Skip to content

Sentinel Commander β€” Technical Documentation (v2026.09.016)

Intended for system architects, L3 support, and infrastructure administrators. Covers internal mechanisms, daemon stability, memory management, and OS integration.


System Requirements

  • Python 3.13+
  • OS: Debian 11+, Ubuntu 22.04+, RHEL 8/9/10, Rocky Linux, AlmaLinux, Raspberry Pi OS
  • Key libraries: Flask, Flask-SocketIO, ChromaDB, paho-mqtt, pyotp, qrcode + pillow, bcrypt, jsonschema
  • Optional: Hailo AI HAT 2+ with hailo-ollama 5.3.0 for NPU inference

SQLite Hot-Patch (RHEL/Rocky)

ChromaDB requires SQLite β‰₯ 3.35.0. On older RHEL systems, __main__.py dynamically replaces the sqlite3 import with pysqlite3-binary before any other import runs β€” prevents segfaults without OS-level recompilation.


Filesystem Layout

Text Only
/opt/Sentinel/
β”œβ”€β”€ sentinel/
β”‚   β”œβ”€β”€ __main__.py         ← entry point, systemd watchdog, faulthandler, LDAP init
β”‚   β”œβ”€β”€ chat_service.py     ← Flask app factory, SocketIO, RBAC
β”‚   β”œβ”€β”€ auth.py             ← authentication, LDAP, 2FA/TOTP, bcrypt, sessions
β”‚   β”œβ”€β”€ state.py            ← SQLite WAL orchestration
β”‚   β”œβ”€β”€ state_base.py       ← DB connection, migrations, composite indexes
β”‚   β”œβ”€β”€ state_agents.py     ← agent registry, watchdog, command allowlist
β”‚   β”œβ”€β”€ state_issues.py     ← issue CRUD, tagging, workflow, TOTP store, thresholds
β”‚   β”œβ”€β”€ watcher.py          ← inotify, config hot-reload, LDAP re-init, FIM
β”‚   β”œβ”€β”€ plugin_manager.py   ← dynamic plugin loading, hot-reload
β”‚   β”œβ”€β”€ ollama_service.py   ← AI worker pool, model switching, llm_semaphore
β”‚   β”œβ”€β”€ rag.py              ← ChromaDB, nomic-embed-text, BM25 fallback
β”‚   β”œβ”€β”€ actions.py          ← Autofix lifecycle, SSH exec, ProxyJump
β”‚   β”œβ”€β”€ notifier.py         ← 13 outbound channels, retry queue, per-severity throttle
β”‚   β”œβ”€β”€ issue_lifecycle.py   ← issue auto-close: age cap, reconciliation, auto-recheck
β”‚   β”œβ”€β”€ ssh_utils.py        ← build_ssh_cmd(), host-key scanning (accept-new)
β”‚   β”œβ”€β”€ analytics.py        ← TTC, Mann-Kendall, Z-Score, health score, forecast
β”‚   β”œβ”€β”€ topology.py         ← agent topology, SNMP CDP/LLDP
β”‚   β”œβ”€β”€ snmp_trap.py        ← SNMP trap receiver
β”‚   β”œβ”€β”€ syslog_receiver.py  ← Syslog UDP/TCP receiver
β”‚   β”œβ”€β”€ safety.py           ← command classifier, allowed_commands
β”‚   β”œβ”€β”€ utils.py            ← rate-limit, IP ban, JSON logging, int_param()
β”‚   β”œβ”€β”€ ai_guard.py         ← prompt-injection defence, action cap, loop detection
β”‚   β”œβ”€β”€ ai_verify.py        ← hallucination check against known infra
β”‚   β”œβ”€β”€ ai_profiles.py      ← context-window profiles per task type
β”‚   β”œβ”€β”€ ai_runtime.py       ← response cache, consistency, token budget, routing
β”‚   β”œβ”€β”€ diagnostics.py      ← fixed read-only command catalog (AI picks IDs, not shell)
β”‚   β”œβ”€β”€ fix_verify.py       ← post-fix verification; failures as anti-patterns
β”‚   β”œβ”€β”€ remediation.py      ← graduated ladder: observeβ†’reloadβ†’restartβ†’reboot
β”‚   β”œβ”€β”€ remediation_plan.py ← rollback, risk, dry-run, work queue
β”‚   β”œβ”€β”€ policy.py           ← block explanation, allowlist/auto-execute proposals
β”‚   β”œβ”€β”€ escalation.py       ← escalation with prior-attempt context
β”‚   β”œβ”€β”€ correlate.py        ← change correlation, causal chains
β”‚   β”œβ”€β”€ incident_analysis.py← timeline, cascades, ranked hypotheses
β”‚   β”œβ”€β”€ trend_detect.py     ← silent degradation, missing signals
β”‚   β”œβ”€β”€ baseline.py         ← per-host normal profile, seasonality
β”‚   β”œβ”€β”€ alert_quality.py    ← false-alarm mining
β”‚   β”œβ”€β”€ playbooks.py        ← procedures from manual fixes
β”‚   β”œβ”€β”€ foresight.py        ← capacity forecast, weekly outlook
β”‚   β”œβ”€β”€ unmatched.py        ← sampling of uncaught log lines
β”‚   β”œβ”€β”€ rag_utils.py        ← hybrid search, citations, chunking
β”‚   β”œβ”€β”€ knowledge.py        ← runbooks, prevention hints, KB transfer
β”‚   β”œβ”€β”€ infra_audit.py      ← config drift, zombies, certs, post-reboot check
β”‚   β”œβ”€β”€ dependencies.py     ← host dependency graph, blast-radius, shutdown sim
β”‚   β”œβ”€β”€ routes/             ← Flask Blueprints
β”‚   β”‚   β”œβ”€β”€ main.py         ← login (2FA flow), dashboard
β”‚   β”‚   β”œβ”€β”€ issues.py
β”‚   β”‚   β”œβ”€β”€ agents.py
β”‚   β”‚   β”œβ”€β”€ actions.py
β”‚   β”‚   β”œβ”€β”€ system.py
β”‚   β”‚   β”œβ”€β”€ export.py
β”‚   β”‚   β”œβ”€β”€ integrations.py ← outbound config + inbound webhooks (Grafana/AM/Zabbix)
β”‚   β”‚   └── chat.py
β”‚   β”œβ”€β”€ plugins/            ← detector modules
β”‚   β”œβ”€β”€ static/             ← CSS/JS (gzip, ETag+304, cache 1 yr, defer; .min.js built at deploy)
β”‚   └── templates/          ← Jinja2, i18n strings
β”œβ”€β”€ build_kb.py             ← RAG knowledge base builder (interactive menu, PDF/DOCX/MD)
β”œβ”€β”€ hailo_models.py         ← Hailo TUI model manager
β”œβ”€β”€ sentinel_init.py        ← interactive installation wizard (refuses default passwords)
β”œβ”€β”€ config.yaml.example
β”œβ”€β”€ Makefile                ← make lint / test / build / ci
β”œβ”€β”€ .gitea/workflows/ci.yml ← CI pipeline (pytest + node --check + build)
└── tests/                  ← 1087 tests (route, security, integration, AI layer, issue lifecycle, benchmark)

/etc/sentinel/
└── config.yaml             ← hot-reloadable configuration (jsonschema-validated)

/var/lib/sentinel/
β”œβ”€β”€ secret_key              ← persistent Flask SECRET_KEY
β”œβ”€β”€ client_api_key          ← auto-generated client token (mode 600)
└── config_backups/         ← automatic pre-restore snapshots (10 kept)

/var/log/sentinel/
β”œβ”€β”€ logs/
β”‚   β”œβ”€β”€ sentinel_state.db   ← SQLite WAL (issues, telemetry, agents, …)
β”‚   └── sentinel.log
└── chroma_db/              ← ChromaDB vector store

Daemon Stability β€” Self-Watchdog

An internal watchdog in __main__.py polls GET localhost:5050/api/health every cycle. Since v2026.06.008 the HTTP check runs in a separate thread over a persistent socket, and the DB uses busy_timeout β€” this removed spurious SIGABRT watchdog kills under load.

Text Only
GET /api/health
  β”‚
  β”œβ”€β”€ 200 OK β†’ reset web_failures=0, systemd WATCHDOG=1
  β”‚
  └── Timeout/Error β†’ web_failures++
        β”‚
        β”œβ”€β”€ failures == 5  β†’ log RAM/CPU diagnostics
        β”œβ”€β”€ failures == 9  β†’ dump all thread stacks
        └── failures >= 10 β†’ self-restart + Teams alert

Systemd integration: unit is generated with WatchdogSec=900 and a WAL-checkpoint ExecStartPre. If WATCHDOG=1 is not sent in time, systemd issues SIGKILL and restarts the daemon.

Diagnostics: faulthandler.enable() dumps all thread tracebacks to the journal on watchdog abort (works even with a held GIL).

Graceful shutdown: SIGTERM flushes the telemetry write buffer and publishes MQTT sentinel/status: offline before exit.

Self-monitoring: a memory watchdog thread warns when RSS > 1.5 GB; self-metrics (RAM, threads, queue depth, issues, agents online, load1) are written to telemetry every minute; AI queue backlog > 50 raises a SENTINEL_SELF_HEALTH issue; startup phase durations are profiled and logged.


Config Hot-Reload

watcher.py monitors /etc/sentinel/config.yaml via inotify. On IN_MODIFY:

  1. time.sleep(1.0) β€” waits for the file to be fully written (inotify fires on first byte, not on close)
  2. load_config() β€” reloads all config sections (with {SECRET:ENV_VAR} substitution)
  3. plugin_manager.load_plugins() β€” reloads detector modules
  4. _reinit_ldap() β€” re-initialises the LDAP manager (required for LDAP login to work after config change)

SIGHUP triggers the same full reload β€” config plus watcher patterns plus plugins (DETECTORS, LOG_GROUPS). The runtime log level can be changed without any reload via /api/admin/log_level.

Schema validation: critical keys (web.port, worker_threads, db_retention_days, …) are validated with jsonschema; an invalid config is rejected instead of half-applied.

Cluster detection (v2026.08.006): host_cluster_rules in config.yaml maps hostnames to clusters via substring patterns. api.get_cluster_from_host() is the shared lookup function used by all plugins (detector_icinga, detector_who, etc.). Priority: explicit cluster field in issue β†’ [CLUSTERNAME] prefix in last_line β†’ config rules β†’ default_cluster (defaults to INFRA). Plugin-specific params: detector_icinga supports ignored_host_services (list of service_pattern@host_pattern) for false-positive suppression; detector_cluster supports slurmctld_threshold (consecutive DOWN pings before reporting).


SQLite WAL β€” Database Architecture

All state is stored in /var/log/sentinel/logs/sentinel_state.db in WAL (Write-Ahead Logging) mode. Key tables:

Table Purpose
problems Active incidents β€” key, status, severity, channel, host, last_line, missing_count
issue_history Archived incidents incl. first_seen β†’ enables resolution-time analytics; 90-day retention
actions Autofix lifecycle β€” command, status, risk_score, dry_run_output, mode, actor
action_audit Append-only compliance log β€” every lifecycle event with actor and timestamp
config_audit Config change audit β€” who, when, IP, which keys
task_queue AI worker queue β€” status, worker_id (single-consumer guarantee)
user_audit Every login, local and LDAP β€” auth_source, first_seen, last_login, last_ip, login_count, online_seconds. Distinct from user_roles, which only holds users whose role someone changed
telemetry TSDB β€” metric, value, timestamp (composite indexes, batched writes)
agents Agent registry β€” hostname, token, last_seen, status, ip_addresses, version
active_sessions Session tracking β€” user, ip, created_at, last_seen, revokable
revoked_sessions Revocations that survive restart
user_totp 2FA/TOTP secrets per user
api_keys REST API keys β€” SHA-256 hash, fine-grained scopes
issue_tags Tag β†’ issue_key mapping
suppress_rules False positive patterns for auto-suppression (with hit_count)
snooze_rules Maintenance windows β€” global or per-host (hosts CSV column)
custom_patterns User-defined regex detection patterns (Pattern Editor)
ssh_execute_log Audit log of all SSH-executed commands
root_audit Root session log β€” server, ip (with reverse DNS), tty, connected_at, is_active, last_seen, origin. connected_at holds the actual login time parsed from the agent's report, not the time of the poll that observed it
agent_thresholds Per-agent alert thresholds (enforced on ingest)
comment_templates Saved comment templates for issue annotations

WAL tuning: wal_autocheckpoint=200, PRAGMA synchronous=NORMAL, explicit checkpoint after telemetry prune. The DB lives outside any inotify-watched directory (a watched DB file caused HPC instability).

Garbage collection (maintenance loop in chat_service.py): batched deletion (LIMIT 1000) to avoid WAL lock contention; prune_issue_history(days=90); VACUUM after > 10 000 deleted rows. Default retention: telemetry 2 days, resolved issues 2 days, issue history 90 days. Additionally, issue_lifecycle.py runs periodic auto-recheck, age-based event issue expiry, and detector reconciliation.

Caching: get_active_issues() has a 5 s TTL in-memory cache guarded by a double-checked threading.Lock (fast lock-free read path, locked rebuild); dashboard sparkline queries cache for 5 min.

Agent watchdog: background thread flips agent status to OFFLINE if no heartbeat within agent_heartbeat_timeout seconds (per-agent or global fallback); offline duration is recorded to telemetry on reconnect.


Connection Lifetime β€” Why the Keeper Exists

sqlite3.Connection.close() does not release the GIL, and closing the last connection to a WAL database makes SQLite run a full checkpoint. With a connection-per-request pattern every close was the last one, so each request froze the whole process for the duration of a checkpoint β€” long enough on a 111 MB database to starve the web thread out of accept(), the systemd watchdog out of its ping, and the signal handler out of running at all.

state_base._ensure_keeper() therefore opens one connection at startup and never closes it. Two details matter:

  • sqlite3.connect() alone does not touch the file β€” SQLite opens it lazily on first use. The keeper issues a real query (SELECT 1 FROM sqlite_master LIMIT 1), otherwise no -wal/-shm appears and the keeper holds nothing. A regression test asserts the WAL session exists after _ensure_keeper().
  • Checkpointing still happens through wal_autocheckpoint inside execute(), which does release the GIL, so the WAL stays bounded without freezing anything.

Presence of sentinel_state.db-wal and -shm while the service runs is the quick way to confirm the keeper is doing its job.

Root Session Reconciliation

Two writers feed root_audit and they store different things in server: a detector writes the cluster name (CLUSTER-A), an on-host agent writes the hostname (hpc-node-04). The origin column (detector / agent) distinguishes them; for detector records the cluster is the server value, so no pattern matching is involved.

The agent path used to close every active session for a host and re-insert them on every report β€” one host produced 126 rows in 36 minutes and the table reached 105 000 rows, which also meant connected_at was reset each poll and "session duration" measured time since the last report. Reports carry full identity:

Text Only
🟒 [ACTIVE] pts/0 from 10.0.0.4 (since 2026-08-20 11:47)

so sessions are now reconciled on (server, tty, login time): an ongoing session is confirmed via last_seen, a new one inserted, an unreported one closed. Two concurrent sessions from the same IP stay separate β€” hence the tty column β€” and re-login on the same pts is a new session because the login time differs.

Sessions a detector simply stops reporting are closed by a per-minute sweep (root_audit_stale_minutes, default 30). This is needed because add_root_audit() is idempotent: an ongoing session is confirmed silently, so without last_seen there is no way to tell it from an abandoned record β€” and the daily pruning only deletes is_active=0, so stale active rows survived indefinitely (records 67 days old were still displayed as live).

Detector Self-Check

load_plugins() compares the enabled detectors in config.yaml against those actually loaded and raises SENTINEL|detectors_missing (channel infra, severity critical) for the difference; the issue clears itself once they load again.

This exists because the failure is silent by construction. A running process keeps imported modules in memory, so detectors whose source files are deleted keep working until the next restart. After that restart nothing reads their logs, the stale cleanup resolves their issues within the hour, and the instance looks healthy while producing nothing. The check turns two days of false calm into one issue.

The self-check cannot abort startup β€” it is wrapped β€” but a failure inside it is logged rather than swallowed.

Login Audit

user_audit is written from _complete_login(), the single point every login passes through, so an LDAP user who was never assigned a role still gets a record. user_roles alone could not serve this: it only holds users whose role someone changed.

Accumulated online time is computed in session_remove() before the row is deleted β€” the duration lives in created_at/last_seen of active_sessions and would disappear with it. Implausible values (negative, longer than a week) are discarded so one broken session cannot skew the total.

Security Layer

Mechanism Implementation
2FA / TOTP pyotp (RFC 6238), user_totp table, two-step login flow, QR enrollment (qrcode + pillow)
Password hashing bcrypt ($2b$ prefix detection in _check_password()); web.password_hash preferred over plaintext
CSRF Session + SameSite=Strict cookie token; global fetch wrapper adds X-CSRF-Token to POST/PUT/DELETE
Brute-force Form login + Basic auth: 5 failures β†’ 300 s IP ban; auto-register 10/min per IP; /api/analyze/* 10 req/min
XSS html.escape() on all AI replies before they reach innerHTML (log content is attacker-controlled)
SSH ssh_utils.build_ssh_cmd() β€” accept-new + pinned UserKnownHostsFile; ssh-keyscan on registration; hostname regex validation; command allowlist pre-validation
API key verify Timing-safe hmac.compare_digest(sha256(submitted), stored_hash); scopes read:issues / write:actions / admin:users
Sessions Absolute 12 h timeout, role refresh from DB every 5 min, revocations persisted in DB
Secrets {SECRET:ENV_VAR} config substitution (env vars wiped after use); /api/config/view masks secrets as ***; persistent SECRET_KEY in /var/lib/sentinel/secret_key
Webhooks HMAC X-Hub-Signature-256 + replay protection via X-Webhook-Timestamp
SSRF /api/admin/validate_url rejects private IP ranges
Input validation int_param(value, default, min, max) on 20+ endpoints β€” no negative values reaching SQL datetime()
Reverse proxy get_real_ip() honours X-Forwarded-For only from TRUSTED_PROXIES
Audit config_audit, ssh_execute_log, action_audit, 403/401 access audit; viewer UI in Settings
FIM SHA-256 of critical files checked every minute β†’ security issue with severity high on change. Implemented in v2026.09.004 β€” before that the config existed but nothing called the check
Symlink containment os.path.realpath() check on all file path inputs
Upload limits 5 MB max, secure_filename(), extension allowlist
CSP headers Content-Security-Policy on all responses
LDAP Fallback to direct ldap3 bind if manager unavailable
Self-check /api/admin/security_check β†’ grade A/B/C/D

Performance Optimisations

  • Static file caching: ETag + max-age + 304 conditional GET; Cache-Control: max-age=31536000, immutable with ?v=<subversion> fingerprinting
  • Script loading: all external <script src> tags use defer; <link rel="preload"> for critical CSS/JS
  • Gzip compression: Flask-Compress level 6, min 2 KB β€” ~75 % transfer reduction
  • Socket.IO transport: ['polling', 'websocket'] with upgrade: true β€” starts over HTTP long-polling (survives all proxies), upgrades to WebSocket when available
  • Backpressure: frontend SocketIO queue bounded at 500 with drop-oldest; duplicate WS messages deduplicated in a 1 s window
  • Telemetry batching: save_telemetry_snapshot() writes through a buffer, flushed periodically and on SIGTERM
  • DB indexes: idx_problems_plugin_ch_ts, idx_issue_hist_plugin_ts, idx_problems_severity, idx_telemetry_cat_metric_ts β€” created automatically on startup
  • AI semaphore: serialisation handled exclusively inside execute_ollama() (an outer re-entrant acquire previously caused a permanent deadlock); Ollama HTTP timeout bounded at 90 s
  • Hot read path: no writes from get_pending_actions() (expiry moved to a background loop) β€” this removed "UI offline" stalls under agent ingest load
  • Bulk ingest: /api/v1/ingest/bulk accepts an array of alerts in one HTTP request

CI & Test Suite

  • 1087 tests: Flask route tests, security tests (brute force, API scopes, hostname injection, secrets masking), integration tests (full issue lifecycle on a real DB), issue lifecycle auto-close tests, dashboard performance benchmark
  • CI: Gitea Actions (.gitea/workflows/ci.yml) β€” pytest + node --check + make build; local pre-push git hook
  • Lint: ruff (pyproject.toml), ESLint with no-redeclare=error; make ci runs lint + tests
  • Build artefacts: .min.js files are built at deploy time and excluded from git

Hailo AI HAT 2+ Integration

When hailo_ollama.enabled: true in config:

  • Inference requests route to hailo-ollama at http://localhost:8000 (NPU)
  • CPU Ollama at :11434 used for embeddings only (nomic-embed-text)
  • Runtime model switch via hailo_models.py TUI without daemon restart
  • hailo_models.py β€” 619-line Unicode TUI: htop-style CPU/Mem bars, RX/TX throughput, NPU architecture + firmware, TPS benchmark graph