Sentinel Commander β Technical Documentation (v2026.09.016)¶
Intended for system architects, L3 support, and infrastructure administrators. Covers internal mechanisms, daemon stability, memory management, and OS integration.
System Requirements¶
- Python 3.13+
- OS: Debian 11+, Ubuntu 22.04+, RHEL 8/9/10, Rocky Linux, AlmaLinux, Raspberry Pi OS
- Key libraries: Flask, Flask-SocketIO, ChromaDB, paho-mqtt, pyotp, qrcode + pillow, bcrypt, jsonschema
- Optional: Hailo AI HAT 2+ with hailo-ollama 5.3.0 for NPU inference
SQLite Hot-Patch (RHEL/Rocky)¶
ChromaDB requires SQLite β₯ 3.35.0. On older RHEL systems, __main__.py dynamically replaces the sqlite3 import with pysqlite3-binary before any other import runs β prevents segfaults without OS-level recompilation.
Filesystem Layout¶
/opt/Sentinel/
βββ sentinel/
β βββ __main__.py β entry point, systemd watchdog, faulthandler, LDAP init
β βββ chat_service.py β Flask app factory, SocketIO, RBAC
β βββ auth.py β authentication, LDAP, 2FA/TOTP, bcrypt, sessions
β βββ state.py β SQLite WAL orchestration
β βββ state_base.py β DB connection, migrations, composite indexes
β βββ state_agents.py β agent registry, watchdog, command allowlist
β βββ state_issues.py β issue CRUD, tagging, workflow, TOTP store, thresholds
β βββ watcher.py β inotify, config hot-reload, LDAP re-init, FIM
β βββ plugin_manager.py β dynamic plugin loading, hot-reload
β βββ ollama_service.py β AI worker pool, model switching, llm_semaphore
β βββ rag.py β ChromaDB, nomic-embed-text, BM25 fallback
β βββ actions.py β Autofix lifecycle, SSH exec, ProxyJump
β βββ notifier.py β 13 outbound channels, retry queue, per-severity throttle
β βββ issue_lifecycle.py β issue auto-close: age cap, reconciliation, auto-recheck
β βββ ssh_utils.py β build_ssh_cmd(), host-key scanning (accept-new)
β βββ analytics.py β TTC, Mann-Kendall, Z-Score, health score, forecast
β βββ topology.py β agent topology, SNMP CDP/LLDP
β βββ snmp_trap.py β SNMP trap receiver
β βββ syslog_receiver.py β Syslog UDP/TCP receiver
β βββ safety.py β command classifier, allowed_commands
β βββ utils.py β rate-limit, IP ban, JSON logging, int_param()
β βββ ai_guard.py β prompt-injection defence, action cap, loop detection
β βββ ai_verify.py β hallucination check against known infra
β βββ ai_profiles.py β context-window profiles per task type
β βββ ai_runtime.py β response cache, consistency, token budget, routing
β βββ diagnostics.py β fixed read-only command catalog (AI picks IDs, not shell)
β βββ fix_verify.py β post-fix verification; failures as anti-patterns
β βββ remediation.py β graduated ladder: observeβreloadβrestartβreboot
β βββ remediation_plan.py β rollback, risk, dry-run, work queue
β βββ policy.py β block explanation, allowlist/auto-execute proposals
β βββ escalation.py β escalation with prior-attempt context
β βββ correlate.py β change correlation, causal chains
β βββ incident_analysis.pyβ timeline, cascades, ranked hypotheses
β βββ trend_detect.py β silent degradation, missing signals
β βββ baseline.py β per-host normal profile, seasonality
β βββ alert_quality.py β false-alarm mining
β βββ playbooks.py β procedures from manual fixes
β βββ foresight.py β capacity forecast, weekly outlook
β βββ unmatched.py β sampling of uncaught log lines
β βββ rag_utils.py β hybrid search, citations, chunking
β βββ knowledge.py β runbooks, prevention hints, KB transfer
β βββ infra_audit.py β config drift, zombies, certs, post-reboot check
β βββ dependencies.py β host dependency graph, blast-radius, shutdown sim
β βββ routes/ β Flask Blueprints
β β βββ main.py β login (2FA flow), dashboard
β β βββ issues.py
β β βββ agents.py
β β βββ actions.py
β β βββ system.py
β β βββ export.py
β β βββ integrations.py β outbound config + inbound webhooks (Grafana/AM/Zabbix)
β β βββ chat.py
β βββ plugins/ β detector modules
β βββ static/ β CSS/JS (gzip, ETag+304, cache 1 yr, defer; .min.js built at deploy)
β βββ templates/ β Jinja2, i18n strings
βββ build_kb.py β RAG knowledge base builder (interactive menu, PDF/DOCX/MD)
βββ hailo_models.py β Hailo TUI model manager
βββ sentinel_init.py β interactive installation wizard (refuses default passwords)
βββ config.yaml.example
βββ Makefile β make lint / test / build / ci
βββ .gitea/workflows/ci.yml β CI pipeline (pytest + node --check + build)
βββ tests/ β 1087 tests (route, security, integration, AI layer, issue lifecycle, benchmark)
/etc/sentinel/
βββ config.yaml β hot-reloadable configuration (jsonschema-validated)
/var/lib/sentinel/
βββ secret_key β persistent Flask SECRET_KEY
βββ client_api_key β auto-generated client token (mode 600)
βββ config_backups/ β automatic pre-restore snapshots (10 kept)
/var/log/sentinel/
βββ logs/
β βββ sentinel_state.db β SQLite WAL (issues, telemetry, agents, β¦)
β βββ sentinel.log
βββ chroma_db/ β ChromaDB vector store
Daemon Stability β Self-Watchdog¶
An internal watchdog in __main__.py polls GET localhost:5050/api/health every cycle. Since v2026.06.008 the HTTP check runs in a separate thread over a persistent socket, and the DB uses busy_timeout β this removed spurious SIGABRT watchdog kills under load.
GET /api/health
β
βββ 200 OK β reset web_failures=0, systemd WATCHDOG=1
β
βββ Timeout/Error β web_failures++
β
βββ failures == 5 β log RAM/CPU diagnostics
βββ failures == 9 β dump all thread stacks
βββ failures >= 10 β self-restart + Teams alert
Systemd integration: unit is generated with WatchdogSec=900 and a WAL-checkpoint ExecStartPre. If WATCHDOG=1 is not sent in time, systemd issues SIGKILL and restarts the daemon.
Diagnostics: faulthandler.enable() dumps all thread tracebacks to the journal on watchdog abort (works even with a held GIL).
Graceful shutdown: SIGTERM flushes the telemetry write buffer and publishes MQTT sentinel/status: offline before exit.
Self-monitoring: a memory watchdog thread warns when RSS > 1.5 GB; self-metrics (RAM, threads, queue depth, issues, agents online, load1) are written to telemetry every minute; AI queue backlog > 50 raises a SENTINEL_SELF_HEALTH issue; startup phase durations are profiled and logged.
Config Hot-Reload¶
watcher.py monitors /etc/sentinel/config.yaml via inotify. On IN_MODIFY:
time.sleep(1.0)β waits for the file to be fully written (inotify fires on first byte, not on close)load_config()β reloads all config sections (with{SECRET:ENV_VAR}substitution)plugin_manager.load_plugins()β reloads detector modules_reinit_ldap()β re-initialises the LDAP manager (required for LDAP login to work after config change)
SIGHUP triggers the same full reload β config plus watcher patterns plus plugins (DETECTORS, LOG_GROUPS). The runtime log level can be changed without any reload via /api/admin/log_level.
Schema validation: critical keys (web.port, worker_threads, db_retention_days, β¦) are validated with jsonschema; an invalid config is rejected instead of half-applied.
Cluster detection (v2026.08.006): host_cluster_rules in config.yaml maps hostnames to clusters via substring patterns. api.get_cluster_from_host() is the shared lookup function used by all plugins (detector_icinga, detector_who, etc.). Priority: explicit cluster field in issue β [CLUSTERNAME] prefix in last_line β config rules β default_cluster (defaults to INFRA). Plugin-specific params: detector_icinga supports ignored_host_services (list of service_pattern@host_pattern) for false-positive suppression; detector_cluster supports slurmctld_threshold (consecutive DOWN pings before reporting).
SQLite WAL β Database Architecture¶
All state is stored in /var/log/sentinel/logs/sentinel_state.db in WAL (Write-Ahead Logging) mode. Key tables:
| Table | Purpose |
|---|---|
problems | Active incidents β key, status, severity, channel, host, last_line, missing_count |
issue_history | Archived incidents incl. first_seen β enables resolution-time analytics; 90-day retention |
actions | Autofix lifecycle β command, status, risk_score, dry_run_output, mode, actor |
action_audit | Append-only compliance log β every lifecycle event with actor and timestamp |
config_audit | Config change audit β who, when, IP, which keys |
task_queue | AI worker queue β status, worker_id (single-consumer guarantee) |
user_audit | Every login, local and LDAP β auth_source, first_seen, last_login, last_ip, login_count, online_seconds. Distinct from user_roles, which only holds users whose role someone changed |
telemetry | TSDB β metric, value, timestamp (composite indexes, batched writes) |
agents | Agent registry β hostname, token, last_seen, status, ip_addresses, version |
active_sessions | Session tracking β user, ip, created_at, last_seen, revokable |
revoked_sessions | Revocations that survive restart |
user_totp | 2FA/TOTP secrets per user |
api_keys | REST API keys β SHA-256 hash, fine-grained scopes |
issue_tags | Tag β issue_key mapping |
suppress_rules | False positive patterns for auto-suppression (with hit_count) |
snooze_rules | Maintenance windows β global or per-host (hosts CSV column) |
custom_patterns | User-defined regex detection patterns (Pattern Editor) |
ssh_execute_log | Audit log of all SSH-executed commands |
root_audit | Root session log β server, ip (with reverse DNS), tty, connected_at, is_active, last_seen, origin. connected_at holds the actual login time parsed from the agent's report, not the time of the poll that observed it |
agent_thresholds | Per-agent alert thresholds (enforced on ingest) |
comment_templates | Saved comment templates for issue annotations |
WAL tuning: wal_autocheckpoint=200, PRAGMA synchronous=NORMAL, explicit checkpoint after telemetry prune. The DB lives outside any inotify-watched directory (a watched DB file caused HPC instability).
Garbage collection (maintenance loop in chat_service.py): batched deletion (LIMIT 1000) to avoid WAL lock contention; prune_issue_history(days=90); VACUUM after > 10 000 deleted rows. Default retention: telemetry 2 days, resolved issues 2 days, issue history 90 days. Additionally, issue_lifecycle.py runs periodic auto-recheck, age-based event issue expiry, and detector reconciliation.
Caching: get_active_issues() has a 5 s TTL in-memory cache guarded by a double-checked threading.Lock (fast lock-free read path, locked rebuild); dashboard sparkline queries cache for 5 min.
Agent watchdog: background thread flips agent status to OFFLINE if no heartbeat within agent_heartbeat_timeout seconds (per-agent or global fallback); offline duration is recorded to telemetry on reconnect.
Connection Lifetime β Why the Keeper Exists¶
sqlite3.Connection.close() does not release the GIL, and closing the last connection to a WAL database makes SQLite run a full checkpoint. With a connection-per-request pattern every close was the last one, so each request froze the whole process for the duration of a checkpoint β long enough on a 111 MB database to starve the web thread out of accept(), the systemd watchdog out of its ping, and the signal handler out of running at all.
state_base._ensure_keeper() therefore opens one connection at startup and never closes it. Two details matter:
sqlite3.connect()alone does not touch the file β SQLite opens it lazily on first use. The keeper issues a real query (SELECT 1 FROM sqlite_master LIMIT 1), otherwise no-wal/-shmappears and the keeper holds nothing. A regression test asserts the WAL session exists after_ensure_keeper().- Checkpointing still happens through
wal_autocheckpointinsideexecute(), which does release the GIL, so the WAL stays bounded without freezing anything.
Presence of sentinel_state.db-wal and -shm while the service runs is the quick way to confirm the keeper is doing its job.
Root Session Reconciliation¶
Two writers feed root_audit and they store different things in server: a detector writes the cluster name (CLUSTER-A), an on-host agent writes the hostname (hpc-node-04). The origin column (detector / agent) distinguishes them; for detector records the cluster is the server value, so no pattern matching is involved.
The agent path used to close every active session for a host and re-insert them on every report β one host produced 126 rows in 36 minutes and the table reached 105 000 rows, which also meant connected_at was reset each poll and "session duration" measured time since the last report. Reports carry full identity:
so sessions are now reconciled on (server, tty, login time): an ongoing session is confirmed via last_seen, a new one inserted, an unreported one closed. Two concurrent sessions from the same IP stay separate β hence the tty column β and re-login on the same pts is a new session because the login time differs.
Sessions a detector simply stops reporting are closed by a per-minute sweep (root_audit_stale_minutes, default 30). This is needed because add_root_audit() is idempotent: an ongoing session is confirmed silently, so without last_seen there is no way to tell it from an abandoned record β and the daily pruning only deletes is_active=0, so stale active rows survived indefinitely (records 67 days old were still displayed as live).
Detector Self-Check¶
load_plugins() compares the enabled detectors in config.yaml against those actually loaded and raises SENTINEL|detectors_missing (channel infra, severity critical) for the difference; the issue clears itself once they load again.
This exists because the failure is silent by construction. A running process keeps imported modules in memory, so detectors whose source files are deleted keep working until the next restart. After that restart nothing reads their logs, the stale cleanup resolves their issues within the hour, and the instance looks healthy while producing nothing. The check turns two days of false calm into one issue.
The self-check cannot abort startup β it is wrapped β but a failure inside it is logged rather than swallowed.
Login Audit¶
user_audit is written from _complete_login(), the single point every login passes through, so an LDAP user who was never assigned a role still gets a record. user_roles alone could not serve this: it only holds users whose role someone changed.
Accumulated online time is computed in session_remove() before the row is deleted β the duration lives in created_at/last_seen of active_sessions and would disappear with it. Implausible values (negative, longer than a week) are discarded so one broken session cannot skew the total.
Security Layer¶
| Mechanism | Implementation |
|---|---|
| 2FA / TOTP | pyotp (RFC 6238), user_totp table, two-step login flow, QR enrollment (qrcode + pillow) |
| Password hashing | bcrypt ($2b$ prefix detection in _check_password()); web.password_hash preferred over plaintext |
| CSRF | Session + SameSite=Strict cookie token; global fetch wrapper adds X-CSRF-Token to POST/PUT/DELETE |
| Brute-force | Form login + Basic auth: 5 failures β 300 s IP ban; auto-register 10/min per IP; /api/analyze/* 10 req/min |
| XSS | html.escape() on all AI replies before they reach innerHTML (log content is attacker-controlled) |
| SSH | ssh_utils.build_ssh_cmd() β accept-new + pinned UserKnownHostsFile; ssh-keyscan on registration; hostname regex validation; command allowlist pre-validation |
| API key verify | Timing-safe hmac.compare_digest(sha256(submitted), stored_hash); scopes read:issues / write:actions / admin:users |
| Sessions | Absolute 12 h timeout, role refresh from DB every 5 min, revocations persisted in DB |
| Secrets | {SECRET:ENV_VAR} config substitution (env vars wiped after use); /api/config/view masks secrets as ***; persistent SECRET_KEY in /var/lib/sentinel/secret_key |
| Webhooks | HMAC X-Hub-Signature-256 + replay protection via X-Webhook-Timestamp |
| SSRF | /api/admin/validate_url rejects private IP ranges |
| Input validation | int_param(value, default, min, max) on 20+ endpoints β no negative values reaching SQL datetime() |
| Reverse proxy | get_real_ip() honours X-Forwarded-For only from TRUSTED_PROXIES |
| Audit | config_audit, ssh_execute_log, action_audit, 403/401 access audit; viewer UI in Settings |
| FIM | SHA-256 of critical files checked every minute β security issue with severity high on change. Implemented in v2026.09.004 β before that the config existed but nothing called the check |
| Symlink containment | os.path.realpath() check on all file path inputs |
| Upload limits | 5 MB max, secure_filename(), extension allowlist |
| CSP headers | Content-Security-Policy on all responses |
| LDAP | Fallback to direct ldap3 bind if manager unavailable |
| Self-check | /api/admin/security_check β grade A/B/C/D |
Performance Optimisations¶
- Static file caching: ETag +
max-age+ 304 conditional GET;Cache-Control: max-age=31536000, immutablewith?v=<subversion>fingerprinting - Script loading: all external
<script src>tags usedefer;<link rel="preload">for critical CSS/JS - Gzip compression: Flask-Compress level 6, min 2 KB β ~75 % transfer reduction
- Socket.IO transport:
['polling', 'websocket']withupgrade: trueβ starts over HTTP long-polling (survives all proxies), upgrades to WebSocket when available - Backpressure: frontend SocketIO queue bounded at 500 with drop-oldest; duplicate WS messages deduplicated in a 1 s window
- Telemetry batching:
save_telemetry_snapshot()writes through a buffer, flushed periodically and on SIGTERM - DB indexes:
idx_problems_plugin_ch_ts,idx_issue_hist_plugin_ts,idx_problems_severity,idx_telemetry_cat_metric_tsβ created automatically on startup - AI semaphore: serialisation handled exclusively inside
execute_ollama()(an outer re-entrant acquire previously caused a permanent deadlock); Ollama HTTP timeout bounded at 90 s - Hot read path: no writes from
get_pending_actions()(expiry moved to a background loop) β this removed "UI offline" stalls under agent ingest load - Bulk ingest:
/api/v1/ingest/bulkaccepts an array of alerts in one HTTP request
CI & Test Suite¶
- 1087 tests: Flask route tests, security tests (brute force, API scopes, hostname injection, secrets masking), integration tests (full issue lifecycle on a real DB), issue lifecycle auto-close tests, dashboard performance benchmark
- CI: Gitea Actions (
.gitea/workflows/ci.yml) β pytest +node --check+make build; localpre-pushgit hook - Lint: ruff (
pyproject.toml), ESLint withno-redeclare=error;make ciruns lint + tests - Build artefacts:
.min.jsfiles are built at deploy time and excluded from git
Hailo AI HAT 2+ Integration¶
When hailo_ollama.enabled: true in config:
- Inference requests route to hailo-ollama at
http://localhost:8000(NPU) - CPU Ollama at
:11434used for embeddings only (nomic-embed-text) - Runtime model switch via
hailo_models.pyTUI without daemon restart hailo_models.pyβ 619-line Unicode TUI: htop-style CPU/Mem bars, RX/TX throughput, NPU architecture + firmware, TPS benchmark graph