Sentinel Commander β Programmer Documentation (v2026.09.016)¶
Intended for developers and DevOps engineers writing custom detection plugins, integrating Sentinel with external tools, or modifying the system core.
Plugin Architecture¶
Sentinel loads plugins dynamically from the plugins/ directory. Every plugin must inherit from BaseDetector.
Log file (disk)
β inotify on_modified / on_moved
βΌ
watcher.py ββlinesβββΆ plugin_manager.py ββmatch(regex)βββΆ MyDetector.process(lines)
β
api.report_problem(key, payload)
β
state.py (SQLite)
Writing a Custom Detector¶
# plugins/my_detector.py
from datetime import datetime, timezone
from sentinel import api
class Detector(api.BaseDetector):
def __init__(self, name, config_params=None):
super().__init__(name, config_params)
self.threshold = self.config_params.get("threshold", 3)
def process(self, lines, file_path):
infra_label = api.get_infrastructure_label(file_path)
for line in lines:
if "MY_CRITICAL_EVENT" not in line:
continue
key = f"MY_EVENT|{infra_label}|{hash(line)}"
api.report_problem(key, {
"status": "active",
"last_line": line.strip(),
"channel_type": "infra",
"severity": "CRITICAL",
"host": infra_label,
"cluster": infra_label,
"log_file": file_path,
"last_seen": datetime.now(timezone.utc).isoformat(),
"missing_count": 0
})
# Optional: queue AI analysis
api.enqueue_ai_task(f"Analyse: {line.strip()}", channel="infra")
Register in config.yaml:
detectors:
my_detector:
enabled: true
file_mask: "myapp.log"
threshold: 5
notify: true # per-detector notification toggle
Core API (api.py)¶
| Method | Description |
|---|---|
report_problem(key, data) | Create or update an incident. Same key = update last_seen + missing_count. |
resolve_problem(key) | Close an incident (auto-heal). Triggers WebSocket resolve event. |
get_problem(key) | Return the current DB row for a key, or None. |
enqueue_ai_task(prompt, channel) | Push a task to the AI worker queue. Non-blocking. |
save_telemetry_snapshot(metric, value, host) | Write a time-series data point (buffered/batched). |
get_infrastructure_label(file_path) | Extract SERVER: <hostname> context from the log line prefix. |
add_root_audit(server, ip) | Idempotent root-session audit entry with reverse DNS. An already-active session has its last_seen refreshed; records are marked origin='detector' (v2026.09.001). |
close_root_audit(server, ip) | Mark matching active records as disconnected. Detectors that do not call it are covered by the age sweep (root_audit_stale_minutes). |
get_cluster_from_host(hostname) | Derive cluster name from hostname using host_cluster_rules config (v2026.08.006). Shared by all plugins. |
notify_teams(message, channel) | Send a Teams webhook message directly (bypass queue). |
Notifier (notifier.py)¶
Outbound notifications were extracted from ChatService into a standalone module:
from sentinel.notifier import send_notification
send_notification(title, message, severity="critical", channel="security")
- Fan-out to all enabled channels: Teams, Slack, PagerDuty, Discord, Telegram, Opsgenie, ntfy.sh, Gotify, SMTP, Matrix, Home Assistant, MQTT, generic Webhook
- Each channel runs through
_with_retry()β failures enter a retry queue (deque maxlen 200) retried by theNotifierRetrythread with 30 s/120 s/300 s backoff (max 3 attempts) - Throttle per severity:
critical/security/root15 min Β·high1 h Β·medium/low/agent/infra4 h - Respects per-detector (
notify:in YAML) and per-channel toggles - Instance name prefixed to all titles:
[Instance] CHANNEL alert
Issue Lifecycle (issue_lifecycle.py)¶
Auto-close mechanisms for issues that no longer apply, integrated into the maintenance loop in chat_service.py:
resolve_aged_event_issues()β archives event-type issues (e.g.SYS_ERR|host) older thanEVENT_ISSUE_MAX_AGE_HOURS(default 48). These issues aggregate events and refreshlast_seenon every new event, so TTL-based cleanup never reaches them.reconcile_detector_issues()β opt-in per detector viaapi.reconcile_issues. Closes issues whose key was not reported in the latest full-state scan. Only enabled for detectors that see the complete state in one run (ha_detector,capacity_detector).auto_recheck_pass()β periodically re-evaluates active issues using detector-specific recheck rules. Closes only issues with a definitiveRESOLVEDverdict; uncertain cases are left for manual review.system_detectorimprovements β word-boundary regexes with a configurable ignore-list replace the old substring match, reducing false positives from benign log noise.
Background maintenance (FIM, heartbeat URLs, self-metrics, session GC, geo-IP cache prune, nightly prune/VACUUM) runs directly in the chat_service.py daily_cleanup_loop.
Plugin Hot-Reload¶
Plugins can be reloaded without restarting the daemon:
From the Web UI: Settings β Plugin hot-reload button Via API: POST /api/plugins/reload (requires superadmin API key or session) Via signal: kill -HUP <pid> β reloads config + watcher patterns + plugins
The Plugin Manager uses importlib.util to load each module into an isolated sys.modules namespace, compiles regex patterns, and instantiates new detector objects.
Pattern Editor (UI-driven patterns)¶
Custom patterns defined in the UI are stored in the custom_patterns table and loaded by the Plugin Manager alongside Python plugins:
CREATE TABLE custom_patterns (
id INTEGER PRIMARY KEY,
name TEXT,
regex TEXT,
file_mask TEXT,
channel TEXT,
severity TEXT,
enabled INTEGER DEFAULT 1
);
These patterns work as lightweight inline detectors β no Python code required. Available via GET /api/patterns and POST /api/patterns. The editor includes a live regex tester and AI pattern suggestions (/api/patterns/suggest) generated from historical issues.
REST API (External Integration)¶
All endpoints require authentication. Methods accepting JSON need Content-Type: application/json. CSRF token (X-CSRF-Token) is required for browser-session POST/PUT/DELETE; Bearer-token API calls are exempt.
Authentication options¶
- Session cookie β obtained via
POST /api/login(+ optional TOTP step) - Bearer token β
Authorization: Bearer <api_key>with fine-grained scopes:read:issues,write:actions,admin:users - Basic Auth β
Authorization: Basic base64(user:pass)(agent ingest only)
Key endpoints¶
| Method | Path | Auth scope | Description |
|---|---|---|---|
| POST | /api/login | β | {username, password} β session cookie (TOTP step if enrolled) |
| GET/POST | /api/2fa/status\|setup\|enable\|disable | session | TOTP enrollment lifecycle |
| GET | /api/v1/issues | read:issues | Active incidents as JSON (incl. plugin_name) |
| POST | /api/v1/agent/ingest | write | Agent telemetry push (+ metrics β threshold check) |
| POST | /api/v1/ingest/bulk | write | Array of alerts in one request |
| GET | /api/issues/<key>/markdown | read:issues | Issue rendered as Markdown |
| POST | /api/issues/<key>/postmortem | read:issues | AI-generated Markdown postmortem |
| GET | /api/v1/actions | read:issues | List pending actions |
| POST | /api/v1/actions/<id>/approve | write:actions | Approve and SSH-execute |
| POST | /api/v1/actions/<id>/reject | write:actions | Reject proposal |
| GET | /api/v1/actions/<id>/output | read:issues | Stream execution output (SSE) |
| POST | /api/ssh/batch | admin | Parallel SSH on up to 50 hosts |
| POST | /api/agents/<hostname>/packages | admin | Package inventory via SSH |
| GET | /api/agents/<hostname>/cve_scan | admin | Pending security updates |
| GET | /api/agents/<hostname>/hw_metrics | admin | net/GPU/SMART/UPS via SSH |
| GET | /api/agents/<hostname>/health_score | read:issues | Composite 0β100 + AβD grade |
| GET/POST/DELETE | /api/agents/<hostname>/ssh_keys | admin | known_hosts management |
| POST | /api/agents/rotate_all_tokens | superadmin | Bulk agent token rotation |
| GET | /api/analytics/resolution_time | read:issues | SLA stats |
| GET | /api/analytics/flapping | read:issues | Most-repeating issues |
| GET | /api/analytics/alert_fatigue | read:issues | False positive stats per plugin |
| GET | /api/analytics/forecast | read:issues | 7-day linear-regression forecast |
| POST | /api/analyze/auto_clusters | read:issues | Algorithmic clustering + AI root cause |
| POST | /api/reports/capacity_plan | read:issues | AI capacity planning |
| GET | /api/predictions/capacity | read:issues | TTC for disk/RAM |
| POST | /api/ansible/run | admin | Validated playbook run with streamed output |
| GET | /api/topology/data | read:issues | Agent topology nodes + edges |
| POST | /api/plugins/reload | admin | Hot-reload all plugins |
| GET/POST | /api/patterns | read/admin | Custom patterns CRUD |
| POST | /api/patterns/suggest | admin | AI regex pattern suggestions |
| POST | /api/apikeys | superadmin | Create API key ({name, scope}) |
| DELETE | /api/apikeys/<id> | superadmin | Revoke API key |
| POST | /api/config/hash_password | superadmin | Generate bcrypt hash |
| GET | /api/config/history/diff?from=X&to=Y | superadmin | Diff two config snapshots |
| GET | /api/admin/backup/download | superadmin | tar.gz backup |
| POST | /api/admin/backup/s3 | superadmin | Upload backup to S3/MinIO |
| GET/POST | /api/admin/log_level | superadmin | Runtime log level |
| GET | /api/admin/audit_trail | superadmin | Config + SSH + action audit |
| GET | /api/admin/security_check | superadmin | Security grade A/B/C/D |
| GET | /api/admin/db_stats | superadmin | DB size + record counts |
| GET | /api/timezone/info Β· /api/timezone/convert | read | Display timezone helpers |
| GET | /metrics | scrape_token | Prometheus metrics endpoint |
| GET | /healthz | β | K8s/UptimeKuma probe (503 on DB failure) |
| GET | /api/docs | β | Swagger UI |
| GET | /api/openapi.json | β | OpenAPI 3.0 spec |
| GET | /api/kb/reindex | admin | Trigger RAG reindex |
Inbound Webhooks (Grafana / Alertmanager / Zabbix)¶
Sentinel accepts inbound webhooks from external monitoring tools on three dedicated endpoints:
| Endpoint | Format |
|---|---|
POST /api/inbound/grafana | Grafana legacy and unified alerting payloads |
POST /api/inbound/alertmanager | Prometheus Alertmanager webhook format |
POST /api/inbound/zabbix | Zabbix Media Type flat JSON |
POST /api/inbound/grafana
Authorization: Bearer <api_key>
Content-Type: application/json
{ "alerts": [ { "labels": {"alertname": "DiskFull", "instance": "hpc-node-01"},
"annotations": {"summary": "disk >90%"}, "status": "firing" } ] }
The payload is normalised and injected into the standard issue pipeline (same as a detector report_problem() call). Webhook auth supports both Bearer token and HMAC X-Hub-Signature-256 with X-Webhook-Timestamp replay protection.
Outbound lifecycle webhooks: configure URLs to be called on issue CREATED / ACKNOWLEDGED / RESOLVED; critical issues can additionally be synced to Gitea (GITEA_URL/TOKEN/REPO) and annotated in Grafana (grafana_annotations.url/api_key).
Functions Added in v2026.09.x¶
| Function | Module | Purpose |
|---|---|---|
state_base._ensure_keeper() | state_base.py | Opens and holds one long-lived DB connection. Without it every per-request close() is the last one and triggers a WAL checkpoint which, because close() does not release the GIL, freezes the whole process. Must issue a real query β connect() alone does not open the file. |
watcher.fim_check() | watcher.py | SHA-256 of watched files against a baseline in kv_settings['fim_state']. Returns changed paths and raises FIM_CHANGE\|<path> issues. The first run only records a baseline; an unreadable file never overwrites it. |
state_agents.parse_root_sessions(msg) | state_agents.py | Parse a root-monitor report into {tty, ip, since} entries. |
state_agents.reconcile_root_sessions(conn, host, sessions, now) | state_agents.py | Reconcile reported sessions against the DB on (server, tty, login time). Returns (new, kept, closed). |
ChatService.build_alerts_context(limit=15) | chat_service.py | Active-issue summary for AI context. Shared by the streaming and non-streaming chat paths β only one of them used to send it, so the UI answered from the knowledge base. |
ChatService._loads_repaired(candidate) | chat_service.py | Last-resort JSON parse when the model used an invalid escape (a shell regex such as 'a\|b'). Doubles lone backslashes. |
config.apply_extra_body(payload) | config.py | Merge ollama_extra_body into an OpenAI /v1 request body. Call only on /v1 branches. |
ChatService.build_logs_context(max_files, lines_per_file, max_chars) | chat_service.py | Tail of the log files that active issues reference, ordered by issue count. The path comes from the database, so it goes through _safe_log_path() β a doctored record must not make the model read arbitrary files. |
ChatService.build_alerts_context(limit) | chat_service.py | Active-issue summary; shared by both chat paths. |
plugin_manager._report_load_result() | plugin_manager.py | Compares enabled detectors in config against those loaded; raises SENTINEL\|detectors_missing for the difference. |
state_agents.user_audit_login(username, ip, auth_source) | state_agents.py | Records a login. Called from _complete_login(), the one point every login passes through. |
state_agents.user_audit_add_online(username, seconds) | state_agents.py | Adds a finished session's duration; implausible values are discarded. |
state_agents.get_user_audit() | state_agents.py | Audit joined with the current role and live session count. Builds rows from cursor.description rather than row_factory β sqlite3 here is stdlib while __main__ swaps pysqlite3 into sys.modules. |
A streaming generator must not touch
gβ it runs after the view returns. Capture what you need into a local first;TestSseGeneratorsAvoidRequestContextenforces it.Never a second y-axis on one chart. Two scales can manufacture any correlation. Compare two measures as two charts sharing the x axis;
TestMetricChartNoDualAxisenforces it.Beware of
ginside streaming generators. An SSE generator is evaluated after the view returns, when Flask's request context is gone βg.usernameinside raisesRuntimeError, the generator dies and the connection is cut mid-stream. Capture the value into a local before defining the generator.TestSseGeneratorsAvoidRequestContextenforces this.
Database Schema β Key Tables¶
problems¶
CREATE TABLE problems (
key TEXT PRIMARY KEY,
status TEXT, -- active | acknowledged | validating | resolved
severity TEXT, -- critical | high | medium | low
channel_type TEXT,
host TEXT,
last_line TEXT,
last_seen TEXT,
missing_count INTEGER DEFAULT 0,
acknowledged_by TEXT,
acknowledged_at TEXT,
depends_on TEXT -- JSON list of dependent issue keys
);
actions (Autofix)¶
CREATE TABLE actions (
id INTEGER PRIMARY KEY,
command TEXT,
status TEXT, -- pending | running | completed | failed | rejected
risk_score INTEGER,
risk_reasons TEXT, -- JSON
mode TEXT, -- dry_run | real
dry_run_output TEXT,
actor TEXT,
created_at TEXT,
executed_at TEXT,
stdout TEXT,
stderr TEXT
);
issue_history (archive)¶
_archive_problem() copies first_seen from problems, enabling resolution-time analytics (/api/analytics/resolution_time). Pruned after 90 days by the scheduler.
RAG Knowledge Base (rag.py + build_kb.py)¶
Adding documents¶
# Rebuild from /opt/Sentinel/knowledge_base/ β interactive menu, PDF/DOCX/MD support
python3 build_kb.py
# Supported formats: .md .txt .pdf .docx .csv
# Or upload directly via the Web UI (Settings β Knowledge Base)
Query flow¶
- User query β
RAGEngine.search(query, top_k=3) - Vectorise query with
nomic-embed-text - HNSW nearest-neighbour search in ChromaDB
- Top-K chunks injected into system prompt
- If ChromaDB unavailable β BM25 TFΓIDF fallback over tokenised text blocks
AI Worker Pool (ollama_service.py)¶
task_queue (DB)
β
βββββ΄ββββββββ
β Worker-0 β β fetches task, locks worker_id
β Worker-1 β
β Worker-N β β configurable via WORKER_THREADS
βββββββββββββ
β
Ollama API
POST /v1/chat/completions (timeout 90 s)
β
Failover: 400 error β legacy /api/chat format
β
Response β append to issue in DB
β emit WebSocket event
Workers rotate indefinitely until state.shutdown_event.is_set(). Tasks are never automatically requeued on error (prevents infinite retry loops on bad input).
Concurrency rule: never acquire
llm_semaphorearound a call intoexecute_ollama()β the function takes it internally; a re-entrant acquire deadlocks the whole AI pipeline permanently.
Development Workflow¶
make lint # ruff + ESLint
make test # pytest β 181 tests
make ci # lint + test (same as CI)
make build # minify JS (.min.js are not in git)
- CI runs in Gitea Actions (
.gitea/workflows/ci.yml): pytest +node --check+ build - A
pre-pushgit hook runs the JS check + pytest locally - See
CONTRIBUTING.mdfor the architecture diagram, dev workflow, and security rules (e.g. every new endpoint needs auth + input validation + a route test)