Skip to content

Sentinel API Reference (v2026.11.014)

Complete API documentation for plugins, agents, and external integrations.


Authentication

Three authentication methods are supported:

Method Header When to use
Session cookie Cookie: session=... Web browser, obtained via POST /api/login
Bearer token Authorization: Bearer <api_key> External apps, agents, scripts
Basic Auth Authorization: Basic base64(user:pass) Legacy β€” agent ingest only

API keys are created by a superadmin in Settings β†’ REST API Keys. Each key carries fine-grained scopes (legacy read/write/admin keys remain valid):

Scope Permissions
read:issues GET endpoints β€” issues, agents, analytics, topology
read:root Same as read:issues, plus the ROOT channel β€” no write access (v2026.11.005+)
write:actions POST β€” agent ingest, approve/reject actions, create issues
admin:users Everything including patterns, reload, user/key management

The ROOT channel is a trap worth knowing about. The role a request runs under is derived from the key's scope, not from the user who created the key. With read or read:issues, issues from the ROOT channel are simply absent from the response β€” no error, no warning. Before v2026.11.005 the only way to see them was write or admin, i.e. a key that can also change things. read:root removes that trade-off and an admin may issue it; the write scopes remain superadmin-only.

CSRF: browser-session POST/PUT/DELETE requests require the X-CSRF-Token header (the web UI adds it automatically via a global fetch wrapper). Bearer-token calls are exempt.

2FA: if TOTP is enrolled for the user, POST /api/login is followed by a second step with the 6-digit code. Manage via /api/2fa/status|setup|enable|disable.

Version format: version is calendar versioning YYYY.MM.NNN, not semver β€” the sequence restarts each month (2026.10.005 β†’ 2026.11.001), so compare it as a tuple of integers. An empty string or a missing key means "unknown" (older servers do not send the field), not "old".

Brute-force protection: 5 failed logins β†’ IP banned 300 s. /api/analyze/* endpoints are rate-limited to 10 req/min per IP.


Plugin API (internal β€” api.py)

Used by detector plugins running inside the Sentinel process.

report_problem(key, data)

Create or update an incident. Same key = update last_seen and reset missing_count.

Python
api.report_problem("DISK_FULL|proxmox01|/data", {
    "status": "active",
    "last_line": "Disk /data 92% full",
    "channel_type": "infra",       # infra | security | agents | root
    "severity": "CRITICAL",        # INFO | WARNING | CRITICAL
    "host": "proxmox01",
    "cluster": "proxmox01",
    "log_file": "/var/log/sentinel/logs/capacity.log",
    "last_seen": datetime.now(timezone.utc).isoformat(),
    "missing_count": 0
})

resolve_problem(key)

Close an incident. Emits a WebSocket resolve event to all connected clients.

Python
api.resolve_problem("DISK_FULL|proxmox01|/data")

get_problem(key) β†’ dict | None

Return the current DB row for a key, or None if it doesn't exist.

enqueue_ai_task(prompt, channel)

Push a task to the AI worker queue. Non-blocking.

Python
api.enqueue_ai_task(
    f"Analyse this SSH attack: {log_line}",
    channel="security"
)

save_telemetry_snapshot(metric, value, host)

Write a time-series data point. Writes are buffered and flushed in batches (and on SIGTERM).

Python
api.save_telemetry_snapshot("cpu_usage", 94.2, "hpc-node-01")

get_infrastructure_label(file_path) β†’ str

Extract the SERVER: <hostname> context injected by Overhealth, or derive from file path.

add_root_audit(server, ip)

Idempotent root-session audit entry (one active record per server+ip) with reverse DNS lookup. Since v2026.09.001 an already-active session has its last_seen refreshed instead of being silently ignored β€” without that timestamp the stale sweep cannot tell an ongoing session from an abandoned record, and rows stayed is_active=1 forever (the daily pruning only deletes is_active=0). Records written through this path are marked origin='detector'.

close_root_audit(server, ip)

Marks matching active records as disconnected. Detectors that track sessions themselves should call it; those that do not are covered by the age-based sweep (root_audit_stale_minutes, default 30).

notifier.send_notification(title, message, severity, channel)

Standalone fan-out to all enabled outbound channels with retry queue and per-severity throttling. See Programmer Documentation.


REST API (HTTP)

Base URL: http://<host>:5050

Authentication

HTTP
POST /api/login
Content-Type: application/json

{"username": "admin", "password": "secret"}

Response sets Set-Cookie: session=... (HMAC-signed; absolute session timeout 12 h). If the account has TOTP enrolled, the response requests a second step with the 6-digit code.

2FA / TOTP

Method Path Auth Description
GET /api/2fa/status session Enrollment status for current user
POST /api/2fa/setup session Generate secret + QR code (base64 PNG)
POST /api/2fa/enable session Confirm with one TOTP code
POST /api/2fa/disable session/admin Disable (admin can disable for any user)

Issues

Method Path Auth Description
GET /api/v1/issues read Active incidents as JSON array β€” see the field notes below
GET /api/v1/issues/<key> read Single incident detail
DELETE /api/v1/issues/<key> admin Hard delete incident
POST /api/v1/issues/<key>/acknowledge admin Acknowledge (β†’ yellow badge)
POST /api/v1/issues/<key>/tag admin {"tag": "storage"} β€” add tag
DELETE /api/v1/issues/<key>/tag/<tag> admin Remove tag
POST /api/v1/issues/<key>/comment admin Add timeline comment
GET /api/v1/issues/<key>/history read Timeline events
GET /api/issues/<key>/markdown read Issue rendered as Markdown
POST /api/issues/<key>/postmortem read AI-generated Markdown postmortem
POST /api/issues/<key>/delete admin Direct REST delete (used by modal buttons)
POST /api/issues/<key>/ignore admin Add to ignore list + delete from DB
POST /api/v1/issues/batch_analyse admin Batch AI analysis (up to 50)
POST /api/analyze/auto_clusters read Algorithmic clustering + optional AI root-cause naming

Agents

Method Path Auth Description
POST /api/v1/agent/ingest write Agent telemetry push (rate-limited; metrics checked against per-agent thresholds)
POST /api/v1/ingest/bulk write Array of alerts in one HTTP request
GET /api/v1/agents read List registered agents with status
GET /api/v1/agents/<hostname> read Agent detail (issues, metrics, version)
POST /api/v1/agents/register admin Register new agent, returns token (+ QR payload)
DELETE /api/v1/agents/<hostname> admin Unregister agent
POST /api/agents/rotate_all_tokens superadmin Bulk token rotation
GET /api/agents/<hostname>/health_score read Composite health 0–100 + grade A–D
POST /api/agents/<hostname>/packages admin Package inventory via SSH (dpkg/rpm, filter param)
GET /api/agents/<hostname>/cve_scan admin Pending security updates via SSH
GET /api/agents/<hostname>/hw_metrics admin net / GPU / SMART / UPS via SSH
GET /api/agents/<hostname>/scheduled_actions read Planned actions for this host
GET/POST/DELETE /api/agents/<hostname>/ssh_keys admin known_hosts view / rescan / delete

Agent ingest payload

HTTP
POST /api/v1/agent/ingest
Authorization: Bearer <agent_token>
Content-Type: application/json

{
  "hostname": "hpc-node-04",
  "version": "a3f8c12",
  "channel": "agents",
  "ip_addresses": ["10.0.0.4"],
  "timestamp": "2026-06-02T18:42:00+02:00",
  "data": {
    "cpu": 94.2,
    "message": "CPU critical: 94.2%",
    "severity": "CRITICAL"
  },
  "metrics": { "cpu_pct": 94.2, "ram_pct": 61.0, "disk_pct": 72.5 }
}

The optional metrics object is evaluated against agent_thresholds rules β€” issues are created on breach and auto-resolved on recovery.


Autofix Actions & SSH

Method Path Auth Description
GET /api/v1/actions read List pending/recent actions
POST /api/v1/actions/<id>/approve admin Approve and SSH-execute
POST /api/v1/actions/<id>/reject admin Reject proposal
GET /api/v1/actions/<id>/output read Stream execution output (SSE)
GET /api/v1/actions/<id>/audit read Action lifecycle audit log
POST /api/ssh/batch admin One allowlisted command on ≀ 50 hosts in parallel (15 s per-host timeout)
POST /api/ansible/run admin Validated ansible-playbook run, streamed output

Analytics & Reports

Method Path Auth Description
GET /api/analytics/resolution_time read SLA β€” average resolution time per plugin
GET /api/analytics/flapping read Most frequently re-appearing issues
GET /api/analytics/alert_fatigue read False positive stats per plugin
GET /api/analytics/forecast read 7-day linear-regression issue forecast
GET /api/analytics/changes_since_login read New/resolved issue counts since last session
GET /api/predictions/capacity read TTC (time-to-full) for disk/RAM
POST /api/reports/capacity_plan read AI capacity plan (HOST/PROBLEM/RECOMMENDATION/PRIORITY)

System, Configuration & Admin

Method Path Auth Description
GET /api/health β€” {"status": "ok"} β€” used by watchdog
GET /api/status_check read Header counts (issues, agent_issues, root_issues, security_issues, ai_sessions) plus version β€” the server release, e.g. "2026.11.008" (v2026.11.005+)
GET /healthz β€” K8s/UptimeKuma JSON probe β€” DB check, 503 on failure
GET /status β€” Public status dashboard (HTML): CPU/RAM/Disk telemetry, anonymized recent incidents, online/offline agent table, infra/agent/security category breakdown
GET /api/status read Detailed system status (DB size, queue depth, AI latency)
POST /api/plugins/reload admin Hot-reload all plugins
GET /api/plugins/<name>/log read Recent issue records produced by one detector (limit, max 500). Backs the Log button in the Detectors & Plugins table
GET /api/root_audit read Root session history. Returns cluster, tty and origin (detector/agent) per record. Limited to the last root_audit_history_per_cluster (default 30) per cluster rather than globally β€” a single busy cluster used to consume the whole window. Active sessions are always included
GET /api/users/roles superadmin Users with their role and audit data β€” auth_source (local/ldap), first_seen, last_login, last_ip, login_count, online_seconds, active_sessions. Users who only ever logged in (never had a role assigned) are included; before v2026.09.012 they were invisible here
GET /api/queue/details read Request queue. requests lists the AI requests actually in flight (status: waiting on the semaphore, processing running) followed by DB task_queue rows. In-flight entries carry cancellable: false β€” they are HTTP threads, not queue rows
GET /api/patterns read List custom detection patterns
POST /api/patterns admin Add pattern
POST /api/patterns/suggest admin AI regex suggestions from historical issues
PUT /api/patterns/<id> admin Update pattern
DELETE /api/patterns/<id> admin Delete pattern
POST /api/apikeys superadmin Create API key {name, scope}
DELETE /api/apikeys/<id> superadmin Revoke API key
POST /api/config/hash_password superadmin Generate bcrypt hash for config.yaml
GET /api/config/view superadmin Config with secrets masked as ***
GET /api/config/history/diff?from=X&to=Y superadmin Diff two config snapshots
POST /api/config/restore superadmin Restore config (auto-backup first, 10 kept)
GET /api/admin/backup/download superadmin Full backup as tar.gz
POST /api/admin/backup/s3 superadmin Upload backup to S3/MinIO
GET/POST /api/admin/log_level superadmin Get/set log level at runtime
GET /api/admin/audit_trail superadmin Config + SSH + action audit viewer
GET /api/admin/security_check superadmin Security headers grade A/B/C/D
GET /api/admin/db_stats superadmin DB size + record counts
POST /api/admin/aggregate_telemetry superadmin Aggregate old telemetry
POST /api/admin/validate_url superadmin URL validation (SSRF guard β€” rejects private ranges)
GET /api/timezone/info read Configured display timezone
GET /api/timezone/convert read Convert timestamp to display TZ
GET /api/kb/reindex admin Trigger RAG knowledge base reindex

AI Sessions

Captured AI agent sessions reported by agent_ai_session_monitor. Stored outside the issue table, so nothing alerts and nothing lingers as an open issue.

Method Path Auth Description
GET /api/ai_sessions read Captured AI sessions β€” running ones always returned, finished ones capped per host

Each row carries host, cluster, user, tool, category (cli / runtime / mcp / api), model, pid, cwd, cmdline, remote, started_at, first_seen, last_seen, ended_at and is_active. A session is identified by (host, pid, started_at); a repeated poll confirms the row instead of inserting a new one. GET /api/status additionally returns ai_sessions β€” the number of currently running sessions, used by the header badge.

Retention is driven by ai_sessions_retention_days (30), ai_sessions_history_per_host (30) and ai_sessions_stale_minutes (30); the last one closes sessions from a host whose agent stopped reporting altogether.


/api/v1/issues β€” fields

Alongside the camelCase names the mobile app expects (channelType, lastSeen, missingCount, logFile, lastLine), the response carries the raw DB values (v2026.11.005+):

Field Meaning
first_seen / firstSeen When the problem was first recorded β€” the basis for its age
occurrence_count / occurrenceCount How many times it has recurred
plugin_name / pluginName Detector name (log_file carries the same value for backwards compatibility)
db_severity / dbSeverity Real severity from the database
acknowledged_by, snoozed_until Who acknowledged it, until when it is snoozed
ignored Whether the key is on the ignore list

severity stays derived from the channel (CRITICAL for security/root, otherwise WARNING) because the mobile app's IssueModel.fromJson depends on it. Sort by db_severity, not by severity.

?include_ignored=1 additionally returns ignored issues, each marked ignored: true. Without the parameter they are dropped, exactly as before β€” the parameter is opt-in so that existing clients do not suddenly start showing items their user has hidden. Accepted values: 1, true, yes (case-insensitive).


API Keys (self-service)

Long-lived keys for scripts, passed in the X-API-Key header. A user manages their own keys; the secret is shown once, at creation.

Method Path Auth Description
GET /api/apikeys read List the caller's keys (metadata only, never the secret)
POST /api/apikeys read Create a key β€” the plaintext secret is returned once
DELETE /api/apikeys/<key_id> read Revoke one of the caller's keys

Pass the key in the header, never in the URL β€” query strings end up in proxy and browser logs.


Request Tracker (RT)

Ticket integration; available when RT is configured.

Method Path Auth Description
GET /api/rt/tickets read List tickets, optional queue and q filters
POST /api/rt/comment write Add a comment to a ticket (recorded with its author)
POST /api/rt/take write Take ownership of a ticket
POST /api/rt/status write Change ticket status

Topology

Method Path Auth Description
GET /api/topology/data read Nodes + edges for force-directed graph

Export, Inbound Webhooks & Integrations

Method Path Auth Description
GET /api/export/incidents read CSV export of incidents
GET /api/export/telemetry read CSV export of telemetry data
POST /api/inbound/grafana write Grafana webhook (legacy + unified alerting)
POST /api/inbound/alertmanager write Prometheus Alertmanager webhook
POST /api/inbound/zabbix write Zabbix Media Type flat JSON
GET/POST /api/integrations/<name>/status\|toggle\|save\|test admin Per-integration config (Teams, Slack, Discord, Telegram, Opsgenie, ntfy, Gotify, SMTP, Matrix, PagerDuty, HA, MQTT)
GET /api/channels/notify admin Per-channel notification toggles
POST /api/channels/<ch>/notify/toggle admin Toggle channel notifications
GET /metrics scrape_token Prometheus metrics endpoint

Inbound webhook auth supports Bearer token or HMAC X-Hub-Signature-256 + X-Webhook-Timestamp (replay protection).

Outbound lifecycle webhooks fire on issue CREATED / ACKNOWLEDGED / RESOLVED; critical issues can sync to Gitea and annotate Grafana dashboards.

Prometheus metrics

Text Only
# GAUGE: active incidents per channel
sentinel_issues_total{channel="infra"} 3
sentinel_issues_total{channel="security"} 1

# GAUGE: agents online
sentinel_agents_online 12

# GAUGE: AI worker queue depth
sentinel_queue_depth 2

# GAUGE: AI response latency (seconds)
sentinel_ai_latency_seconds 1.24

# GAUGE: log parser throughput
sentinel_lines_parsed_per_min 1840

Self-metrics can also be pushed to a Prometheus pushgateway (prometheus.pushgateway_url).


API Documentation (auto-generated)

Path Description
GET /api/docs Swagger UI (interactive browser)
GET /api/openapi.json OpenAPI 3.0 JSON spec

WebSocket Events (Socket.IO)

Connect to ws://<host>:5050/socket.io/. Transport order is ['polling', 'websocket'] with upgrade β€” works behind any reverse proxy. Server-side the frontend queue is bounded (500, drop-oldest) and duplicate messages within 1 s are deduplicated.

Event (server β†’ client) Payload Description
new_issue {key, severity, channel, host, message} New incident created
resolve {key} Incident resolved (auto or manual)
new_action {id, command, risk_score} New Autofix pending
action_result {id, status, stdout} Autofix completed
agent_status {hostname, status} Agent online/offline transition
telemetry_update {metric, value, host} Telemetry data point
chat_message {user, text, device_id} P2P admin chat message
Event (client β†’ server) Payload Description
chat {message} Send chat query to AI
ping_presence {device_id, user} Mobile heartbeat (every 60 s)