Sentinel Commander — Documentation (v2026.08.001)¶

                   ##
               ++########++
              ++++###+++++++
             +++++++++++-++-.
            ++++++#++++++-+-
             ++######+++++--
             +++  ####+  -+-
              ++++######++-
              ++++#######+-#
               ++++######++-+.-+
              ++++++###+++##+---++#
            +++++++++###+#######++++++++
         -+++++++++++###+#####+#+++++----..-
      ++-++++++++#######+######++++++++--..  .
     ---++++##+++#####++####++++#++++---.....
    --+++++############++###++++#+++---........
   --++++++##++##########++++++++++----.........
   --+++++#####++#####+#+++++++++++++-.-.........
   --++++#####+########++++++-+++++-++--------.-..
   ++++##########+####++++++-+++++++-+++---------.

        S E N T I N E L   C O M M A N D E R
                    v2026.08.001

Hybrid AI Log Monitor & Analyzer for Linux Infrastructure

Quick Navigation¶

• Deployment Guide — Production deployment of the entire system
• API Reference — Plugin and integration API
• Troubleshooting — Common issues and solutions
• Host Setup — Preparing monitored hosts for AI diagnostics

What is Sentinel?¶

Sentinel Commander is an advanced hybrid monitoring system with a full AI layer designed for enterprise Linux infrastructure. It combines a Pull approach (inotify log tailing) with a Push approach (remote Python agents POSTing telemetry). Core integrations: Hailo AI HAT 2+ NPU (hailo-ollama), local/external Ollama LLM, ChromaDB (RAG).

As of v2026.08.001 the system has accumulated 1 050 automated tests and 80/100 AI roadmap items across 24 new AI modules covering diagnostics, verification, remediation planning, correlation, baselining, forecasting, and knowledge management.

What's New in v2026.08.001 (2026-08-03)¶

What's New in v2026.07.001 (2026-07-31)¶

AI Layer — 21 New Modules¶

Scale¶

• Tests: 317 → 938 (+621 new tests covering the full AI layer)
• AI roadmap: 66/100 items complete
• Full decision audit trail for every AI action
• All AI modules fully integrated into the Scheduler maintenance loop

What's New in v2026.06.024 (since v2026.06.005)¶

Security Hardening¶

Notifications & Integrations¶

Analytics, Agent Fleet, Operations¶

• Health score per host (A–D grade), 7-day issue forecast, SLA & alert-fatigue reports
• Batch SSH (50 hosts, ThreadPoolExecutor), per-agent thresholds, CVE scanner, package inventory
• /healthz probe, config backup/restore with snapshots, SIGHUP hot-reload, FIM, Ansible runner
• Composite DB indexes, WAL tuning, HTTP caching (ETag + 304), SocketIO backpressure, virtual scroll

Engineering Quality¶

• 181 automated tests (v2026.06.024 baseline) — now 1 050 in v2026.08.001
• CI pipeline — pytest + node --check + make build, pre-push git hook
• Linting — ruff (Python), ESLint, pinned requirements.txt

Feature Overview¶

Architecture¶

  Log files ──inotify──▶ watcher.py ──▶ plugins[] ──▶ api.report_problem()
  Remote agents ──POST──▶ /api/v1/agent/ingest              │
  Grafana/AM/Zabbix ──POST──▶ /api/inbound/*                │
                                                            ▼
                                                   state.py (SQLite WAL)
                                                            │
                         ┌──────────────────────────────────┤
                         ▼                                  ▼
               ollama_service.py                   Flask + SocketIO :5050
               (AI worker pool)                    (chat_service.py + routes/)
                         │                                  │
         ┌───────────────┼───────────────┐         ┌────────┴────────┐
         ▼               ▼               ▼         ▼                 ▼
  hailo-ollama      Ollama CPU      external   scheduler.py     notifier.py
  (NPU :8000)       (:11434)        API        (maintenance)    (13 channels)

Core modules¶

AI layer modules (v2026.07+)¶

Data Flow¶

A. Ingest & Detection¶

1. Log line written to /var/log/sentinel/logs/ → inotify event
2. watcher.py reads new lines via mmap
3. PluginManager routes lines to matching detectors by file mask
4. Detector generates unique key (e.g. DISK_FULL|proxmox01|/data)
5. State written to problems table → task pushed to task_queue
6. Push path: agents POST /api/v1/agent/ingest (or /api/v1/ingest/bulk for batches); metrics payload is checked against per-agent thresholds

B. AI Inference + RAG¶

1. AI worker fetches task from task_queue
2. Loads prompt template by channel (security, clusters, infra, root, icinga)
3. RAG: ChromaDB query → relevant context injected into system prompt
4. Prompt dispatched to Ollama (NPU/CPU/external)
5. If response contains remediation script → actions.py creates pending action

C. Remediation Feedback Loop¶

1. actions.py creates DB record with proposed SSH command
2. Frontend emits new_action WebSocket event to Web UI
3. Admin with admin or superadmin clicks Approve or Reject
4. On Approve: SSH connection to management node → command execution (allowlist pre-validated)
5. STDOUT/STDERR logged → incident enters validating state

D. Notification Pipeline¶

1. Issue saved → notifier.send_notification() fan-out to all enabled channels
2. Per-detector and per-channel toggles checked first
3. Per-severity throttle applied (critical 15 min … low 4 h)
4. Failures enter the retry queue (30 s → 120 s → 300 s backoff)
5. Lifecycle webhooks fired on CREATED / ACKNOWLEDGED / RESOLVED

Issue Workflow¶

  detected ──▶  active  ──▶  acknowledged (✓✓ button)
                   │                │
                   │           validating ──▶ resolved
                   │
                   └──▶ (auto-resolved by detector or expiry rule)

Escalation rules: If an issue stays active or acknowledged for more than N hours without resolution, its severity is automatically raised to the next level.

Lifecycle webhooks can notify external systems on every transition, and critical issues can be mirrored into Gitea.

Installation¶

git clone <repo> /opt/Sentinel
sudo bash /opt/Sentinel/install.sh        # Debian/Ubuntu/RHEL/Rocky/Pi OS
sudo python3 /opt/Sentinel/sentinel_init.py   # interactive config wizard
sudo systemctl enable --now sentinel

The setup wizard refuses default passwords, generates the systemd unit with WatchdogSec=900 and a WAL-checkpoint ExecStartPre, and creates /var/lib/sentinel.

Key configuration (config.yaml)¶

web:
  port: 5050
  password_hash: "$2b$12$..."      # bcrypt — preferred over plaintext `password`

security:
  login_max_attempts: 5
  login_ban_time: 300
  session_max_hours: 12

ollama:
  url: "http://localhost:11434"
  model: "llama3.2"
  workers: 3

hailo_ollama:
  enabled: false
  url: "http://localhost:8000"

ldap:
  enabled: false
  host: "ldaps://ldap.example.com"
  base_dn: "dc=example,dc=com"

prometheus:
  enabled: true
  scrape_token: "{SECRET:PROM_TOKEN}"   # env-var substitution

telemetry_alerts:
  cpu_critical: 95
  disk_critical: 95
  temp_critical: 85

fim:
  enabled: true
  paths: [/etc/passwd, /etc/shadow, /etc/ssh/sshd_config]

Requirements¶

• Python 3.13+ · Flask · Flask-SocketIO · ChromaDB · paho-mqtt
• pyotp (2FA) · qrcode + pillow (QR enrollment) · bcrypt · jsonschema
• Ollama with nomic-embed-text (for embeddings)
• Optional: Hailo AI HAT 2+ with hailo-ollama 5.3.0 for NPU inference

Component Ecosystem¶