Přeskočit obsah

Sentinel Ecosystem — Deployment Guide (v2026.11.014)

Complete guide for deploying the entire Sentinel monitoring ecosystem in production.


Architecture Overview

Text Only
  ┌─────────────────────────────────────────────────┐
  │              Central Server                     │
  │  Sentinel Core :5050  +  Overhealth (cron)      │
  │  sentinel-plugins (co-located)                  │
  └──────────────────────────┬──────────────────────┘
                             │
          ┌──────────────────┼──────────────────┐
          │ SSH pull         │ Push (HTTPS)      │ optional
          ▼                  ▼                   ▼
   Monitored nodes    Agent nodes         sentinel-alert :5056
   (Overhealth)       (sentinel-agent)    sentinel-hw :5055

Installing from Packages

Every component is published as a .deb and an .rpm. Everything is pure Python, so the packages are architecture independent: one .deb covers Ubuntu 22.04/24.04/26.04 and Debian 12/13, one .rpm covers RHEL 8/9/10 and derivatives. Only the dependency metadata differs, and for RPM a rich dependency (python3.12 or python3.11 or python3.10) handles the fact that RHEL 8 defaults to Python 3.6 and RHEL 9 to 3.9.

A release is a file to download; a registry is what lets a machine fetch the package through apt or dnf, upgrades included — that is how it should be installed in production.

Bash
# Debian / Ubuntu — with signature verification, not trusted=yes
curl -s https://<gitea>/api/packages/<owner>/debian/repository.key \
  | gpg --dearmor > /etc/apt/keyrings/sentinel-gitea.gpg
echo "deb [signed-by=/etc/apt/keyrings/sentinel-gitea.gpg] https://<gitea>/api/packages/<owner>/debian stable main" \
  > /etc/apt/sources.list.d/sentinel.list
apt update && apt install sentinel

# RHEL and derivatives — the generated .repo sets gpgkey itself
dnf config-manager --add-repo https://<gitea>/api/packages/<owner>/rpm.repo
dnf install sentinel

Gitea signs the Release file, so [trusted=yes] is unnecessary — without the key apt only warns NO_PUBKEY. Anonymous access works only when the instance does not require sign-in (REQUIRE_SIGNIN_VIEW in app.ini); otherwise the token has to go into the repository URL as well.

Components whose dependencies exist only on PyPI (sentinel, sentinel-alert, sentinel-console) build a virtualenv in postinst, which requires network access. If that fails, the package install fails loudly rather than leaving a configured package with a service that cannot start.

Two components' install paths are worth knowing: Sentinel installs into /opt/Sentinel (capital S, matching install.sh and the docs), and sentinel-alert keeps its runtime data in /var/lib/sentinel-alert rather than next to the code, so an upgrade cannot overwrite the attack database or the inventories.

Version numbers are calendar based, YYYY.MM.NNN, per component — the sequence restarts every month, so sentinel-alert 2026.11.001 is newer than 2026.10.001 and has nothing to do with Sentinel's own number. A trailing -N is the package revision: it goes up when only the packaging changed and the program did not.


Step Component Where
1 Sentinel (core server) Central monitoring server
2 sentinel-plugins Same server as Sentinel
3 sentinel-overhealth Same server as Sentinel
4 sentinel-agent Every monitored node
5 sentinel-alert Any machine (optional, standalone)
6 sentinel-hw Raspberry Pi Zero 2W (optional)
7 sentinel-app Android device (optional)
8 sentinel-console Admin workstations (optional)
9 sentinel-docs Internal docs server (optional)

1. Sentinel — Central Server

Requirements

  • Python 3.13+, Debian 11+ / Ubuntu 22.04+ / RHEL 8/9/10 / Rocky Linux
  • Ollama running with nomic-embed-text model (for RAG embeddings)
  • 4 GB RAM minimum, 8 GB recommended
  • 10 GB disk for DB + ChromaDB

Installation

Bash
git clone git@github.com:foxik0070/Sentinel.git /opt/Sentinel
sudo bash /opt/Sentinel/install.sh
sudo python3 /opt/Sentinel/sentinel_init.py   # interactive wizard
sudo systemctl enable --now sentinel

Key configuration (/etc/sentinel/config.yaml)

YAML
web:
  host: 0.0.0.0
  port: 5050
  # bcrypt hash preferred over plaintext password — generate via
  # Settings → "Hash password" or POST /api/config/hash_password
  password_hash: "$2b$12$..."

security:
  login_max_attempts: 5
  login_ban_time: 300
  session_max_hours: 12

ollama:
  url: "http://localhost:11434"
  model: "llama3.2"
  workers: 3

teams_channels:
  webhooks:
    security: "https://outlook.office.com/webhook/..."

prometheus:
  enabled: true
  scrape_token: "{SECRET:PROM_TOKEN}"   # env-var substitution supported

telemetry_alerts:
  cpu_critical: 95
  disk_critical: 95
  temp_critical: 85

fim:
  enabled: true
  paths: [/etc/passwd, /etc/shadow, /etc/ssh/sshd_config]

# Root session audit (v2026.09.001+)
root_audit_stale_minutes: 30        # close sessions a detector stopped reporting
root_audit_history_per_cluster: 30  # history shown per cluster, not globally

# Log excerpts sent to the model as chat context (v2026.09.016+)
chat_logs_max_files: 5              # files, picked by how many active issues reference them
chat_logs_lines: 40                 # last N lines from each

Chat context is capped. Only files that active issues actually reference are sent, ordered by issue count, and the whole block is truncated at 6 000 characters. The path is read from the database and forced through the LOG_DIR containment check — a doctored record must not be able to make the model read arbitrary files.

FIM only started working in v2026.09.004. The fim: block above existed and was documented earlier, but nothing ever called the check — its only caller was a dead module removed in v2026.08.003. If you configured FIM before that release, it was silently doing nothing; the first run after upgrading records a baseline without alerting, so expect no findings until something actually changes.

Reasoning models (ollama_extra_body)

Models that emit chain-of-thought need it switched off, otherwise the reasoning ends up in chat answers, in the daily digest and in auto-severity classification. qwen3-32b writes <think> blocks inside content rather than into reasoning_content, where Sentinel could separate them:

YAML
ollama_model: qwen3-32b
ollama_extra_body:
  chat_template_kwargs:
    enable_thinking: false

The fields are merged into the OpenAI /v1 request body only. Hailo and the legacy /api/generate branch do not understand them and would reject the request.

Measured against a LiteLLM proxy, median of four runs, severity classification:

model total first token throughput
qwen2.5_coder:32b 2.77 s 0.05 s 70 chars/s
qwen3-32b + enable_thinking=false 1.61 s 0.06 s 82 chars/s

With thinking left on, the same query takes 8.4 s and produces 1 346 characters, most of it reasoning.

Embeddings follow ollama_url. RAG derives the embedding endpoint from it unless embedding_ollama_url is set. If you point ollama_url at a server that only serves a chat model, embeddings break and RAG silently stops returning context.

The setup wizard (sentinel_init.py) refuses default passwords, creates /var/lib/sentinel, and generates the systemd unit with WatchdogSec=900 plus a WAL-checkpoint ExecStartPre. Config files are validated with jsonschema — invalid critical keys are rejected at load.

Verify

Bash
systemctl status sentinel
curl -s http://localhost:5050/api/health
# → {"status": "ok"}
curl -s http://localhost:5050/healthz
# → {"status": "ok", "db": "ok"}   (HTTP 503 on DB failure — for K8s/UptimeKuma)

Detectors kept in a separate repository need their own deploy step. If your detectors live outside the Sentinel package, no deploy of it will place them into PLUGIN_DIR — deploy that repository separately. Never rsync --delete into the plugin directory: it would remove the plugins that ship with Sentinel. Since v2026.09.013 a detector that the config asks for but that fails to load raises SENTINEL|detectors_missing, so a missing deploy step is visible within a minute instead of going unnoticed.

2. sentinel-plugins

Sentinel ships only the base — base.py plus system_detector and services_detector. The other ten detectors come from this package.

Bash
apt install sentinel-plugins     # Debian / Ubuntu
dnf install sentinel-plugins     # RHEL and derivatives

They land in /opt/sentinel-plugins. Point Sentinel at that directory by making plugin_dir a list (v2026.11.010 and later):

YAML
plugin_dir:
  - /opt/Sentinel/sentinel/plugins   # the base that ships with Sentinel
  - /opt/sentinel-plugins            # this package
  - /opt/sentinel-plugins-it4i       # site-specific sets, if any

Directories are searched in the order given, so on a name clash the one listed first wins. A plain string still behaves as before.

Install this before upgrading Sentinel to v2026.11.010

A detector that config.yaml requires but whose file is in none of the directories raises SENTINEL|detectors_missing (channel infra, severity critical). That is why plugins are step 2 and the Sentinel upgrade is step 1 only on a fresh install — on an existing instance, install the package first.

Plugins hot-reload — no restart needed after adding a new detector.


3. sentinel-overhealth

Bash
git clone git@github.com:foxik0070/sentinel-overhealth.git \
  /opt/sentinel-overhealth
cd /opt/sentinel-overhealth

# SSH key for remote access
ssh-keygen -t ed25519 -f conf/id_ed25519 -N ""
# Copy public key to all monitored nodes:
ssh-copy-id -i conf/id_ed25519.pub root@<node>

Cron deployment

Bash
# /etc/cron.d/sentinel-overhealth
*/5 * * * * root /opt/sentinel-overhealth/sentinel_orchestrator.py \
  --clean --workers 25 \
  >> /var/log/sentinel/orchestrator.log 2>&1

Systemd timer deployment

Bash
sudo cp sentinel-collector.service sentinel-collector.timer \
  /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now sentinel-collector.timer

Verify

Bash
/opt/sentinel-overhealth/sentinel_orchestrator.py --workers 5
ls -la /var/log/sentinel/logs/
# should show: availability.log capacity.log services.log ...

4. sentinel-agent (on each monitored node)

Bash
# On the monitored node
git clone git@github.com:foxik0070/sentinel-agent.git /opt/sentinel-agent
cd /opt/sentinel-agent

# Interactive install (generates config + systemd service)
sudo ./sentinel_agent_init.py -l en

# Verify
systemctl status sentinel-agent
journalctl -u sentinel-agent -f

Agent registration

Agents authenticate via a Bearer token. Generate tokens in Sentinel Web UI:
Settings → Agents → Register new agent → copy token

Set in /etc/sentinel/agent_config.yaml:

YAML
sentinel:
  url: "https://sentinel.example.com"
  api_key: "<token_from_ui>"
  interval: 30

5. sentinel-alert (optional, standalone)

Bash
git clone git@github.com:foxik0070/sentinel-alert.git /opt/sentinel-alert
cd /opt/sentinel-alert
sudo ./setup.sh

Required: GeoLite2-City.mmdb in data/ (free MaxMind account).

Verify: http://<host>:5056


6. sentinel-hw (optional, Raspberry Pi)

Bash
# On the Raspberry Pi
git clone git@github.com:foxik0070/sentinel-hw.git /home/pi/sentinel-hw
cd /home/pi/sentinel-hw
chmod +x install.sh
./install.sh

Configure config.yaml with Sentinel URL and API key.
Register device in Sentinel Web UI: Sentinel Satellites → HW Devices → Register


Firewall Rules

Service Port Protocol Direction
Sentinel Web UI 5050 TCP inbound to central server
sentinel-alert 5056 TCP inbound to alert server
sentinel-hw Web UI 5055 TCP inbound to RPi
Agent ingest 5050 TCP outbound from all nodes
Overhealth SSH 22 TCP outbound from central server
Ollama (local) 11434 TCP loopback only
hailo-ollama 8000 TCP loopback only

Nginx Configuration File
server {
    listen 443 ssl;
    server_name sentinel.example.com;

    ssl_certificate     /etc/ssl/certs/sentinel.crt;
    ssl_certificate_key /etc/ssl/private/sentinel.key;

    location / {
        proxy_pass http://127.0.0.1:5050;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # WebSocket support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 120s;
        proxy_next_upstream off;
    }

    # Serve static files directly with long cache — protects the app
    # server from connection storms and speeds up asset delivery
    location /static/ {
        alias /opt/Sentinel/sentinel/static/;
        expires 1y;
        add_header Cache-Control "public, immutable";
    }

    # Connection limit per client IP
    limit_conn addr 10;
}

Note: Socket.IO is configured to start with HTTP long-polling before upgrading to WebSocket — it works correctly behind all proxies including Cloudflare.

Trusted proxies: set TRUSTED_PROXIES in config.yaml so X-Forwarded-For is honoured only from your proxy — otherwise clients could spoof their IP and bypass IP bans.


Production Checklist

  • Set web.password_hash (bcrypt) — Sentinel logs a CRITICAL warning + UI banner on default passwords
  • Enable 2FA/TOTP for all admin accounts (Settings → 2FA)
  • Set teams_channels.webhooks (or Discord/Telegram/Opsgenie/…) for alert notifications
  • Set prometheus.scrape_token if Prometheus is used — prefer {SECRET:ENV_VAR} syntax
  • Configure ldap section if using LDAP auth
  • Deploy SSL certificate (Nginx or sentinel_init.py)
  • Set TRUSTED_PROXIES when running behind a reverse proxy
  • Set up log rotation for /var/log/sentinel/
  • Add monitored nodes to /etc/hosts on Overhealth server
  • Distribute Overhealth SSH public key to all nodes
  • Download GeoLite2-City.mmdb for sentinel-alert
  • Register all agents in Sentinel Web UI (QR code or copy-token)
  • Configure allowed_commands allowlist for Autofix
  • Set telemetry_alerts thresholds (+ per-agent thresholds where needed)
  • Test Autofix with a low-risk command before enabling auto_execute
  • Schedule backups — /api/admin/backup/download or S3/MinIO upload
  • Enable fim (File Integrity Monitoring) for critical system files
  • Run /api/admin/security_check — aim for grade A
  • Point your uptime monitor (UptimeKuma/K8s) at /healthz

Updating

Bash
# Sentinel core
cd /opt/Sentinel
git pull
make build          # builds .min.js artefacts (not stored in git)
sudo systemctl restart sentinel
# or hot-reload config+patterns+plugins without restart:
sudo systemctl kill -s HUP sentinel

# Agents (manual)
cd /opt/sentinel-agent
git pull
sudo systemctl restart sentinel-agent

# Overhealth + plugins
git -C /opt/sentinel-overhealth pull
git -C /opt/Sentinel/sentinel/plugins pull
# Plugins hot-reload via Web UI: Settings → Plugin hot-reload

Rocky Linux / RHEL — Notes

  • Python 3.13+ may need to be installed separately: dnf install python3.13
  • Run Sentinel with explicit interpreter: python3.13 -m sentinel -e
  • SELinux: may need setsebool -P httpd_can_network_connect 1 if using Nginx
  • firewalld: firewall-cmd --add-port=5050/tcp --permanent && firewall-cmd --reload