Prometheus Monitoring
This deliverable adds Prometheus-based monitoring for the Python Wactorz runtime.
Scope
Included:
- Python REST API metrics at
/metrics - actor health and runtime metrics from the Python registry
- process/runtime metrics from the Python process
- Prometheus in Docker Compose, and Alertmanager to deliver its alerts
- optional Mosquitto availability probe controlled by
.env
What Is Monitored
Python app
Prometheus scrapes the Python REST service and records:
- HTTP request counts
- HTTP response counts by status
- HTTP request duration
- running actor count
- actor state
- actor heartbeat age
- actor restart count
- actor messages processed
- actor errors
- actor tasks completed, failed, and timed out (counted among the failed too)
- LLM input tokens
- LLM output tokens
- LLM cost in USD
- process/runtime metrics exported by
prometheus_client
And, for what the dashboard does not show:
| Metric | What it says |
|---|---|
wactorz_mqtt_connected |
1 while the server's broker connection is up |
wactorz_mqtt_outbox_queued |
Messages in memory waiting to be sent to the broker |
wactorz_mqtt_outbox_backlog |
Stored messages waiting on disk for room in that queue |
wactorz_mqtt_publish_failures_total |
Publishes that failed on a live connection and were held to retry |
wactorz_mqtt_outbox_dropped_total |
Messages discarded because the outbox was full |
wactorz_mqtt_outbox_discarded_total |
Messages given up on: unsendable, expired undelivered, or failing every try |
wactorz_actor_mailbox_depth{actor_name} |
Messages waiting in an actor's mailbox |
wactorz_actor_messages_refused_total{actor_name} |
Messages a full mailbox had no room for: notifications dropped, anything else refused after a wait |
wactorz_actor_handling_seconds{actor_name} |
How long an actor has been on the message it is handling; 0 when idle. An actor's heartbeat carries on while it waits on one message, so this is what shows it stuck |
wactorz_actor_queue_wait_seconds{actor_name} |
Time from a message being made to the actor taking it: in the mailbox, or with its sender for room in a full one. An actor takes one message at a time, so a slow handler shows here as the wait of everything behind it, and a high wait beside wactorz_actor_messages_refused_total is a full mailbox |
wactorz_actor_message_duration_seconds{actor_name} |
Time an actor's handler took over one message. A generated agent hands each task to a task of its own, so its handling time is in wactorz_agent_task_duration_seconds instead |
wactorz_ha_requests_total{command,outcome} |
Home Assistant WebSocket requests by command (get_states, call_service, the registry lists, subscribe_events) and how they ended: ok, error or timeout. The events a subscription then waits for are not requests, and are not counted |
wactorz_ha_request_duration_seconds{command} |
Time from a Home Assistant request to its answer, or to giving up on it |
wactorz_ha_connect_duration_seconds / wactorz_ha_connect_failures_total |
Time to open and authenticate a WebSocket to Home Assistant, and the attempts that failed (unreachable, or a refused token) |
wactorz_event_loop_lag_seconds |
Histogram of how long the event loop took to run a callback it was asked to run at once. Every agent shares the loop, so a long lag is all of them waiting |
wactorz_nodes{state} |
Edge nodes that are up and down |
wactorz_node_up{node} |
1 while a node's heartbeat is recent |
wactorz_node_heartbeat_age_seconds{node} |
Seconds since a node's last heartbeat |
wactorz_node_agents{node} |
Agents a node reported running |
wactorz_node_info{node,version,runtime} |
The version and runtime a node reported |
wactorz_llm_requests_total{provider,outcome} |
LLM requests by how they ended: ok, unavailable (the provider kept failing through every retry) or error (anything else: a rejected request, a bad key) |
wactorz_llm_request_duration_seconds{provider} |
Time from a request to its answer or failure, retries included; for a streamed answer, to its last chunk |
wactorz_http_requests_total{server,method,route} |
HTTP requests, by server: rest (this interface) or dashboard (the dashboard's server, chat included). route is the registered pattern, never the path asked for |
wactorz_http_request_duration_seconds{server,method,route} |
Time a request took to answer. A WebSocket is counted but not timed, since it lasts as long as the connection |
wactorz_ws_connections |
Dashboard WebSocket connections open now |
wactorz_chat_first_reply_seconds{kind} |
Time from a chat message reaching the server to the first words of its reply, what a person notices while an answer streams in. kind is where it went: command, local (an agent in this process, main among them), remote (an agent on a node) or unrouted |
wactorz_chat_turn_duration_seconds{kind} |
Time from a chat message reaching the server to its reply being complete. A turn the person stops is not counted |
wactorz_chat_log_rows_dropped_total |
Chat turns shown on the page but not kept in the chat log, because its writer had more queued than it holds |
wactorz_agent_task_duration_seconds{agent,outcome} |
Time a generated agent's handle_task took, by how it ended: completed, failed or timed_out |
wactorz_agent_process_duration_seconds{agent} |
Time one cycle of a generated agent's process() took, whether it returned or raised |
wactorz_agent_process_timeouts_total{agent} |
process() cycles still running when their time ran out |
wactorz_actor_slow_restarts{actor_name,node} |
1 for an agent that crashed so often in a row that its supervisor now restarts it slowly; node is empty for one on this server, and a node's come from its heartbeat |
wactorz_llm_spend_usd{period} |
Spent on models so far this spend period (daily, weekly or monthly), as Settings shows it |
wactorz_llm_spend_limit_usd{period} |
The spend limit for the period; absent when none is set. At it, model requests pause until the period ends or the limit is raised |
wactorz_model_score{model,side,metric} |
Each side of a model's shadow trial, active and shadow, on the figure the promotion policy compares (f1, macro_f1, mae, or the policy's), from the labels so far; set at every check of the trial and dropped when it ends |
wactorz_model_shadow_agreement{model} |
The share of a trial's pairs both sides answered alike |
wactorz_model_drift{model,test,column} |
A drift test's statistic on a column of the model's readings at the drift agent's last check: PSI, KS's D, or how many changes ADWIN found in the scores (docs/design/drift.md) |
wactorz_model_drift_alarm{model,test,column} |
1 while that test is over its threshold on that column, so an alert rule needs no threshold of its own |
wactorz_model_train_seconds{model} |
How long the last training run of the model took |
wactorz_model_train_runs_total{model,outcome} |
Training runs by how they ended: ok or failed |
A request counts once however many attempts it took, and one the caller cancelled is not counted. The wactorz_agent_* series cover the agents that are running: an agent's go when it stops, so one-off agents do not accumulate, and one started again under the same name begins afresh, which Prometheus reads as a counter reset. Main forgets a node that stays silent, so the nodes named in your deploy targets are reported as down until they are heard from, rather than disappearing; a node started by hand shows only while main knows it.
The app exposes these at GET /metrics on two ports:
- the dashboard's,
8888by default (MONITOR_PORT), however Wactorz is started — the CLI, a library call, the Home Assistant add-on; - the REST interface's,
8000by default, whenINTERFACE=rest, as in the compose stacks, whose Prometheus scrapes this one.
Both pages hold the same metrics. Once API_KEY is set, both ask for it, and a scraper presents it as Authorization: Bearer.
Mosquitto
Mosquitto is an optional Prometheus target.
It is monitored with the Blackbox Exporter:
- Mosquitto: TCP connect probe to
mosquitto:1883
This is availability monitoring, not deep service-specific exporter telemetry.
Authentication
/metrics is served by the API on port 8000, and once API_KEY is set every
route there except the health probes requires it. An unauthenticated scrape gets 401
and the target goes down with nothing written to the log, so a keyed install
loses its metrics silently unless the scrape carries the key.
The endpoint accepts either X-API-Key or Authorization: Bearer.
Bundled Prometheus needs no setup: the compose stack passes API_KEY
through and the scrape config is rendered with the header when a key is set.
Your own Prometheus must be told:
- job_name: wactorz-python
static_configs:
- targets: ["wactorz:8000"]
authorization:
type: Bearer
credentials: 'your-api-key'
Prefer credentials_file where the scraper reads its configuration from a
shared location — Prometheus redacts credentials from its own config API, but
a file keeps the key out of the config entirely.
⚠ The key is the only credential there is, so a scraper holding it can reach
every API route, not only /metrics. That is reasonable where Prometheus runs
beside the app and inside the same trust boundary; it is worth more thought
across a network.
Environment Flags
Add or adjust these in .env:
PROMETHEUS_EXTERNAL_PORT=9090
PROMETHEUS_SCRAPE_INTERVAL=15s
PROMETHEUS_PYTHON_TARGET=wactorz-python
PROMETHEUS_MONITOR_MOSQUITTO=0
REST_EXTERNAL_PORT=8000
Notes:
PROMETHEUS_PYTHON_TARGETchooses the host or service name Prometheus scrapes for Python metrics;REST_EXTERNAL_PORTis appended as the scrape port.- If Wactorz runs in Compose, use the service name such as
wactorz-pythonorwactorz. - If Wactorz runs from the terminal on the host, use
host.docker.internal. PROMETHEUS_MONITOR_MOSQUITTO=1enables the Mosquitto TCP probe.
Docker Compose
Main stack
Prometheus, Alertmanager and the Blackbox Exporter belong to the python profile:
docker compose --profile python up -d
The full profile adds Home Assistant and does not start them; give both profiles to have both:
docker compose --profile python --profile full up -d
Development stack
There they belong to the app profile:
docker compose -f compose.dev.yaml --profile app up -d
Prometheus is available at:
http://localhost:${PROMETHEUS_EXTERNAL_PORT:-9090}
Simple Ways To Run It
1. Wactorz in Compose, Prometheus in Compose
Leave:
PROMETHEUS_PYTHON_TARGET=wactorz-python
Then run:
docker compose --profile python up -d prometheus alertmanager blackbox-exporter wactorz-python
2. Wactorz from terminal, Prometheus in Compose
Set:
PROMETHEUS_PYTHON_TARGET=host.docker.internal
Start Wactorz locally in REST mode, then run:
docker compose --profile python up -d --no-deps prometheus alertmanager blackbox-exporter
This starts only the monitoring containers and points Prometheus at the Wactorz process running on your host.
Verification
1. Check Python metrics directly
curl -fsS http://localhost:8000/metrics | head
# With API_KEY set:
curl -fsS -H "Authorization: Bearer $API_KEY" http://localhost:8000/metrics | head
# Without the REST interface, from the dashboard's port:
curl -fsS -H "Authorization: Bearer $API_KEY" http://localhost:8888/metrics | head
You should see Prometheus-formatted output such as wactorz_actors_total, wactorz_http_requests_total, and process metrics.
2. Check Prometheus targets
Open:
http://localhost:9090/targets
Expected:
prometheusisUPwactorz-pythonisUP- optional probe targets appear only when enabled in
.env
3. Check optional probes
When PROMETHEUS_MONITOR_MOSQUITTO=1, Prometheus should show a mosquitto-blackbox target.
If a flagged dependency is not running, that target will correctly show as failing.
Alert Rules
Basic Prometheus alert rules are included for:
- Python app down
- actor heartbeat stale
- an actor on one message for over 30 minutes
- the event loop blocked for more than 5 seconds
- the broker connection lost for 2 minutes
- more than 100 outgoing messages waiting for 10 minutes
- outgoing messages dropped or given up on
- an edge node down for 5 minutes
- more than half the requests to an LLM provider failing for 10 minutes
- an agent recording more than 10 errors in 15 minutes
- an agent restarted slowly after repeated crashes, for 5 minutes, on the server or a node
- model spend past 80% of its limit, and at the limit (critical: model requests are paused)
- optional dependency probe failing
They live in infra/prometheus/alerts.yml. Prometheus evaluates them and hands the ones that fire to Alertmanager, which the compose stack starts beside it.
Where Alerts Go
Nowhere, until you say. Out of the box Alertmanager lists what is firing on its own page and delivers nothing:
http://localhost:${ALERTMANAGER_EXTERNAL_PORT:-9093}
To be told about an alert, name a webhook in .env:
ALERT_WEBHOOK_URL=https://example.org/hooks/wactorz
# Optional. Sent as "Authorization: Bearer <token>".
ALERT_WEBHOOK_TOKEN=
and restart the service (docker compose --profile python up -d alertmanager). Every alert, and its resolution, is then sent there as an HTTP POST carrying Alertmanager's JSON. The method and the body are Alertmanager's and are not settings; the token is the one credential it sends. Alerts with the same name are grouped into one notification, sent 30 seconds after the first fires and repeated every 4 hours while it lasts.
The address and the token are written to files inside the container, in memory, and the configuration names those files, so neither shows on Alertmanager's status page.
For anything a single webhook cannot express, such as email, Slack, several receivers, custom headers or routing by severity, write your own configuration to infra/alertmanager/alertmanager.yml. It is used exactly as written, the two settings above are then not read, and the file is ignored by git because it usually holds a credential. The Alertmanager documentation describes the format.
Alertmanager's page has no login and can silence an alert, so it is published to this host only, like Prometheus.