Observability

FrogDB exposes Prometheus metrics, OTLP export, distributed tracing, HTTP health/status endpoints, structured logging, and optional USDT probes. The metric catalog itself lives in Metrics Reference — this page covers how to enable and configure each observability channel. For interactive diagnostics (slowlog, LATENCY, hot-key detection, the debug web UI), see Diagnostics.

Prometheus metrics

FrogDB exports metrics in Prometheus text format (version=0.0.4) on the HTTP server:

GET http://<host>:9090/metrics

The listener is controlled by [http], not [metrics] — [metrics] only configures OTLP export (below). /metrics shares the same HTTP server as the health, status, and debug endpoints:

[http]
enabled = true
bind = "127.0.0.1"
port = 9090

/metrics is always unauthenticated and always returns 200, regardless of http.token. See Metrics Reference for the complete, generated list of exported metrics — this page does not duplicate metric names.

OTLP export

In addition to the pull-based /metrics endpoint, FrogDB can push metrics to any OpenTelemetry-compatible collector via OTLP/gRPC on an interval:

[metrics]
otlp-enabled = false
otlp-endpoint = "http://localhost:4317"
otlp-interval-secs = 15

Setting	Type	Default	Description
otlp-enabled	boolean	false	Enable OTLP metrics export
otlp-endpoint	string	http://localhost:4317	gRPC endpoint of the OTLP collector
otlp-interval-secs	integer	15	Export interval in seconds

This otlp-endpoint is distinct from tracing.otlp-endpoint below — metrics and traces are exported independently and can point at different collectors.

See Configuration Reference → Metrics.

Distributed tracing

FrogDB supports OpenTelemetry distributed tracing with configurable span granularity:

[tracing]
enabled = false
otlp-endpoint = "http://localhost:4317"
sampling-rate = 1.0
service-name = "frogdb"
scatter-gather-spans = false  # Child spans per shard for MGET/MSET
shard-spans = false           # Spans inside shard workers
persistence-spans = false     # WAL/snapshot spans

Tracing is disabled by default; sampling-rate defaults to 1.0 (sample everything) once enabled — lower it in production to control span volume.

Span hierarchy

When tracing is enabled, FrogDB emits spans at three levels:

1. Request span — created at the connection handler, covers the full command lifecycle
2. Shard execution span — created in the shard worker, covers command processing
3. Store operation span — individual data structure operations

Semantic conventions

FrogDB uses OpenTelemetry database semantic conventions:

Attribute	Example
db.system	frogdb
db.operation	GET, SET, MGET
db.statement	SET mykey myvalue
frogdb.shard_id	3
frogdb.connection_id	42
frogdb.key_count	1

Recent traces

FrogDB keeps the last tracing.recent-traces-max traces (default 100) in memory for debugging. These feed the recent-traces view in the debug web UI.

See Configuration Reference → Distributed Tracing.

Health and status endpoints

All endpoints below are served on the [http] listener (default port 9090) and are unauthenticated regardless of http.token:

Endpoint	Description	Response
GET /health/live	Liveness — process is running	200 if alive, 503 after shutdown begins
GET /health/ready	Readiness — accepting commands	503 until the server signals ready, 200 once ready, 503 again during shutdown
GET /healthz	Alias for /health/live	Same as /health/live
GET /readyz	Alias for /health/ready	Same as /health/ready
GET /status/json	Full server status (JSON)	200 with the status document

/health/ready starts not ready and only flips to 200 once the server finishes startup — this is intentional for readiness-gated rollouts (don’t route traffic to a node that hasn’t finished recovery). /status/json is documented in full in Diagnostics → Server status; it’s listed here only to complete the endpoint table.

Bearer-token-protected routes (/debug/*, /admin/*) only require Authorization: Bearer <token> when http.token is configured — otherwise they’re open on whatever http.bind allows (127.0.0.1 by default, so loopback-only out of the box). See Diagnostics → Debug web UI for the full auth story.

Kubernetes probe configuration:

livenessProbe:
  httpGet:
    path: /health/live
    port: 9090
  initialDelaySeconds: 5
readinessProbe:
  httpGet:
    path: /health/ready
    port: 9090
  initialDelaySeconds: 5

INFO sections

INFO returns the standard Redis-compatible sections plus FrogDB extras. Default sections (returned by a bare INFO):

INFO server|clients|memory|persistence|stats|replication|cpu|keyspace|ratelimit

ratelimit is a FrogDB extension: it only renders when per-user ACL rate limiting is active (fields ratelimit_users_configured, ratelimit_total_commands_rejected, ratelimit_total_bytes_rejected — see Security).

Extra sections available via INFO all / INFO everything:

INFO commandstats|errorstats|latencystats|latency_baseline|tiered|keysizes

For the meaning of the standard Redis fields within each section, see the Redis INFO documentation; this page documents only where FrogDB’s section list differs.

Logging

FrogDB uses structured logging with configurable output format and level:

[logging]
level = "info"                  # trace, debug, info, warn, error
format = "pretty"               # "pretty" or "json"
output = "stdout"               # "stdout", "stderr", or "none"
per-request-spans = false       # Per-request tracing spans
file-path = "/var/log/frogdb/frogdb.log"

[logging.rotation]
max-size-mb = 100
frequency = "daily"             # "daily", "hourly", or "never"
max-files = 5

per-request-spans adds a tracing span around every request; enabling it adds measurable CPU overhead and is intended for short diagnostic windows rather than continuous production use. Rotation requires file-path to be set and at least one of max-size-mb or frequency to be active.

Runtime level changes

The log level can be changed at runtime without a restart:

redis-cli CONFIG SET loglevel debug

loglevel accepts the same five levels as logging.level.

See Configuration Reference → Logging.

USDT / DTrace probes

FrogDB includes USDT (User Statically Defined Tracing) probes for low-overhead, production-safe tracing on macOS and Linux. They compile to no-ops unless the server is built with --features usdt-probes.

Available probes

Probe	Arguments	Description
command__start	command, key, conn_id	Fired when a command begins execution
command__done	command, latency_us, status	Fired when a command completes
shard__message__sent	from_shard, to_shard, msg_type	Inter-shard message dispatched
shard__message__received	shard, msg_type, queue_depth	Shard received a message
key__expired	key, shard_id	Key expired via TTL
key__evicted	key, shard_id, policy	Key evicted due to memory pressure
memory__pressure	used, max, action	Memory pressure event
wal__write	shard_id, key, bytes	WAL write for a key
scatter__start	command, shard_count, txid	Scatter-gather operation started
scatter__done	command, latency_us, shard_count	Scatter-gather operation completed
pubsub__publish	channel, subscribers	Message published to a channel
connection__accept	conn_id, addr	New client connection accepted

Probes are defined in frogdb-server/crates/core/src/probes.rs under the frogdb USDT provider.

Usage examples

# List all FrogDB probes (macOS)
sudo dtrace -l -n 'frogdb*:::'

# Trace all commands
sudo dtrace -n 'frogdb*:::command-start { printf("%s %s\n", copyinstr(arg0), copyinstr(arg1)); }'

# Measure command latency
sudo dtrace -n 'frogdb*:::command-done { printf("%s %d us\n", copyinstr(arg0), arg1); }'

# Watch memory pressure events
sudo dtrace -n 'frogdb*:::memory-pressure { printf("used=%d max=%d action=%s\n", arg0, arg1, copyinstr(arg2)); }'

# With bpftrace (Linux)
sudo bpftrace -e 'usdt:./frogdb-server:frogdb:command__start { printf("%s %s\n", str(arg0), str(arg1)); }'

Key metrics to watch

A short, qualitative starting point — treat the thresholds as examples to tune for your workload, not SLAs:

Metric	Watch for
frogdb_memory_used_bytes / frogdb_memory_maxmemory_bytes	Approaching the configured limit
histogram_quantile(0.99, rate(frogdb_commands_duration_seconds_bucket[5m]))	Sustained increases in tail command latency
frogdb_persistence_errors_total	Any increase (WAL/snapshot I/O trouble)
frogdb_connections_rejected_total	Any increase (hitting maxclients or auth failures)

All four metric names are validated against the generated Metrics Reference catalog.

See also

Metrics Reference Complete, generated list of Prometheus metrics

Diagnostics Slowlog, LATENCY, hot-key detection, and the debug web UI

Configuration Reference All [http], [metrics], [tracing], and [logging] keys

Clustering Cluster topology and coordination