Observability

msb-metrics is a separate binary that continuously exports sandbox CPU, memory, disk, and network metrics to an OpenTelemetry-compatible backend. Run one collector on each host alongside microsandbox. For a one-time reading, use msb metrics. To read metrics in application code, use Sandbox::metrics().

How it works

All three interfaces read the same host-local registry and can run together. This page covers continuous export with msb-metrics.

Setup

Install

msb-metrics is not included with the main msb installer. Download it from the latest release and place it on your PATH.

Requirements

The collector reads the shared-memory registry directly:

Run it as the same Unix user as msb. The registry is owner-only.
Use the same $MSB_HOME as msb. If needed, pass --msb-home; the default is ~/.microsandbox.
Run one collector per host. Each registry contains only that host’s sandboxes.

Quick start

Start the collector

gRPC
HTTP

Use port 4317 for most local collectors and sidecars.

msb-metrics otel --endpoint=http://localhost:4317

Use a complete metrics endpoint, commonly on port 4318.

msb-metrics otel --endpoint=https://example.com/otlp/v1/metrics --protocol=http

Run a sandbox

msb run alpine

Check your backend

Metrics appear after the next collection and export intervals.

Backends

For production, use Grafana Alloy or another local collector as a forwarder. You can also export directly to a hosted backend. End-to-end setup walkthroughs live under Examples:

Grafana Alloy

Forward metrics through a local Alloy process.

Grafana Cloud

Export directly to Grafana Cloud.

Prometheus

Use Prometheus’s native OTLP receiver.

OpenTelemetry Collector

Run a local OpenTelemetry Collector.

Datadog

Export through the Datadog Agent.

Metrics

Sandbox metrics

Metrics use the microsandbox.* namespace. The table shows only the suffix.

Suffix	Unit	Description
`cpu.utilization`	`1`	vCPU-seconds per wall-second. A fully used 2-vCPU sandbox reports `2.0`.
`memory.usage`	`By`	Resident memory.
`memory.limit`	`By`	Configured guest memory limit.
`disk.bytes_read`	`By`	Cumulative bytes read by the sandbox process.
`disk.bytes_written`	`By`	Cumulative bytes written.
`network.bytes_received`	`By`	Cumulative bytes from runtime to guest.
`network.bytes_sent`	`By`	Cumulative bytes from guest to runtime.
`upper.used`	`By`	Used space on the OCI upper filesystem when reporting is available.
`upper.free`	`By`	Available space on the OCI upper filesystem when reporting is available.
`upper.host_allocated`	`By`	Host-allocated space for the writable upper image when observable.
`uptime`	`s`	Sandbox uptime.

Disk and network totals are gauges containing absolute cumulative values. Use rate() in PromQL for throughput.

Attributes

Every datapoint identifies its source and sandbox.

Attribute	Default	Description
`service.name`	`microsandbox`	Service resource attribute. Override with `--resource`.
`service.instance.id`	Hostname	Host identity, when available. Override with `--resource`.
`sandbox.name`	on	Sandbox name.
`sandbox.id`	on	Sandbox catalog ID.
`sandbox.run_id`	off	Enable with `--emit-run-id`; creates a series after every restart.
`sandbox.pid`	off	Enable with `--emit-pid`; creates a series after every restart.

Sandbox labels are also included by default. Disable all labels with --no-labels, or repeat --exclude-label-key <key> to omit specific keys from metrics without removing them from the sandbox catalog.

High-cardinality labels such as user.id, sandbox.run_id, and sandbox.pid can increase active-series counts and backend costs. Include only the dimensions you query.

Label lookup is best-effort. If the catalog is unavailable, the collector exports that sample without labels and tries again on the next interval.

Labels in queries

Prometheus replaces dots in OpenTelemetry names with underscores. For example, user.id becomes user_id, and microsandbox.cpu.utilization becomes microsandbox_cpu_utilization.

# CPU by user
sum by (user_id) (microsandbox_cpu_utilization)

# One user's sandboxes
microsandbox_cpu_utilization{user_id="alice"}

# CPU by user in one tenant
sum by (user_id) (microsandbox_cpu_utilization{tenant="acme"})

Restarts and stops

Disk and network totals reset when a sandbox restarts. PromQL rate() handles these resets, though a brief negative interval can appear before the next sample. Stopped sandboxes stop producing samples, so their series become stale. A crashed sandbox behaves the same way: readers retire its abandoned registry entry on the next collection. See Sandbox metrics for the underlying lifecycle behavior.

Operations

Collector health

The collector exports its own health metrics through the same OTLP pipeline.

Suffix	Type	Description
`collector.exports.success`	counter	Successful exports since startup.
`collector.exports.failure`	counter	Failed exports since startup.
`collector.collections.dropped`	counter	Collections dropped because the buffer was full.
`collector.last_success_timestamp`	gauge	Unix time of the latest successful export.

All use the OTel scope microsandbox-metrics-collector. Exports are flowing

rate(microsandbox_collector_exports_success_total[1m])

Export failure ratio

rate(microsandbox_collector_exports_failure_total[5m])
  /
clamp_min(
  rate(microsandbox_collector_exports_success_total[5m]) +
  rate(microsandbox_collector_exports_failure_total[5m]),
  1
)

No successful export for five minutes

time() - microsandbox_collector_last_success_timestamp_seconds > 300

Outages and retries

Failed exports return to the front of the buffer and retry with capped exponential backoff, from --flush-interval up to 32 times that interval. Oldest collections are dropped when the buffer fills. The collector reports the loss through collector.collections.dropped and continues running.

Scaling

Collector memory grows with the number of active sandboxes and buffered collections. Lower --max-buffered to reduce memory use during backend outages, at the cost of dropping samples sooner. Monitor collector.collections.dropped when choosing a buffer size.

Shutdown

On SIGINT or SIGTERM, the collector stops collecting, exports buffered samples, closes the OTLP transport, and exits. --export-timeout bounds the final export.

Troubleshooting

EACCES opening shared memory

Run msb-metrics as the Unix user that owns the registry.

No sandboxes appear

Confirm a sandbox is running and both processes use the same $MSB_HOME. Pass --msb-home explicitly if necessary. Use --log-level=debug to see the registry name.

The backend returns 401, 403, or 422

Check authentication headers and the endpoint protocol. gRPC commonly uses port 4317; HTTP needs the complete metrics URL expected by your backend, often ending in /v1/metrics.

Restarts create new series

Disable --emit-run-id and --emit-pid to keep one series per sandbox identity across restarts.

Reference

Use msb-metrics <command> --help to see the same options in your terminal.

msb-metrics otel

Export metrics over OTLP.

msb-metrics otel --endpoint=<URL>
                 [--protocol=grpc|http]
                 [--compression=none|gzip]
                 [--ca-cert=<path>]
                 [--header=KEY=VALUE]...
                 [--resource=KEY=VALUE]...
                 [--emit-run-id] [--emit-pid]
                 [--collect-interval=<dur>]
                 [--flush-interval=<dur>]
                 [--max-buffered=<n>]
                 [--export-timeout=<dur>]
                 [--msb-home=<path>]
                 [--no-labels]
                 [--exclude-label-key=<key>]...

Connection

Flag	Default	Description
`--endpoint`	required	OTLP endpoint. For HTTP, provide the complete metrics URL.
`--protocol`	`grpc`	`grpc` or `http`.
`--compression`	`none`	`gzip` or `none`; available for gRPC only.
`--ca-cert`	none	Additional PEM-encoded CA certificate; available for gRPC only.
`--header`	none	Repeatable `KEY=VALUE` header, commonly used for authentication.

Attributes

Flag	Default	Description
`--resource`	none	Override or add a resource attribute; repeatable.
`--emit-run-id`	off	Include `sandbox.run_id`.
`--emit-pid`	off	Include `sandbox.pid`.
`--no-labels`	off	Do not include sandbox labels.
`--exclude-label-key`	none	Omit one label key from metrics; repeatable.

Collection

Flag	Default	Description
`--collect-interval`	`1s`	How often to read the registry.
`--flush-interval`	`10s`	Scheduled export interval.
`--max-buffered`	`60`	Collection buffer limit per exporter.
`--export-timeout`	`30s`	Timeout for one export.
`--msb-home`	`$MSB_HOME`, then `~/.microsandbox`	Used to locate the registry.

msb-metrics stdout

Print one human-readable line per snapshot without an OTLP receiver.

msb-metrics stdout [--collect-interval=<dur>]
                   [--flush-interval=<dur>]
                   [--max-buffered=<n>]
                   [--export-timeout=<dur>]
                   [--msb-home=<path>]
                   [--no-labels]
                   [--exclude-label-key=<key>]...

The output is intended for inspection, not production parsers.

Global flags

Flag	Default	Description
`--log-level`	`info`	`error`, `warn`, `info`, `debug`, or `trace`. `RUST_LOG` takes precedence.
`--log-format`	`text`	`text` or newline-delimited `json`.

Getting Started

Sandboxes

Networking

Deep dive

Troubleshooting

How it works

Setup

Install

Requirements

Quick start

Backends

Grafana Alloy

Grafana Cloud

Prometheus

OpenTelemetry Collector

Datadog

Metrics

Sandbox metrics

Attributes

Labels in queries

Restarts and stops

Operations

Collector health

Outages and retries

Scaling

Shutdown

Troubleshooting

Reference

​How it works

​Setup

​Install

​Requirements

​Quick start

​Backends

Grafana Alloy

Grafana Cloud

Prometheus

OpenTelemetry Collector

Datadog

​Metrics

​Sandbox metrics

​Attributes

​Labels in queries

​Restarts and stops

​Operations

​Collector health

​Outages and retries

​Scaling

​Shutdown

​Troubleshooting

​Reference

How it works

Setup

Install

Requirements

Quick start

Backends

Metrics

Sandbox metrics

Attributes

Labels in queries

Restarts and stops

Operations

Collector health

Outages and retries

Scaling

Shutdown

Troubleshooting

Reference