Copy page

Architecture

A tour of the code. The reasons behind the choices are in DESIGN.md.

training script ── vigily (Python) ── vigily-py (pyo3) ─┐
                                                            ├── vigily-core ── kv0 ── Lance datasets
browser ── vigily-ui (embedded) ── vigily-server (axum) ─┘

vigily-core

Module	What it holds
model.rs	Run, RunStatus, RunPatch, Series, Points, Summary, metric keys (loss[val]), JSON flattening
store.rs	The Store and RunWriter traits
memory.rs	MemoryStore, used by the tests
kv0_store.rs	Kv0Store, the adapter to kv0::tracking
catalog.rs	Catalog and Snapshot: the in-memory index of runs, flattened parameters and series summaries; experiments, metric and parameter key lists; select (filter and sort); column ids
query.rs	The run query language: tokenizer, parser, evaluation against a RunView
chart.rs	X axes, align (exact union or min/max envelope on a shared grid), aggregate (mean, min, max per group)
records.rs	The RecordStore trait; MemoryRecords; Kv0Records on kv0's records table
evals.rs	Datasets and versions, eval results, labels, feature samples, schedule runs and state, alerts, baselines; the key layout
stats.rs	Mann–Whitney U, bootstrap intervals, PSI, Kolmogorov–Smirnov, chi-squared, Jensen–Shannon
drift.rs	Suites, behaviour drift and data drift as forward passes over a suite's history

Column ids name what the runs table shows and sorts by: run:name, param:hparams.lr, metric:min:loss[val]. A metric key is the metric name plus its context values in key order.

vigily-server

router() builds the axum app; serve() refreshes the catalog, starts the 2-second refresh loop and listens. Handlers are thin: parse, take a snapshot, call core, serialise.

Endpoint	Notes
GET /api/meta	counts, experiments, metric keys with run counts, parameter keys with kinds and distinct values
GET /api/runs?q=&sort=&desc=&archived=	all matching rows; a query error is a 400 with position
GET /api/runs/{id}	the row plus the parameter tree and series with summaries
PATCH /api/runs/{id}	name, experiment, description, tags, archived
POST /api/chart	{runs, metric, x, range, points, group_by, aggregate} → {x, series[{run_id, name, group, y, last}], groups[{label, runs, mean, min, max, n}], domain, exact, missing}

Evaluation endpoints (evals_api.rs): GET /api/datasets, GET /api/datasets/{name}, GET /api/runs/{id}/items, POST /api/runs/{id}/labels, GET /api/runs/{id}/audit, GET /api/compare?a=&b=, GET /api/schedules, GET /api/schedules/{name}/runs[/{id}], GET /api/suites, GET /api/suites/{key}, PUT /api/suites/{key}/baseline, GET /api/suites/{key}/data-drift, GET /api/alerts, POST /api/alerts/{id}. The scheduler (scheduler.rs), drift monitor and webhook (monitor.rs) and the configuration file (config.rs) start with serve().

Tracing endpoints (traces_api.rs, model and storage in vigily-core/src/traces.rs): POST /api/traces/ingest, POST /v1/traces (OTLP/HTTP JSON), GET /api/traces?run=&model=&status=&since=&until=&q=, GET /api/traces/{id}, DELETE /api/traces/{id}.

Raw points of finished series are kept in a bounded cache (8 million points), so zooming re-samples without reading storage. Anything that is not /api/... is the embedded web app; client-side routes get index.html.

vigily-ui

Path	What it holds
src/index.css	Tailwind v4 tokens for both themes (see brand/DESIGN.md), uPlot and browser-surface theming
src/lib/api.ts	API types and the fetch client
src/lib/columns.ts	Runs-table column definitions and the default column choice
src/lib/format.ts	Numbers (four significant digits, integers in full for parameters), durations, times
src/components/metric-chart.tsx	The chart: uPlot, the readout, zoom, smoothing, outliers, keyboard
src/components/query-input.tsx	The filter field: completion, syntax help, error position
src/pages/*.tsx	Runs, Metrics, Run, Experiments
scripts/qa.mjs	Screenshots in both themes at two widths, computed checks, a keyboard-only pass

build.rs embeds dist/ into the binary; build the app before cargo.

vigily-py

src/lib.rs is the native module vigily._native (Repo, RunHandle); values cross as JSON text. python/vigily/__init__.py is the public API: Run (parameters by item assignment, track, metadata properties, finish, context manager, finishing at exit) and Repo (reading back). One native repository handle is shared per location and process.

Tests

• cargo nextest run: core unit tests (model, query, chart, catalog against MemoryStore, and a round trip through Lance in a temporary directory) and server tests that drive the router in memory.
• uv run --no-sync pytest: the SDK against the built extension, including exit handling in subprocesses.
• npm run qa in crates/vigily-ui: the browser pass against a running server.

Scale test

bench/scale.py logs synthetic runs; then time the server:

uv run --no-sync python bench/scale.py --runs 2000 --points 2000 --repo /tmp/vigily-scale
vigily up --repo /tmp/vigily-scale --port 7601

bench/scale.sh starts the server and times the API (UP_ARGS=... passes options to vigily up). 2 000 runs in 20 experiments, three series each (loss[train] with 2 000 points, loss[val] and accuracy[val] with 200), 4.8 million points, about 1.1 GB on disk. Release build, 20-core Linux workstation, local SSD. Logging, start-up and memory with kv0 8c33ef2 (the tracking additions, now merged into kv0 main); request times with d2fd932:

	Start as logged (default)	Optimizing before the first refresh (earlier default)
Logging the 2 000 runs, one after another	107 s
Start-up (first catalog refresh)	0.27 s	3.41 s (optimize included)
GET /api/meta	0.002 s	0.002 s
GET /api/runs, all 2 000 rows	0.008 s	0.007 s
GET /api/runs, filtered and sorted by a metric	0.003 s	0.005 s
POST /api/chart, 200 runs, cold	0.103 s	0.061 s
POST /api/chart, 200 runs, cached	0.016 s	0.017 s
POST /api/chart, 200 runs, zoomed	0.006 s	0.006 s
Server memory (RSS)	244 MB	1 019 MB (right after optimizing)

The catalog reads each table once and the stored series statistics, and no metric points, so start-up no longer grows with the number of points. kv0 compacts small files in the background while runs are logged, which keeps logging at a steady rate (100 s for this repository with kv0 3a88fe0).

Memory

A full optimize (sorting points by series, indexing them, folding superseded run rows) makes cold charts faster and keeps the runs table small, but it is the most memory-hungry thing the server does. Measured on the repository above, release build, kv0 3a88fe0 (with its lower-memory optimize), resident memory of vigily up:

	glibc allocator	glibc, memory returned after each optimize (default)	mimalloc
After start-up	109 MB	110 MB	348 MB
After the first background optimize	907 MB	706 MB	574 MB
After three optimizes	911 MB	707 MB	514 MB
Peak during an optimize	907 MB	1 009 MB	990 MB
With --optimize-at-start: start-up time, then memory	3.3 s, 901 MB	3.2 s, 773 MB	3.2 s, 836 MB

Decisions, from these numbers:

• No optimize at start (the default): start-up is 0.28 s instead of 3.2 s, and the server stays near 110 MB until the first background optimize, 15 minutes later. --optimize-at-start brings it back for anyone who wants indexed charts from the first request.
• Memory is returned after each optimize (malloc_trim on Linux with glibc): the server settles at about 700 MB instead of keeping the 900 MB it reached. mimalloc settles lower still (about 520 MB) but costs 240 MB more at rest, so the default stays glibc.
• The rest (about 600 MB above rest) is kv0's and Lance's caches after an optimize; noted as kv0 work. --optimize-every sets the interval (default 15m) and --no-optimize turns it off, for small machines.

Before kv0's tracking additions (per-run reads, statistics computed from every point, no automatic compaction), the same repository took 250 to 386 s to start uncompacted and 10.4 s after vigily optimize, and logging it took 27 minutes.