Experiment tracking evaluation drift

Log every run, evaluate it every night, and know by morning what drifted.

vigily is a self-hosted experiment tracker that keeps watching after you stop. Log training runs and evaluations from Python, compare them in a fast web app, run evaluations on a schedule, and get the evidence when behaviour or inputs move. One binary; your storage.

Get started Concepts GitHub

The runs table: summary figures, loss sparklines and four selected runs lettered A to D — **Fig. 1** The runs table: what is training, what finished, what failed, and a loss curve per run on one scale.Synthetic data from `examples/demo.py`

The runs table in the dark theme — **Fig. 1** The runs table: what is training, what finished, what failed, and a loss curve per run on one scale.Synthetic data from `examples/demo.py`

Track runs and long curves

Filter like you think lr<0.01 status:finished loss[val].min<0.25 Reading Fig. 2 Labels sit at the line ends instead of a legend; the run you are looking at is in ink blue.

A small Python SDK logs parameters and metric series. The runs table filters and sorts thousands of runs with a short query language; charts ask for about as many points as they have pixels, so 200 000 points draw as fast as 200. Select runs and they become A, B, C, D, the letters they keep when you compare them.

Fig. 2 Training loss of three runs by step, labelled at the line ends. Illustrative data.

Evaluate on versioned datasets

Datasets are content-addressed and immutable. evaluate() runs your target in your process with exact match, regex, numeric, JSON schema or any Python function as the evaluator. Two runs compare item by item, regressions first.

Per-item results of an evaluation run with pass and fail verdicts — **Fig. 3** One evaluation run, item by item, with its verdicts.

Per-item results of an evaluation run in the dark theme — **Fig. 3** One evaluation run, item by item, with its verdicts.

Keep watch overnight

Schedules run your evaluation commands on cron in your time zone. Each run is compared with its baseline item by item, and a drift raises an alert in the app and to a webhook, with the items that flipped from pass to fail.

A drift suite: exact match per run below its baseline band, with the items that flipped — **Fig. 4** A nightly evaluation that drifted: the score per run against its baseline band.

A drift suite in the dark theme — **Fig. 4** A nightly evaluation that drifted: the score per run against its baseline band.

One binary, your storage

The web app, the API, the scheduler and the drift monitor are one Rust binary.

# train.py
from vigily import Run

run = Run(experiment="cifar10")
run["hparams"] = {"lr": 3e-3}
for step, loss in train():
    run.track(loss, name="loss", step=step)
run.finish()

# serve the app, the API and the schedules
vigily up --config vigily.toml

# list runs in a terminal
vigily runs "exp:cifar10 loss[val]<0.3"

# run a schedule now
vigily schedules run nightly-intents

Questions

Does any data leave my machine?

No. vigily writes to the directory or bucket you choose, has no telemetry, and never calls a model: evaluators that need one call your own client.

Do I need a database server?

No. Storage is Lance tables through kv0, on local disk, S3, GCS or Azure.

Can the web app start commands?

No. Schedules come only from the server's vigily.toml; the API and the app can read them but not create, change or start them.