Inference
Must match
- Model and weights
- Hardware and topology
- Request mix
Also recorded
Precision, engine revision, power policy.
GPU performance investigations
Find what changed in GPU performance.
Compare workloads, investigate the evidence, and verify better outcomes for inference, training, and GPU fleet operations.
- Self-hosted server with SQLite storage
- Comparability checked before any delta
- Evidence bundles you can recheck offline
One comparison workflow, three outcome contracts
Each team keeps its own metrics and guardrails. The comparison, evidence and verification steps are the same.
Inference example: engine upgrade
Synthetic demonstration. Illustrative values, not measurements.
- Same model (Llama 3.1 8B)
- Same hardware (1× H100)
- Same request mix (512 prompt / 128 output tokens)
- Intended change: engine v0.8.2 → v0.8.3
110 ms time to first token, p95
- Queue
- 40 ms
- Prefill
- 70 ms
- Decode
- 90 ms
230 ms +120 ms (+109%) time to first token, p95
- Queue
- 160 ms
- Prefill
- 70 ms
- Decode
- 90 ms
Observation, not cause
Queue time grew from 40 ms to 160 ms. Prefill and decode took the same time in both runs. The added latency happens before the model starts work on the request.
Training example: framework upgrade
Synthetic demonstration. Illustrative values, not measurements.
- Same model and objective
- Same rank count (8)
- Same hardware (8× H100 SXM5)
- Same effective batch
- Intended change: PyTorch 2.3 → 2.4-rc
420 ms mean step time
- Input
- 25 ms
- Compute
- 260 ms
- Communication
- 115 ms
- Checkpoint
- 20 ms
570 ms +150 ms (+35.7%) mean step time
- Input
- 25 ms
- Compute
- 260 ms
- Communication
- 265 ms
- Checkpoint
- 20 ms
Observation, not cause
Compute stayed at 260 ms; communication grew from 115 ms to 265 ms. Evaluation loss held (2.14 vs 2.15). A straggling rank is a hypothesis to test with a controlled rerun, not a finding.
GPU fleet example: two observation windows
Synthetic demonstration. Illustrative values, not measurements.
- Same pool (64 GPUs)
- Same device types
- Same scheduler policy (gang priority)
- Eligibility history recorded
- Compared: window A vs window B
14.2 s mean queue wait
- Service attainment
- 98.5%
- Allocated GPUs
- 60 / 64
- Waiting jobs
- 0
85.6 s +71.4 s mean queue wait
- Service attainment
- 89.2%
- Allocated GPUs
- 56 / 64
- Waiting jobs
- 6, each needing a 16-GPU gang
Observation, not cause
Eight GPUs sat unallocated while six jobs waited, and every waiting job needed 16 GPUs at once. Without eligibility history, idle capacity cannot be assumed usable.
Comparability comes before the delta
Each lens declares what must match. Every dimension is reported as the same, intentionally changed, different or unknown, and a pair that cannot be compared is refused with a reason.
Training
Must match
- Model and objective
- Hardware and topology
- Rank count
- Effective batch
Also recorded
Optimizer, evaluation metric.
GPU fleet
Must match
- Pool
- Device types
- Scheduler policy
- Eligibility history
Also recorded
Workload mix.
From question to shareable evidence
Six stages, the same for every team. Read how each stage works.
- FramePick the lens, the decision question and the guardrails.
- CompareDeclare the intended change and check comparability first.
- InspectFollow the outcome into aligned windows, phases and operations.
- HypothesizePin observations and keep them apart from hypotheses.
- VerifyAttach rerun evidence under the declared guardrails.
- ShareExport a bundle with a SHA-256 checksum anyone can recheck.
Evidence that can be checked later
An exported evidence bundle is JSON. It carries a SHA-256 checksum over its canonical form, the digests of the stored recipe and results, and the provenance of the original captures. The server's own command rechecks a bundle without the server running.
Bundles are not signed. The check detects a changed bundle; it does not prove who produced it, and a valid bundle is not by itself a verified or improved result.
$ graphsignal-server evidence verify bundle.jsonStart with a comparison you already need to make.
Import a capture or point a client at the server, then compare two runs.