The measurement company for stochastic systems. We map where AI agents, robot policies, and automated workflows fail—and prove, with paired seeds and statistical bounds you can defend, whether a change actually held. Graded by code and math, never by another model.
Whether you are shipping an AI agent, a robot controller, or a learned policy, a single headline score averages out localized failures—and moves by a few points between runs with nothing changed at all.
Poisson is a statistical measurement engine, not an eval wrapper. We test your system before and after a change, condition by condition, on the exact same seeds, correcting for testing every regime at once. You see exactly what improved, what regressed, and what is still underpowered before the commit merges.
report.md · change in fall rate, by condition
significantly safersignificantly worseno significant changethe average
● OpenResearch result · CI workflow previewPromote checkpoint 5999
microduck-walking@v1
build
unit tests
poisson cloud / provefailed
5 of 14 conditions got worse.
8 better · 5 worse · 1 too close to call, below a 8-point limit
These files are as Prove 0.0.1 wrote them, in that version’s green and red.
Merge blocked
A bipedal walking policy, checkpoint 2250 → checkpoint 5999: the aggregate fall rate dropped from 50.5% to 20.8% (-29.7 pts on average), which looks like a ship-ready win. Condition-by-condition paired testing reveals 5 of 14 friction-and-push conditions got significantly worse—blocking the merge. Read the study →
01Where it sits
Your runs in. A verdict out.
Keep the stack, simulator, and runner you already have. Poisson ingests your rollout traces and adds the mathematical layer they leave out: controlled conditions, failure boundary search, and a statistically rigorous verdict on every change.
Builds the conditions: seeded tasks across calibrated difficulty knobs.
BreakIn development
Hunts across parameter spaces for the conditions that make your system fail.
TracePrivate wheel
Reads the runs you already have, step by step, to isolate where trajectories die.
Prove
Reruns every condition before and after a change on identical seeds, corrected for testing them all at once.
GymIn development
Converts isolated failure regimes and verified rollouts into deterministic environments for post-training.
What you get · Shipping gate
A verdict per condition
Improved
Regressed
Too small to tell, with the MDE limit
In CI, the check blocks merge on a significant regression.
What you get · Training loopIn development
Trainable failure data
Seeded failure tasks
Exact regimes where the system broke, packaged as reproducible seeds.
Code-verified rollouts
Graded trajectories ready for RL post-training or fine-tuning, without LLM-judge reward hacking.
Re-run Prove after training to verify the fix held.
Runs where you do. On your machine, on Modal, or across a cluster in your AWS, GCP or Azure account. Every integration
02Four surfaces
One method. Four kinds of system.
Whether a rollout is a tool-calling agent, a 50Hz MuJoCo control loop, an RL training episode, or a back-office pipeline, it reduces to the same mathematical object: a stochastic trajectory over controlled conditions. One engine measures all four.
Prompts, models, and tool schemas change daily, while evaluation scores bounce around on run-to-run variance. We isolate failure modes across tool-call depth and distractor load, trace multi-step trajectories to the exact step that broke, and prove whether a prompt or model swap actually improved reliability—without LLM-as-a-judge noise.
Empirical proofOn τ²-bench airline: running the exact same agent twice at temperature 0 disagrees on 22% of tasks—showing why unpaired leaderboard deltas are noise.
A policy retrain that improves average reward often introduces catastrophic failures in untested physical corners. We sweep across friction, payload, terrain, and external perturbation parameters in simulation to map your controller's exact failure boundary and show, regime by regime, whether a new checkpoint is statistically safer, worse, or too close to call.
Empirical proofOn a simulated Unitree Go1: sweeping 400 friction-and-push conditions shows a retrained policy significantly safer in 60 regimes, worse in 0, with no significant change in the other 340.
Static benchmarks saturate quickly, and policies learn to exploit weak or LLM-graded reward signals. We generate algorithmically parameterized task distributions with deterministic code verifiers—giving you reward channels the policy can't see for RL post-training, and calibrated difficulty curves to measure true generalization.
Empirical proofAcross 4,200 seeded episodes: strict vs. leaky verification exposes where models reach the goal state but fail to stop.
Teams either review every automated action by hand or flip automation on blind and hope for the best. We read your pipeline's runs and the approvals behind them, measure step-by-step survival and error rates, and compute how many cases it takes for the confidence bound to clear a threshold you choose, before you take human review off a step—or block a bad model downgrade.
● OpenSynthetic back office · CI workflow previewSwitch to the cheaper model
600 actions per version, a person approves each
build
unit tests
poisson cloud / provefailed
1 of 4 action types got worse.
Harmful reroute drafts 4.0% → 22.7%
Merge blocked
03The tools
Five tools. One evidence trail.
From synthetic stress-testing to production trace survival and CI regression gates. Every tool grades by deterministic code and exact statistics—never by another model—and sends nothing home.
shell · poisson-trace
poisson-trace fix-check OLD NEW --store ROOT --event NAME \ ( --failing-from RUN | --inputs all ) --max-mde X
Show more commands
pip install poisson-tracepip install poisson-prove# where do runs die, over the step indexpoisson-trace where RUN --store ROOT --event NAME# do two runs' survival curves differpoisson-trace compare A B --store ROOT --event NAME --max-mde-hr X# did the change make runs fail less, with the old version as the controlpoisson-trace fix-check OLD NEW --store ROOT --event NAME \ ( --failing-from RUN | --inputs all ) --max-mde X# which inputs moved, corrected across all of thempoisson-trace changes A B --store ROOT --event NAME
whereSee how many runs survive each step, and which steps end them.
fix-checkCompare the old and new versions on the same inputs, with the old one re-run as the control.
readersImport existing DeepEval or Ragas results without rerunning the model. How.
pip install poisson-provepoisson-prove init --example# 1. sweep the conditions, before the changepoisson-prove sweep --yes# 2. sweep again after it, on the same seedspoisson-prove sweep --yes \ --target poisson_prove.examples.mock_agent:rollout_patched \ --pair-with latest# 3. compare the pair into report.md, heatmap.svg, receipt.htmlpoisson-prove report --event violated --x tool_fail_rate --y task_id# in CI: exit 4 if any condition came back significantly worsepoisson-prove check --event violated --fail-on worse
report.mdWhich conditions moved, in words, with the method and counts.
heatmap.svgEvery condition, improved or regressed, at a glance.
receipt.htmlRuns, seeds and hashes, so anyone can check the result.
These files are as Prove 0.0.1 wrote them, in that version’s green and red.
calibration card · gpt-5-nano · depth
Depth
Pass@1
Strict, 95% interval · ○ leaky
vs. previous level
d1
54.7%
baseline
d2
39.7%
−15pp · distinct
d3
23.7%
−16pp · distinct
d5
17.7%
too small to call
d8
87.0%
+69pp · distinct
4,200 episodesSeeded tasks at every level, each with its own verifier.
$2.52The whole card, at the rates read on 2026-09-25.
2 modelsgpt-5-nano and Qwen3-8B are calibrated so far. The cards.
break · in development
Break runs your agent many times while it changes what the agent meets in production, keeps only the changes that fail more often than an untouched control, and hands each one over as a replayable condition a fix can be checked against.
What breaks it before a user does?A hunt before a release, and a short list of the conditions that fail, worst first.
What happens when my tools misbehave?Errors, timeouts, slow answers, rate limits, empty or malformed results, and an instruction planted in a tool result.
I fixed it. Did I?Each finding re-run on the changed agent, with Trace’s answer: fails less, fails more, or too small to tell.
Keep what we found.Findings are files, and re-run in CI as a standing check.
forAgents on the OpenAI Agents SDK, LangGraph and CrewAI, run in a test environment.
testsTrace’s; the correction across findings is Prove’s. Break computes none of its own.
notA jailbreak catalogue, a model judge or a certificate. A hunt that finds nothing says what it tried.
gym · in development
The gym trains an agent against checks it cannot see, draws its practice from where it currently fails, and reports for each checkpoint whether training helped, at a stated confidence.
TasksSeeded worlds, the tools the agent acts through, and the hidden checks that give the reward.
BreakSearches for where the current checkpoint fails. Those failures become the curriculum.
TraceReads every recorded episode and says at which step the remaining failures happen.
ProveCompares each checkpoint with its starting point on held-out conditions training never touched.
forLabs and teams training agents, on their own machines or their own cloud.
firstThe tool-use agent pack, a local sandbox and one trainer adapter. The gym page.
notA trainer or a benchmark. No model grades an episode.
