CVFuzz.
Find the smallest realistic change that breaks your computer-vision model — local-first robustness testing from failure-boundary search to full-stream video evaluation.
Overview
Accuracy tells you if a model is good; it doesn't tell you where it breaks
Production vision systems meet degraded optics, motion, bad weather, compression, and partial visibility that a clean validation set never covers. A single accuracy number says nothing about which specific conditions a given detector can survive.
CVFuzz applies controlled, configurable visual transformations to every baseline detection and walks each one's severity levels until it finds the point where the prediction meaningfully changes — the failure boundary. Numeric refinement narrows that boundary further wherever a transform supports interpolation.
Two ways to ask the same question
The `run` command searches for the least-severe break per object, using baseline detections as metamorphic references when ground truth isn't available. The `video-run` command and local web app instead render one complete video per transformation and evaluate every frame, producing a single self-contained run with annotated MP4s, per-frame data, and aggregate metrics — the shape you want for a full robustness review rather than a single boundary number.
Nine degradations, target-aware failure analysis
Exposure, low-light noise, motion blur, defocus, JPEG compression, resolution loss, fog, target-aware partial occlusion, and glare — each configured through versioned YAML with an ordered search parameter and independent render parameters. Detection-aware analysis distinguishes missed objects, confidence collapse, class changes, and localization drift using IoU-based matching, so a changed prediction is classified rather than just flagged.
How it works
Everything runs on the machine that has the model and the footage. A model adapter produces baseline detections; nine transformations sweep configured severities against them; a failure detector flags the first meaningful prediction change. No database, no cloud call, no synthetic demo data — just file-backed runs you can inspect afterward.
In the terminal
$ cvfuzz run yolo11n.pt street.mp4 --config cvfuzz.yaml Loading model adapter... ultralytics/yolo11n Baseline detections: 214 objects across 812 frames Searching motion_blur (kernel_size)... boundary at k=9 Searching fog (density)... boundary at 0.42 Searching occlusion (target-aware)... boundary at 31% Run written to .cvfuzz/runs/2f9a1c/
$ cvfuzz video-run yolo11n.pt street.mp4 --config cvfuzz.yaml Rendering 9 transformed streams... Evaluating original + 9 transformed videos frame-by-frame missed_objects: 12 confidence_collapse: 34 class_change: 3 localization_drift: 21 Annotated MP4s + metrics.json written to .cvfuzz/runs/8b71e0/
Install & use
$ python3.11 -m venv .venv && source .venv/bin/activate$ python -m pip install -e '.[dev,yolo]'
$ git clone https://github.com/baselhusam/CVFuzz.git$ cd CVFuzz/backend
$ cvfuzz init-config cvfuzz.yaml$ cvfuzz validate-config cvfuzz.yaml$ cvfuzz run /path/to/yolo11n.pt /path/to/street.mp4 --config cvfuzz.yaml$ cvfuzz inspect .cvfuzz/runs/<run-id>
Commands
cvfuzz init-configScaffold a versioned YAML transform configurationcvfuzz validate-configCheck a configuration before spending a run on itcvfuzz transformsList the 9 available transformations and their parameterscvfuzz runFailure-boundary search over an image, directory, or videocvfuzz video-runFull-stream evaluation across every enabled transformationcvfuzz inspectReview a boundary-search run's resultscvfuzz inspect-videoReview a video-run's annotated MP4s and metricscvfuzz serveStart the local API behind the Next.js web dashboard