PROVE: Xiaomi’s Perceptual Evaluation Suite for Object Removal (RC‑S, RC‑T + PROVE‑Bench)
Removing an object from a photo or video rarely has a single pixel‑perfect answer. Many different fills can look fine to a human viewer even though they diverge at the pixel level. Because of that one‑to‑many nature, Xiaomi’s MiLM Plus team built PROVE, a no‑reference evaluation suite that judges only the edited region by comparing local deep‑feature distributions rather than pixelwise differences.
What PROVE contains and why it matters
PROVE (Perceptual RemOVal cohErence) bundles two metrics and a real‑world video benchmark. RC‑S measures spatial coherence inside the edited region and RC‑T measures short‑term temporal consistency. They rely on sliding‑window Maximum Mean Discrepancy (MMD) over DINOv2 features, score only the masked area (masks are mandatory, white pixels mark the removed region), and do not require a paired ground‑truth video. The project is open source under Apache‑2.0 and was accepted to ACM MM 2026. Code and dataset links appear below.
“Yes, as an evaluation harness, not a product feature.”, MiLM Plus / Xiaomi research
Quick practicality: how teams should use PROVE (move these up in your checklist)
- CI gating strategy (example): run a lightweight RC‑S check on a small representative subset per commit. Run full RC‑S and RC‑T weekly or nightly on a larger batch. RC‑S reported runtime is ≈134.6 ms/frame on an RTX 4090 measured at 1080p in single‑frame batches. That implies about 11 seconds per 81‑frame clip for RC‑S only, so ~18 minutes for 100 clips on one GPU. RC‑T and full pipelines will add cost.
- Calibration: don’t let a single RC‑S or RC‑T number decide release. Calibrate pass/fail thresholds with a small in‑house human study (30-50 representative clips is a reasonable starter) and recheck visual examples during rollout.
- Mask discipline: PROVE scores only masked regions, so invest in reliable mask generation. Mask jitter or misalignment will distort results. PROVE‑H explicitly stresses unrefined masks to surface that brittleness.
- Audit for metric‑gaming: add adversarial tests that blur, add texture, or swap regions to catch models that optimize the metric without producing genuinely plausible content.
How RC‑S and RC‑T work (short intuition)
Think of PROVE as checking whether the replaced patch “feels” like its surroundings in a perceptual feature space rather than matching exact pixels. Concretely:
- RC‑S (spatial): computes sliding‑window MMD between local DINOv2 patch features inside the mask and nearby context. The sliding window localizes the test so background areas don’t drown out the edited region.
- RC‑T (temporal): evaluates temporal coherence by computing MMD across adjacent frames inside the temporally stabilized intersection of restored regions. Frames are jointly cropped to the union of masks to align comparisons.
MMD (Maximum Mean Discrepancy) is a kernel‑based two‑sample test used to compare distributions rather than individual points, which is exactly what you want when many plausible fills exist. DINOv2 is used for features because its self‑supervised patch representations capture semantics and texture at a level that aligns well with human perception.
PROVE‑Bench: paired and hard real videos
- PROVE‑M: 80 paired tripod‑captured clips (81 frames, 1080p). For each clip the authors collected target‑free footage within two minutes and refined SAM3 masks frame‑by‑frame. They applied synchronized Ken Burns‑style panning and zoom augmentation to triplets and ran a three‑stage quality control.
- PROVE‑H: 100 unpaired “hard” videos (crowds, flowing water, flames, textured terrain, multi‑puddle reflections, fast motion). These intentionally use unrefined SAM3 masks to stress real‑world failure modes.
How it lines up with human judgment
The team ran a 20‑participant human study aggregated by Borda count to compare metric rankings to subjective preference. Higher correlation numbers mean a metric’s ranking matches human preference better. Reported results (paper tables) show:
- RC‑S: average Kendall’s τ = 0.59 and Spearman’s ρ = 0.66.
- ReMOVE (prior no‑reference metric): τ = 0.26, ρ = 0.29.
- CFD (another prior metric): τ = 0.16, ρ = 0.18.
RC‑S ranked first on five of six benchmark splits in the paper’s tables. In the RORD‑Val region‑robustness test, RC‑S scored the clean image higher than blurred and region‑swapped variants for every evaluated sample in that test set, according to the paper. These numbers are promising, but remember the human study had N=20, so treat them as encouraging rather than definitive across all domains and demographics.
Performance, ablations and leaderboard highlights
- Runtime: RC‑S reported ≈134.6 ms/frame on a single RTX 4090 (measured at 1080p, single‑frame batches). This is RC‑S only; RC‑T runtime was not reported in the summary and will add cost.
- Speed advantage: RC‑S is reported as ~13.7× faster than CFD on the spatial tests used in the paper.
- Ablations (reported): switching the feature extractor matters. DINOv2 yields avg τ ≈ 0.59, DINOv3 drops to ≈ 0.51, SAM features ≈ 0.44. Removing the sliding window or swapping MMD for cosine similarity reduces correlation performance, with reported deltas in the paper.
- Leaderboard notes: the public leaderboard reports combined RC‑S and RC‑T scores. SVOR (1.3B) leads combined RC‑S at 0.5197 and EffectErase leads RC‑T at 0.2525. These leaderboard values are relative rankings rather than absolute quality thresholds, useful for comparing systems but not a direct proxy for user satisfaction without calibration.
Limitations and pragmatic caveats
- Mask dependence: PROVE requires masks (white pixels = removed region). If your mask pipeline jitters or clips the boundary, metric values will shift. Test sensitivity to noisy masks before relying on absolute thresholds.
- Human study size: the perceptual validation used 20 participants. That provides useful signals but not broad population coverage, so larger subjective tests remain necessary for final product decisions.
- Hyperparameters matter: sliding‑window size w, kernel bandwidth for the RBF MMD, which DINOv2 layer(s) are used, patch stride and normalization all affect behavior. Defaults live in the repo. Verify and experiment for your content mix.
- Compute and memory: DINOv2‑giant with sliding windows is nontrivial. Expect GPU time and memory requirements. PROVE is intended for CI and evaluation, not low‑latency on‑device scoring.
- Metric‑gaming risk: any metric used as a gate becomes a target. Examples to watch for include overly smooth fills that reduce distribution variance, or adversarial textures that match the local feature distribution but look fake to humans. Pair PROVE with spot visual audits and adversarial test cases.
- Unanswered points: the paper and repo should be consulted for exact window size and kernel bandwidth defaults, the RC‑T runtime, and systematic sensitivity to mask noise. See links below for the canonical sources.
Where to run it and what to check in the repo
PROVE is public; the GitHub repository includes code, scripts, and instructions (Apache‑2.0). The PROVE‑Bench dataset is available on Hugging Face, and the paper preprint is on arXiv. Before integrating, verify the precise CLI names, default hyperparameters, and any precomputed feature caches or batching recommendations in the repo.
Practical next steps by audience
- Researchers: add RC‑S and RC‑T as no‑reference baselines in papers on removal, and reproduce ablations on your data to see feature‑extractor sensitivity.
- Engineering teams: pilot PROVE in CI on a single model family for about four weeks. Calibrate fail thresholds with 30-50 representative clips and schedule nightly full runs plus lightweight per‑commit checks.
- Product/ops: budget GPU time for evaluation (the RC‑S timing suggests ~11 seconds per 81‑frame 1080p clip on one RTX 4090). Include periodic human audits to catch metric blind spots.
Key questions: short answers
- What problem does PROVE solve?
It evaluates object removal in a perception‑aligned way by scoring only the edited region and comparing local deep‑feature distributions (sliding‑window MMD over DINOv2), avoiding full‑reference bias that favors copy‑paste solutions.
- Do I need ground‑truth videos to use PROVE?
No. RC‑S and RC‑T are no‑reference metrics. Masks are mandatory and must mark the removed region (white pixels by convention).
- How well does PROVE match human judgments?
RC‑S correlates substantially better with a 20‑participant Borda study than prior no‑reference metrics (reported Kendall’s τ ≈ 0.59, Spearman’s ρ ≈ 0.66). These are encouraging results but come from a modest‑size study and should be validated on your content mix.
- Can I run PROVE in CI for fast feedback?
Yes, it’s intended for evaluation and CI. Expect nontrivial GPU cost: RC‑S was reported at ≈134.6 ms/frame at 1080p on an RTX 4090. Use lightweight per‑commit checks and periodic full evaluations rather than full runs on every change.
- What are the main caveats?
Dependence on mask quality, compute and memory cost, a small human study, and potential for metric‑gaming. Hyperparameters (window size, kernel bandwidth, DINOv2 layer choice) affect behavior and need validation for your use cases.
PROVE advances evaluation for object removal by aligning the metric with the perceptual, one‑to‑many reality of the task. Use it as a sharper automated gate in development and CI, but keep humans in the loop and validate thresholds on representative content before letting scores drive production decisions.