LingBot‑Vision: Open‑source 1.1B ViT that learns edges first with masked boundary modeling

Robbyant open‑sources LingBot‑Vision: a 1.1B ViT that learns edges first

Robbyant, the embodied‑AI team inside Ant Group, has published LingBot‑Vision, a family of self‑supervised Vision Transformer encoders trained to prioritize dense spatial structure (contours, boundaries, depth discontinuities). The flagship ViT‑g/16 is roughly 1.1 billion parameters. Robbyant released model weights and inference code under the Apache‑2.0 license on Hugging Face and a companion GitHub repository (see the project entries published by Robbyant on Hugging Face and GitHub for the model cards, technical report and usage examples). The release bundle includes multiple sizes (ViT‑g/16 and distilled ViT‑L, ViT‑B, ViT‑S), an interactive technical report, and example inference scripts.

What sets LingBot‑Vision apart

Most self‑supervised vision methods emphasize semantic invariance: push different views of the same object together and tolerate small pixel differences. LingBot‑Vision reframes pretraining to make geometry, where objects end and surfaces change, a primary signal. That matters for any application that needs reliable per‑pixel geometry: depth completion, robotic grasping, AR scene understanding and video object propagation.

“LingBot‑Vision … is a family of self‑supervised Vision Transformers built for dense spatial perception.”, Robbyant.

The core trick: masked boundary modeling

Masked boundary modeling forces the network to pay attention to boundary‑bearing patches during masked prediction. Key pieces:

  • During training an exponential moving average (EMA) teacher, the same self‑distillation paradigm used by DINO and iBOT, predicts a dense categorical boundary field that serves as an online target. (EMA = exponential moving average. DINO/iBOT = self‑distillation contrastive / online teacher frameworks.)
  • Boundary‑bearing tokens identified by the teacher are forced into the mask so the student must reconstruct or predict them. The masking rule is written succinctly as: M⁺ = M ∪ B, a random mask M union the boundary set B.
  • The boundary attribute per pixel is encoded as a(p) = (d, θ, φ¹, φ²), a small geometric vector capturing distance and orientation cues. Each channel is discretized into K = 32 bins so boundary prediction becomes a categorical per‑pixel task.
  • The full pretraining objective is the sum of several terms (contrastive/distillation plus boundary and regularization losses): L = L_DINO + λᵢ · L_iBOT + λᵦ · L_bnd + λₖ · L_KoLeo. L_DINO and L_iBOT are the global self‑distillation/contrastive components. L_bnd is the categorical boundary prediction loss and L_KoLeo is a diversity/regularizer term. See the paper’s appendix for the exact λ hyperparameters.
  • Decoded segments from the teacher are validated with a Number‑of‑False‑Alarms (NFA) test from the a‑contrario tradition to filter spurious structure. The a‑contrario/NFA rule tests whether observed structure is unlikely under a uniform (no‑structure) null model and controls false alarms in a principled way.

Why these choices matter

Masked image modeling (MIM) and contrastive self‑supervision tend to underweight the patches that carry geometric discontinuities. By forcing boundary patches into the mask and supervising them with a teacher‑predicted geometric field, LingBot‑Vision explicitly teaches the encoder to represent fine spatial structure. The NFA validator reduces the chance that noisy teacher predictions introduce hallucinated edges during training, although NFA methods are best understood as highly principled rather than free of implementation choices (discretization, evaluation thresholds and ε decisions still matter in practice).

Empirical highlights (as reported by Robbyant)

Robbyant evaluates LingBot‑Vision primarily as a frozen encoder (frozen features with a single linear readout) for dense tasks. Across multiple benchmarks the geometry‑first backbone shows strong returns versus much larger or widely used alternatives.

  • Pretraining corpus: Robbyant reports a curated pretraining set of ~161 million images sampled from a ~2 billion image pool, trained without human labels, external edge detectors, or a pretrained backbone.
  • Depth (NYU‑Depth v2): LingBot‑Vision (ViT‑g/16) achieves RMSE = 0.296 (frozen encoder + linear readout) compared with 0.309 reported for a 7B DINOv3 variant and 0.307 for V‑JEPA 2.1 (2B). Robbyant frames this as parity or better with much larger models on dense depth estimation.
  • KITTI: Robbyant reports that LingBot‑Vision is “the best model below 2B parameters” on KITTI evaluations (see the model paper tables for numeric RMSE entries and exact evaluation protocol).
  • Semantic segmentation: On ADE20K the model trails a distilled DINOv3 ViT‑H+ by 1.3 mIoU, matches on Cityscapes, and leads on VOC12. Compared to a same‑size DINOv2 baseline, LingBot‑Vision improves by 4+ mIoU on ADE20K, Cityscapes and VOC12 (check the paper tables for precise numbers and eval settings).
  • Video object segmentation (training‑free propagation): Using frozen features and a label‑propagation setup, Robbyant reports J&F = 70.0 on DAVIS‑2017 and 73.5 on YouTube‑VOS (compare to DINOv3 ViT‑H+ at 71.1 / 74.0; differences are small and may be within experimental variance).
  • ImageNet‑1K: Linear probe = 86.32. k‑NN = 83.39, useful sanity checks that the model retains solid semantic capability despite its geometry emphasis.
  • Distillation and deployment: A distilled 0.3B ViT‑L student reportedly matches a 7B DINOv3 on NYU‑Depth (0.310 vs 0.309), showing the geometry signal can be compressed into smaller, deployable models.
  • Downstream payoff, LingBot‑Depth 2.0: Swapping LingBot‑Vision encoders into their depth‑completion stack and scaling training data from 3M public samples to 150M led to large gains. For example, on block‑masked DIODE‑Indoor RMSE reportedly fell from 0.132 to 0.062. ClearGrasp transparent‑object RMSEs are cited at 0.010 / 0.012. Robbyant reports improvements across 14 depth‑completion benchmarks.

All numeric claims above are reported by Robbyant. Consult the project GitHub and Hugging Face model cards and the technical report for exact evaluation protocols (input sizes, frozen vs finetuned readouts, splits) and full tables.

Engineer checklist, fast adoption steps and red flags

  • Grab the released assets: locate the Robbyant GitHub repo (look for the lingbot‑vision codebase) and the Hugging Face collection published by Robbyant to review model cards, license statements (Apache‑2.0), and dataset notes before use.
  • Sanity checks to run locally: run the ImageNet linear probe and a small k‑NN test at your target resolution to verify behavior on your infrastructure and confirm feature shapes (patch_tokens are [B, H*W, C]. Example printed shape: torch.Size([1, 1024, 384]) for a 32×32 patch grid and 384‑dim channels).
  • Domain validation: evaluate frozen features on a 200-500 sample held‑out set representative of your production edge cases (transparent objects, motion blur, low light, sensor noise). Check depth RMSE, segmentation mIoU and visual quality of propagated masks.
  • Latency/memory budget: measure inference throughput and peak memory for the ViT‑g/16 and distilled students on your target hardware, and plan to use ViT‑L/ViT‑B students if on‑device constraints are strict.
  • NFA ablation: toggle the NFA validation on and off and observe noisy‑edge propagation in your domain. The paper’s NFA step reduces false positives, but results can be sensitive to discretization and thresholds.

Caveats, open questions and things to check before production

  • Dataset provenance and bias: Robbyant reports training on a curated ~161M image set drawn from a ~2B pool, inspect the model card and the paper appendix for exact filtering heuristics, license provenance, and any third‑party data restrictions.
  • Compute and reproducibility: the release emphasizes data efficiency relative to some large baselines. Total compute (steps, epochs, batch size, hardware cost) is essential to estimate reproducibility and cost; find those details in the technical report.
  • NFA is principled but not magical: the NFA a‑contrario test controls false alarms under a uniform null model, but discretization choices (K = 32) and implementation specifics still influence outcomes, don’t assume parameter‑free means no tuning is needed.
  • Task trade‑offs: a geometry‑first objective can shift representational priorities. ImageNet linear probes are healthy, but task‑specific finetuning and detection/instance segmentation parity should be validated for your application.
  • Robustness gaps: the teacher’s boundary predictions and resulting student features may behave differently under extreme lighting, heavy occlusion, artistic or non‑photorealistic imagery. Run targeted stress tests.

Key takeaways, quick Q&A

  • What is masked boundary modeling and why does it matter?
    It forces boundary‑bearing tokens into the masked set and supervises them with a teacher‑predicted categorical boundary field, making geometric discontinuities and contours explicit pretraining targets, useful where per‑pixel geometry matters.
  • Do these geometry gains require labeled edges or external detectors?
    No, Robbyant emphasizes “no human labels, no external edge detectors, and no pretrained backbone to bootstrap from, ” using an EMA teacher to produce dense targets online.
  • Can a ~1.1B model match much larger models?
    On several dense spatial tasks reported by Robbyant (depth, segmentation, video propagation) the ViT‑g/16 matches or outperforms much larger models in frozen‑encoder evaluations, though performance is task‑dependent and differences can be small.
  • Will smaller, deployable models inherit the gains?
    Yes, Robbyant reports a distilled 0.3B ViT‑L student that matches a 7B DINOv3 on NYU‑Depth RMSE (0.310 vs 0.309), indicating the geometry signal compresses well for many dense tasks.
  • How should product teams evaluate LingBot‑Vision for their use case?
    Pull the Robbyant GitHub and Hugging Face model cards, run the ImageNet linear probe as a sanity check, then validate frozen feature performance on a small representative set of 200-500 held‑out images that include your edge cases (transparent objects, motion blur, low light). Measure latency and memory, and test NFA on and off to understand false‑alarm behavior.

Final perspective

Reorienting pretraining from “what is this?” toward “where does it end?” is a pragmatic shift for any team that needs reliable, label‑lean scene geometry. LingBot‑Vision shows that a geometry‑first ViT trained without external edge labels can produce frozen features that deliver tangible downstream improvements, and that those gains can be distilled into small models suitable for deployment.

Before you commit, inspect the Robbyant model card (Hugging Face) and GitHub repo for licensing and dataset provenance, run the engineer checklist above, and validate on the specific edge cases your product must handle. If boundary sensitivity transfers, you’ll likely reduce downstream labeling and gain cleaner depth and segmentation behavior, and that is an operational advantage worth testing.

0