Count Anything: a pragmatic, text‑guided model that actually counts across photos, drones and microscopes

What if a single AI could take a text prompt—“count the cars,” “count wheat ears,” “count cell nuclei”—and mark every instance across a parking lot image, a drone shot, or a microscope slide? Count Anything aims to do exactly that. It’s not a flashy leap toward general intelligence; it’s a focused, practical step: combine the right tools for different visual counting problems and train them on a truly cross‑domain dataset.

Why visual counting is harder than it sounds

Humans count by fusing recognition, grouping and context. Machines must do the same across wildly different scales, viewpoints and occlusion patterns. A crowded stadium, a drone photo of a field, and a stained tissue slide look nothing alike to an algorithm. Historically, engineers built separate, specialist models for each domain—one for crowd counting, another for cell counting, another for satellite vehicle detection. That creates fragmentation: multiple pipelines, multiple vendors, and lots of integration work.

How Count Anything works (text‑guided counting)

The core insight is simple and effective: use two complementary “eyes.” One predictor draws bounding boxes for large, distinct objects; the other drops points for small, dense targets. A merge rule reconciles overlaps by preferring whichever prediction has higher confidence, preventing double counts when a large object contains many small points.

Rather than retraining a massive vision backbone, the team built on Meta’s SAM3 and added small adapter modules—lightweight add‑ons that tweak the preexisting model without full retraining. That makes the system easier and cheaper to extend than rebuilding a network from scratch.

“Combining a box‑based detector for large objects with a point‑based counter for dense, small targets yields complementary strengths.”

CLOC: the cross‑domain training diet

Data matters. The team assembled CLOC, a mixed‑domain dataset with roughly 220,000 images, 619 categories and about 15 million annotated object instances across six domains: everyday photos, satellite/drone imagery, medical tissue scans, microscopy, agricultural scenes (wheat ears, etc.), and bacterial cultures. That breadth is why a single, text‑guided model can generalize from parking lots to petri dishes.

How well it performs (and what the numbers mean)

On the authors’ benchmark, Count Anything reports an average absolute error (MAE) of about nine objects per queried category. MAE is the mean of |predicted − actual| across examples, so on average the model missed or double‑counted nine items per query. By their tests that’s roughly half the error of the best competing generalist methods they evaluated (CountGD, CLIP‑Count, Grounding DINO).

Interpretation matters: a nine‑object MAE is excellent for images with hundreds of targets (single‑digit percent error), but for small counts (e.g., counting five rare defects) it’s obviously unacceptable. The model is competitive in crowd counting but doesn’t always outperform top specialist models tuned for those tasks.

Practical comparison

• Generalist rivals (CountGD, CLIP‑Count, Grounding DINO): higher MAE on the mixed benchmark.
• Specialist models (crowd counting, cell counting): can beat Count Anything within a single, narrow domain.

Business use cases that move from niche to unified workflow

Count Anything’s strength is consolidation. Instead of maintaining separate detectors for drones, microscopes and CCTV, teams could centralize many counting tasks into one text‑guided endpoint—reducing operational friction and speeding experimentation.

• Agriculture: estimate yield by counting ears of wheat from drone imagery. A centralized counter can speed seasonal forecasting, though highly occluded crops or dense canopies still require field validation.
• Retail & logistics: run rapid inventory audits from shelf or pallet photos. Use the point predictor for dense SKUs and the box predictor for larger pallets.
• Environmental monitoring: count vehicles or deforestation patches in satellite images for urban planning and compliance reporting.
• Clinical research (non‑diagnostic): accelerate cell and colony counts for lab workflows—while keeping humans in the loop for regulatory decisions.

Limitations and deployment checklist

Count Anything is a powerful generalist, not a drop‑in replacement for domain specialists in safety‑critical contexts. Common failure modes translate directly into business risk:

• Ambiguous labels: prompts like “bushes” vs. “shrubs” can map inconsistently to annotations and produce unreliable counts.
• Extreme density and occlusion: packed crowds, overlapped crops, or dense cell clusters reduce matching accuracy and increase missed or merged instances.
• Regulatory and privacy considerations: medical and satellite imagery require strict governance, anonymization and audit trails.

Deployment checklist for teams:

• Validate per‑domain MAE and set acceptable thresholds (absolute and percentage error).
• Run a pilot with ~1,000 labeled images representative of production data.
• Use human‑in‑the‑loop gating for decisions above a risk threshold (e.g., >5% deviation from expected counts).
• Monitor drift: track MAE, false positives, and class‑specific failure rates over time.
• Consider ensembling: route known‑hard domains to specialist models while using the generalist for broad coverage.
• Apply model compression or conditional routing (run only the box or point predictor when appropriate) to control latency for edge deployments.

Quick ROI thinking

Example framework: if a manual inventory audit costs $X per week and Count Anything reduces human time by Y%, multiply monthly savings by expected accuracy adjustments (rework costs for miscounts). A short pilot will reveal the real Y and the error‑driven rework factor; many teams find automated pre‑audit plus human verification eliminates low‑value labor while preserving quality.

Reproducibility and where to explore

The code and paper are public. Explore the implementation and reproduce experiments at the project repo: github.com/Mengqi-Lei/count-anything, and read the preprint on arXiv for methodological details. Because the system builds on SAM3 with lightweight adapters, engineering teams can prototype faster and extend models to new classes without expensive retraining.

Bottom line: a clever hybrid architecture plus a broad cross‑domain dataset moves practical, text‑guided counting closer to mainstream adoption—faster experimentation, fewer fragmented pipelines, and smarter automation—so long as teams validate domain performance and keep humans in the loop for high‑stakes decisions.

If you’re responsible for operations, analytics or automation, start with a focused pilot: pick two domains, label ~1,000 representative images, measure MAE and error types, and compare total cost of ownership against existing specialist pipelines. Count Anything probably won’t replace every niche model, but it can centralize many counting tasks and accelerate ROI from AI for business and AI automation.