A tabular model that keeps your training rows in memory, and scores at the top of leaderboards
Nums AI released Causilo, a pretrained tabular “foundation model” with a scikit‑learn‑style API, Apache‑2.0 source code, and pretrained weights hosted on Hugging Face. According to Nums AI and a TabArena maintainer who re‑ran the submission, Causilo achieved an overall Elo of about 1794 among single models on TabArena; Nums AI also submitted v1.0.1 to ScoringBench and reports first place by CRPS, R² and RMSE on the ScoringBench protocol (101 datasets, 5 folds, capped at 3, 000 samples). See the project page and artifacts at the Causilo repo (https://github.com/nums-ai/causilo) and the model card on Hugging Face. The TabArena codebase and verification process are documented at https://github.com/autogluon/tabarena, and the ScoringBench submission is viewable in the ScoringBench PR (https://github.com/JonasLandsgesell/ScoringBenchOutput/pull/2).
What Causilo is, quickly
Causilo is a pretrained tabular model for classification and regression that follows an in‑context learning pattern, where calling fit stores your labeled rows as read‑only context instead of changing model weights. At predict time, query rows attend to that context to produce outputs. The code is Apache‑2.0; the pretrained weights are on Hugging Face but distributed under a separate Causilo License v1.0 that restricts commercial, hosted, and API use. You must negotiate a separate license with Nums AI for production or hosted deployments.
Why the in‑context design matters for product teams
Benefit: you can adapt to new tables quickly without retraining weights. That is useful for many small, separate datasets or rapid prototyping across customer tables. Cost: inference memory and latency tend to grow with the number of training rows because those rows are part of the active context the model attends to. Practical tradeoffs for latency, memory, and throughput depend heavily on your table sizes and how you use the model flags described below. Benchmark with your actual workload before committing to production.
How Causilo works (high‑level, technical bits you’ll care about)
- Pretraining: Nums AI states Causilo was pretrained only on synthetic data; public materials do not disclose the full synthetic corpus generation details.
- API & runtimes: consumes NumPy arrays or pandas DataFrames (categorical features and missing values supported). Requirements: Python 3.10-3.12 and PyTorch 2.13+; the first fit downloads the checkpoint automatically.
- Classification & regression: classification supports up to 10 classes; regression returns mean by default and v1.0.1 adds median and quantile outputs (999 native quantiles reported by Nums AI).
- Architecture phases: refinement, compression, in‑context learning. Values are embedded using 16 learned sine and cosine frequencies; missing values have a learned vector. Features are grouped (sets of 3), summarized in two column stages using 128 latent slots; a row stage with 4 latent tokens enables cross‑attention between groups; pooling compresses each row into a 512‑dim vector; a 12‑layer prediction block lets query rows attend to labeled training rows.
- Ensemble behavior: default of 8 ensemble members share weights but cycle normalization types (none, rank2gaussian, robust, power) and use seeded permutations of features/classes to diversify predictions.
- In‑context semantics: training rows are read‑only context; query rows can attend to that context but cannot change it or each other. The model supports an option use_kv_cache (caches key/value representations at fit to speed repeated predictions at the cost of additional memory).
Benchmarks and how to read them
TabArena aggregates pairwise comparisons across many tabular datasets using an Elo‑style ranking. Its process requires authors to submit code and a maintainer to re‑run the full evaluation for verification. Nums AI submitted Causilo to TabArena and a maintainer re‑ran the evaluation, reporting an overall Elo ≈1794 among single models (official TabArena setup: 51 datasets, 816 Full splits, 8 estimators, seed 42). Prior Labs’ TabPFN-3 previously reported an overall Elo of about 1636.2 on TabArena.
Important nuance: TabArena separates “models” (single methods under a common protocol) from “systems” (full AutoML pipelines and ensembles). The “highest Elo” claim applies to single models under TabArena’s protocol. When system entries are included, the TabArena maintainer’s re‑run placed Causilo 3rd of 88 overall. Elo confidence intervals for the top models overlap, so the lead is narrow rather than definitive.
On ScoringBench, which emphasizes probabilistic regression metrics like CRPS, Nums AI’s v1.0.1 submission ran on 101 datasets with 5 folds and a 3, 000‑sample cap. Nums AI reports ranking first by CRPS, R² and RMSE, and the ScoringBench maintainer independently checked the submission artifacts before merging. That shows strong probabilistic performance under ScoringBench’s protocol and caps, but the results are bounded by the dataset selection and the 3, 000‑row limit used for the benchmark.
Performance and resource notes (what Nums AI reported)
Nums AI re‑ran three models on a single H100 (80 GB) GPU with 8 CPU cores per job and reported that Causilo was fastest on both fit and predict in that experiment and used the least GPU memory in their published table. These are single‑run measurements reported by Nums AI. Full reproducibility requires the benchmarking script, batch and context sizes, and environment details (precision, use_kv_cache setting, etc.). Request those artifacts or reproduce locally before sizing production infrastructure.
Licensing and pretraining caveats (read this before you pilot)
- Code license: Apache‑2.0, you can inspect and run the code freely.
- Weights license: Causilo License v1.0 on Hugging Face restricts commercial, hosted, and API use, businesses must negotiate a separate license with Nums AI for production or hosted deployments. See the Hugging Face model card: https://huggingface.co/nums-ai/causilo.
- Pretraining data: Nums AI reports synthetic‑only pretraining; the public materials do not describe the synthetic corpus in detail (generation method, scale, diversity). That matters for domain shift and real‑world generalization. Ask Nums AI for the synthetic data methodology if you plan broad production use.
How to evaluate Causilo for your team (a short pilot plan)
Suggested 1‑week pilot for product or data teams:
- Pick 3 representative datasets: one small (≈1k rows), one medium (≈10k rows), and one matching your typical production table (or the largest realistic sample you expect to supply as context).
- Run baseline comparisons: your current pipeline, an off‑the‑shelf AutoML system, and Causilo with use_kv_cache both off and on.
- Measure: wall‑clock fit and predict latency, peak GPU and CPU memory, and throughput for realistic batch sizes. For regression tasks, record CRPS, Pinball loss at relevant quantiles, RMSE, R², and calibration plots for interval coverage.
- Operational checks: simulate typical update patterns (how often training rows change), estimate costs for in‑context inference (compression, caching, or approximate search strategies if you have many support rows), and validate licensing for your deployment model.
Practical example
Imagine a fintech team maintaining hundreds of small loan tables across regions. Causilo could let them prototype models for each table without retraining weights: ship the per‑table rows as context and get calibrated quantile forecasts for risk assessment. But before replacing tuned production models, the team must validate tail risk behavior and latency for the typical table sizes, especially if some tables exceed the 3, 000‑row scale used in ScoringBench’s capped runs.
Responsible adoption: what to watch for
- Domain shift: synthetic pretraining helps privacy and scale