Sound embeddings: why envelope vs spectral encoders flip winners between retrieval and classification

Two tiny encoders, two different winners, and a lesson for every team that cares about sound embeddings

Run a tiny experiment. One encoder measures loudness over time, the other summarizes the spectrum. Ask four different questions of their vectors: classification, clustering, retrieval, and segmentation. The scoreboard flips depending on which metric you inspect. That matters when you choose an embedding for audio search in a podcast app versus a sound‑classification product.

Important: the demonstration this piece follows is from a compact, fully synthetic, CPU‑only notebook. Results illustrate tradeoffs in representation, not real‑world performance. Because the corpus controls loudness envelopes and timbre deliberately, the relative winners may change on natural recordings (reverberation, mic variation, background noise). If you want to verify behavior on real data, run the same battery on datasets such as ESC‑50, VGGSound or LibriSpeech.

How MSEB is organized (the contract and the evaluators)

  • types, the shared data shapes: Sound, SoundEmbedding, Score, TaskMetadata.
  • encoder, the contract your model implements via MultiModalEncoder (you implement _setup, _check_input_types, _encode).
  • evaluators, task families that consume embedding caches and emit Scores (classification, clustering, retrieval, segmentation, and more).

“MSEB is three layers, and a benchmark run walks down them:”, Google Research MSEB tutorial notebook

The notebook installs mseb==0.1.0 and uses SR = 16000 as the sampling rate constant. It intentionally exercises only the lighter evaluators (classification, clustering, retrieval, segmentation) because they depend only on NumPy and scikit‑learn. Transcription and reranking evaluators pull in heavier dependencies such as Whisper, TensorFlow and apache‑beam.

The minimal experiment: two encoders and a synthetic corpus

Rather than download audio, the notebook synthesises everything so it can run on CPU:

“Everything below runs on CPU with no dataset download: we synthesise the audio.”, Google Research MSEB tutorial notebook

  • Classes: tone, chirp, noise.
  • 12 examples per class → 36 documents. Each example rendered twice (a cleaner “take 0” for documents and a noisier “take 1” for queries), producing 36 documents + 36 queries.
  • Waveforms are normalized to unit RMS before applying a per‑item envelope so loudness differences are controlled entirely by the envelope. This isolates envelope vs spectral cues.
  • RNGs are deterministic: the notebook seeds per‑item and per‑take so runs are repeatable. Exact seed formulas and code are available in the referenced notebook (see the GitHub repo below).

The notebook implements two lightweight MultiModalEncoder subclasses by providing the three required methods (_setup, _check_input_types, _encode):

  • EnergyEnvelopeEncoder, pools average energy into n_bins (default 16). Docstring: “Baseline: average energy in `n_bins` equal time slices. Loud/quiet, nothing about timbre.”
  • SpectralProfileEncoder, pools mean log‑magnitude spectrum into n_bands (default 16) using frame windows (default frame=512). Docstring: “Contender: mean log-magnitude spectrum pooled into `n_bands` bands. Describes timbre.”

Why the encoders produce different winners (short)

The core point is simple and practical: different evaluators reward different invariances. The EnergyEnvelopeEncoder captures loudness patterns, an envelope fingerprint that distinguishes recordings and helps identity‑sensitive retrieval. The SpectralProfileEncoder captures timbre, which aligns better with class semantics and thus improves classification and clustering on this synthetic task.

“The winner changes column to column ({‘, ‘.join(flips)} goes the other way). That is the whole argument for a MASSIVE benchmark: a single headline number would have hidden it. An encoder that cannot name a sound can still recognise it, and vice versa.”, Google Research MSEB tutorial notebook

That is why multi‑task benchmarking matters. Optimizing for one objective, for example top‑1 classification accuracy, can degrade another, for example retrieval MRR, unless you explicitly train or tune for both.

What the evaluators actually measure

Below is a compact summary of the four lightweight evaluators the notebook runs, the questions they ask, and typical metrics (acronyms explained on first use):

  • ClassificationEvaluator, What class does this embedding belong to? Metrics: Accuracy, Top‑k Accuracy (the notebook sets top_k_value=2), Balanced Accuracy, Weighted Precision/Recall/F1. The example uses class prototypes (mean unit vector per class) and a dot‑product similarity readout.
  • ClusteringEvaluator, Do embeddings form clusters that align with labels? Uses MiniBatchKMeans and reports V‑measure (the harmonic mean of homogeneity and completeness). Note: if MiniBatchKMeans is constructed without random_state it uses NumPy’s global RNG, so seed explicitly for repeatability.
  • RetrievalEvaluator, Which document(s) match this query embedding? Metrics: MRR (mean reciprocal rank, rewards putting a relevant item high in the ranking), EM (exact match, top‑1 hit), Recall@5, NDCG@10 (normalized discounted cumulative gain, sensitive to rank positions and graded relevance). The notebook uses a BruteForceSearcher over document embeddings.
  • SegmentationEvaluator, What terms occurred where in time? The notebook treats segments as term strings with timestamps and reports TimestampsAccuracy, EmbeddingsAccuracy, combined scores, Word Error Rate (WER), and mean Average Precision (mAP). It uses a tolerance tau = 0.05s (50 ms) for timestamp matching.

MSEB also exposes helper metric functions (compute_word_errors, compute_reciprocal_rank, compute_ndcg_at_k, compute_lp_norm, compute_dynamic_time_warping_distance) so you can run diagnostics or custom aggregations.

Practical gotchas and advice for engineering teams

  • Make the synthetic limit explicit. Results from the toy corpus show mechanism and tradeoffs, but do not prove transfer to natural audio. Validate on at least one real dataset (speech, environmental sound, or your product data) before procurement or deployment.
  • Seed everything that matters. The notebook demonstrates deterministic per‑item and per‑take seeds; do the same for numpy, Python’s random, and whichever framework you use (PyTorch, TensorFlow). For stochastic evaluators (clustering), run multiple seeds and report mean ± std.
  • NDCG spelling and semantics. NDCG (not “NDGC”) is sensitive to input format. In the notebook compute_ndcg_at_k assumes a single relevant document and compares by equality, so pass a single string ID, not a list of IDs, to avoid a silent zero score.
  • Dot product vs cosine. If embeddings are unit‑normalized, dot product equals cosine similarity. If they are not normalized, dot product encodes both direction and magnitude. For retrieval tasks that rely on an identity fingerprint, preserving magnitude can help. For classification you often want invariance and normalize vectors.
  • Segmentation shapes. SoundEmbedding.embedding may carry N numeric vectors or N strings (terms). Timestamps are an M×2 array. M == N means frame‑aligned spans, and M == 1 is an utterance‑level vector. Misalign these and segmentation scoring breaks.
  • Compression claims need context. EncodingStats records input_size_bytes and embedding_size_bytes and exposes compression_ratio. As a worked example: 1 second of float32 audio at SR=16000 is 16, 000 samples × 4 bytes = 64, 000 bytes. A 128‑dim float32 embedding is 128 × 4 = 512 bytes → 64, 000 / 512 ≈ 125× reduction. For 10 seconds that ratio approaches ≈1, 250×, which explains the notebook’s illustrative “thousandfold” remark for longer clips.
  • Runtime and cost vary wildly. Toy spectral baselines run fast on CPU. Replacing them with wav2vec, Whisper or CLAP increases compute and memory. Expect to use GPUs for throughput on large corpora and to measure encoding latency and FLOPs as part of procurement.

How to extend the experiment

  • Implement the encoder contract by defining _setup, _check_input_types, and _encode, with those three methods your encoder becomes a first‑class citizen of the benchmark and the framework handles batching, statistics and validation.
  • Swap in higher‑capacity encoders wrapped by the package (examples include wav2vec, Whisper, CLAP, EnCodec and others) to see how pre‑trained models behave on the same battery.
  • Scale evaluations using mseb.runner with apache‑beam if you need to run published tasks on large datasets or cloud pipelines.
  • Compare against the public leaderboard to contextualize scores: https://huggingface.co/spaces/google/mseb-leaderboard and inspect the repo at https://github.com/google-research/mseb for code and the tutorial notebook.

Who should care, and what to change right away

If your product needs identity‑sensitive audio search, prioritise retrieval metrics (MRR, Recall@k, NDCG) and consider preserving magnitude in embeddings (avoid mandatory L2 normalization before indexing). If you need robust semantic labels, prioritise class discrimination metrics (balanced accuracy, weighted F1) and test invariances to loudness and recording conditions.

For procurement and R&D: don’t pick a model on a single “main_score.” Publish a compact metric vector (for example: [MRR@5, Recall@5, V‑measure, Accuracy] ± std) alongside EncodingStats (bytes per embedding, estimated FLOPs, latency on CPU/GPU). Report run‑to‑run variance for stochastic evaluators and include at least one test on real‑world data.

Key takeaways, questions you should ask (and short answers)

  • What minimum work is required to add a new encoder to MSEB?

    Implement the MultiModalEncoder contract methods: _setup, _check_input_types, and _encode. Once implemented, the framework handles batching, validation and evaluation plumbing.

  • Why do two encoders disagree on which is “better”?

    Different evaluators reward different invariances: retrieval favors identity cues (envelope/recording fingerprint), classification favors semantic cues (timbre). A model can excel at one task and lag on another depending on what its representation preserves.

  • How do I avoid non‑deterministic clustering results?

    Pass a fixed random_state to the clustering algorithm or seed NumPy’s global RNG (np.random.seed). Repeat experiments across multiple seeds and report mean ± std for metrics like V‑measure.

  • What does M vs N mean in SoundEmbedding timestamps?

    M is the number of timestamp spans and N is the number of embeddings. M == N indicates frame‑aligned timestamps; M == 1 indicates a single utterance‑level embedding with one span.

  • How do I avoid the NDCG silent‑zero trap?

    Pass the relevant document as a single string ID (not a list) when compute_ndcg_at_k expects one relevant item. The utility compares by equality and supplying a list can silently produce 0.0 while other metrics (like MRR) still behave sensibly.

Benchmarks shape engineering priorities. If you use sound embeddings in production, match your evaluation battery to product needs, report a few complementary metrics plus encoding costs, and codify reproducibility practices so comparisons are meaningful. The MSEB design, types, encoder contract, and evaluators, gives a lightweight, extensible way to do that; the synthetic notebook shows the principle. For code, examples and the original tutorial notebook, see the MSEB repo: https://github.com/google-research/mseb and the public leaderboard at https://huggingface.co/spaces/google/mseb-leaderboard.

0