Imagine spending months training a neural network, tuning hyperparameters, filing the paper, and then someone runs a library-lookup script from a decade ago and beats you on the leaderboard. That is not a hypothetical. It is, according to Nguyen, Overstreet, King, and Ciesielski writing in the Journal of the American Society for Mass Spectrometry, roughly what is happening in machine learning for small-molecule structure elucidation via tandem mass spectrometry. The finding is counterintuitive enough to stop you mid-scroll: in a domain where AlphaFold's success has primed everyone to expect deep learning to steamroll classical methods, ML models are struggling to beat simple baselines. That result deserves more than a footnote. ## What Mass Spectrometry Actually Asks of a Model Mass spectrometry is the technique scientists use to identify a molecule by fragmenting it and measuring the mass-to-charge ratios of the resulting pieces. Think of it as identifying a shredded document by weighing the confetti. For small molecules, including metabolites, drugs, and environmental contaminants, the standard workflow involves matching an observed spectrum against a reference library of known spectra. As Nguyen et al. explain in their JASMS paper, this library-matching strategy is popular but fundamentally limited by whatever molecules happen to already be in the library. That coverage gap is precisely why researchers got excited about ML: if you could predict a spectrum for any molecule from its structure alone, you could build a synthetic library covering chemical space far beyond what experimentalists have measured. The promise is real. The execution is where things get complicated. The core difficulty, according to Nguyen et al., is that tandem MS/MS data is noisy, sparse, and deeply sensitive to experimental conditions. ML predictions are especially unreliable at low collision energies, and models struggle to generalize across the wide structural diversity of small molecules. That diversity is not a minor inconvenience: a model trained on one chemical class can fail entirely on another. And the data quality problems do not announce themselves in a loss curve. ## The Benchmarking Trap, Explained Without Mercy Here is where the lesson gets broadly applicable. Nguyen et al. identify what they call "generic machine learning benchmarking tactics" as a primary driver of misleading accuracy scores in this field. The mechanics are familiar to anyone who has read enough ML papers: you partition your dataset, train on the majority, evaluate on a held-out slice, report a strong number, and submit. The problem, as the JASMS paper makes explicit, is that this approach does not account for the particular structure of mass spectrometry data. When your training and test sets share similar chemical scaffolds because you split randomly rather than by molecular structure, your model essentially memorizes patterns it will never see in deployment. The benchmark looks great. The real-world performance does not. This is not a niche complaint about one subfield. It is a specific, named instance of a general failure mode: evaluation sets that are too similar to training sets, producing numbers that flatter the method rather than test it. The MassSpecGym benchmark, introduced at NeurIPS 2024 by Bushuiev and colleagues from institutions including the Czech Academy of Sciences, Czech Technical University, Wageningen University, and the University of Toronto, represents a direct attempt to address this by providing a shared, rigorous evaluation framework for molecule discovery and identification tasks. Structured benchmarks that force genuine generalization are how a field earns the right to claim progress. ## What Good Evaluation Actually Looks Like Nguyen et al. are specific about what needs to change, and their recommendations are worth treating as a checklist rather than a suggestion box. First: curate your datasets carefully, because garbage-in guarantees garbage-benchmark. Second: restrict predictions to sufficiently high collision energies where the signal is cleaner and the task is better defined. Third, and perhaps most importantly: work more closely with experimental mass spectrometrists. That last point is less about humility and more about epistemics. Domain experts know which failure modes matter in practice and which benchmark wins are purely academic. Ignoring them is how you end up with a model that posts strong numbers on a leaderboard while a lookup table beats it in a real lab. The self-supervised approach reported by Bittremieux and Noble in Nature Biotechnology offers a complementary angle: training a foundation model called DreaMS on large-scale, publicly available MS/MS repositories using a two-stage self-supervised framework. The idea is that learning rich representations from massive unlabeled data before fine-tuning could reduce the model's dependence on narrowly curated labeled sets. It is a promising direction, and it also illustrates that the field is actively self-correcting rather than ignoring the problem. ## What This Means for ML Practitioners The mass spectrometry story is a clean, well-documented case study in a pattern that shows up across applied ML: a complex domain with limited labeled data, high structural variability, and experimental noise is a hostile environment for generic benchmarking. The models are not necessarily bad. The evaluation frameworks are often just not measuring what they claim to measure. Every time you see a paper reporting large accuracy improvements over prior work in a specialized scientific domain, the first question worth asking is not "what model did they use?" but "how did they split the data, and does that split reflect real deployment conditions?" For learners building their ML intuition, this episode is genuinely useful. It suggests that reading the evaluation section of a paper as carefully as the architecture section is not pedantry; it is the skill that separates practitioners who can transfer methods to new problems from those who reproduce benchmark numbers and wonder why nothing works in production. Watch the MassSpecGym benchmark for how the community responds to structured evaluation, and watch whether the next wave of MS/MS papers actually tests generalization across chemical classes. That will be the real signal. ## Sources - Advancing the Prediction of MS/MS Spectra Using Machine Learning

Sources