groundy
Culture & Society

When AI Designs Antibodies, What Does the Binding Score Leave Unverified?

AbFlowNet preprint shows controlled RMSD gains over DiffAb, but authors note the energy estimator is unreliable. The evidence supports ranking candidates, not replacing wet-lb

Published 6 references
Three ivory ceramic antibody-like Y forms with green tips stand in a copper-colored dish beside a separate black ceramic form with a hollow recess. Hard shadows fall across a warm ivory background.
On this page11 sections

An energy-optimized design objective can reorder an antibody shortlist; it cannot settle whether the candidates bind. In the single-team, unreplicated AbFlowNet preprint, a different-seed retraining check on RAbD’s L3 CDR region reports RMSD of 4.06 Å for DiffAb versus 1.362 Å for AbFlowNet, with gradient updates matched to isolate the effect of the training objective.

That is a substantial structural-benchmark gap, but it does not justify replacing diffusion sampling on the strength of the binding score. The practical choice is to keep diffusion as the generative backbone and treat an AbFlowNet-style energy objective as a possible shortlist ranker. Adoption makes sense only when the validation budget still covers affinity, specificity, and developability experiments. Otherwise, the pipeline changes the order of unconfirmed candidates without changing what remains unconfirmed.

The controlled comparison supports a narrower claim than “better binding”

AbFlowNet combines diffusion-based antibody design with a trajectory-balance objective that optimizes estimated binding energy. Its comparison with DiffAb has an unusually useful control: the authors say they matched the number of gradient updates “to isolate the effect of optimizing binding energy via trajectory balance.” They also retrained and evaluated both systems with a different seed to check the reported discrepancy, according to the AbFlowNet paper.

Within that configuration, the design strengthens attribution to the objective rather than to a larger training-update budget. It does not constitute external replication. A same-team, different-seed check addresses robustness to one training choice; another group reproducing the result would address a different question. Independent engagement exists, but on a different benchmark: the ConTact authors retrained AbFlowNet as one of eight baselines on ChiMERa-Bench. That comparison does not test the L3/RAbD figure, so the different-seed check remains the only direct evidence on that number.

The second distinction is between the target of the comparison and the target of discovery. RAbD’s L3 result concerns structural deviation from a native antibody. A discovery team ultimately needs to know whether a designed candidate binds the intended antigen, whether it does so specifically, and whether it is developable. Those are separate evidentiary steps, and the reported RMSD gap does not close them.

Geometry and sequence recovery leave binding chemistry unresolved

Two common leaderboard metrics are particularly easy to overread.

RMSD primarily assesses backbone-atom alignment. It does not include the side chains that mediate antigen, antibody interactions. Amino-acid recovery, or AAR, is a sequence-recovery measure rather than a functional readout, and the AbDPO authors warn that it is susceptible to manipulation and does not precisely gauge generated-antibody quality. They also state that the relationship between in-silico preferences and wet-lab results remains an unresolved scientific issue.

DiffForce, an independent force-guided diffusion paper, repeats the basic limitation: RMSD overlooks side chains crucial to the specificity and strength of antigen, antibody interactions, while AAR may not reliably reflect functional performance.

The consequence is not that these metrics are useless. A backbone comparison can reveal a structural modeling failure, and sequence recovery can characterize resemblance to known antibodies. The problem arises when either number is promoted into evidence of affinity. A candidate can score well on backbone agreement or native-sequence recovery while the experimental questions about its interactions remain open. Developability sits outside both readouts as well.

The binding objective also needs an audit

The energy score is not an independent experimental measurement waiting at the end of the pipeline. It is the computational signal being optimized, and AbFlowNet’s authors are unusually direct about its weakness:

“As the in vitro affinity data for all training complexes are not available and for fair comparison with existing methods, we used Pyrosetta InterfaceAnalyzer, which is an unreliable estimator of binding energy.”

That statement appears in the AbFlowNet preprint itself. The authors add that in-silico methods such as Rosetta and OpenMM Yank have only moderate correlation with real binding energy. They also caution that generated CDRs are not guaranteed to be geometrically plausible, which could reduce estimator reliability further.

Two different uncertainties therefore sit inside an apparently decisive “binding energy” improvement. First, the estimator may correspond only imperfectly to measured binding. Second, its input structures may deviate from geometric plausibility in ways that make the estimate less dependable.

Moderate correlation does not imply zero information. An estimator might still help order candidates. What the evidence does not support is converting a given improvement in that score into a proportional improvement in measured affinity. An optimizer can improve its reward while leaving real binding unresolved; that is an inference from the mismatch between proxy and experimental endpoint, not a demonstration that AbFlowNet’s candidates bind poorly.

Map each computational result to its experimental obligation

A validation plan becomes clearer when every metric is assigned a limited claim and a remaining obligation.

Computational readoutWhat it can supportWhat it still leaves unverified
RMSDBackbone-atom agreement with a reference structure in the tested benchmark configurationSide-chain-mediated interaction quality, measured affinity, specificity, and developability
Amino-acid recoveryRecovery of native sequence patterns; useful as a bounded sequence-similarity measureFunctional performance, binding affinity, specificity, and developability; the metric is also open to manipulation
In-silico binding energyCandidate ordering under a named estimator such as Pyrosetta InterfaceAnalyzerMeasured binding energy and affinity; the estimator is described as unreliable, with only moderate correlation reported for related methods
Native-contact recoveryAspects of where a designed loop sits relative to the antigenAntigen-specific residue identities and, consequently, the downstream functional result

The last row supplies a useful warning against fixing the leaderboard by simply choosing a more interface-focused metric. ConTact reports an 11–16% improvement in native-contact recovery while matching the best sequence recovery. Yet its authors state that better loop placement does not translate into antigen-specific residue identities. Improving an intermediate readout can therefore leave the end goal untouched.

The same discipline helps when interpreting smaller headline deltas. On SAbDab’s CDR-H3 task versus DiffAb, AbMEGD reports a 10.13% increase in amino-acid recovery, a 3.32% rise in “improvement percentage,” and a 0.062 Å RMSD reduction. Those figures describe its reported benchmark comparison; they do not supply affinity, specificity, or developability evidence. Nor should they be ranked directly against AbFlowNet’s L3 numbers as though different tasks and metrics formed one continuous scale.

Experimental benchmarks argue for enrichment, not exemption

The strongest independent check among these papers is AbBiBench, which evaluates computational binding-affinity models using 216,560 harmonized experimental measurements across 13 antibodies and 9 antigens. Its evaluation unit is the antibody, antigen complex, rather than an isolated structural-plausibility score.

Its conclusion cuts directly against leaderboard-driven validation exemptions: sequence or structural plausibility does not necessarily imply improved antibody, antigen binding affinity. Its operational recommendation is equally specific:

“Current scores should therefore be used primarily to enrich experimental shortlists; a low model rank alone should not be treated as sufficient evidence to exclude a variant from experimental testing.”

That rule cuts both ways. A high score is not permission to skip the assay, and a low score is not, by itself, sufficient reason to discard a variant. AbBiBench does not directly validate or invalidate AbFlowNet’s generated candidates; it supplies experiment-anchored evidence about how computational scores should inform testing.

The experimental failure record adds another reason to preserve that gate. DiffForce reports that many antibodies sampled in silico from diffusion models fail to demonstrate functionality when tested in vitro. “Many” is not a universal failure rate, and the cited papers do not establish how often any particular pipeline fails. It is enough to reject the assumption that computational sampling routinely produces functional candidates.

What should change in the validation plan?

On this evidence, I would trial energy optimization as a ranking layer, not adopt it as a replacement for diffusion sampling or experimental screening. The adoption criteria are concrete:

  • Keep the computational claim bounded. Report the benchmark, CDR region, baseline, estimator, gradient-update control, and seed check alongside the result. Do not relabel an RMSD or InterfaceAnalyzer improvement as measured affinity.
  • Evaluate ranking against experimental outcomes. If the objective is meant to improve discovery, its useful test is whether its ordering improves the experimental shortlist relative to the team’s current ranking process. The RAbD result alone cannot answer that.
  • Retain affinity, specificity, and developability gates. A candidate should not acquire a functional claim merely by receiving a favorable structural or energy score.
  • Do not use low rank as an automatic exclusion rule. AbBiBench explicitly warns against that practice, so a hard computational cutoff needs evidence beyond model rank alone.
  • Keep in-silico and measured results visibly separate in methods reporting. Readers should be able to tell which claims came from an estimator and which came from an assay.

This also changes the economics of adoption. Computation may produce a better-ordered queue, but nothing in these preprints establishes that it reduces the number of experiments required. Until a team has prospective measurements connecting its chosen score to its own assay outcomes, it cannot responsibly book the RMSD or energy gain as avoided screening work. A larger set of attractive computational candidates may instead move cost and attention toward triage and testing.

That is the second-order consequence reviewers should ask authors to address. If a paper claims more efficient discovery, the reporting obligation extends beyond generation speed or benchmark deltas to what happened after shortlisting. If it claims better binding, measured binding evidence is the relevant support.

What this evidence cannot settle

Every AbFlowNet figure here is bounded to the tested RAbD CDR-design configuration and same-team retraining checks. The other reported comparisons use their own tasks, including SAbDab, and do not establish general performance across antibody-design settings. All of the cited papers are arXiv preprints, several at particular versions, so revision, peer review, or independent replication could change the figures and interpretations.

Most importantly, this analysis does not show that AbFlowNet’s candidates fail experimentally. The preprint reports in-silico evaluation and states that in vitro affinity data were unavailable for the training complexes; it reports no measured binding results for its generated candidates. That is an evidentiary gap, not a demonstrated failure.

The defensible reading is therefore narrower and more useful: AbFlowNet reports a controlled structural-benchmark improvement while optimizing an estimator its authors consider unreliable. That may justify a place in the shortlist-ranking toolkit. It does not retire any part of the experimental validation plan. A leaderboard can help decide what to test next; it cannot decide what the candidate has already proven.

Frequently Asked Questions

What does the AbFlowNet preprint say about the reliability of its binding energy estimator?

“As the in vitro affinity data for all training complexes are not available and for fair comparison with existing methods, we used Pyrosetta InterfaceAnalyzer, which is an unreliable estimator of binding energy.”

How should computational scores be used when deciding which antibody variants to test experimentally?

“Current scores should therefore be used primarily to enrich experimental shortlists; a low model rank alone should not be treated as sufficient evidence to exclude a variant from experimental testing.”

What limitation does RMSD have when evaluating antibody design?

RMSD primarily assesses backbone-atom alignment. It does not include the side chains that mediate antigen, antibody interactions.

References

Follow the links in the article for context. The supporting material is collected here for further reading.

  1. AbFlowNet preprintarxiv.orgAccessed
  2. ConTactarxiv.orgAccessed
  3. AbDPO authorsarxiv.orgAccessed
  4. DiffForcearxiv.orgAccessed
  5. AbMEGDarxiv.orgAccessed
  6. AbBiBencharxiv.orgAccessed

Join the discussion

Share a useful perspective or ask a question about this article.

Discussion guidelinesComments privacy