Do LLM Agents Actually Improve Protein Design...

Do LLM Agents Actually Improve Protein Design? A Controlled Benchmark Study

Author Name : Aditya Agarwal

DOI: https://doi.org/10.56025/IJARESM.141026210

ABSTRACT LLM-based agents are increasingly proposed for autonomous scientific discovery, including protein engineering. Yet no controlled comparison exists between agentic and non-agentic approaches on established protein fitness benchmarks with experimental ground truth. We introduce PROTAGENT, an LLM agent that iteratively proposes protein mutations using memory, reflection, and domain knowledge, scored by ESM-2 masked marginal likelihood. We evaluate PROTAGENT against five baselines—random search, greedy hill climbing, AdaLead, and Gaussian process Bayesian optimization (GP-UCB)—on eight ProteinGym deep mutational scanning (DMS) benchmarks under a fixed budget of 200 oracle calls per protein. Our results show that PROTAGENT does not significantly outperform any baseline (Cohen’s d = −0.279 vs. random, 95% CI crossing zero). Systematic ablation reveals that neither memory, reflection, nor domain knowledge meaningfully contributes to optimization performance. We identify oracle quality—the correlation between ESM-2 scores and experimental fitness (Spearman ρ ranging from 0.05 to 0.73 across proteins)—as the dominant factor explaining 79.4% of total variance, far outweighing method differences (0.9%). These findings establish that for single mutation protein engineering with current protein language model oracles, the generative model is the bottleneck, not the search strategy.