A Controlled Benchmark of Open-Source and Proprietary LLMs for Few-Shot Microbiome IBD Classification

Phenotyping inflammatory bowel disease (IBD) from gut microbiome profiles remains challenging due to 93% genus-level zero-inflation, skewed amplicon count distributions, and the practical cost of assembling labelled cohorts. Few-shot in-context learning (ICL) with large language models (LLMs) sidesteps the annotation bottleneck, yet existing benchmarks test only proprietary APIs—a deployment model incompatible with the data governance constraints of most clinical sites. We benchmark six frontier LLMs under identical few-shot conditions—three proprietary (GPT-4o, Claude claude-opus-4-5, Gemini 2.5 Flash Lite) and three open-source (Mistral 7B, LLaMA-3 8B, DeepSeek-R1-Distill-Qwen 1.5B)—evaluated against supervised baselines (Random Forest, XGBoost, LightGBM, soft-voting Ensemble) on 16S rRNA amplicon data (n = 1,316; holdout n = 30). All six LLMs received identical prompts, the same log-normalised top-20 features, and the same random shot selection strategy with a fixed seed, ensuring that observed differences are attributable to model capacity rather than experimental conditions. The supervised Ensemble led holdout performance (Macro-F1: 0.7948; AUC: 0.7725). Among LLMs, Mistral 7B achieved the highest Macro-F1 (0.5417), surpassing all three proprietary models without any parameter update. The mean performance gap between open-source (0.4711) and proprietary (0.5101) groups was only 0.039 Macro-F1 points—too narrow to justify exclusive reliance on commercial APIs in privacy-sensitive deployments. Within-open-source variance (0.130 Macro-F1 points) substantially exceeded this inter-family gap, indicating that model selection within the open-source ecosystem is the more consequential practical decision. These results suggest that open-source LLMs running on local hardware are a workable option when labeled data is limited and routing patient-derived data to external APIs is not permitted.

@article{Najih2026,
  title     = {A Controlled Benchmark of Open-Source and Proprietary LLMs for Few-Shot Microbiome IBD Classification},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {17},
  number    = {5},
  year      = {2026},
  publisher = {The Science and Information Organization},
  author    = {Nouhaila En Najih and Soufiane Hamida and Ahmed Moussa},
  doi       = {10.14569/IJACSA.2026.0170596},
  url       = {https://doi.org/10.14569/IJACSA.2026.0170596}
}