(PTFS06-04-26) Model Architecture Dominates Nutritional Estimation Accuracy in Vision-Language Systems

Objectives: Vision-language models (VLMs) enable automated dietary assessment from food photographs, yet optimal deployment strategies remain unestablished. We systematically evaluated the relative contributions of model architecture, image configuration, prompt strategy, and image quality to nutritional estimation accuracy, benchmarking VLM performance against professional nutritionists.

Methods: We evaluated 40 VLMs across eight providers using the Nutrition5k dataset and photographs of original dishes. Performance was assessed across image configurations (1–5 angles), prompt strategies (7 approaches), and image quality (laboratory vs. smartphone). Root mean squared logarithmic error (RMSLE) served as the primary metric for caloric and macronutrient estimation. Professional nutritionists provided reference estimates. Variance decomposition quantified the relative contribution of each factor. Multiple testing correction was applied to all pairwise comparisons.

Results: Model architecture dominated performance variance (99.6%). Multi-angle photography provided no benefit over single images (RMSLE: 0.627 vs. 0.623, p = 0.182), and prompt engineering showed no significant effects after correction (all p > 0.05). High-quality consumer smartphone photography significantly outperformed controlled laboratory imaging (RMSLE: 0.548 vs. 0.616, p = 0.020, Cliff's δ = −0.58) despite a substantially smaller sample. Providing ingredient descriptions slightly improved aggregate performance (RMSLE: 0.548 → 0.516, p = 0.039), with heterogeneous model-level responses (−19.1% to +3.4%). Professional nutritionists substantially outperformed all VLMs (RMSLE: 0.176 vs. 0.443 for the best-performing model, a 152% relative gap), with protein estimation showing the largest discrepancy (672% higher error for VLMs).

Conclusions: Model selection and image quality are the dominant factors in VLM-based nutritional estimation accuracy, while multi-view imaging and prompt complexity contribute negligibly. Current VLMs may suit consumer-level calorie tracking, but achieving clinical-grade macronutrient profiling will require advances in model architecture rather than deployment optimization.

Funding Sources: No external funding