(P38-04-26) Development and Evaluation of Computer Vision and Language Models for Food Group and Price Analysis of Menu Images Collected From Diverse Food Outlets in Bangkok, Thailand

Objectives: Using publicly available AI models, we developed a software pipeline for analyzing menu images to quantify availability and price of nutritionally-relevant food groups at diverse food outlets in Bangkok.

Methods: We evaluated and iteratively optimized performance of optical character recognition (OCR), large language models (LLM), and Visual Question Answering models for extracting and structuring food names and prices from a subset of 30 manually labelled menu images (containing 512 foods) collected as part of a participatory mapathon of Bangkok food environments. We also compared two approaches, Retrieval-Augmented Generation and a deterministic search engine, for mapping extracted foods to a purpose-built “food quality database” of 15,307 Thai and international foods coded at the ingredient level against 25 Global Diet Quality Score food groups. Nutritional analysis of the full pipeline involved comparing food group breakdowns between manually assigned database entries and those automatically mapped by the pipeline.

Results: The completed pipeline employs a bilingual (Thai/English) Vision-Language Model (Typhoon-OCR-7B) for extracting food names and prices, Mistral Nemo Instruct-2407 LLM and automated post-processing for structuring results, a weighted ensemble of 5 string-similarity matching algorithms and Thai Unicode normalization to map food names to food quality database entries, and optimization for high-throughput processing of high-resolution images. At a 50% fuzzy similarity threshold, 88.2% of food names identified by extraction and structuring models accurately matched their manual labels (precision) while 71.1% of all foods actually on the menus were accurately enumerated (recall). For 60.4% of menu items, food group breakdowns were identical between manually assigned and automatically mapped database entries, while error among the 39.6% of imperfect matches was generally modest (often consisting of near-neighbor food group classifications that largely preserved the overall nutrition profile).

Conclusions: The pipeline reliably extracts food names and prices in menu images from Bangkok food outlets. Ongoing research involves large-scale implementation for spatial analysis of food quality and price using menus collected across the city.

Funding Sources: Mahidol University Strategic Research Fund