(P38-24-26) An Analysis of Transparency, Rigor, Reproducibility, and Verifiability of Randomized Controlled Trials of Beef Consumption and Health Outcomes

Objectives: No systematic assessment of rigor, reproducibility, transparency (RRT) and verifiability of randomized controlled trial (RCT) studies examining beef consumption and health outcomes exists. By evaluating these dimensions, we aimed to characterize the state of the field and establish methods to do so in other areas of nutrition research.

Methods: We systematically searched the literature from inception until January 2024 for RCTs examining the effects of beef consumption on health outcomes in humans. This yielded 92 eligible studies. We included a subset of 48 eligible trials using a stratified random sampling procedure. We contacted the authors of each trial to request individual-level data to computationally reproduce the reported results of the main health outcome analysis. If a study was reproducible, we assessed whether the statistical methods and interpretations were valid for the design (verification). We also assessed each trial for transparency (29 items) and rigor (14 items) drawn from existing reporting guidelines, checklists and experience from our team.

Results: Most studies were not reproducible nor verifiable (n=41/48), largely due to datasets not being made available. Therefore, we could not directly assess the reproducibility of those trials. Of the studies for which we received data (n=10), 7 were reproducible and verifiable. Overall, the extent to which studies adhered to the standard reporting criteria and best design and implementation practices assessed varied. For example, reporting of design (parallel vs. crossover), sample characteristics, and protocol for the intervention were described for all studies, whereas only 12 reported their method of randomization. One trial described as randomized did not use a random method of allocation, 2 trials contained statistical errors related to within-group testing, and 3 failed to account for non-independence of their data.

Conclusions: The minority of beef consumption and health outcomes RCTs reviewed were reproducible and verifiable and also have some areas for improvement for transparency and rigor standards. Improved data sharing practices, fully transparent reporting, and adherence to rigorous research practices are necessary in studies on this topic.

Funding Sources: This work was funded by the Beef Checkoff and supported in part by NIH grants.