For decades, the Electronic Health Record (EHR) has served as the digital ledger of modern medicine. While these databases contain vast amounts of longitudinal information—from blood pressure readings to medication histories—they represent only a fraction of the human health story. Today, a landmark collaboration between Verily Health, NVIDIA, and the National Institutes of Health (NIH) All of Us Research Program is shifting the paradigm, moving beyond the EHR to create the first multimodal foundation model that fuses clinical history with genomic data.
This breakthrough, materialized in a new tool called Forecast™ 1.0, marks a significant leap toward a truly holistic understanding of patient health, promising to transform how we predict, diagnose, and treat chronic disease.
The Main Facts: Bridging Nature and Nurture
At its core, the project addresses the "nature versus nurture" divide that has long hampered clinical AI. EHRs provide the "nurture" aspect—the lived clinical experience, environmental exposures, and treatment responses. Genomic data, conversely, captures the "nature"—the inherited biological predispositions that dictate disease risk long before a symptom ever manifests.
Until now, these two data streams existed in silos. Genomic data is static and complex, while clinical data is dynamic and often unstructured. Verily’s new foundation model successfully integrates these disparate sources, allowing the AI to learn broad, cross-modal patterns. By training on the massive, diverse dataset provided by the All of Us Research Program, the model identifies complex risk markers that were previously invisible to systems relying solely on clinical notes or billing codes.
A Chronological Evolution of Clinical AI
The journey to this multimodal milestone did not happen overnight. The evolution of healthcare AI can be broken down into three distinct eras:
- The Era of Unimodal Analytics (2010–2018): Early clinical AI focused on single-modality tasks, such as using natural language processing (NLP) to read physician notes or using computer vision to analyze X-rays. These systems were powerful but narrow, often failing to account for the systemic context of a patient’s health.
- The Foundation Model Shift (2019–2022): The industry began to move toward "foundation models"—large-scale, pre-trained AI models capable of performing a variety of downstream tasks. This allowed researchers to move away from building custom models for every specific disease and toward more generalizable, adaptable intelligence.
- The Multimodal Integration (2023–Present): With the advent of high-performance computing, the focus shifted to "modality fusion." The Verily-NVIDIA project represents the zenith of this phase, utilizing the All of Us dataset to prove that combining genetic risk scores with longitudinal EHR data yields a superior predictive performance that neither modality could achieve alone.
Supporting Data: Quantifying the Breakthrough
The efficacy of the Forecast™ 1.0 model is backed by rigorous performance metrics that highlight both clinical and computational superiority.
Clinical Performance
The primary test case for the model was Type 2 diabetes prediction. When the team integrated genetic risk scores into their EHR-based model, the results were transformative:
- Accuracy: The model demonstrated a substantially higher ability to identify patients at high risk for Type 2 diabetes.
- Precision: By reducing false positives (incorrect alerts), the model ensured that clinicians could focus on high-risk patients who truly needed intervention.
- Temporal Reach: Unlike conventional models that struggle with short-term vs. long-term projections, this model successfully predicted disease risk over clinically meaningful windows of five to ten years, offering a roadmap for early prevention rather than just reactive treatment.
Computational Efficiency
Processing genomic and clinical data simultaneously requires immense power, which has historically been a barrier to entry for research institutions. By leveraging NVIDIA H100 GPUs and the NVIDIA NeMo AutoModel framework, the team achieved a threefold increase in pre-training efficiency compared to standard workflows like Hugging Face Accelerate. This acceleration is critical; it reduces the "time-to-insight," allowing researchers to iterate faster and bring innovations to the bedside with unprecedented speed.
Official Responses and Expert Perspectives
Jonathan Amar, Senior Manager of Verily Data Science, emphasizes that the EHR is only one chapter in a patient’s life. "To truly understand disease risk, AI needs to learn from multiple dimensions of human health, including both our biology at birth and our lived clinical experience," Amar stated during the project’s announcement.
He further highlighted the strategic importance of the collaboration: "By bringing together two types of typically mismatched data, we demonstrated how multimodal AI can improve disease prediction while laying the groundwork for a future that incorporates many more health data modalities."
The involvement of the NIH All of Us program was equally pivotal. By providing a diverse, large-scale dataset, the program allowed the researchers to bypass the historical limitation of data scarcity. As Amar noted, this democratizes the process, providing a "preview of what’s possible in areas such as pharmaceutical biomarker discovery, diagnostic risk stratification, and health system precision medicine programs."
Implications: The Future of Precision Medicine
The release of Forecast™ 1.0 on GitHub is not merely an open-source contribution; it is a call to action for the global research community. By providing researchers with access to these tools within a secure Trusted Research Environment (TRE), Verily is facilitating a transition from fragmented research to a unified, scalable approach to precision medicine.
1. Beyond Diabetes: The Blueprint for Multimodality
The success with Type 2 diabetes is a proof-of-concept. The real-world application of this architecture could eventually incorporate:
- Wearable Data: Integrating continuous glucose monitoring or heart rate variability from smartwatches to track real-time patient health.
- Medical Imaging: Using MRI or CT scans as inputs alongside genomic and clinical data for early cancer detection.
- Patient-Reported Outcomes: Incorporating subjective data regarding quality of life, which is often missing from clinical databases.
2. Democratizing Advanced Research
One of the most significant implications is the lowering of the technical barrier. Computational complexity has long been a "gatekeeper" in medical AI. By optimizing the model for H100 GPUs and making the methodology transparent, the project ensures that smaller research hospitals and academic institutions—not just tech giants—can leverage foundation models to improve patient outcomes in their specific populations.
3. The Shift to Proactive Care
Perhaps the most profound implication is the shift in the clinical timeline. Current healthcare systems are largely designed to manage disease after symptoms appear. By identifying high-risk individuals years in advance, clinicians can initiate lifestyle interventions, early screenings, or preventative pharmacological treatments. This shift from "sick-care" to "preventative-care" has the potential to alleviate the immense financial and social burden of chronic disease.
Conclusion: A High-Resolution View of Health
As Verily continues to evolve the Forecast™ platform, the ultimate goal remains clear: to create an increasingly high-resolution, dynamic picture of human health. By continuously enriching the datasets—moving from EHRs and genomics into the realm of unstructured clinician notes and real-time biometric sensors—the model will move closer to a "digital twin" of patient health.
The collaboration between Verily, NVIDIA, and the NIH serves as a critical milestone. It proves that the future of medicine lies not in deeper specialization of one data type, but in the sophisticated, intelligent integration of all data types. As this technology matures, we are entering an era where the "story" of a patient’s health will be written with a precision that was once thought to be the realm of science fiction. Researchers interested in exploring the underlying architecture and results of this project are encouraged to review the official whitepaper and engage with the Forecast™ 1.0 repository to begin building the next generation of predictive medicine.
About the Author
