🧬 RareGraph

RareGraph is a reproducible pipeline for building a MONDO-centered rare-disease knowledge graph from OMIM, GeneReviews, and Orphadata/Orphanet. It uses Gemini 2.5 Pro through Vertex AI batch inference to structure narrative OMIM and GeneReviews content, parses Orphadata deterministically, normalizes concepts to HPO and MONDO, and merges the sources into a disease-centered graph.

The final knowledge graph v1.0.0 can be downloaded from HuggingFace at quannguyenminh103/RareGraph.

Important

This repository contains code and safe placeholders only. Licensed source data, API keys, credentials, intermediate files, and generated graphs are excluded from Git.

📚 Documentation

Guide	Use it for
Running RareGraph	Installation, Google Cloud setup, preflight checks, smoke tests, full runs, SLURM, resuming, and output verification
Obtaining source data	OMIM access and API keys, GeneReviews bulk NXML, Orphadata, HPO, MONDO, licenses, and exact file locations
Data dictionary	Record identifiers, all clinical and biological fields, controlled vocabularies, nullability, and finalization rules

✨ Main features

• Extracts structured, evidence-bearing records from OMIM and GeneReviews with field-grouped Gemini prompts.
• Parses Orphadata XML without an LLM, including disease identifiers, genes, phenotypes, classifications, epidemiology, natural history, and HOOM content.
• Normalizes diseases and phenotypes to MONDO and HPO and preserves OMIM and ORPHA cross-references.
• Integrates phenotypes, genes, inheritance, demographics, diagnosis, testing, management, counseling, pathways, animal models, prognosis, and differential diagnosis.
• Validates model responses against source-specific schemas before normalization and merging.
• Supports resumable request generation, Vertex AI batch submission, response collection, embedding caches, and per-record outputs.
• Provides one configuration-driven CLI and an ordered SLURM workflow.
• Applies deterministic post-merge quality controls documented in the data dictionary.

🗺️ Pipeline overview

OMIM files + API key -> OMIM API -> Gemini batch -> validate -> normalize ----+
                                                                            |
GeneReviews NXML -----> parse -----> Gemini batch -> validate -> normalize --+--> MONDO merge --> RareGraph.json
                                                                            |
Orphadata XML --------> deterministic extraction ----------------------------+
                                                                            |
HPO + MONDO ----------> identifiers, hierarchy, and semantic normalization --+

Gemini is used for narrative OMIM and GeneReviews content. Orphadata is already structured and is processed deterministically.

✅ Requirements

• Linux or an HPC login node with Python 3.10 or newer
• Internet access for installation and public-source downloads
• A Google Cloud project with Vertex AI and Cloud Storage access
• Google Application Default Credentials, workload identity, or an attached service account
• Authorized OMIM API/download access
• The GeneReviews bulk NXML package
• Optional CUDA GPU access for faster BioLORD normalization; CPU is supported

See Obtaining source data before setup. It explains how to request an OMIM API key, where the key belongs, and how to obtain every source.

🚀 Quick start

From a fresh clone:

cd RareGraph
bash scripts/bootstrap.sh

Then:

1. Follow Obtaining source data to configure .env and place the authorized OMIM and GeneReviews inputs.

2. Run the no-execution preflight check:

   bash scripts/preflight.sh

3. Run a five-record smoke test:

   raregraph --config config/pipeline.json run --limit 5

4. Run the complete pipeline interactively or through SLURM:

   raregraph --config config/pipeline.json run
   # or
   bash slurm/submit_all.slurm

The smoke test and full-run commands perform real downloads, API calls, and cloud work. Read Running RareGraph before executing them.

📦 Output

The final disease-keyed knowledge graph is written to:

data/output/RareGraph.json

Normalized source graphs, the structured MONDO hierarchy, per-disease records, batch manifests, and reusable caches remain under data/. This directory is ignored by Git.

🗂️ Repository layout

RareGraph/
|-- config/                    # portable pipeline configuration
|-- datasets/
|   |-- omim/                  # OMIM download, extraction, and normalization
|   |-- genereviews/           # GeneReviews parsing, extraction, and normalization
|   `-- orphanet/              # deterministic Orphadata extraction
|-- integration/               # MONDO construction, final merge, cleanup, and audits
|-- raregraph/                 # unified orchestration CLI
|-- scripts/                   # bootstrap and no-execution preflight helpers
|-- slurm/                     # ordered HPC jobs and dependency launcher
|-- tests/                     # pipeline-contract and post-merge regression tests
|-- RUNNING_RAREGRAPH.md
|-- SOURCE_ACQUISITION.md
`-- DATA_DICTIONARY.md