diff --git a/README.md b/README.md
index 4218a8a79..f1af10192 100644
--- a/README.md
+++ b/README.md
@@ -235,6 +235,10 @@ This tutorial demonstrates how to construct a training workflow of [HoVerNet](ht
##### [Nuclei Classification](./pathology/nuclick#nuclei-classification-model)
The notebook demonstrates examples of training and inference pipelines with interactive annotation for pathology, NuClick is used for delineating nuclei, cells and a squiggle for outlining glands.
+#### **Multimodal**
+##### [Multimodal Early-Fusion Network](./multimodal/nakaseke_multimodal_early_fusion)
+An end-to-end tutorial fusing a 2D radiograph stream (MONAI `DenseNet121`) with a low-dimensional clinical tabular stream via dictionary-based transforms and `torch.cat`, using a fully synthetic, locally generated dataset.
+
#### **Acceleration**
##### [fast_model_training_guide](./acceleration/fast_model_training_guide.md)
The document introduces details of how to profile the training pipeline, how to analyze the dataset and select suitable algorithms, and how to optimize GPU utilization in single GPU, multi-GPUs or even multi-nodes.
diff --git a/multimodal/nakaseke_multimodal_early_fusion/README.md b/multimodal/nakaseke_multimodal_early_fusion/README.md
new file mode 100644
index 000000000..a5832c6ba
--- /dev/null
+++ b/multimodal/nakaseke_multimodal_early_fusion/README.md
@@ -0,0 +1,54 @@
+# Multimodal Early-Fusion Network: Radiographs + Clinical Tabular Data
+
+[](https://colab.research.google.com/github/Project-MONAI/tutorials/blob/main/multimodal/nakaseke_multimodal_early_fusion/multimodal_early_fusion_tutorial.ipynb)
+
+This tutorial demonstrates an **early-fusion** architecture that combines a 2D medical image stream
+with a low-dimensional clinical tabular stream in a single MONAI dictionary-based pipeline, using
+[`multimodal_early_fusion_tutorial.ipynb`](./multimodal_early_fusion_tutorial.ipynb).
+
+## Motivation
+
+Most MONAI tutorials focus on a single imaging modality. In real deployments, especially in
+resource-constrained clinics, an image is rarely read in isolation -- a clinician also has vitals,
+labs, or a short structured history on hand, and a model that ignores that context is throwing away
+signal it doesn't have to. This tutorial shows the minimum end-to-end MONAI/PyTorch pattern for
+combining the two without hand-rolling a custom `Dataset`.
+
+The clinical feature schema (age, BMI, salivary pH, systolic blood pressure) and the binary screening
+task are modeled after a hypertension-screening workflow explored at Nakaseke Hospital, Uganda. The
+underlying patient data is confidential and is **not** included in or downloaded by this tutorial.
+
+## Dataset
+
+**Fully synthetic, generated locally, no download required.** The notebook's
+`simulate_nakaseke_multimodal_dataset()` function creates, on the fly:
+
+- one synthetic 2D radiograph per patient, saved as a NIfTI (`.nii.gz`) file with `nibabel`
+- a 4-dimensional tabular vector `[age, bmi, salivary_ph, systolic_bp]`
+- a binary label
+
+Both modalities are generated from a shared hidden "risk factor" per patient, so neither the image
+nor the tabular vector is fully predictive of the label on its own -- this is what motivates fusing
+them. There is no claim that the notebook's results reflect real-world diagnostic performance; the
+synthetic cohort exists only to exercise the pipeline end-to-end without any real or downloadable
+data.
+
+## What the notebook covers
+
+1. Reproducible setup with `monai.utils.set_determinism`.
+2. Synthetic multimodal data generation and a MONAI-style data manifest
+ (`{"image": ..., "nakaseke_tabular": ..., "label": ...}`).
+3. A dictionary-based `Compose` pipeline where image-only transforms (`LoadImaged`,
+ `EnsureChannelFirstd`, `ScaleIntensityRanged`) touch only the `"image"` key, while the tabular and
+ label keys are cast to tensors with `EnsureTyped` and otherwise left untouched.
+4. `ResilientMultimodalClassifier`: a MONAI `DenseNet121` visual stream (512-d embedding) fused via
+ `torch.cat` with a small tabular projection stream (16-d embedding) into a 528-d representation,
+ followed by a dropout-regularized classification head.
+5. A short training loop (`max_epochs`, `val_interval`) showing validation accuracy improve as the
+ model learns to use both streams.
+
+## Requirements
+
+Everything needed is installed by the notebook's own `Setup environment` cell
+(`monai-weekly[nibabel, tqdm]`, `matplotlib`). No GPU is required -- the default image size (64x64)
+and cohort size (200 patients) are chosen to train in well under a minute on CPU.
diff --git a/multimodal/nakaseke_multimodal_early_fusion/multimodal_early_fusion_tutorial.ipynb b/multimodal/nakaseke_multimodal_early_fusion/multimodal_early_fusion_tutorial.ipynb
new file mode 100644
index 000000000..95b0bd1db
--- /dev/null
+++ b/multimodal/nakaseke_multimodal_early_fusion/multimodal_early_fusion_tutorial.ipynb
@@ -0,0 +1,544 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Copyright (c) MONAI Consortium \n",
+ "Licensed under the Apache License, Version 2.0 (the \"License\"); \n",
+ "you may not use this file except in compliance with the License. \n",
+ "You may obtain a copy of the License at \n",
+ " http://www.apache.org/licenses/LICENSE-2.0 \n",
+ "Unless required by applicable law or agreed to in writing, software \n",
+ "distributed under the License is distributed on an \"AS IS\" BASIS, \n",
+ "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. \n",
+ "See the License for the specific language governing permissions and \n",
+ "limitations under the License.\n",
+ "\n",
+ "# Multimodal Early-Fusion Network: Radiographs + Clinical Tabular Data\n",
+ "\n",
+ "[](https://colab.research.google.com/github/Project-MONAI/tutorials/blob/main/multimodal/nakaseke_multimodal_early_fusion/multimodal_early_fusion_tutorial.ipynb)\n",
+ "\n",
+ "This tutorial builds an end-to-end **early-fusion** classifier that combines a 2D medical image stream with a low-dimensional clinical tabular stream, using only MONAI's dictionary-based transform and dataset APIs plus a small PyTorch fusion network.\n",
+ "\n",
+ "The clinical schema (age, BMI, salivary pH, systolic blood pressure) and the binary screening task are modeled after a real hypertension-screening workflow at Nakaseke Hospital, Uganda. Because that patient data is confidential and cannot be published, **this notebook generates a fully synthetic cohort locally** -- no downloads, no external services, no real patient data -- while keeping the same feature schema, tensor shapes, and modeling problem, so the pipeline is a drop-in template for a real (IRB-approved, de-identified) dataset with matching keys.\n",
+ "\n",
+ "The synthetic generator ties both modalities to a shared hidden \"risk factor\" per patient, so neither the image nor the tabular vector is fully predictive on its own -- this is what makes the fusion architecture worth demonstrating, rather than a stream that could win alone.\n",
+ "\n",
+ "**Why early fusion, and why it fits this problem specifically.** Fusion strategies are usually grouped into early (feature-level), late (decision-level), and joint/intermediate variants. Late fusion -- training an image classifier and a tabular classifier independently and averaging their predictions -- is the more common shortcut, but it caps the model's expressive power at the sum of two univariate opinions: it cannot learn that, say, a mildly ambiguous radiograph combined with a specific BMI/BP combination is jointly more informative than either reading alone. Early fusion, by concatenating learned embeddings from both streams before a shared classification head, lets the network learn exactly those cross-modal interactions. The cost is exactly what this notebook is designed to show how to pay safely: image and tabular data live on very different scales and shapes, and MONAI's dictionary transforms are what make combining them straightforward without writing a custom `Dataset`.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Setup environment"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!python -c \"import monai\" || pip install -q \"monai-weekly[nibabel, tqdm]\"\n",
+ "!python -c \"import matplotlib\" || pip install -q matplotlib\n",
+ "%matplotlib inline"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Setup imports"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "import shutil\n",
+ "import tempfile\n",
+ "from typing import Any\n",
+ "\n",
+ "import matplotlib.pyplot as plt\n",
+ "import nibabel as nib\n",
+ "import numpy as np\n",
+ "import torch\n",
+ "import torch.nn as nn\n",
+ "from monai.config import print_config\n",
+ "from monai.data import DataLoader, Dataset\n",
+ "from monai.metrics import ROCAUCMetric\n",
+ "from monai.networks.nets import DenseNet121\n",
+ "from monai.transforms import Compose, EnsureChannelFirstd, EnsureTyped, LoadImaged, ScaleIntensityRanged\n",
+ "from monai.utils import set_determinism\n",
+ "\n",
+ "print_config()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Reproducibility\n",
+ "\n",
+ "MONAI's `set_determinism` seeds Python's `random`, NumPy, and PyTorch (including CUDA) in one call, so the synthetic cohort, model initialization, and training loop below are reproducible run to run."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "set_determinism(seed=42)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Setup data directory\n",
+ "\n",
+ "You can specify a directory with the `MONAI_DATA_DIRECTORY` environment variable. \n",
+ "This allows you to save results and reuse the generated files. \n",
+ "If not specified, a temporary directory is used and removed at the end of the notebook."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "directory = os.environ.get(\"MONAI_DATA_DIRECTORY\")\n",
+ "if directory is not None:\n",
+ " os.makedirs(directory, exist_ok=True)\n",
+ "root_dir = tempfile.mkdtemp() if directory is None else directory\n",
+ "print(root_dir)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## The clinical data generator (Nakaseke-inspired, fully synthetic)\n",
+ "\n",
+ "`simulate_nakaseke_multimodal_dataset` creates, entirely on disk locally:\n",
+ "- one 2D synthetic radiograph slice per patient, saved as a standard NIfTI (`.nii.gz`) file with `nibabel`\n",
+ "- a 4-dimensional tabular vector `[age, bmi, salivary_ph, systolic_bp]`\n",
+ "- a binary screening label\n",
+ "\n",
+ "and returns a MONAI-style data manifest: a list of dictionaries `{\"image\": path, \"nakaseke_tabular\": array, \"label\": int}`.\n",
+ "\n",
+ "**On the radiograph itself:** this is a *stylized synthetic density map*, not a rendering of real anatomy -- generated by low-pass filtering white noise into a smooth spatial field, then applying a radial vignette so density fades toward the edges (see the visualization below). The point is not to look like a diagnostic-quality X-ray; it is to give the CNN stream a spatially *smooth*, *bounded* structure whose average density is deliberately correlated with `risk_factor` -- unlike independent per-pixel noise, which a convolutional network could still exploit through its mean but which would not visually resemble anything a clinician would recognize as tissue."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def _generate_smooth_density_field(rng: np.random.Generator, image_size: int, cutoff: float = 0.12) -> np.ndarray:\n",
+ " \"\"\"Low-pass filter white noise into a smooth, blob-like spatial field.\n",
+ "\n",
+ " Real tissue density varies smoothly in space; independent per-pixel noise\n",
+ " does not, and looks like static rather than anatomy. An FFT low-pass\n",
+ " filter is the simplest way to get that smoothness using only NumPy, so\n",
+ " the notebook stays free of extra plotting/image dependencies.\n",
+ " \"\"\"\n",
+ " noise = rng.normal(0.0, 1.0, size=(image_size, image_size))\n",
+ " freqs = np.fft.fftfreq(image_size)\n",
+ " freq_x, freq_y = np.meshgrid(freqs, freqs)\n",
+ " radial_frequency = np.sqrt(freq_x**2 + freq_y**2)\n",
+ " low_pass_mask = (radial_frequency < cutoff).astype(np.float32)\n",
+ " smoothed = np.fft.ifft2(np.fft.fft2(noise) * low_pass_mask).real\n",
+ " return smoothed / (smoothed.std() + 1e-6)\n",
+ "\n",
+ "\n",
+ "def simulate_nakaseke_multimodal_dataset(\n",
+ " root_dir: str,\n",
+ " num_patients: int = 200,\n",
+ " image_size: int = 64,\n",
+ " seed: int = 42,\n",
+ ") -> list[dict[str, Any]]:\n",
+ " \"\"\"Generate a synthetic multimodal cohort mimicking the Nakaseke Hospital\n",
+ " hypertension-screening schema (2D radiograph + 4 clinical features).\n",
+ "\n",
+ " No real patient data is used or required. A shared latent ``risk_factor``\n",
+ " per patient drives both the systolic blood pressure and the mean\n",
+ " radiograph density, so that neither modality alone is fully predictive\n",
+ " of the label.\n",
+ " \"\"\"\n",
+ " image_dir = os.path.join(root_dir, \"nakaseke_synthetic_images\")\n",
+ " os.makedirs(image_dir, exist_ok=True)\n",
+ "\n",
+ " rng = np.random.default_rng(seed)\n",
+ " affine = np.eye(4)\n",
+ " data_manifest: list[dict[str, Any]] = []\n",
+ "\n",
+ " # Radial vignette: fades density toward the edges so each synthetic scan\n",
+ " # reads as one bounded structure rather than texture filling the frame.\n",
+ " # It is identical for every patient, so it is computed once, outside the loop.\n",
+ " yy, xx = np.mgrid[0:image_size, 0:image_size]\n",
+ " center = (image_size - 1) / 2\n",
+ " radius_from_center = np.sqrt((xx - center) ** 2 + (yy - center) ** 2)\n",
+ " vignette = np.clip(1.0 - (radius_from_center / radius_from_center.max()) ** 1.5, 0.0, 1.0)\n",
+ "\n",
+ " for patient_idx in range(num_patients):\n",
+ " risk_factor = rng.normal(loc=0.0, scale=1.0)\n",
+ "\n",
+ " age = float(np.clip(rng.normal(45, 15), 18, 90))\n",
+ " bmi = float(np.clip(rng.normal(24, 5), 15, 45))\n",
+ " salivary_ph = float(np.clip(rng.normal(6.8, 0.4), 5.5, 8.0))\n",
+ " systolic_bp = float(np.clip(125 + 15 * risk_factor + rng.normal(0, 8), 90, 200))\n",
+ " tabular_features = np.array([age, bmi, salivary_ph, systolic_bp], dtype=np.float32)\n",
+ "\n",
+ " base_intensity = 90.0 + 25.0 * risk_factor\n",
+ " density_field = _generate_smooth_density_field(rng, image_size)\n",
+ " structure = 55.0 * density_field * vignette\n",
+ " fine_grain = rng.normal(0, 6, size=(image_size, image_size))\n",
+ " radiograph = np.clip(base_intensity + structure + fine_grain, 0, 255).astype(np.float32)\n",
+ "\n",
+ " label_logit = risk_factor + rng.normal(0, 0.6)\n",
+ " label = int(label_logit > 0)\n",
+ "\n",
+ " scan_path = os.path.join(image_dir, f\"patient_{patient_idx:04d}.nii.gz\")\n",
+ " nib.save(nib.Nifti1Image(radiograph, affine), scan_path)\n",
+ "\n",
+ " data_manifest.append(\n",
+ " {\n",
+ " \"image\": scan_path,\n",
+ " \"nakaseke_tabular\": tabular_features,\n",
+ " \"label\": label,\n",
+ " }\n",
+ " )\n",
+ "\n",
+ " return data_manifest\n",
+ "\n",
+ "\n",
+ "data_manifest = simulate_nakaseke_multimodal_dataset(root_dir, num_patients=200, image_size=64)\n",
+ "train_files, val_files = data_manifest[:160], data_manifest[160:]\n",
+ "\n",
+ "print(\n",
+ " f\"Generated {len(data_manifest)} synthetic patient records \"\n",
+ " f\"({len(train_files)} train / {len(val_files)} validation).\"\n",
+ ")\n",
+ "print(\"Example record keys:\", list(data_manifest[0].keys()))\n",
+ "print(\"Example tabular vector [age, bmi, salivary_ph, systolic_bp]:\", data_manifest[0][\"nakaseke_tabular\"])\n",
+ "print(\"Example label:\", data_manifest[0][\"label\"])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Advanced dictionary transforms: preserving the tabular stream\n",
+ "\n",
+ "The `Compose` pipeline below only ever targets the `\"image\"` key for image-specific processing\n",
+ "(`LoadImaged`, `EnsureChannelFirstd`, `ScaleIntensityRanged` to normalize scanner variation that is\n",
+ "common across rural-clinic radiograph equipment). The `\"nakaseke_tabular\"` and `\"label\"` entries never\n",
+ "pass through any image transform -- they are only explicitly cast to PyTorch tensors with `EnsureTyped`,\n",
+ "which is what lets a single dictionary-based pipeline carry heterogeneous, non-image data safely\n",
+ "alongside imaging data."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "multimodal_transforms = Compose(\n",
+ " [\n",
+ " LoadImaged(keys=\"image\", image_only=True),\n",
+ " EnsureChannelFirstd(keys=\"image\"),\n",
+ " ScaleIntensityRanged(keys=\"image\", a_min=0, a_max=255, b_min=0.0, b_max=1.0, clip=True),\n",
+ " EnsureTyped(keys=[\"nakaseke_tabular\", \"label\"]),\n",
+ " ]\n",
+ ")\n",
+ "\n",
+ "train_ds = Dataset(data=train_files, transform=multimodal_transforms)\n",
+ "val_ds = Dataset(data=val_files, transform=multimodal_transforms)\n",
+ "\n",
+ "train_loader = DataLoader(train_ds, batch_size=8, shuffle=True, num_workers=0)\n",
+ "val_loader = DataLoader(val_ds, batch_size=8, shuffle=False, num_workers=0)\n",
+ "\n",
+ "sanity_batch = next(iter(train_loader))\n",
+ "print(\"Sanity check on one batch:\")\n",
+ "print(\" image shape: \", sanity_batch[\"image\"].shape)\n",
+ "print(\" nakaseke_tabular shape: \", sanity_batch[\"nakaseke_tabular\"].shape)\n",
+ "print(\" label shape: \", sanity_batch[\"label\"].shape)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Visualize one synthetic sample\n",
+ "\n",
+ "Expect a soft, blobby grayscale pattern that fades out toward the edges -- not a sharp anatomical image, and not flat noise either. If it instead looks like pure salt-and-pepper static, `_generate_smooth_density_field`'s FFT low-pass step did not run; if it looks like a hard geometric grid, the vignette exponent is too aggressive for the chosen `image_size`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sample = val_ds[0]\n",
+ "plt.imshow(sample[\"image\"][0], cmap=\"gray\")\n",
+ "plt.title(f\"Synthetic radiograph | label={int(sample['label'])}\")\n",
+ "plt.axis(\"off\")\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## The multimodal early-fusion architecture\n",
+ "\n",
+ "`ResilientMultimodalClassifier` has two independent streams and a fusion junction:\n",
+ "\n",
+ "- **Stream 1 (visual):** a MONAI `DenseNet121` (`spatial_dims=2, in_channels=1, out_channels=512`) that\n",
+ " turns each radiograph into a 512-dimensional embedding.\n",
+ "- **Stream 2 (tabular context):** a small feed-forward projection network that compresses the\n",
+ " 4-dimensional Nakaseke tabular vector into a 16-dimensional embedding.\n",
+ "- **Fusion junction:** the two embeddings are concatenated with `torch.cat` into a single\n",
+ " 528-dimensional (512 + 16) fused representation.\n",
+ "- **Classification head:** a dropout-regularized linear head maps the fused representation to the\n",
+ " final diagnostic logit.\n",
+ "\n",
+ "**Design intuition worth calling out explicitly:**\n",
+ "\n",
+ "- *Why the 512 vs. 16 embedding asymmetry?* An image carries far more raw entropy than a 4-value\n",
+ " vector, so collapsing both streams to the same width would either waste capacity on the tabular side\n",
+ " or force the image into an unnecessarily narrow bottleneck. 16 dimensions is enough for a linear\n",
+ " projection of 4 clinical features to be useful in the fused vector without dominating it. This is a\n",
+ " practical default, not a tuned optimum -- a natural extension is a learned gating or attention layer\n",
+ " that lets the network decide the relative weight of each stream per patient instead of fixing it\n",
+ " through embedding width alone.\n",
+ "- *Why Dropout in the head, specifically?* Real single-site clinical tabular cohorts are usually small\n",
+ " (tens to a few hundred patients), while `DenseNet121` alone carries roughly 7-8 million parameters.\n",
+ " Dropout on the fused representation is a cheap, standard regularizer for exactly this\n",
+ " small-data/large-model regime; for a production deployment on a real cohort this size, pairing it\n",
+ " with weight decay and early stopping on a held-out validation loss is advisable.\n",
+ "- *Why not just concatenate the raw 4 tabular values instead of projecting them?* Because the raw\n",
+ " values live on very different scales (age in years vs. salivary pH in single digits), and\n",
+ " concatenating unprojected features next to a 512-d embedding would let the image stream dominate the\n",
+ " fused gradient by sheer dimensionality. A small learned projection gives the optimizer a\n",
+ " same-order-of-magnitude representation to work with on both sides of `torch.cat`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "class ResilientMultimodalClassifier(nn.Module):\n",
+ " \"\"\"Early-fusion network combining a 2D radiograph stream with a tabular clinical stream.\"\"\"\n",
+ "\n",
+ " def __init__(\n",
+ " self,\n",
+ " tabular_in_features: int = 4,\n",
+ " tabular_embed_dim: int = 16,\n",
+ " image_embed_dim: int = 512,\n",
+ " num_classes: int = 1,\n",
+ " dropout: float = 0.3,\n",
+ " ) -> None:\n",
+ " super().__init__()\n",
+ "\n",
+ " self.image_stream = DenseNet121(\n",
+ " spatial_dims=2,\n",
+ " in_channels=1,\n",
+ " out_channels=image_embed_dim,\n",
+ " )\n",
+ "\n",
+ " self.tabular_stream = nn.Sequential(\n",
+ " nn.Linear(tabular_in_features, 32),\n",
+ " nn.ReLU(),\n",
+ " nn.Linear(32, tabular_embed_dim),\n",
+ " nn.ReLU(),\n",
+ " )\n",
+ "\n",
+ " fused_dim = image_embed_dim + tabular_embed_dim\n",
+ " self.classification_head = nn.Sequential(\n",
+ " nn.Dropout(dropout),\n",
+ " nn.Linear(fused_dim, 64),\n",
+ " nn.ReLU(),\n",
+ " nn.Dropout(dropout),\n",
+ " nn.Linear(64, num_classes),\n",
+ " )\n",
+ "\n",
+ " def forward(self, image: torch.Tensor, tabular: torch.Tensor) -> torch.Tensor:\n",
+ " image_embedding = self.image_stream(image)\n",
+ " tabular_embedding = self.tabular_stream(tabular)\n",
+ " fused = torch.cat([image_embedding, tabular_embedding], dim=1)\n",
+ " return self.classification_head(fused)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Prove the fusion graph with a single mock forward pass"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
+ "model = ResilientMultimodalClassifier().to(device)\n",
+ "\n",
+ "mock_images = torch.randn(4, 1, 64, 64, device=device)\n",
+ "mock_tabular = torch.randn(4, 4, device=device)\n",
+ "mock_logits = model(mock_images, mock_tabular)\n",
+ "\n",
+ "print(\"Mock fused output shape:\", mock_logits.shape)\n",
+ "assert mock_logits.shape == (4, 1), \"Fusion graph produced an unexpected output shape.\"\n",
+ "print(\"Fusion graph OK: image (512-d) + tabular (16-d) -> 528-d fused vector -> 1 logit.\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Train the fusion model\n",
+ "\n",
+ "A short training loop over the synthetic cohort. `max_epochs` and `val_interval` follow the MONAI\n",
+ "tutorial convention so the automated notebook-execution tests can safely reduce them for CI."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "max_epochs = 5\n",
+ "val_interval = 1\n",
+ "learning_rate = 1e-3\n",
+ "\n",
+ "loss_function = nn.BCEWithLogitsLoss()\n",
+ "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
+ "\n",
+ "for epoch in range(max_epochs):\n",
+ " model.train()\n",
+ " epoch_loss = 0.0\n",
+ " for batch in train_loader:\n",
+ " images = batch[\"image\"].to(device)\n",
+ " tabular = batch[\"nakaseke_tabular\"].to(device)\n",
+ " labels = batch[\"label\"].to(device).float().unsqueeze(1)\n",
+ "\n",
+ " optimizer.zero_grad()\n",
+ " logits = model(images, tabular)\n",
+ " loss = loss_function(logits, labels)\n",
+ " loss.backward()\n",
+ " optimizer.step()\n",
+ " epoch_loss += loss.item()\n",
+ "\n",
+ " epoch_loss /= len(train_loader)\n",
+ " print(f\"epoch {epoch + 1}/{max_epochs} average training loss: {epoch_loss:.4f}\")\n",
+ "\n",
+ " if (epoch + 1) % val_interval == 0:\n",
+ " model.eval()\n",
+ " correct, total = 0, 0\n",
+ " with torch.no_grad():\n",
+ " for batch in val_loader:\n",
+ " images = batch[\"image\"].to(device)\n",
+ " tabular = batch[\"nakaseke_tabular\"].to(device)\n",
+ " labels = batch[\"label\"].to(device).float().unsqueeze(1)\n",
+ "\n",
+ " logits = model(images, tabular)\n",
+ " predictions = (torch.sigmoid(logits) > 0.5).float()\n",
+ " correct += (predictions == labels).sum().item()\n",
+ " total += labels.numel()\n",
+ "\n",
+ " print(f\"epoch {epoch + 1}/{max_epochs} validation accuracy: {correct / total:.4f}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Report a threshold-independent clinical metric\n",
+ "\n",
+ "Accuracy depends on the 0.5 decision threshold and is a poor summary for screening tasks with any\n",
+ "class imbalance -- it is easy to look good on accuracy by mostly predicting the majority class. The\n",
+ "ROC-AUC is threshold-independent and is the metric clinical ML work is generally expected to report,\n",
+ "so this notebook computes it once on the full validation set with MONAI's native `ROCAUCMetric`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "auc_metric = ROCAUCMetric()\n",
+ "model.eval()\n",
+ "with torch.no_grad():\n",
+ " for batch in val_loader:\n",
+ " images = batch[\"image\"].to(device)\n",
+ " tabular = batch[\"nakaseke_tabular\"].to(device)\n",
+ " labels = batch[\"label\"].to(device).float().unsqueeze(1)\n",
+ "\n",
+ " probabilities = torch.sigmoid(model(images, tabular))\n",
+ " auc_metric(y_pred=probabilities, y=labels)\n",
+ "\n",
+ "final_val_auc = auc_metric.aggregate()\n",
+ "auc_metric.reset()\n",
+ "print(f\"Final validation ROC-AUC: {final_val_auc:.4f}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Cleanup data directory\n",
+ "\n",
+ "Remove the temporary directory if one was used."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "if directory is None:\n",
+ " shutil.rmtree(root_dir)"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.12"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}