Skip to content

kyegomez/Med-PaLM

Repository files navigation

Multi-Modality

Med-PaLM

A responsible path to generative AI in healthcare: Unleash the power of Med-PaLM 2 to revolutionize medical knowledge, answer complex questions, and enhance healthcare experiences with accuracy, safety, and equitable practices.

GitHub issues GitHub forks GitHub stars GitHub license Share on Twitter Share on Facebook Share on LinkedIn Discord Share on Reddit Share on Hacker News Share on Pinterest Share on WhatsApp

Med palm

Getting Started

pip install MedPalm

Usage

import torch
from medpalm.model import MedPalm

#usage
img = torch.randn(1, 3, 256, 256)
text = torch.randint(0, 20000, (1, 4096))

model = MedPalm()
output = model(img, text)
print(output.shape)

📝 Note: Modify the examples to suit your data and project needs.

📚 Datasets

Here is a comprehensive markdown table containing metadata and details for all the datasets mentioned in the MED PALM paper

Dataset Modality Description # Training examples # Test examples Tasks
MultiMedBench Multimodal Benchmark for biomedical AI 14 biomedical tasks
MedQA Text US licensing exam questions 10,178 1,273 Question answering
MedMCQA Text Indian medical exam questions 182,822 4,183 Question answering
PubMedQA Text Biomedical literature questions 0 500 Question answering
MIMIC-III Radiology reports Radiology reports for ICU patients 58,405 reports 13,057 reports Report summarization
VQA-RAD Radiology images QA pairs on radiology images 1,797 QA pairs 451 QA pairs Visual question answering
Slake-VQA Radiology images English-Chinese QA pairs 9,849 samples 2,070 samples Visual question answering
Path-VQA Pathology images QA pairs on pathology images 19,755 QA pairs 6,761 QA pairs Visual question answering
MIMIC-CXR Chest X-ray Images and reports 353,542 4,834 Report generation, classification
PAD-UFES-20 Dermatology images Skin lesion smartphone images 1,838 images 460 images Image classification
CBIS-DDSM (mass) Mammography Mammogram mass patches 1,318 images 378 images Image classification
CBIS-DDSM (calcification) Mammography Mammogram calcification patches 1,544 images 326 images Image classification
VinDr-Mammo Mammography Mammogram studies 16,000 images 4,000 images Image classification
PrecisionFDA (training) Genomics Genomic variant images 197,038 images Image classification
PrecisionFDA (evaluation) Genomics Genomic variant images 13,030 images Image classification
Montgomery County Chest X-ray Chest X-rays 0 138 images TB detection evaluation
MIMIC-CXR (human evaluation) Chest X-ray Chest X-ray images and reports 246 cases Human evaluation

💼 Commercial Use-Cases

Med Palm has thousands of potential use cases the 3 below are simple, for more detailed applications check out my new blog article on MedPalm's use in the real world. Click here to learn more

  • Clinical Diagnostics: Combining medical imaging, patient tales 📖, and genes, we're aiming for top-notch diagnostic solutions.

  • Healthcare Research: Dive deep into diverse datasets and discover something new with Med-PaLM by your side! 🤿

  • Telemedicine: Quick, reliable, and remote! 🌍 Med-PaLM's here to revolutionize telehealth.

Contributing to Med Palm 🤖🌟

Help with the todo list!


License

Med-PaLM's is under the MIT license. Check out the details here.

Citation

@misc{2307.14334,
Author = {Tao Tu and Shekoofeh Azizi and Danny Driess and Mike Schaekermann and Mohamed Amin and Pi-Chuan Chang and Andrew Carroll and Chuck Lau and Ryutaro Tanno and Ira Ktena and Basil Mustafa and Aakanksha Chowdhery and Yun Liu and Simon Kornblith and David Fleet and Philip Mansfield and Sushant Prakash and Renee Wong and Sunny Virmani and Christopher Semturs and S Sara Mahdavi and Bradley Green and Ewa Dominowska and Blaise Aguera y Arcas and Joelle Barral and Dale Webster and Greg S. Corrado and Yossi Matias and Karan Singhal and Pete Florence and Alan Karthikesalingam and Vivek Natarajan},
Title = {Towards Generalist Biomedical AI},
Year = {2023},
Eprint = {arXiv:2307.14334},
}

Todo

  • Implement a training script using Huggingface's accelerate and deepspeed