Multi-modal Deep Learning

About

This article investigates deep learning methodologies for single-modality clinical data analysis, as a crucial precursor to multi-modal medical research. Building on Guo JingYuan's work, the study refines clinical data processing through Compact Convolutional Transformer (CCT), Patch Up, and the innovative CamCenterLoss technique, establishing a foundation for future multimodal investigations. The proposed methodology demonstrates improved prediction accuracy and at tentiveness to critically ill patients compared to Guo JingYuan's ResNet and StageNet approaches. Novelty that using image-pretrained vision transformer backbone to perform transfer learning time-series clinical data.The study highlights the potential of CCT, Patch Up, and novel CamCenterLoss in processing single modality clinical data within deep learning frameworks, paving the way for future multimodal medical research and promoting precision and personalized healthcare

Chen Yuhua · 2024

Related benchmarks

Task	Dataset	Result
Multimodal Classification	MST Missing Modalities	Accuracy99.96	28
Multimodal Classification	PolyMNIST Missing Rate η=0.6	Accuracy97.5	16
Multimodal Classification	PolyMNIST Missing Rate η=0.8	Accuracy89.86	16
Multimodal Classification	PolyMNIST Missing Rate η=0	Accuracy99.94	16
Multimodal Classification	MST Missing Modalities {S,T}	Accuracy0.9833	14
Multimodal Classification	CelebA Missing Modalities {T}	Accuracy89.71	14
Multimodal Classification	CelebA Missing Modalities	Accuracy89.98	14
Multimodal Classification	CelebA Missing Modalities {I}	Accuracy72.04	14