Domain-Specific Self-Supervised and Multimodal Learning for Brain Tumor Segmentation from MRI and Radiology Reports
Baydemir, Berat (2026)
2026
Master's Programme in Computing Sciences and Electrical Engineering
Informaatioteknologian ja viestinnän tiedekunta - Faculty of Information Technology and Communication Sciences
This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Hyväksymispäivämäärä
2026-05-12
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:tuni-202605125460
https://urn.fi/URN:NBN:fi:tuni-202605125460
Tiivistelmä
Brain tumor segmentation from magnetic resonance imaging (MRI) is an important task in medical image analysis because accurate delineation of tumor regions can support diagnosis, treatment planning, and disease monitoring. Recent multimodal approaches have also explored the use of radiology reports together with imaging data, but the effectiveness of such methods depends strongly on visual representation learning, model initialization, and fusion design.
This thesis investigates a three-stage framework for brain tumor segmentation using the TextBraTS dataset. In the first stage, domain-specific self-supervised pretraining was performed on a leakage-aware deduplicated subset of BraTS2021 in order to learn visual representations from unlabeled multi-modal brain MRI volumes. Two self-supervised strategies were studied: masked reconstruction alone, and masked reconstruction combined with contrastive learning. In the second stage, the learned representations were transferred to a vision-only SwinUNETR segmentation model trained on TextBraTS. In the third stage, multimodal segmentation models were developed by incorporating radiology-report-derived text features into the segmentation framework. For multimodal fusion, both the original bidirectional cross-attention design from TextBraTS and a simpler Feature-wise Linear Modulation (FiLM)-based conditioning strategy were evaluated.
The experiments show that domain-specific self-supervised pretraining improves downstream vision-only segmentation performance, and that reconstruction-based pretraining was more effective than the combined reconstruction and contrastive objective in the final experimental setting. The strongest vision-only model, initialized from reconstruction-only self-supervised pretraining, achieved an average Dice score of 85.0 and an average HD95 of 3.38 on the TextBraTS test set. In the multimodal stage, FiLM-based conditioning outperformed the bidirectional cross-attention formulation under the controlled training pipeline used in this thesis. The best multimodal model was obtained with FiLM-based fusion initialized from all compatible self-supervised MRI-pretrained weights, reaching an average Dice score of 85.2 and an average HD95 of 3.56 on the test set.
The results indicate that effective brain tumor segmentation in this setting benefits from domain-specific MRI pretraining and from a staged training strategy in which self-supervised learning, supervised vision-only adaptation, and multimodal refinement are treated as complementary steps. The findings further suggest that lightweight text conditioning can be more effective than heavier cross-attention-based fusion for report-guided brain tumor segmentation when strong visual initialization is available.
This thesis investigates a three-stage framework for brain tumor segmentation using the TextBraTS dataset. In the first stage, domain-specific self-supervised pretraining was performed on a leakage-aware deduplicated subset of BraTS2021 in order to learn visual representations from unlabeled multi-modal brain MRI volumes. Two self-supervised strategies were studied: masked reconstruction alone, and masked reconstruction combined with contrastive learning. In the second stage, the learned representations were transferred to a vision-only SwinUNETR segmentation model trained on TextBraTS. In the third stage, multimodal segmentation models were developed by incorporating radiology-report-derived text features into the segmentation framework. For multimodal fusion, both the original bidirectional cross-attention design from TextBraTS and a simpler Feature-wise Linear Modulation (FiLM)-based conditioning strategy were evaluated.
The experiments show that domain-specific self-supervised pretraining improves downstream vision-only segmentation performance, and that reconstruction-based pretraining was more effective than the combined reconstruction and contrastive objective in the final experimental setting. The strongest vision-only model, initialized from reconstruction-only self-supervised pretraining, achieved an average Dice score of 85.0 and an average HD95 of 3.38 on the TextBraTS test set. In the multimodal stage, FiLM-based conditioning outperformed the bidirectional cross-attention formulation under the controlled training pipeline used in this thesis. The best multimodal model was obtained with FiLM-based fusion initialized from all compatible self-supervised MRI-pretrained weights, reaching an average Dice score of 85.2 and an average HD95 of 3.56 on the test set.
The results indicate that effective brain tumor segmentation in this setting benefits from domain-specific MRI pretraining and from a staged training strategy in which self-supervised learning, supervised vision-only adaptation, and multimodal refinement are treated as complementary steps. The findings further suggest that lightweight text conditioning can be more effective than heavier cross-attention-based fusion for report-guided brain tumor segmentation when strong visual initialization is available.