AFTer-SAM: Adapting SAM with Axial Fusion Transformer for Medical Imaging Segmentation

The Segmentation Anything Model (SAM) has demon- strated effectiveness in various segmentation tasks. How- ever, its application to 3D medical data has posed chal- lenges due to its inherent design for both 2D and natural im- ages. While there have been attempts to apply SAM to medi- cal images on a slice-by-slice basis, the outcomes have been less than optimal. In this study, we introduce AFTer-SAM, an adaptation of SAM designed for volumetric medical im- age segmentation. By incorporating an Axial Fusion Trans- former, AFTer-SAM is capable of capturing both intra-slice details and inter-slice contextual information, essential for accurate medical image segmentation. Given the potential computational challenges of training this enhanced model, we utilize Low-Rank Adaptation (LoRA) to efficiently fine- tune the weights of the Axial Fusion Transformer. This en- sures a streamlined training process without compromising on performance. Our results indicate that AFTer-SAM of- fers significant improvements in volumetric medical image segmentation, suggesting a promising direction for the ap- plication of large pre-trained models in medical imaging.