Evaluating accuracy of striatal, pallidal, and thalamic segmentation methods: Comparing automated approaches to manual delineation

Carolina Makowski, Sophie Béland, Penelope Kostopoulos, Nikhil Bhagwat, Gabriel A. Devenyi, Ashok K. Malla, Ridha Joober, Martin Lepage, M. Mallar Chakravarty

Accurate automated quantification of subcortical structures is a greatly pursued endeavour in neuroimaging. In an effort to establish the validity and reliability of these methods in defining the striatum, globus pallidus, and thalamus, we investigated differences in volumetry between manual delineation and automated segmentations derived by widely used FreeSurfer and FSL packages, and a more recent segmentation method, the MAGeT-Brain algorithm. In a first set of experiments, the basal ganglia and thalamus of thirty subjects (15 first episode psychosis [FEP], 15 controls) were manually defined and compared to the three automated methods. Our results suggest that all methods overestimate volumes compared to the manually derived "gold standard", with the least pronounced differences produced using MAGeT. The least between-method variability was noted for the striatum, whereas marked differences between manual segmentation and MAGeT compared to FreeSurfer and FSL emerged for the globus pallidus and thalamus. Correlations between manual segmentation and automated methods were strongest for MAGeT (range: 0.51 to 0.92; p<0.01, corrected), whereas FreeSurfer and FSL showed moderate to strong Pearson correlations (range 0.44-0.86; p<0.05, corrected), with the exception of FreeSurfer pallidal (r=0.31, p=0.10) and FSL thalamic segmentations (r=0.37, p=0.051). Bland-Altman plots highlighted a tendency for greater volumetric differences between manual labels and automated methods at the lower end of the distribution (i.e. smaller structures), which was most prominent for bilateral thalamus across automated pipelines, and left globus pallidus for FSL. We then went on to examine volume and shape of the basal ganglia structures using automated techniques in 135 FEP patients and 88 controls. The striatum and globus pallidus were significantly larger in FEP patients compared to controls bilaterally, irrespective of the method used. MAGeT-Brain was more sensitive to shape-based group differences, and uncovered widespread surface expansions in the striatum and globus pallidus bilaterally in FEP patients compared to controls, and surface contractions in bilateral thalamus (FDR-corrected). By contrast, after using a recommended cluster-wise thresholding method, FSL only detected differences in the right ventral striatum (FEP>Control) and one cluster of the left thalamus (Control>FEP). These results suggest that different automated pipelines segment subcortical structures with varying degrees of variability compared to manual methods, with particularly pronounced differences found with FreeSurfer and FSL for the globus pallidus and thalamus.