We report on the significant effect that intermodal dispersion can have on spatially and spectrally resolved interferometric (${\mathrm{S}}^2$) fiber mode analysis. This dispersion can significantly broaden the measured intermodal group delay and could be misinterpreted as distributed scattering. In our new approach, the spectral interference data is analyzed over multiple wavelength windows staggered across the measurement bandwidth and assembled together to form a spectrogram that reveals the wavelength dependence of the intermodal group delay. Measurements on standard telecom single-mode and large-mode-area fibers are presented. This spectrogram analysis is a more accurate map of mode conversion along the fiber and is essential for evaluating fibers and fiber devices.