The effect of surface roughness on the ellipsometric response of semiconductor surfaces is investigated. $\mathrm{CdTe}\left(211\right)\mathrm{B}$ layers were grown on $\mathrm{Ge}\left(211\right)$ by molecular beam epitaxy using less than optimal growth conditions to enhance the formation of surface roughness. Their optical properties, measured by rotating-compensator ellipsometry, showed small but significant sample-to-sample differences not explainable in terms of nanometer-scale roughness. A critical-point analysis established that the critical-point structure of the dielectric function was the same for all samples. This result suggested that the observed sample-to-sample variations were due to macroscopic roughness, which scatters off-specular light into the detector, thereby causing errors. We introduced tentative corrections for off-specular reflection that fitted the observed differences and thus supported the idea that off-specular reflection was responsible for the observed differences. These results were obtained using $\mathrm{CdTe}$ but are easily extensible to other rough opaque materials.