Imaging polarimeters are often designed and optimized by assuming that the polarization properties of the optics are nearly ideal. For example, we often assume that the linear polarizers have infinite extinction ratios. It is also usually assumed that the retarding elements have retardances that do not vary either spatially or with the angle of incidence. We consider the case where the polarization optics used to develop an imaging polarimeter are imperfect. Specifically, we examine the expected performance of a system as the extinction ratio of the diattenuators degrades, as the retardance varies spatially, and as the retardance varies with incidence angle. It is found that the penalty in the signal-to-noise ratio for using diattenuators with low extinction ratios is not severe, as an extinction ratio of 5 causes only a 2.0\text{\hspace{0.17em} dB} increase in the noise in the reconstructed Stokes parameter images compared with an ideal diattenuator. Likewise, we find that a system can be optimized in the presence of spatially varying retardance, but that angular positioning error is far more important in rotating retarder imaging polarimeters.