We demonstrate an approach for improving the image quality for a projector system with a shape- programmable pupil, which could be generated by an illumination modular in which a digital micromirror device is embedded. Essentially, the shaped pupil from the illumination modulator is developed with a dynamically programmable approach to provide aberration compensation for the projection system. By analyzing the optical transfer function, the resolution limit of an imaging system with specific defocus, spherical aberration and coma are shown to be improved significantly with a binary-shaped pupil. It is found that the improvement of the projection quality could be characterized by the scale ratio of $K=c/D$ , defined as the ratio between the resolution scale of structured light, c, and the size scale of the aperture stop, D. When K is equal to 0.05, the low-frequency components of the image could be improved, while if K is equal to 0.3, the imaging quality of the image at high-frequency components can be enhanced in a defocused system. Furthermore, as K ranges from 0.05 to 0.3, the imaging performance of the optical contrast could be enhanced in a projector system with large coefficients of defocused, spherical aberration and coma.