We investigated the pupil functions of specific-shaped apertures generated by Lambertian illumination with a rectangular light pipe to derive the corresponding optical transfer functions (OTFs) in aberration-free and defocused optical systems. The semianalytical results indicate that the curves of the OTF of the optical system vary with the form of the shaped apertures that are generated by illumination with different geometric structures of light pipes and light sources. It was found that the OTF values of even-peak frequencies decrease when the Lambertian light source decreases. If there are a total of $n\times n$ individual apertures within a pupil, then n near-periodical peaks will appear on the OTF curve. It is evident that the values of the OTF remain almost unchanged even when the lengths of the light pipes are different. Furthermore, the geometric structure of the light pipe does not affect the resolution limit of the optical system, and under the condition of a larger defocused coefficient ${\omega }_{20}$ , the results of the defocused system can coincide with those of the aberration-free systems.