Analysis and optimization of diffraction effects in nanolithography through multilayered media with a fast and accurate field-theoretical approach is presented. The scattered field through an arbitrary two-dimensional (2D) mask pattern in multilayered media illuminated by a TM-polarized incident wave is determined by using an electric field integral equation formulation. In this formulation the electric field is represented in terms of complex images Green’s functions. The method of moments is then employed to solve the resulting integral equation. In this way an accurate and computationally efficient approximate method is achieved. The accuracy of the proposed method is vindicated through comparison with direct numerical integration results. Moreover, the comparison is made between the results obtained by the proposed method and those obtained by the full-wave finite-element method. The ray tracing method is combined with the proposed method to describe the imaging process in the lithography. The simulated annealing algorithm is then employed to solve the inverse problem, i.e., to design an optimized mask pattern to improve the resolution. Two binary mask patterns under normal incident coherent illumination are designed by this method, where it is shown that the subresolution features improve the critical dimension significantly.