We have developed a set of techniques, referred to as scatterfield microscopy, in which the illumination is engineered in combination with appropriately designed metrology targets to extend the limits of image-based optical metrology. Previously we reported results from samples with sub- 50\text{-nm} -sized features having pitches larger than the conventional Rayleigh resolution criterion, which resulted in images having edge contrast and elements of conventional imaging. In this paper we extend these methods to targets composed of features much denser than the conventional Rayleigh resolution criterion. For these applications, a new approach is presented that uses a combination of zero-order optical response and edge-based imaging. The approach is, however, more general and a more comprehensive set of analyses using theoretical methods is presented. This analysis gives a direct measure of the ultimate size and density of features that can be measured with these optical techniques. We present both experimental results and optical simulations using different electromagnetic scattering packages to evaluate the ultimate sensitivity and extensibility of these techniques.