We propose a quantum switch for telecom-band applications that is composed of a ${\chi }^{(2)}$ microdisk coupled to two fibers (or waveguides). The idea is to apply a pump pulse to shift the microdisk out of resonance, thereby switching the device between the cross and bar states in an interaction-free manner. As an example, a 2.5-μm-thick, $10\mathrm{\mu m}$ radius GaAs microdisk with an intrinsic Q of $\sim {10}^8$ and a fiber-cavity-coupling Q of $\sim {10}^4$ can achieve low-loss ( $\lesssim 1\%$ ) switching for gigahertz-rate O-band quantum signals with milliwatt-peak-power pumps in the C band.