A volume holographic correlator is capable of inner product calculations between the input image and multiple stored images in parallel. The inner product that is the center value of the correlation can provide a scalar measure of change between two images. The inner product values are directly acquired by measuring the intensities of the correlation peaks on the CCD. However, the measured intensities are not exactly equal to the theoretical inner product values due to the redundancy correlation. The structure of the correlation peak for randomly interleaved images is analyzed. It can be regarded as two volumes, one pyramid and one prism. The relative inner product value is only determined by the height of the pyramid. The prism, caused by the redundancy correlation, appears as the background noise, which is the main source of the inner product calculation error. A calibration method is proposed to remove the prism from the measured intensity. Based on the geometric structure of the correlation peak, the theoretical expression of the inner product value for the pyramid is derived. A white image is employed as the input image and the measured correlation peak intensity is used to calibrate the inner product value. The calibration method can effectively eliminate the error caused by the redundancy correlation to achieve a high output accuracy of the volume holographic correlator. Experiments are demonstrated for the validity of the method.