Effect of "mixer bounce" or "sampler bounce" on high dynamic range measurements.

Nearly all VNAs suffer to some degree from a measurement problem known as "mixer bounce" or "sampler bounce". This manifests itself as a false response in the measurement of a very good filter, making the filter's stopband rejection appear worse than it really is. The problem is caused by leakage of the VNA's LO signal and its harmonics out of the test ports, and IF frequency modulation sidebands on that LO leakage.

If you set a VNA to a CW frequency and an S12 measurement, you can connect a spectrum analyzer to port 1 and find the LO leakage. If you connect a coupler between ports 1 and 2 so that there is a large signal going into port 1 and you can observe the port 1 LO leakage through the coupled arm, you may be able to see that the LO leakage now has sidebands on it at the VNA's IF frequency. Better VNAs have high LO to RF isolation in their receivers to minimize this leakage.

So why is this modulated LO leakage a problem? Suppose you are measuring a highpass or bandpass filter with excellent stopband rejection (there is seldom a problem with lowpass filters). While the VNA is measuring S21 at the stopband frequencies, there is a large RF signal going into port 1, assuming the filter is reflective. Harmonics of the port 1 LO leakage land in the filter's passband, and are passed through to port 2. The IF sidebands on these LO harmonics are down converted in the port 2 receiver channel, and cause an IF response, which appears on the screen.

What can be done about this?
First, look for an operating mode which "turns off" the unused receiver channels. If the port 1 receiver is not working, it will not leak the modulated LO signal out of the test port. In some HP network analyzers, this mode is available and is called "alternate A and B" (A and B refer to the receiver channels for ports 1 and 2 respectively).
Second, if your VNA has direct receiver access jumpers, you can remove these and place pads in front of the R channel and A channel receivers, probably 12 to 20 dB is a safe bet. These will both reduce the RF signal going into these receiver channels, and reduce the LO leakage coming out of them. You will need to re-calibrate the VNA with the pads in place.
Finally, try reducing the VNA power level a little bit below maximum. Although you usually want to use maximum power for best dynamic range, you may find for some VNAs and some DUTs, that reducing the power a few dB actually gives you better dynamic range by decreasing problems like mixer bounce.