I'm having trouble splitting up quotes, so I'll just paste them into regular quote blocks.
That's really weird. So just applying a single EQ to a single channel doesn't give the expected measured response? I don't know what could be causing that.
Edit: Can you try this with just a PEQ and no all-pass filter in the path? I'll go back and re-check the biquad coefficients for the all-pass filters using Matlab.
Internally, MSO computes the filter responses as analog filters. When it computes the biquad coefficients, it uses the bilinear transformation to get the z-domain transfer function. There is a nonlinear scale factor that "predistorts" e.g. the center frequency of the PEQ such that after the bilinear transformation the z-domain transfer function has the exact same center frequency as the analog one. There is some "bandwidth warping" going on due to the nonlinearity of the frequency transformation, but when the critical frequencies of the filter are so much less than the Nyquist frequency of the DSP, there is negligible error, on the order of a couple hundredths of a dB. In order to ensure the integrity of the calculated biquad coefficients, I compared my results with the spreadsheet by Charlie Laub at the miniDSP site.
When I was checking out the biquad coefficient code, the results agreed with Charlie's spreadsheet down to 14 digits for all filter types except the PEQ. To chase the difference down further, I wrote a Matlab script and a set of functions to compute the response of the analog PEQ, the coefficients of the digital PEQ, and the response of the digital PEQ. Center frequency and boost/cut at the center frequency were dead on, as expected by the "pre-distortion" of the analog filter center frequency. Maximum error between the analog filter response and the computed digital filter response was a couple hundredths of a dB over the frequency band to 200 Hz. (Edit: I just went back and reinstalled Matlab and ran that script again. Maximum error between analog and digital filter is 1.4 * 10^(-4) dB.)
I'm not saying Charlie's spreadsheet for the PEQ is wrong though. There is potential ambiguity in the definition of "Q" for the analog PEQ. I use the one that Bruno Putzeys uses in his Hypex sub amps. For those, the "Q" is assigned to the analog filter "pole Q" when there is a boost, and to the analog filter "zero Q" when there is a cut. This gives the result that two PEQs with the same Q and center frequency, one with a boost of X dB and the other with a cut of X dB will exactly cancel each other out when combined. My guess, although I have not proven it, is that the discrepancy between my results and Charlie's for the PEQ may be due to a possibly different definition of "Q" being used for the analog PEQ filter. At any rate, I got excellent agreement between the analog and digital filter responses, so I am satisfied the calculations are good for the PEQ.
Let's hope that was the cause of the problem. Is there any dynamic EQ sneaking into the AVR?
That should give an error message. I think the 4-way advanced plugin can have 6 PEQs, but the 2x4 advanced is limited to 5. When you tell MSO how many biquads your DSP has per channel, and you use more than that, it will still give you all the coefficients in order to not lose information. I believe it prints out a warning in the filter report when that happens. I'll check that, and if it doesn't, I'll fix it so it does. I should probably display a warning message when this happens when exporting biquad text as a file too. The purpose of specifying the maximum number of biquads is that if you use less, it will pad them with "through connection" biquads up to the maximum number in the exported text and the filter report.
It could be that they're using a small number of frequencies and lots of smoothing so that the peak or dip is missed and smoothed over. That's just a guess on my part. With regard to MSO PEQ biquad computations, the center frequency and gain/attenuation of a single PEQ digital filter at the center frequency ought to be dead nuts on.
I'm glad to hear that at least, but sorry to hear that you're having these other problems.