I'm having trouble splitting up quotes, so I'll just paste them into regular quote blocks.
In contrast, when I sweep an individual channel, apply a simple eq and sweep again, the results are close but not quite right. Lots of eq points and lots of channels compounds the errors significantly. Obviously you can't do things like boost away a null, but I'm not trying to do that. I would expect simple eqs of a single channel in "minimum phase" regions to behave predictably, just as changing the master volume on the avr produces a nearly identical sweep at a different level.
That's really weird. So just applying a single EQ to a single channel doesn't give the expected measured response? I don't know what could be causing that.
Edit: Can you try this with just a PEQ and no all-pass filter in the path? I'll go back and re-check the biquad coefficients for the all-pass filters using Matlab.
Internally, MSO computes the filter responses as analog filters. When it computes the biquad coefficients, it uses the bilinear transformation to get the z-domain transfer function. There is a nonlinear scale factor that "predistorts" e.g. the center frequency of the PEQ such that after the bilinear transformation the z-domain transfer function has the exact same center frequency as the analog one. There is some "bandwidth warping" going on due to the nonlinearity of the frequency transformation, but when the critical frequencies of the filter are so much less than the Nyquist frequency of the DSP, there is negligible error, on the order of a couple hundredths of a dB. In order to ensure the integrity of the calculated biquad coefficients, I compared my results with the spreadsheet by Charlie Laub at the miniDSP site.
When I was checking out the biquad coefficient code, the results agreed with Charlie's spreadsheet down to 14 digits for all filter types except the PEQ. To chase the difference down further, I wrote a Matlab script and a set of functions to compute the response of the analog PEQ, the coefficients of the digital PEQ, and the response of the digital PEQ. Center frequency and boost/cut at the center frequency were dead on, as expected by the "pre-distortion" of the analog filter center frequency. Maximum error between the analog filter response and the computed digital filter response was a couple hundredths of a dB over the frequency band to 200 Hz. (Edit: I just went back and reinstalled Matlab and ran that script again. Maximum error between analog and digital filter is 1.4 * 10^(-4) dB.)
I'm not saying Charlie's spreadsheet for the PEQ is wrong though. There is potential ambiguity in the definition of "Q" for the analog PEQ. I use the one that Bruno Putzeys uses in his Hypex sub amps. For those, the "Q" is assigned to the analog filter "pole Q" when there is a boost, and to the analog filter "zero Q" when there is a cut. This gives the result that two PEQs with the same Q and center frequency, one with a boost of X dB and the other with a cut of X dB will exactly cancel each other out when combined. My guess, although I have not proven it, is that the discrepancy between my results and Charlie's for the PEQ may be due to a possibly different definition of "Q" being used for the analog PEQ filter. At any rate, I got excellent agreement between the analog and digital filter responses, so I am satisfied the calculations are good for the PEQ.
Finding no. 1:
I discovered last night that THX "Boundary Gain Compensation" was enabled on my receiver. It's hidden in its own menu folder far away from things like audyssey and bass management. Grr. I thought this was a simple shelving filter, but before and after sweeps reveal some really strange effects on different sub channels. I turned that off and took yet more before sweeps, so I'm hopeful I'll see improvements when I'm able to next take some measurements. This means I'll probably want to remeasure and adjust the whole signal chain again as well.
Let's hope that was the cause of the problem. Is there any dynamic EQ sneaking into the AVR?
I'm growing skeptical of the "import REW biquads" function in my minidsp with 2x4 advanced plugin. First, it will accept 6 biquads for both the input and output channels, but there are only 5 peq slots. I'm guessing the 5th is dropped, but it's very hard to tell from the interface.
That should give an error message. I think the 4-way advanced plugin can have 6 PEQs, but the 2x4 advanced is limited to 5. When you tell MSO how many biquads your DSP has per channel, and you use more than that, it will still give you all the coefficients in order to not lose information. I believe it prints out a warning in the filter report when that happens. I'll check that, and if it doesn't, I'll fix it so it does. I should probably display a warning message when this happens when exporting biquad text as a file too. The purpose of specifying the maximum number of biquads is that if you use less, it will pad them with "through connection" biquads up to the maximum number in the exported text and the filter report.
Secondly, while biquad computations seem to match fairly well from the minidsp display graph to the MSO software, they aren't fully one to one. The frequencies of adjustments seem to line up, but the magnitude may be slightly different, especially on large adjustments, e.g. a deep cut that equates to -15 dB in MSO might register as roughly -12 dB in the minidsp graph. I'm thinking the minidsp plugin graph is pretty small and might have some display/rounding issues, but it has me wondering.
It could be that they're using a small number of frequencies and lots of smoothing so that the peak or dip is missed and smoothed over. That's just a guess on my part. With regard to MSO PEQ biquad computations, the center frequency and gain/attenuation of a single PEQ digital filter at the center frequency ought to be dead nuts on.
Besides the disappointing and confusing results so far with before/after sweeps, I think the software is fantastic. If I give it a few peqs and shelf/LT/all-pass filters to work with, it's amazing what it can do. With a first-order all-pass to play with on each sub channel, it's able to align them such that the low-end extends down to a 8-9 hz -3dB point with almost no boost on any channel, and there is more headroom across the entire bandwidth. Setting things the old-fashioned way, my -3 point has been in the tweens with a 12 dB LT.
I'm glad to hear that at least, but sorry to hear that you're having these other problems.