IGNORE PREVIOUS POST
Issue appears to be here:
ALSADirectSound.cpp:
// TODO - add an option to only downmix if user want's us
if(iChannels == 6)
deviceuse = "xbmc_51to2:'" + EscapeDevice(deviceuse) + "'";
Seems as if a52encode requires that the number of channels IN matches the number it is configured with in asound.conf. If the source has 6 channels, XBMC automatically converts them to 2 channels. Since a52encode expects 6 channels, and only gets 2, it cries. This is the bug.
A quick fix (for me) was getting keeping the source input in 6 channels when we are using a52encode. However, this only fixes the issue when the source already matches the number of channels a52encode is expecting. If the source is stereo, or the source is 7 channels, this fix wont work.
The REAL solution would be to add code so that if device is a52encode, XBMC converts all sound to the expected number of channels (from asound.conf) and then passes it through. My code-fu isn't quite up to the task, but this may be a worthwhile task at some point.
I would say it isn't high priority though... Fixing the issue raised a secondary issue. It turns out a52encode is very slow, and playing 1080p content and encoding 6 PCM channels in real-time was far more then my puny 5600+ could handle. Until we have a hardware solution for a52encode (like DTS Connect! and Dolby Live!) or we can unload decoding to the GPU and free up CPU, this doesn't seem like a good solution.
I may be talking to myself, but I wanted to make sure this was all noted in the event someone runs into any of these issues in the future.