I've been testing the new mplayer dll and found definite improvements in h264 decoding speed/playback, with so far no new bugs encountered (both from a self compiled version on 2006.12.27 and the dll downloaded from the start of this thread)...
I am not the most knowledgeable when it comes to tweaking codec settings for encoding - I have been using Staxrip for encoding DVD rips to H264, using Sharktooth's Xbox Hi-res profile (no CABAC or deblocking) to avoid dropped frames.
I have now found that I am able to encode using the Xbox Lo-res profile (Sharktooth's once again), which enables CABAC and deblocking, though I have to disable deblocking. The lo-res profile also alters other settings, such as using b-frame pyramid and weighted prediction, that aren't used in the hi-res profile.
I have converted some PAL DVDs (16:9 tv series) to H264 at 720x400 res, using AC3 2.0, aac and mp3 audio in different test files, at approx 1250 kbps avg (video) bitrate. All have played without any dropped frames so far, from HDD and via SMB or ccx over a LAN (AC3 passed to an amp for decoding though).
I'm currently encoding another PAL DVD of a 4:3 tv series to H264 @ 704x528 res - approx 2000kbps video bitrate, with mp3 sound using the same settings... fingers crossed it plays fine too...
Hope this helps any other relative noobs out there who want decent quality encodes but still use an all-in-one type gui like Staxrip - Sharktooth's profiles can also be imported into megui.
A huge thankyou to elupus and any other developers who have worked on porting the mplayer updates to xbmc... you just keep on making an already great app even better