Thanks for the info, very interesting stuff !
I've started to look into DXVA2 myself a bit. Thing is that there's a bug in XBMC/FFMPEG's DXVA2 implementation for WMV9 (it doesn't work). WMP plays WMV9 fine with DXVA2, so I figured, if there's an easy way to hook into the DXVA2 calls, it shouldn't be too hard to compare what WMP does and how FFMPEG tries to decode.
I first started looking at that 'DXVA checker' tool, since that has an option to trace DXVA2 calls, it seemed to be a great start. The guy who wrote it encrypted his exe (probably to keep guys like me out, lol) but since it's an .NET assembly it was a joke to dump the decrypted version from memory
It turns out that he just uses a simple user mode event tracing, supplied by Advapi:
http://www.geoffchappell.com/viewer.htm?.../index.htm
So that wasn't what I had hoped for (had hoped for something in kernel mode !), since I can't use that to dump function parameters etc. Then I found that MPC-HC has some code called 'ipinhook.c' that seemed to be exactly what I was looking for (DXVA2 hooking). However it turned out that it just overwrites its own Vtables. Again not something I could use to hook into WMP (although I guess it should be possible to inject code into WMP and then overwrite the vtables the same way, but that's quite a bit of work).
In short, been busy quite a bit on a route that didn't yield a quick/easy result ... Probably should have used that time instead to study videodecoding
Oh, and btw, if anybody is interested: I found by looking at wmvdecod.dll that if you set :
HKCU\Software\Microsoft\Scrunch\WMVideo
DWORD "DXVATRACE" = 1
then wmvdecod.dll (used by WMP to decode WMV/9) writes some event logging to c:\dxva.log. However, again, not really stuff I could use ...