[ringgh0st]

I don's see any weird stuff going on in the logs. As you said before playing 60fps on 60 hz is fine and playing 24fps on 24 hz is also fine.

Therefore I suspect the TV of causing the "telecine judder". To get the answer what is causing it we can do a couple of things.

Perquisites:
1: Set the desktop to 60p
2: find some samples of 720p or 1080p 24p files. For instance trailers from the apple site. these are alle L3.0 AVC files I think so not that heavy on the Mini. Also they are official AVC file so they will work with VDA.
3: VLC for mac
4: play these files from the local HD instead of from SMB
5: post the brand and model of the TV

Options
1: Instead of XBMC use VLC to play a clip in fullscreen. Please note that VLC does not use VDA so use a 720p clip of 1080p clips running at L3.0 AVC
2: If you have the option to connect the Mac Mini to a different Plasma or LCD tv. Parents, Family etc.
3: Connect a different computer (not a Mac Mini) to the plasma, and see if playing the same files also cause judder.

With these options I suspect the TV to be the cause of the judder

Possible solution
1: Enable AV-sync in XBMC and try all three different sync-methodes.
2: Disable AV sync in XBMC and enable Intelligent Frame Creation to see if it makes a difference.
3: Enable AV-sync AND enable Intelligent Frame Creation on the TV

Good luck !

good idea with VLC! Will do. thanx for your input