There is a page on the wiki containing info on supported formats and codecs:
http://wiki.xbmc.org/index.php?title=XBM...ats/Codecs
However there is a note at the top of that page which states that it is "incomplete". I'm not sure if that is with regard to the list supported formats / codecs - it probably is though as it was last updated in 2008 (edit: re-checked this and I was wrong; last update to supported formats/codecs was actually Jan 2011 - to add BluRay support to the list. my bad). In any case, as stated on
this page of the wiki XBMC uses MPlayer as its core player for video/audio so you can probably just check the
MPlayer support page for supported codecs for more up-to-date info on this. Personally, I've never come across a video file that XBMC can't play (any failings in this area have generally been due to hardware limitations rather than codec support)
The supported codecs/formats of ATV Flash are listed (in minimal detail) at the bottom of the
ATV Flash homepage. If you need more detailed information from ATV Flash I'd suggest that you ask on the
Firecore forums. I can't say that I know either way, but I'd be surprised if ATV Flash supports codecs which XBMC doesn't. Any differences in playback performance are most likely to be due to a combination of the properties of the source file (e.g. bitrate, or method used to encode the file), and resources necessary to play back the file. I can't claim to know much about ATV Flash, but my guess is that it may play back some files more smoothly than XBMC as it is less feature-rich and so ties up less resources elsewhere. Having said that, I'd imagine that any such differences are minimal.