XBMC Benchmark Suite as internal feature or addon script for testing video playback?

I was wondering if anyone had recommendations for this. I think it would be useful if there was an XBMC independent set of benchmarks one could run on a set of hardware to get a concrete idea of its capability as it relates to functionality provided by XBMC. I realize the most appropriate test is running XBMC, so perhaps there's a way to use XBMC in a mode where it's a benchmark itself. Maybe if we had some standard, open or CC media that we could provide then we could have an exhaustive benchmark mode that would spit out some text file with scores.

Any thoughts?

I think that you could maybe extend XBMC's DVDPlayer use its FFmpeg to output benchmark results, then all you need is a set of standard video files that everyone can use, (some with video and audio, and some with only video).
http://wiki.xbmc.org/?title=DVDPlayer
Go to www.ffmpeg.org to see what kinds of options that FFmpeg already offers.

The interface itself for a such benchmarking suit under XBMC would best be written as a python script IMO, that script would then 'simply' playback those set of standard video files with an additional parameter given to XBMC's DVDPlayer in order for it to output the result which the python script could both save to a text-file as well as print it to the screen.
http://wiki.xbmc.org/?title=Building_Python_Scripts
http://wiki.xbmc.org/?title=HOW-TO:_ALEX...n_Tutorial
http://wiki.xbmc.org/?title=Category:Python

PS! It is really only video (and video with audio) decoding that you need to test as that is what is the most CPU intensive for XBMC and thus the main bottleneck, but if you wanted to add a bonus feature that checks which GPU you have exacly and what OpenGL extensions it supports then from a list the user a recommendation on whether or not you should to buy a new graphic-adapter or not. Supper would also be if it could have an option to automaticly upload the result to a common website on the internet which put all results into a database for analysing trends.

It might also be useful to benchmark the audio visualizations. Those can suck up a lot of CPU/GPU, especially at higher resolutions. It wouldn't be difficult to rig up a call to the visualization to have it return the FPS.

You could use the new open movie "Big Buck Bunny" as a nice starting point. They provide their movie in standard and high definition (720 & 1080) encoded using four different codecs.

Check it out at: http://www.bigbuckbunny.org/index.php/download/

Actually I thought of that and it's a great idea. I also found some other 1080p video licensed under the creative commons as well. I'm personally not trying to do the benchmarking in XBMC, but thought a thread to discuss the topic would be good. I guess I will at some point if no one else does, but right now I'm working on something else.

This seems to be exacly what you are looking for to use as a base, (you should now only have to figure out how to use execte this from within XBMC via python and then save/display the result):

http://www.phoronix-test-suite.com

Quote:Phoronix Test Suite

The Phoronix Test Suite is the most comprehensive testing and benchmarking platform available for Linux and is designed to carry out qualitative and quantitative benchmarks in a clean, reproducible, and easy-to-use manner. This software is based upon the internal tools and extensive Linux benchmarking work done by Phoronix since 2004, with input from tier-one computer hardware vendors. This software is open-source and licensed under the GNU GPLv3 and consists of a lightweight core (pts-core) with each benchmark consisting of an XML-based profile with related utility scripts. The process from the benchmark installation, to the actual benchmarking, to the parsing of important hardware and software components is heavily automated and completely repeatable, asking users only for confirmation of actions.

The Phoronix Test Suite can be used for simply comparing your computer's performance with your friends and co-workers or can also be used at your company for internal quality assurance purposes under Linux. Results from the Phoronix Test Suite are displayed in a results viewer with optional support for uploading them to PTS Global. PTS Global allows you to browse all uploaded results, search these results (coming soon), and comparing other results against your own system.

This benchmarking software with all benchmarking profiles can be found on the downloads page. The latest development code is housed at Phorogit, the public git repository hosted by Phoronix. All support inquiries and discussions can be directed to the Phoronix Test Suite Forum.

Also part of the Phoronix Test Suite is pts-grapher (git repository) and the pts-results-viewer.

If you're interested in getting involved with the Phoronix Test Suite as either a developer or profile/suite maintainer for your favorite software package(s) (or other software that you develop), contact phoronix [at] phoronix.com as we do welcome user contributions.

http://www.phoronix.com/scan.php?page=ar..._090&num=1

Quote:Phoronix Test Suite 0.9.0 Released

New test profiles in Phoronix Test Suite 0.9.0 include mencoder, ffmpeg, compliance-sensors, and compliance-acpi. The mencoder test uses the libavcodec family of codecs to measure the time needed to decode/encode an AVI video file and the ffmpeg test is also similar but is using FFmpeg with the libavcodec package. For checking system compatibility, the compliance-sensors test checks whether LM_Sensors is properly configured and specifically checks to see whether some critical sensors are properly working. Compliance-sensors will attempt to check the CPU core voltage, +3.3V / +5V / +12V line voltages, CPU fan speed, CPU temperature, and the system temperature. The compliance-acpi test is similar to compliance-sensors but is used for checking the available features of ACPI, the Advanced Configuration and Power Interface. This test will check if the monitor brightness control, CPU power management, thermal zones, and suspend (S3 and S4 modes) features are supported through ACPI. All of the compliance-* tests are now part of the system-compliance test suite.

For ideas on how to execute third-party linux applications from within XBMC using python checkout this topic-thread => http://forum.xbmc.org/showthread.php?tid=30659

It descibes how to get get emuLauncer python script to work under XBMC for Linux by executing a MAME emulator for Linux with parameters.

san Wrote:You could use the new open movie "Big Buck Bunny" as a nice starting point. They provide their movie in standard and high definition (720 & 1080) encoded using four different codecs.

Check it out at: http://www.bigbuckbunny.org/index.php/download/

Elephants Dream is another one, multiple formats and resolutions available on http://www.elephantsdream.org

Quote:Elephants Dream is the world’s first open movie, made entirely with open source graphics software such as Blender, and with all production files freely available to use however you please, under a Creative Commons license.

The short film was created by the Orange Open Movie Project studio in Amsterdam during 2005/2006, bringing together a diverse team of artists and developers from all over the world. More about the project...

A full list of other open content film can be found here:
http://en.wikipedia.org/wiki/List_of_open_content_films



PS! You could make the python script for XBMC download the videos, that way those would not have to be hosted at xbmc-scripts.com

Nice! There is some excellent stuff in this thread! I will definitely check out the Phoronix Test Sute.

Nice! There is some excellent stuff in this thread! I will definitely check out the Phoronix Test Sute.

Phoronix.com Releases Linux Test Suite 1.0

Quote:Today marks Phoronix.com's 4th anniversary and in celebration we have announced the release of Phoronix Test Suite "Trondheim" 1.0. Phoronix Test Suite 1.0 is the most comprehensive testing and benchmarking platform available for Linux and is designed to carry out qualitative and quantitative tests in a clean, reproducible, and easy-to-use manner.

This GPLv3 software is shipping with 57 test profiles and 23 test suites covering many aspects of the Linux desktop. Furthermore, independent users can easily write their own profiles using this framework and the documented, standardized XML specification. In forthcoming releases we'll be adding even more profiles and features! The Phoronix Test Suite supports reporting test results from any (official or user-created) profile to PTS Global, a publicly-controlled repository of all results with advanced sorting, graphing, and display features.

There are many other features to the Phoronix Test Suite such as sensor monitoring, hardware/software detection, and batch mode support. For more information on Phoronix Test Suite 1.0, check out the press release at http://www.phoronix.com/vr.php?view=12427 or by visiting http://www.phoronix-test-suite.com/

All the best,
Phoronix Media
http://www.phoronix.com/
http://www.phoronixforums.com/
http://www.phoronix-test-suite.com/