build, deploy, test, Grrr, refactor
#1
Now that XBMC is maturing and the code is starting to move in different directions, is it time that we start thinking about unit testing?

I have. Trying to unravel the mysteries of the microphone in the Karaoke functionality has left me dazed battered and bruised (not all self inflicted).

I am seriously considering setting up an automated build process (continuous integration) so that i will know i have broken something when i check it in locally.

The build, deploy, test, refactor is starting to take its toll.

any thoughts?

BOB.SMITH
Reply
#2
Unit testing works nicely for small simple classes, but anything much more complicated and it often gets very difficult to even write tests (or at least to write ones that can be automated)

Ofcourse I (and the team) support any and all efforts to aid in any way at all. Anything that makes it easier to not make mistakes is great.

And assuming you have all the debugging stuff setup correctly, just hitting F5 from visual studio does build + deploy to xbox - one hopes you're not developing without the debugger active?

Cheers,
Jonathan
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#3
No John its ok, i'm using debugger, i'm just not a developer, my C++ days were a long, long time ago, so its a learning curve for me.

What i do need though, is at least one other xbox, as i only currently have the one. This means development is confined to the wee small hours so i don't interrupt the missus watching dexter (or whatever is flavour of the month). I keep watching eBay for that elusive (non buggered about with) bargain.

I have taken a step back from thinking about the unit tests, and i have been thinking about the functional system tests.

The project needs some test documentation writing. Even as an ardent fan of XBMC i don't know all of the features and more importantly i don't know if they work correctly. There is a high probability of the Oracle assumption rearing its ugly head here.

Perhaps this would be a better starting place for me. Create the functional test documentation. That way people would have valid scripts to follow when the team releases a significant build. Make defect tracking easier (we would have test case number), and then work can be started on automating the manual tests.

Your thoughts?

BOB.SMITH
Reply
#4
I've been doing a lot of video testing using a ton of zero byte files that I named to match movies, tv shows, and music videos in various forms. It started where I actually had a user send me a directory listing of their huge movie repository (it was about 1000 movies) which I then quickly perl scripted to touch as zero byte files. This has been working nicely.

The downside is they don't play. So instead, it may be wise to get some very short creative commons licensed videos in various formats, make a lot of copies of each and name them accordingly. The same could be done for music files as well.
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
Reply
#5
kraqh3d Wrote:So instead, it may be wise to get some very short creative commons licensed videos in various formats, make a lot of copies of each and name them accordingly. The same could be done for music files as well.
Yes, the same was suggested here for benchmarking as well => http://forum.xbmc.org/showthread.php?tid=33645

it lists a few sources for creative commons licensed multimedia material as well.
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
Reply
#6
good ideas.

i've used the divx test cd in the past as that has various encodes of divx and xvid clips on it with various audio encodes. its meant for testing the capabilities of standalone players but its a good litmus test as xbmc normaly plays the whole disc no problem.

the idea of having a media drive with contrived data is a good one.

data driven testing.

BOB.SMITH
Reply
#7
Any and all efforts would be extremely welcome Smile
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#8
hi mr marshall,

i have read through the user guide, but i'm worried that its either

1) not correct
2) out of date

Given that there are no functional specifications i was going to use these as a starting point for the test cases. That leaves me with using the oracle assumption - use the existing system for comparison.

On the whole formalising the testware for the project will be non trivial, and i will seek some input from my test peers to find a way in which we can implement software testing on the project that brings real benefit - i.e reduces risk, increases quality.

BOB.SMITH
Reply
#9
Sounds like the best way to go about it. Yes, the manual is in some places out of date, so the best plan of attack would be starting where we are now.
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#10
Besides the creative commons licensed files available there are also several sites that host multimedia files for audio and video that the copyright has expired for, such sites as archive.org

PS! I have a few CDs with test samples, I do no know the license of them but if anyone wants them to check that out then please provide a server for me to upload them to and then send me a PM.
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
Reply

Logout Mark Read Team Forum Stats Members Help
build, deploy, test, Grrr, refactor0