NZB Video Streamer script in development, help wanted...

I was frustrated with my options for streaming HD video to my HTPC, so I decided to develop an NZB Video Streamer script for XBMC.

I adapted a front-end from switch's XBMC plugin designed to control SabNZBd (for queuing NZBs). That plugin didn't support streaming so I replaced the SabNZBd backend with my own implementation. Streaming support requires getting info about the progress and status of the download which I deliver with callback functions. Also, this plugin doesn't require you to install any external packages (i.e. SabNZBd!).

Switch's front-end was already integrated with several NZB RSS or search sources: Newzbin, Binsearch, TVNZB, and NZBIndex. There is still a lot of work to be done on the front-end though.

The plugin is currently on the SVN addon repository at:
https://code.google.com/p/xbmc-addons/so...20Streamer

It is accessible from XBMC's addon installer. I've tested it on two Windows boxes, one 32-bit XP and one 64-bit Windows 7. I have yet to test it on Linux.

Things still left to develop:
- improve reliability for non-Windows platforms; par2.exe is currently only checked in for Windows
- add file sizes and useful metadata about whether the NZB in question is complete or what language it's in, etc. to the NZB list
- add intelligent filtering to not list NZBs that aren't really videos (i.e. they just hold PARs and/or NFO)
- clean up release names so they're nice to read
- detect passworded RARs and inform the user about those

> My goals:

Why develop a new backend for this? SABNZBd is cross-platform and already capable of doing most of this. I wrote a plugin myself which allows one to browse the content of the fill threads database (a dutch usenet indexing community). The plugin links the content to the SABNZBd plugin for XBMC such that the selected content is automatically downloaded, verified, extracted etc. This way only new front-ends need to be added (as either a plugin or as a script).

I was thinking along the lines of allowing SABNZBd to extract content on-the-fly. The last update apparently already performs on-the-fly PAR checking.
The next step is to extract the files as long as this check succeeds and otherwise terminate the on-the-fly extraction and await a repair.
If this content is directly extracted to the Video directory and XBMC allows playback of incomplete videos then a streaming solution is ready.

Using SABNZBd gives you: NZB parsing, sorting, downloading, decoding, verification, repair, extraction, multiple-threads, etc...

> Using the PARs is not really an option because AFAIK one must have all the PARs in order to start repairing.

No not all the PARs, you just need enough blocks. Blocks come either from the actual content or the PAR blocks. In practice this means you can only repair after the download completed.

I just started looking into scripting XBMC myself, so I'm afraid I have little advice on how the play the RARs.

My experience in the past was that XBMC refused to play incomplete multi-rar archives or incomplete movie files (using on-the-fly extraction) which makes streaming rather useless. Has this changed?

Good luck to you, I've been looking for something like this for ages - finally you've got the know how to build it! Awesome! Keep me updated!

Cheers Sion

Caine Wrote:> My goals:

Why develop a new backend for this? SABNZBd is cross-platform and already capable of doing most of this. I wrote a plugin myself which allows one to browse the content of the fill threads database (a dutch usenet indexing community). The plugin links the content to the SABNZBd plugin for XBMC such that the selected content is automatically downloaded, verified, extracted etc. This way only new front-ends need to be added (as either a plugin or as a script).

I considered using existing backends but they were either poor cross-platform (HellaNZB) or somewhat overwrought for this purpose. I'm not sure that the backend for this streaming-oriented script should be a daemon since it has modal activity based on the current state of the download (i.e. it needs to abort on error and start playing once the first RAR is done).

Coding the download stuff with nntplib was really not that hard and like I said it's mostly already done. And there is no need to do extraction, the XBMC core handles that.

Quote:No not all the PARs, you just need enough blocks. Blocks come either from the actual content or the PAR blocks. In practice this means you can only repair after the download completed.

Exactly, broken files break the stream. This is a bummer and it's why I had my idea about re-uploading existing releases in the new streaming-friendly container. Making that idea a reality would also require changes to existing news readers (to handle on-the-fly parity).

Quote:My experience in the past was that XBMC refused to play incomplete multi-rar archives or incomplete movie files (using on-the-fly extraction) which makes streaming rather useless. Has this changed?

It will only play a RAR set once all the RARs exist. They don't have to be the correct size, they just have to exist. So my first step after parsing an NZB is to create all the files in the NZB with 0 bytes.

This would be the most useful plugin ever...at least for me. I have been looking for a plugin similar to this for years. I am so excited to see that you are taking on the challenge! I have subscribed to the thread and can't wait to see this project grow.

I suggest you look into: nzbplayer.com
I once contacted to get the source code to implement that program into NAS devices.
Had no time.
But if you contact the producers of this app, maybe they can help you.
just a suggestion

OK, I figured out how to play a RAR archive like can be done interactively. I do this from the Python API.
An example:
rar://c%3a%5cUsers%5c<username>%5cAppData%5cRoaming%5cXBMC%5cuserdata%5cscript%5fdata%5cNZB%5cflhd%2dfringes​01e02%2erar/flhd-fringes01e02.mkv

The problem now is that if you stop the player after playing a RAR like this and you try to start the same RAR again, it will sometimes hang and give these errors:
12:24:12 T:3176 M:2338213888 NOTICE: -->Python Interpreter Initialized<--
12:24:12 T:2560 M:2331361280 NOTICE: DVDPlayer: Opening: rar://C%3A%5CUsers%5C<username>%5CAppData%5CRoaming%5CXBMC%5Cuserdata%5Cscript_data%5CNZB%5Cflhd%2dfringes01​e02%2erar/flhd-fringes01e02.mkv
12:24:12 T:2560 M:2331361280 WARNING: CDVDMessageQueue(player)::Put MSGQ_NOT_INITIALIZED
12:24:12 T:2188 M:2331361280 NOTICE: Creating InputStream
12:24:12 T:2188 M:2331095040 NOTICE: Creating Demuxer
12:24:12 T:2188 M:2330771456 ERROR: XFILE::CFile::Read : Access violation at 0x008e422a: Reading location 0x0cc46d8f
12:24:42 T:2188 M:2332160000 ERROR: XFILE::CFileRar::Seek - Timeout waiting for buffer to empty
12:25:12 T:2188 M:2335555584 ERROR: XFILE::CFileRar::Seek - Timeout waiting for buffer to empty
12:25:42 T:2188 M:2340700160 ERROR: XFILE::CFileRar::Seek - Timeout waiting for buffer to empty
12:25:42 T:2188 M:2340700160 NOTICE: Opening video stream: 0 source: 256

My first guess is that it may be trying to resume to the spot you stopped from and is having some related issue. I believe there is a setting to turn off that default behavior to see if you can tie the error specifically to resume.

I've looked into streaming from nzb quite a bit in the past. What are your plans for handling multiple video files in the RAR set? Unfortunately, a lot of usenet posters insist on breaking videos in 700Mb chunks (WHO actually burns to CD)

My main use case for this capability is for streaming HD video, which is rarely split into multiple videos. I've had trouble with a pure python rarfile library not closing the file handles and trying to read 0-byte RARs (I create all the RARs as 0 byte because that's what XBMC needs to play from them - another reason I had to write my own backend) Right now I just take the basename of the RARs and try to play several extensions of that in the RARs (.mkv, .avi, .mpg, but that can be adjustable). Eventually I'll get the RAR file listing working of course (but it'll still only be able to list the files available in the first RAR).

I'll update the first post with my progress.

OK, I submitted the plugin to SVN and I thought it was working but it seems my recurring issue with the fresh playback stopping as soon as it starts is still a problem. When I finish the first RAR and start playing it, it shows the first second of video and then stops. My script detects that the player didn't keep playing the video, so it aborts the download, thinking that there's no movie in the RARs or that the RARs are passworded. But when I retry the same NZB, a par2.exe check will prevent having to redownload the first RAR again and it will immediately start playing - that time it always works even though the commands to play are the same.

Here's the log from the fresh play, when it stops working after a second:
22:29:04 T:3460 M:1969049600 NOTICE: DVDPlayer: Opening: rar://C%3A%5CUsers%5CMatt%20Chambers%5CAppData%5CRoaming%5CXBMC%5Cuserdata%5Cplugin_da​ta%5CVideo%5CNZB%20Streamer%20svn%5Cbetter%2eoff%2eted%2es02e09%2ehdtv%2exvid%2d​xii%5Cbetter%2eoff%2eted%2es02e09%2ehdtv%2exvid%2dxii%2erar/better.off.ted.s02e09.hdtv.xvid-xii.avi
22:29:04 T:3460 M:1969049600 WARNING: CDVDMessageQueue(player)::Put MSGQ_NOT_INITIALIZED
22:29:04 T:3612 M:1969049600 NOTICE: Creating InputStream
22:29:04 T:3612 M:1968779264 NOTICE: Creating Demuxer
22:29:04 T:3612 M:1967456256 NOTICE: Opening video stream: 0 source: 256
22:29:04 T:3612 M:1967456256 NOTICE: Creating video codec with codec id: 13
22:29:04 T:3612 M:1967435776 NOTICE: CDVDVideoCodecFFmpeg::Open() Using codec: MPEG-4 part 2
22:29:04 T:3612 M:1967415296 NOTICE: Creating video thread
22:29:04 T:3612 M:1967403008 NOTICE: Opening audio stream: 1 source: 256
22:29:04 T:3612 M:1967398912 NOTICE: Finding audio codec for: 86017
22:29:04 T:5008 M:1967390720 NOTICE: running thread: video_thread
22:29:04 T:3612 M:1967349760 NOTICE: Creating audio thread
22:29:04 T:3056 M:1967337472 NOTICE: running thread: CDVDPlayerAudio::Process()
22:29:04 T:3056 M:1967153152 NOTICE: Creating audio device with codec id: 86017, channels: 2, sample rate: 48000, no pass-through
22:29:04 T:3056 M:1966075904 WARNING: CDVDMessageQueue(audio)::Get - retrieved last data packet of queue
22:29:04 T:5008 M:1965883392 NOTICE: fps: 23.976000, pwidth: 640, pheight: 352, dwidth: 640, dheight: 352
22:29:04 T:5008 M:1965883392 NOTICE: Display resolution DESKTOP : 1280x1024 @ 60.00 - Full Screen (12)
22:29:04 T:5008 M:1967808512 WARNING: CDVDMessageQueue(video)::Get - retrieved last data packet of queue
22:29:04 T:3612 M:1967804416 NOTICE: CDVDPlayer::OnExit()
22:29:04 T:3612 M:1967804416 NOTICE: DVDPlayer: eof, waiting for queues to empty
22:29:04 T:3612 M:1967804416 NOTICE: DVDPlayer: closing audio stream
22:29:04 T:3612 M:1967804416 NOTICE: Closing audio stream
22:29:04 T:3612 M:1967804416 NOTICE: CDVDMessageQueue(audio)::WaitUntilEmpty
22:29:04 T:3056 M:1967804416 NOTICE: thread end: CDVDPlayerAudio::OnExit()
22:29:04 T:3612 M:1967804416 NOTICE: Waiting for audio thread to exit
22:29:04 T:3612 M:1967808512 NOTICE: Closing audio device
22:29:04 T:3612 M:1967673344 NOTICE: Deleting audio codec
22:29:04 T:3612 M:1967673344 NOTICE: DVDPlayer: closing video stream
22:29:04 T:3612 M:1967673344 NOTICE: Closing video stream
22:29:04 T:3612 M:1967673344 NOTICE: CDVDMessageQueue(video)::WaitUntilEmpty
22:29:04 T:3612 M:1967673344 NOTICE: waiting for video thread to exit
22:29:04 T:5008 M:1967673344 ERROR: Got MSGQ_ABORT or MSGO_IS_ERROR return true
22:29:04 T:5008 M:1967673344 NOTICE: thread end: video_thread
22:29:04 T:3612 M:1967710208 NOTICE: deleting video codec
22:29:04 T:3612 M:1968033792 NOTICE: CDVDPlayer::OnExit() deleting demuxer
22:29:04 T:3612 M:1967828992 NOTICE: CDVDPlayer::OnExit() deleting input stream
22:29:05 T:3460 M:1964855296 NOTICE: CDVDPlayer::CloseFile()
22:29:05 T:3460 M:1964040192 WARNING: CDVDMessageQueue(player)::Put MSGQ_NOT_INITIALIZED
22:29:05 T:3460 M:1964040192 NOTICE: DVDPlayer: waiting for threads to exit
22:29:05 T:3460 M:1964040192 NOTICE: DVDPlayer: finished waiting


Here's the log from when I play it the second time and it works fine:
22:30:22 T:3460 M:1961615360 NOTICE: DVDPlayer: Opening: rar://C%3A%5CUsers%5CMatt%20Chambers%5CAppData%5CRoaming%5CXBMC%5Cuserdata%5Cplugin_da​ta%5CVideo%5CNZB%20Streamer%20svn%5Cbetter%2eoff%2eted%2es02e09%2ehdtv%2exvid%2d​xii%5Cbetter%2eoff%2eted%2es02e09%2ehdtv%2exvid%2dxii%2erar/better.off.ted.s02e09.hdtv.xvid-xii.avi
22:30:22 T:3460 M:1961615360 WARNING: CDVDMessageQueue(player)::Put MSGQ_NOT_INITIALIZED
22:30:22 T:1924 M:1961615360 NOTICE: Creating InputStream
22:30:22 T:1924 M:1961353216 NOTICE: Creating Demuxer
22:30:22 T:1924 M:1960583168 NOTICE: Opening video stream: 0 source: 256
22:30:22 T:1924 M:1960583168 NOTICE: Creating video codec with codec id: 13
22:30:22 T:1924 M:1960562688 NOTICE: CDVDVideoCodecFFmpeg::Open() Using codec: MPEG-4 part 2
22:30:22 T:1924 M:1960562688 NOTICE: Creating video thread
22:30:22 T:1924 M:1960562688 NOTICE: Opening audio stream: 1 source: 256
22:30:22 T:1924 M:1960562688 NOTICE: Finding audio codec for: 86017
22:30:22 T:4876 M:1960562688 NOTICE: running thread: video_thread
22:30:22 T:1924 M:1960554496 NOTICE: Creating audio thread
22:30:22 T:4588 M:1960554496 NOTICE: running thread: CDVDPlayerAudio::Process()
22:30:22 T:4588 M:1958944768 NOTICE: Creating audio device with codec id: 86017, channels: 2, sample rate: 48000, no pass-through
22:30:22 T:4588 M:1956040704 WARNING: CDVDMessageQueue(audio)::Get - retrieved last data packet of queue
22:30:22 T:4876 M:1956294656 NOTICE: fps: 23.976000, pwidth: 640, pheight: 352, dwidth: 640, dheight: 352
22:30:22 T:4876 M:1956294656 NOTICE: Display resolution DESKTOP : 1280x1024 @ 60.00 - Full Screen (12)

The only difference I see is an extra "retrieved last data packet of queue" in the fresh play, but I don't understand what would cause that. The first RAR is complete in both cases.