Hi,
I have started development on adding support for nzb files to xbmc and hope to have something resembling a working prototype in the next week.
I have done some exploritory work and believe that i can make it work.
Will be implimented as CFileNzb / CNzbDirectory (with supporting classes) types and submitted as a patch to those who are interested... hopefully presuming it works and gets stable, it will make it into trunk.
Scope:
- open an nzb file from a supported url/path (probably typically being initiated from a python script which is outside the scope of what i am doing)
- treat an nzb file as a directory (similar to rar) listing contents of nzb using file mask (if just 1 media file, just play it... maybe take setting from rar setting/behaviour)
- Streaming a file from within an nzb is done by opening a url in the following format (similar to rar):
sample url: nzb://http%3a%2f%2fx.y.com%2fdir%2fnzbfile.nzb/SubjectOfFileToDownloadFromNZB.avi (URL Encoded Part = http://x.y.com/dir/nzbfile.nzb)
- stream media content in nzb file via nntp
- yDecode, etc on the fly (part by part - generally < parts 500k i think so doable)
- rar files?? see section on Issues i really don't know about yet
- the idea is that this a streaming solution - not downloading
- this will not save downloaded content anywhere (except maybe temp files - see below)
- might need to save yenc parts temporarilly on HD to avoid taking up too much RAM if there will be a couple of threads downloading
- Settings page for nntp settings (maybe incl. number of threads to run - or maybe better to derive number of threads from size of internet streaming cache/nzb/yEnc part size, or possibly hard code to 2 (current/next) )
Not in Scope
- I will not be creating any nzb site scraper type thing for the moment, although this functionality will rely on a python script like this to really be useful
- This is not a download tool like the SABnzb script, etc.... it's more of a filesystem/driver type thing. The SABnzb script could probably be modified however to use this instead of SABnzb. (i don't know any python however but it could be a nice learning project!
)
Issues i think i can solve:
- NZB files can list rar parts in any old order - need to sort to download in correct order to stream
- NNTP isn't supported by cUrl... need to write this (nntp is easy tho so shouldn't be difficult i hope
)
- Need to store nntp server and authentication details somewhere - will probably add this to settings page (or is this a source?)
- After a part is downloaded, it needs to be decoded... this could make for a skip if multi-threading isn't used for parts (and buffer isn't big enough)
Issues not thought out fully / tested / know about yet (feedback appreciated ):
- Broken/incomplete usenet posts are common; hence the existance of par files. Seeking will probably not be an option day 1 so a broken post will result in playback stopping and it being impossible to seek to your original position... maybe can detect broken posts/parts and skip forward to next rar / part, making for a jump and some lost viewing time... suspect any fix/workaround will not be in initial release!
- Using temp files feels like a nice solution for allowing CFileRar, etc to handle streamed rar files... i understand that CFileRar needs a real file handle for the underlying magic to be able to work. My initial thought was that streaming really means no temp files but there seems to be a lot of positive reasons for using temp files
- yEnc parts need to be downloaded in full before they can be decoded so we will never be streaming raw data from the internet but rather downloading a 500k part (size is an example), decoding that part, unraring that part (as part of a bigger rar file & set) and passing decoded & unrared bytes to the mplayer buffer
- if a rar file can be written to and read from at the same (ish) time, this will work (don't know if locking will be a problem with this, but i know xbmc can play partial rars) (this might be a long shot) (can always download and play full rar files (usually 15mb/file)- but this hardly qualifies as streaming... but it would work ok!
- temp files would also solve how to handle multi-part archives
- CFIleNzb would contain and manage a CFile member pointing to the the temp file
- This would make the nzb support much more managable by not duplicating portions of the rar support specifically for unraring in memory within the nzb code. The issues surrounding unraring in memory (and handling multi-part, etc) sound complicated so would be nice to side step that one!!
- this would probably require a minimum of 30MB of space for temp files (for current and next rar)
- Not using temp files implies the following
- need enough ram available for two downloaded parts & whatever it takes to decode and unrar (part size could vary)
- need to impliment code to unrar from memory, either by extending CFileRar or using RarManager / unrar dlls directory (not sure exactly what this will take)... this feels like a complicated coding job & would result in duplication of some functionality surrounding rar files = not too managable
- Many NZB sites use cookie authentication and/or get/post variables to access an nzb... need to be able to work with/around this somehow while staying nzb site neutral (haven't looked at how to manage this at all yet... may or may not already be addressed somewhere else for another application... pointers appreciated)
- Rar file recursion, etc... need to figure out how to handle rar files in rar files, etc and exceptions to the norm of rar files containing avi/mp3 files, etc...
I spoke briefly with cptspiff in the IRC channel about this functionality and he mentioned that to his knowledge, this has not been done before; and in particular that it had not been tried and failed for a good reason... if i'm wasting my time here, please let me know!
As i've been writing this, i've been figuring out things, and figuring out that there's a lot i don't know! If there are stupid statements above, or things which just miss the point, please let me know but be gentle!! Hopefully, i'll get this working and it will be a useful addon!
Regards,
Patrick
I have started development on adding support for nzb files to xbmc and hope to have something resembling a working prototype in the next week.
I have done some exploritory work and believe that i can make it work.
Will be implimented as CFileNzb / CNzbDirectory (with supporting classes) types and submitted as a patch to those who are interested... hopefully presuming it works and gets stable, it will make it into trunk.
Scope:
- open an nzb file from a supported url/path (probably typically being initiated from a python script which is outside the scope of what i am doing)
- treat an nzb file as a directory (similar to rar) listing contents of nzb using file mask (if just 1 media file, just play it... maybe take setting from rar setting/behaviour)
- Streaming a file from within an nzb is done by opening a url in the following format (similar to rar):
sample url: nzb://http%3a%2f%2fx.y.com%2fdir%2fnzbfile.nzb/SubjectOfFileToDownloadFromNZB.avi (URL Encoded Part = http://x.y.com/dir/nzbfile.nzb)
- stream media content in nzb file via nntp
- yDecode, etc on the fly (part by part - generally < parts 500k i think so doable)
- rar files?? see section on Issues i really don't know about yet
- the idea is that this a streaming solution - not downloading
- this will not save downloaded content anywhere (except maybe temp files - see below)
- might need to save yenc parts temporarilly on HD to avoid taking up too much RAM if there will be a couple of threads downloading
- Settings page for nntp settings (maybe incl. number of threads to run - or maybe better to derive number of threads from size of internet streaming cache/nzb/yEnc part size, or possibly hard code to 2 (current/next) )
Not in Scope
- I will not be creating any nzb site scraper type thing for the moment, although this functionality will rely on a python script like this to really be useful
- This is not a download tool like the SABnzb script, etc.... it's more of a filesystem/driver type thing. The SABnzb script could probably be modified however to use this instead of SABnzb. (i don't know any python however but it could be a nice learning project!
)Issues i think i can solve:
- NZB files can list rar parts in any old order - need to sort to download in correct order to stream
- NNTP isn't supported by cUrl... need to write this (nntp is easy tho so shouldn't be difficult i hope
)- Need to store nntp server and authentication details somewhere - will probably add this to settings page (or is this a source?)
- After a part is downloaded, it needs to be decoded... this could make for a skip if multi-threading isn't used for parts (and buffer isn't big enough)
Issues not thought out fully / tested / know about yet (feedback appreciated
):- Broken/incomplete usenet posts are common; hence the existance of par files. Seeking will probably not be an option day 1 so a broken post will result in playback stopping and it being impossible to seek to your original position... maybe can detect broken posts/parts and skip forward to next rar / part, making for a jump and some lost viewing time... suspect any fix/workaround will not be in initial release!
- Using temp files feels like a nice solution for allowing CFileRar, etc to handle streamed rar files... i understand that CFileRar needs a real file handle for the underlying magic to be able to work. My initial thought was that streaming really means no temp files but there seems to be a lot of positive reasons for using temp files
- yEnc parts need to be downloaded in full before they can be decoded so we will never be streaming raw data from the internet but rather downloading a 500k part (size is an example), decoding that part, unraring that part (as part of a bigger rar file & set) and passing decoded & unrared bytes to the mplayer buffer
- if a rar file can be written to and read from at the same (ish) time, this will work (don't know if locking will be a problem with this, but i know xbmc can play partial rars) (this might be a long shot) (can always download and play full rar files (usually 15mb/file)- but this hardly qualifies as streaming... but it would work ok!
- temp files would also solve how to handle multi-part archives
- CFIleNzb would contain and manage a CFile member pointing to the the temp file
- This would make the nzb support much more managable by not duplicating portions of the rar support specifically for unraring in memory within the nzb code. The issues surrounding unraring in memory (and handling multi-part, etc) sound complicated so would be nice to side step that one!!
- this would probably require a minimum of 30MB of space for temp files (for current and next rar)
- Not using temp files implies the following
- need enough ram available for two downloaded parts & whatever it takes to decode and unrar (part size could vary)
- need to impliment code to unrar from memory, either by extending CFileRar or using RarManager / unrar dlls directory (not sure exactly what this will take)... this feels like a complicated coding job & would result in duplication of some functionality surrounding rar files = not too managable
- Many NZB sites use cookie authentication and/or get/post variables to access an nzb... need to be able to work with/around this somehow while staying nzb site neutral (haven't looked at how to manage this at all yet... may or may not already be addressed somewhere else for another application... pointers appreciated)
- Rar file recursion, etc... need to figure out how to handle rar files in rar files, etc and exceptions to the norm of rar files containing avi/mp3 files, etc...
I spoke briefly with cptspiff in the IRC channel about this functionality and he mentioned that to his knowledge, this has not been done before; and in particular that it had not been tried and failed for a good reason... if i'm wasting my time here, please let me know!
As i've been writing this, i've been figuring out things, and figuring out that there's a lot i don't know! If there are stupid statements above, or things which just miss the point, please let me know but be gentle!! Hopefully, i'll get this working and it will be a useful addon!
Regards,
Patrick
