[optip]

progress today...

it took a bit of trial and error but i got a good start made on the socket programming, to produce a proper directory listing from the contents of an NZB file...

it works by downloading the header and first couple of lines for all of the articles referenced in the nzb file from the news server... this gives an accurate listing of the file names and file sizes as if it was a real directory.

Having dug into this a bit, its not necessary to sort the xml nzb... instead, on opening the nzb, the file names for all the parts are scanned into an array/vector so when a file within the nzb is requested in the nzb://smb://x.y.z/filename.ext the filename is matched against the nzb part from the array and that part is downloaded immediately

Im getting the feeling that seeking will be at least technically possible, if not relatively easy! We'll see; but this would be great to have presuming broken/incomplete posts will be an issue that will affect nzb support

[C-Quel]

how you getting along? hope all is well and looking forward to seeing some output from this

Kudos!

[optip]

Hi,

I havent been able to spend so much time on this this weekend, but progress is good.

all this basics are in place now. if you open an NZB, it is treated as a directory in a similar way to RARs. pretty much all the functionality around this part is done.
- nzb is understood
- (basic) nntp library is implimented
- when an nzb is opened as a directory, the CFileNzb class makes nntp connections to the newsserver to check what's really in the file from the yEnc headers
- directory listing includes filesizes from this header

I've almost got the playing part working too. I've pretty much finished the open and stat methods of the IFile... im currently working on the Read method.. the key one i guess! My limited knowledge of socket programming in c++ is causing me some trouble, with recvs throwing the occasional exception after a delay and im not sure why. I'll figure it out tho!

things i haven't started on yet are or arent happy with are

- to get things moving quickly, i used static variables in the CFile::Open to persist the contents of the directory so it doesn't have to parse the nzb or go back to nntp when requests are made for the subs, etc. might be a better way to cache this then in static variables in ram. gonna go back to if if deemed necessary once ive seen something playing from this, and getting past a couple of yEnc parts!

- will need to so some type of settings page for nntp service settings. using a class with hard coded values at the moment.

I tested with my newsreaders throughput when using just one download thread - and instead of its usual ~320KB/s download speed, it was only getting 260... this might mean it will be necessary to use two download threads (gets 308 - close enough) and a bigger download buffer (/tmp file) to get the most efficiency from the bandwidth

there is a possible side effect of treating nzbs as directories in that if something looks in them as part of a directory scan, etc it will result in connections to nntp, and the inclusion of that content from usenet being part of that search... many nzb postings also contain the nzb inside the post so i have lots in my media directories. i don't expect that people will intend those to be part of such scans. (i havent tested that yet tho!)

anyway... today's not good for time so hopefully tomorrow, i'll have something at least functional, if not polished.

[nate12o6]

how difficult would it be to create a setting for the number of threads opened for download? A lot of ISP will limit your bandwidth per thread but give you 4 threads.

PS. Good to hear that your making progress!

[optip]

how difficult would it be to create a setting for the number of threads opened for download? A lot of ISP will limit your bandwidth per thread but give you 4 threads.

i thought about this alright... the problem with more threads in the streaming context is that unless you keep a very big buffer, the multiple download threads are competing against eachother for bandwidth for the streaming buffer... this means you need to keep enough buffer for all 4 (eg) streams times two... one for buffer while downloading, and another for buffer with data unencoded to be passed to mplayer... does that make sense? (maybe some of this can be kept in a tmp file instead of ram buffer)

i think you're right though... its not much more technically complicated to set up a user definable number of threads as to set up 2 so it's probably the right way to go.

[optip]

another progress update...

i'l sooooooooo close to having this functional! it's not quite there yet though.

The Read() method now successfully downloads yenc parts as required. It also yDecodes the content and saves the yDecoded in a disk based buffer for reading to mplayer.

the problem i have now however is that for some reason, the yDecoded content that is saved on the disk is slightly different from the original of what i downloaded... this makes the files unreadable to mplayer.

For example, with mp3 files, the ID3 header seem relatively intact but spaces are replaced with @ symbols... im sure there are other examples similar to this that span the whole file.

i've started trying to debug where this problem is coming from but ive got no solid ideas as yet...

if there are any common yEnc decoding problems that my problem might fit into, let me know! still - should hopefully have something streaming soon!

[optip]

below is my yenc part decoder... if anybody has an idea as to why it doesn't work, i'd appreciate it!!

characters in the output loaded into unEncFileContents have nulls instead of spaces, etc

Code:

int CFileNZB::yEncDecodeBlock(byte* Block, int size, byte* unEncFileContents)
{
  byte x;
  byte* O = (byte*) malloc(size); ////byte[Block.Length];
  byte* Out = (byte*) malloc(size);
  int count = 0;
  for (int i=0;i < size; i++)
  {
    x = Block[i];
    if (x=='\r')
    {

      i++;
      if (Block[i] == '\n')
      {
        continue;
      }
      else
      {
        CLog::Log(LOGINFO, "There was a carriage return without a linefeed detected");
        return NULL;
//        throw new Exception("There was a carriage return without a linefeed detected");
      }
    }
    else if (x == '=')
    {
      i++;
      byte r = x;
      x = Block[i];
      O[count] = (byte)((x-64)-42);
     }
    else
    {
      O[count] = (byte)(x-42);
    }
    count++;
  }
  if (count != 0)
  {
    //Out = new byte[count];
    //Buffer.BlockCopy(O,0,Out,0,count);
    //unEncFileContents =  (byte*) malloc(count);
    memcpy(unEncFileContents,O,count);
    free(O);
    return count;
   }
  free(O);
  return NULL;
}

PS. please ignore the inneficient malloc, etc will clean it up when it works properly!

[nate12o6]

i really wish i could help you.

[optip]

aha... i figured it out!

the problem stemmed from using CStdString to store the data coming from the socket... moved things to byte* data structures and the decoding now seems to be perfect!

fingers crossed, i'll have streamed music for the first time with this in the next hour

will have to do multithreading for video to work but the concept will have been proven!

will report back when it works

[optip]

it works!

have successfully opened an nzb file (from my filesystem) which contains references to mp3 files, and successfully streamed the content inside

it's got issues! its only single threaded and the download next part is not asynchronous to there are skips and pops in the sound... there are also memory leaks from my somewhat amateur attempts from moving to a nice managed CStdString to a malloc'd byte*

there no problems remaining that can't be solved tho!

will start on the cleanup and at least async downloading tomorrow!

[geot]

Superb, excellent effort.
I've been watching this thread since it started.
If there is anything that I can help with in the near future, let me know. It maybe 2 to 3 weeks before I can offer any programming (v.rusty) help.

[nate12o6]

This is great news. I just hope that the devs will put in in the svn so that it will be in the t3ch releases!
Good job!

[optip]

thanks guys!... i hope so anyway

ive still got a bit to do though... i started cleaning up and commenting what i have so far today but i dont have a lot of time for it.

its memory footprint is waaaay too big. i originally downloaded and yDecoded in RAM. allowing for multithreading, etc this uses way too much RAM. My first priority is to move everything into temporary files and pass file pointers between methods instead of pointers to decodeded content, etc. At the same time, i'll make the code the downloads from usenet and decodes thread aware - so they make temp files that include their threadid in the filename or something like that... i was worried for a while yesterday that with decoding problems i was having but now that i have heard music playing through an nzb file, its full steam ahead again to make this compact and hopefully soon enough, i'll have a patch which anybody can integrate with their source library & i'll raise getting this into trunk with those who can make such decisions!

I did speak briefly with cptspiff on the IRC channel before i started on this about whether this functionality would be suitable for trunk inclusion and he indicated that on the presumption that it was stable & worked, etc; he didn't see why not. That is certainly my goal with this, all going well!

I'm a little bit surprised about how little interest there seems to be generally with direct nzb support... i wonder if people don't know what they are, or don't like usenet (i guess you do have to pay!), or prefer bittorrent, or have their own system, etc! If this works well, personally my mind boggles at the thought of a good python script pointed a newzbin (or whereever) working with this! talk about video, etc on demand! i can't wait until it's stable!... gotta get a balance between working on this, and the rest of my life tho!!

thanks anyway for the support!

(btw...i wonder how complicated bittorrent is!... i have a vague idea how p2p works so i know it's complicated but i'm thinking that a bittorrent client done in a similar way to this really would be killer app!.. it'd be crap on ratio tho )

[nate12o6]

From what i hear bit torrent uses a lot of ram. But who knows, i don't think anyone has every tried to implement one native to xbmc. Just a bunch of python scripts.

I don't think that a lot of people or aware of the fact that most isp give free usenet access. Even to binary groups! They just have to go to there isp's website and do a little research. I use cox communications and we get 4 threads at 44KB each for no extra charge. Granted its not as good as easynews or other pay group sites but it is still quick enough to stream a movie or music for free I bet after this is in trunk the interest will definitely grow.

[chunk_1970]

thanks guys!... i hope so anyway
If this works well, personally my mind boggles at the thought of a good python script pointed a newzbin (or whereever) working with this! talk about video, etc on demand! i can't wait until it's stable!... gotta get a balance between working on this, and the rest of my life tho!!

Hi, I have been working on my script Torrent-X to add support for the sabnzbd downloader and the newzbin site. Its working but not fully tested yet. I dont see that it would be too difficult to add in support to pass the data to your code in xbmc. It would be no great shakes to pass the nzb to your code especially if there was a httpapi command that called your code passing the nzb file, people could basically click the file on the site and select an option like (play now) and your code then does its business..

It does sound like the whole thing could generate a huge buzz in the newsgroup area. Like you say many people dont seem to be stating a interest in whats happenning but there are alot of people out there. Theyre probably just waiting to see what happens..

I only have a 1mb connection so my direct streaming capabilities are useless anyways..