Windows [WIP] FreakOut - NZB Downloader/Updater for Games

(2012-03-22 14:54)HenryFord Wrote:  So - I began to implement ROM-support for earlier consoles (like N64, SNES, Amiga, etc.).
I am thinking about using coolrom.com for ROM-downloads since it is the most convenient site to use.
Other sites either use sharehoster like mediafire, etc. or use captcha-methods to prevent users from downloading without visiting.
coolrom only has a counter - seems to be fairly easy to circumvent, but I can't figure out how... Can someone help me out here?

Another option would be to setup an own server for this - a server which practically mirrors the ROMs and only makes them available for this program to use (at some point probably opening a website though).
For this however I would need money... I can do it first on my own VPS, but once it would reach a certain traffic-point I would have to collect donations in order to keep it up.
So as of now (without knowledge how to circumvent the countdown) I will provide a few ROMs on my VPS.

I hope you're allowing searching for nzb's for these kinds of roms too.

I'll always prefer nzbs thus those Downloads will only happen via http if there is nothing else available.
The url resolver won't help by the way (at least not with coolrom) because they use some strange, variable Post-data. But i think i figured it out and sending my own post will probably help resolve this issue. I'll keep you posted (pun intended)

Right, the urlresolvers grab the original page, build the post url, submit it, and scrape the link from the resulting page. Don't think we have one for mediafire yet, but the techniques should be the same

Just hit a roadblock with coolrom.com - I can't for the life of me figure out what the hell they are doing to prevent me from fullfilling the request...
The thing is - I think I figured out how they do it:
They use two hidden-fields in the form which will send POST-Data to the server after you click on "Start download". I can see with a sniffer that these POST-Data get transmitted.
So now I figured I just download the page, parse it and get the values from those two hidden fields. That is totally working...
As a result you will get a variable with randomname=randomvalue - exactly like on the original page. The second field is called "sid" which holds your PHP-Session-ID. I set this with the use of my cookie I generated after visiting the page. Now - everything seems to be fine: Matching randomname=randomvalue variable and a matching sid=phpsessid . Oh - of course I also use the cookie to be used.
Now I went ahead and resembled THE COMPLETE HTTP-Header of the request made by my browser. It is literally the same header (with different values for randomname=randomvalue and sid=phpsessid obviously though), with the same User-Agent, Content-Length, etc. - yet the site refuses to deliver the octet-stream and redirects me to the rom-detail-page instead.
I...just don't know how the hell they are doing it. Frustrating.

(@Bstrdsmkr: The problem with coolrom seems to be that they are using the webserver to deliver the contents of the file. You don't directly access the file itself via a direct link but the server delivers a stream which would have to be written to the disk manually. So - after posting the correct URL to the server you get a octet-stream as a response, not a HTML page )

Eww that sounds painful. Disable and clear your cookies/temp files and see if you now get that behavior in the browser. That'll tell you if it's cookie based. There's also a Firefox addon that lets you disable headers, so you can use that to narrow down the problem. They've only got 3 vectors to track you so it has to be either:
1. Cookies
2. Headers
3. Page content

I've seen sites that track your session id across pages. If they don't see it show up on a logical progression of pages, they kick you to the wrong page. Could that be it? Maybe you've done everything right on the page itself, but they know you didn't visit a previous page?

(2012-03-16 18:17)xecutionx Wrote:  Please, please consider moving to Python so this can be used on a variety of systems. The reason that audio-matic failed to gain as many followers as Headphones did was due to the fact it was Windows only.

That being said, it looks amazing. I can definitely see myself using this if it had Rom support.

+1 .... please consider python!!

cheers
c_bb

(2012-03-23 20:58)Bstrdsmkr Wrote:  Eww that sounds painful. Disable and clear your cookies/temp files and see if you now get that behavior in the browser. That'll tell you if it's cookie based. There's also a Firefox addon that lets you disable headers, so you can use that to narrow down the problem.

I used Fiddler to deeply analyze the packages that are being sent, I even went so far to do a real indepth analysis with tcpdumps - but well... I can't figure it out. They definetly use a cookie for the phpsession - but methinks that's not all the magic here (although I cannot see anything else being sent...)

Quote:I've seen sites that track your session id across pages. If they don't see it show up on a logical progression of pages, they kick you to the wrong page. Could that be it? Maybe you've done everything right on the page itself, but they know you didn't visit a previous page?

Well...technically I do visit the site before (I have to in order to parse the webpage) and I use a cookie there as well - but to no avail. I also set the correct refferer-header, so that can't be it... I'm stumped


@Cool_bombom: Seriously? Please consider reading the thread before throwing stupid stuff like that around.

(2012-03-25 03:17)HenryFord Wrote:  @Cool_bombom: Seriously? Please consider reading the thread before throwing stupid stuff like that around.

Sorry about that ... just wanted to show my support for a python conversion! Anyways brillant work and idea :-)

cheers
C_bb

Well - no problem
Had one of my grumpy days...

Currently not much time to continue the work, but eastern is coming along and I am off work for the week afterwards, probably a lot will happen in that time
Stay tuned!

Any plans to include support for creating launchers for Steam games ? however I suspect and understand it might be too niche an use and it might be just up to me to make the launcher bats manually.