[RELEASE] Scraper Editor (Based on ScraperXML open source C# Library) - Help wanted!

  Thread Rating:
  • 1 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #91
DuMbGuM Wrote:Just a quick question guys wondering if you can help me out, I was trying to use this tool to edit the tvdb scraper to scrape episode ratings from tv.com but have gotten completely lost trying to do it. Any idea if this had already been done or how I might do it? I'm not sure if it's a simple task or not.

I just want the same tvdb scraper but one which scrapes all episode ratings from tv.com as the ratings there are much more comprehensive.

Thanks, and nice work on this by the way!

I'm actually personally working on a rewrite of the tvdb scraper (since some of the way its written makes it incompatible with ScraperXML Editor)

My suggestion would to be to make an include file to get the tv.com ratings. You'd have to find a way to be able to find the exact tvshow and episode from the tv.com given the tvshow/episode and find the right page given the tvdb episode & tvshow title.

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
DuMbGuM Offline
Fan
Posts: 448
Joined: Sep 2008
Reputation: 1
Location: Ireland
Post: #92
Ah okay, good thinking, i'll give it a shot thanks for the help

[Image: watched-clearlogo.jpg]
find quote
wikid Offline
Junior Member
Posts: 9
Joined: Jan 2010
Reputation: 0
Post: #93
Hi, I just created these two threads, that maybe should help a little bit in regards to helping the development and debugging your scraper:

http://forum.xbmc.org/showthread.php?tid=66189

AND

http://forum.xbmc.org/showthread.php?tid=66191

Cheers!

WiKiD
find quote
flobbes Offline
Senior Member
Posts: 142
Joined: Mar 2009
Reputation: 0
Post: #94
I cant get the editor running neither under windows nor linux.

Using Linux I at least get an error message, dont know if it helps you any further.

http://paste.ubuntu.com/363832/
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #95
it seems you did get the editor running since its gotten to the point of moving between treenodes (on the scraper tree), when does this error occur?

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
flobbes Offline
Senior Member
Posts: 142
Joined: Mar 2009
Reputation: 0
Post: #96
Nicezia Wrote:it seems you did get the editor running since its gotten to the point of moving between treenodes (on the scraper tree), when does this error occur?

I See the Editor and it immediatly crashes before I can do anything.
If I delete the config files, a dialog to choose the scraper dir appears.
After doing that it crashes.

Under Windows I get this error:

http://paste.ubuntu.com/364731/
(This post was last modified: 2010-01-28 20:53 by flobbes.)
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #97
delete your old config.xml in the app folder, the configuration format has changed

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
flobbes Offline
Senior Member
Posts: 142
Joined: Mar 2009
Reputation: 0
Post: #98
Im starting it from a clean install, dont have an old .cfg

Just wanted to mention when I delete the created config files, I at least see the "choose scraper dir" dialog, after doing that the program crashes.

Edit: Tried it with an empty dir under windows, that worked.
Wow the program is really looking nice!
Grats and thanks for your effort!
(This post was last modified: 2010-01-29 01:14 by flobbes.)
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #99
then it should ask you for your scraper folder before it even starts did you at least get that far?

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #100
oh i see it says you do get that far, are you choosing the base folder? (the structure must match XBMC's folder ./video and ./music with common folders as subfolders of these... or just choose the xbmc/system/scrapers folder)

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
flobbes Offline
Senior Member
Posts: 142
Joined: Mar 2009
Reputation: 0
Post: #101
At the moment im using an empty folder, that works fine.

But I have a problem, using the tester.

THe program crashes when enabling "buffer edit". And it crashes when I load a file from cache and start the testing.

Error enabling buffer edit:
http://paste.ubuntu.com/365114/
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #102
uploading 3.6.0 now should fix that problem

also this version fixes all scraper testing (tv show scraper whould work for both tvshow and tvepisode testing now)

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
(This post was last modified: 2010-01-29 23:47 by Nicezia.)
find quote
flobbes Offline
Senior Member
Posts: 142
Joined: Mar 2009
Reputation: 0
Post: #103
Nice Buffer edit works now.

When testing my own scrapers, I always get:

Code:
Could not download webpage.
If you were running auto mode it will be halted

Error returned:
Ung├╝ltiger URI: Das URI-Schema ist ung├╝ltig

The last one translates to:

Invalid URI: The URI-syntax is invalid

Any idea what the problem could be?
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #104
i need to see the url its trying to download in the way the scraper returns it in order to know why

and if you're trying to use gzip, that's still something i haven't added as of yet, since i don't truly know how to handle gzipped webpages, still learning when it comes to programming internet interaction.

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
(This post was last modified: 2010-01-30 19:30 by Nicezia.)
find quote
flobbes Offline
Senior Member
Posts: 142
Joined: Mar 2009
Reputation: 0
Post: #105
Its a scraper for a german adult site, so not safe for work.

The CreateSearchUrl is:

http://e.orgazmik.ch/film/list/dvd/SRCH....SRCHSTR=\1

The scraper (and my others are working fine) with xbmc, but the editor gives my that error message on all my scrapers.

The whole scraper can be found here:
http://pastebin.com/m72cf8c38
(This post was last modified: 2010-01-30 23:58 by flobbes.)
find quote
Post Reply