[RELEASE] Scraper Editor (Based on ScraperXML open source C# Library) - Help wanted!

  Thread Rating:
  • 1 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Thumbs Up  [RELEASE] Scraper Editor (Based on ScraperXML open source C# Library) - Help wanted!
Post: #1
Scraper editor partially finished (finished enough to make scrapers)

Haven't added the tester yet, as i'm having problems with scraperXML's http protocol in c#

Runnning on Linux via Mono (Notice there were a few bugs i had to work out on it namely the bindings, but now it runs error free on linux)[Image: ScraperEditorLinuxviaMono.png]


Running on windows (Notice the highlighting of Matches and turning text of match groups to Red)
[Image: ScraperEditorOnWindowsExpressionEdi.png]

Download


Just a slight note, use ctrl+v to paste text into box below as i haven't included a context menu for that yet, in the expression box i haven't yet finished the context menu, (which will include common regular expression which will be configurable via an XML file)

This will load edit and save any scraper made from scratch or that comes with XBMC. In it is an enhanced version of the Excalibur Scraper (which gets linked actors AND non-linked actors - and instead of guessing at image names actually goes and downloads images from the actor page)

oh and as a reminder !!!!DO NOT!!!!! use XML entities in the editor, put the pure xml in and entities will be managed on compiling the XML file!

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
(This post was last modified: 2009-07-05 00:30 by Nicezia.)
find quote
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #2
nice! just not having to write those xml tags will save tons of time Smile
find quote
DonJ Offline
Team-Kodi Member
Posts: 524
Joined: May 2005
Reputation: 5
Post: #3
Looks great, thanks for the effort!

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find quote
zag Offline
Team-Kodi Member
Posts: 1,681
Joined: Oct 2007
Reputation: 20
Location: UK
Post: #4
Excellent work, very useful
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #5
Currently adding xml editing for those "purists" and integrating ScraperXML for testing scrapers. All previously broken features are fixed, and hope to release the full version of it by end of next week. ScraperXML will be updated as soon as finish all its integration featues (using this for an integration demonstration.

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
C-Quel Offline
Retired Team-Kodi Member
Posts: 1,375
Joined: Aug 2004
Reputation: 0
Post: #6
Big Grin

Zotac ID89 + 4GB + 160GB Intel SSD + Samsung UE40D7000 + DS411+II / 2 x 3TB WD RED CAVIAR (TVHeadend Package + 4 Tuners) + Fibaro HC2 Home Automation Intergration!

^^^

Fucking awesome springs to mind :)

iNerd Store

iNerd Forum
find quote
Gamester17 Offline
Team-XBMC Forum Moderator
Posts: 10,523
Joined: Sep 2003
Reputation: 10
Location: Sweden
Thumbs Up  Nice Nicezia!
Post: #7
Great stuff, thanks! Cool

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find quote
DonJ Offline
Team-Kodi Member
Posts: 524
Joined: May 2005
Reputation: 5
Post: #8
A type of 'debug mode' would be brilliant for the app. So you can throw a query search term or nfo at the app and step through the regexp expressions and functions while seeing the values of the buffers.

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #9
DonJ Wrote:A type of 'debug mode' would be brilliant for the app. So you can throw a query search term or nfo at the app and step through the regexp expressions and functions while seeing the values of the buffers.

Well i'm making it so that one can test each functon individually you can give a url (or local file) to test the function with, as far as stepping through i'd have to make a few modifications to the scraperxml library just for the tester, but i'm already doing that with other options (like the individual function testing took a few mods to the source of ScraperXML)


As for the buffers, i curretly have it so you can watch up to three buffers at once (each watch can be selected via combobox) anything more than three at a time can get too crowded

DonJ if you give me a list of your ideas for this "debug mode" i can work on integrating them into the program

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
(This post was last modified: 2009-06-18 19:09 by Nicezia.)
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #10
New Release coming monday
(I figure i'll give myself the weekend to work out the bugs in linux)


Function Editor
[Image: FunctionEditor.jpg]

RegExp Editor
[Image: RegExpEditor.jpg]

Scraper Info Editor
[Image: ScraperInfoEditor.jpg]

Tester
[Image: Tester.jpg]
(Tester Fully Functional in windows, Linux has a problem with the walk mode)

One can throw anything at the Tester, Simulate XBMC's passed values, open NFO's, XML's saved html.. or set the buffers to whatever you want. As shown below

[Image: Tester_QueryMode.jpg]


Is this what you had in mind DonJ Confused

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
(This post was last modified: 2009-06-20 21:33 by Nicezia.)
find quote
DonJ Offline
Team-Kodi Member
Posts: 524
Joined: May 2005
Reputation: 5
Post: #11
Yes, looks great. Looking forward to the next release.

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #12
Thanks for the sticky!!!

Version 1.5 is ready

Here ya go new release.
Download and ENJOY!!!

I'm working on a Help file that will not only explain the use of ScraperXML Editor, but give a detailed tutorial from start to finish of writing a scraper, some basic guidelines on do's and don'ts and good scraper writing tips.

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
(This post was last modified: 2009-06-23 01:11 by Nicezia.)
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #13
Is anybody using this?
I need a little feedback, as to if there are any errors , and some suggestions for improvment would be nice as well

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
Eso Rimmer Offline
Junior Member
Posts: 7
Joined: Jul 2009
Reputation: 0
Post: #14
Will there be any tutorial for this tool? When I open working scraper, how can I test it see how it works? I have no idea how to use it.
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #15
Eso Rimmer Wrote:Will there be any tutorial for this tool? When I open working scraper, how can I test it see how it works? I have no idea how to use it.

Yes there's a whole suite coming soon with Help files

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
Post Reply