• 1
  • 2(current)
  • 3
  • 4
  • 5
  • 22
ScraperXML (Open Source XML Web Scraper C# Library) please help verify my work...
#16
Nicezia Wrote:I was considering integrating this into a catalog manager that i wrote, it handles books and comics, movies, TVShows , the only thing is with it so far is that all info has to be put in manually (except for movies, which uses theMovieDB Api).

But it would definately be a good idea for everyone to have a unified and forwards thinking scraper library.
@Nicezia, I think that you (and other working on similar things) should checkout this "Unified Media Manager" project and consider joining it:
http://forum.xbmc.org/showthread.php?tid=50388

Nod
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
Reply
#17
nailing this is far more important than getting any unified media manager started, or rather a prerequisite. keep your focus here mate Smile
Reply
#18
I'm on it spiff , its my primary goal as well i started coding the library last night
wasn't that hard, it was pretty much a cut and paste of all the functions i had already written, and from that point adding properties and making sure everything still worked as i t was working as a console program (only without the interface options - so i just took all the items left over in from the console program and transferred them into a test program , everything still works and custom functions are accounted for, now its just the post/get and referrer handling that's left.
Reply
#19
Okay I'm trying to document my library for those who may potentially use it.

I need to know all the possible types of settings so that i can make suggestions on how to implement them.

So far working with the scrapers that currently exist in XBMC i have only run across labelenum, bool, sep and text. Are there any more?
Reply
#20
'integer'. there are a few properties on the text fields (in particular hidden is of interest). see GUIDialogPluginSettings.cpp
Reply
#21
The Library itself is complete, but I'm trying to make a test program to demonstrate its use. If you'd like just the library to try to feel your way through it then PM Me
Reply
#22
Rainbow 
Once the library is ready for release (as open source under the GPL?) may I suggest that you also announce it here:
http://forums.thetvdb.com
and here:
http://forums.themoviedb.org

as well as here:
http://forum.team-mediaportal.com/improv...ing-35312/
and here:
http://www.meedios.com/forum/viewtopic.php?t=2238

With more software using the same scraper framework and API we will all eventually be able to share XML scrapers files 'as is' between different project!

Very cool IMHO! Cool

PS! Will you make this library available on SourceForge and/or Google Code or a similar site as its on project?

Huh
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
Reply
#23
I'm having two problems with the library,
the first is with theTvdb.com, after running the "GetLanguages" custom function, i end up with an extra nesting in settings
Code:
<settings>
   <settings>
      <setting></setting>
   </settings>
   <setting></setting>
   <setting></setting>
</settings>
Seeing how it runs everything else fine (except tv.com which seems to have updated their pages) I'm wondering if this is just caused by a malformed scraper, or is there some bug i need to track down?

And yes i plan to release it under the GPL.


right now i just need to find out if this is an error on my part or not.
Reply
#24
Gamester17 Wrote:Once the library is ready for release (as open source under the GPL?) may I suggest that you also announce it here:
http://forums.thetvdb.com
and here:
http://forums.themoviedb.org

as well as here:
http://forum.team-mediaportal.com/improv...ing-35312/
and here:
http://www.meedios.com/forum/viewtopic.php?t=2238

With more software using the same scraper framework and API we will all eventually be able to share XML scrapers files 'as is' between different project!

Very cool IMHO! Cool

PS! Will you make this library available on SourceForge and/or Google Code or a similar site as its on project?

Huh

Lots of questions my head's gonna explode...

Well Yes i want to put it under some open source lisence, but i'm new to actually releasing any software into the world... so i don't know where to start on that.
Reply
#25
I suggest starting by giving the library its own unique name and then registering it as its own project on Google Code or SourceForge, uploading the code to its SVN for continued working on it there, maintaining both code and documentation there, and zip up the first 'alpha' version releasing it as a rebuilt download in a .zip file.

With it being available on Google Code or SourceForge in an SVN more people can work on it and the code will never get lost if someone quits the project.
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
Reply
#26
ok i have my answer, to that question, its not an error in the scraper, however, this is going to take some pretty interesting programing to work around.

The tvdb is th only one i have this problem with and i'm afraid i might break the others if i screw with it, i don't understand why put the <settings> tag on it, instead of doing a element replace with the data gathered from the custom function...

I've been trying to program around this all morning, so far haven't broke the other scrapers ability to run...

maybe if i just addit as a child element so that it comes out like

Code:
<settings>
    <setting label="Language" type="labelenum" id="language" values="da|fi|nl|de|it|es|fr|pl|hu|el|tr|ru|he|ja|pt|zh|cs|sl|hr|ko|en|sv|no|" default="en"></setting>
</settings>
<settings>
    <setting label="Use DVD Order" type="bool" id="dvdorder" default="false">
    </setting>
    <setting label="Enable Fanart" type="bool" id="fanart" default="true">
    </setting><setting label="Prefer Posters" type="bool" id="posters" default="false">
   </setting><setting type="sep"></setting><setting label="Language Override for Banners" type="bool" id="override" default="false">
   </setting>
</settings>

so far its the only way i see to work around it, but not sure if that works with XBMC
Reply
#27
Allright,
I just registered my library at sourceforge under the name ScraperXML waiting for approval , as soon as they ok it, i'm going to have it up there


I'm still not happy with a few of the work-arounds i used for settings, i couldn't quite understand how XBMC knows when and how to send those settings to the scraper process, but the way i implemented it is to have a persistent XMLElement in the ScraperParser, that the program implementing it can allow the settings to be set, and then sent to the scraper parser, when the scraper parser comes across the request for info $INFO[whateversetting.@id] it requests that setting's value from the persistent xmlElement.

I think XBMC pulls that info from a sqllite db, right? If that's the case my work-arounds should be decent enough to substitute.
Reply
#28
xbmc has a general class that parses the settings into its structures. it can parse several xml blocks and concate them. when we want to save the settings, we ask the class to regenerate the xml. it will of course stick everything in a single tag.

this is also exactly how the other chains works. we parse several <details> blocks
Reply
#29
spiff Wrote:xbmc has a general class that parses the settings into its structures. it can parse several xml blocks and concate them. when we want to save the settings, we ask the class to regenerate the xml. it will of course stick everything in a single tag.

this is also exactly how the other chains works. we parse several <details> blocks

ah geez, now i feel a little stupid, i didn't even think of looking to see if linq supported concatenating, which it does. Silly me.
Reply
#30
Is the default action for expression repeat "yes" or "no" ?

oops nevermind
Reply
  • 1
  • 2(current)
  • 3
  • 4
  • 5
  • 22

Logout Mark Read Team Forum Stats Members Help
ScraperXML (Open Source XML Web Scraper C# Library) please help verify my work...0