+- XBMC Community Forum (http://forum.xbmc.org)
+-- Forum: Development (/forumdisplay.php?fid=32)
+--- Forum: Scraper Development (/forumdisplay.php?fid=60)
+--- Thread: ScraperXML (Open Source XML Web Scraper C# Library) please help verify my work... (/showthread.php?tid=50055)
- Gamester17 - 2009-05-05 17:10
If you open source this code/library and could pitch this concept in a better way to MediaPortal and MeediOS developers then we could all use the same scrapers and share the development of them, checkout:
http://forum.team-mediaportal.com/improvement-suggestions-46/suggestion-use-xbmcs-xml-scrapers-http-scraping-35312/
and:
http://www.meedios.com/forum/viewtopic.php?t=2238
Just the same all other open source media managers that are dotnet based could use this as well:
http://forum.xbmc.org/tags.php?tag=media+manager
Possible making this scraper method (as well as XBMC's NFO formatting) the open standard for all open source media center and media management applications
- Nicezia - 2009-05-06 06:14
spiff Wrote:great!
i would put alot of effort into having your parser work as a library if i were you. it will make it alot more useful, in particular i sincerly hope that stuff like MIP will pick it up.
I think that's going to be the only way seeing as how Monodevelop has absolutely no support for creating a GUI with .NET, save for C# which i don't know the slightest bit about.
As far as it goes with just the console module, all i have to finish up at this moment is my considerations of custom functions(which i'm working on at this moment) and then go back and account for error handling and streamline the code.
- Nicezia - 2009-05-06 06:28
Gamester17 Wrote:If you open source this code/library and could pitch this concept in a better way to MediaPortal and MeediOS developers then we could all use the same scrapers and share the development of them, checkout:
http://forum.team-mediaportal.com/improvement-suggestions-46/suggestion-use-xbmcs-xml-scrapers-http-scraping-35312/
and:
http://www.meedios.com/forum/viewtopic.php?t=2238
Just the same all other open source media managers that are dotnet based could use this as well:
http://forum.xbmc.org/tags.php?tag=media+manager
Possible making this scraper method (as well as XBMC's NFO formatting) the open standard for all open source media center and media management applications
Not A Bad Idea at all i hadn't even considered that, but I was considering integrating this into a catalog manager that i wrote, it handles books and comics, movies, TVShows , the only thing is with it so far is that all info has to be put in manually (except for movies, which uses theMovieDB Api).
But it would definately be a good idea for everyone to have a unified and forwards thinking scraper library.
Since my end goal is making an editor for the ScraperXMLs (and now that i have a better understanding myself of how these things work, that goal seems simpler than it did last week) maybe that would calm the people who are saying its too difficult to program for .... (And i actually dissagree with that now considering that I have had no formal programming training myself and have only been self-teaching myself programming for about 11 months)
- Nicezia - 2009-05-06 14:20
Ummm, i kinda coded it this way without thinking about it, but just to make sure... the Function dest only tells XBMC where to look for the Information that was gathered by its root RegExp Element... right?
I was just looking back through my code and checking for errors when i realized there was a skip in the chain between regExp and the function, but i was using the Function Dest to pull the info to send to either get a page or pull results (based on what function is in play)
And last thing i need to verify is... while a custom function is running it uses a seperate Bufferspace from the main one with a completely fresh set of buffers $$1-$$20?
- spiff - 2009-05-06 14:33
yes on the first one.
the second one; depends on whether or not the calling function has clearbuffers="no" set. if it isn't set, clear buffers after function execution, if it is set do not and hence the next function should be called with the previous buffer state (excluding the first one which holds the data of course).
- Gamester17 - 2009-05-06 15:25
Nicezia Wrote:I was considering integrating this into a catalog manager that i wrote, it handles books and comics, movies, TVShows , the only thing is with it so far is that all info has to be put in manually (except for movies, which uses theMovieDB Api).@Nicezia, I think that you (and other working on similar things) should checkout this "Unified Media Manager" project and consider joining it:
But it would definately be a good idea for everyone to have a unified and forwards thinking scraper library.
http://forum.xbmc.org/showthread.php?tid=50388
- spiff - 2009-05-06 16:44
nailing this is far more important than getting any unified media manager started, or rather a prerequisite. keep your focus here mate
- Nicezia - 2009-05-06 17:15
I'm on it spiff , its my primary goal as well i started coding the library last night
wasn't that hard, it was pretty much a cut and paste of all the functions i had already written, and from that point adding properties and making sure everything still worked as i t was working as a console program (only without the interface options - so i just took all the items left over in from the console program and transferred them into a test program , everything still works and custom functions are accounted for, now its just the post/get and referrer handling that's left.
- Nicezia - 2009-05-08 06:33
Okay I'm trying to document my library for those who may potentially use it.
I need to know all the possible types of settings so that i can make suggestions on how to implement them.
So far working with the scrapers that currently exist in XBMC i have only run across labelenum, bool, sep and text. Are there any more?
- spiff - 2009-05-08 10:37
'integer'. there are a few properties on the text fields (in particular hidden is of interest). see GUIDialogPluginSettings.cpp