+- XBMC Community Forum (http://forum.xbmc.org)
+-- Forum: Development (/forumdisplay.php?fid=32)
+--- Forum: Scraper Development (/forumdisplay.php?fid=60)
+--- Thread: ScraperXML (Open Source XML Web Scraper C# Library) please help verify my work... (/showthread.php?tid=50055)
- Nicezia - 2009-07-10 22:49
a question for users of this, would you rather the urlencoded trailer link or should i change it to a simple link if its urlencoded?
- smeehrrr - 2009-07-10 23:33
Nicezia Wrote:a question for users of this, would you rather the urlencoded trailer link or should i change it to a simple link if its urlencoded?I don't understand this question.
- Nicezia - 2009-07-11 04:49
well some trailers come urlencoded (i.e. http%3A%2F%2Fwww.foo.com%2Furl.flv)
would you rather me leave it like that before sending to final details or would you like to decode the urlencoding? (seeing as how i'm creating a charset convertor)
- smeehrrr - 2009-07-11 07:06
It would make more sense for your API to return a decoded URL in the MovieTag object, I think. At any rate it should be consistent, so if some scrapers return an encoded url and others return a decoded one, you should pick one and normalize to that form always.
- xyber - 2009-07-11 13:23
Nicezia Wrote:Will edit it to log scraper return values
However what exactley is the error/failure, i haven't seen this error and can't fix it if i don't know the full details of it. it retrives fanart just fine for me.
It was http://www.themoviedb.org/ that was returning a server 500 error as someone else mentioned. Not error in your lib. What I was thinking though is that it would be nice if you could create a class one could access to query errors that occured.
For example,
Code:
public string GetDetails(string strResultsEntity)Code:
//Exception handling Web
catch (System.Net.WebException wex)
{
if (LoggingEnabled)
{
IO.Log(ScraperLogFile, "WebException :" + wex.Message);
}
return null;
}It was one of the XElement SecondPass = CustomFunctionParse(item2); calls that caused the exception. Can't remember if it was the first or second pass one.
Anycase, now that I think bout it, it might be hard to actually tell what went wrong as it could be a query to a server for trailers, or one for fanart or another server for posters. So it won't be easy knowing what to tell the user to turn off in the scraper settings.
- xyber - 2009-07-11 13:25
Nicezia Wrote:well some trailers come urlencoded (i.e. http%3A%2F%2Fwww.foo.com%2Furl.flv)
would you rather me leave it like that before sending to final details or would you like to decode the urlencoding? (seeing as how i'm creating a charset convertor)
Don't really care. Long as I can use it to download the trailer via my app when I finally get around coding that part. Guess it would be better to show an unencoded version to the user if an app where to show the user the url.
- xyber - 2009-07-11 16:28
Does the TV Show section of the lib work? I get very strange results or none at all using TvScraper.CreateSearch and TvScraper.GetDetails
this is TvScraper.GetDetails for Heroes on IMDB
Code:
<tvshow>
<plot>tt0813715</plot>and rest of it is only episodeguide, thumbs, backdrops and actor tags.
and I just spot this in scraper.log
Code:
2009/07/11 - 16:37:59 : XML Exception: There are multiple root elements. Line 1, position 88. - unable to parse <url function="GetSeriesPremiered">http://akas.imdb.com/title/tt0813715/episodes</url><url function="GetSeriesPlot">http://akas.imdb.com/title/tt0813715/plotsummary</url><url cache="tt0813715-credits.html" function="GetSeriesCast">http://akas.imdb.com/title/tt0813715/</url><url cache="tt0813715-posters.html" function="GetIMPALink">http://akas.imdb.com/title/tt0813715/posters</url><url cache="tt0813715-posters.html" function="GetIMDBPoster">http://akas.imdb.com/title/tt0813715/posters</url><episodeguide><url>http://www.imdb.com/title/tt0813715/episodes</url></episodeguide>- ultrabrutal - 2009-07-11 19:12
when will we see some of these fixes hitting svn and a new build? (don't have C# compiler installed). I still get crashes in GetDetails and they are not solely because of fanart
- Nicezia - 2009-07-12 02:56
Well i'm in the process of testing a new release now, i suppose somewhere between midweek and the weekend I'll be submitting it to svn,
no TV shows isn't working just yet in the version that's in svn, however i have it working fully in the version i'm testing. and hope to upload really soon.
- Nicezia - 2009-07-12 02:59
If anyone knows anything about ISO charsets & windows codepages (for conversion purposes) i need someone to help with a charset conversion utility from non-latin character sets to unicode