[WIP] AniDB.net Anime Video Scraper - Printable Version
+- XBMC Community Forum (http://forum.xbmc.org)
+-- Forum: Help and Support (/forumdisplay.php?fid=33)
+--- Forum: Add-ons Help and Support (/forumdisplay.php?fid=27)
+---- Forum: Metadata scrapers (/forumdisplay.php?fid=147)
+---- Thread: [WIP] AniDB.net Anime Video Scraper (/showthread.php?tid=64587)
- d3us - 2011-03-03 18:16
i already tried that, anidb client and xbmc wiki, but i couldn't find answer... as i said before, everything used to work fine till season 10! but now...
- salival - 2011-03-03 22:40
For those intrested, here is the code I use for the anidb java client in anidb.net. (in the client, go to options > use tagging system > edit tagging system)
With the following regexp in your advancedsettings.xml I get most anime, including specials and movies. For movies you will not get fanart (because the tvdb doesn't have fanart for movies).
Also, to improve the chance of this scraper to get the right anime, name your folder exactly as the anime is named on anidb. You can also use one of the names in the titles field of the animedescription.
- pathw - 2011-03-11 14:38
two questions regarding how we rely on tvdb
1) For example I have this show http://anidb.net/perl-bin/animedb.pl?show=anime&aid=1719. This however gets identified as http://thetvdb.com/?tab=series&id=89491&lid=7 .
Can we identify the root cause? I think it's because of special characters. The anidb lookup is correct because I have nfo files everywhere with the right aid. it's hard to do the lookup using the data xbmc gives us because it cleans up the string "a.li.ce" as "a li ce"
But if we use the anidb information for the tvdb lookup. This problem of inconsistency should go away.
2) Regarding images
This show http://anidb.net/perl-bin/animedb.pl?show=anime&aid=6111, has no analogue in tvdb.
In this case the view is inconsistent because some images are unavailable and so on. Can we choose anidb images scaled up when tvdb images arent available?
3) Images for some shows like a.li.ce as mentioned before look scaled up disproportionately in the "tv show information" context menu. Is this something we can fix?
Lastly, a more general problem. I've started modifying my anidb.xml file for my own needs. Specifically I want specials to be episodes with SXX, TXX, CXX, OXX and PXX. (specials, trailers, openings, other and parodies). I've used the same general strategy of identifying these as belonging to a different season. Anything other than S gets a different season in the high 100s. I then set the displayseason using the same strategy of 0 or 2. Should I push this back.
- pathw - 2011-03-27 10:59
Finalspace Wrote:That issue for infinte loop and crashes xbmc scrapping process is definitly annoying and i want to do something to get this thing working.
So i did some research into this problem to try and figure out how this infinite loop happens. It's really quite strange. It has to do with 3 functions.
after analyzing the show from the anidb page, GetTVDBLookupAPI is called to get any fanart and such.
GetTVDBLookupAPI scrapes the anib page for information to guess the tvdb name. It also stores all the prequels. It returns a call to GetFanartAPI with the tvdb search url
GetFanartAPI tries to get fanart from the tvdb data. It make a call to GetTVDBLookupLoopAPI to lookup prequels when there is no tvdb data
GetTVDBLookupLoopAPI makes a call to GetTVDBLookupAPI with the prequel info.
This problem of an infinite loop happens when you have a show that has no data on TVDB and it has a prequel that ALSO has no data on TVDB. But the error that occurs seems to be related to a bug in XBMC. I say this because the bug doesnt happen everytime, but seems non deterministic. It happens MOST times however.
I've appended 2 log sections. The first time is when the lookup did not fail. The second time when it fails. (the foo and bar attributes are some debugging information that I added. foo is the id of any known prequel, bar is the shows name). when xbmc crashes, for some reason when the tvdblookuploop calls tvdblookup with a prequel, the curl call doesnt happen. So the tvdb lookup returns the same show again, and this just goes on forever.
Quote:13:47:09 T:3236 M:2237095936 DEBUG: scraper: GetTVDBLookupAPI returned <details><url function="GetFanartAPI" cache="tvdb-s5592-To%20Heart%202%20ad.xml"foo="4806"bar="To Heart 2 ad">http://www.thetvdb.com/api/GetSeries.php?seriesname=To%20Heart%202%20ad</url></details>
Quote:13:49:44 T:2680 M:2241540096 DEBUG: scraper: GetTVDBLookupAPI returned <details><url function="GetFanartAPI" cache="tvdb-s5592-To%20Heart%202%20ad.xml"foo="4806"bar="To Heart 2 ad">http://www.thetvdb.com/api/GetSeries.php?seriesname=To%20Heart%202%20ad</url></details>
notice how foo and bar are related to the show being looked up even when we are looking up the prequel.
- bambi73 - 2011-03-27 17:58
Your observations are correct, there is bug in XBMC (at least from my point of view :oo. I posted some description for developers here.
I'll try to add some workaround to scraper and make version 1.0.1 ... after almost 3/4 year
- pathw - 2011-03-27 19:33
oh so that's the problem? do you think we should eliminate the cache calls in that area?
- bambi73 - 2011-03-27 20:14
Turning cache off anywhere is bad idea, you will get banned on AniDB.net in no time. It's enough to clear $$1 at the end of each function, so new content from site/cache is appended to empty string.
If you want to test my solution you can try v1.0.1b1. I'll welcome to see your test results.
- salival - 2011-03-27 22:13
Some time ago I believe I sent you a PM. It was about the animelist.xml
You can get the file from anidb.net by using the following url: http://anidb.net.nyud.net/api/animetitles.xml
This routes it through a cdn. I contacted the anidb staff to check if this was allowed, which it was. (here's the thread)
If you find yourself lacking time to maintain the anidb.xml on google sites you could substitute the url for the one above.
- bambi73 - 2011-03-28 17:30
salival Wrote:Hi Bambi,Hello,
sorry for not responding to your PMs, i was quite busy with my personal life and work in past 3/4 years and had almost no time to spend on XBMC related work.
About Coral CDN, i'll check its potential.
- salival - 2011-03-28 19:28
No need to apologize. I just took the opportunity while you're here.