+- XBMC Community Forum (http://forum.xbmc.org)
+-- Forum: Help and Support (/forumdisplay.php?fid=33)
+--- Forum: Add-ons Help and Support (/forumdisplay.php?fid=27)
+---- Forum: Metadata scrapers (/forumdisplay.php?fid=147)
+---- Thread: [WIP] AniDB.net Anime Video Scraper (/showthread.php?tid=64587)
- d3us - 2011-03-03 18:16
i already tried that, anidb client and xbmc wiki, but i couldn't find answer... as i said before, everything used to work fine till season 10! but now...
- salival - 2011-03-03 22:40
Finalspace Wrote:(...)
From the anidb client page you can find an simply java client for uses that for renaming files/adding to anidb.
http://anidb.net (you need an acc to use the java client)
Therefore you can rename all your files in an format which can be read by xbmc.
Check xbmc wiki page (TVShow) for more details.
For those intrested, here is the code I use for the anidb java client in anidb.net. (in the client, go to options > use tagging system > edit tagging system)
Code:
Trunc(str, len):=$repl(%str%, ".{" $len($repl(%str%, "(.?){" %len% "}$", "")) "}$", "")
TruncEllipse(str, len):= { $len(%str%) = $len($Trunc(%str%, %len%)) ? %str% : $Trunc(%str%, %len%) "…" }
AT:=[%ATr%,%ATe%] #anime title (romanji or english)
ET:=[%ETe%,%ETr%] #episode title (english or romanji)
ET:=$TruncEllipse(%ET%,"64") #truncate long titles
GT:="[" [%GTs%,%GTl%] "]" #group tag (short or long)
EpNoPad:=$pad(%EpNo%,$max($len(%EpHiNo%),$len(%EpCount%)),"0") #episode number padding
EpNoPad:="ep"%EpNoPad% #add "ep" in front
EpNoPad:=$repl(%EpNoPad%,'ep[sS]',"S00E0") #rename specials from "epS" to "S00E"
Src:="["%Source%"]" #set [source] e.g. [Blu-ray]
Ver:={%Ver%="1"?"":"v"%Ver%} #set version if appliccable
Res:="["%FVideoRes%"]" #set resolution e.g. [1280x720]
MEpNo:={%ET%="Complete Movie"?%EpNoPad% %Ver%:%EpNoPad% %Ver%" - "%ET%} #Only show title when not "Complete Movie"
Movie := %AT%" "%MEpNo%" "%Res%%Src%%GT% #for movies set to "animetitle ep01 [resolution][source][group]
Other := %AT%" "%EpNoPad% %Ver% " - "%ET%" "%Res%%Src% %GT% #all else set to "animetitle ep## - episode title [resolution][source][group]
FileName:= {%Type% = "Movie"? %Movie% : %Other%} #check if the file is a movie or not and rename appropiatelyWith the following regexp in your advancedsettings.xml I get most anime, including specials and movies. For movies you will not get fanart (because the tvdb doesn't have fanart for movies).
Code:
<advancedsettings>
<tvshowmatching action="prepend">
<regexp>ep([0-9]+)</regexp>
</tvshowmatching>
</advancedsettings>Also, to improve the chance of this scraper to get the right anime, name your folder exactly as the anime is named on anidb. You can also use one of the names in the titles field of the animedescription.
- pathw - 2011-03-11 14:38
two questions regarding how we rely on tvdb
1) For example I have this show http://anidb.net/perl-bin/animedb.pl?show=anime&aid=1719. This however gets identified as http://thetvdb.com/?tab=series&id=89491&lid=7 .
Can we identify the root cause? I think it's because of special characters. The anidb lookup is correct because I have nfo files everywhere with the right aid. it's hard to do the lookup using the data xbmc gives us because it cleans up the string "a.li.ce" as "a li ce"
But if we use the anidb information for the tvdb lookup. This problem of inconsistency should go away.
2) Regarding images
This show http://anidb.net/perl-bin/animedb.pl?show=anime&aid=6111, has no analogue in tvdb.
In this case the view is inconsistent because some images are unavailable and so on. Can we choose anidb images scaled up when tvdb images arent available?
3) Images for some shows like a.li.ce as mentioned before look scaled up disproportionately in the "tv show information" context menu. Is this something we can fix?
Lastly, a more general problem. I've started modifying my anidb.xml file for my own needs. Specifically I want specials to be episodes with SXX, TXX, CXX, OXX and PXX. (specials, trailers, openings, other and parodies). I've used the same general strategy of identifying these as belonging to a different season. Anything other than S gets a different season in the high 100s. I then set the displayseason using the same strategy of 0 or 2. Should I push this back.
- pathw - 2011-03-27 10:59
Finalspace Wrote:That issue for infinte loop and crashes xbmc scrapping process is definitly annoying and i want to do something to get this thing working.
I can simply reproduce that for the following folders:
"To Heart 2 ad"
"Bishoujo Senshi Sailor Moon R"
"Bishoujo Senshi Sailor Moon R (1993)"
Therefore i think it happens for all animes which dont exists by the anidb name in TVDB.
So i did some research into this problem to try and figure out how this infinite loop happens. It's really quite strange. It has to do with 3 functions.
after analyzing the show from the anidb page, GetTVDBLookupAPI is called to get any fanart and such.
GetTVDBLookupAPI scrapes the anib page for information to guess the tvdb name. It also stores all the prequels. It returns a call to GetFanartAPI with the tvdb search url
GetFanartAPI tries to get fanart from the tvdb data. It make a call to GetTVDBLookupLoopAPI to lookup prequels when there is no tvdb data
GetTVDBLookupLoopAPI makes a call to GetTVDBLookupAPI with the prequel info.
This problem of an infinite loop happens when you have a show that has no data on TVDB and it has a prequel that ALSO has no data on TVDB. But the error that occurs seems to be related to a bug in XBMC. I say this because the bug doesnt happen everytime, but seems non deterministic. It happens MOST times however.
I've appended 2 log sections. The first time is when the lookup did not fail. The second time when it fails. (the foo and bar attributes are some debugging information that I added. foo is the id of any known prequel, bar is the shows name). when xbmc crashes, for some reason when the tvdblookuploop calls tvdblookup with a prequel, the curl call doesnt happen. So the tvdb lookup returns the same show again, and this just goes on forever.
successful lookup.
Quote:13:47:09 T:3236 M:2237095936 DEBUG: scraper: GetTVDBLookupAPI returned <details><url function="GetFanartAPI" cache="tvdb-s5592-To%20Heart%202%20ad.xml"foo="4806"bar="To Heart 2 ad">http://www.thetvdb.com/api/GetSeries.php?seriesname=To%20Heart%202%20ad</url></details>
13:47:09 T:3236 M:2237095936 DEBUG: FileCurl::Open(086EF130) http://www.thetvdb.com/api/GetSeries.php?seriesname=To%20Heart%202%20ad
13:47:10 T:3236 M:2237100032 DEBUG: FileCurl::Close(086EF130) http://www.thetvdb.com/api/GetSeries.php?seriesname=To%20Heart%202%20ad
13:47:10 T:3236 M:2237116416 DEBUG: scraper: GetFanartAPI returned <details><url function="GetTVDBLookupLoopAPI" cache="5592.xml"foo="4806">&</url></details>
13:47:10 T:3236 M:2237095936 DEBUG: scraper: GetTVDBLookupLoopAPI returned <details foo=""><url function="GetTVDBLookupAPI" gzip="yes" cache="4806.xml">http://api.anidb.net:9001/httpapi?request=anime&client=xbmcscrap&clientver=1&protover=1&aid=4806</url></details>
13:47:10 T:3236 M:2237095936 DEBUG: FileCurl::Open(086EF130) http://api.anidb.net:9001/httpapi?request=anime&client=xbmcscrap&clientver=1&protover=1&aid=4806
13:47:11 T:3236 M:2237095936 DEBUG: FileCurl::Close(086EF130) http://api.anidb.net:9001/httpapi?request=anime&client=xbmcscrap&clientver=1&protover=1&aid=4806
13:47:11 T:3236 M:2237095936 DEBUG: scraper: GetTVDBLookupAPI returned <details><url function="GetFanartAPI" cache="tvdb-s4806-To%20Heart%202%20OVA.xml"foo=""bar="To Heart 2 OVA">http://www.thetvdb.com/api/GetSeries.php?seriesname=To%20Heart%202%20OVA</url></details>
13:47:11 T:3236 M:2237095936 DEBUG: FileCurl::Open(086EF130) http://www.thetvdb.com/api/GetSeries.php?seriesname=To%20Heart%202%20OVA
13:47:14 T:3236 M:2237870080 DEBUG: FileCurl::Close(086EF130) http://www.thetvdb.com/api/GetSeries.php?seriesname=To%20Heart%202%20OVA
13:47:14 T:3236 M:2237886464 DEBUG: scraper: GetFanartAPI returned <details><url function="GetTVDBLookupLoopAPI" cache="4806.xml"foo="">&</url></details>
13:47:14 T:3236 M:2237853696 DEBUG: scraper: GetTVDBLookupLoopAPI returned <details foo=""><url function="GetFanartDataReturnAPI" cache="4806.xml">&</url></details>
13:47:14 T:3236 M:2237853696 DEBUG: scraper: GetFanartDataReturnAPI returned <details></details>
failed lookup
Quote:13:49:44 T:2680 M:2241540096 DEBUG: scraper: GetTVDBLookupAPI returned <details><url function="GetFanartAPI" cache="tvdb-s5592-To%20Heart%202%20ad.xml"foo="4806"bar="To Heart 2 ad">http://www.thetvdb.com/api/GetSeries.php?seriesname=To%20Heart%202%20ad</url></details>
13:49:44 T:2680 M:2241536000 DEBUG: scraper: GetFanartAPI returned <details><url function="GetTVDBLookupLoopAPI" cache="5592.xml"foo="4806">&</url></details>
13:49:44 T:2680 M:2241536000 DEBUG: scraper: GetTVDBLookupLoopAPI returned <details foo=""><url function="GetTVDBLookupAPI" gzip="yes" cache="4806.xml">http://api.anidb.net:9001/httpapi?request=anime&client=xbmcscrap&clientver=1&protover=1&aid=4806</url></details>
13:49:44 T:2680 M:2241503232 DEBUG: scraper: GetTVDBLookupAPI returned <details><url function="GetFanartAPI" cache="tvdb-s4806-To%20Heart%202%20ad.xml"foo="4806"bar="To Heart 2 ad">http://www.thetvdb.com/api/GetSeries.php?seriesname=To%20Heart%202%20ad</url></details>
13:49:44 T:2680 M:2241499136 DEBUG: scraper: GetFanartAPI returned <details><url function="GetTVDBLookupLoopAPI" cache="4806.xml"foo="4806">&</url></details>
13:49:44 T:2680 M:2241499136 DEBUG: scraper: GetTVDBLookupLoopAPI returned <details foo=""><url function="GetTVDBLookupAPI" gzip="yes" cache="4806.xml">http://api.anidb.net:9001/httpapi?request=anime&client=xbmcscrap&clientver=1&protover=1&aid=4806</url></details>
13:49:44 T:2680 M:2241499136 DEBUG: scraper: GetTVDBLookupAPI returned <details><url function="GetFanartAPI" cache="tvdb-s4806-To%20Heart%202%20ad.xml"foo="4806"bar="To Heart 2 ad">http://www.thetvdb.com/api/GetSeries.php?seriesname=To%20Heart%202%20ad</url></details>
13:49:44 T:2680 M:2241499136 DEBUG: scraper: GetFanartAPI returned <details><url function="GetTVDBLookupLoopAPI" cache="4806.xml"foo="4806">&</url></details>
13:49:44 T:2680 M:2241499136 DEBUG: scraper: GetTVDBLookupLoopAPI returned <details foo=""><url function="GetTVDBLookupAPI" gzip="yes" cache="4806.xml">http://api.anidb.net:9001/httpapi?request=anime&client=xbmcscrap&clientver=1&protover=1&aid=4806</url></details>
13:49:44 T:2680 M:2241499136 DEBUG: scraper: GetTVDBLookupAPI returned <details><url function="GetFanartAPI" cache="tvdb-s4806-To%20Heart%202%20ad.xml"foo="4806"bar="To Heart 2 ad">http://www.thetvdb.com/api/GetSeries.php?seriesname=To%20Heart%202%20ad</url></details>
....
notice how foo and bar are related to the show being looked up even when we are looking up the prequel.
- bambi73 - 2011-03-27 17:58
Your observations are correct, there is bug in XBMC (at least from my point of view :oo
. I posted some description for developers here.I'll try to add some workaround to scraper and make version 1.0.1 ... after almost 3/4 year
- pathw - 2011-03-27 19:33
oh so that's the problem? do you think we should eliminate the cache calls in that area?
- bambi73 - 2011-03-27 20:14
Turning cache off anywhere is bad idea, you will get banned on AniDB.net in no time. It's enough to clear $$1 at the end of each function, so new content from site/cache is appended to empty string.
If you want to test my solution you can try v1.0.1b1. I'll welcome to see your test results.
- salival - 2011-03-27 22:13
Hi Bambi,
Some time ago I believe I sent you a PM. It was about the animelist.xml
You can get the file from anidb.net by using the following url: http://anidb.net.nyud.net/api/animetitles.xml
This routes it through a cdn. I contacted the anidb staff to check if this was allowed, which it was. (here's the thread)
If you find yourself lacking time to maintain the anidb.xml on google sites you could substitute the url for the one above.
- bambi73 - 2011-03-28 17:30
salival Wrote:Hi Bambi,Hello,
Some time ago I believe I sent you a PM. It was about the animelist.xml
You can get the file from anidb.net by using the following url: http://anidb.net.nyud.net/api/animetitles.xml
This routes it through a cdn. I contacted the anidb staff to check if this was allowed, which it was. (here's the thread)
If you find yourself lacking time to maintain the anidb.xml on google sites you could substitute the url for the one above.
sorry for not responding to your PMs, i was quite busy with my personal life and work in past 3/4 years and had almost no time to spend on XBMC related work.
About Coral CDN, i'll check its potential.
- salival - 2011-03-28 19:28
No need to apologize. I just took the opportunity while you're here.