+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Information Providers (scrapers) (https://forum.kodi.tv/forumdisplay.php?fid=147)
+----- Forum: TV Show Scrapers (https://forum.kodi.tv/forumdisplay.php?fid=305)
+----- Thread: [WIP] AniDB.net Anime Video Scraper (/showthread.php?tid=64587)
- bambi73 - 2011-11-08
@hentai23, @ZERO<ibis>: Best what you can do is turn debug logging on (check wiki if you don't know how) and check it yourself. XBMC scraper engine logs almost eveything. If you don't find anything strange in logs you can post it there (over pastebin), someone will look at it.
@ZERO<ibis>: Maybe you should check your anidb.net scraper cache, yesterday http API server was online but returned Internal server error in all xmls. Files in cache cleared after 48hr, so maybe it's reason why you can scraper some series and some not.
- hentai23 - 2011-11-08
sure as hell ill wait its unrivaled in its anime capabilities ANN doesnt even come close -.-'
debugging wont do anything in this situation
anyhow here youre debug log
http://pastebin.com/8gi3mMCg
its an anidb prob
- bambi73 - 2011-11-08
Even when there are no real errors in log it doesn't mean it's ok. For example following line is completely wrong
Code:
10:03:08 T:5372 DEBUG: scraper: GetDetailsExt returned <details><id></id><plot></plot><episodeguide><url gzip="yes" cache=".xml">http://api.anidb.net:9001/httpapi?request=anime&client=xbmcscrap&clientver=1&protover=1&aid=</url></episodeguide><url function="GetTVDBLookupAPI" cache=".xml">&</url></details>id empty, plot empty, everything else missing. This happen when you can connect to anidb.net http api, but it returns "empty" responses (for example if you are banned or site return Internal server error response like in past days). Of course these responses are hold in cache like regular ones and are reused (for 48hrs) when you ask for same anime again and IMHO it's your problem. Try to clear anidb.net craper cache and try it again. I tried to scrape 2 series now and it works fine again.
- hentai23 - 2011-11-08
hmmm one sec there how do i clear the cache ?
ok found out i got banned ...how long it take to get unbanned? xD
how in hell did i get banned anyhow i only scrape theyre fn database for info -.-'
- salival - 2011-11-08
could you try this link?
It's the url the scraper uses to call the anime info. If the link returns an xml list with the information you're not banned and if it returns an empty file you're banned.
Most bans last only a few minutes (5 to 30) most of the time and occur when you try to request information from anidb for a lot of anime at the same time, for example when you try to scrape a collection for the first time.
Anidb does this to prevent the server from overloading.
edit: to clear your cache go to your userfolder and then cache\scrapers\metadata.anidb.net and remove all files in there.
- bambi73 - 2011-11-08
As salival said, try that URL and if you get something like (don't remember exact form)
Code:
<error>You are banned</error>then you are banned, otherwise you will get long xml file with anime info. But i'm not so sure about that short unbanning time, IMHO it's more like few hours.
And how he said, they are defending server agains too frequent requests. Unfortunatelly there is now way how to tell XBMC to insert some delay between single scrapes, so it sometimes happen. Just be patient and wait
- hentai23 - 2011-11-09
no ive checked the cache .xml file said i was banned ...<error>You are banned</error>....and this has been goin on for the last couple of days , so once you get banned xml in youre cache you always gotta delete it first then wait about 1 day and then retry ? ..since ive had this since the last couple of days thou i asume if i dont delete the banned xml i get rebanned?....until i delete the banned xml and wait as said?......-.-'
....how in hell can you get banned this is seriously ridiculous they need a server upgrade if its this bad ..... -.-'
p.s: its not like its a problem i can simply switch my ip i got enough since i use a vpn -.-'...thou this shouldnt even be happening
- Finalspace - 2011-11-24
hentai23 Wrote:no ive checked the cache .xml file said i was banned ...<error>You are banned</error>....and this has been goin on for the last couple of days , so once you get banned xml in youre cache you always gotta delete it first then wait about 1 day and then retry ? ..since ive had this since the last couple of days thou i asume if i dont delete the banned xml i get rebanned?....until i delete the banned xml and wait as said?......-.-'
....how in hell can you get banned this is seriously ridiculous they need a server upgrade if its this bad ..... -.-'
p.s: its not like its a problem i can simply switch my ip i got enough since i use a vpn -.-'...thou this shouldnt even be happening
Unfortunatly anidb have really hard restrictions for the HTTP and UDP api.
There is currently no way to fix the problem in the scraper itself
Maybe by using a sleep in the scraper, for each result, to ensure for giving enough time for the api restriction checks would fix this?
Or making a simple php webservice, which encloses the anidb api calls to enforce sleep intervals or something like that.
- salival - 2011-11-24
@hentai23:
You indeed have to delete you the cache. Else XBMC will look at the cache folder, sees there already is a file and reads it. But of course it is unable to get any data from it.
So to make sure XBMC will not use the empty files in cache you should empty the scraper cache.
You also will not get banned again if there's an xml file with the "you are banned" info in your cache. Only XBMC will not look any further than that file if the cache hasn't expired.
- bambi73 - 2011-11-24
Finalspace Wrote:Unfortunatly anidb have really hard restrictions for the HTTP and UDP api.There is no such thing as sleep in scraper, scraper itself has no way to affect scraping process. Personally i hardcoded 10sec between starts of each scrape, but it's rough solution not suited for posting patch.
There is currently no way to fix the problem in the scraper itself
Maybe by using a sleep in the scraper, for each result, to ensure for giving enough time for the api restriction checks would fix this?
Finalspace Wrote:Or making a simple php webservice, which encloses the anidb api calls to enforce sleep intervals or something like that.Maybe, but i don't know how and don't feel any need of it. Of course you can prepare something yourself, changing scaper should be easy then.
- hentai23 - 2011-11-25
exactly as said clear the cache and wait for a couple days then retry and should work
but for a quick fix just change youre ip and you can scrape once again problem free no biggy if you are using a paid vpn xD
or perhaps someone could integrate a better solution cause i can imagine this pissing off quite a few anime lovers that have more than 10 anime to scrape and ..bam !! ban -.-'
making them have to wait for 1 or 2 days to be able to rescrape ..not funny at all i would say -.-'
- salival - 2011-11-25
Yeah, it's not ideal.
But then again you will only encounter this with the initial scan. I haven't had any problems with my daily use.
- ZERO <ibis> - 2011-11-30
hentai23 Wrote:exactly as said clear the cache and wait for a couple days then retry and should work
but for a quick fix just change youre ip and you can scrape once again problem free no biggy if you are using a paid vpn xD
or perhaps someone could integrate a better solution cause i can imagine this pissing off quite a few anime lovers that have more than 10 anime to scrape and ..bam !! ban -.-'
making them have to wait for 1 or 2 days to be able to rescrape ..not funny at all i would say -.-'
That is why you scrap manually only. Well at least that is the approach I took, besides this way you can quality check the process to ensure everything imported correctly.
On another note, I have a request:
Can we get character support? Like for example there already is a "cast" that lists the voice actors but how about having a list of characters that we can search to find other series they appeared in? It could also give a description of them from the anidb database. Second this would allow a character image to go with the cast listing to make it easier for people that are just bad with names.
What would be nice is a way to ultimately be able to select a character and see not only the other series that character was in but also for the voice side be able to see some pics of the characters a given actor has played.
- bambi73 - 2011-12-03
@ZERO <ibis>: Only character list supported by XBMC is cast list. When I added cast list to scraper I was thinking about some kind of switch between either voice actor or character list, but found following problems:
1/ Voice actors pictures has somehow consistent aspect ratio, but same can't be said about characters pictures. They goes from 16:9 to 1:5 which looks really bad in XBMC.
2/ Most skins shows cast list as for example "Saiga Mitsuki as Tsukasa" and i found possible options "Tsukasa as Tsukasa" or "Tsukasa as Saiga Mitsuki" for character list as bad.
Character description isn't part of xml returned by http api.
Requested features can be done (maybe) by some kind of addon, not scraper, it's posibilities are limited.
- hentai23 - 2011-12-07
ZERO <ibis>' Wrote:That is why you scrap manually only. Well at least that is the approach I took, besides this way you can quality check the process to ensure everything imported correctly.
On another note, I have a request:
Can we get character support? Like for example there already is a "cast" that lists the voice actors but how about having a list of characters that we can search to find other series they appeared in? It could also give a description of them from the anidb database. Second this would allow a character image to go with the cast listing to make it easier for people that are just bad with names.
What would be nice is a way to ultimately be able to select a character and see not only the other series that character was in but also for the voice side be able to see some pics of the characters a given actor has played.
manual scrape every single anime? what madness is this? you dont know what to do with youre time it would seem , hell i betcha you manual scrape youre 1000´s of movies too then ? id rather spend my life on watching movies instead of having a 100% perfect library and never getting to watch every movie in my library due to manually scrapping and sorting my collection 8 hours a day xD,wich btw mine is already at 95% perfection WITHOUT manual scraping lolz
only manual scrape i do is if i have games i add to advanced launcher or a couple of anime that wont scrape without removing a few japanese words wich then scrapes the correct title but otherwise HELL NO!!!
....
and how in our society nowadays where you hardly have time to take a crap let alone alot of free time with work ,family and all ,life is too short to waste it -.-'