+- XBMC Community Forum (http://forum.xbmc.org)
+-- Forum: Help and Support (/forumdisplay.php?fid=33)
+--- Forum: Add-ons Help and Support (/forumdisplay.php?fid=27)
+---- Forum: Metadata scrapers (/forumdisplay.php?fid=147)
+---- Thread: [WIP] AniDB.net Anime Video Scraper (/showthread.php?tid=64587)
RE: [WIP] AniDB.net Anime Video Scraper - Curufinwe - 2012-03-26 22:27
Hi!
I am running into trouble when adding new anime episodes recently. This happens when adding a new series as well when adding episodes from a series where the adding of episodes previously worked. Here is some log information when trying to refresh an episode that was earlier added successfully but now no longer is:
Code:
22:10:28 T:4720 M:1130360832 DEBUG: ------ Window Deinit (DialogVideoInfo.xml) ------
22:10:28 T:4720 M:1130360832 DEBUG: VideoInfoScanner: No NFO file found. Using title search for 'x:\xx\xxx\xxxS01E01.mkv'
22:10:28 T:4720 M:1130360832 INFO: Loading skin file: DialogProgress.xml
22:10:28 T:4720 M:1130356736 DEBUG: ------ Window Init (DialogProgress.xml) ------
22:10:28 T:4720 M:1130237952 DEBUG: CVideoDatabase::GetEpisodeId (x:\xx\xxx\xxxS01E01.mkv), query = select idEpisode from episode where idFile=1176
22:10:28 T:4720 M:1130229760 DEBUG: VideoInfoScanner: Found episode match x:\xx\xxx\xxxS01E01.mkv (s1e1) [[Ss]([0-9]+)[][._-]*[Ee]([0-9]+)([^\\/]*)$]
22:10:28 T:4720 M:1130229760 DEBUG: scraper: GetEpisodeList returned <episodeguide></episodeguide>
22:10:28 T:4720 M:1130221568 DEBUG: CVideoDatabase::GetEpisodeId (x:\xx\xxx\xxxS01E01.mkv), query = select idEpisode from episode where idFile=1176
22:10:28 T:4720 M:1130213376 DEBUG: VideoInfoScanner: No NFO file found. Using title search for 'x:\xx\xxx\xxxS01E01.mkv'
22:10:28 T:4720 M:1130213376 ERROR: VideoInfoScanner: Asked to lookup episode x:\xx\xxx\xxxS01E01.mkv online, but we have no episode guide. Check your tvshow.nfo and make sure the <episodeguide> tag is in place.
22:10:28 T:4720 M:1130213376 DEBUG: CVideoDatabase::GetStackedTvShowList query: select idShow from tvshow where c00 like (select c00 from tvshow where idShow=144) order by idShow
22:10:28 T:4720 M:1130213376 DEBUG: CVideoDatabase::GetSeasonsNav query: select episode.c12,path.strPath,tvshow.c00,tvshow.c08,tvshow.c14,tvshow.c13,count(1),count(files.playCount) from episode join tvshowlinkepisode on tvshowlinkepisode.idEpisode=episode.idEpisode join tvshow on tvshow.idShow=tvshowlinkepisode.idShow join files on files.idFile=episode.idFile join tvshowlinkpath on tvshowlinkpath.idShow = tvshow.idShow join path on path.idPath = tvshowlinkpath.idPath where tvshow.idShow IN (143,144) group by episode.c12
22:10:28 T:4720 M:1130213376 INFO: Caching image from: to special://masterprofile/Thumbnails/Video/8/8da5fa75.tbn with width 512 and height 512
22:10:28 T:4720 M:1130213376 DEBUG: Win32DllLoader::Load(special://xbmcbin/system/ImageLib.dll)
22:10:28 T:4720 M:1130012672 DEBUG: msg: PICTURE::CreateThumbnail: Unable to open image: Error:
22:10:28 T:4720 M:1130012672 ERROR: CPicture::CacheImage Unable to create new image special://masterprofile/Thumbnails/Video/8/8da5fa75.tbn from image
22:10:28 T:4720 M:1130008576 DEBUG: CVideoDatabase::GetEpisodeId (x:\xx\xxx\xxxS01E01.mkv), query = select idEpisode from episode where idFile=1176
22:10:28 T:4720 M:1129947136 DEBUG: ------ Window Init (DialogVideoInfo.xml) ------
22:10:28 T:4720 M:1129947136 INFO: Loading skin file: DialogVideoInfo.xml
22:10:28 T:4704 M:1129910272 DEBUG: thread start, auto delete: 1
22:10:28 T:4704 M:1129910272 DEBUG: CThumbExtractor::DoWork - trying to extract filestream details from video file x:\xx\xxx\xxxS01E01.mkv
22:10:28 T:4704 M:1129353216 DEBUG: CTextureCache::GetImageHash - unable to stat url x:\xx\xxx\xxx\folder.jpg
22:10:29 T:4720 M:1129664512 DEBUG: ------ Window Deinit (DialogProgress.xml) ------
22:10:30 T:4720 M:1129877504 DEBUG: SDLKeyboard: scancode: 56, sym: 308, unicode: 0, modifier: 0
22:10:30 T:4720 M:1129881600 DEBUG: CWinEventsWin32::WndProcWindow is active
22:10:30 T:4720 M:1129881600 DEBUG: CWinEventsWin32::WndProcWindow lost focus
22:10:34 T:5124 M:1129881600 ERROR: XFILE::CFileRar::Seek - Timeout waiting for buffer to empty
22:10:34 T:5124 M:1129881600 ERROR: CDVDDemuxFFmpeg::Open - error reading from input stream, rar://yyy.mkv
22:10:34 T:5488 M:1129881600 DEBUG: Thread 5488 terminating
22:10:34 T:5124 M:1130000384 ERROR: CDVDFileInfo::ExtractThumb - Error creating demuxer
22:10:34 T:5124 M:1130000384 DEBUG: CThumbExtractor::DoWork - trying to extract thumb from video file x:\xx\xxx\xxxS01E03.mkv
22:10:39 T:4720 M:1130037248 DEBUG: CWinEventsWin32::WndProcWindow is active
22:10:39 T:4720 M:1130037248 DEBUG: CWinEventsWin32::WndProcWindow gained focus
22:10:40 T:4720 M:1137393664 DEBUG: SDLKeyboard: scancode: 56, sym: 308, unicode: 0, modifier: 0
22:10:40 T:4720 M:1137418240 DEBUG: CWinEventsWin32::WndProcWindow is active
22:10:40 T:4720 M:1137418240 DEBUG: CWinEventsWin32::WndProcWindow lost focusI do not know if this is enough or if additional information is needed. It seems the episode is identified but somehow no related information is found. When using the TVDB scraper the series seems to be identified correctly.
RE: [WIP] AniDB.net Anime Video Scraper - Curufinwe - 2012-03-28 00:26
Never mind, it is working again now.
RE: [WIP] AniDB.net Anime Video Scraper - bambi73 - 2012-03-29 12:49
(2012-03-15 13:40)shinner Wrote: Having trouble scraping episode info for some newer anime though (eg. Another). I pinpointed the problem to anime-list.xml, it's not mapped in there. Am I right that this file has not been updated for a while, like since late 2011?I was completelly busy with my work and real life for last 4-5 months so all my XBMC related projects was stoped, i was using it only for occasional watching
So I'll have to use my own personal anime mapping? (It also works if I edit the xml in the cache, but then it'll get overwritten)
. I hope it'll be better in following months, so I plan to update anime-list.xml with newest Animes (btw if you already created your own personal list please send it to me by PM, i'll use it as base for my updates), but can't promise when it'll be done 
.Anyway I'm not sure if anime-list.xml is source of your problems. It's only mapping between anidb.net and thetvdb.com, so it affect only loading informations from thetvdb.com, like background images or episode plots, rest informations from anidb.net should be there.
(2012-03-15 13:40)shinner Wrote: Edit:When I added this feature to scraper I tested it with XBMC web server and it was working fine, but don't ask me details, it was 1,5y ago
If anyone has used personal anime mapping before, I could use a little help. If I enable the XBMC web server, can I place the pers-anime-list.xml locally? If so, where do I have to place it and what's the url? (http://localhost. something) And do I have to disable the default xml file?
RE: [WIP] AniDB.net Anime Video Scraper - lad1337 - 2012-03-29 13:13
i just wanted to let you guys know that i am working on a project with a few others to takle this problem once and for all
i am talking about http://thexem.de/
it is ment to map every episode on one entity with the appropriate one on any other entity
for now there is only a proxy service for the tvdb api but i am personally interested to do the same for anidb
what the proxy service does: you give a origin(e.g. tvdb) and a destination(e.g. anidb) then use the api logic of the origin get a response just like the origin would give you but with the numbering of the destination...
we are still in a closed testing phase but its looking good for a tvdb->scene relation already
but you can already get every map with a json api
what i could do is implement a auto generated anime-list.xml based on the info provided in xem
my question is would this be something of use or would you rather prefer something else to improve the scraper
or would you accept/wish another xml structure or json? could you even use a non local xml for the anime-list.xml ?
if not how could xem provide a anidb_id<->tvdb_id mapping in a way that this scraper would benefit ?
lad1337
RE: [WIP] AniDB.net Anime Video Scraper - shinner - 2012-03-31 12:56
(2012-03-29 12:49)bambi73 Wrote: I hope it'll be better in following months, so I plan to update anime-list.xml with newest Animes (btw if you already created your own personal list please send it to me by PM, i'll use it as base for my updates), but can't promise when it'll be done
Anyway I'm not sure if anime-list.xml is source of your problems. It's only mapping between anidb.net and thetvdb.com, so it affect only loading informations from thetvdb.com, like background images or episode plots, rest informations from anidb.net should be there.
My bad, I should've been clearer in explaining. Yup, the series, actors etc. scrape ok, it's just the episode plots are empty. I only just added one anime in manually to test the issue, so no point sending it to you, sorry ^^' Thanks for your work!
Can I ask how you generated your anime-list.xml? Did you write everything manually, or there's a way to automate it by parsing the files returned by tvdb and anidb API with a script? Sorry if I don't make sense, I don't really get the hang of how the scraper works yet, so help is really appreciated.
(2012-03-29 12:49)bambi73 Wrote:I'll have to do some trial and error when I'm free then. I'll update you with the settings when I can get it to work XP(2012-03-15 13:40)shinner Wrote: Edit:When I added this feature to scraper I tested it with XBMC web server and it was working fine, but don't ask me details, it was 1,5y ago
If anyone has used personal anime mapping before, I could use a little help. If I enable the XBMC web server, can I place the pers-anime-list.xml locally? If so, where do I have to place it and what's the url? (http://localhost. something) And do I have to disable the default xml file?
@lad1337 Not a programmer, so can't really answer your questions, but that sounds really promising in simplifying episode mapping.
RE: [WIP] AniDB.net Anime Video Scraper - bambi73 - 2012-03-31 16:40
@shinner: I have modified version of scraper which outputs results (in XML form) into debug log. But putting it togerther is manual work.
@lad1337: I'll check your site again when it's populated with data and will see if it can be used in scraper in some way. Anyway it can't be full-fledged replacement for anime-list.xml because it contains some additional informations or corrections for existing data.
RE: [WIP] AniDB.net Anime Video Scraper - wentzelfcp - 2012-03-31 16:59
Quote:I'll explain the basics, I really am short on time to solve the problem totally >.<. Do you know how to read XML files? The file in question is
C:\Users\<your user name>\AppData\Roaming\XBMC\cache\scrapers\metadata.anidb.net\anime-list.xml
Basically what the anidb scraper does is it gets some data from Anidb, then it maps the episode to TVDB and gets episode plot info from there. The mapping is done in anime-list.xml, eg.
<anime anidbid="6239" tvdbid="88651" defaulttvdbseason="1">
maps anidb ID 6239 to TVDB ID 88651. It's the same show (07-Ghost in this case).
If you know how to, you can copy one of the animes, modify the name, anidbid and tvdbid (get those from the sites yourself). Then scrape. It's more complicated for multi-season anime, so I'm not going to talk about it for now.
Do a backup of your edited file, because the cache will be cleared after a while. The personal anime mapping setting of the addon is supposed to work-around this, but I can't get it working yet
hi can someone please help me understand what i have to do here? i dont really know what i must do
. Im trying to scrape episode info for shows like one piece, naruto shippuuden, another... any help, would be much appreciated
RE: [WIP] AniDB.net Anime Video Scraper - Reser - 2012-04-01 23:33
(2012-03-13 17:32)shinner Wrote: Your anime names are originally absolute numbering right? eg. Naruto 305, not S15E02 or something? If so, try to use regex, it's much less trouble.
Now, create an advancedsettings.xml file, and add this in:
Edit the "Anime" in the code above to the name of your source folder. This regex should handle multi-episode files.Code:
<advancedsettings>
<tvshowmatching action = "prepend">
<regexp>Anime[\\/].*[/\._ \-]()([0-9]+)(-[0-9]+)?</regexp>
</tvshowmatching>
</advancedsettings>
Thanks to gates' post.
Ok, I am trying to get my AniDB scraper working, and I have down the above for my advanced settings.
I have also named half my anime using the anidb own file renamer script.
An example of how my files are named now is : Code Geass Hangyaku no Lelouch - 04 - His Name is Zero - [gg](49715425)
An Example of how My files used to be named ( and work with TVDB) is : K-ON!.S01E08.Freshman Reception!
Now with the new Regex, Code Geass for example with the new episode name setup does not show any episodes at all, all my anime wit the old setup ( S00e00) show up the episodes but with about 10 of the same episode ove rand over.
Why is this not working?
Also another problem I am having is it is refusing to use Banners, no matter how many times I tell it to in the settings. And TVDB does have the banners as before I changed to AniDB they all had banners perfectly.
I have noticed the banners options keeps deactivating in anidb options and I have to keep re-enabling it. It just does not want to stay on or get any banners...
Only anime it has got a banner for is Code Geass, which is also the one anime I am trying the new name setup on and is showing 0 episodes.
RE: [WIP] AniDB.net Anime Video Scraper - Reser - 2012-04-02 21:55
I finally got it working by using Regex of
<regexp> - ([0-9]+)</regexp>
However anidb plugin even with banners enabled in settings still REFUSES to download banners, and I am left with Large Posters SQUASHED into banner size.
Really driving me nuts now..
RE: [WIP] AniDB.net Anime Video Scraper - bambi73 - 2012-04-02 23:22
@Reser: When you enable banners it doesn't mean they will be default automatically. If banner exist for anime it's only added as another possibility, so you can choose it from Info window. Problem with banners is that they don't exists for each anime, so I did it working this way. Additionally default poster is always from anidb.net and make default something from thetvdb.com, like posters, will make scraper handling even more complicated than it's now.