[WIP] AniDB.net Anime Video Scraper

  Thread Rating:
  • 3 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
wentzelfcp Offline
Junior Member
Posts: 15
Joined: Mar 2012
Reputation: 0
Post: #391
i cant beleive this, your shiner, you have a video demo of your xbmc setup on youtube? dont you? hahaha im Usumario, you replied to one of my comments a few days ago, omg , how the internet world is small.
Moving on, my setting goes:

TVshows
anime show folder
anime season folder
anime show: xxxxxx.S01E01

what should i do next? Big Grin

P.S i was going to subscribe to your channel, but a message appeared,saying that it no longer existed.
find quote
wentzelfcp Offline
Junior Member
Posts: 15
Joined: Mar 2012
Reputation: 0
Post: #392
i have already started to rename my anime series, just like you told me, but i still dont know waht to copy paste? oh and btw, does anidb scrape fanart or posters?
find quote
shinner Offline
Member
Posts: 52
Joined: Jan 2012
Reputation: 2
Post: #393
@wentzelfcp: haha, yes indeed, I put a demo video up, because I didn't see many people separating their anime from other TV shows Wink

Quote:P.S i was going to subscribe to your channel, but a message appeared,saying that it no longer existed.
I have no idea about this, could be something about my youtube setting :\ Currently no plan to add videos, but if I do I'll pass you a link Wink Maybe one detailed one on adding anime from start to end if I get the time...

On topic, your anime are showing up in XBMC now? If so, you can start working on getting the episodes to show up. There's no need to rename your files if you use custom regex.

And yes,fanart and posters are scraped (they come from tvdb).
find quote
wentzelfcp Offline
Junior Member
Posts: 15
Joined: Mar 2012
Reputation: 0
Post: #394
yes i tested it with a single folder, and it recognized, but now im already half way trough renaming them,(sheez) . what do you suggest? should i leave it like this , because, my one piece folder has 15 seasons. btw some of my naruto episodes, consist in 2 episodes for file,pex 47-48, what do i do to scrape them? what do i copy paste for regex?

thx
find quote
shinner Offline
Member
Posts: 52
Joined: Jan 2012
Reputation: 2
Post: #395
Your anime names are originally absolute numbering right? eg. Naruto 305, not S15E02 or something? If so, try to use regex, it's much less trouble.
Now, create an advancedsettings.xml file, and add this in:

Code:
<advancedsettings>
    <tvshowmatching action = "prepend">
        <regexp>Anime[\\/].*[/\._ \-]()([0-9]+)(-[0-9]+)?</regexp>
     </tvshowmatching>
</advancedsettings>
Edit the "Anime" in the code above to the name of your source folder. This regex should handle multi-episode files.

Thanks to gates' post.
(This post was last modified: 2012-03-13 17:35 by shinner.)
find quote
wentzelfcp Offline
Junior Member
Posts: 15
Joined: Mar 2012
Reputation: 0
Post: #396
is that all i need to do?

thanks again , and sorry if i was a little impatient.Tongue
find quote
shinner Offline
Member
Posts: 52
Joined: Jan 2012
Reputation: 2
Post: #397
Yup, haha. Good luck and enjoy your anime Wink
We really need a beginner's guide to adding anime, the wiki's are too wall-of-text-y for my taste xp
find quote
wentzelfcp Offline
Junior Member
Posts: 15
Joined: Mar 2012
Reputation: 0
Post: #398
Quote:Edit the "Anime" in the code above to the name of your source folder. This regex should handle multi-episode files.

what do i edit the anime part? or the one with the brackets and backspaces? what do i need to change? because i tried and i didnt workFrown
(This post was last modified: 2012-03-13 17:52 by wentzelfcp.)
find quote
shinner Offline
Member
Posts: 52
Joined: Jan 2012
Reputation: 2
Post: #399
Quote:what do i edit the anime part? or the one with the brackets and backspaces? what do i need to change? because i tried and i didnt workFrown
Do not touch the backslashes or brackets! "Anime" is my folder name, just change it. eg. if your folder is called TV Shows, change the stuff between the <regexp></regexp> tags to
Code:
TV Shows[\\/].*[/\._ \-]()([0-9]+)(-[0-9]+)?
(This post was last modified: 2012-03-15 13:41 by shinner.)
find quote
shinner Offline
Member
Posts: 52
Joined: Jan 2012
Reputation: 2
Post: #400
wentzelfcp's issue has been solved via pm (just following the steps listed in the previous posts, really).

Having trouble scraping episode info for some newer anime though (eg. Another). I pinpointed the problem to anime-list.xml, it's not mapped in there. Am I right that this file has not been updated for a while, like since late 2011?
So I'll have to use my own personal anime mapping? (It also works if I edit the xml in the cache, but then it'll get overwritten Tongue)

Edit:
If anyone has used personal anime mapping before, I could use a little help. If I enable the XBMC web server, can I place the pers-anime-list.xml locally? If so, where do I have to place it and what's the url? (http://localhost. something) And do I have to disable the default xml file?
(This post was last modified: 2012-03-18 12:20 by shinner.)
find quote
Curufinwe Offline
Junior Member
Posts: 2
Joined: Mar 2012
Reputation: 0
Post: #401
Hi!

I am running into trouble when adding new anime episodes recently. This happens when adding a new series as well when adding episodes from a series where the adding of episodes previously worked. Here is some log information when trying to refresh an episode that was earlier added successfully but now no longer is:


Code:
22:10:28 T:4720 M:1130360832   DEBUG: ------ Window Deinit (DialogVideoInfo.xml) ------
22:10:28 T:4720 M:1130360832   DEBUG: VideoInfoScanner: No NFO file found. Using title search for 'x:\xx\xxx\xxxS01E01.mkv'
22:10:28 T:4720 M:1130360832    INFO: Loading skin file: DialogProgress.xml
22:10:28 T:4720 M:1130356736   DEBUG: ------ Window Init (DialogProgress.xml) ------
22:10:28 T:4720 M:1130237952   DEBUG: CVideoDatabase::GetEpisodeId (x:\xx\xxx\xxxS01E01.mkv), query = select idEpisode from episode where idFile=1176
22:10:28 T:4720 M:1130229760   DEBUG: VideoInfoScanner: Found episode match x:\xx\xxx\xxxS01E01.mkv (s1e1) [[Ss]([0-9]+)[][._-]*[Ee]([0-9]+)([^\\/]*)$]
22:10:28 T:4720 M:1130229760   DEBUG: scraper: GetEpisodeList returned <episodeguide></episodeguide>
22:10:28 T:4720 M:1130221568   DEBUG: CVideoDatabase::GetEpisodeId (x:\xx\xxx\xxxS01E01.mkv), query = select idEpisode from episode where idFile=1176
22:10:28 T:4720 M:1130213376   DEBUG: VideoInfoScanner: No NFO file found. Using title search for 'x:\xx\xxx\xxxS01E01.mkv'
22:10:28 T:4720 M:1130213376   ERROR: VideoInfoScanner: Asked to lookup episode x:\xx\xxx\xxxS01E01.mkv online, but we have no episode guide. Check your tvshow.nfo and make sure the <episodeguide> tag is in place.
22:10:28 T:4720 M:1130213376   DEBUG: CVideoDatabase::GetStackedTvShowList query: select idShow from tvshow where c00 like (select c00 from tvshow where idShow=144) order by idShow
22:10:28 T:4720 M:1130213376   DEBUG: CVideoDatabase::GetSeasonsNav query: select episode.c12,path.strPath,tvshow.c00,tvshow.c08,tvshow.c14,tvshow.c13,count(1),co​unt(files.playCount) from episode join tvshowlinkepisode on tvshowlinkepisode.idEpisode=episode.idEpisode join tvshow on tvshow.idShow=tvshowlinkepisode.idShow join files on files.idFile=episode.idFile  join tvshowlinkpath on tvshowlinkpath.idShow = tvshow.idShow join path on path.idPath = tvshowlinkpath.idPath where tvshow.idShow IN (143,144)  group by episode.c12
22:10:28 T:4720 M:1130213376    INFO: Caching image from:  to special://masterprofile/Thumbnails/Video/8/8da5fa75.tbn with width 512 and height 512
22:10:28 T:4720 M:1130213376   DEBUG: Win32DllLoader::Load(special://xbmcbin/system/ImageLib.dll)
22:10:28 T:4720 M:1130012672   DEBUG:   msg: PICTURE::CreateThumbnail: Unable to open image:  Error:
22:10:28 T:4720 M:1130012672   ERROR: CPicture::CacheImage Unable to create new image special://masterprofile/Thumbnails/Video/8/8da5fa75.tbn from image
22:10:28 T:4720 M:1130008576   DEBUG: CVideoDatabase::GetEpisodeId (x:\xx\xxx\xxxS01E01.mkv), query = select idEpisode from episode where idFile=1176
22:10:28 T:4720 M:1129947136   DEBUG: ------ Window Init (DialogVideoInfo.xml) ------
22:10:28 T:4720 M:1129947136    INFO: Loading skin file: DialogVideoInfo.xml
22:10:28 T:4704 M:1129910272   DEBUG: thread start, auto delete: 1
22:10:28 T:4704 M:1129910272   DEBUG: CThumbExtractor::DoWork - trying to extract filestream details from video file x:\xx\xxx\xxxS01E01.mkv
22:10:28 T:4704 M:1129353216   DEBUG: CTextureCache::GetImageHash - unable to stat url x:\xx\xxx\xxx\folder.jpg
22:10:29 T:4720 M:1129664512   DEBUG: ------ Window Deinit (DialogProgress.xml) ------
22:10:30 T:4720 M:1129877504   DEBUG: SDLKeyboard: scancode: 56, sym: 308, unicode: 0, modifier: 0
22:10:30 T:4720 M:1129881600   DEBUG: CWinEventsWin32::WndProcWindow is active
22:10:30 T:4720 M:1129881600   DEBUG: CWinEventsWin32::WndProcWindow lost focus
22:10:34 T:5124 M:1129881600   ERROR: XFILE::CFileRar::Seek - Timeout waiting for buffer to empty
22:10:34 T:5124 M:1129881600   ERROR: CDVDDemuxFFmpeg::Open - error reading from input stream, rar://yyy.mkv
22:10:34 T:5488 M:1129881600   DEBUG: Thread 5488 terminating
22:10:34 T:5124 M:1130000384   ERROR: CDVDFileInfo::ExtractThumb - Error creating demuxer
22:10:34 T:5124 M:1130000384   DEBUG: CThumbExtractor::DoWork - trying to extract thumb from video file x:\xx\xxx\xxxS01E03.mkv
22:10:39 T:4720 M:1130037248   DEBUG: CWinEventsWin32::WndProcWindow is active
22:10:39 T:4720 M:1130037248   DEBUG: CWinEventsWin32::WndProcWindow gained focus
22:10:40 T:4720 M:1137393664   DEBUG: SDLKeyboard: scancode: 56, sym: 308, unicode: 0, modifier: 0
22:10:40 T:4720 M:1137418240   DEBUG: CWinEventsWin32::WndProcWindow is active
22:10:40 T:4720 M:1137418240   DEBUG: CWinEventsWin32::WndProcWindow lost focus

I do not know if this is enough or if additional information is needed. It seems the episode is identified but somehow no related information is found. When using the TVDB scraper the series seems to be identified correctly.
find quote
Curufinwe Offline
Junior Member
Posts: 2
Joined: Mar 2012
Reputation: 0
Post: #402
Never mind, it is working again now.
find quote
bambi73 Offline
Senior Member
Posts: 195
Joined: Jan 2010
Reputation: 0
Location: Czech Republic
Post: #403
(2012-03-15 13:40)shinner Wrote:  Having trouble scraping episode info for some newer anime though (eg. Another). I pinpointed the problem to anime-list.xml, it's not mapped in there. Am I right that this file has not been updated for a while, like since late 2011?
So I'll have to use my own personal anime mapping? (It also works if I edit the xml in the cache, but then it'll get overwritten Tongue)
I was completelly busy with my work and real life for last 4-5 months so all my XBMC related projects was stoped, i was using it only for occasional watching Smile. I hope it'll be better in following months, so I plan to update anime-list.xml with newest Animes (btw if you already created your own personal list please send it to me by PM, i'll use it as base for my updates), but can't promise when it'll be done Wink.
Anyway I'm not sure if anime-list.xml is source of your problems. It's only mapping between anidb.net and thetvdb.com, so it affect only loading informations from thetvdb.com, like background images or episode plots, rest informations from anidb.net should be there.

(2012-03-15 13:40)shinner Wrote:  Edit:
If anyone has used personal anime mapping before, I could use a little help. If I enable the XBMC web server, can I place the pers-anime-list.xml locally? If so, where do I have to place it and what's the url? (http://localhost. something) And do I have to disable the default xml file?
When I added this feature to scraper I tested it with XBMC web server and it was working fine, but don't ask me details, it was 1,5y ago Big Grin
find quote
lad1337 Offline
Junior Member
Posts: 19
Joined: May 2010
Reputation: 0
Post: #404
i just wanted to let you guys know that i am working on a project with a few others to takle this problem once and for all

i am talking about http://thexem.de/
it is ment to map every episode on one entity with the appropriate one on any other entity

for now there is only a proxy service for the tvdb api but i am personally interested to do the same for anidb
what the proxy service does: you give a origin(e.g. tvdb) and a destination(e.g. anidb) then use the api logic of the origin get a response just like the origin would give you but with the numbering of the destination...
we are still in a closed testing phase but its looking good for a tvdb->scene relation already

but you can already get every map with a json api

what i could do is implement a auto generated anime-list.xml based on the info provided in xem
my question is would this be something of use or would you rather prefer something else to improve the scraper
or would you accept/wish another xml structure or json? could you even use a non local xml for the anime-list.xml ?
if not how could xem provide a anidb_id<->tvdb_id mapping in a way that this scraper would benefit ?

lad1337
find quote
shinner Offline
Member
Posts: 52
Joined: Jan 2012
Reputation: 2
Post: #405
(2012-03-29 12:49)bambi73 Wrote:  I hope it'll be better in following months, so I plan to update anime-list.xml with newest Animes (btw if you already created your own personal list please send it to me by PM, i'll use it as base for my updates), but can't promise when it'll be done Wink.
Anyway I'm not sure if anime-list.xml is source of your problems. It's only mapping between anidb.net and thetvdb.com, so it affect only loading informations from thetvdb.com, like background images or episode plots, rest informations from anidb.net should be there.

My bad, I should've been clearer in explaining. Yup, the series, actors etc. scrape ok, it's just the episode plots are empty. I only just added one anime in manually to test the issue, so no point sending it to you, sorry ^^' Thanks for your work!
Can I ask how you generated your anime-list.xml? Did you write everything manually, or there's a way to automate it by parsing the files returned by tvdb and anidb API with a script? Sorry if I don't make sense, I don't really get the hang of how the scraper works yet, so help is really appreciated.

(2012-03-29 12:49)bambi73 Wrote:  
(2012-03-15 13:40)shinner Wrote:  Edit:
If anyone has used personal anime mapping before, I could use a little help. If I enable the XBMC web server, can I place the pers-anime-list.xml locally? If so, where do I have to place it and what's the url? (http://localhost. something) And do I have to disable the default xml file?
When I added this feature to scraper I tested it with XBMC web server and it was working fine, but don't ask me details, it was 1,5y ago Big Grin
I'll have to do some trial and error when I'm free then. I'll update you with the settings when I can get it to work XP

@lad1337 Not a programmer, so can't really answer your questions, but that sounds really promising in simplifying episode mapping.
(This post was last modified: 2012-03-31 12:58 by shinner.)
find quote
Post Reply