2014-01-15, 23:42
For full automatic scraping, the only real source for me is TMDB (everything else doesn't seem to work well or gives wrong results). I have set up my Ember for Frodo, got my TMDB API and only set up this source for my scraper. So far (after the porn stuff doesn't appear anymore) everything seems to work fine except the fact that scraping one single movie takes about 4-5 minutes... no need to calculate the total amount of time for a database of 2500 movies. 
I have a 50MBit so what is actually happening here? What takes so long? I don't even download trailers and according to my network adapter there isn't any bandwidth used at all, only for a short time at the beginning (for the poster, banner, fanart and nfo I guess), but what is taking so long after that?
BTW, I am using BETA Ember 1.4 - 20140110
Thanks already for your help and advice,
best regards,
Torm
Update 1
To find out what's the problem, I just scraped the poster, the fanart, the banner and the info for the movie and deactivated everything else. Scraping "12 Monkeys" automatically still took me 5 minutes.
Update 2
Only scraped the info of the movie, took Ember also 5 minutes to finish. It obviously seems to be a problem with the program itself and not the source. To make sure I am right, I will try to scrape from a different source than TMDB.
Update 3
Scraped with only IMDB activated and only the movie information, nothing else. Finished in 10 seconds. The good thing is that it perfectly works also with images - and much faster, the bad is that it uses english texts and genres where I would like to have them in my native language german. It obviously has something to do with TMDB scraper or api but this is something I can't track down.
I have a 50MBit so what is actually happening here? What takes so long? I don't even download trailers and according to my network adapter there isn't any bandwidth used at all, only for a short time at the beginning (for the poster, banner, fanart and nfo I guess), but what is taking so long after that?BTW, I am using BETA Ember 1.4 - 20140110
Thanks already for your help and advice,
best regards,
Torm
Update 1
To find out what's the problem, I just scraped the poster, the fanart, the banner and the info for the movie and deactivated everything else. Scraping "12 Monkeys" automatically still took me 5 minutes.
Update 2
Only scraped the info of the movie, took Ember also 5 minutes to finish. It obviously seems to be a problem with the program itself and not the source. To make sure I am right, I will try to scrape from a different source than TMDB.
Update 3
Scraped with only IMDB activated and only the movie information, nothing else. Finished in 10 seconds. The good thing is that it perfectly works also with images - and much faster, the bad is that it uses english texts and genres where I would like to have them in my native language german. It obviously has something to do with TMDB scraper or api but this is something I can't track down.
