[slyi]

I started playing with it as well and noticed the Unicode issue but I can work around these for the moment. I'm using ice films as my target, integrating asynchronous meta calls rather your sample serial calls later I hope to add paging. One thing I notice an issue on tmdb function update_imdb_meta dictionary keys are incorrect named eg:imdb_poster should be thumb_url
Btw your latest code is much cleaner but for imdb searchs I feel you should initialize with imdbapi first then update with tvdb
Btw2: davilla made a fix for slow python on atv2 so everything is much faster now

[k_zeon]

I started playing with it as well and noticed the Unicode issue but I can work around these for the moment. I'm using ice films as my target, integrating asynchronous meta calls rather your sample serial calls later I hope to add paging. One thing I notice an issue on tmdb function update_imdb_meta dictionary keys are incorrect named eg:imdb_poster should be thumb_url
Btw your latest code is much cleaner but for imdb searchs I feel you should initialize with imdbapi first then update with tvdb
Btw2: davilla made a fix for slow python on atv2 so everything is much faster now

what is the fix davilla made for slow python on atv2 so everything is much faster.
Is it something that i can implement on my ATV2
tks

[rogerthis]

what is the fix davilla made for slow python on atv2 so everything is much faster.
Is it something that i can implement on my ATV2
tks

The python compile files weren't getting created, so it slowed up addons on the atv2. More info here http://forum.xbmc.org/showthread.php?tid=110698

[Eldorado]

I started playing with it as well and noticed the Unicode issue but I can work around these for the moment. I'm using ice films as my target, integrating asynchronous meta calls rather your sample serial calls later I hope to add paging. One thing I notice an issue on tmdb function update_imdb_meta dictionary keys are incorrect named eg:imdb_poster should be thumb_url
Btw your latest code is much cleaner but for imdb searchs I feel you should initialize with imdbapi first then update with tvdb
Btw2: davilla made a fix for slow python on atv2 so everything is much faster now

Thanks I will take a look

When you say initialize with imdb first, then tvdb.. did you mean tvdb or tmdb? I have for movies I can generally get everything I need from TMDB, and maybe 10% of the time I need to use IMDB to fill in the blanks..

From what I have seen so far, TMDB seems to be consistently more complete

But I noticed a couple things yesterday in how the code does scraping when it doesn't have IMDB id's.. the whole get tmdb method needs to be restructured to work on all scenarios.. need to white board it

[k_zeon]

Hey Eldorado.
Got things working ie i can load menus and when moving up and down shows the thumnail, but on initial loading of info it is very very slow.

for aprox 186 movies took approx 10 mins
am now trying the A's with 680 and will see how long it takes....

Edit:
It is now 50 mins later and only on 450. and also the encoding error has not raised its head
as i dont call the function create_infolabels any more. Maybe the str() function was a prob dont know
and i have not changed anything.

[Eldorado]

Hey Eldorado.
Got things working ie i can load menus and when moving up and down shows the thumnail, but on initial loading of info it is very very slow.

for aprox 186 movies took approx 10 mins
am now trying the A's with 680 and will see how long it takes....

That's pretty slow.. what is your connection speed? Are you specifying to download the covers/fanart with the preparezip=true?

I have found on my end I could do approx. 200 in a couple minutes

Everything is getting saved in the DB?

This will be the main reason why addons that use this will need to do an initial meta data pack to minimize all the scraping on first time use

[k_zeon]

Hey Eldorado

I have basically done the same as projectfreetv, except at the moment i am only displaying the coverart. The info is being stored in the db.

Code:

metaget=metahandlers.MetaData()
            for url,name,sYear in match:
                sYear = sYear.replace('(','')
                sYear = sYear.replace(')','')
                meta = metaget.get_meta('', 'movie', name, sYear)
                addon.add_directory({'mode' : 'GetMovieSource', 'url' : url }, CleanHTML(name) + '  (' +  sYear + ')', total_items=len(match), img=meta['cover_url'] )

preparezip=true where would that be set, did not see that in projectfreetv

What does it do? and How do i use it.

[rogerthis]

Hey Eldorado.
Got things working ie i can load menus and when moving up and down shows the thumnail, but on initial loading of info it is very very slow.

for aprox 186 movies took approx 10 mins
am now trying the A's with 680 and will see how long it takes....

Edit:
It is now 50 mins later and only on 450. and also the encoding error has not raised its head
as i dont call the function create_infolabels any more. Maybe the str() function was a prob dont know
and i have not changed anything.

Wow, that's slow. Can you go onto the next page, or do you have to wait for the whole list to finish?

[k_zeon]

no i sit and watch the progress bar.
Over 1 hour to load 650 movies and wait till finished.

XBMC on Apple TV2 Eden Pre 10.09

[rogerthis]

no i sit and watch the progress bar.
Over 1 hour to load 650 movies and wait till finished.

XBMC on Apple TV2 Eden Pre 10.09

I think slyi's idea of asynchronous is that way to go. ie downloading in the background, and you still have full control of xbmc, so you can go into the next page. Even better asynchronous with caching, so it keeps downloading after you move onto the next page, but puts the new page on the top of the list. You would need some dialog on the top right, so that you knew it was working away, like it does for scanning the library.

Did you try out the python compile files fix on the ATV2?

[k_zeon]

Hey rogerthis

yes i tried the speed up fix and yes i did notice a difference on 2 ATV2's. Menus load a bit quicker.

I still dont know what preparezip=true is and how to use it.

Any ideas

Re the asynchronous , Eldorado is the one who is rejigging the Metadata addon.

So at present i run his addon to get the info into my db but as of yet am only using the thumb to display when selecting a menu entry. this is working but it takes ages to get this info.

I have added a settings file to my addon to enable/disable the meta and if i have already scraped the info with it enable and then switch it off and try again, the menu load very quickly and even the thumbs are displayed even though i have not used the img tage

ie

Code:

if meta_setting=='false':
            for url,name,sYear in match:
                sYear = sYear.replace('(','')
                sYear = sYear.replace(')','')
                addon.add_directory({'mode' : 'GetMovieSource', 'url' : url }, CleanHTML(name) + '  (' +  sYear + ')', total_items=len(match) )
            addon.end_of_directory()

        elif  meta_setting=='true':
            metaget=metahandlers.MetaData()
            for url,name,sYear in match:
                sYear = sYear.replace('(','')
                sYear = sYear.replace(')','')
                meta = metaget.get_meta('', 'movie', name, sYear)
                addon.add_directory({'mode' : 'GetMovieSource', 'url' : url }, CleanHTML(name) + '  (' +  sYear + ')', total_items=len(match), img=meta['cover_url'] )
            addon.end_of_directory()

[Eldorado]

Hey rogerthis

yes i tried the speed up fix and yes i did notice a difference on 2 ATV2's. Menus load a bit quicker.

I still dont know what preparezip=true is and how to use it.

Any ideas

Re the asynchronous , Eldorado is the one who is rejigging the Metadata addon.

So at present i run his addon to get the info into my db but as of yet am only using the thumb to display when selecting a menu entry. this is working but it takes ages to get this info.

I have added a settings file to my addon to enable/disable the meta and if i have already scraped the info with it enable and then switch it off and try again, the menu load very quickly and even the thumbs are displayed even though i have not used the img tage

ie

Code:

if meta_setting=='false':
            for url,name,sYear in match:
                sYear = sYear.replace('(','')
                sYear = sYear.replace(')','')
                addon.add_directory({'mode' : 'GetMovieSource', 'url' : url }, CleanHTML(name) + '  (' +  sYear + ')', total_items=len(match) )
            addon.end_of_directory()

        elif  meta_setting=='true':
            metaget=metahandlers.MetaData()
            for url,name,sYear in match:
                sYear = sYear.replace('(','')
                sYear = sYear.replace(')','')
                meta = metaget.get_meta('', 'movie', name, sYear)
                addon.add_directory({'mode' : 'GetMovieSource', 'url' : url }, CleanHTML(name) + '  (' +  sYear + ')', total_items=len(match), img=meta['cover_url'] )
            addon.end_of_directory()

The preparezip=True is a setting only the dev will set and will do it just once in order to create a metadata pack to distribute, with the amount of content you have on your addon I would strongly recommend that you distribute one so that users aren't scraping so much data

To use you set it when you initialize metahandlers

eg.
metaget=metahandlers.MetaData(preparezip=True)

This will download the covers to your cache folder, then all you need to do is zip of the folder

Is it possible for you to send me a copy of your addon (are you on git?) so I can run a scrape on my connection? I currently don't have any movie lists so big

An hour is VERY long.. would like to read thru the logs as well

edit - I should note I'm doing all my testing on a windows pc, are you using one of the more recent nightly's on your atv?

[k_zeon]

Hi Eldorado

Thanks. I am running XBMX ATV2 XBMC Eden Pre 10.9

Git is https://github.com/kzeonG/TVShack

some code may not be required etc and this is still in development.

let me know what you think and if i could do things better

[Eldorado]

Running it through letter 'A' right now.. chugging along

One thing I've spotted so far is some bad names being passed which is eating up a bit of time:

Code:

10:18:23 T:2308  NOTICE: Requesting TMDB : http://api.themoviedb.org/2.1/Movie.search/en/json/b91e899ce561dd19695340c3b26e0a02/A%20Dispatch%20from%20Reuter%26amp%3B%23x27%3Bs+1940
10:18:25 T:2308  NOTICE: Error connecting to TMDB: HTTP Error 404: Not Found

You are passing in names with bad characters:

Code:

A Dispatch from Reuter's
A Hard Day's Night
A Midsummer Night's Dream

Use your CleanHTML method on the name before you pass it in, I modified your addon on my end to do this and results came back much better

I've spotted a few other things I can clean up as well in my own code, plus one bug

My full load took approx. 25mins, which I think compares well against the built-in scrapers

[k_zeon]

Running it through letter 'A' right now.. chugging along

One thing I've spotted so far is some bad names being passed which is eating up a bit of time:

Code:

10:18:23 T:2308  NOTICE: Requesting TMDB : http://api.themoviedb.org/2.1/Movie.search/en/json/b91e899ce561dd19695340c3b26e0a02/A%20Dispatch%20from%20Reuter%26amp%3B%23x27%3Bs+1940
10:18:25 T:2308  NOTICE: Error connecting to TMDB: HTTP Error 404: Not Found

You are passing in names with bad characters:

Code:

A Dispatch from Reuter's
A Hard Day's Night
A Midsummer Night's Dream

Use your CleanHTML method on the name before you pass it in, I modified your addon on my end to do this and results came back much better

I've spotted a few other things I can clean up as well in my own code, plus one bug

My full load took approx. 25mins, which I think compares well against the built-in scrapers

Hey Eldorado

Thanks, should have spotted that re the Clean(name)...

What other things have you found to clean up in your own code, plus what bug.