[t0mm0]

Are you referring to the current Icefilms DB that users have and merging to this one?

If so the two will not be compatible due to the amount of changes, new fields, changed field names and datatypes, but perhaps some sort of conversion script could be written

Or were you thinking of sending periodic updates for the db? That might have to be something for a future release, definitely would be useful

no i was thinking more along the lines of....

imagine there are 2 addons all using your library.

user installs addon1, browses around (therefore downloading some metadata and storing it in meta database)
user installs addon2, it prompts him to download a metadata pack which he agrees to. meta module downloads the pack and replaces the database.

now if addon1 had inserted metadata into the db that doesn't appear in the addon2 metadata pack, the data from addon1 is lost and will need to be redownloaded

if when the pack is downloaded it is merged in to the existing db in some way rather than just replacing it, addon1's obscure metadata is preserved and doesn't need to be re-downloaded.

which i guess is the same as being able to use incremental updates to the data.

hope that makes sense!

t0mm0

[Eldorado]

no i was thinking more along the lines of....

imagine there are 2 addons all using your library.

user installs addon1, browses around (therefore downloading some metadata and storing it in meta database)
user installs addon2, it prompts him to download a metadata pack which he agrees to. meta module downloads the pack and replaces the database.

now if addon1 had inserted metadata into the db that doesn't appear in the addon2 metadata pack, the data from addon1 is lost and will need to be redownloaded

if when the pack is downloaded it is merged in to the existing db in some way rather than just replacing it, addon1's obscure metadata is preserved and doesn't need to be re-downloaded.

which i guess is the same as being able to use incremental updates to the data.

hope that makes sense!

t0mm0

Gotcha, I was thinking about how to handle this and open to ideas

Should it be up to the individual addons to download the metadata packs, or should it be forced on the initial use of the library?

As in if you want to use this library, a metadata pack must be downloaded/installed and contain as much as we can no matter the site.. no blank DB option? I kind of like this option, but haven't thought too far into the implications of doing so..

The option of leaving it up to the addon presents an interesting idea, each addon dev can scrape the entire site that they are writing the addon for, and offer it up as a pack which can be added to an existing DB if one exists

[Eldorado]

I understand your using the icefilms meta code as a base, and it may sound like a lot to change it for little benefit, but it would mean one far cleaner code for Would you consider using some common functions for get/set db items.

To be honest, i have not been able read the code to well partly because i only started learning python a couple of weeks ago and i wanted to start with clean slate for my own knowledge but there seems to over lapping functions in each scraper tvdb & tmdb. I review your git, so more at the weekend and may send you on patch for review for common code i find.

I can assure you I won't be the one to rework the code, doing the cleaning that I'm doing right now is enough of a job

Keep in mind the code is currently under constant development, almost daily I'm working on it cleaning things up, adding comments, removing unnecessary code etc. I'll try to post my updates to git more frequently

Also, the TVDB scraper was not written by any of us but taken from an open source project elsewhere.. I don't think I want to mess with it too much, I've been focusing more on how they are called and the supporting functions

I believe you want castandrole attribute to show the role details aswell. Looking at http://wiki.xbmc.org/?title=InfoLabels

I would like to see item below

CastAndRole
ListItem.Episode
ListItem.Season
ListItem.TVShowTitle
ListItem.Property(TotalSeasons)
ListItem.Property(TotalEpisodes)
ListItem.Property(WatchedEpisodes)
ListItem.Property(UnWatchedEpisodes)
ListItem.Property(NumEpisodes)
Container.FolderThumb
Container.TvshowThumb
Container.SeasonThumb
Fanart.Image
ListItem.StarRating

Some of these I don't believe are available to me from the scrapers, don't mix properties that are available to the skin vs what can be supplied via scrapers

eg.
ListItem.Property(TotalSeasons)
ListItem.Property(TotalEpisodes)
ListItem.Property(WatchedEpisodes)
ListItem.Property(UnWatchedEpisodes)
ListItem.Property(NumEpisodes)

I believe all are out of my control but instead handled internally via xbmc/skin functions

Most of the others are already included, I'll post soon as to what metadata I am currently collecting for TV Shows/Episodes

The reason i mentioned a json file it that is text rather than binary and partly i don't yet fully understand the sqllite advantages/api's.

My main objection to one massive download is that say i watch x-files re-runs i don't see the need to have all jersey shore data and images polluting my limited HD. ;-)

BTW: For thumbnails/fanart what image format do you use png or dds?

I hear you about jersey shore

The metadata download pack I believe is almost necessary to stay friendly with the sites we are scraping, though for those of us more tech savvy we can go in and manually remove the data we don't want

The covers/fanart I believe are all jpg, it's dependent on what the scraping sites have available

[t0mm0]

Gotcha, I was thinking about how to handle this and open to ideas

Should it be up to the individual addons to download the metadata packs, or should it be forced on the initial use of the library?

As in if you want to use this library, a metadata pack must be downloaded/installed and contain as much as we can no matter the site.. no blank DB option? I kind of like this option, but haven't thought too far into the implications of doing so..

The option of leaving it up to the addon presents an interesting idea, each addon dev can scrape the entire site that they are writing the addon for, and offer it up as a pack which can be added to an existing DB if one exists

i would say that it might be a good idea to provide a fairly reasonable pack with the most popular stuff though the meta module (or maybe 2 - 1 for tv, 1 for movies) but i think it would also be cool to allow addons to provide their own packs.

for example, the main pack would probably include the popular US shows and english language films (you obviously can't include everything as it would be huge) but wouldn't it be cool if say an anime addon could distribute an anime metadata pack, or a french film addon could distribute a french film metadata pack etc?

i would also make it optional (just like icefilms, although there should be a setting to stop it from keep asking) and maybe also make the images separate and optional too. some of us have slow connection speeds and/or small hard drives

t0mm0

[Eldorado]

i would say that it might be a good idea to provide a fairly reasonable pack with the most popular stuff though the meta module (or maybe 2 - 1 for tv, 1 for movies) but i think it would also be cool to allow addons to provide their own packs.

for example, the main pack would probably include the popular US shows and english language films (you obviously can't include everything as it would be huge) but wouldn't it be cool if say an anime addon could distribute an anime metadata pack, or a french film addon could distribute a french film metadata pack etc?

i would also make it optional (just like icefilms, although there should be a setting to stop it from keep asking) and maybe also make the images separate and optional too. some of us have slow connection speeds and/or small hard drives

t0mm0

Good stuff, gives me a lot to think about

As of now, the way it was done the metadata pack is purely optional, it's up to the addon to make a call to metacontainer() to download and install a zip file, though currently it would overwrite whatever DB might be there, should be easy enough to add ability to add a data pack to an existing DB.. unzip db to a temp dir, select/insert..

If you don't download/install a data pack, a blank DB is created on init, they even built functionality in to create a zip pack very easily!

The images is definitely on the todo list, I believe they are downloading only the movie covers.. I would like to give the option to the addon to do this, the DB stores the URL to the cover and fanart

[Eldorado]

Need some advice on how to save the cast information

InfoLabel 'CastandRole' requires to send in a tuple as so:

cast = [('joe blow', 'sheriff'), ('jane sue', 'waitress')]

Now to save this in SQLite, what is the best option?

Convert to string and save in a text field? Then convert back to tuple when reading from the DB?

My other option is to just scrap the cast idea and leave it as was already done in the script and store it in with the plot

eg.

Code:

Starring: Joe Blow, Jane Sue

Plot:
A whole bunch of random crap happens

[t0mm0]

Need some advice on how to save the cast information

InfoLabel 'CastandRole' requires to send in a tuple as so:

cast = [('joe blow', 'sheriff'), ('jane sue', 'waitress')]

Now to save this in SQLite, what is the best option?

Convert to string and save in a text field? Then convert back to tuple when reading from the DB?

My other option is to just scrap the cast idea and leave it as was already done in the script and store it in with the plot

eg.

Code:

Starring: Joe Blow, Jane Sue

Plot:
A whole bunch of random crap happens

you could have a separate 'CastAndRole' table with fields for movieId, name, and role.

[Eldorado]

you could have a separate 'CastAndRole' table with fields for movieId, name, and role.

Had considered that, but then it gives me more work

For now I've simply converted it to a string to store in the table, then use eval() when I select from it which converts it back to a tuple.. hopefully this is ok and would be good if anyone can point out any performance/space issues by doing it like this

Cast is working well now, though I'm not sure how to enable the cast pictures - which I probably won't do, but also you can click on a cast member and I'm assuming it brings up a profile of the actor.. probably won't do either, but would be nice to know if and how it can be done

More cleanup items to go in soon, I've reworked the code a bit and pulled all TVDB calls out of the TMDB class and into the metahandler class, didn't think it made sense to be in there..

Now tv shows are slightly broken, I think I'm now happy with movies and will focus on finalizing tv show metadata

Then it's on to the more practical things such as downloading images, creating metadata packs, inserting updates etc...

[Eldorado]

Many more updates, also first post updated with more details and 1-2 slight changes on how the methods are called

All movie, tv show - season, episode meta data looks to be working pretty well and I've collected all that I believe I can, please check the first post for the list

Anyone out there testing? So far I've been running only against Project Free TV, would like to see results from other sites

[t0mm0]

Anyone out there testing? So far I've been running only against Project Free TV, would like to see results from other sites

sorry i haven't got to it yet. i have been busy for the last few of weeks but that's out of the way now. i'm gonna spend the weekend getting urlresolver back up to speed and then i'll try adding meta support to letmechoosethis.

t0mm0

[k_zeon]

Hi Eldorado

did some more testing and i have run into errors

http://pastebin.com/S9eCAyg2

I have used the code in your addon plugin.video.projectfreetv as an example to get movie info.
At the moment i am only using the below to get the img. the rest will come when t0mm0 updates his code to add to the Dir

Code:

elif mode == 'GetMovieTitle':
    
        html = net.http_GET(url).content
        match=re.compile('<li class="searchList"><a href="(.+?)">(.+?)</a> <span>(.+?)</span>').findall(html)

        metaget=metahandlers.MetaData()
        
        for url,name,sYear in match:
            sYear = sYear.replace('(','')
            sYear = sYear.replace(')','')
            
            meta = metaget.get_meta('', 'movie', name, sYear)
            infoLabels = create_infolabels(meta, name)
          
            addon.add_directory({'mode' : 'GetMovieSource', 'url' : url }, CleanHTML(name) + '  (' +  sYear + ')', total_items=len(match), img=infoLabels['thumb'] )

as you will see in my Log, the info is being returned but an error is generated.
I get the impression its happening on

infoLabels = create_infolabels(meta, name)

any idea's

Edit: May have just found the reason. Delete the s off genres to make it genre and also for studios. then the error goes away

Code:

def create_infolabels(meta, name):
    infoLabels = {}
    infoLabels['title'] = name
    infoLabels['plot'] = str(meta['plot'])
    infoLabels['genre'] = str(meta['genres'])
    infoLabels['duration'] = str(meta['duration'])
    infoLabels['premiered'] = str(meta['premiered'])
    infoLabels['studio'] = meta['studios']

[k_zeon]

after some more testing, have another error under plot

http://pastebin.com/Fp1XJ94n

I am using your latest version but still get the encoding error

last word in plot is fianc\xe9e.

[Eldorado]

Hi Eldorado

did some more testing and i have run into errors

http://pastebin.com/S9eCAyg2

I have used the code in your addon plugin.video.projectfreetv as an example to get movie info.
At the moment i am only using the below to get the img. the rest will come when t0mm0 updates his code to add to the Dir

Code:

elif mode == 'GetMovieTitle':
    
        html = net.http_GET(url).content
        match=re.compile('<li class="searchList"><a href="(.+?)">(.+?)</a> <span>(.+?)</span>').findall(html)

        metaget=metahandlers.MetaData()
        
        for url,name,sYear in match:
            sYear = sYear.replace('(','')
            sYear = sYear.replace(')','')
            
            meta = metaget.get_meta('', 'movie', name, sYear)
            infoLabels = create_infolabels(meta, name)
          
            addon.add_directory({'mode' : 'GetMovieSource', 'url' : url }, CleanHTML(name) + '  (' +  sYear + ')', total_items=len(match), img=infoLabels['thumb'] )

as you will see in my Log, the info is being returned but an error is generated.
I get the impression its happening on

infoLabels = create_infolabels(meta, name)

any idea's

Edit: May have just found the reason. Delete the s off genres to make it genre and also for studios. then the error goes away

Code:

def create_infolabels(meta, name):
    infoLabels = {}
    infoLabels['title'] = name
    infoLabels['plot'] = str(meta['plot'])
    infoLabels['genre'] = str(meta['genres'])
    infoLabels['duration'] = str(meta['duration'])
    infoLabels['premiered'] = str(meta['premiered'])
    infoLabels['studio'] = meta['studios']

My Project Free TV code is a bit out of date right now, need to push in new updates to that code

[Eldorado]

after some more testing, have another error under plot

http://pastebin.com/Fp1XJ94n

I am using your latest version but still get the encoding error

last word in plot is fianc\xe9e.

Thanks, I'll take a look soon... odd that unicode errors are still popping..

[Eldorado]

Looks like we need to get t0mm0 a letter 't' icon for his profile now to keep up with us

So, k_zeon has another unicode error.. t0mm0, is there any chance of adding more to your HttpResponse class to possibly catch this one? I'm not sure what else we can do other than adding some coding to convert specific characters, which I'd like to avoid doing.. unicode stuff is a bit outside my territory right now

Character is 'é' in the word finacée, last word in the overview:
http://api.themoviedb.org/2.1/Movie.getI...0a02/17134

It is coming over as 'fianc\xe9e'.. which I'm surprised by as I had this issue earlier with this character on other movies but was resolved when I switched to your library to make the http get's