[dev] Project Free TV

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
Eldorado Offline
Fan
Posts: 520
Joined: May 2009
Reputation: 14
Post: #16
t0mm0 Wrote:yeah, please do, as long as it's generic enough that you can pass anything around - maybe it will be useful in other situations too.

t0mm0

Will do, I created a fork for myself and will put in this change as well as my Zshare plugin
find quote
t0mm0 Offline
Fan
Posts: 486
Joined: Mar 2011
Reputation: 8
Location: UK
Post: #17
Eldorado Wrote:Will do, I created a fork for myself and will put in this change as well as my Zshare plugin

great! look forward to the pull request Wink

i promise i will try and fix the sorting (and videoid) stuff this weekend

t0mm0
find quote
rogerthis Offline
Donor
Posts: 217
Joined: Apr 2011
Reputation: 1
Location: Connacht
Post: #18
Eldorado, have you played around with any of the metadata yet?
find quote
Eldorado Offline
Fan
Posts: 520
Joined: May 2009
Reputation: 14
Post: #19
rogerthis Wrote:Eldorado, have you played around with any of the metadata yet?

I have, but nothing substantial yet..

I grabbed the latest from the Icefilms master branch, moved it into it's own script folder all setup to be it's own script addon

The fun part now is stripping all Icefilms specific coding as well as some items that were left on their to-do list

I was wondering if I should throw it up on my git repository for all to start working on, or pass it over to t0mm0 to host as he seems to have a much better understanding of python..?

Also, seems that it can only scrape tmdb and tvdb using IMDB id's.. would need to add in scraping based on just movie or tv show name, not sure how difficult it will be to add that in
find quote
lefty420 Offline
Senior Member
Posts: 160
Joined: Jul 2009
Reputation: 4
Post: #20
Not sure if it would help but have you seen the modified icefilms that comes with a 5gb meta data collection. It can be found here http://forum.xbmc.org/showthread.php?p=8...post879777
find quote
t0mm0 Offline
Fan
Posts: 486
Joined: Mar 2011
Reputation: 8
Location: UK
Post: #21
Eldorado Wrote:I was wondering if I should throw it up on my git repository for all to start working on, or pass it over to t0mm0 to host as he seems to have a much better understanding of python..?
haha Rofl but thanks for the compliment Wink

by all means bung it up on github so everyone can take a look. personally i probably won't get to it for a while Sad i'm really trying to nail the video id stuff whch is turning out to be more complicated than i first hoped (i've already completely rewritten everything several times but am never happy with it as it ends up too complicated) once that is done and people are happy i think we should do some kind of release so people can actually use all this stuff!

t0mm0
find quote
rogerthis Offline
Donor
Posts: 217
Joined: Apr 2011
Reputation: 1
Location: Connacht
Post: #22
Eldorado Wrote:I have, but nothing substantial yet..

I grabbed the latest from the Icefilms master branch, moved it into it's own script folder all setup to be it's own script addon

The fun part now is stripping all Icefilms specific coding as well as some items that were left on their to-do list

I was wondering if I should throw it up on my git repository for all to start working on, or pass it over to t0mm0 to host as he seems to have a much better understanding of python..?

Also, seems that it can only scrape tmdb and tvdb using IMDB id's.. would need to add in scraping based on just movie or tv show name, not sure how difficult it will be to add that in

Yeah, could you throw it up on your git, and I'll take a look at it and see what I can do. The plan is either add it to the t0mm0 repo or keep it separate. I guess it all depends on how big it gets. Could you start a new thread for it too?

We would need to make a plan on how it will function. eg Will the info get added to the library, or have a cache for each addon, or it gets redone at each page load.
Metadata for favourites only might be a good idea. Where your favourites would get reloaded at addon start \ xbmc reboot.
find quote
Eldorado Offline
Fan
Posts: 520
Joined: May 2009
Reputation: 14
Post: #23
rogerthis Wrote:Yeah, could you throw it up on your git, and I'll take a look at it and see what I can do. The plan is either add it to the t0mm0 repo or keep it separate. I guess it all depends on how big it gets. Could you start a new thread for it too?

We would need to make a plan on how it will function. eg Will the info get added to the library, or have a cache for each addon, or it gets redone at each page load.
Metadata for favourites only might be a good idea. Where your favourites would get reloaded at addon start \ xbmc reboot.

Well unless I'm reading the code wrong, and it's also an area of xbmc/python I have not gone yet, it appears it creates a database or container (if that's the correct term) in the userdata directory to store all meta data, then references/adds to it as you navigate Icefilms

So my first thought would be to have a central meta data DB that all addons reference and update

I'll pop this up on my repository tonight so people can start taking a look
find quote
Eldorado Offline
Fan
Posts: 520
Joined: May 2009
Reputation: 14
Post: #24
Need a bit of help on this one..

This page is giving me some issues regarding ascii characters being used, likely the same error some complain about in Icefilms:
http://www.free-tv-video-online.me/inter...son_4.html

Code:
13:22:10 T:4732   ERROR: Error Type: <type 'exceptions.UnicodeEncodeError'>
13:22:10 T:4732   ERROR: Error Contents: 'ascii' codec can't encode character u'\x92' in position 6: ordinal not in range(128)
13:22:10 T:4732   ERROR: Traceback (most recent call last):
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\plugin.video.projectfreetv\default.p​y", line 310, in <module>
                                                addon.add_video_item(url,{'title':name},{'section': 'tvshows', 'video': name})
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.t0mm0.common\lib\t0mm0​\common\addon.py", line 560, in add_video_item
                                                playlist, item_type='video')
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.t0mm0.common\lib\t0mm0​\common\addon.py", line 535, in add_item
                                                play = self.build_plugin_url({'play': play}) + self.build_url_query(queries)
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.t0mm0.common\lib\t0mm0​\common\addon.py", line 211, in build_url_query
                                                return '&' + urllib.urlencode(queries)
                                              File "C:\Program Files (x86)\XBMC\system\python\Lib\urllib.py", line 1269, in urlencode
                                                v = quote_plus(str(v))
                                            UnicodeEncodeError: 'ascii' codec can't encode character u'\x92' in position 6: ordinal not in range(128)
13:22:10 T:4732    INFO: -->End of Python script error report<--

I'm guessing it's the single quote mark in the episode - "3. If You Love Me, Why Am I Dyin'?"

I've tried adding a .encode('utf-8') from some google searches, but doesn't get any better:

Code:
13:37:30 T:6680   ERROR: Error Type: <type 'exceptions.UnicodeDecodeError'>
13:37:30 T:6680   ERROR: Error Contents: 'ascii' codec can't decode byte 0xc2 in position 19: ordinal not in range(128)
13:37:30 T:6680   ERROR: Traceback (most recent call last):
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\plugin.video.projectfreetv\default.p​y", line 309, in <module>
                                                addon.add_video_item(url,{'title':name},{'section': 'tvshows', 'video': name})
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.t0mm0.common\lib\t0mm0​\common\addon.py", line 560, in add_video_item
                                                playlist, item_type='video')
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.t0mm0.common\lib\t0mm0​\common\addon.py", line 546, in add_item
                                                self.log_debug('adding item: %s - %s' % (infolabels['title'], play))
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.t0mm0.common\lib\t0mm0​\common\addon.py", line 283, in log_debug
                                                self.log(msg, xbmc.LOGDEBUG)
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.t0mm0.common\lib\t0mm0​\common\addon.py", line 260, in log
                                                msg = unicodedata.normalize('NFKD', unicode(msg)).encode('ascii',
                                            UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 19: ordinal not in range(128)
13:37:30 T:6680    INFO: -->End of Python script error report<--
find quote
Eldorado Offline
Fan
Posts: 520
Joined: May 2009
Reputation: 14
Post: #25
t0mm0 Wrote:haha Rofl but thanks for the compliment Wink

by all means bung it up on github so everyone can take a look. personally i probably won't get to it for a while Sad i'm really trying to nail the video id stuff whch is turning out to be more complicated than i first hoped (i've already completely rewritten everything several times but am never happy with it as it ends up too complicated) once that is done and people are happy i think we should do some kind of release so people can actually use all this stuff!

t0mm0

Once you release them, I'll be releasing this addon.. no pressure Smile

Any word on getting them added to the official repository?
find quote
rogerthis Offline
Donor
Posts: 217
Joined: Apr 2011
Reputation: 1
Location: Connacht
Post: #26
Eldorado Wrote:Need a bit of help on this one..

This page is giving me some issues regarding ascii characters being used, likely the same error some complain about in Icefilms:
http://www.free-tv-video-online.me/inter...son_4.html

Code:
13:22:10 T:4732   ERROR: Error Type: <type 'exceptions.UnicodeEncodeError'>
13:22:10 T:4732   ERROR: Error Contents: 'ascii' codec can't encode character u'\x92' in position 6: ordinal not in range(128)
13:22:10 T:4732   ERROR: Traceback (most recent call last):
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\plugin.video.projectfreetv\default.p​y", line 310, in <module>
                                                addon.add_video_item(url,{'title':name},{'section': 'tvshows', 'video': name})
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.t0mm0.common\lib\t0mm0​\common\addon.py", line 560, in add_video_item
                                                playlist, item_type='video')
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.t0mm0.common\lib\t0mm0​\common\addon.py", line 535, in add_item
                                                play = self.build_plugin_url({'play': play}) + self.build_url_query(queries)
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.t0mm0.common\lib\t0mm0​\common\addon.py", line 211, in build_url_query
                                                return '&' + urllib.urlencode(queries)
                                              File "C:\Program Files (x86)\XBMC\system\python\Lib\urllib.py", line 1269, in urlencode
                                                v = quote_plus(str(v))
                                            UnicodeEncodeError: 'ascii' codec can't encode character u'\x92' in position 6: ordinal not in range(128)
13:22:10 T:4732    INFO: -->End of Python script error report<--

I'm guessing it's the single quote mark in the episode - "3. If You Love Me, Why Am I Dyin'?"

I've tried adding a .encode('utf-8') from some google searches, but doesn't get any better:

Code:
13:37:30 T:6680   ERROR: Error Type: <type 'exceptions.UnicodeDecodeError'>
13:37:30 T:6680   ERROR: Error Contents: 'ascii' codec can't decode byte 0xc2 in position 19: ordinal not in range(128)
13:37:30 T:6680   ERROR: Traceback (most recent call last):
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\plugin.video.projectfreetv\default.p​y", line 309, in <module>
                                                addon.add_video_item(url,{'title':name},{'section': 'tvshows', 'video': name})
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.t0mm0.common\lib\t0mm0​\common\addon.py", line 560, in add_video_item
                                                playlist, item_type='video')
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.t0mm0.common\lib\t0mm0​\common\addon.py", line 546, in add_item
                                                self.log_debug('adding item: %s - %s' % (infolabels['title'], play))
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.t0mm0.common\lib\t0mm0​\common\addon.py", line 283, in log_debug
                                                self.log(msg, xbmc.LOGDEBUG)
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.t0mm0.common\lib\t0mm0​\common\addon.py", line 260, in log
                                                msg = unicodedata.normalize('NFKD', unicode(msg)).encode('ascii',
                                            UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 19: ordinal not in range(128)
13:37:30 T:6680    INFO: -->End of Python script error report<--

Have you tried Anarchintosh's htmlcleaner.py? https://github.com/icefilms-xbmc/htmlcle...cleaner.py This is what icefilms used to remove those annoying characters. It would be useful to be in t0mm0's resolver.
(This post was last modified: 2011-09-06 21:54 by rogerthis.)
find quote
t0mm0 Offline
Fan
Posts: 486
Joined: Mar 2011
Reputation: 8
Location: UK
Post: #27
rogerthis Wrote:Have you tried Anarchintosh's htmlcleaner.py? https://github.com/icefilms-xbmc/htmlcle...cleaner.py This is what icefilms used to remove those annoying characters. It would be useful to be in t0mm0's resolver.

actually, it looks like this is a bug in the web site rather than my code (but i would say that wouldn't i Wink)

the page says that is is using the ISO-8859-1 character set, but \x92 is not a valid ISO-8859-1 character (looks like it is really CP-1252). maybe we should add an override in t0mm0.common.net where if you know that the web page is lying about the character set you can specify it?

re: htmlcleaner.py - all the functionality i think is useful is already available in t0mm0.common.addon (though the list of entities could do with being expanded!). generally where possible my code is trying to preserve all weird characters because what is a weird character to me might be important in someone elses language. we should not be stripping stuff, we should be making sure it is proper utf-8 unicode instead. if the web page identifies the correct character set in either the http header or a meta tag then this should already happen - the problem only occurs at the moment when the web page lies Wink

t0mm0
find quote
Eldorado Offline
Fan
Posts: 520
Joined: May 2009
Reputation: 14
Post: #28
t0mm0 Wrote:actually, it looks like this is a bug in the web site rather than my code (but i would say that wouldn't i Wink)

the page says that is is using the ISO-8859-1 character set, but \x92 is not a valid ISO-8859-1 character (looks like it is really CP-1252). maybe we should add an override in t0mm0.common.net where if you know that the web page is lying about the character set you can specify it?

re: htmlcleaner.py - all the functionality i think is useful is already available in t0mm0.common.addon (though the list of entities could do with being expanded!). generally where possible my code is trying to preserve all weird characters because what is a weird character to me might be important in someone elses language. we should not be stripping stuff, we should be making sure it is proper utf-8 unicode instead. if the web page identifies the correct character set in either the http header or a meta tag then this should already happen - the problem only occurs at the moment when the web page lies Wink

t0mm0

I might be in a bit of a catch then, it seems to be just this one page on the site that is giving me this issue.. and I think it's rather recent as I know I've pulled in the page before

If I switch the character set being used to satisfy this one page, I could likely break the others? How do you change the character set when requesting a page?
find quote
t0mm0 Offline
Fan
Posts: 486
Joined: Mar 2011
Reputation: 8
Location: UK
Post: #29
hi eldorado,
Eldorado Wrote:I might be in a bit of a catch then, it seems to be just this one page on the site that is giving me this issue.. and I think it's rather recent as I know I've pulled in the page before

If I switch the character set being used to satisfy this one page, I could likely break the others?

well in this case i believe that ISO-8859-1 is a subset of CP1252 so it might actually be worth forcing CP1252 on any page that says it is ISO-8859-1 just in case (unless someone who knows more about this stuff than me has a better suggestion?) which we could do in t0mm0.common.net

i would think it is likely that all pages on a site would be encoded the same way so if you forced an encoding it would be okay across the whole site, although of course i could be wrong....
Eldorado Wrote:How do you change the character set when requesting a page?
you can't at the moment, but i was thinking maybe adding something like a force_page_encoding argument to http_GET() might be useful?

t0mm0
find quote
Eldorado Offline
Fan
Posts: 520
Joined: May 2009
Reputation: 14
Post: #30
t0mm0 Wrote:hi eldorado,


well in this case i believe that ISO-8859-1 is a subset of CP1252 so it might actually be worth forcing CP1252 on any page that says it is ISO-8859-1 just in case (unless someone who knows more about this stuff than me has a better suggestion?) which we could do in t0mm0.common.net

i would think it is likely that all pages on a site would be encoded the same way so if you forced an encoding it would be okay across the whole site, although of course i could be wrong....
you can't at the moment, but i was thinking maybe adding something like a force_page_encoding argument to http_GET() might be useful?

t0mm0

I'm thinking it could be, we could use this one page as a solid test
find quote