XBMC Community Forum
script.module.urlresolver development - Printable Version

+- XBMC Community Forum (http://forum.xbmc.org)
+-- Forum: Development (/forumdisplay.php?fid=32)
+--- Forum: Python Add-on Development (/forumdisplay.php?fid=26)
+--- Thread: script.module.urlresolver development (/showthread.php?tid=105707)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22


- DragonWin - 2011-08-21 22:15

t0mm0 Wrote:looks like you're making some interesting progress there!

It' coming along nicely it's now able to test the proxy's it has found, and choose the optimal proxy, and returns ip: port, location like:
Code:
184.73.63.96:80, United States  (Seattle)

I did run into an error that I'm unsure if it should be the net class that handles it.

Error showed itself a few times when I did a HEAD to http://www.google.com going through the proxy
Code:
Traceback (most recent call last):
  File "G:\udvikling\Git\anon-proxy-scraper\anonscraper.py", line 106, in <module>
    scrape_proxy_info('gb')
  File "G:\udvikling\Git\anon-proxy-scraper\anonscraper.py", line 51, in scrape_proxy_info
    working, time = test_proxy(ip)
  File "G:\udvikling\Git\anon-proxy-scraper\anonscraper.py", line 75, in test_proxy
    test = net.http_HEAD('http://www.google.com')
  File "G:\udvikling\Git\anon-proxy-scraper\t0mm0\common\net.py", line 225, in http_HEAD
    response = urllib2.urlopen(req)
  File "C:\Program Files\python\lib\urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "C:\Program Files\python\lib\urllib2.py", line 394, in open
    response = self._open(req, data)
  File "C:\Program Files\python\lib\urllib2.py", line 412, in _open
    '_open', req)
  File "C:\Program Files\python\lib\urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "C:\Program Files\python\lib\urllib2.py", line 1199, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "C:\Program Files\python\lib\urllib2.py", line 1170, in do_open
    r = h.getresponse(buffering=True)
  File "C:\Program Files\python\lib\httplib.py", line 1027, in getresponse
    response.begin()
  File "C:\Program Files\python\lib\httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "C:\Program Files\python\lib\httplib.py", line 371, in _read_status
    raise BadStatusLine(line)
BadStatusLine: ''

t0mm0 Wrote:i'm afraid i haven't done much on this code today as i've been slacking Wink

one thing i did try and fail at was using stacking for multi-part videos. looks like you can't use stack:// on plugin:// urls and i guess links would expire if you resolve them first. i guess i'll have to do something with playlists unless someone has any ideas?

t0mm0

I'm way to inexperienced to be of any use regarding stack:// or plugin:// Confused

I have put the 3 plugins I have done up on github, and for now until I'm satisfied that they actually work I'll keep them separate from urlresolver, so I can update them without having to ask you to update your git repository, but feel free to abuse / break them and let me know if you succeed Big Grin

https://github.com/DragonWin/xbmc-urlresolver-plugins



@rogerthis
I wish I could be of help there, but I'm just starting with the xbmc stuff and python ... it's way over my head ... all though I must agree that the functionality would be great to have available for addons.


- t0mm0 - 2011-08-21 22:34

DragonWin Wrote:It' coming along nicely it's now able to test the proxy's it has found, and choose the optimal proxy, and returns ip: port, location like:
Code:
184.73.63.96:80, United States  (Seattle)
cool Cool
DragonWin Wrote:I did run into an error that I'm unsure if it should be the net class that handles it.

Error showed itself a few times when I did a HEAD to http://www.google.com going through the proxy
not something i've come across before, don't think i can help much without running the code.


DragonWin Wrote:I have put the 3 plugins I have done up on github, and for now until I'm satisfied that they actually work I'll keep them separate from urlresolver, so I can update them without having to ask you to update your git repository, but feel free to abuse / break them and let me know if you succeed Big Grin

https://github.com/DragonWin/xbmc-urlresolver-plugins
great! i'll take a peek as soon as i get the chance.

though it will be easier for both you and me if you fork my repo and commit the plugins in there instead (make branches if you want to work on different things at the same time). then when you are happy with them you can send a pull request and github keeps track of everything.

thanks,

t0mm0


- Eldorado - 2011-08-22 18:13

t0mm0 Wrote:i did some playing around and have come up with a proposal....

without changing the interface at all it can be done fairly simply.

i was playing with the videoweed plugin. i changed valid_url() to:
Code:
def valid_url(self, web_url):
        return re.match('http://(www.)?videoweed.(es|com)/file/[0-9a-z]+',
                        web_url) or re.match('videoweed.*>.+',
                                             web_url, re.IGNORECASE)


and added the following to the top of get_media_url():
Code:
#construct url from video_id
        r = re.search('videoweed.*>(.+)', web_url, re.IGNORECASE)
        if r:
            video_id = r.group(1)
            web_url = 'http://videoweed.es/file/' + video_id

so you can call urlresolver.resolve('videoweed>crirmdz3tj116') or urlresolver.resolve('videoweed.com>crirmdz3tj116') for example.

i chose '>' as the separator as it is not valid in a URL. can anyone see a problem with this approach? if not i'll add this facility to the other plugins and document the convention. i think it might also be a good time to make a page in the docs with a list of plugins and what can be used to call them.

let me know what you think.....

t0mm0

Sounds good!

Are you restricted to do it this way, using > as the separator, or can you overload the method and do something like this:

urlresolver.resolve('videoweed.com', 'crirmdz3tj116')

And still at the same time allow it to be called supplying a standard full url:

urlresolver.resolve('http://www.videoweed.com/videoid=crirmdz3tj116')

I really need to start reading thru the urlresolver code Smile

Btw, excellent stuff so far, my addon is progressing quite well mainly due to making use of your common classes and of course the urlresolver, the majority of the time spent was really scraping of the site and learning how to do things in Python.. everything else from adding directories, video links, and of course playing video links was a piece of cake!


- whufclee - 2011-08-22 21:36

Just noticed you were talking about stacking different sources. I have no idea how he done it but whilst testing out all the plugins recently I found that the ones on AJ Repo all had this feature. You can choose to watch parts (as you can in all other plugins) but there's also an option to watch the whole thing by somehow stacking. It works a treat so if you want to take a look at his code I think all of his ones had this feature. His repo is hosted here http://code.google.com/p/apple-tv2-xbmc/source/browse/#svn%2Ftrunk%2Faddons


- t0mm0 - 2011-08-22 22:11

Eldorado Wrote:Sounds good!

Are you restricted to do it this way, using > as the separator, or can you overload the method and do something like this:

urlresolver.resolve('videoweed.com', 'crirmdz3tj116')

And still at the same time allow it to be called supplying a standard full url:

urlresolver.resolve('http://www.videoweed.com/videoid=crirmdz3tj116')

I really need to start reading thru the urlresolver code Smile
i think you might be right. i was trying to do as little change to the plugins as possible, and thought that doing it that way would mean writing new methods for each plugin and complicate the interface, but looking at it now i think it doesn't have to. i think it is still needed to send this to the plugin itself, but the urlresolver.resolve() could take two arguments and add the '>' before passing it on to the plugins.

the only issue i see is with the stuff like urlresolver.filter_list() and choose_source() etc. how would you pass 2 separate arguments to those? you could have a class that represents a host and video id or perhaps?

dunno, other ideas are welcome!
Eldorado Wrote:Btw, excellent stuff so far, my addon is progressing quite well mainly due to making use of your common classes and of course the urlresolver, the majority of the time spent was really scraping of the site and learning how to do things in Python.. everything else from adding directories, video links, and of course playing video links was a piece of cake!

thanks! that was the plan so i'm glad it's working out Wink i'm sure we can make it simpler though as people such as yourself think of different ways of using stuff or other useful bits of code that can be abstracted out of addons and into the modules so they can be reused.

whufclee Wrote:Just noticed you were talking about stacking different sources. I have no idea how he done it but whilst testing out all the plugins recently I found that the ones on AJ Repo all had this feature. You can choose to watch parts (as you can in all other plugins) but there's also an option to watch the whole thing by somehow stacking. It works a treat so if you want to take a look at his code I think all of his ones had this feature. His repo is hosted here http://code.google.com/p/apple-tv2-xbmc/source/browse/#svn%2Ftrunk%2Faddons

thanks for the pointer - i just had a look and it looks like he is using playlists too. all the code to do this is in t0mm0.common already so i will definitely go that way. i just thought i'd have a play with stack:// as it exists and i hadn't tried it before Wink

t0mm0.


- Eldorado - 2011-08-23 00:01

t0mm0 Wrote:i think you might be right. i was trying to do as little change to the plugins as possible, and thought that doing it that way would mean writing new methods for each plugin and complicate the interface, but looking at it now i think it doesn't have to. i think it is still needed to send this to the plugin itself, but the urlresolver.resolve() could take two arguments and add the '>' before passing it on to the plugins.

the only issue i see is with the stuff like urlresolver.filter_list() and choose_source() etc. how would you pass 2 separate arguments to those? you could have a class that represents a host and video id or perhaps?

dunno, other ideas are welcome!

For those 2 methods, would you be only passing in strings or would they be collections? (are they called dicts in python?)

t0mm0 Wrote:thanks! that was the plan so i'm glad it's working out Wink i'm sure we can make it simpler though as people such as yourself think of different ways of using stuff or other useful bits of code that can be abstracted out of addons and into the modules so they can be reused.t0mm0.

I'm sure this could expand quite a bit once more people jump in, especially if this can be added to the main repo

I'm starting to take a peak into the metadata code in Icefilms, another piece Anarchintosh was thinking of pulling out on it's own as part of Video Falcon.. hopefully we can eventually move onto that one and have a central meta data container for all addons that wish to use it!


- rogerthis - 2011-08-23 09:52

In this addon VODie http://code.google.com/p/xbmc-vodie/ there is a function for recently played files (and favourites too). This is another option that would be handy for t0mm0.common, for people with bad memories like myself.

It seems to be well coded, definitely worth a look.


- matmabro - 2011-08-23 17:11

iDude Wrote:I personally think that favorites should be handled by the addons separately. That's because many sites have a "favorites" section once you sign up. letmewatchthis.ch has this for example and it's quite useful because you can add whole TV shows to your favorites and they update themselves when a new episode comes out. This in combination with XBMC.MyLibrary and you have a SickBeard for streaming (Works with movies too)

I already modified t0mm0s addon so that instead of an a-z listing i get my favorite TV shows. Add a simple search filter for XBMC.MyLibrary and that's it. Now I can add tv shows on the road and don't need any servers running at home, don't need to download anything and everything just works automagically Laugh
This should also work with movies but I haven't tried it yet.

But the more I think it about the more it seems that this is maybe a site specific feature and should be handled separately. I don't know. Maybe I'm just talking nonsense (been awake for the last 36 hours)
iDude, could you briefly explain how you were implement this and get it to work? I have been messing around with XBMC.Mylibrary for a little bit and have had only limited success with it, although I think it's because I was trying to use Bluecop's Hulu plugin, which now seems ti be not really working. I think that if I could get the letmewatch.ch favorites into mylibrary, my wife would be one super happy camper.


- t0mm0 - 2011-08-23 22:14

rogerthis Wrote:In this addon VODie http://code.google.com/p/xbmc-vodie/ there is a function for recently played files (and favourites too). This is another option that would be handy for t0mm0.common, for people with bad memories like myself.

It seems to be well coded, definitely worth a look.

yes a very nice and simple favourites implementation - i might be persuaded to add something like that into t0mm0.common.addon after all Wink

is 'recently watched' useful? actually that might form the basis of a nice way to handle watched status too.... does anyone know what you need to set to get the tick that pops up on watched items?

by the way i am really busy this week so probably won't be able to actually write any more code till after the weekend Sad

t0mm0


- DragonWin - 2011-08-25 10:14

Hey T0mm0

Sorry for being absent on the project without any word, but unfortunately a customer called (same as last), and asked how fast I could be at the airport Eek, and they would arrange my travel plans. I'm still on the business trip, so I have not had any chance to look at forking / branching the code, but I will def. look into it once I'm back.

I'll also try and look into how I build Proxyscraper (anon proxy what ever we are going to call it) as a class that can be implemented into the common part or net class to be used by other addons. I'm thinking of just stealing the framework from the net class and work from there.

I'll look through all the post once I get back to catch all the tips you put there while I have been away Wink


Zshare plugin - Eldorado - 2011-08-25 19:42

I'm working on a ZShare plugin but having a bit of trouble

Code:

Code:
import re
from t0mm0.common.net import Net
import urllib2
from urlresolver import common
from urlresolver.plugnplay.interfaces import UrlResolver
from urlresolver.plugnplay.interfaces import PluginSettings
from urlresolver.plugnplay import Plugin

class ZShareResolver(Plugin, UrlResolver, PluginSettings):
    implements = [urlResolver, PluginSettings]
    name = "zshare"

    def __init__(self):
        p = self.get_setting('priority') or 100
        self.priority = int(p)
        self.net = Net()

    def get_media_url(self, web_url):
        #find key
        try:
            html = self.net.http_GET(web_url).content
        except urllib2.URLError, e:
            common.addon.log_error('zshare: got http error %d fetching %s' %
                                    (e.code, web_url))
            return False
      
        r = re.search('<iframe src="(.+?)" .+? id="videoframe"',html)

        if r:
            videourl = r.group(1)
        else:
            common.addon.log_error('zshare: filename and filekey not found')
            return False

        try:
            html = self.net.http_GET(videourl).content
        except urllib2.URLError, e:
            common.addon.log_error('zshare: got http error %d fetching %s' %
                                    (e.code, api))
            return False

        r = re.search('file: "(.+?)",',html)
        if r:
            stream_url = r.group(1) + '?start=0'
        else:
            common.addon.log_error('zshare: stream url not found')
            return False
            
        return stream_url
        
    def valid_url(self, web_url):
        return re.match('http://(www.)?zshare.net/video/(?:[0-9a-zA-Z]+)',
                        web_url)

Rather simple as I don't know how to do any fancy python stuff yet Smile

First - when I pass in zshare links to either urlresolver.choose_source() or urlresolver.filter_urls() they don't come back in the filtered list

**Edit - oops, was a typo error in my addon!




Second - the link I am returning doesn't seem to be playable when used via urlresolver

eg.

addon.add_video_item('http://www.zshare.net/video/93296873f03a68ec',
{'title': 'zshare'})

Logs show errors trying to open the stream

Code:
13:20:39 T:1740 M:1387261952  NOTICE: DVDPlayer: Opening: http://dl005.zshare.net/stream/e0fa06c3aa46583713037cbe84b53d78/93296873/1314292839/QSSDivX_lap-bdtc.avi.flv//5257303228/?start=0
13:20:39 T:1740 M:1387261952 WARNING: CDVDMessageQueue(player)::Put MSGQ_NOT_INITIALIZED
13:20:39 T:8036 M:1387261952   DEBUG: thread start, auto delete: 0
13:20:39 T:8036 M:1387261952  NOTICE: Creating InputStream
13:20:39 T:8036 M:1387261952   DEBUG: CFileCache::Open - opening <stream/e0fa06c3aa46583713037cbe84b53d78/93296873/1314292839/QSSDivX_lap-bdtc.avi.flv//5257303228/> using cache
13:20:39 T:8036 M:1387261952   DEBUG: Using proxy http://mcapproxy2:8080
13:20:39 T:8036 M:1387261952   DEBUG: FileCurl::Open(085E1928) http://dl005.zshare.net/stream/e0fa06c3aa46583713037cbe84b53d78/93296873/1314292839/QSSDivX_lap-bdtc.avi.flv//5257303228/?start=0
13:20:39 T:1740 M:1387266048   DEBUG: ------ Window Init (DialogBusy.xml) ------
13:20:39 T:8036 M:1387257856    INFO: CDVDFactorySubtitle::GetSubtitles, searching subtitles
13:20:39 T:8036 M:1387257856   DEBUG: CUtil::CacheSubtitles: START
13:20:39 T:8036 M:1387257856    INFO: CDVDFactorySubtitle::GetSubtitles, searching subtitles done
13:20:39 T:8036 M:1387257856  NOTICE: Creating Demuxer
13:20:39 T:2532 M:1387257856   DEBUG: thread start, auto delete: 0
13:20:39 T:2532 M:1387257856    INFO: CFileCache::Process - Hit eof.
13:20:39 T:8036 M:1387257856   DEBUG: CDVDDemuxFFmpeg::Open - probing failed, re-probing with probe size [2048]
13:20:39 T:8036 M:1387257856   DEBUG: CDVDDemuxFFmpeg::Open - probing failed, re-probing with probe size [4096]
13:20:39 T:8036 M:1387257856   DEBUG: CDVDDemuxFFmpeg::Open - probing failed, re-probing with probe size [8192]
13:20:39 T:8036 M:1387257856   ERROR: CDVDDemuxFFmpeg::Open - error probing input format, http://dl005.zshare.net/stream/e0fa06c3aa46583713037cbe84b53d78/93296873/1314292839/QSSDivX_lap-bdtc.avi.flv//5257303228/?start=0
13:20:39 T:8036 M:1387257856   ERROR: CDVDPlayer::OpenDemuxStream - Error creating demuxer

Yet when I put this same resolved link into a STRM file it plays fine:

Code:
13:20:15 T:1740 M:1408155648  NOTICE: DVDPlayer: Opening: http://dl005.zshare.net/stream/f16aaf43283b78078c651d5ffc00de7a/93296873/1314216558/QSSDivX_lap-bdtc.avi.flv//5256998104/?start=0
13:20:15 T:1740 M:1408151552 WARNING: CDVDMessageQueue(player)::Put MSGQ_NOT_INITIALIZED
13:20:15 T:2376 M:1408151552   DEBUG: thread start, auto delete: 0
13:20:15 T:2376 M:1408147456  NOTICE: Creating InputStream
13:20:15 T:2376 M:1408110592   DEBUG: CFileCache::Open - opening <stream/f16aaf43283b78078c651d5ffc00de7a/93296873/1314216558/QSSDivX_lap-bdtc.avi.flv//5256998104/> using cache
13:20:15 T:2376 M:1408110592   DEBUG: Using proxy http://mcapproxy2:8080
13:20:15 T:2376 M:1408110592   DEBUG: FileCurl::Open(036BFC70) http://dl005.zshare.net/stream/f16aaf43283b78078c651d5ffc00de7a/93296873/1314216558/QSSDivX_lap-bdtc.avi.flv//5256998104/?start=0
13:20:15 T:2376 M:1408114688    INFO: CDVDFactorySubtitle::GetSubtitles, searching subtitles
13:20:15 T:2376 M:1408110592   DEBUG: CUtil::CacheSubtitles: START
13:20:15 T:2760 M:1408110592   DEBUG: thread start, auto delete: 0
13:20:15 T:2376 M:1408110592    INFO: CDVDFactorySubtitle::GetSubtitles, searching subtitles done
13:20:15 T:2376 M:1408106496  NOTICE: Creating Demuxer
13:20:15 T:2376 M:1408081920   DEBUG: Win32DllLoader::Load(special://xbmcbin/system/players/dvdplayer/avutil-50.dll)
13:20:15 T:2376 M:1408057344   DEBUG: Win32DllLoader::Load(special://xbmcbin/system/players/dvdplayer/avcodec-52.dll)
13:20:15 T:2376 M:1407942656   DEBUG: Win32DllLoader::Load(special://xbmcbin/system/players/dvdplayer/libfaad.dll)
13:20:15 T:2376 M:1407922176   DEBUG: Win32DllLoader::Load(special://xbmcbin/system/players/dvdplayer/avformat-52.dll)
13:20:15 T:1740 M:1407807488   DEBUG: ------ Window Init (DialogBusy.xml) ------
13:20:15 T:1740 M:1407807488    INFO: Loading skin file: DialogBusy.xml
13:20:15 T:2376 M:1404284928   DEBUG: CDVDDemuxFFmpeg::Open - probing detected format [flv]
13:20:15 T:2376 M:1404252160   DEBUG: CDVDDemuxFFmpeg::Open - av_find_stream_info starting
13:20:15 T:2376 M:1404129280   DEBUG: CDVDDemuxFFmpeg::Open - av_find_stream_info finished
13:20:15 T:2376 M:1404129280    INFO: ffmpeg[948]: Input #0, flv, from 'http://dl005.zshare.net/stream/f16aaf43283b78078c651d5ffc00de7a/93296873/1314216558/QSSDivX_lap-bdtc.avi.flv//5256998104/?start=0':

Again.. something I'm missing? The url does look odd ending with '//5256998104/?start=0', but this is what I found works and is what URL Snooper reports as the file being played


- rogerthis - 2011-08-25 21:08

Hey Eldorado, I have been playing with your code, but I haven't found the solution.

When I add only this to my strm file
Code:
http://dl005.zshare.net/stream/3b53563b806f9d64901f3be33531c263/93296873/1314298987/QSSDivX_lap-bdtc.avi.flv//5257327820/?start=0
It wouldn't play for me. Did you add anything else to your strm file?

Also what was the typo? I started working on it before you changed it.


- t0mm0 - 2011-08-25 21:38

DragonWin Wrote:Hey T0mm0

Sorry for being absent on the project without any word, but unfortunately a customer called (same as last), and asked how fast I could be at the airport Eek, and they would arrange my travel plans. I'm still on the business trip, so I have not had any chance to look at forking / branching the code, but I will def. look into it once I'm back.

I'll also try and look into how I build Proxyscraper (anon proxy what ever we are going to call it) as a class that can be implemented into the common part or net class to be used by other addons. I'm thinking of just stealing the framework from the net class and work from there.

I'll look through all the post once I get back to catch all the tips you put there while I have been away Wink

no worries DragonWin, i am not able to do much coding this week either Sad though i should be back at full(-ish) speed after the weekend.

t0mm0


- t0mm0 - 2011-08-25 21:44

hi eldorado,

i had written something almost exactly the same a few weeks back which worked fine but i hadn't released it as i hadn't done much testing. i tried my old code and that failed too (which i was expecting as it was pretty much the same as yours!)

anyway, i looked at the requests in a browser and the only real difference seemed to be the http headers. so after a small bit of testing it seems they don't like the xbmc user-agent any more. replacing it with a normal browser user-agent seems to fix it:

Code:
import re
from t0mm0.common.net import Net
import urllib2
from urlresolver import common
from urlresolver.plugnplay.interfaces import UrlResolver
from urlresolver.plugnplay.interfaces import PluginSettings
from urlresolver.plugnplay import Plugin

class ZShareResolver(Plugin, UrlResolver, PluginSettings):
    implements = [urlResolver, PluginSettings]
    name = "zshare"

    def __init__(self):
        p = self.get_setting('priority') or 100
        self.priority = int(p)
        self.net = Net()

    def get_media_url(self, web_url):
        #find key
        try:
            html = self.net.http_GET(web_url).content
        except urllib2.URLError, e:
            common.addon.log_error('zshare: got http error %d fetching %s' %
                                    (e.code, web_url))
            return False
      
        r = re.search('<iframe src="(.+?)" .+? id="videoframe"',html)

        if r:
            videourl = r.group(1)
        else:
            common.addon.log_error('zshare: filename and filekey not found')
            return False

        try:
            html = self.net.http_GET(videourl).content
        except urllib2.URLError, e:
            common.addon.log_error('zshare: got http error %d fetching %s' %
                                    (e.code, api))
            return False

        r = re.search('file: "(.+?)",',html)
        if r:
            [color=red]stream_url = r.group(1) + '|user-agent=Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.109 Safari/535.1'[/color]
        else:
            common.addon.log_error('zshare: stream url not found')
            return False
            
        return stream_url
        
    def valid_url(self, web_url):
        return re.match('http://(www.)?zshare.net/video/(?:[0-9a-zA-Z]+)',
                        web_url)

is zshare fast enough to also stream the original file? might be worth adding as a setting if it is.....

t0mm0.


- Eldorado - 2011-08-25 22:49

rogerthis Wrote:Hey Eldorado, I have been playing with your code, but I haven't found the solution.

When I add only this to my strm file
Code:
http://dl005.zshare.net/stream/3b53563b806f9d64901f3be33531c263/93296873/1314298987/QSSDivX_lap-bdtc.avi.flv//5257327820/?start=0
It wouldn't play for me. Did you add anything else to your strm file?

Also what was the typo? I started working on it before you changed it.

Sorry I should have written more, I had a typo in my addon which calls urlresolver, not related to the plugin