Kodi Community Forum
script.module.urlresolver development - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32)
+--- Forum: Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=26)
+--- Thread: script.module.urlresolver development (/showthread.php?tid=105707)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28


- Eldorado - 2011-08-25

t0mm0 Wrote:hi eldorado,

i had written something almost exactly the same a few weeks back which worked fine but i hadn't released it as i hadn't done much testing. i tried my old code and that failed too (which i was expecting as it was pretty much the same as yours!)

anyway, i looked at the requests in a browser and the only real difference seemed to be the http headers. so after a small bit of testing it seems they don't like the xbmc user-agent any more. replacing it with a normal browser user-agent seems to fix it:

Perfect, that seems to have fixed it!

Should I bother submitting this one if you have already done one?

t0mm0 Wrote:is zshare fast enough to also stream the original file? might be worth adding as a setting if it is.....

t0mm0.

Not sure what you mean by your last question, original file?


- t0mm0 - 2011-08-25

Eldorado Wrote:Perfect, that seems to have fixed it!

Should I bother submitting this one if you have already done one?
please do - mine is much more messy and is from before i started adding the error handling stuff Wink
Eldorado Wrote:Not sure what you mean by your last question, original file?
if you click download on the zshare page you can download the full quality avi file (like megaupload) instead of the low quality flash version - just tried it on the link you posted earlier and seem to get good enough speed to stream the high quality version so it's probably worth adding that. maybe you can make a setting to choose whether you want flash or original file?


- Eldorado - 2011-08-26

t0mm0 Wrote:please do - mine is much more messy and is from before i started adding the error handling stuff Wink

if you click download on the zshare page you can download the full quality avi file (like megaupload) instead of the low quality flash version - just tried it on the link you posted earlier and seem to get good enough speed to stream the high quality version so it's probably worth adding that. maybe you can make a setting to choose whether you want flash or original file?

Ah very nice, never noticed that!

Will do

Edit - looks like might be a bit more work than I had thought, can anyone watching give some hints on how to build the final download url or get me close? Seems to work much in the way of MegaUpload, download counter of 50 seconds for unregistered users, shorter wait time if you are registered etc..

I did find one other addon with a zshare handler mentioned earlier, it might be slightly out of date but not too far off

Zshare: http://code.google.com/p/xbmc-xstream-plugin/source/browse/trunk/plugins/video/plugin.video.xstream/hosters/zshare.py
Handler: http://code.google.com/p/xbmc-xstream-plugin/source/browse/trunk/plugins/video/plugin.video.xstream/resources/lib/handler/requestHandler.py

Looks like he builds a URL (if i'm reading it right) of something like:

Code:
www.zshare.net/download/<videoid>&referer2=<blank?>&download=1&imagefield.x=76&imagefield.y=28

Which doesn't seem to get me anywhere right now, but I do see these values being used in the page source


- Eldorado - 2011-08-26

Ok I think I found it

Going to this page: http://www.zshare.net/download/93296873f03a68ec

I have a form submit button, which gives me a new page (same address).. and I can grab the link prior to the countdown finishing

They try to hide it in an array:

Code:
var link_enc=new Array('h','t','t','p',':','/','/','d','l','0','0','5','.','z','s','h','a','r','e','.','n','e','t','/','d','o','w','n','l','o','a','d','/','a','4','7','4','c','7','d','1','e','5','0','6','4','d','0','2','c','3','1','2','d','e','6','2','8','9','5','6','f','a','2','b','/','1','3','1','4','3','6','9','7','2','2','/','9','3','2','9','6','8','7','3','/','Q','S','S','D','i','v','X','_','l','a','p','-','b','d','t','c','.','a','v','i');

So just need to figure out how to do the submit form button to get me to this page..


- rogerthis - 2011-08-26

Eldorado Wrote:Ok I think I found it

Going to this page: http://www.zshare.net/download/93296873f03a68ec

I have a form submit button, which gives me a new page (same address).. and I can grab the link prior to the countdown finishing

They try to hide it in an array:

Code:
var link_enc=new Array('h','t','t','p',':','/','/','d','l','0','0','5','.','z','s','h','a','r','e','.','n','e','t','/','d','o','w','n','l','o','a','d','/','a','4','7','4','c','7','d','1','e','5','0','6','4','d','0','2','c','3','1','2','d','e','6','2','8','9','5','6','f','a','2','b','/','1','3','1','4','3','6','9','7','2','2','/','9','3','2','9','6','8','7','3','/','Q','S','S','D','i','v','X','_','l','a','p','-','b','d','t','c','.','a','v','i');

So just need to figure out how to do the submit form button to get me to this page..

It's not in nice url resolver code, but this is able to access the page:
Code:
import urllib2, urllib, re

url = 'http://www.zshare.net/download/93296873f03a68ec'

req = urllib2.Request(url)
req.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3')
response = urllib2.urlopen(req)
link2=response.read()
values = {'referer2' : ' ', 'download' : '1', 'imageField.x' : '120', 'imageField.y' : '37'}
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
headers = { 'User-Agent' : user_agent }
data = urllib.urlencode(values)
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
html = response.read()
p = re.compile('enc=new Array((.+?));').findall(html)[0]
print p



- Eldorado - 2011-08-26

Yep just got it worked out!

Though I'm having troubles getting the page using t0mm0's net.http_GET()

Currently is not returning the proper page, but it is in my test script

Full code:
Code:
import re
from t0mm0.common.net import Net
import urllib2
from urlresolver import common
from urlresolver.plugnplay.interfaces import UrlResolver
from urlresolver.plugnplay.interfaces import PluginSettings
from urlresolver.plugnplay import Plugin

class ZShareResolver(Plugin, UrlResolver, PluginSettings):
    implements = [urlResolver, PluginSettings]
    name = "zshare"
    
    def __init__(self):
        p = self.get_setting('priority') or 100
        self.priority = int(p)
        self.net = Net()

    def get_media_url(self, web_url):

        #check quality setting
        q = int(self.get_setting('q'))  
        print 'Q=' + str(q)    
        
        #if standard FLV
        if q == 0:
        
            #retrieve URL
            try:
                html = self.net.http_GET(web_url).content
            except urllib2.URLError, e:
                common.addon.log_error('zshare: got http error %d fetching %s' %
                                        (e.code, web_url))
                return False
                
            r = re.search('<iframe src="(.+?)" .+? id="videoframe"',html)
    
            if r:
                videourl = r.group(1)
            else:
                common.addon.log_error('zshare: filename not found')
                return False
    
            try:
                html = self.net.http_GET(videourl).content
            except urllib2.URLError, e:
                common.addon.log_error('zshare: got http error %d fetching %s' %
                                        (e.code, api))
                return False
    
            r = re.search('file: "(.+?)",',html)
            if r:
                stream_url = r.group(1) + '|user-agent=Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.109 Safari/535.1'
    
            else:
                common.addon.log_error('zshare: stream url not found')
                return False

        #if higher quality AVI
        if q == 1:
            
            #create http headers
            header = {}
            header['referer2'] = ''
            header['download'] = 1
            header['imageField.x'] = 76
            header['imageField.y'] = 28
            
            #url requires /download/
            web_url = web_url.replace('/video/','/download/')
            
            #retrieve URL
            try:
                html = self.net.http_GET(web_url, header).content
            except urllib2.URLError, e:
                common.addon.log_error('zshare: got http error %d fetching %s' %
                                        (e.code, web_url))
                return False
            
            r = re.compile('new Array.(.+?).;').findall(html)
            
            if r:
                stream_url = r[0].replace("'","").replace(",","")
    
            else:
                common.addon.log_error('zshare: stream url not found')
                return False
                      
        return stream_url  
        
    def valid_url(self, web_url):
        return re.match('http://(www.)?zshare.net/video/(?:[0-9a-zA-Z]+)',
                        web_url)
                        
    def get_settings_xml(self):
        xml = PluginSettings.get_settings_xml(self)
        xml += '<setting label="Highest Quality" id="ZShareResolver_q" '
        xml += 'type="enum" values="FLV|AVI" default="0" />\n'
        return xml



- t0mm0 - 2011-08-26

Eldorado Wrote:Yep just got it worked out!

Though I'm having troubles getting the page using t0mm0's net.http_GET()

Currently is not returning the proper page, but it is in my test script

without running the code but comparing it to rogerthis code, i think you want to use http_POST() instead of http_GET(). the 'header' variable you are creating is not actually headers but form data to POST.

urllib2 POSTs automatically when you pass it form data whereas my stuff expects you to explicitly choose GET or POST (i did that to try and make it clearer what the code does)

t0mm0


- DragonWin - 2011-08-27

Eldorado Wrote:Yep just got it worked out!

Though I'm having troubles getting the page using t0mm0's net.http_GET()

Currently is not returning the proper page, but it is in my test script

T0mm0 is right (surprise Laugh)

Code:
<td class="text1"><form name="form1" method="post" action="">
<input name="referer2" type="hidden" id="referer2" value="http://forum.xbmc.org/showthread.php?tid=105707&page=13">
<input name="download" type="hidden" id="download" value="1">
<input name="imageField" type="image" id="imageField" src="/images/download.gif" width="219" height="57" border="0">
</form></td>

Post data from a tcpdump
Code:
referer2=http%3A%2F%2Fforum.xbmc.org%2Fshowthread.php%3Ft%3D105707%26page%3D13&download=1&imageField.x=109&imageField.y=27

So some thing along the lines of
Code:
form_data = { 'referer2' : http://www.google.com, 'download' : 1, 'imageField.x' : 109, 'imageField.y : 27}
postdata = self.net.http_POST(web_url, form_data).content



- DragonWin - 2011-08-27

T0mm0

I think I have to scratch the proxyscraper project. I have run into some issues I can't see how to over come.

1) Antivirus
The antivirus manufacture have marked 99% of the sites I have found as dangerous, so if xbmc is running on a machine with TrendMicro, McAfee, Norten, it will not work unless the end user allows the explicit domain, in the antivirus.

End users should never be bothered with such stuff.

2) Proxy lists are flaky
Even though they look good, 90% of the proxies are dead. Taken into consideration that we only want anonymous or highly anonymous proxies for this to work (eg. no x-forwarder or via in the header), the number of working proxies diminish to almost 0


If I ever find out how to circumvent the antivirus issue I might take it up again. I were trying to put it along side net and addon under common.

Ohh well I'll save the code just in case.


- t0mm0 - 2011-08-27

DragonWin Wrote:T0mm0

I think I have to scratch the proxyscraper project. I have run into some issues I can't see how to over come.

1) Antivirus
The antivirus manufacture have marked 99% of the sites I have found as dangerous, so if xbmc is running on a machine with TrendMicro, McAfee, Norten, it will not work unless the end user allows the explicit domain, in the antivirus.

End users should never be bothered with such stuff.

2) Proxy lists are flaky
Even though they look good, 90% of the proxies are dead. Taken into consideration that we only want anonymous or highly anonymous proxies for this to work (eg. no x-forwarder or via in the header), the number of working proxies diminish to almost 0


If I ever find out how to circumvent the antivirus issue I might take it up again. I were trying to put it along side net and addon under common.

Ohh well I'll save the code just in case.

that's a shame. nice idea though DragonWin. (i never have an issue with antivirus software as i use linux so i sometimes forget about that sort of stuff Wink)

i think we still need to think of a nice way of handling proxies though.

do we need to be able to specify proxies for each resolver plugin? i think probably yes (especially if we add resolvers for things like tv network websites that restrict access to their own countries), but i guess most resolver plugins will not need proxy settings.

i wonder if it would be a nice idea to have a central proxy setting where you could set up custom proxies (or x-forwarded-for settings) for any number of countries, sort of like 'custom presets', which you could easily choose to use from each plugin's settings (so if you had several plugins that required US proxies for example, you would just set a US preset centrally, then each plugin that requied US proxies you would just choose the US preset rather than having to enter the details into each plugin).

any thoughts on whether that seems like a good idea and how it might be implemented would be most welcome! (if any of it makes sense of course Wink)

i'm still busy with other non-xbmc stuff at the moment but hope to start writing more code again on monday.

thanks,

t0mm0.


- DragonWin - 2011-08-27

t0mm0 Wrote:i think we still need to think of a nice way of handling proxies though.

do we need to be able to specify proxies for each resolver plugin? i think probably yes (especially if we add resolvers for things like tv network websites that restrict access to their own countries), but i guess most resolver plugins will not need proxy settings.

i wonder if it would be a nice idea to have a central proxy setting where you could set up custom proxies (or x-forwarded-for settings) for any number of countries, sort of like 'custom presets', which you could easily choose to use from each plugin's settings (so if you had several plugins that required US proxies for example, you would just set a US preset centrally, then each plugin that requied US proxies you would just choose the US preset rather than having to enter the details into each plugin).

any thoughts on whether that seems like a good idea and how it might be implemented would be most welcome! (if any of it makes sense of course Wink)

i'm still busy with other non-xbmc stuff at the moment but hope to start writing more code again on monday.

thanks,

t0mm0.

If we put the functions in common.addon, to make it available for authors not using urlresolver, and then in urlresolver made use of those functions to create a preset for different countries (only seen US and GB used so far), would make it very handy to have.

I like the idea of having a "master" setting for different countries. Each plugin that requires a proxy from eg. US, would have to "announce" to urlresolver that it's a member of the "need US proxy" group.

[edit]
btw, I never did find a way to force the player to use a proxy, so far I was only able to get the http requests to use a proxy, so we would have to figure out if those sites that require a proxy will allow the streaming directly. Otherwise I don't see the point Sad
[/edit]


Just some thing else I feel I'm lacking in the common.addon, and that's a small popup like icefilms has when a user logs into megaupload. Think I'm going to see if I can't stuff that in there some where.


- t0mm0 - 2011-08-27

DragonWin Wrote:If we put the functions in common.addon, to make it available for authors not using urlresolver, and then in urlresolver made use of those functions to create a preset for different countries (only seen US and GB used so far), would make it very handy to have.

I like the idea of having a "master" setting for different countries. Each plugin that requires a proxy from eg. US, would have to "announce" to urlresolver that it's a member of the "need US proxy" group.
yes i guess that would fit nicely in t0mm0.common.net but it means we will have to have a settings page for t0mm0.common.net too.

i'm sure we can come up with something workable....

DragonWin Wrote:Just some thing else I feel I'm lacking in the common.addon, and that's a small popup like icefilms has when a user logs into megaupload. Think I'm going to see if I can't stuff that in there some where.

yeah i already have some dialog box stuff in t0mm0.common.addon so other stuff like that would be good too.

t0mm0.


- DragonWin - 2011-08-28

I'm having a bit of a hard time with the urlresolver.choose_source(sources)

I'm trying to figure out how I can make it play directly from that choice, currently when I choose a source for the video it just gives me a new directory with the same links once more.

The addon itself is based on the test addon.

This source is very messy as I have been trying to figure this out for a while now.
Code:
html = net.http_GET(addon.queries['url']).content
    count = 0
    sources = {}
    expr = re.compile('[a|;"]\s\shref="/link/.+/(\d+)/">(.+)</a>')
    match = expr.findall(html)
    if len(match) > 0:
        for linkid, name in match:
            if debug >= 1:
                msg = 'Solarmovie1: Found ' + name + ' = ' + linkid
                addon.log_error(msg)
            url = base_url + '/movie/playlink/id/' + linkid + '/part/1/'
            url, title = FindIframeLink(url,name)
            if url: # url here is the hosting provider url.
                pl = addon.get_playlist(1)
                #addon.add_item(url, {'title' : title })
                addon.add_item(url, {'title' : title }, 'special://foo', 'specisal://bar', False, 0, pl)
                sources[url] = title
                count += 1
                if count > 5:
                    break
        stream_url = urlresolver.choose_source(sources)
        addon.resolve_url(stream_url)
        
    else:
        if debug >= 1:
            myerror = 'missed: ' + addon.queries['url']
            addon.log_error(myerror)



- t0mm0 - 2011-08-28

DragonWin Wrote:I'm having a bit of a hard time with the urlresolver.choose_source(sources)

I'm trying to figure out how I can make it play directly from that choice, currently when I choose a source for the video it just gives me a new directory with the same links once more.

The addon itself is based on the test addon.

This source is very messy as I have been trying to figure this out for a while now.
Code:
html = net.http_GET(addon.queries['url']).content
    count = 0
    sources = {}
    expr = re.compile('[a|;"]\s\shref="/link/.+/(\d+)/">(.+)</a>')
    match = expr.findall(html)
    if len(match) > 0:
        for linkid, name in match:
            if debug >= 1:
                msg = 'Solarmovie1: Found ' + name + ' = ' + linkid
                addon.log_error(msg)
            url = base_url + '/movie/playlink/id/' + linkid + '/part/1/'
            url, title = FindIframeLink(url,name)
            if url: # url here is the hosting provider url.
                pl = addon.get_playlist(1)
                #addon.add_item(url, {'title' : title })
                addon.add_item(url, {'title' : title }, 'special://foo', 'specisal://bar', False, 0, pl)
                sources[url] = title
                count += 1
                if count > 5:
                    break
        stream_url = urlresolver.choose_source(sources)
        addon.resolve_url(stream_url)
        
    else:
        if debug >= 1:
            myerror = 'missed: ' + addon.queries['url']
            addon.log_error(myerror)

check out the letmewatchthis addon which uses choose_source()

not sure what part of the addon you have got that code, but you seem to be doing addon.add_item() as well as urlresolver.choose_source()

you should have one part that adds video items (https://github.com/t0mm0/xbmc-urlresolver/blob/d82472e76bd9fb66dbdbd3079bf0b8866d9291a0/plugin.video.letmewatchthis/default.py#L177) - this will result in one entry in the xbmc list for each episode/video

then when you want to actually play a video you make the dictionary of hoster urls and titles for that one episode/video and pass it to urlresolver.choose_source() (https://github.com/t0mm0/xbmc-urlresolver/blob/d82472e76bd9fb66dbdbd3079bf0b8866d9291a0/plugin.video.letmewatchthis/default.py#L40)

hope that makes sense!

t0mm0


- k1m0s - 2011-08-29

First off I must say, Probably the best creation for making addons ever. Really appreciate your guys work. Makes thing easier on a meat head like me. I spet a cpl weeks trying to get megaup file to play with no luck at all till I tried this and it took just minutes. I have been messing with the test addon and was wondering if there is a way to add icons to the links in the format given
Quote:addon.add_video_item('http://www.megaupload.com/?d=TQPQJM5H',
{'title': 'megaupload'})
addon.add_video_item('http://www.megavideo.com/?v=LYWNYM1J',
{'title': 'megavideo'})
Thanks again for all the hard work fellas. Smile