XBMC Community Forum
script.module.urlresolver development - Printable Version

+- XBMC Community Forum (http://forum.xbmc.org)
+-- Forum: Development (/forumdisplay.php?fid=32)
+--- Forum: Python Add-on Development (/forumdisplay.php?fid=26)
+--- Thread: script.module.urlresolver development (/showthread.php?tid=105707)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31


- t0mm0 - 2011-08-21 13:15

DragonWin Wrote:Javascript I found
Code:
</script></td></tr></table><script type="text/javascript">h8z6=2391;n4t0=5349;f6x4=4457;g7j0=3397;w3s9=6722;o5b2=6330;c3h8=6005;p6d4=55​21;k1a1=5412;d4y5=1599;m3q7r8=0^h8z6;r8h8z6=1^n4t0;s9w3p6=2^f6x4;x4k1e5=3^g7j0;q​7p6y5=4^w3s9;j0t0c3=5^o5b2;w3m3q7=6^c3h8;l2z6k1=7^p6d4;e5d4m3=8^k1a1;h8n4t0=9^d4​y5;</script>

<script type="text/javascript">document.write("<font class=spy2>:<\/font>"+(r8h8z6^n4t0)+(s9w3p6^f6x4)+(s9w3p6^f6x4)+(r8h8z6^n4t0))</script>

^ is a bitwise XOR (in both python and javascript) which (rather handily!) is it's own inverse function (so if you do a bitwies XOR twice you end up back where you started).

so for example the first number is found using:
Code:
n4t0=5349
r8h8z6=1^n4t0
we want:
Code:
x = r8h8z6^n4t0
so we can combine to get:
Code:
x = 1^n4t0^n4t0
the 2 bitwise XORs cancel each other out! so we end up with
Code:
x = 1

so basically no calculations are required at all - all you have to do is grab the numbers out of the initial javascript line and it becomes a simple process of substitution to find the port number.

Code:
...
[b]m3q7r8=0[/b]^h8z6;[b]r8h8z6=1[/b]^n4t0;[b]s9w3p6=2[/b]^f6x4;[b]x4k1e5=3[/b]^g7j0;[b]q7p6y5=4[/b]^w3s9;[b]j0t0c3=5[/b]^o5b2;[b]w3m3q7=6[/b]^c3h8;[b]l2z6k1=7[/b]^p6d4;[b]e5d4m3=8[/b]^k1a1;[b]h8n4t0=9[/b]^d4y5
...

i knew all those maths classes would be useful some time Wink

t0mm0


- DragonWin - 2011-08-21 13:29

haha awesome thanks :-)


- DragonWin - 2011-08-21 19:18

It's getting there, at least I seem to have the US proxies under control now. Below is a sample output of my ehmm Eek I think it's called dict in python but hash with hashes Cool

As you can see it contains more information than what it just required right now, but it might come in handy at a later stage, like making a filter on proxytype (HIA = high anonymity, ANM = Anonymous, NOA = no anonymity at all)

The socket timeout were set to 2 sec during that run, to also test for timeouts etc.

Code:
{'208.101.63.210:8118': {'proxytype': 'HIA', 'location': 'United States  (Dallas)', 'testtime': 0.312000036239624}, '205.209.188.235:80': {'proxytype': 'NOA', 'location': 'United States  (Redwood City)', 'testtime': 0.38499999046325684}, '173.0.50.237:3128': {'proxytype': 'HIA', 'location': 'United States  (Kansas City)', 'testtime': 1.8899998664855957}, '198.36.222.8:80': {'proxytype': 'ANM', 'location': 'United States  (Rush City)', 'testtime': 0.31799983978271484}, '204.93.211.219:80': {'proxytype': 'ANM', 'location': 'United States  (Skokie)', 'testtime': 0.41899991035461426}, '69.114.243.229:8909': {'proxytype': 'HIA', 'location': 'United States  (Brooklyn)', 'testtime': 0.42099976539611816}, '198.36.222.8:3128': {'proxytype': 'ANM', 'location': 'United States  (Rush City)', 'testtime': 1.0750000476837158}}

With a bit of luck I can reuse the function for all the other countries.


- rogerthis - 2011-08-21 20:00

I posted here http://forum.xbmc.org/showthread.php?tid=108110 about adding metadata and watched status through to addons, like favourites is.

I think both functionality would be a big advantage to our addons. It would save a lot of work on our side, and it is something that would hopefully be easy enough to implement on there side. I might be the fact that they are presently blocking it to addons, I don't know.

I have put it up as a feature request http://forum.xbmc.org/showthread.php?tid=108304. Please feel free to make any comments you think are needed.


- t0mm0 - 2011-08-21 21:33

DragonWin Wrote:It's getting there, at least I seem to have the US proxies under control now. Below is a sample output of my ehmm Eek I think it's called dict in python but hash with hashes Cool

As you can see it contains more information than what it just required right now, but it might come in handy at a later stage, like making a filter on proxytype (HIA = high anonymity, ANM = Anonymous, NOA = no anonymity at all)
looks like you're making some interesting progress there!
rogerthis Wrote:I posted here http://forum.xbmc.org/showthread.php?tid=108110 about adding metadata and watched status through to addons, like favourites is.

I think both functionality would be a big advantage to our addons. It would save a lot of work on our side, and it is something that would hopefully be easy enough to implement on there side. I might be the fact that they are presently blocking it to addons, I don't know.

I have put it up as a feature request http://forum.xbmc.org/showthread.php?tid=108304. Please feel free to make any comments you think are needed.
i don't know enough about the metadata stuff but the watched status would be really nice. i can't think of a way of doing it in the addon itself other than marking as watched as soon as you press play (using setResolvedUrl() you can't specify what player is used (i posted a thread about that a while ago) so i don't think you can use your own sub-classed player to mark something as watched when it gets near the end) but maybe that would be ok with an option in the context menu to 'mark as unwatched'?

i'm afraid i haven't done much on this code today as i've been slacking Wink

one thing i did try and fail at was using stacking for multi-part videos. looks like you can't use stack:// on plugin:// urls and i guess links would expire if you resolve them first. i guess i'll have to do something with playlists unless someone has any ideas?

t0mm0


- DragonWin - 2011-08-21 22:15

t0mm0 Wrote:looks like you're making some interesting progress there!

It' coming along nicely it's now able to test the proxy's it has found, and choose the optimal proxy, and returns ip: port, location like:
Code:
184.73.63.96:80, United States  (Seattle)

I did run into an error that I'm unsure if it should be the net class that handles it.

Error showed itself a few times when I did a HEAD to http://www.google.com going through the proxy
Code:
Traceback (most recent call last):
  File "G:\udvikling\Git\anon-proxy-scraper\anonscraper.py", line 106, in <module>
    scrape_proxy_info('gb')
  File "G:\udvikling\Git\anon-proxy-scraper\anonscraper.py", line 51, in scrape_proxy_info
    working, time = test_proxy(ip)
  File "G:\udvikling\Git\anon-proxy-scraper\anonscraper.py", line 75, in test_proxy
    test = net.http_HEAD('http://www.google.com')
  File "G:\udvikling\Git\anon-proxy-scraper\t0mm0\common\net.py", line 225, in http_HEAD
    response = urllib2.urlopen(req)
  File "C:\Program Files\python\lib\urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "C:\Program Files\python\lib\urllib2.py", line 394, in open
    response = self._open(req, data)
  File "C:\Program Files\python\lib\urllib2.py", line 412, in _open
    '_open', req)
  File "C:\Program Files\python\lib\urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "C:\Program Files\python\lib\urllib2.py", line 1199, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "C:\Program Files\python\lib\urllib2.py", line 1170, in do_open
    r = h.getresponse(buffering=True)
  File "C:\Program Files\python\lib\httplib.py", line 1027, in getresponse
    response.begin()
  File "C:\Program Files\python\lib\httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "C:\Program Files\python\lib\httplib.py", line 371, in _read_status
    raise BadStatusLine(line)
BadStatusLine: ''

t0mm0 Wrote:i'm afraid i haven't done much on this code today as i've been slacking Wink

one thing i did try and fail at was using stacking for multi-part videos. looks like you can't use stack:// on plugin:// urls and i guess links would expire if you resolve them first. i guess i'll have to do something with playlists unless someone has any ideas?

t0mm0

I'm way to inexperienced to be of any use regarding stack:// or plugin:// Confused

I have put the 3 plugins I have done up on github, and for now until I'm satisfied that they actually work I'll keep them separate from urlresolver, so I can update them without having to ask you to update your git repository, but feel free to abuse / break them and let me know if you succeed Big Grin

https://github.com/DragonWin/xbmc-urlresolver-plugins



@rogerthis
I wish I could be of help there, but I'm just starting with the xbmc stuff and python ... it's way over my head ... all though I must agree that the functionality would be great to have available for addons.


- t0mm0 - 2011-08-21 22:34

DragonWin Wrote:It' coming along nicely it's now able to test the proxy's it has found, and choose the optimal proxy, and returns ip: port, location like:
Code:
184.73.63.96:80, United States  (Seattle)
cool Cool
DragonWin Wrote:I did run into an error that I'm unsure if it should be the net class that handles it.

Error showed itself a few times when I did a HEAD to http://www.google.com going through the proxy
not something i've come across before, don't think i can help much without running the code.


DragonWin Wrote:I have put the 3 plugins I have done up on github, and for now until I'm satisfied that they actually work I'll keep them separate from urlresolver, so I can update them without having to ask you to update your git repository, but feel free to abuse / break them and let me know if you succeed Big Grin

https://github.com/DragonWin/xbmc-urlresolver-plugins
great! i'll take a peek as soon as i get the chance.

though it will be easier for both you and me if you fork my repo and commit the plugins in there instead (make branches if you want to work on different things at the same time). then when you are happy with them you can send a pull request and github keeps track of everything.

thanks,

t0mm0


- Eldorado - 2011-08-22 18:13

t0mm0 Wrote:i did some playing around and have come up with a proposal....

without changing the interface at all it can be done fairly simply.

i was playing with the videoweed plugin. i changed valid_url() to:
Code:
def valid_url(self, web_url):
        return re.match('http://(www.)?videoweed.(es|com)/file/[0-9a-z]+',
                        web_url) or re.match('videoweed.*>.+',
                                             web_url, re.IGNORECASE)


and added the following to the top of get_media_url():
Code:
#construct url from video_id
        r = re.search('videoweed.*>(.+)', web_url, re.IGNORECASE)
        if r:
            video_id = r.group(1)
            web_url = 'http://videoweed.es/file/' + video_id

so you can call urlresolver.resolve('videoweed>crirmdz3tj116') or urlresolver.resolve('videoweed.com>crirmdz3tj116') for example.

i chose '>' as the separator as it is not valid in a URL. can anyone see a problem with this approach? if not i'll add this facility to the other plugins and document the convention. i think it might also be a good time to make a page in the docs with a list of plugins and what can be used to call them.

let me know what you think.....

t0mm0

Sounds good!

Are you restricted to do it this way, using > as the separator, or can you overload the method and do something like this:

urlresolver.resolve('videoweed.com', 'crirmdz3tj116')

And still at the same time allow it to be called supplying a standard full url:

urlresolver.resolve('http://www.videoweed.com/videoid=crirmdz3tj116')

I really need to start reading thru the urlresolver code Smile

Btw, excellent stuff so far, my addon is progressing quite well mainly due to making use of your common classes and of course the urlresolver, the majority of the time spent was really scraping of the site and learning how to do things in Python.. everything else from adding directories, video links, and of course playing video links was a piece of cake!


- whufclee - 2011-08-22 21:36

Just noticed you were talking about stacking different sources. I have no idea how he done it but whilst testing out all the plugins recently I found that the ones on AJ Repo all had this feature. You can choose to watch parts (as you can in all other plugins) but there's also an option to watch the whole thing by somehow stacking. It works a treat so if you want to take a look at his code I think all of his ones had this feature. His repo is hosted here http://code.google.com/p/apple-tv2-xbmc/source/browse/#svn%2Ftrunk%2Faddons


- t0mm0 - 2011-08-22 22:11

Eldorado Wrote:Sounds good!

Are you restricted to do it this way, using > as the separator, or can you overload the method and do something like this:

urlresolver.resolve('videoweed.com', 'crirmdz3tj116')

And still at the same time allow it to be called supplying a standard full url:

urlresolver.resolve('http://www.videoweed.com/videoid=crirmdz3tj116')

I really need to start reading thru the urlresolver code Smile
i think you might be right. i was trying to do as little change to the plugins as possible, and thought that doing it that way would mean writing new methods for each plugin and complicate the interface, but looking at it now i think it doesn't have to. i think it is still needed to send this to the plugin itself, but the urlresolver.resolve() could take two arguments and add the '>' before passing it on to the plugins.

the only issue i see is with the stuff like urlresolver.filter_list() and choose_source() etc. how would you pass 2 separate arguments to those? you could have a class that represents a host and video id or perhaps?

dunno, other ideas are welcome!
Eldorado Wrote:Btw, excellent stuff so far, my addon is progressing quite well mainly due to making use of your common classes and of course the urlresolver, the majority of the time spent was really scraping of the site and learning how to do things in Python.. everything else from adding directories, video links, and of course playing video links was a piece of cake!

thanks! that was the plan so i'm glad it's working out Wink i'm sure we can make it simpler though as people such as yourself think of different ways of using stuff or other useful bits of code that can be abstracted out of addons and into the modules so they can be reused.

whufclee Wrote:Just noticed you were talking about stacking different sources. I have no idea how he done it but whilst testing out all the plugins recently I found that the ones on AJ Repo all had this feature. You can choose to watch parts (as you can in all other plugins) but there's also an option to watch the whole thing by somehow stacking. It works a treat so if you want to take a look at his code I think all of his ones had this feature. His repo is hosted here http://code.google.com/p/apple-tv2-xbmc/source/browse/#svn%2Ftrunk%2Faddons

thanks for the pointer - i just had a look and it looks like he is using playlists too. all the code to do this is in t0mm0.common already so i will definitely go that way. i just thought i'd have a play with stack:// as it exists and i hadn't tried it before Wink

t0mm0.