How do you send cookies with urllib?

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
SofaKng Offline
Donor
Posts: 388
Joined: Mar 2005
Reputation: 0
Post: #1
i need to send cookies to a website that i'm connecting to and i was told that i could use cachedhttp.py to accomplish this. i've tried searching and looking at the documentation but i still can't figure it out.

can anybody help me? all i want is to send some cookies (like 4) every time i connect with urllib. (or i could switch to urllib2 or whatever....)
find quote
Bernd Offline
Senior Member
Posts: 211
Joined: Mar 2005
Reputation: 0
Post: #2
i encourage every xbmc script author to use cachedhttp.py!!!
its an easy and ready to use lib which provides lots of functionality ready to use incl cookies, user agent and timeouts.

bernd
find quote
SofaKng Offline
Donor
Posts: 388
Joined: Mar 2005
Reputation: 0
Post: #3
is there any documentation for this? (or really basic examples?)

i've taken a look at it but couldn't figure out how to use it...
find quote
BigBellyBilly Offline
Skilled Python Coder
Posts: 900
Joined: Feb 2005
Reputation: 2
Location: UK
Post: #4
it can be done with urllib.urlopener()

Quote:cookies = [
['cookie', 'cfid=768572'],
['cookie', 'cftoken=45453733'],
['cookie', 'person_id=1686446'],
['cookie', 'token=1ff4ddc946758852308e4fc869486fc2']
]

agent = urllib.urlopener()
for cook in cookies:
name, value = cook
agent.addheader(name, value)

f = agent.retrieve(url, file)
find quote
SofaKng Offline
Donor
Posts: 388
Joined: Mar 2005
Reputation: 0
Post: #5
after that would i just do:

f = agent.urlopen('http://www.myurl.com')
data = f.read()
...blah...

?
find quote
hattmall Offline
Junior Member
Posts: 32
Joined: Nov 2005
Reputation: 0
Post: #6
can anyone post an updated code example. I am unable to get this to work. This is my first attempt at a python script and I am piecing together from tutorials, etc.

Quote:import xbmc, xbmcgui, urllib, urllib2 , re
from string import split, replace, find


Base_URL = "http://ftp.iinet.net.au/pub/games/movies/Red_Vs_Blue/Season1/"

LinkDescription = []
LinkURL = []
cookies = [
['cookie','user=XXXXXX-%59%44%73%45%46%65%4B%4C%76%53']
]

class MyClass(xbmcgui.Window):
def __init__(self):
self.list = xbmcgui.ControlList(200, 150, 300, 400)
self.addControl(self.list)
self.list.addItem('Item 1')
self.list.addItem('Item 2')
self.list.addItem('Item 3')
self.setFocus(self.list)


def onAction(self, action):
if action.getButtonCode() == 275:
self.close()

if action.getButtonCode() == 257:
self.list.addItem('1')
WebSock = urllib.urlopen(Base_URL) # Opens a 'Socket' to URL
WebHTML = WebSock.read() # Reads Contents of URL and saves to Variable
WebSock.close()

self.list.addItem('2')

Temp_Web_URL = re.compile('<a href=["](.*)[.]zip["]>', re.IGNORECASE).findall(WebHTML) # Using find all mentions of stuff using regexp to use wildcards
Temp_Web_Desc = re.compile('<a href=["].*[.]zip["]>(.*)</a>').findall(WebHTML) # find it

self.list.addItem('3')

for urls, desc in zip(Temp_Web_URL,Temp_Web_Desc):
LinkURL.append(urls[9:-2]) # Adds urls to a list # note need to add extention for these links to really work
LinkDescription.append(desc)


self.list.addItem('4')
self.list.addItem(LinkURL[1])

if action.getButtonCode() == 256:
webfile = 'http://rapidshare.com/files/172380200/webearth.jpg'
localfile = 'Q:\\scripts\\webearth.jpg'
self.downloadURL(webfile,localfile)
self.addControl(xbmcgui.ControlImage(0,0,800,600, 'Q:\\scripts\\webearth.jpg'))

def downloadURL(self,source, destination):
try:
loc = urllib.URLopener()
for cook in cookies:
name, value = cook
loc.addheader(name, value)
loc.retrieve(source, destination)
self.message('download ok')
except:
self.message('download failed')

def onControl(self, control):
if control == self.list:
item = self.list.getSelectedItem()
self.message('You selected : ' + item.getLabel())


def message(self, message):
dialog = xbmcgui.Dialog()
dialog.ok(" My message title", message)




mydisplay = MyClass()
mydisplay .doModal()
del mydisplay
find quote
Voinage Offline
Banned
Posts: 846
Joined: Mar 2008
Location: England
Post: #7
Try this :

Code:
import urllib2

req = urllib2.Request(url)
req.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14')
page = urllib2.urlopen(req);response=page.read();page.close()
cookie=page.info()['Set-Cookie']
# top is for obtaining the cookie via the info get.
req = urllib2.Request(url)#send the new url with the cookie.
req.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14')
req.add_header('Cookie',cookie)
page = urllib2.urlopen(req)
response=page.read();page.close()
find quote
hattmall Offline
Junior Member
Posts: 32
Joined: Nov 2005
Reputation: 0
Post: #8
Now Have it working I think sending the cookie, but I need to be able to Retrieve the URL while sending the cookie.
This way I can store the file, or play it.
Like the urllib.retrieve

which will work, but will not pass the cookies.

So I need the Urllib2 way to retrieve a file or whatever the method is.
(This post was last modified: 2008-12-12 09:13 by hattmall.)
find quote
hattmall Offline
Junior Member
Posts: 32
Joined: Nov 2005
Reputation: 0
Post: #9
Ok, I managed to Make it Write the page Downloaded to a file.
To me this seems like a bad way to do it, because it will be dealing with large file
100 MEG Archrives, Large Video Files, etc and I want to stream them.

What is the better way??
Can I have the file writing, as well as playing at the same time?

Code Now looks like this:


Code:
def downloadURL(self,source, destination):        
    try:
      req = urllib2.Request(source)
      req.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14')
      page = urllib2.urlopen(req);response=page.read();page.close()
      self.message(page.geturl())
      #cookie=page.info()['user=XXXXXX-%59%44%73%45%46%65%4B%4C%76%53']
      #Dosen't seem to need the above line and it caused it to fail
      # top is for obtaining the cookie via the info get.
      req = urllib2.Request(source)#send the new url with the cookie.
      req.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14')
      req.add_header('Cookie','user=XXXXXXX-%59%44%73%45%46%65%4B%4C%76%53')
      page = urllib2.urlopen(req)#;response=page.read();page.close()
      my_picture = page.read()
      #Writing The File
      fout = open(destination, "wb")
      fout.write(my_picture)
      fout.close()

     self.message('download ok')
     except:
     self.message('download failed')

Thank you for any help.
(This post was last modified: 2008-12-12 09:50 by hattmall.)
find quote
Voinage Offline
Banned
Posts: 846
Joined: Mar 2008
Location: England
Post: #10
Have you read my plugin tutorial ?

The set-cookie only gets the true cookie sent by the site, if it isnt there it will not be able to find it and cause an error.

Copy the addlink def from one of my plugins then addlink(name you want to display, url of file, thumbnail)

when you press (A) xbox or click(windows/other) it will stream the file.
(This post was last modified: 2008-12-12 14:52 by Voinage.)
find quote
hattmall Offline
Junior Member
Posts: 32
Joined: Nov 2005
Reputation: 0
Post: #11
** Resolved - Explaining In Next Post, with new questions.

I have read your Tutorial and the documentation in the wiki multiple times, wouldn't have gotten this far without it by any means, and search the forums as best I can.
Thank you to all who provide such information.

Ok, I think I understand, here is my issue. I need to send a cookie. That I already have on my computer. The site does not set a cookie. The cookie actually includes the username and an authorization code. I copy this information from my Windows computer to the script. Each time I access something on the server I have to send this cookie otherwise I can't get the file. (It seems)
This is the line that is allowing it to work.

req.add_header('Cookie','user=XXXXXXX-%59%44%73%45%46%65%4B%4C%76%53')

XXXX = My Username for Rapidshare.com

So far, I think I am only able to do this with urllib2, which does not support the retrieve method.

I am also not linking to direct video files, but to RAR Archives. So I believe I will need to download the first file completely before it can begin playing.

I am able to do this for files less than 64MB using.
page = urllib2.urlopen(req)
fout = open(destination, "wb") #destination is location to save file
fout.write(page.read())
fout.close()

But, if the file is larger than 64MB the XBOX runs out of Ram and locks up.

As of now I am making a script, because I plan to add more features. I haven't made it a plugin. If I was to have a plugin, and a direct link to video/audio files. Is there a way to pass the cookie before getting the file to play.

As best I can tell the addlink def uses urllib.urlretrieve which I do not think can send the cookie header?

Thank you Voinage for all your help, please let me know where I am right and wrong and what I can do.

I am looking for a way to download the file in increments of perhaps 20% write it, and then append another 20% etc.

Thanks
(This post was last modified: 2008-12-13 02:06 by hattmall.)
find quote
hattmall Offline
Junior Member
Posts: 32
Joined: Nov 2005
Reputation: 0
Post: #12
I was able to achieve what I needed, which was to retrieve a url and send a header.

My code is
Code:
pac = urllib.URLopener();
      pac.addheader('Cookie','user=XXXXXX-%59%44%73%45%46%65%4B%4C%76%53')
      pac.retrieve(page.geturl(),destination)

Since it is rapidshare. I have to use page.geturl() because the first request, with the header returns redirets to a New Url different then what is originally specified because a random "premium" link is generated on the fly.

I will make a new topic since any further questions I have will be off topic.
find quote
Voinage Offline
Banned
Posts: 846
Joined: Mar 2008
Location: England
Post: #13
Code:
This spilts the file into 3 you can split it more if you want the sends the correct range requests to get each part of the file and download it.

def DAP(url):
    name=url.split('/')
    req = urllib2.Request(url)
    req.add_header('User-Agent','Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14')
    response = urllib2.urlopen(req)
    size=int(response.info().get('Content-Length'));split=size/3
    p1='bytes=0-'+str(split)
    p2='bytes='+str((split+1))+'-'+str((split*2))
    p3='bytes='+str((split*2+1))+'-'+str(size)
    req = urllib2.Request(url)
    req.add_header('User-Agent','Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14')
    req.add_header('Range',p1)
    response1 = urllib2.urlopen(req)
    buf1=response.read()
    req = urllib2.Request(url)
    req.add_header('User-Agent','Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14')
    req.add_header('Range',p2)
    response2 = urllib2.urlopen(req)
    buf2=response.read()
    req = urllib2.Request(url)
    req.add_header('User-Agent','Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14')
    req.add_header('Range',p3)
    response3 = urllib2.urlopen(req)
    buf3=response.read()
    print time.asctime(),'Downloading: bytes= %s of %s' % (str(len(buf1)), str(size))
    fileout=open(name[-1],'wb')
    fileout.write(buf1)
    if len (buf1)>=split-1:
        buf1=buf1+buf2
    if len(buf1)>=split*2-1:
        buf1=buf1+buf3
    if len(buf1)>=size:
        response1.close()
        response2.close()
        response3.close()
        fileout.flush()
        fileout.close()
find quote