[RELEASE] OzWeather - Australian Weather Addon using BOM data

scotia · 2012-02-27 05:48

Hmm I looked under "downloads", not "source"

My bad.

Thanks for the add-on!

Scott

bossanova808 · 2012-02-27 05:49

No probs.

scotia · 2012-02-27 06:56

Hi Bossa,

I was having the same trouble as the chap before with a script error when searching a postcode.

Bearing in mind in know very little about python...

The returned url from weather zone has a www. at the front which is missing from the script.
Also, unfortunately there are TWO HTML lists with class typ2, which breaks the script. It only checks one of the two lists, which is the one containing news. Because the list you want is in the "structure_middle" div I included a step before extracting the list to limit the html to just that div.
The third thing is that the length of skimmed is 1, even though the search returns many results. So I changed it to the length of the locations array.

To that end, I've created a patch to fix things. It has some log() calls which really shouldn't be there, feel free to remove them. I'm sure there are ten better ways to accomplish the same thing (like checking for (www\.)? at the start of the URL.

I hope you don't mind me fiddling with your code.

Regards
Scott

Code:

--- default.py.dist     2012-02-27 15:49:06.000000000 +1100

+++ default.py  2012-02-27 15:44:40.000000000 +1100

@@ -521,7 +521,7 @@

         text = keyboard.getText()

         #need to submit the postcode to the weatherzone search

-        searchURL = 'http://weatherzone.com.au/search/'

+        searchURL = 'http://www.weatherzone.com.au/search/'

         user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'

         host = 'www.weatherzone.com.au'

         headers = { 'User-Agent' : user_agent, 'Host' : host }

@@ -530,9 +530,11 @@

         req = urllib2.Request(searchURL, data, headers)

         response = urllib2.urlopen(req)

         resultPage = str(response.read())

+#        log("OzWeather: got response", resultPage);

         #was there only one match?  If so it returns the page for that match so we need to check the URL

         responseurl = response.geturl()

-        if responseurl != 'http://weatherzone.com.au/search/':

+#        log("OzWeather: got responseurl", responseurl);

+        if responseurl != 'http://www.weatherzone.com.au/search/':

             #we were redirected to an actual result page

             locationName = common.parseDOM(resultPage, "h1", attrs = { "class": "unenclosed" })

             locationName = str.split(locationName[0], ' Weather')

@@ -540,17 +542,24 @@

             locationids = [responseurl]

         else:

             #we got back a page to choose a more specific location

-            skimmed = common.parseDOM(resultPage, "ul", attrs = { "class": "typ2" })

+            middle = common.parseDOM(resultPage, "div", attrs = { "id": "structure_middle" })

+#            log("OzWeather: got middle", middle);

+            skimmed = common.parseDOM(middle, "ul", attrs = { "class": "typ2" })

+#            skimmed = common.parseDOM(resultPage, "ul", attrs = { "class": "typ2" })

+#            log("OzWeather: got skimmed", skimmed);

             #ok now get two lists - one of the friendly names

             #and a matchin one of the URLs to store

             locations = common.parseDOM(skimmed[0], "a")

+#            log("OzWeather: got locations", locations);

+#            log("OzWeather: got locations length", len(locations));

             templocs = common.parseDOM(skimmed[0], "a", ret="href")

+#            log("OzWeather: got templocs", templocs);

             #build the full urls

             locationids = []

             for count, loc in enumerate(templocs):

                 locationids.append(WeatherZoneURL + loc)

             #if we did not get enough data back there are no locations with this postcode

-            if len(skimmed)<=1:

+            if len(locations)<=1:

                 locations = []

                 locationids = []

bossanova808 · 2012-02-27 07:18

Not at all, patches welcome! But what postcode was giving you errors - I check about 7 by default and I haven't hit the issue.

Um....also, I am stupid with git - not immediately sure how to apply a patch Wink

can you just post the changed function as you have it now? I'll also try and learn...

scotia · 2012-02-27 07:19

Hi Bossa,

and I have a feature request.

The UV value is the forecast from the BOM. Is there any chance of displaying the current. I couldn't find any source on the BOM site, but there is a real time capital city reading in XML here:
http://www.arpansa.gov.au/uvindex/realti...values.xml

Is there any chance you could read that data in and display it alongside the forecast?

Cheers,
Scott

bossanova808 · 2012-02-27 07:25

Hi Scott - the problem with that is tying it to the user's location - it's no good to me if it is just capital cities, I really need a Weatherzone source for it and I couldn't find one. Otherwise how do I deal with Yackandanda's UV, for example?

Basically, I need a BOM ID of some sort, or a weahterzone one, or it gets pretty hard....

Also, please supply postcodes with issues as I want to look at that a bit more and can't find any...

scotia · 2012-02-27 07:29

Bossa,

I was searching 3122. 3000 was also failing however. The search page in my browser had (http://www.) at the front, which was not matching your script.

Here is the new function. Changed lines are commented out with '#' with the new one below. I also added a line.

I'd rather the responseurl check for both a url WITH www. and with out it. I'm too lazy to google python regex!

Regards
Scott

Code:

##############################################

### NOW ACTUALLTY RUN THIS PUPPY - this is main() in the old language...

socket.setdefaulttimeout(100)

#the being called from the settings section where the user enters their postcodes

if sys.argv[1].startswith('Location'):

    keyboard = xbmc.Keyboard('', 'Enter your 4 digit postcode e.g. 3000', False)

    keyboard.doModal()

    if (keyboard.isConfirmed() and keyboard.getText() != ''):

        text = keyboard.getText()

        #need to submit the postcode to the weatherzone search

#        searchURL = 'http://weatherzone.com.au/search/'

        searchURL = 'http://www.weatherzone.com.au/search/'

        user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'

        host = 'www.weatherzone.com.au'

        headers = { 'User-Agent' : user_agent, 'Host' : host }

        values = {'q' : text, 't' : '3' }

        data = urllib.urlencode(values)

        req = urllib2.Request(searchURL, data, headers)

        response = urllib2.urlopen(req)

        resultPage = str(response.read())

        #was there only one match?  If so it returns the page for that match so we need to check the URL

        responseurl = response.geturl()

#        if responseurl != 'http://weatherzone.com.au/search/':

        if responseurl != 'http://www.weatherzone.com.au/search/':   

            #we were redirected to an actual result page

            locationName = common.parseDOM(resultPage, "h1", attrs = { "class": "unenclosed" })

            locationName = str.split(locationName[0], ' Weather')

            locations = [locationName[0] + ', ' + text]

            locationids = [responseurl]

        else:

            #we got back a page to choose a more specific location

# next line added

            middle = common.parseDOM(resultPage, "div", attrs = { "id": "structure_middle" })

#            skimmed = common.parseDOM(resultPage, "ul", attrs = { "class": "typ2" })

            skimmed = common.parseDOM(middle, "ul", attrs = { "class": "typ2" })

            #ok now get two lists - one of the friendly names

            #and a matchin one of the URLs to store

            locations = common.parseDOM(skimmed[0], "a")

            templocs = common.parseDOM(skimmed[0], "a", ret="href")

            #build the full urls

            locationids = []

            for count, loc in enumerate(templocs):

                locationids.append(WeatherZoneURL + loc)

            #if we did not get enough data back there are no locations with this postcode

#            if len(skimmed)<=1:

            if len(locations)<=1:

                locations = []

                locationids = []

### no changes past this point

bossanova808 · 2012-02-27 07:36

What was in your browser isn't really relevant to be honest, it's what the result of the Python urlopen and it was coming back without the www hence the code.

But more importantly, both those codes work here on my Windows box, and 3000 at least definitely works on my linux box, so I can't see why they are failing for you...

What's your platform details?

bossanova808 · 2012-02-27 07:43

(that said I am happy to make the changes as they don't appear to break anything, it's just odd and hard for me to test as they all work fine here anyway!)

scotia · 2012-02-27 07:49

Ok, when I was debugging the script I got:

Code:

NOTICE: Oz Weather-0.3.5: Exception: OzWeather: got responseurl [http://www.weatherzone.com.au/search/]

(I added a line to print out the response url)

But my browser returns the www. as well.

What the site returns is really up to it. I am using a squid proxy may alter the request.

Also of some use:

Code:

[root]# curl -D /tmp/curl-headers "http://weatherzone.com.au/search"

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<html><head>

<title>301 Moved Permanently</title>

</head><body>

<h1>Moved Permanently</h1>

<p>The document has moved <a href="http://www.weatherzone.com.au/search">here</a>.</p>

</body></html>

scotia Junior Member Posts: 22 Joined: Oct 2005 Reputation: 0	2012-02-27 05:48 Post: #131 Hmm I looked under "downloads", not "source" My bad. Thanks for the add-on! Scott
	find quote

bossanova808 Member+ Posts: 1,542 Joined: Sep 2009 Reputation: 20 Location: Melbourne, Australia	2012-02-27 05:49 Post: #132 No probs. My XBMC stuff: ...inc: My Addons: XSqueeze - Squeezebox for XBMC, XZen - Zenfolio for XBMC and OzWeather - weather with Australian BOM data & radars Notes on my fanless, silent, powerful XBMC build Get Win7 to run RemoteApps
	find quote