Kodi Community Forum
Release OzWeather - Australian Weather Addon using BOM data inc. animated radar support - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Weather Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=155)
+---- Thread: Release OzWeather - Australian Weather Addon using BOM data inc. animated radar support (/showthread.php?tid=116905)



RE: OzWeather - Australian Weather Addon using BOM data inc. animated radar support - bossanova808 - 2021-04-26

Right, well that would explain things I guess!

Where's the link to the old one - I have lost track a bit....I don't use Estuary, only Estouchy (phone, tablets) and Confluence (main machines).  I may well have stuffed it up somehow although when I search for the error your logs were showing I didn't find that code in the patcher folder, so it's a bit odd.

In the meantime, I am making (slow) progress on the BOM api side of things.


RE: OzWeather - Australian Weather Addon using BOM data inc. animated radar support - DavidFW1960 - 2021-04-27

(2021-04-26, 06:29)bossanova808 Wrote: Right, well that would explain things I guess!

Where's the link to the old one - I have lost track a bit....I don't use Estuary, only Estouchy (phone, tablets) and Confluence (main machines).  I may well have stuffed it up somehow although when I search for the error your logs were showing I didn't find that code in the patcher folder, so it's a bit odd.

In the meantime, I am making (slow) progress on the BOM api side of things.
Ok so I just checked. If you follow the manual install instructions from the Wiki it is still downloading the same xml files I downloaded a few years ago (downloads a zip file with MyWeather.xml and DialogSeekBar.xml) but the skin patcher seems to be downloading a different version of MyWeather.xml. I just am overwriting the MyWeather.xml from the original zip file for manual install into the skin patcher folder and that made it work for both Vero/OSMC and my ChromeCast with GTV.

The patcher doesn't seem to copy over the DialogSeekBar.xml file. Any reason why?


RE: OzWeather - Australian Weather Addon using BOM data inc. animated radar support - bossanova808 - 2021-04-27

Yep so I'm in an awkward stage of mess as I rebuild everything from the ground up so for now just carry on as you were and I will fix it all up when I get through the BOM API changes, basically.  Realistically it's likely to be a few weeks as there's a lot to do and I haven't got a lot of time (business, kids, etc...)


RE: OzWeather - Australian Weather Addon using BOM data inc. animated radar support - DavidFW1960 - 2021-04-27

(2021-04-27, 03:01)bossanova808 Wrote: Yep so I'm in an awkward stage of mess as I rebuild everything from the ground up so for now just carry on as you were and I will fix it all up when I get through the BOM API changes, basically.  Realistically it's likely to be a few weeks as there's a lot to do and I haven't got a lot of time (business, kids, etc...)
All good mate. Take your time and live your life.


RE: OzWeather - Australian Weather Addon using BOM data inc. animated radar support - bossanova808 - 2021-04-28

IMPORTANT UPDATE:

There are lots of changes going on with the BOM data/services right now.  I am busy re-writing things to use the new BOM API for weather observation/forecast data.


But - also - after being able to download the radars just fine for ~10 years now, this morning instead of radar backgrounds, I see this message:

Potential automated request detected! We are making changes to our website therefore web scraping is no longer supported. Please contact us by filling in the details at http://reg.bom.gov.au/screenscraper/screenscraper_enquiry_form/ and we will get in touch with you.

Looks like I may have even more re-writing to do!  I had hoped just to release the new BOM API data part first (and am a fair way through that now) - without having to re-code the radar side of things.  But that is now looking less likely.  The message is odd as no scraping is involved in the radars, it's done with anonymous FTP as per the BOMs process for this.  But looks like they are blocking anonymous access.  I have contacted them and will see what they say, anyway...


Thus I suspect there is going to be a few weeks of breakage with OzWeather before things are working again (and I am presuming there will be public, free access to the radars again, somehow...). 
I suppose it is not surprising - 10 years is a long time to go without major changes, especially for something that scrapes, rather than uses a proper API.

So...the good news is weather data will soon use the BOM API (primarily because WeatherZone forecast data is no longer coming from the BOM it seems - and also because using an API is much better than scraping, anyway!).
The bad news is I have to find out what is going to happen with radars and that is much trickier coding so might take a while for me to work it all out. 

When the new release comes, everyone will have to re-configure the addon so that the addon knows the geohash for you location.  This will be built in, but it will mean you need to visit settings and update each of your locations to get things working again.


RE: OzWeather - Australian Weather Addon using BOM data inc. animated radar support - DavidFW1960 - 2021-04-28

(2021-04-28, 00:31)bossanova808 Wrote: IMPORTANT UPDATE:

There are lots of changes going on with the BOM data/services right now.  I am busy re-writing things to use the new BOM API for weather observation/forecast data.


But - also - after being able to download the radars just fine for ~10 years now, this morning instead of radar backgrounds, I see this message:

Potential automated request detected! We are making changes to our website therefore web scraping is no longer supported. Please contact us by filling in the details at http://reg.bom.gov.au/screenscraper/screenscraper_enquiry_form/ and we will get in touch with you.

Looks like I may have even more re-writing to do!  I had hoped just to release the new BOM API data part first (and am a fair way through that now) - without having to re-code the radar side of things.  But that is now looking less likely.  The message is odd as no scraping is involved in the radars, it's done with anonymous FTP as per the BOMs process for this.  But looks like they are blocking anonymous access.  I have contacted them and will see what they say, anyway...


Thus I suspect there is going to be a few weeks of breakage with OzWeather before things are working again (and I am presuming there will be public, free access to the radars again, somehow...). 
I suppose it is not surprising - 10 years is a long time to go without major changes, especially for something that scrapes, rather than uses a proper API.

So...the good news is weather data will soon use the BOM API (primarily because WeatherZone forecast data is no longer coming from the BOM it seems - and also because using an API is much better than scraping, anyway!).
The bad news is I have to find out what is going to happen with radars and that is much trickier coding so might take a while for me to work it all out. 

When the new release comes, everyone will have to re-configure the addon so that the addon knows the geohash for you location.  This will be built in, but it will mean you need to visit settings and update each of your locations to get things working again.

Yeah. The original Home Assistant BOM integration used web scraping for the observational data and it stopped working a few days ago as well. Fortunately Home Assistant had deprecated that a few months ago and another user wrote a new custom integration using the API as well but still some people didn't bother switching over to that and now it's just stopped working they have to reconfigure their systems...

We found this announcement:
```
Service Announcements
All scheduled release dates are subject to change due to technical and operational requirements.
Website notification of change
Scheduled Release Date: 3 March 2021A web application firewall policy has been implemented for www.bom.gov.au which will block screen scraping activity.The Bureau is monitoring screen scraping activity on the site and will commence interrupting, and eventually blocking, this activity on www.bom.gov.au from Wednesday, 3 March 2021. This is aimed at protecting infrastructure, system access and security, intellectual property and server/service load.Web or screen scraping is the act of copying information that shows on a digital display so it can be used for another purpose. This activity has always been at odds with the Bureau’s terms and conditions. We understand www.bom.gov.au contributes significantly to the work of many individuals and organisations and we are committed to continuing to provide access through our registered user’s channel.For further information, or to discuss the ongoing use of our materials, please make contact with us via [email protected].
```


RE: OzWeather - Australian Weather Addon using BOM data inc. animated radar support - bossanova808 - 2021-04-28

(Technically Oz Weather has never scraped anything from BOM as such.  It scraped from WeatherZone....but it did use the anonymous FTP for radars).

At the moment it seems all they have actually cut off is the radar backgrounds....but I noticed my Android weather app is not working this morning either....


RE: OzWeather - Australian Weather Addon using BOM data inc. animated radar support - bossanova808 - 2021-04-28

As trivia, I actually think the Home Assistant stuff was forked off Oz Weather years ago...I remember the dev asking about it long ago. I told him/her that that's kinda the point of open source...have at it!  I might be wrong though - it was definitely one of those home automation things, just can't remember which one...


RE: OzWeather - Australian Weather Addon using BOM data inc. animated radar support - DavidFW1960 - 2021-04-28

(2021-04-28, 04:32)bossanova808 Wrote: As trivia, I actually think the Home Assistant stuff was forked off Oz Weather years ago...I remember the dev asking about it long ago. I told him/her that that's kinda the point of open source...have at it!  I might be wrong though - it was definitely one of those home automation things, just can't remember which one...

Dunno.. Home Assistant is Python but so who knows. They canned the core integration (when someone logged an issue with the radar a few months ago) - rather than fix the issue they said... well this integration uses web scraping so instead of fixing it we will remove it... job done... In any case we have a new custom component for the obs and forecast (written in Python and using the API I linked you to) and a new radar card I already linked you to (javascript)


RE: OzWeather - Australian Weather Addon using BOM data inc. animated radar support - bossanova808 - 2021-04-28

Yeah the radar stuff is a fair bit more complicated...so I'll be working one step at a time here.  It's been a good 10 years but it was bound to happen eventually I guess.

The big question, really, is will this new JSON API remain open as it is now?  It does not seem super likely that it will, given that statement above (http://www.bom.gov.au/weather-services/announcements/). 

If they do add auth, that's not a huge drama as long as they issue a free access key...this would mean they can track things on their end (cf. the various art services) - or even a system for users to get their own keys is fine.  But their data charges are soon steep and clearly I aint gonna be paying....

....it all feels a bit iffy right now, basically.  But I am slugging even so and making steady progress.


RE: OzWeather - Australian Weather Addon using BOM data inc. animated radar support - DavidFW1960 - 2021-04-28

(2021-04-28, 05:22)bossanova808 Wrote: The big question, really, is will this new JSON API remain open as it is now?  It does not seem super likely that it will, given that statement above (http://www.bom.gov.au/weather-services/announcements/). 
I hope so. Their own app uses the same API so I would think it should remain open but who knows.


RE: OzWeather - Australian Weather Addon using BOM data inc. animated radar support - bossanova808 - 2021-04-28

Re: the Home Assistant radars - how do you configure what radar to show with that?  Do you have to enter lat/long, or does it work it out from the geohash??

It looks like it has a list of radar locations in lat/long terms, actually (https://github.com/theOzzieRat/bom-radar-card/blob/9ab11e7e5ba3edffbd2ee0b4d1edcf2069baca6b/src/bom-radar-card.ts#L130) - perhaps compiled from here - http://www.bom.gov.au/australia/radar/about/radar_site_info.shtml ?

In theory one could decode the geohash to lat/long, then find the closest from that list I suppose.  But generally I'd prefer to just let the user choose the radar they want...


RE: OzWeather - Australian Weather Addon using BOM data inc. animated radar support - DavidFW1960 - 2021-04-28

(2021-04-28, 06:17)bossanova808 Wrote: Re: the Home Assistant radars - how do you configure what radar to show with that?  Do you have to enter lat/long, or does it work it out from the geohash??

It looks like it has a list of radar locations in lat/long terms, actually (https://github.com/theOzzieRat/bom-radar-card/blob/9ab11e7e5ba3edffbd2ee0b4d1edcf2069baca6b/src/bom-radar-card.ts#L130) - perhaps compiled from here - http://www.bom.gov.au/australia/radar/about/radar_site_info.shtml ?

In theory one could decode the geohash to lat/long, then find the closest from that list I suppose.  But generally I'd prefer to just let the user choose the radar they want...

Yeah he hard codes the radar locations. You can display the location of the Radars as well. Can also pick the type of map, the Center of the map display too. It works out the radar to use from that location and you can pan and zoom as well. I'd be in favour of using the geohash to determine the radar as then all you need to do is set one location and it configures both. Can also zoom the map too... it's pretty cool how it works.


RE: OzWeather - Australian Weather Addon using BOM data inc. animated radar support - bossanova808 - 2021-04-28

Yep so I have got as far as configuring it via input a location (suburb, postcode, etc) -> get back a list of geohashes, choose one, then get lat/long from that geohash.

And a function to work out distance between lat/long pairs...so yes, in short, should then be able to come up with the closes radar lat long from that list.  Meaning a one step config which is nice.

Only disadvantage is it stops people choosing a specific radar, but I think that is ok really, as really 99.99% of folks will want the default radar for the locality.

(Without a LOT of work anyway...) Kodi won't support a complex radar control like that...it will be much as it is, just more modern version thereof, if all goes to plan.  It's just an animation of frames that the addon spits out, to the standard Kodi multi-image control designed for such...obviously in a full browser environment you can do much more.   But really, I'd guess mostly the default view will be just fine.


RE: OzWeather - Australian Weather Addon using BOM data inc. animated radar support - DavidFW1960 - 2021-04-28

(2021-04-28, 07:30)bossanova808 Wrote: Yep so I have got as far as configuring it via input a location (suburb, postcode, etc) -> get back a list of geohashes, choose one, then get lat/long from that geohash.

And a function to work out distance between lat/long pairs...so yes, in short, should then be able to come up with the closes radar lat long from that list.  Meaning a one step config which is nice.

Only disadvantage is it stops people choosing a specific radar, but I think that is ok really, as really 99.99% of folks will want the default radar for the locality.

(Without a LOT of work anyway...) Kodi won't support a complex radar control like that...it will be much as it is, just more modern version thereof, if all goes to plan.  It's just an animation of frames that the addon spits out, to the standard Kodi multi-image control designed for such...obviously in a full browser environment you can do much more.   But really, I'd guess mostly the default view will be just fine.

Yeah totally OK and if they want a different location they can just add a different locale anyway.... Sounds good to me.