+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Information Providers (scrapers) (https://forum.kodi.tv/forumdisplay.php?fid=147)
+----- Forum: General Scraping (https://forum.kodi.tv/forumdisplay.php?fid=306)
+----- Thread: Req Sports scraper (/showthread.php?tid=174789)
RE: Sports scraper - enen92 - 2015-02-03
I assume all the events have the time according to the same timezone? Pregmatch the start time and convert it to a constant timezone presenting it in the format %H:%M". Unix timestamps would be a nice idea but since I assume you want to use this also as a database for recordings there will be matches before 1970.
Here's the code snipped I use to use in python (don't know much about php):
Code:
import pytzimp
d = pytzimp.timezone(str(pytzimp.timezone('Atlantic/Azores'))).localize(datetime.datetime(2014, 11, 1, hour=hora, minute=minuto)))
my_location=pytzimp.timezone(pytzimp.all_timezones["my_location_timezone"])
converted=d.astimezone(my_location)
fmt =%H:%M"
time=converted.strftime(fmt)In this case it receives the hour and minute as integers and convert it from timezone "Atlantic/Azores" to a timezone I specify.
I think trying to get the user location and convert it automatically to the corresponding timezone is sort of a waste of effort at the moment. Better decide a common timezone and present the results for this timezone only. Later on you can probably add to the webinterface something like the new wiziwig has (automatically detect the timezone and present a way to change it - or just the last):
http://www.getyourfixtures.com/
RE: Sports scraper - zag - 2015-02-05
The times seem completely random from the provider so I won't be including them for now unfortunately.
But I have done a few improvements:
- Front page now shows latest results, edits and fixtures
- Player page now shows last 5 tweets
- Soccer results are all now automated
- Player Clearart (From championship manager)
I spent a couple of hours searching for Basketball and other sports results but it is so hard to find anything reliable or get API keys. I'm thinking of just providing the feature to enter a result yourself. Not sure how many people will do this but at least we would have an easy way of creating an event result for something you want to scrape into Kodi.
RE: Sports scraper - Ovokx - 2015-02-05
(2015-02-05, 13:14)zag Wrote: I'm thinking of just providing the feature to enter a result yourself. Not sure how many people will do this but at least we would have an easy way of creating an event result for something you want to scrape into Kodi.
Also thinking as the best way to get results into the DB for all the stuff
RE: Sports scraper - enen92 - 2015-02-05
Automated user scripts would be cool. Can u let us know how we can provide this data for easy scrapping on your side?
There are free hosters that allow python web app hosting.
Building a scrapper to create a csv is pretty easy. And they would run only once a day or each 2 days.
I wont fill results on the db but automated scripts seem fine for me, and ill make some if u wish.
Let me just know how you want that data, preferably with a template or similar.
RE: Sports scraper - Ovokx - 2015-02-05
Updated all winter transfers on Spanish La Liga and added to the sheet the missing ones
RE: Sports scraper - zag - 2015-02-06
(2015-02-05, 21:20)enen92 Wrote: Automated user scripts would be cool. Can u let us know how we can provide this data for easy scrapping on your side?
There are free hosters that allow python web app hosting.
Building a scrapper to create a csv is pretty easy. And they would run only once a day or each 2 days.
I wont fill results on the db but automated scripts seem fine for me, and ill make some if u wish.
Let me just know how you want that data, preferably with a template or similar.
I've been importing them using CSV so far:
https://docs.google.com/spreadsheets/d/1vEWAqLz36DO8NF8EC_1LJRzb6AIwUjJHT1YPvz_oQoU/edit?usp=sharing
https://docs.google.com/spreadsheets/d/13MrVa24usJ0ImoOrMQtyGN7njIh84483ahK2pBxU7Dw/edit?usp=sharing
I really don't think its viable though, as all the team names have to match to import successfully.
I'll get working on a manual results feature.
I'll also do a bit of work on the English leagues now the transfer window is closed
RE: Sports scraper - enen92 - 2015-02-06
I think it might be viable. Spent 1 hour making a scrapper so here are the results. Portuguese league only to start and for a proof-of-concept.
You can check the full folder here:
http://web.ist.utl.pt/~ist157804/zag/ptprimliga/
Each cvs file is named round.csv. Example 19.csv:
http://web.ist.utl.pt/~ist157804/zag/ptprimliga/19.csv
All those files are made in a batch, only one time, and can be updated only when needed or the round we need.
The actual, being played or to be played round is identified in the file beingrefreshed.txt:
http://web.ist.utl.pt/~ist157804/zag/ptprimliga/beingrefreshed.txt
This means that, in this case as the current round is the 20, the 20.csv file will be updated 2 ou 3 times a day.
Last but not least, 1 or 2 times a day, league tables would be updated:
http://web.ist.utl.pt/~ist157804/zag/ptprimliga/tables.csv
I can use the exact same script to do the same for:
-Portugal: primeira liga, segunda liga
-England: Premier league, championship (and cups with some modification)
-spain
-italy
-netherlands
-france
-germany
-greece
Other sports are also available, such as nba or motorsports.
RE: Sports scraper - zag - 2015-02-06
Amazing! Will take a closer look at that this weekend
For now i've written a new feature to add scores manually.
Just click on the side menu function on the league page.
You can then add any individual event or result
RE: Sports scraper - Ovokx - 2015-02-07
Wow! Pretty nice feature! Congrats!
RE: Sports scraper - zag - 2015-02-08
Bit more progress today:
- Fixed a number of reported bugs
- Added player date of birth edit
- Added player born country edit
- Added French LNB basketball league (thx Ovokx)
- Imported Australian NRL Players and Events (thx mjccox)
- Improved Manual events/results feature
RE: Sports scraper - n1md4 - 2015-02-08
From what I understand, a lot of the development caters for team games in general, and specifically football (soccer). Being a motorsports fan though, I'm very interested the development in that area.
Zag, do you have some roadmap for this, even if only vaguely?
RE: Sports scraper - el_Paraguayo - 2015-02-09
enen92,
I had this request on my live scores thread:
Quote:can you add widgets to the kodi's main screen that will show metadata for your favorite team?I've explained that this is better handled by thesportsdb rather than my service. However, it is a good idea and something I'd like to implement if I can.
Before I start having a go at coding this (which will be slow!) I just wanted to check whether that's something you're already intending to do.
RE: Sports scraper - enen92 - 2015-02-10
I sort of have this planned but for the home menu of the addon I'm developing. That's why the team and nick appear on the top left corner. For the homescreen I don't know...I think the skins relly on "skin widgets" addon? Anything that is not native will certainly look weird (plus the added work of making it work in other skins/resolution).
But it's doable though! And I know more a less how that could be done. The "sportscenter" (hate this name) code is too small yet and due to this, it's easy to learn something about python/ xml gui design for kodi.
Ping me if you need help with it
cheers
RE: Sports scraper - el_Paraguayo - 2015-02-10
It all comes down to how you want this to be implemented.
Skin Widgets is, I think, a service that generates a heap of infolabels which skinners can then choose how to implement in their skin. This means that, if the functionality has been included in the skin, there's no need to run a script to launch the display. However, this basically passes the work of coding this to the skin developers/maintainers and I suspect this will not be an important enough feature for them to want to focus on.
The sportscenter (the name's not that bad, but I'd prefer it to be centre with "re"... it's a British thing!) is slightly different, it seems to run as a standalone script and comes with skin specific implementations as part of the script.
For the favourite team widget part - I, personally, would rather this was done similarly to the Skin Widgets - i.e. have a service creating and updating infolabels - so I'll start on that basis. On the plus side, either route will require the creation of some xml files so it shouldn't be too big a job to adapt anything I do for use in the sportscentre!
RE: Sports scraper - zag - 2015-02-10
- Manually add future events feature
- Manually update scores for events feature
- Homepage redesign to show latest 5 results for each sport
- Added Golf and Baseball sports
Next up is detailed event editing. Maybe the motorsport guys want to get in with the feature requests now.
I'm thinking racetrack image, TV Broadcaster, event "report", score, top 3 podium positions
Let me know if you think of any more?