Posts: 1,466
Joined: Jul 2005
Reputation:
29
Basje
Team-Kodi Member
Posts: 1,466
I recently discovered that there are a number of IP addresses that are hammering my repository. They keep downloading the same updates over and over again and download the addons.xml file far to often. It actually turned out that some IP addresses download an update (10MB) more than 200x on a single day. That is 2 GB of bandwidth and is really causing me to wonder what is wrong.
Now, most (95%) of the IP's actually download them once or twice (if they have multiple XBMC's). So I am wondering what could lead to this behaviour. I double checked, but I can't really related it to a specific XBMC or OS version.
Posts: 7,650
Joined: Jun 2011
Reputation:
287
did you "whois" those IPs? Maybe its one of those meta repositories...
Posts: 1,466
Joined: Jul 2005
Reputation:
29
Basje
Team-Kodi Member
Posts: 1,466
Hi, I would like to continue this topic. Last tuesday I updated my repository from version 1.0.2 -> 1.0.3 -> 1.0.4 (the 1.0.4 was due to a missing icon) and I updated my addon from 3.3.7 -> 3.4.0.
What I see today is (these are only todays counts):
10000 downloads of the 1.0.4 repository
900 downloads of the old 1.0.2 repository by only 8 clients
700 downloads of the old 3.3.7 add-on by the same 8 clients (that is about 7 GB of data)
In total those 8 clients are responsible for a total of 2100 downloads in 24 hours.
I did not check this yesterday, but I guess it will be the same. So I thinks something is not OK at all for those 8 IP addresses. I would really like to know what is wrong with these addresses, but how can this be figured out?
Posts: 26,215
Joined: Oct 2003
Reputation:
187
The only thing I can think of is they have some sort of corrupt add-on database. Thus, the update time of the repository in the database is never updated (can't write to the database) so it continually hits.
I'm not sure what could be done about that to be honest - the database open's fine, but writes to it are probably broken.
Posts: 1,466
Joined: Jul 2005
Reputation:
29
Basje
Team-Kodi Member
Posts: 1,466
So can I advise those people to delete their addons db?
Besides that: would it not be possible to detect this problem when XBMC cannot write to a DB? In case of the addon db it could just be re-created?
Posts: 26,215
Joined: Oct 2003
Reputation:
187
If you know who they are, then sure, see if that's the issue.
I'm not sure we could detect this reliably. When an error occurs, it's not always clear whether that's a coding error on our part or some sort of corruption of the database (which may also be some coding error - I have never managed to achieve this so I don't know!). Worst case all we could do is flag the database as having something wrong with it.
The first step is determining where it's failing - there should be some errors thrown and thus errors in the log. If we can confirm that the database is corrupt then we're in a better position to be able to fix it. Ideally we'd have an example of such a corrupt database to play with.
Posts: 1,466
Joined: Jul 2005
Reputation:
29
Basje
Team-Kodi Member
Posts: 1,466
I would like to re-address this issue, as even with recent versions of Kodi the problem persists.
I have users downloading add-on updates over 500x per day. With a 0.5 MB per download, that makes 250 MB per user per day!
So this is just my simple add-on with 50.000 active users. I would assume that if this is the case on your repository servers you would be wasting gigabytes of bandwidth reach day.
Created some analysis tools (SQL stuff) for my apache logs to analyse the millions of rows of requests. Would you guys be willing to have me have a look at your logs (or some of them) to see how often this occurs for you?
Let me know.