[Razze]

In my world, companies are spending millions trying to aggregate multiple disparate dbs into a single edw. For a single version of truth and to have analytics across entire company, vs silos.

Kodi is essentially the analytics/reporting platform, so why break out into multiple dbs, or stay that way for the sake of it?

I have to disagree, Kodi has nothing to do with an dwh.
In my world pretty much anybody is heading in the other direction, if we're talking about applications.

Monolithic Architectures Fail By Increasing Friction Through Centralized Databases

The centralized database becomes a massive coupling mechanism. Every single thing in an entire system is locked together because each component must use the database to manage state.

If we need to decouple as we did on the internet and federate so everyone can make changes independently of each other, then the database can’t be system wide.

Centralized databases are a high friction architecture.

Microservices Succeed By Reducing Friction

Microservices remove the database as a coupling device. The scary and counter-intuitive idea is that each service has its own database.

Quoted from:
http://highscalability.com/blog/2015/4/2...oserv.html

Also see http://microservices.io/patterns/data/da...rvice.html for some explanation.

While I don't say one database isn't a solution, it has to be thought through. Would be bad to do such a step and realize we just devalued Kodi.
Personal, I wouldn't go for one database, but I think it's up for the team to decide.

[Tolriq]

Your quote is collector

The scary and counter-intuitive idea is that each service has its own database.

And this is exactly what you propose

Anyway Video / Audio / XXX are not microservice they are the exact same thing, assets with meta data (name / art / ....) associated with user data (watched / ....). Even the concepts of tvshows / albums / singles / movies can be abstracted to the exact same thing (collections)

[Razze]

Your quote is collector

The scary and counter-intuitive idea is that each service has its own database.

And this is exactly what you propose

Anyway Video / Audio / XXX are not microservice they are the exact same thing, assets with meta data (name / art / ....) associated with user data (watched / ....). Even the concepts of tvshows / albums / singles / movies can be abstracted to the exact same thing (collections)

Not sure if you get what that quote is about. It's scary and conter-intuitiv if you have never worked in that way. If you take a young programmer, he won't go for this approach, but do everything in one database and probably use classes as few as possible. Don't think that makes it right. There are reasons for patterns, abstraction and design decisions.

They are like them or better should be more like microservices.
And your right, they are almost the exact same thing. But we will loose out on some things that are not the same.
And don't forget the higher friction.

[Tolriq]

Well the other part of the same sentence maybe :

Microservices remove the database as a coupling device.

.

Your quote, quoted like is was is clear, about what a micro service is and why it should be decoupled from database and that's the reason why 1 database per micro service is bad and scary.
I perfectly understand the quote and this and I also know that Kodi video and audio are clearly not 2 independent microservices.

Anyway the question after your answer is what are the things that are not the same? And what are the friction ?

This thread is a one year thinking around that with response and schema that fit all the know actual and future needs with lots of arguments discussion and thinking.

Throwing it away, with an unrelated quote and no argument is not cool for all the work that was done here.

Please note that I was not involved in this and have no idea if the schema is correct or not, but I do know the amount of work that was done, and maybe the need to some respect for it.

[phate89]

First of all I don't see Razze's intervention as lack of respect. He entered the dev team not a long ago and he's actually entered in the discussion that didn't have a lot of "official" dev opinions..

But I still agree with tolriq. video and audio aren't separated services..
Imho this is one of the main kodi limits. They intersect with each other everywhere.. Movie have soundtracks, music have videoclips, a person could be an actor and a singer etc etc. Of course right now there aren't too many interactions like these in the code so the separation isn't a big problem now but the actual separation makes harder to bring them and will hurt kodi in the future.
The single database is imho more future.-proof. It will make these kinds of changes simpler and reduce the amount of db changes in the future..

I also understand why high scalability with microservices is better in a lot of cases but we're talking a "small", local, almost always single access database... There are less advantages here..

[zag]

A point to remember as well is its a lot easier to develop at the moment using the separate databases

[Tolriq]

As phate89 says, all depends on what to develop, for bug fixes of course as the database are already split and have different bugs.

But for future things that are asked since long time, like user ratings, user play states this means all have to be done X times with X different implementation since each database is fundamentally different.
Each time adding more difference to each database instead of having common things more efficient and more stable since more thought.

[Razze]

Well the other part of the same sentence maybe :

Microservices remove the database as a coupling device.

.

Your quote, quoted like is was is clear, about what a micro service is and why it should be decoupled from database and that's the reason why 1 database per micro service is bad and scary.
I perfectly understand the quote and this and I also know that Kodi video and audio are clearly not 2 independent microservices.

Anyway the question after your answer is what are the things that are not the same? And what are the friction ?

This thread is a one year thinking around that with response and schema that fit all the know actual and future needs with lots of arguments discussion and thinking.

Throwing it away, with an unrelated quote and no argument is not cool for all the work that was done here.

Please note that I was not involved in this and have no idea if the schema is correct or not, but I do know the amount of work that was done, and maybe the need to some respect for it.

If I would not respect the work I wouldn't have said anything and let this lay around, without discussion.
The schema itself might be fine, but I get a weird feeling, that there has not been that much work/proof, when I (still) find typos in the headers. But thats just a remark and has nothing to do with my point.
As you might know, I wasn't around a year ago and even if I were, I probably wouldn't have been involved in database decissions for Kodi itself.
Also I'm not saying, this is a bad idea, I'm saying consider what this will cause. Both negative and positive.

Things not being the same can be solved by moving all additional data to one generic table. Generic as in much rows not columns. So that's to consider, but not a show stopper, as long as nobody starts to create nullable fields or use fields for usecases that are not in the column name.
Friction will be higher as everybody will have to work with the same database and it might get harder to overview. Also if some addon like emby for example edits your data it will potentially not only break movies/tvshows then, but music also.

First of all I don't see Razze's intervention as lack of respect. He entered the dev team not a long ago and he's actually entered in the discussion that didn't have a lot of "official" dev opinions..

But I still agree with tolriq. video and audio aren't separated services..
Imho this is one of the main kodi limits. They intersect with each other everywhere.. Movie have soundtracks, music have videoclips, a person could be an actor and a singer etc etc. Of course right now there aren't too many interactions like these in the code so the separation isn't a big problem now but the actual separation makes harder to bring them and will hurt kodi in the future.
The single database is imho more future.-proof. It will make these kinds of changes simpler and reduce the amount of db changes in the future..

I also understand why high scalability with microservices is better in a lot of cases but we're talking a "small", local, almost always single access database... There are less advantages here..

Well that person case is in fact something that should be splitted into an own database in my world. Then both music and video and whatever could use that service/repo.

As phate89 says, all depends on what to develop, for bug fixes of course as the database are already split and have different bugs.

But for future things that are asked since long time, like user ratings, user play states this means all have to be done X times with X different implementation since each database is fundamentally different.
Each time adding more difference to each database instead of having common things more efficient and more stable since more thought.

I don't think that the database implementation is our problem here. Like when I added user ratings, for every part music/video, the amount of sql I wrote was like 3% of the changes I guess?

[phate89]

Well that person case is in fact something that should be splitted into an own database in my world. Then both music and video and whatever could use that service/repo.

To understand better your point of view what is the limit of separation? For example user data (ratings, bookmarks, resumes, users) should be splitted? if yes everyone in its own database or in a single one?

Thanks for our opinion btw even if it's the opposite of what I think.. Did you know if the dev team agrees with you?

[Razze]

Well that person case is in fact something that should be splitted into an own database in my world. Then both music and video and whatever could use that service/repo.

To understand better your point of view what is the limit of separation? For example user data (ratings, bookmarks, resumes, users) should be splitted? if yes everyone in its own database or in a single one?

Thanks for our opinion btw even if it's the opposite of what I think.. Did you know if the dev team agrees with you?

In theory there is no limit, but it's down to what is sensible/makes sense. It obviously has overhead, so that might be another minus.
It actually might make sense to split user data off, never thought of that before. Will have to think a little more about it, but it doesn't sound stupid
I would think splitting them from their own databases into one "user_data" database would make the most sense.

I can't really speak about the dev team. I have spoken if some of them about this, I know some think the same as I do. But there are others that want to go in the other direction.
It's a really deep decision, which will have consequences for years, so we might want to be sure, what we're doing.

[Tolriq]

From a coding point of view splitting user data to it's own database, how do you link all the stuff ?

This means multiple database opened to make joins (not efficient) how do you handle unique ids ?
You have to recreate temporary things in this table since a movie and an album can have the same id since they came from 2 database 2 tables.
This means any change to any database will break this other database and that the guy that handle user data will cry trying to keep up to date with any change to other database ?

Duplicating tables / code would be way less worse than this.

[phate89]

In theory there is no limit, but it's down to what is sensible/makes sense. It obviously has overhead, so that might be another minus.
It actually might make sense to split user data off, never thought of that before. Will have to think a little more about it, but it doesn't sound stupid
I would think splitting them from their own databases into one "user_data" database would make the most sense.

I can't really speak about the dev team. I have spoken if some of them about this, I know some think the same as I do. But there are others that want to go in the other direction.
It's a really deep decision, which will have consequences for years, so we might want to be sure, what we're doing.

Isn't query between 2 databases slower? Right now everything is based on views that contains all the basic data and they're the base for almost everything. If separating it in 2 databases makes it slower it will mean that almost every single query will be slower...
Also kodi is built on that views... If you split the db in 2 parts you can't use the views anymore because tables are in 2 different databases so you need to rewrite almost everything and find another way to filter the library

[whitebelly]

In my own selfish opinion, I like seperate music and video dbs because I like my music in a common MySQL db but prefer my movie dbs to remain unique to each client. But I suppose I could find an acceptable setup if there were only one db.

[Razze]

Btw, while music and video are quiet similar. Don't forget about the epg data, tv, addons, adsp, textures and viewmodes databases.

From a coding point of view splitting user data to it's own database, how do you link all the stuff ?

This means multiple database opened to make joins (not efficient) how do you handle unique ids ?
You have to recreate temporary things in this table since a movie and an album can have the same id since they came from 2 database 2 tables.
This means any change to any database will break this other database and that the guy that handle user data will cry trying to keep up to date with any change to other database ?

Duplicating tables / code would be way less worse than this.

Well we will need rules, that call each service and link stuff. After all it's just about having the same id on the correct items.

If you want unique ids, use guids they are even faster created then incremental ids.

In theory there is no limit, but it's down to what is sensible/makes sense. It obviously has overhead, so that might be another minus.
It actually might make sense to split user data off, never thought of that before. Will have to think a little more about it, but it doesn't sound stupid
I would think splitting them from their own databases into one "user_data" database would make the most sense.

I can't really speak about the dev team. I have spoken if some of them about this, I know some think the same as I do. But there are others that want to go in the other direction.
It's a really deep decision, which will have consequences for years, so we might want to be sure, what we're doing.

Isn't query between 2 databases slower? Right now everything is based on views that contains all the basic data and they're the base for almost everything. If separating it in 2 databases makes it slower it will mean that almost every single query will be slower...
Also kodi is built on that views... If you split the db in 2 parts you can't use the views anymore because tables are in 2 different databases so you need to rewrite almost everything and find another way to filter the library

We wouldn't be querying between the two databases (thats a no go) we would ask the service that for that database. Funny thing is views (like we're using them right now) are really slow already.
At my work we actually fix performance problems by replacing views with tables. Joining between different databases would be quiet fast, nothing to worry about, but as I said a no go in this architecture. Also I'm not sure if sqlite can do that.
And yes, we'll be rewriting major chunks of everything that has anything to do with the database, but that would also be true for the one database approach.

[phate89]

We wouldn't be querying between the two databases (thats a no go) we would ask the service that for that database. Funny thing is views (like we're using them right now) are really slow already.
At my work we actually fix performance problems by replacing views with tables. Joining between different databases would be quiet fast, nothing to worry about, but as I said a no go in this architecture. Also I'm not sure if sqlite can do that.
And yes, we'll be rewriting major chunks of everything that has anything to do with the database, but that would also be true for the one database approach.

Actually with one database you need to change some query but you can keep the views and the current structure.. It's actually quite a smaller change use a db like this than separate user data in a db.
But I agree that the actual database interaction is probably the most problematic part. The problem is that it's not only the video/musicdatabase code but it's deep in the libraries code. I looked into with my separation of user data and right now I'm going towards having one view for each user because removing the actual method is a huge task that I can't do alone..