Option to make a distributed Client / Server XBMC setup

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
xbmcg Offline
Senior Member
Posts: 152
Joined: Nov 2011
Reputation: 0
Location: Europe
Lightbulb  Option to make a distributed Client / Server XBMC setup
Post: #1
There were request in the forum for more then one library.
I would like to put it a little further:

I have a distributed environment with more than one media server (a large server for archiving all older stuff, a xbmc media center in my living room with current media / tv recordings and recent dvd / br content, a NAS with other video stuff / TV shows etc.

I also have small xbmc boxes with no local media in my home - they just stream from the backend servers.

Not all of this media sources are turned on all the time. They can be fired up by remote Power-On signals and switch off when not in use after some hours or on a remote shutdown command.

One can have xmas movies on one NAS and Helloween / Horror stuff on another, or adult content on a 3rd. They would be seldom used sometimes - however I don't want them to be always visible in the library, but only when they are turned on, some of them even password protected.

It would be awesome, if each media server could have it's own repository of media information stored localy along with the media (thumbs, infos, databaseĀ“s and everything needed to represent / feed the xbmc UI of the players), because the media and the meta data belong together and make no sense one without the other.

On the other side, the players recognize if a media source is online, and if - it automatically uses the remote database to update it's UI cache (front end xbmc boxes will show only available content). If another source goes online, the information is merged, if a source goes offline - the information is purged from the UI.

The backend systems could also take care for the scraping and updating of their databases. If they are just "dumb" NAS devices or usb hard drives, a central server instance could do the job for these tasks and delivers the mySQL engine / database of online available content and the corresponding share locations for the metadata / media currently available online.

This could allow a seamless plug&play integration in a distributed environment.

It also enables a common setup for all xbmc front end players and a 'start playback in one room, continue watching in another' - scenario... A dream would be a zero conf xbmc front end, booting from ROM or flash or over the network with no local storage at all. it would get a dhcp ip and would scan the local net for a xbmc server containing all config information, skins etc. from the repository.

Dynamic custom categories could filter then according to the needs of each user (smart filters, favorites...) based on profiles stored in the repository. Advantage: same custom, user specific UI in all rooms - whereever you go.

Most of the stuff needed is already there thanks to the migration to a mySQL based library. What is needed is a abstraction layer between content storage with library and presentation layer, that aggregates the currently available libraries / Metadata to the UI. This could be done as a windows or linux service, that polls all sources and updates the local SQL instance on startup and every 5 minutes or so if something changes on the network.

We have already abstraction layers there.

We got the data layer (called add source and supporting almost all network protocols available)
We got the metadata stored along with the media (using amber media manager)
We got also the (lokal) mySQL Database for search and categories like artists, genre, year etc.
We also have the concept of independant profiles with own configs and databases.
While allowing skins, we have a layer between this data and the presentation engine.
And finaly there is the media playback engine (based on mplayer) that plays back all this stuff on the frontend.
We also have the straming ability and the remote control stub using http servers.

But all this is running on the front end on a single machine.

So what is missing:

1. we need to migrate to a distributed database. The database parts must be stored together with the media on the data layer.
2. we need a aggregation database, that caches the currently available metadata for performance reasons.
3. we need a service on the frontend, that scans the network for changes and updates the SQL cache.
4. for the last optional issue (no local store, net boot for xbmc front ends) a config master (share?) needs to be setup, that caches user profiles, skins, plug-ins etc.

What are the advantages:

- one seemless, consistent UI on all xbmc instances everywhere in the house.
- metadata and library consistency (no additional scans required per instance)
- easy possibility for roaming of media playback
- easy remote control for one room by a UI in another room
- easy zone setup (sync playback / multiroom)
- easy integration of future home automation system
- easy maintenance and customization of all devices from one central point
- concept of 'satelite units' can be realized (e.g: UI: Android, Windows Mobile, iOS, Windows, Linux, hidden playback box without human input interfaces like keyboard, mouse, IRremote controlled only over the net, intelligent media servers) not just for one player, but for all in the house. The user can choose what modules to install on what machine, so the hardware requirements could be adjusted to the needs (25$ credit card sized PC, ION Boards, 20 Disk NAS Beasts, Tablets whatever...).

So what separate Parts would make sense:

- Data Storage
- Scraper engine
- sqlite3 db | mySQL Server instance(s) for the Data Layer
- sqlite3 db | mySQL Server instances for the presentation layer(s)
- Network discovery Service for the presentation layer
- central Config Storage for Profiles, Addons, Skins, Settings
- Playback engine
- UI engine

you can have each on one instance or combinations on separate machines, there could be one config storage (master), many data storages with or without scraper engines and mySQL instances, many playback engines and many UI engines either with a local SQL cache or using a centralized mySQL Server instance / network discovery.

Does this sounds reasonable?
(This post was last modified: 2012-01-13 00:37 by xbmcg.)
find quote
Robotica Offline
Banned
Posts: 1,202
Joined: Aug 2010
Post: #2
Making seperate libraries out of XBMC is the core to many advantage feature like distributed setups, services for a whole lot of functionality, downstream-usage, etc. But it's a hell of a job with needs a lot of coordination by the team.

See: topfs his view: media server: http://wiki.xbmc.org/index.php?title=Goo..._Code_2011
(edit: and the 2013 teams view: http://wiki.xbmc.org/index.php?title=Goo...uted_model)

If you want to get started: https://github.com/xbmc/xbmc/pull/542
This pull request disables GUI...

This topic leads to most available info: http://forum.xbmc.org/showthread.php?tid=114612
(This post was last modified: 2013-06-12 17:09 by Robotica.)
find quote
xbmcg Offline
Senior Member
Posts: 152
Joined: Nov 2011
Reputation: 0
Location: Europe
Post: #3
Thank you for the links.

The server part seems to be in development some how. I think, a modular approach would be better than just the -server switch, so the setup does not enforce the setup of 3D open GL / overlays... Assume, there is a headless server with no graphics at all, but only a remote shell.

I would like to suggest to have a clean, defined client / server interface spec between the parts and to keep them sane and dedicated to their main task, instead of having the full blown beast everywhere. This also easies a distributed development of the components.

Also a Option to have a hub & spoke setup (central storage, database, config on dedicated hardware with satelite clients) or a full meshed environment (every xbmc instance can work as server and as client in the same time).
(This post was last modified: 2012-01-07 17:42 by xbmcg.)
find quote
ghostelement Offline
Fan
Posts: 444
Joined: Sep 2010
Reputation: 3
Location: Portland, Oregon
Post: #4
Very good post. I think we both have similar ideas in what we would like to see accomplished when it comes to an XBMC server. As far as configuration for something like this, it could have a built in WebUI much like Sickbeard or similar services.

[Image: tgdb-buttonlogosmall.png]
thegamesdb.net - An open video games database.
scottbrant.net - Blog
find quote
aNiceGuy Offline
Junior Member
Posts: 5
Joined: Aug 2011
Reputation: 0
Post: #5
I cannot speak to the authorization component lightly touched on, Horror, Adult, ect. Very valid and important.

For the distributed part of this conversation, after a lot of research and thought I agree that moving the meta-data and storage to a distributed system is great in theory. Going this route either adds complexity to the developer side or the user side depending on the scenario.

What are the pressures that make distributing meta-data and possibly storage so interesting?

Consider a world were power is infinite and free. In this world would people move the file storage and meta-data to central system that is always on?

I spent a day or two and looked into distributed database technologies and storage technologies. Most of the work in this arena is to enhance performance, up-time, and move to commodity hardware. My XBMC does not have unduly pressure in this area, it is very stable, performance is great, and runs on almost any hardware I throw at it. My unRAID storage box suspends and boots up very quickly. Hosting mySQL and file storage on this one system seems very effective.

Help me understand the pressures to go more distributed and maybe a better path can be found.

Finding an easier way to make the setup of these pieces easier would increase adoption. I think that might be a worthwhile tangent for XBMC effort.
(This post was last modified: 2012-03-07 20:03 by aNiceGuy.)
find quote
xbmcg Offline
Senior Member
Posts: 152
Joined: Nov 2011
Reputation: 0
Location: Europe
Post: #6
OK, this is the henn-egg question. Some people state : "there is need for exact 5 computers in the whole world..." This about the centralized approach. Well the world changes. We do client-server, distribute power to the client - then again centralisation (cloud hype).

The truth is in the middle. There are notebooks / smart pads, whatever - able to run xbmc and have enough storage for content.
Also in a family or students home there are many independent people living together, having their own content / xbmc installations.

Would it not be great, if they can just connect to a common network and share their content, without copying all stuff to a central storage?

There are advantages / disadvantages of both scenarios, but to get most of the available devices and storage around it would be great
to be able to joun the resources, when they are available online and to remove them from the library when offline - beside usb drives and other storages if your nas is full....

There is also another use case. Assume there is a "working set" with common content for all, but also offline disks with some special content, not used frequently, but immediately available in the library, as soon as the storage goes online... (e.g. some home made movies, or special stuff)

I think, dynamic discovery and merging / removing content auto-magically (only registered of course) would make xbmc even better.
find quote