[jacintech.fire]

***MOD EDIT: ***



I have built a samba (JBOD) array with an installed storage capacity of 512TB, spread across eight (8) nodes (i.e 16x 4TB SATA HDD per node) to house our ever expanding media library.

Currently we have 5 HD Displays/XBMC Clients (Win7 HTPCs) in the house connected via a 1Gbps Switch (Actually, each client is connected to a PowerLine 500Mbps adapter, as my wife refuses to allow me wire the house in a more conventional way; something about "putting holes" in her brand-new house..). There is a dedicated MYSQL server and shared userdata dir for library synch across all the clients.

As you can imagine, it has become increasingly resource intensive to manage such a large setup (considering that each 4TB HDD is further partitions into 4x 1TB partitions; and each partition represent a difference media source, the sources.xml file is rather large).

I have begun to investigate the possibility of using an Object Store mechanism (i.e OpenStack w/GlusterFS, HDFS, etc.) as an alternative to a Samba array/Windows Share.

My question is: Are there any plans for XBMC to support distributed, cloud storage mechanism that do not rely on the traditional File(s)/Directory(ies) mechanism of FTP, Samba, and the rest?

Thank you in advance for your assistance

[joelbaby]

each partition represent a difference media source

Why is each media source limited to 1TB? Or ... Why do you need 512 separate media sources?

[wsnipex]

no plan for xbmc to support this directly as its already possible on the OS level on real operating systems .

[jacintech.fire]

No reason whatsoever. I guess is force of habit from back in the days as a slackware sysadmin when we built the servers with partitioned HDD. We found that when a drive got clobbered, it was easier to recover the data if it was partitioned in smaller chunks...and sometimes only one partition on a drive would fail rather than the whole thing. I guess it stuck...

[jacintech.fire]

Oh, I can setup the distributed file system (or rather, Object Store) on my own. That's the easy part. What I would like is not to have to rely on mounting the individual blocks; exporting the distributed volumes as a samba share; then mounting these shares on each individual client box...

I was thinking about a mechanism that would interact with (and manipulate) the object store directly...

Can you imagine the possibilities?
Horizontally scalable Media Library...
Adding an encryption/decryption layer to the XBMC so that it can access all the Media (Objects) that are stored encrypted in the Object Store...
...the sky is the limit..

[.:B:.]

The Unix philosophy suits best here imho - XBMC is great at what it does: media playback. It shouldn't be handling storage and certainly not encryption and stuff. That is already done at OS level. Or at least: can be done at OS level already.

[DrDaxxy]

I take it you can't mount distributed file systems on a single mountpoint?
You might want to submit a feature request then, but I wouldn't be too hopeful. Your use case is all but common

What on earth do you need half a petabyte for anyway, if I may ask?

[jacintech.fire]

Please forgive my ignorance; but what I am talking about is a mechanism to access and manipulate an Object Storage, the same way it does now with a mechanism to manipulate files and directories...

Adding a basic mechanism to read/decrypt an encrypted object is not, imho, a large leap or deviation.

The basic idea of software is to continue to evolve and stay relevant. Given the popularity and obvious benefits of cloud storage, distributed file systems and object stores, I would think the XBMC team would take advantage of these benefits...

XBMC core audience is not the mass market (average user); as this market is already served by the likes of Netflix, Hulu, Amazon and the rest. The Average XBMC user will tinker, modify and otherwise roll-his/her-own media streaming solution. Given that eventually every media library out-grows its storage, adding these forward-looking functionality will ensure XBMC retains its core audience (user base)...

...but then again, what do I know :-)

[jacintech.fire]

It isn't just the issue of mounting the file system. It is rather the inherently antiquated files/folders/directories mechanism. Can you imagine how labour intensive is to manage a library with 50,000+ titles (Music, Music Videos, Movies, etc).

Most Object Stores were specifically designed to deal with large amount of data like a growing media library...

Why 512TB...? Why NOT :-)

**Bragging rights, mostly**

[DrDaxxy]

I don't think object stores would fix that. You'd still have to import your media - the only difference I'm seeing is whether metadata and data are in the same datastore.

[solamnic]

....connected to a PowerLine 500Mbps adapter, ....

What brand is your Powerline...?
is it adequate to stream in all your xbmc boxes at the same time?

[Ned Scott]

UnRaid. What you want is UnRaid. Expandable storage across many many drives that appear as an unchanged (but growing, as you add drives)) drive to XBMC.

I'm not sure why on earth you would partition each 4TB HDD into 4 partitions. What you've described is an unmanageable mess, and it's not because XBMC doesn't have some "cloud" feature. You've fundamentally made this harder than it needs to be. This is exactly why things like UnRaid exist.

[Milhouse]

Or ZFS, which is perfect for this number of disks and would give outstanding performance with some solid state caching. I too am left scratching my head at this frankly bizarre installation - it's an impressive number of disks but that's about it...

[jacintech.fire]

Any brand would do. I turn encryption on just for $#!7s and giggles. I turn QoS on both the routers and the individual adapters. For anything other than uncompressed BluRay it works fine. It also helps to have decent electrical wiring in the house...
I can get up to 23Mbps at the farthest connection in the house...

[jacintech.fire]

@DrDaxxy you need to import the media into XBMC one way or another (files, folders, images, nfo files). When dealing with thousands upon thousands of titles dealing with individuals files is a pain. You can surely see that. Case in point, managing the media metadata (library) with a MYSQL database vs local files...