Best way to store a large media library?

  Thread Rating:
  • 1 Votes - 4 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
GJones Offline
Fan
Posts: 438
Joined: Nov 2009
Reputation: 0
Post: #31
The biggest factor is determining if you want a storage server or a transcode/storage server. An atom (dual-core) is more than adequate for storage but would not be the brightest choice if you are doing anything other than a direct backup.
find quote
Bongo123 Offline
Junior Member
Posts: 6
Joined: Oct 2009
Reputation: 0
Post: #32
Theres a lot of great tips in this post to mull over when starting a backup solution from scratch but what if like me you have roughly 4tb's of data spread over 4 external USB drives and are looking to come up with a backup solution.

I take it would need to go out and buy another spare 1tb for additional media plus 5tb to have a backup of my original media plus any additional, this coupled with a case that can hold 10+ drives, plus software and this is assuming i can just remove the usb drives from their cases and bung em into a tower case is gonna cost a pretty packet.

I've just started looking into this and to be honest its bloody daunting, with everyone having a different soltuion to meet their specific needs and with so many software/hardware combinations i feel im waaay in over my head with this one.

If anyone could reccomend a specific path that would suit my needs which is basically a backup solution/home server to keep all my files in one place and share with addtional computers i would greatly appreciate it because at the minumte im just going round in circles with too many options to consider
(This post was last modified: 2010-03-24 12:30 by Bongo123.)
find quote
vikjon0 Offline
---
Posts: 2,497
Joined: Apr 2009
Reputation: 8
Location: Sweden
Post: #33
Quote:Theres a lot of great tips in this post to mull over when starting a backup solution from scratch but what if like me you have roughly 4tb's of data spread over 4 external USB drives and are looking to come up with a backup solution.

I am thinking the same. Of course we can setup to servers with double capacity, but I would prefere a traditional tape backup. Since I mostly store non-unique media it would be enough with a tape every week. The problem seem to be to find an affordable tape station.
find quote
TugboatBill Offline
Posting Freak
Posts: 829
Joined: Oct 2009
Reputation: 7
Post: #34
I went with unraid for several reasons. The plan is to use XBMC clients in 3 places and a centralized NAS was the best solution. I wanted a NAS device so I knew I would be building something from scratch. I also knew that my data requirements would be expanding over time. For a start I was able to use the free version (limited to 3 drives = 4TB of storage space) of unraid. Since I use it primarily for videos/music speed wasn't a consideration. I liked the fact that if I lost 2 drives I wouldn't be losing all my data, I could use different size drives, and it was energy efficient (shuts down unused drives). I don't do backups because of the cost. I'm already at 4TB and that leaves tape as the only solution and tape is expensive. If you've ever priced out a tape system that can backup TBs of data in a reasonable number of tapes you'll agree. Shocked

If I was going to an existing computer that's already used for other tasks, IE a desktop w/Windows on it I would consider flexraid. It has many of the features of unraid but can be loaded on top of many OSes.

Personally, I kept away from traditional RAID because they keep the drives running all the time. That's ok for a business but at home I pay for the electricity (well, actually somewhat - but that's another story) and it really didn't make sense running a drive 24*7 when it may be accessed once every week.
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 26,181
Joined: Oct 2003
Reputation: 176
Post: #35
If you have the disks already (in external USB disks) then just use them in your server box. You need only buy the case, motherboard etc. You will need 1 spare disk if you're going for unraid (or enough capacity to free up one of your current disks).

Cheers,
Jonathan

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
TeknoJnky Offline
Fan
Posts: 424
Joined: Aug 2008
Reputation: 0
Post: #36
for those looking for a single device backup (or even primary storage), you might consider western digitals my book studio ii, 4tb quad interface (usb/firewire400/800/esata). 3.63 tb formated, requires vista/win7 or mac (xp not compatible with GPT partition)

I'm using one as my primary audio storage (backing up to nas) and so far its great. It didn't come with an esata cable, but I picked up one at monoprice and it works great and is way faster than usb/firewire.

its a dual drive, 2tb mirror or 4tb stripe. Obviously if your using stripe, you should always have it backed up, but so far its been more stable than my previous generation 1 & 2 tb my books dual drives.

http://www.amazon.com/gp/product/B002BH3...ss_product
find quote
Seoras Offline
Junior Member
Posts: 49
Joined: Sep 2009
Reputation: 0
Post: #37
I personally did it on the cheap using a couple of Western Digital MyBook II 2TB units (2 x 1TB drives inside) in a RAID 1 setup. It's not ideal, but I am taking the calculated risk that both drives in each unit will not fail at the same time, thus I *should* be able to recover by simply popping in a new 1TB drive into the chassis. Of course I haven't actually tested this yet.. and that's the think about backups.. your plan is only as good as your restore plan.. if you don't test it then it's garbage! Which means.. yeah.. I need to get on that Smile

Both are currently connected to a machine via USB2 (no eSata on that old cranky box). Read speeds over USB2 were not a concern since the network that it is streaming over is 100Mbps wired.. i.e. USB2 isn't the bottleneck. (Which incidentally is an off topic rant, why have MOCA devices that support theoretical limits of 270Mbps but stick a 100Mbps ethernet jack in them sigh).

As others have said, you have to weigh your risks vs cost. Once you start talking about multiple TB's of data, backups become extremely expensive.. online doesn't really cut it for cost/time, and tape solutions are ridicuously overpriced... RAID is the cheapest way to add some level of safety net to your data but it's no substitute... so think about your data and a worst case scenario then decide how much that is worth to you... then get depressed Smile

You can always go out and build an unRaid or flexRaid system using all those old cheap throwaway drives that everyone seems to have lying around.. If you store your data on large external usb drives or whatever and then back them up to an unRaid system built from all the small crappy old drives you have lying around, you might be able to get out of it all relatively cheaply.
find quote
korvgryta Offline
Junior Member
Posts: 5
Joined: Mar 2010
Reputation: 0
Post: #38
Depending on your internet connection, you can do what me and a couple of friends do. Instead of us all spending a lot of cash on storage we've built a server and added about 6TB of storage to it (RAID5) and then simply mount it remotely over the internet from my friend's place with SSHFS or Expandrive for Win7.

This works flawlessly even with 1080p mkv material peaking 25Mbit. Browsing becomes slightly slower (not much) but playback is no problem at all.

This way I can also access my movies/music/pictures anywhere I go with my laptop Smile Xvid-files works over 3G.
find quote
xaos01 Offline
Junior Member
Posts: 35
Joined: Apr 2009
Reputation: 0
Post: #39
My current setup is listed in my sig. I bought a dual-proc G5 simply because it ran MacOS and had a PCI-X slot. Got it off ebay for $500 bucks. In that PCI-X slot I put a 4-Port eSATA Host Adapter with Port Multiplier Compatibility. Hanging off 2 of the ports are Five-Bay, Hot-Swap, External, SATA Port Multiplier Enclosures. Two ports are unused. I have 9 1TB Caviar Blacks, and 1 2TB Caviar Green all as separate drives, no RAID. The 2TB green is sketchy. I suppose it could be a bad drive, every once in a while, some file I put there bit-rots and becomes unplayable and locks up the Finder when it tries to create a thumb for it. I personally think that I was prolly writing to it too fast or something and the data was laid down wrong to begin with. I have a 2TB Black from NewEgg ready to go when I figure out what to do.

I either need to start swapping out 1TBs with 2TBs or get another chassis to continue collecting.

Nothing is redundant. It's bugged me since day 1, and now that I have like 1500 full rips, it _really_ bugs me. RAID never did everything I wanted. So I was looking around and thinking maybe I would build a Solaris ZFS box when I stumbled on unRAID. It looks like it would be _perfect_ for me.

So I think Im going to go that route and sell my 2 Chassis' and the card on eBay. It will be nice to have 3 boxes down to 1 with some level of fault tolerance. From the site fora, it appears that AFP is working or in the works.

Got a question about unRAID tho'.

How in the frak can I have unlimited media drives and a single parity drive and still survive a single drive failure? I just don't get it. Seems like some kinda' pocket-universe/bag-of-holding thing goin' on...

Oo

Platform - AppleTV
AppleTV Version - 3.0.1
Launcher version - 3.2.4
Install Method - Patchstick
SVN Revision - 9.11 R28256
(2)Seritek 5PM Chassis' :: 12TB
(This post was last modified: 2010-03-30 01:37 by xaos01.)
find quote
Kode Offline
fanart.tv admin
Posts: 749
Joined: Mar 2010
Reputation: 14
Post: #40
xaos01 Wrote:Got a question about unRAID tho'.

How in the frak can I have unlimited media drives and a single parity drive and still survive a single drive failure? I just don't get it. Seems like some kinda' pocket-universe/bag-of-holding thing goin' on...

Oo

Its just simple XOR

basically it reads all of the 0's and 1's on each drive and puts the result on the parity drive, ie

Drive 1: 0 1 0 0 1
Drive 2: 1 0 1 1 0
Drive 3: 1 0 0 1 0

Parity : 0 1 1 0 1

therefore if Drive 2 fails you are left with

Drive 1: 0 1 0 0 1
Drive 2: ? ? ? ? ?
Drive 3: 1 0 0 1 0

Parity : 0 1 1 0 1

All it has to do is work out what each ? needs to be in order for the value to equal the value stored on the parity drive
(This post was last modified: 2010-04-01 14:01 by Kode.)
find quote
teeedubb Offline
Donor
Posts: 2,367
Joined: Mar 2010
Reputation: 58
Location: Down Under
Post: #41
I use a amd le-1200 with 2 gig ram with ubuntu desktop installed (looking at switching to xfce soon). its more than adequate for serving media and downloading torrents, it takes around 8hrs to encode a dvd to mkv, but that isnt a issue for me as its on 24/7. id recommend using a pc over a nas box due to flexibility. ie you can add tv tuners and stream tv to any pc in your house

My Add-ons, only available for XBMC:
Steam Launcher|Music Party Mode|Reboot2oOS|Triple J Radio
find quote
olympia Offline
Team-XBMC Member
Posts: 2,497
Joined: May 2008
Reputation: 32
Post: #42
Kode Wrote:Its just simple XOR


Parity : 0 1 1 0 1

therefore if Drive 2 fails you are left with

Parity : 0 1 0 1 0

Parity remains the same, but I guess this is just a typo... Wink
find quote
Kode Offline
fanart.tv admin
Posts: 749
Joined: Mar 2010
Reputation: 14
Post: #43
olympia Wrote:Parity remains the same, but I guess this is just a typo... Wink

Yeah, i had to wait for my post to be moderated so i couldnt change anything, my point was as long as only 1 drive fails the missing drive can be worked out based on what would be required for all the disks to be the value stored on the parity drive
find quote
woodeye18 Offline
Junior Member
Posts: 5
Joined: Jan 2009
Reputation: 0
Post: #44
I use a WRT160nl Linksys router with usb support. It has my external drives attached to it, is always on and operates as a low energy NAS. It can be flashed with an opensource WRT OS also.
find quote
Lakestang Offline
Junior Member
Posts: 25
Joined: May 2009
Reputation: 0
Post: #45
I will cast a vote for Windows Home Server. I have been running a home built box since the beta and love it. I also have experience with the HP Mediasmart and really think it is the best network storage solution out there for the money.

I setup a MediaSmart on a very non-techie friends network more than two years ago and have not had to do anything to it, other than turn it on after power outages and add drives to it. They forget it is there. "oh, the little box in the spare bedroom"

You do not get a traditional raid, but, you do decide what folders to backup and it will do this automatically while serving all your network media needs AND automatically backing up all computers on your network.

The bonus is that if you lose a backed up computer, it is VERY simple to get it back with the server recovery feature. Also, if you want to upgrade your system drive, this is the same as a backup and painless. I have recovered from two hard drive failures with my WHS.

To me, this may be the most overlooked software/hardware solution out there.
find quote