+ Post New Thread
Results 1 to 13 of 13
Hardware Thread, Ceph? Anyone tried it yet? in Technical; I am looking at rebuilding a distributed robust storage solution and was considering two SANs with block level replication such ...
  1. #1

    Join Date
    Jun 2010
    Location
    Berkshire
    Posts
    111
    Thank Post
    18
    Thanked 9 Times in 9 Posts
    Rep Power
    10

    Question Ceph? Anyone tried it yet?

    I am looking at rebuilding a distributed robust storage solution and was considering two SANs with block level replication such as the Fujitsu Eternius 90 but then someone mentioned Ceph to me and so I was wondering if any of you fine people had any experience with it?
    I like its (apparent) simplicity and scalability but it's a big investment for our school so I want as much reassurance as I can find....

  2. #2

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,611
    Thank Post
    1,226
    Thanked 772 Times in 670 Posts
    Rep Power
    234
    Quote Originally Posted by sparker View Post
    I am looking at rebuilding a distributed robust storage solution and was considering two SANs with block level replication such as the Fujitsu Eternius 90 but then someone mentioned Ceph to me and so I was wondering if any of you fine people had any experience with it? I like its (apparent) simplicity and scalability but it's a big investment for our school so I want as much reassurance as I can find....
    I've been looking at Ceph, wondering if I'll do a test install on my home server (which is currently running OpenStack). How much experience of Linux-based servers and so forth do you have? As far as I can make out, Ceph's scaleable, expandable "block storage" is very good, but not usable as a /dev/xxx-style device from within Linux, although you can see a block as a file via a FUSE module, which would (I think) let you place a file system on that block and mount it via a loopback interface (performance, I imagine, will be poor). You're expected to attach each block of storage to a virtual machine, so Ceph is something to use on conjunction with a VM or cloud system (OpenStack seems like a popular choice).

    When you say "a big investement", how much money is that? Kind of the idea of Ceph is that it can run on commodity hardware - two or three HP Microservers would do it, each with four harddrives, you don't have to spend more than a couple of thousand pounds for a reasonable-sized (10TB - 20TB) starter system, and you can add on to that as you go. If you'd prefer block-level replication, DRBD can do that for free, too.

  3. Thanks to dhicks from:

    sparker (11th December 2013)

  4. #3

    glennda's Avatar
    Join Date
    Jun 2009
    Location
    Sussex
    Posts
    7,784
    Thank Post
    272
    Thanked 1,129 Times in 1,025 Posts
    Rep Power
    348
    I haven't read much of that page but it appears to be Object based storage rather than file system based storage?#

    Take a look at Open-E Active Active ISCSI Software solutions these can be low cost and high capacity!

  5. Thanks to glennda from:

    sparker (11th December 2013)

  6. #4

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,611
    Thank Post
    1,226
    Thanked 772 Times in 670 Posts
    Rep Power
    234
    Quote Originally Posted by glennda View Post
    I haven't read much of that page but it appears to be Object based storage rather than file system based storage?#
    Ceph is an object-based storage system, but it has both a block-level storage device layer built on top of that and the CephFS filesystem, which has modules integrated with the Linux kernel so you don't have to use FUSE.

  7. Thanks to dhicks from:

    sparker (11th December 2013)

  8. #5


    Join Date
    May 2009
    Posts
    2,877
    Thank Post
    258
    Thanked 766 Times in 581 Posts
    Rep Power
    269
    Quote Originally Posted by glennda View Post
    Take a look at Open-E Active Active ISCSI Software solutions these can be low cost and high capacity!
    And rather poorly supported by the vendor.

  9. #6

    Join Date
    Jun 2010
    Location
    Berkshire
    Posts
    111
    Thank Post
    18
    Thanked 9 Times in 9 Posts
    Rep Power
    10
    My current quote is for 22k for three servers( Dell) with 16Tb in each. I currently have a san with about 10 tb but will be able to arrange it more efficiently when I migrate it to whatever I end up with but what I want to end up with is a highly robust distributed solution ( I have two server rooms about 100mtrs apart) which will give me nearline reduncancy and then use my old SAN to replace my full to capacity NAS. I am running VMware atthe moment with HA enabled so my main weakness is that everything is in one rack and is at risk from theft fire flood as a result and if that happened I reckon the downtime would be 6-8 weeks - assuming I could rebuild it all as it isn't all backed up because I don't have room.
    I am looking for block level replication and need a supported solution as I may not be at the school for ever plus my linux skills are somewhat rudimentary. If it offers performance and scalability advantages over a 'standard 'pair of block level replicating sans then it seems good but hard to evaluate from a position of no experience hence the shout out to your kind expertise!
    The company offering me Ceph also want 2200 a year for support as well ( 20%) which seems a bit steep.
    What would you recommend?

  10. #7

    glennda's Avatar
    Join Date
    Jun 2009
    Location
    Sussex
    Posts
    7,784
    Thank Post
    272
    Thanked 1,129 Times in 1,025 Posts
    Rep Power
    348
    Quote Originally Posted by pcstru View Post
    And rather poorly supported by the vendor.
    What issues do you have? i've always found the support brilliant, but i'm also a trained engineer on the system!

    Sorry for taking thread off topic

  11. #8


    Join Date
    May 2009
    Posts
    2,877
    Thank Post
    258
    Thanked 766 Times in 581 Posts
    Rep Power
    269
    Quote Originally Posted by glennda View Post
    What issues do you have? i've always found the support brilliant, but i'm also a trained engineer on the system!
    It's rather difficult to say what the issues are when Open E themselves fail to identify them. I think in the end they were muttering about Raid subsystems not correctly reporting span sizes blah-blah. The upshot of that for us was total functional loss of a dual node system, a weekend spent reconfiguring to run one node and then a month without any ability to make changes or replication. I've had to use their support in anger and while I'm not a trained Open-E engineer, I know when I'm getting quality support on a business critical system ... and when I'm not :-).

  12. #9

    glennda's Avatar
    Join Date
    Jun 2009
    Location
    Sussex
    Posts
    7,784
    Thank Post
    272
    Thanked 1,129 Times in 1,025 Posts
    Rep Power
    348
    Quote Originally Posted by pcstru View Post
    I've had to use their support in anger and while I'm not a trained Open-E engineer, I know when I'm getting quality support on a business critical system ... and when I'm not :-).
    Fair play - i havnt had an issue with the support, what hardware do you have it running on?

  13. #10


    Join Date
    May 2009
    Posts
    2,877
    Thank Post
    258
    Thanked 766 Times in 581 Posts
    Rep Power
    269
    Quote Originally Posted by glennda View Post
    Fair play - i havnt had an issue with the support, what hardware do you have it running on?
    We run two Dell 720xd boxes - as recommended by our local Open-E reseller (who I hope as a reseller is trained). I wish our experience was different because otherwise it would be good value. I've had much more expensive (Datacore) systems go wrong in past lives, which was infuriating (especially since an expensive dual node system 'crashed' due to a single disc in an array failing to quite fail), but I never doubted they understood the position we were in.

  14. #11

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,611
    Thank Post
    1,226
    Thanked 772 Times in 670 Posts
    Rep Power
    234
    I currently have a san with about 10 tb but will be able to arrange it more efficiently when I migrate it to whatever I end up with but what I want to end up with is a highly robust distributed solution
    Block-level replication isn't quite the same thing as a distributed "cloud" system - you have one server mirror another and take over in case of failure. We have this set up with DRBD here (Linux-specific, in our case running under XenServer, commercial support is available from LinBit, and you'll probably find that the underlying block-level replication mechanism on many Linux-based SANs is actually DRBD), with both our main XenServer machines mirroring to a third backup server. Interestingly, you might be able to get better read/write performance out of a Ceph cluster - I think the default settings are for 3 replicas of data (two local, one remote), so you have more disks to do read operations on - in a DRBD setup the "slave" replica isn't a live, usable block device so doesn't help with read operations, and depending on the replication algorythm used can require remote writes to finish before a write operation is complete, slowing write operations. Bear in mind that you'll probably wind up with less usable disk space using Ceph than with a RAID array - Ceph replicates data, it doesn't just checksum it like a RAID controller does.

    Ceph (and other cloud storage solutions) might not the kind of thing to base your virtual infrastructure on with no prior experience. I've had a bit of experience tinkering with my own OpenStack instance at home and am confident that I could at least get a system back up if it conked out - I wouldn't want to spend 22,000 on something if I wasn't quite sure how to use it, though. Ceph's kernel-based drivers are only in the most up-to-date Linux distributions - Fedora 18 and 19 (I think I had to use a release candidate of Fedora 19 just to get OpenStack installed at home in the end). Debian and Red Hat are very stable but can be several years behind in kernel versions - Ceph and similar are still somewhat at the cutting edge.

    I am looking for block level replication and need a supported solution as I may not be at the school for ever plus my linux skills are somewhat rudimentary. If it offers performance and scalability advantages over a 'standard 'pair of block level replicating sans then it seems good but hard to evaluate from a position of no experience hence the shout out to your kind expertise!
    Yes, Ceph should hopefully offer more scaeability and better performance than block-level replication, but you're going to need support - both DRBD and Ceph are the kind of things which can be fiddly to set up and keep running.

    The company offering me Ceph also want 2200 a year for support as well ( 20%) which seems a bit steep.
    If that's what their expertise costs then there you are - if you don't have the expertise yourself you'll have to pay someone. Is the company concerned InkTank (basically the team that built Ceph in the first place)? If not, it might be worth getting a comparative quote from them - even if they are pricy, they are the ones who built Ceph and should know exactly what they are talking about.

  15. Thanks to dhicks from:

    sparker (11th December 2013)

  16. #12

    Join Date
    Jun 2010
    Location
    Berkshire
    Posts
    111
    Thank Post
    18
    Thanked 9 Times in 9 Posts
    Rep Power
    10
    Thanks Dave - good advice - i'll drop InkTank a line and see what they say.

  17. #13

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,611
    Thank Post
    1,226
    Thanked 772 Times in 670 Posts
    Rep Power
    234
    Quote Originally Posted by sparker View Post
    My current quote is for 22k for three servers( Dell) with 16Tb in each.
    Just coming back round to this subject: the more I read about Ceph, the more like a useful system it seems. It seems to be popular as a component in other systems - as a backing store for VM disk images for XenServer and as a drop-in replacement storage system for OpenStack and Hadoop. It would seem to be usable as a standard network-available share, too, if I mount the Ceph volume on a server and then share it out with Samba.

    I'm interested in doing some Hadoop-style distributed processing but without the bother of installing Hadoop, just using each Ceph node for processing as well as storage. If I can get object location information from Ceph then I can distribute the computation task to where its data is stored. This seems to be possible, from what I can read in the documentation, so I'm planning to set up some servers to give it a go. Has anyone else tried anything similar? What sort of hardware are other people running Ceph on? If you have servers with multiple disks, do you find it best to amalgamate those disks into a RAID array (via hardware or via md RAID) first, then assign them to a Ceph OSD, or is it best to let Ceph run one OSD per disk? Do you have SSDs for journals in your servers, if so, what kind?

SHARE:
+ Post New Thread

Similar Threads

  1. System Center 2012... Anyone Tried It Yet?
    By Gongalong in forum Thin Client and Virtual Machines
    Replies: 8
    Last Post: 20th April 2012, 08:50 AM
  2. DFS and VHD: Anyone tried it?
    By Mr.Ben in forum Wireless Networks
    Replies: 5
    Last Post: 19th April 2011, 01:01 PM
  3. Replies: 4
    Last Post: 2nd November 2010, 09:33 PM
  4. Replies: 4
    Last Post: 13th October 2008, 11:36 AM
  5. GroundWork Monitor - anyone tried it?
    By gshaw in forum Wireless Networks
    Replies: 25
    Last Post: 9th October 2008, 09:33 AM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •