Ceph? Anyone tried it yet?

[sparker]

I am looking at rebuilding a distributed robust storage solution and was considering two SANs with block level replication such as the Fujitsu Eternius 90 but then someone mentioned Ceph to me and so I was wondering if any of you fine people had any experience with it?

I like its (apparent) simplicity and scalability but it's a big investment for our school so I want as much reassurance as I can find....

[dhicks]

I am looking at rebuilding a distributed robust storage solution and was considering two SANs with block level replication such as the Fujitsu Eternius 90 but then someone mentioned Ceph to me and so I was wondering if any of you fine people had any experience with it? I like its (apparent) simplicity and scalability but it's a big investment for our school so I want as much reassurance as I can find....

 

I've been looking at Ceph, wondering if I'll do a test install on my home server (which is currently running OpenStack). How much experience of Linux-based servers and so forth do you have? As far as I can make out, Ceph's scaleable, expandable "block storage" is very good, but not usable as a /dev/xxx-style device from within Linux, although you can see a block as a file via a FUSE module, which would (I think) let you place a file system on that block and mount it via a loopback interface (performance, I imagine, will be poor). You're expected to attach each block of storage to a virtual machine, so Ceph is something to use on conjunction with a VM or cloud system (OpenStack seems like a popular choice).

 

When you say "a big investement", how much money is that? Kind of the idea of Ceph is that it can run on commodity hardware - two or three HP Microservers would do it, each with four harddrives, you don't have to spend more than a couple of thousand pounds for a reasonable-sized (10TB - 20TB) starter system, and you can add on to that as you go. If you'd prefer block-level replication, DRBD can do that for free, too.

[glennda]

I haven't read much of that page but it appears to be Object based storage rather than file system based storage?#

 

Take a look at Open-E Active Active ISCSI Software solutions these can be low cost and high capacity!

[dhicks]

I haven't read much of that page but it appears to be Object based storage rather than file system based storage?#

 

Ceph is an object-based storage system, but it has both a block-level storage device layer built on top of that and the CephFS filesystem, which has modules integrated with the Linux kernel so you don't have to use FUSE.

[pcstru]

Take a look at Open-E Active Active ISCSI Software solutions these can be low cost and high capacity!

And rather poorly supported by the vendor.

[sparker]

My current quote is for 22k for three servers( Dell) with 16Tb in each. I currently have a san with about 10 tb but will be able to arrange it more efficiently when I migrate it to whatever I end up with but what I want to end up with is a highly robust distributed solution ( I have two server rooms about 100mtrs apart) which will give me nearline reduncancy and then use my old SAN to replace my full to capacity NAS. I am running VMware atthe moment with HA enabled so my main weakness is that everything is in one rack and is at risk from theft fire flood as a result and if that happened I reckon the downtime would be 6-8 weeks - assuming I could rebuild it all as it isn't all backed up because I don't have room.

I am looking for block level replication and need a supported solution as I may not be at the school for ever plus my linux skills are somewhat rudimentary. If it offers performance and scalability advantages over a 'standard 'pair of block level replicating sans then it seems good but hard to evaluate from a position of no experience hence the shout out to your kind expertise!

The company offering me Ceph also want £2200 a year for support as well ( 20%) which seems a bit steep.

What would you recommend?

[glennda]

And rather poorly supported by the vendor.

 

What issues do you have? i've always found the support brilliant, but i'm also a trained engineer on the system!

 

Sorry for taking thread off topic

[pcstru]

What issues do you have? i've always found the support brilliant, but i'm also a trained engineer on the system!

It's rather difficult to say what the issues are when Open E themselves fail to identify them. I think in the end they were muttering about Raid subsystems not correctly reporting span sizes blah-blah. The upshot of that for us was total functional loss of a dual node system, a weekend spent reconfiguring to run one node and then a month without any ability to make changes or replication. I've had to use their support in anger and while I'm not a trained Open-E engineer, I know when I'm getting quality support on a business critical system ... and when I'm not :-).

[glennda]

I've had to use their support in anger and while I'm not a trained Open-E engineer, I know when I'm getting quality support on a business critical system ... and when I'm not :-).

 

Fair play - i havnt had an issue with the support, what hardware do you have it running on?

[pcstru]

Fair play - i havnt had an issue with the support, what hardware do you have it running on?

We run two Dell 720xd boxes - as recommended by our local Open-E reseller (who I hope as a reseller is trained). I wish our experience was different because otherwise it would be good value. I've had much more expensive (Datacore) systems go wrong in past lives, which was infuriating (especially since an expensive dual node system 'crashed' due to a single disc in an array failing to quite fail), but I never doubted they understood the position we were in.

[dhicks]

I currently have a san with about 10 tb but will be able to arrange it more efficiently when I migrate it to whatever I end up with but what I want to end up with is a highly robust distributed solution

 

Block-level replication isn't quite the same thing as a distributed "cloud" system - you have one server mirror another and take over in case of failure. We have this set up with DRBD here (Linux-specific, in our case running under XenServer, commercial support is available from LinBit, and you'll probably find that the underlying block-level replication mechanism on many Linux-based SANs is actually DRBD), with both our main XenServer machines mirroring to a third backup server. Interestingly, you might be able to get better read/write performance out of a Ceph cluster - I think the default settings are for 3 replicas of data (two local, one remote), so you have more disks to do read operations on - in a DRBD setup the "slave" replica isn't a live, usable block device so doesn't help with read operations, and depending on the replication algorythm used can require remote writes to finish before a write operation is complete, slowing write operations. Bear in mind that you'll probably wind up with less usable disk space using Ceph than with a RAID array - Ceph replicates data, it doesn't just checksum it like a RAID controller does.

 

Ceph (and other cloud storage solutions) might not the kind of thing to base your virtual infrastructure on with no prior experience. I've had a bit of experience tinkering with my own OpenStack instance at home and am confident that I could at least get a system back up if it conked out - I wouldn't want to spend £22,000 on something if I wasn't quite sure how to use it, though. Ceph's kernel-based drivers are only in the most up-to-date Linux distributions - Fedora 18 and 19 (I think I had to use a release candidate of Fedora 19 just to get OpenStack installed at home in the end). Debian and Red Hat are very stable but can be several years behind in kernel versions - Ceph and similar are still somewhat at the cutting edge.

 

I am looking for block level replication and need a supported solution as I may not be at the school for ever plus my linux skills are somewhat rudimentary. If it offers performance and scalability advantages over a 'standard 'pair of block level replicating sans then it seems good but hard to evaluate from a position of no experience hence the shout out to your kind expertise!

 

Yes, Ceph should hopefully offer more scaeability and better performance than block-level replication, but you're going to need support - both DRBD and Ceph are the kind of things which can be fiddly to set up and keep running.

 

The company offering me Ceph also want £2200 a year for support as well ( 20%) which seems a bit steep.

 

If that's what their expertise costs then there you are - if you don't have the expertise yourself you'll have to pay someone. Is the company concerned InkTank (basically the team that built Ceph in the first place)? If not, it might be worth getting a comparative quote from them - even if they are pricy, they are the ones who built Ceph and should know exactly what they are talking about.

[sparker]

Thanks Dave - good advice - i'll drop InkTank a line and see what they say.

[dhicks]

My current quote is for 22k for three servers( Dell) with 16Tb in each.

 

Just coming back round to this subject: the more I read about Ceph, the more like a useful system it seems. It seems to be popular as a component in other systems - as a backing store for VM disk images for XenServer and as a drop-in replacement storage system for OpenStack and Hadoop. It would seem to be usable as a standard network-available share, too, if I mount the Ceph volume on a server and then share it out with Samba.

 

I'm interested in doing some Hadoop-style distributed processing but without the bother of installing Hadoop, just using each Ceph node for processing as well as storage. If I can get object location information from Ceph then I can distribute the computation task to where its data is stored. This seems to be possible, from what I can read in the documentation, so I'm planning to set up some servers to give it a go. Has anyone else tried anything similar? What sort of hardware are other people running Ceph on? If you have servers with multiple disks, do you find it best to amalgamate those disks into a RAID array (via hardware or via md RAID) first, then assign them to a Ceph OSD, or is it best to let Ceph run one OSD per disk? Do you have SSDs for journals in your servers, if so, what kind?

[mavhc]

Just been looking at Ceph and Proxmox this week.

 

Seems best to run one OSD per disk.

 

Main question is what to use to network them together, Mellanox MHQH29B-XSR 40Gbps 2Port Infiniband cards are cheap, so 3 of those with 3 servers and 6 DACs would make a full mesh.

[dhicks]

Just been looking at Ceph and Proxmox this week.

 

Six years on - we stuck with DRBD mirrored volumes on XenServer for our (school / work) production server setup, and although it's now looking dated we're still only just sorting out moving the VMS and storage off, so it's lasted very well! For the new server setup we've gone with basic Windows Server 2019 with Storage Spaces - we needed to run Windows Server VMs, so the licensing needed to be in place anyway, and we get vendor-supported hardware.

 

It's getting less relevant for the average school / business to host their own server hardware - even our new file server is probably going to mostly be moved for backups once we get the working file shares on to Google Drive.

 

In terms of the kind of computational / storage problem that Ceph is built to handle (if I understand correctly, AWS Lambda-style jobs with VMs that start up, connect to a block-based storage pool, then quit when done), I'm having trouble finding tasks suitible for that kind of computing model. Day-to-day tasks still tend to best be solvable (as far as I can see) with a program that gathers inputs (generally, files from a traditional file system), does some processing, then produces some outputs (again, often back out to a file system).

[mavhc]

Well, seems to be used for VM storage a lot, mostly what proxmox is using it for, HCI.

 

Are you using S2D?

[dhicks]

Are you using S2D?

 

Storage Spaces Direct? I can't remember which bit of functionality that is - I know we configured the Windows server to have a local "pool" of storage with an underlying tier of larger disks (several 8TB 3.5" drives) with a faster tier above that (two 1TB SSDs), so most writes (and a lot of the reads) will happen on the fast SSD drives. We don't have any real-time failover for VMs set up - both our email and MIS are now web-based, we're starting to move the file servers the same way, so there really isn't going to be that much left running locally.

[mavhc]

S2D is the cluster storage part, so similar to Ceph, except it has RDMA working, which is why you need specific HW for it to work really. AIUI