CHiLL Posted November 4, 2020 Posted November 4, 2020 (edited) We have 2x HP 2040 MSA enclosures, running 12x 600GB 15k SAS disks for OS and 12x 2TB 7.2k MDL drives for storage respectively, both using RAID5. We are considering replacing the storage with SSDs and I'm just researching best practice. HP state that the SATA controller in the unit is the limiting factor, so SSD potential is, well, limited. Current configuration: Enclosure 1 = Pool A - OS Disks - 11x 600GB 15k SAS drives - 5.4TB total storage Disk group A01 = 6x 600GB RAID5 = 3TB, 5x read speed, no write speed gain Disk group A02 = 5x 600GB RAID5 = 2.4TB, 4x read speed, no write speed gain Enclosure 2 = Pool B - Data Disks - 11x 2TB 7.2k MDL drives - 18TB total storage Disk group B01 = 6x 2TB RAID5 = 10TB, 5x read speed, no write speed gain Disk group B02 = 5x 2TB RAID5 = 8TB, 4x read speed, no write speed gain This was configured by an external company during an upgrade in 2015. I don't know why each pool has two RAID arrays for the same storage type. The only thing I can think of that is that each array can suffer 1 drive failure at the same time, whereas if in one large array, two drives dying at the same time would be a bad time. Assuming that's the only reason they chose that over RAID10, I've come up with the following so far. First draft upgrade: Enclosure 1 - We're using about 4TB of this pool Disk group A01 = 6x 1TB RAID10 = 3TB, 6x read and 3x write speed gain Disk group A02 = 4x 1TB RAID10 = 2TB, 4x read and 2x write speed gain Enclosure 2 - We're using about 6.5TB of this pool Disk group B01 = 4x 2TB RAID10 = 4TB, 4x read and 2x write speed gain Disk group B02 = 4x 2TB RAID10 = 4TB, 4x read and 2x write speed gain Plus: Multiple arrays to combat multiple drive failures Negative: Expensive 2TB drives. Unsure if the write speed gains can be utilised over an SSDs raw speed vs controller limitations. Alternatively, I could remove the multiple disk groups and have: Enclosure 1 Disk group A01 = 10x 1TB RAID10 = 5TB, 10x read and 5x write speed gain Enclosure 2 Disk group B01 = 8x 2TB RAID10 = 5TB, 8x read and 4x write speed gain I have a couple of questions; 1) Given the SATA controller limitation, will there be a difference in the performance between RAID5 and RAID10? 2) Is it worth replacing the MSA with new storage device, like one that is designed for SSD storage? I'm just wondering, if we splash out on SSD drives, are we shooting ourselves in the foot by putting them in the 2040? 3) What type of SSDs do we need to consider? Are generic drives fine, or do we need specific enterprise/NAS level drives? 4) Is there anything else I need to consider, other than migrating the data to a temporary unit while the drives are replaced and the associated configuration requirements? Edited November 4, 2020 by CHiLL
Steve21 Posted November 4, 2020 Posted November 4, 2020 Do you know how your IOPS etc are currently? Given the speed difference from HDD to SSD do you even need to worry about RAID10 vs RAID6 etc which should still be a lot faster and more secure than 5 etc but with the added storage too meaning could go with cheaper/smaller SSDs Steve
CHiLL Posted November 5, 2020 Author Posted November 5, 2020 Do you know how your IOPS etc are currently? Given the speed difference from HDD to SSD do you even need to worry about RAID10 vs RAID6 etc which should still be a lot faster and more secure than 5 etc but with the added storage too meaning could go with cheaper/smaller SSDs Steve I have no idea what the IOPS are at the moment. I've just downloaded DiskSpd, though will need to wait for out of hours to run it. I hadn't considered RAID6, but that'd give us the 2 disk failure that we currently have with 2x RAID5 arrays. I'm liking this idea. With that in mind: Enclosure 1 Disk group A01 = 11x 1TB RAID6 = 9TB, 9x read speed, no write speed gain & 1 extra hot spare in bay 12 Enclosure 2 Disk group B01 = 11x 1TB RAID6 = 9TB, 9x read speed, no write speed gain & 1 extra hot spare in bay 12 My only concern would be future expansion, as that'd mean replacing all the disks again due to no spare bays. However, we'd have 18TB of usable storage...which should be fine, given we're using 6.5TB at present and never gone above 7TB.
titch Posted November 9, 2020 Posted November 9, 2020 Are you planning on setting the SSDs up as a performance tier or just their own group?
CHiLL Posted November 10, 2020 Author Posted November 10, 2020 (edited) Are you planning on setting the SSDs up as a performance tier or just their own group? They would be the only storage for live data, so which ever setup is best (I'm not entirely sure) I've got the IOPS results from using Diskspd, but the output log file is a bit more than I understand! Command Line: C:\DiskSpd-2.0.21a\amd64\diskspd.exe -c18G -d300 -r -w40 -t8 -o32 -b64K -Sh -L C:\Cache\diskspdtmp.dat Input parameters: timespan: 1 ------------- duration: 300s warm up time: 5s cool down time: 0s measuring latency random seed: 0 path: 'C:\Cache\diskspdtmp.dat' think time: 0ms burst size: 0 software cache disabled hardware write cache disabled, writethrough on performing mix test (read/write ratio: 60/40) block size: 65536 using random I/O (alignment: 65536) number of outstanding I/O operations: 32 thread stride size: 0 threads per file: 8 using I/O Completion Ports IO priority: normal System information: computer name: SERVER start time: 2020/11/07 05:05:06 UTC Results for timespan 1: ******************************************************************************* actual test time: 300.00s thread count: 8 proc count: 2 CPU | Usage | User | Kernel | Idle ------------------------------------------- 0| 31.85%| 9.60%| 22.25%| 68.15% 1| 28.87%| 8.87%| 20.00%| 71.13% ------------------------------------------- avg.| 30.36%| 9.24%| 21.13%| 69.64% Total IO thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file ----------------------------------------------------------------------------------------------------- 0 | 650444800 | 9925 | 2.07 | 33.08 | 966.636 | 86.579 | C:\Cache\diskspdtmp.dat (18GiB) 1 | 650706944 | 9929 | 2.07 | 33.10 | 966.378 | 86.564 | C:\Cache\diskspdtmp.dat (18GiB) 2 | 651952128 | 9948 | 2.07 | 33.16 | 967.062 | 87.063 | C:\Cache\diskspdtmp.dat (18GiB) 3 | 650379264 | 9924 | 2.07 | 33.08 | 966.660 | 86.358 | C:\Cache\diskspdtmp.dat (18GiB) 4 | 651558912 | 9942 | 2.07 | 33.14 | 966.308 | 87.105 | C:\Cache\diskspdtmp.dat (18GiB) 5 | 650117120 | 9920 | 2.07 | 33.07 | 966.481 | 86.156 | C:\Cache\diskspdtmp.dat (18GiB) 6 | 650641408 | 9928 | 2.07 | 33.09 | 966.684 | 87.348 | C:\Cache\diskspdtmp.dat (18GiB) 7 | 650117120 | 9920 | 2.07 | 33.07 | 966.649 | 86.409 | C:\Cache\diskspdtmp.dat (18GiB) ----------------------------------------------------------------------------------------------------- total: 5205917696 | 79436 | 16.55 | 264.79 | 966.607 | 86.699 Read IO thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file ----------------------------------------------------------------------------------------------------- 0 | 391774208 | 5978 | 1.25 | 19.93 | 969.148 | 87.736 | C:\Cache\diskspdtmp.dat (18GiB) 1 | 389283840 | 5940 | 1.24 | 19.80 | 967.701 | 84.773 | C:\Cache\diskspdtmp.dat (18GiB) 2 | 391774208 | 5978 | 1.25 | 19.93 | 969.394 | 86.748 | C:\Cache\diskspdtmp.dat (18GiB) 3 | 394264576 | 6016 | 1.25 | 20.05 | 968.258 | 86.925 | C:\Cache\diskspdtmp.dat (18GiB) 4 | 389152768 | 5938 | 1.24 | 19.79 | 969.499 | 87.101 | C:\Cache\diskspdtmp.dat (18GiB) 5 | 390791168 | 5963 | 1.24 | 19.88 | 966.871 | 86.099 | C:\Cache\diskspdtmp.dat (18GiB) 6 | 389218304 | 5939 | 1.24 | 19.80 | 966.929 | 85.536 | C:\Cache\diskspdtmp.dat (18GiB) 7 | 392101888 | 5983 | 1.25 | 19.94 | 968.177 | 85.199 | C:\Cache\diskspdtmp.dat (18GiB) ----------------------------------------------------------------------------------------------------- total: 3128360960 | 47735 | 9.94 | 159.12 | 968.248 | 86.277 Write IO thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file ----------------------------------------------------------------------------------------------------- 0 | 258670592 | 3947 | 0.82 | 13.16 | 962.833 | 84.656 | C:\Cache\diskspdtmp.dat (18GiB) 1 | 261423104 | 3989 | 0.83 | 13.30 | 964.407 | 89.127 | C:\Cache\diskspdtmp.dat (18GiB) 2 | 260177920 | 3970 | 0.83 | 13.23 | 963.551 | 87.417 | C:\Cache\diskspdtmp.dat (18GiB) 3 | 256114688 | 3908 | 0.81 | 13.03 | 964.200 | 85.420 | C:\Cache\diskspdtmp.dat (18GiB) 4 | 262406144 | 4004 | 0.83 | 13.35 | 961.575 | 86.895 | C:\Cache\diskspdtmp.dat (18GiB) 5 | 259325952 | 3957 | 0.82 | 13.19 | 965.892 | 86.238 | C:\Cache\diskspdtmp.dat (18GiB) 6 | 261423104 | 3989 | 0.83 | 13.30 | 966.320 | 89.977 | C:\Cache\diskspdtmp.dat (18GiB) 7 | 258015232 | 3937 | 0.82 | 13.12 | 964.327 | 88.164 | C:\Cache\diskspdtmp.dat (18GiB) ----------------------------------------------------------------------------------------------------- total: 2077556736 | 31701 | 6.60 | 105.67 | 964.137 | 87.273 total: %-ile | Read (ms) | Write (ms) | Total (ms) ---------------------------------------------- min | 658.249 | 652.165 | 652.165 25th | 912.309 | 907.597 | 910.395 50th | 958.279 | 953.866 | 956.576 75th | 1011.472 | 1007.126 | 1009.785 90th | 1073.024 | 1070.731 | 1072.306 95th | 1134.677 | 1136.793 | 1135.181 99th | 1238.374 | 1232.926 | 1236.248 3-nines | 1328.290 | 1326.648 | 1326.891 4-nines | 1410.584 | 1425.395 | 1421.708 5-nines | 1455.508 | 1445.149 | 1455.508 6-nines | 1455.508 | 1445.149 | 1455.508 7-nines | 1455.508 | 1445.149 | 1455.508 8-nines | 1455.508 | 1445.149 | 1455.508 9-nines | 1455.508 | 1445.149 | 1455.508 max | 1455.508 | 1445.149 | 1455.508 Edited November 10, 2020 by CHiLL
titch Posted November 10, 2020 Posted November 10, 2020 I'm no pro. But would suggest setting them up as a performance tier which then means data is dynamically moved to the SSDs when it's being used and back to midline/storage when it's finished. Kind of like a cacheing drive but the data resides on the SSDs. Out of interest, have you got any kind of redundancy for your SAN. We recently had a major balls up with faulty drives in our SAN. It had never really occured to me before then stupidly that this was a single point of failure despite having a decent cluster.
CHiLL Posted November 11, 2020 Author Posted November 11, 2020 I'm no pro. But would suggest setting them up as a performance tier which then means data is dynamically moved to the SSDs when it's being used and back to midline/storage when it's finished. Kind of like a cacheing drive but the data resides on the SSDs. Out of interest, have you got any kind of redundancy for your SAN. We recently had a major balls up with faulty drives in our SAN. It had never really occured to me before then stupidly that this was a single point of failure despite having a decent cluster. We only have the dual MSA for production storage. Other than the RAID configuration, there is no other redundancy, except for our nightly backups to a separate QNAP NAS. We were of the idea that all spinning disks of the MSA would be replaced with SSDs. 1
titch Posted November 11, 2020 Posted November 11, 2020 We only have the dual MSA for production storage. Other than the RAID configuration, there is no other redundancy, except for our nightly backups to a separate QNAP NAS. We were of the idea that all spinning disks of the MSA would be replaced with SSDs.Very Gucci going for all SSDs! What sort of prices are they coming in at? Do you think all your data deserves to sit on SSDs or could you make do with cacheing and save a heap of money? Word of warning, and this is just me being bitter with burnt fingers. But I would now be cautious of having all drives of the same model in any production SAN without appropriate redundancy. If something is wrong with those drives you may find they all go pop at the same time meaning no RAID is going to help you.
CHiLL Posted November 12, 2020 Author Posted November 12, 2020 Very Gucci going for all SSDs! What sort of prices are they coming in at? Do you think all your data deserves to sit on SSDs or could you make do with cacheing and save a heap of money? Word of warning, and this is just me being bitter with burnt fingers. But I would now be cautious of having all drives of the same model in any production SAN without appropriate redundancy. If something is wrong with those drives you may find they all go pop at the same time meaning no RAID is going to help you. We haven't got to the stage of pricing yet, I'm just putting feelers out to see if it's something we can do or if it's worth doing, with the infrastructure we have.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now