Advice please - CCTV server bandwidth issue

[CharlieKES]

Hi all, I'm after some advice for our school CCTV server.

 

Setup:

Hardware: Dell R510, 8gb ram (on average 4gb used), 12TB set to raid 1 storage (so we get great write speed), two gigabit Nic's(only one is currently used to pull data from cameras)

Software: Windows Server 2012, Luxriot Evo S

Switches: Cisco 3750's

 

Problem:

The server and software works well and the ram, cpu doesn't seem to be straining under average load, but the video playback isn't as smooth as we'd like. The past few months I've been eliminating possible bottlenecks,which had led to my concluding its a network bandwidth issue. By my calculations, if I have 85 cameras feeding my server 4Mbps a second, I need at least 340Mbps of bandwidth. I obviously can't get that with a single gigabit connection (it seems to hit 200Mbps and not go any further, as that's a limit for gigabit connection).

 

So my plan is to activate the other nic and have dual gigabit to help with the bandwidth load from all the cameras. The problem is, I'm not sure the best way to go about this:

• Should I team up the ports on the Cisco switch?
  
• Or should I have half my cameras use 10.110.100.x and the other have use 10.110.101.x?
  

 

What do you guys think?

 

Thanks for any advice you could give.

Edited July 17, 2017 by CharlieKES

[psydii]

I'd question whether you need to be recording at that resolution/frame rate. Dial it back to 720 5fps and you've got plenty of storage and bandwidth on a gig connection.

 

As to a direct answer to your question, link aggregation with src-dst-ip load balancing as this seems to offer the greatest compatibility in various OS/Switch combinations. YMMV.

 

Just to check though, are you talkng about 200/340 Mega BITS or Mega Bytes per second? 125 Mbytes/sec is the absolute theoretical maximum for gigabit, before taking into account packet size, and protocol overheads. If bits, then you really ought to be getting more that what you are seeing.

 

If Bytes one also has to consider disk performance because 340Mbytes random IO per second is quite a push for disks.

[CharlieKES]

Thanks for your reply, I've attached a screenshot if it helps.

[Blue_Cookeh]

LAG would improve that. I'd look at getting a quad port NIC and initially bonding two of the links on a switch located somewhere in your network that has decent uplinks to the camera switches.

 

Also FWIW we see a lot higher speeds on our gigabit links, typically over 500Mbps on our SCCM server when imaging machines... and that's a Hyper-V VM.

[CharlieKES]

Thanks for your help, I'll look into checking the configuration of our switch.

[AlanD]

Setting up a Team/LAG between server and switch seems like a no brainer....I'll assume its a core switch and cameras are well spread out between the ports and edge switches so there is no other potential bottleneck.

 

I'd be interested to know what the data rate is per camera,

 

Its possible that the problem is with the Disk IO - which might well cope with sequential read/writes of the amount required - but are probably writing to 85 different files at a time - so the access time and IOPs begin to get critical....especially if these are non enterprise SATA disks in the raid....perhaps running at 7K2

Its also possible with non enterprise disks that there are "poor" sectors which are not marked as failures - but are causing re-read/writes,

 

...and then if you want to view a file at the same time as all this is going on ....

[MkII]

You're scaring me now AlanD.. we're about to buy a 64 channel Hikvision NVR with a couple of disks to record feeds from 60 cameras.

[AlanD]

Sorry - I have little experience with CCTV except for the few we have - which are not IP based ...and not HD or high frame rate.

 

I'm sure if boxes are sold which support 64 video streams they will work - or at least will work provided you install the correct "Video" if not "Enterprise" disks as they require. A back of the envelope calculation tells me that video streams are potentially around 5Mb/s (without compression and dependent on frame rate of course). 100 cameras are going to potentially take away half of your 1Gb/s networking ...and writing 500Mb/s is seriously quick - but should be possible with sequential access to RAID 1 disk set. In practice I am guesing that with slower frame rates and compression its going to be a lot less than this..perhaps under 100Mb/s.

 

I'd be asking about exactly how much data is being transferred per camera per second to check its impact on the network.

[psydii]

There is usually a bandwidth calculator available at the manufacturer's website.

 

https://www.google.co.uk/search?q=hikvision+bandwidth+calculator&oq=hikvision+band&aqs=chrome.0.0j69i57j0l4.7727j0j7&sourceid=chrome&ie=UTF-8

 

Generally speaking for a 1080p 25fps stream you are looking at 6MBit per second.

 

Quite why you'd need to record at that resolution I am not sure, it could get expensive really quickly, with 60 cameras you'd need about 700IOPs which could require about 5-7 disks. I've never heard of anyone recording at that resolution, and you'd need 32TB to store a months worth of footage.

 

 

720 at 5FPs, with record on motion detection, IOPS and storage come back down to something reasonable.

[MkII]

Quick Overview

 

8 SATA interfaces and 1 eSATA interface connectable for recording and backup.

 

Max. HDD support - 48TB (8 x 6TB)

Max: supported IP cameras - 64

Max: Megabits per second (Mbps) - 320

 

As a guide when selecting a suitable NVR, HikVision's IP cameras typically use this much bandwidth at 25fps:

 

720P - 2Mbps

1.3MP - 2Mbps

2MP - 4Mbps

3MP - 5Mbps

5MP - 9Mbps

 

Incoming Bandwidth 320 Mbps (200 Mbps when RAID is enabled)

Outgoing Bandwidth 256 Mbps (200 Mbps when RAID is enabled)

[AlanD]

This is all interesting - even though I don't want to deploy this stuff....

 

I hear what is said about lower resolutions and slower frame rates - but from the odd cameras we use - I find they are often used to cover large areas with a lot going on - and while I would be happy to sacrifice highest frame rates I would want to keep the resolution at HD.

 

So it looks like just a "couple of disks" is not going to work - or at least not work well for 60 cameras. You will need more disks - even if they are smaller capacity....and I would say they need to be 24x7 capable "video" drives (or better still enterprise disks) - probably 7K2....not cheaper desktop drives. Desktop drives typically don't work well in RAIDs - especially when sectors start to go bad and get remapped - which is a normal and expected process...because desktop drives keep trying to read bad sectors and these retries - which may work - but degrade the performance of the RAID which might lead to "stuttering" in Video playback - or worse recording.

 

I'd be nervous about the impact of 320Mb/s taken out of any existing 1Gb/s capacity links - so I would be considering setting up some LAGs between my switches to provide extra capacity....unless you already of 10G links of course.

[MkII]

The supplier recommends WD Purple which I have reservations about. The new cameras allegedly have much better compression. We're going for 3mp. With motion capture record and low frame rate at 720p.

[MkII]

 

Supplier says: 64 x 3MP cameras recording at 10fps each using the h.264 CODEC recording continuously will use 256Mbps bandwidth and you will need 36TB to store recordings for 14 days

[mavhc]

Why are you worried about WD Purple disks? Also it's 2017, shouldn't they be selling h265 cameras? Halve your storage costs

[MkII]

We are getting H.265 cameras. The WD Purple disks fault tolerance means you could potentially get write failures on raided disks. Red are better.

 

 

We're thinking of going for 8 x 4TB WD Purples for £1024

Edited July 19, 2017 by MkII

[mavhc]

Good, you said h264 before.

 

Seems like it's a waste of time using RAID for multi camera recording, just put x/y cameras on each drive.

[MkII]

Yes I was quoting the supplier - good catch - he's raving about the new codec usually - saying 70% improvement in compression? I'm guessing his calculator isn't up to date.

 

That's what I'm planning to do... ditch RAID.

 

Cheers