Any AXIS Camera Station experts - load balancing

[iSteve]

Hi all,

 

We have a large installation of Axis cameras at our site. We're looking to try and load balance these across two servers to help with some recent performance issues. It would be great to hear from anyone who has implemented similar. From what I've read, it seems we install two copies of Camera Station, and split the cameras across each server, we then configure the client to see both servers in it's list. Is that correct?

[Steve21]

Basically yep!

 

Depending on the issues though if it’s things like write speed you could stilll use a single server but make the storage point for a group of cameras a different server and then wouldn’t need to do the dual client setup etc

 

Steve

[iSteve]

The issue was after upgrading from 5.0xx to 5.3, PTZ reaction speed is sluggish through camera station, and playback buffers the whole time. Out of hours, playback is fine. We have 127 cameras and many viewers. It's all on a separate VLAN but the update has caused a massive performance hit.

[gszech]

The issue was after upgrading from 5.0xx to 5.3, PTZ reaction speed is sluggish through camera station, and playback buffers the whole time. Out of hours, playback is fine. We have 127 cameras and many viewers. It's all on a separate VLAN but the update has caused a massive performance hit.

 

We had around 120 cameras and now reduced them to around 100 hoping that this will improve the performance. Playback buffers for us and also some of the cameras are skipping frames. The system was working fine for years, so it's probably one of the updates caused the problem. When we emailed Axis they said that the server is designed to support around 60 CCTV units - which is fair, but everything was working fine previously. They also said that motion detection creates a lot of pressure on the server and they recommend to record continuously, but this didn't change anything, server load is at max 15% with or without motion.

 

We tried changing video stream quality, toggle record to continuous / motion, factory reset all CCTV cameras, maintain database and update the app - no luck.

I would love to know the solution for this, but I don't want to spend weeks with Axis support, so we ended up slowly replacing old Axis Cameras with UniFi ones hoping that the system will work fine with less load.

[iSteve]

This sounds similar to our experience. We're hoping that splitting the load between two machines will help but am not clear on the process

[gszech]

This sounds similar to our experience. We're hoping that splitting the load between two machines will help but am not clear on the process

 

Just a quick suggestion, instead of configuring load balancing you can try to disconnect half of the cameras on the switch for one hour(disable power to them) to see if this sorts out the performance. If it does - I would ask Axis support on some guidance how to configure load balancing.

[RooiValkUSA]

I realize this thread is old but I wanted to see if anyone found a solution to this or similar problems.

 

The problem we are having is not with live viewing. You can bring up 30 cameras simultaneously onscreen without issue. Nothing involving live viewing seems to be a a problem.

 

However when we do something that requires the software to "get data", it chokes and splutters for sometimes minutes. Here are a few examples of what sets this off: Right-clicking on a camera and selecting edit can take minutes (no joke). Logging out of a server can take minutes, depending on the server. Selecting any kind of configuration change takes minutes. Selecting a camera map (even something as small as a few Mb) takes forever. Clicking the show recordings icon. In fact it is just about any action in the software except selecting a camera or multiple cameras, for live view.  If you generate a server report from the client, it takes forever to push it to your desktop. generating the report is normal, but transferring to desktop is excrutiatingly slow, despite networking telling is that they see no issues.

 

Here's some more strangeness:

We have 96 cameras on each server. 1 of the servers has zero issues while the other 3 are problem children, despite all having 96 cameras on them and being setup identically.

This is far worse during work hours. It mostly feels "normal" after hours, but chokes during work hours, and there are NOT that many people using the software.

The problem does not exist on the server if you log in there and run the client.

The server itself is not under any stress whatsoever. CPU and memory see minimal usage ... etc.

 

We're getting almost no answers from networking or Axis on this one and right now the servers are basically non-usable at client level, for anything except live viewing.

[iSteve]

Our solution with guidance from Axis, was that motion detection was killing our performance. We had 120 cameras all recording via motion trigger. Once we set the cameras to record continuously during school hours, and switch to motion out of hours, the problems went away. We are now up to 143 cameras on one server, and everything is fine.

[RobFuller]

We encountered a similar issue where the client application would crash under certain conditions. After some investigation, we discovered that disabling hardware decoding resolved the problem.

Interestingly, the issue never occurred on the server side, likely because we were connected via RDP.