Let's talk monitoring!

[Blue_Cookeh]

Hi Guys,

 

I'm looking at improving our monitoring situation at the moment and was wondering what you guys use in your environments?

 

So far I've given Nagios and LibreNMS a go, Nagios was nice but it was far too time consuming to configure and right now LibreNMS seems pretty slick albeit basic. Any other suggestions?

 

I'm looking for something that can do Windows/Linux/HP switches/Ubiquiti (hopefully?) gear and possibly tie back into our UPS' and iDRAC Enterprise cards.

[Vegas]

We use Mutiny

Network monitoring solutions|performance tools | Mutiny

[CyberNerd]

It's worth spending time on Nagios. It is very powerful because it is so well supported by open plugins and easy to write scripts for. once configured will do it's job without intervention.

[IanT]

PRTG and SCOM

[unixman_again]

We use Servers alive but I don't know how good it is. I know it texts the bosses phone when something goes down or comes back up and it sends me an email. I don't know what the annual fee is.

 

+1 Nagios. If I was in charge, that's what I would use having used it successfully at three sites previously. I written plugins for devices too.

[Geoff]

I am using open source solutions, designed for the cloud so they are somewhat overkill however they scale massively. I have 3 Linux VMs:

 

VM 1 (Centralised Logging) - Nxlog (Windows Clients), ElasticSearch, Logstash & Kibana. End result looks like this (here is a demo you can play with):

 

http://www.neteye-blog.com/wp-content/uploads/2014/10/kibana-citrix.png

 

VM 2 (Time Series Metrics) - CollectD, Graphite Powershell Functions (Windows Clients), Graphite & Grafana. End result looks like this (here is a demo you can play with):

 

 

VM 3 (Alerting) - RabbitMQ, Redis, Sensu (Sensu Client runs on Windows, I wrote my own checks in Powershell) & Uchiwa. End result looks like this:

 

Edited July 28, 2015 by Geoff
links

[Fazza]

Mutiny all the way! As an educational user we get around a 90% discount! We have a wallboard shown on a wall mounted monitor in our room so we can see at a glance what is going on and we get email alerts from it telling us when RAM and CPU are running high (you can set what percentages it notifies you at) along with telling us if a server has just rebooted or has disconnected from the network.

 

It's free to use for up to 10 devices so you can give it a go and bung on a server or two along with any switches you have and see how you get on.

[pete]

Icinga (Nagios fork) for up/down notifications. Observium for graphing and network mapping. OSSEC for centralised logging and file monitoring.

 

I keep meaning to sit down and feed OSSEC into graylog, but then I find something better to do than write regexes to kludge ossec output into something graylog understands.

[Dos_Box]

Moving to Network and Classroom Management forum.

[cpjitservices]

PRTG here

 

All is well.

[Tall_Paul]

PRTG here as well with the free 100 sensors. It's brilliant.

 

When we're virtualised shortly (don't ask) I'll run VEEAM One again too.

[rpettit]

Hi Geoff,

 

Is this all open source? Was it time consuming to build these 3 VM's?

 

What documentation did you use?

 

Many thanks

[ITGURU]

I use a mixture - PRTG for Traffic monitoring.

HP System Insight Manager for monitoring Servers and anything with a static IP - emails me on any server HDD failure or device becoming unreachable/reachable.

[Geoff]

Hi Geoff,

 

Is this all open source? Was it time consuming to build these 3 VM's?

 

What documentation did you use?

 

Many thanks

 

Yes this is all off the shelf open source software running on 3 Ubuntu VMs (A larger deployment more fault tolerant setup would require more, eg multiple rabbitmq and a HA pair for redis + sensu, but the system scales horizontally). I followed the Digital Ocean guides as a starting point then 'added on' extras after Googling.

 

https://www.digitalocean.com/community/tutorials/how-to-configure-sensu-monitoring-rabbitmq-and-redis-on-ubuntu-14-04

https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-ubuntu-14-04

https://www.digitalocean.com/community/tutorials/how-to-install-and-use-graphite-on-an-ubuntu-14-04-server (Does not include the Grafana setup)

https://www.digitalocean.com/community/tutorials/how-to-configure-collectd-to-gather-system-metrics-for-graphite-on-ubuntu-14-04

 

There are lots and lots and lots of other guides out there too. I suggest you just start reading.

[markwilfan]

It's worth spending time on Nagios. It is very powerful because it is so well supported by open plugins and easy to write scripts for. once configured will do it's job without intervention.

+1 for nagios. For less time consuming setup use check_mk to do the heavy lifting http://mathias-kettner.com/check_mk.html. We have nagvis installed too to give us a live weather map of data

[rpettit]

Are you using the raw edition?

[Elison]

using 10-strike network monitor (more info)

it's easy to set and use in comparasion with nagios

[synaesthesia]

Looks easy but that pricing structure is hilarious!

 

"Yes thanks, I'll take a Ford Focus please"

 

"No problem. It's £12,000 for the base, you'll need to pay another £500 annually for the doors and then £2 for each mile you plan to drive"....

[Fazza]

SCOM is fairly cheap for Education so that's always worth a shot.

[strawberry]

I use the Dude. Really easy to setup and use.

[localzuk]

@Geoff - With Elasticsearch/logstash/nxlog, what do you have set up in terms of:

 

Default shards per index?

How many servers are forwarding their logs?

How many logs per hour is it handling?

What spec are you running the server on?

 

Only, I've set this up, per the how to you mentioned, and I'm seeing a heck of a lot of shards (there seems to be an index per day for logstash), and the CPU is going crazy processing all the logs (as its getting all the historical data too, luckily this'll only go back to 2012 for our servers).

[Geoff]

One shard per node is optimal. There's plenty of reading on the subject here.

I only have about 20 servers forwarding logs at the moment. This is still in development so I'm not covering everything (around 80 devices/servers).

250k log lines/hour

A VM with 2 vCPU, 8Gb Ram, 300Gb VHD for my indexes. It's important to tune the Java VM to your memory sizes. If you are using ubuntu you need to edit /etc/default/elasticsearch.

 

Also consider adding more nodes (I'd probably go for 3 + 1 client node), this system is designed to scale horizontally. As I originally indicated more elasticsearch nodes should be used in a production setup as you have no fault tolerance with a single node. Additionally consider having the node running kibana in client mode as this will load balance the kibana requests between the nodes in your cluster. You will also have to reshard (by re-indexing), see above for that.

 

Finally with a multi-node setup you should use RabbitMQ or Redis between your logstash clients and your logstash servers. This will load balance the log data as it comes into the cluster.

 

RabbitMQ and Logstash - Dopey's Corner

How to Setup Logstash on Linux with ElasticSearch, Redis, Nginx

 

If this is too much for your internal infrastructure to handle. Put the Redis/RabbitMQ onsite and put the Elasticsearch cluster on Amazon S3 or similar. In the event of a link failure your Redis/RabbitMQ will buffer the logs until your elasticsearch cluster is available. Bonus points if you use elastic compute on Amazon, so you can size your cluster up and down based on load (eg, the firewall gets hammered whenever we or our customers do PCI scans, which causes a lot of logging to be produced. It would be awsome to spin up extra elasticsearch and logstash nodes to handle these spikes).

Edited September 17, 2015 by Geoff

[localzuk]

Hmm... Think that may be a bit more than our capacity can handle at the moment. Would never be given authorisation to start using cloud servers - ongoing costs are disliked.

 

I'll just have to scrap the project for now, and next time our servers are up for replacement look at it all again, and spec our servers to handle the load - or, if we've started using the cloud by then, do it that way.

 

Thanks for the info.

[Geoff]

If you cannot handle storing and processing *everything* just be selective. You should also think about what you would need realistically store. There's little point storing logs that will never see the light of day (other than for compliance or regulatory reasons).

 

Production nodes I'm probably looking at 8 vCPU and 64Gb Ram with 1 Tb index storage. The client node can have less ram and storage obviously. I'd template all this so I can spin up extra VMs as required.

Edited September 17, 2015 by Geoff

[SHimmer45]

Having been a Ranger for networks users would any of these tools give a live console for showing which users are logged into which computers in realtime with what Apps are running etc?

 

 

Cacti is pretty good as well for monitoring swithes and the like again requires some time investment.

PTRG looks a little more polished

Edited September 17, 2015 by SHimmer45