Unifi APs in Grafana using SNMP

This is kind of goofy with how Ubiquiti doesn’t do well at supporting SNMP.  For one thing, they don’t support it through the controller, only directly to each AP.  But, you have to enable it at the controller to have it flip the switch on the APs so they’ll respond.  They really want you to use the API, which is great if you’re a programmer.  I am not.  I’m a router jockey, so I like SNMP.  Anyway, after finding and downloading the MIBs I had a look through them and sorted out a couple of OIDs I was interested in.  Specifically, client count per radio and Eth0 bits in and bits out.  Here’s what I loaded into Telegraf.  You need a separate inputs section for each AP you want to monitor.  Nope, not really an “Enterprise” approach.

[[inputs.snmp]]
agents = [ “192.168.x.x:161” ]  ## The IP of a single AP.
timeout = “5s”
retries = 3
version = 1
community = “RO_Community”
max_repetitions = 10
name = “UnifiWiFiOffice”
[[inputs.snmp.field]]
name = “Bits.Out”
oid = “1.3.6.1.4.1.41112.1.6.2.1.1.12.0”
[[inputs.snmp.field]]
name = “Bits.In”
oid = “1.3.6.1.4.1.41112.1.6.2.1.1.6.0”
[[inputs.snmp.field]]
name = “2.4.Clients”
oid = “1.3.6.1.4.1.41112.1.6.1.2.1.8.0”
[[inputs.snmp.field]]
name = “5.0.Clients”
oid = “1.3.6.1.4.1.41112.1.6.1.2.1.8.3”

Unraid shell script for getting stats into Grafana

Continuing the documentation effort.  This is a shell script you run from Unraid in a cron job to feed stats to InfluxDB.  You can then present them in Grafana.  Note about that, I was having a lot of trouble getting the Grafana graphs to present correctly for anything coming from this script.  I had to change the Fill from “null” to “none” in the graph.  Not sure why that’s happening, but “none” gets it to behave just like everything else.

## Assembled from this post: https://lime-technology.com/forum/index.php?topic=52220.msg512346#msg512346

## add to cron like:

## * * * * * sleep 10; /boot/custom/influxdb.sh > /dev/null 2>&1

## //0,10 * * * * /boot/custom/influxdb.sh > /dev/null 2>&1
#

# Set Vars

#

DBURL=http://192.168.x.x:8086 ## IP address of your InfluxDB server

DBNAME=dashboard ## Easier if you pick an existing DB

DEVICE=”UNRAID”

CURDATE=`date +%s`

# Current array assignment.

# I could pull the automatically from /var/local/emhttp/disks.ini

# Parsing it wouldnt be that easy though.

DISK_ARRAY=( sdn sdl sdf sdc sdj sde sdo sdh sdi sdd sdk sdm sdg sdp sdb )

DESCRIPTION=( parity disk1 disk2 disk3 disk4 disk5 disk6 disk7 disk8 disk9 disk10 disk11 disk12 disk13 cache )

#

# Added -n standby to the check so smartctl is not spinning up my drives

#

i=0

for DISK in “${DISK_ARRAY[@]}”

do

smartctl -n standby -A /dev/$DISK | grep “Temperature_Celsius” | awk ‘{print $10}’ | while read TEMP

do

curl -is -XPOST “$DBURL/write?db=$DBNAME” –data-binary “DiskTempStats,DEVICE=${DEVICE},DISK=${DESCRIPTION[$i]} Temperature=${TEMP} ${CURDATE}000000000” >/dev/null 2>&1

done

((i++))

done
# Had to increase to 10 samples because I was getting a spike each time I read it. This seems to smooth it out more

top -b -n 10 -d.2 | grep “Cpu” | tail -n 1 | awk ‘{print $2,$4,$6,$8,$10,$12,$14,$16}’ | while read CPUusr CPUsys CPUnic CPUidle CPUio CPUirq CPUsirq CPUst

do

top -bn1 | head -3 | awk ‘/load average/ {print $12,$13,$14}’ | sed ‘s/,//g’ | while read LAVG1 LAVG5 LAVG15

do

curl -is -XPOST “$DBURL/write?db=$DBNAME” –data-binary “cpuStats,Device=${DEVICE} CPUusr=${CPUusr},CPUsys=${CPUsys},CPUnic=${CPUnic},CPUidle=${CPUidle},CPUio=${CPUio},CPUirq=${CPUirq},

CPUsirq=${CPUsirq},CPUst=${CPUst},CPULoadAvg1m=${LAVG1},CPULoadAvg5m=${LAVG5},CPULoadAvg15m=${LAVG15} ${CURDATE}000000000” >/dev/null 2>&1

done

done
if [[ -f byteCount.tmp ]] ; then
# Read the last values from the tmpfile – Line “eth0”

grep “eth0” byteCount.tmp | while read dev lastBytesIn lastBytesOut

do

cat /proc/net/dev | grep “eth0” | grep -v “veth” | awk ‘{print $2, $10}’ | while read currentBytesIn currentBytesOut

do

# Write out the current stats to the temp file for the next read

echo “eth0” ${currentBytesIn} ${currentBytesOut} > byteCount.tmp
totalBytesIn=`expr ${currentBytesIn} – ${lastBytesIn}`

totalBytesOut=`expr ${currentBytesOut} – ${lastBytesOut}`
curl -is -XPOST “$DBURL/write?db=$DBNAME” –data-binary “interfaceStats,Interface=eth0,Device=${DEVICE} bytesIn=${totalBytesIn},bytesOut=${totalBytesOut} ${CURDATE}000000000” >/

dev/null 2>&1
done

done
else

# Write out blank file

echo “eth0 0 0” > byteCount.tmp

fi
# Gets the stats for boot, disk#, cache, user

#

df | grep “mnt/\|/boot\|docker” | grep -v “user0\|containers” | sed ‘s/\/mnt\///g’ | sed ‘s/%//g’ | sed ‘s/\/var\/lib\///g’| sed ‘s/\///g’ | while read MOUNT TOTAL USED FREE UTILIZATION DISK

do

if [ “${DISK}” = “user” ]; then

DISK=”array_total”

fi

curl -is -XPOST “$DBURL/write?db=$DBNAME” –data-binary “drive_spaceStats,Device=${DEVICE},Drive=${DISK} Free=${FREE},Used=${USED},Utilization=${UTILIZATION} ${CURDATE}000000000” >/dev/null 2>&

1

done

Telegraf mixed SNMP config

Following my previous post about Grafana, once everything is installed you’ll want to capture some data.  Otherwise, what’s the point.  Telegraf is a data gathering tool made by Influxdata.  It’s stupid simple to get working with InfluxDB.  After following the previous script, go to /etc/telegraf/ and edit telegraf.conf.  Near the top is the Output Plugins section.  Make sure that’s modified for your InfluxDB install.  From there, scroll down to Input Plugins.  There’s a ridiculous number of input plugins available.  We’re focused on SNMP today, but it’s worth looking through the list to see if a “need” can be solved with Telegraf before using some other custom script.

For me, I needed to add SNMP for my Ubiquiti ER-X firewall and my Nutanix CE cluster.  Here’s my SNMP config section with the obvious security bits redacted:

# # Retrieves SNMP values from remote agents
# [[inputs.snmp]]
[[inputs.snmp]]
agents = [ “192.168.x.x:161” ] ##Nutanix CE CVM IP
timeout = “5s”
version = 3

max_repetitions = 50

sec_name = “username”
auth_protocol = “SHA” # Values: “MD5”, “SHA”, “”
auth_password = “password”
sec_level = “authPriv” # Values: “noAuthNoPriv”, “authNoPriv”, “authPriv”

priv_protocol = “AES” # Values: “DES”, “AES”, “”
priv_password = “password”

name = “nutanix”
[[inputs.snmp.field]]
name = “host1CPU”
oid = “1.3.6.1.4.1.41263.9.1.6.1”
[[inputs.snmp.field]]
name = “host2CPU”
oid = “1.3.6.1.4.1.41263.9.1.6.2”
[[inputs.snmp.field]]
name = “host3CPU”
oid = “1.3.6.1.4.1.41263.9.1.6.3”
[[inputs.snmp.field]]
name = “host4CPU”
oid = “1.3.6.1.4.1.41263.9.1.6.4”
[[inputs.snmp.field]]
name = “ClusterIOPS”
oid = “1.3.6.1.4.1.41263.506.0”
[[inputs.snmp.field]]
name = “Host1MEM”
oid = “1.3.6.1.4.1.41263.9.1.8.1”
[[inputs.snmp.field]]
name = “Host2MEM”
oid = “1.3.6.1.4.1.41263.9.1.8.2”
[[inputs.snmp.field]]
name = “Host3MEM”
oid = “1.3.6.1.4.1.41263.9.1.8.3”
[[inputs.snmp.field]]
name = “Host4MEM”
oid = “1.3.6.1.4.1.41263.9.1.8.4”

[[inputs.snmp]]
agents = [ “192.168.0.1:161” ] ##Firewall IP
timeout = “5s”
retries = 3
version = 2
community = “RO_community_string”
max_repetitions = 10

name = “ERX”
[[inputs.snmp.field]]

name = “Bytes.Out”
oid = “1.3.6.1.2.1.2.2.1.10.2”
[[inputs.snmp.field]]
name = “Bytes.In”
oid = “1.3.6.1.2.1.2.2.1.16.2”

You’ll have to get Telegraf to read in the config again.  The sledgehammer method would be a reboot.  I think a Telegraf service restart would also do the trick.  Reboots for me take about 5 seconds (yep, really), so it’s useful to make sure it’s coming up clean on a reboot anyway.

Grafana on Ubuntu 16.04…easy, I think

Just went through setting up Grafana on Ubuntu 16.04 and thought I would grab the steps I went through.  I’m using a combination of Telegraf and some custom remote scripts to get data into InfluxDB.

curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add –
source /etc/lsb-release
echo “deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable” | sudo tee /etc/apt/sources.list.d/influxdb.list
sudo apt-get update && sudo apt-get install influxdb
sudo service influxdb start
echo “deb https://packagecloud.io/grafana/testing/debian/ wheezy main” | sudo tee /etc/apt/sources.list.d/grafana.list
curl https://packagecloud.io/gpg.key | sudo apt-key add –
sudo apt-get update && sudo apt-get install grafana
sudo service grafana-server start
wget https://dl.influxdata.com/telegraf/releases/telegraf_1.2.1_amd64.deb
sudo dpkg -i telegraf_1.2.1_amd64.deb
telegraf -sample-config > telegraf.conf
nano telegraf.conf
telegraf -config telegraf.conf
sudo cp telegraf.conf /etc/telegraf/telegraf.conf
sudo systemctl enable grafana-server.service
sudo systemctl enable telegraf.service
sudo reboot

This gets things installed.  I’ll have another post to describe other configuration that’s required.