Bailing on Nutanix CE

I’ve had a number of problems with CE lately that look like they’ll be difficult to fix.  Not sure I’ll have a better experience with a different platform, but I figure it’s about time to try.  If nothing else, I’ll have some backup copies of my VMs, which is not an obvious thing in CE.

So, here’s what I’ve had to go through to export VMs:

Log into a CVM and then “acli” to find the vmdisk_uuid.  Do a “vm.get [vm name] include_vmdisk_paths=1” to see a list of parameters.  Copy the vmdisk_uuid from about the middle of the output.  Exit the acli.

Run this:

1
qemu-img convert -c nfs://127.0.0.1/PrismCentral-CTR/.acropolis/vmdisk/[vmdisk_uuid from the previous step] -O qcow2 ./data/stargate-storage/disks/drive-scsi0-0-0-1/NameOfVM.qcow2

This command might be different depending on your setup.  I’m running PrismCentral so I had to use this location to find the disk.  The path is listed above the uuid in the acli command.  The output target will need to be adjusted depending on your space requirements.  If you leave out the target destination I believe it will save it in your current directory.  That might be ok or it might be too small for the VM.  I checked mine and decided to save it to the slow tier disk.  Depending on the disk size it might take a very long time.

Once it completes you can use SCP (not SFTP) to copy the file off.  I used WinSCP to connect to the same CVM.  The path for the above command is /home/nutanix/data/stargate-storage/disks/drive-scsi0-0-0-1.  The disk copy is in there for me and I can SCP it to somewhere else.  I tried sending it directly to an NFS share I have running on the network but it failed permissions, despite being whitelisted.  This process is cumbersome, but it works.  I’m sure there are better ways…

Grafana and Chromecasting to a TV

I’ve wanted to use simple Chromecast dongles for pumping a Grafana dashboard to a TV for a while now.  The challenge has been how to effectively manage the casting source.  Chromecasts can’t manage any of their own content, they can only be a casting target.  I don’t want a mobile device sitting in the rack with it’s sole purpose being the casting function.  Management of that would be difficult.  I also want to be able to cast to multiple Chromecasts with the same content or different content.

Google makes this difficult by limiting the signing certificate in the casting protocol.  However, some people have worked around it.  I’ve tried two different casting servers and I’m having success with:

https://mrothenbuecher.github.io/Chromecast-Kiosk/

I set up a dedicated VM with pretty light resources, installed Tomcat and then added the Kiosk server.  It works really well with one caveat.

The Chromecast dongles will arbitrarily decide if the TV is 720P or 1080P.  For most video content this doesn’t have a dramatic impact, but when you’re trying to display a dense Grafana dashboard it can make all the difference.  Unfortunately, this isn’t controllable in any way.  You have to test it against the TV and hope it works.

I now have a 32″ TV in the kitchen which is 1080P (also hard to find at 32″) and displaying a pretty dense Grafana dashboard.  I’ll try to add a picture here later.  I think this could be incredibly useful for business monitoring scenarios and is a lot less expensive than putting a PC on a TV.

Ubiquiti USG site to site VPN with a single controller

Quick note about how to make this work. If you want to have two Unifi Security Gateways connect to a single controller at one location, you need to open up a couple of ports. Specifically, 8080 and 8443 need to be open to the controller. I strongly suggest you make sure you have a fixed IP at the remote side and you lock down the ACL (port forward) to only allow traffic to 8080 and 8443 from that remote public IP. Once you have that in place, you can have the remote USG be adopted by the controller’s public IP. Be sure to add it to a different site.

After adoption is successful in the controller, turning on the site to site VPN is trivial. In Networks you create a new network. Select Site-to-site VPN from the “home” site network configuration. You should see the new remote site listed at the bottom. Simple as that.

Poor trunk labeling

Just a reference for anyone trying to create a trunk between Dell and Ubiquiti switches. In my case, I wanted to create a trunk between a Dell N2048P and a Ubiquiti Edge Switch, with a native VLAN in use across the trunk. They both seem to have the notion of Access, General, and Trunk modes. They also both suggest that General is essentially the same as Trunk mode, where tags will be retained across the link, and any untagged frames will dump into the PVID, or Native VLAN.

Unfortunately, this doesn’t appear to be the case. The only way I could get it to work was to set both to Trunk, despite some suggestions out there about untagged traffic being ignored. It seems that as long as you don’t set the trunk to ignore untagged traffic manually, it will act like a regular old trunk port. What’s worse is that in the General mode it will sort of work.

Raspberry Pi Zero W headless setup

There seems to be conflicting info out there for how to accomplish this.  Compounded with the Zero’s different micro ports, it’s easier if you can set it up as a headless device.  Unfortunately, I found that if you try to do this locally, with a monitor and keyboard, the order of operations causes the ssh keys to be faulty.  So, let’s make it easy and just do it all from the start.  Download Raspbian Jessie Lite.  I believe the version I got is 03.02.  In Windows I’m using Rufus to write the disk image.  Select the disk image from the folder icon in the lower right.  You need to search for all file types, as it’s not an ISO.  Once you select it, Rufus will automatically determine that it needs to be a DD write for the file.  Fire it off on your micro SD card and let it finish.  It will take a few minutes.

When it’s done you’ll have a single partition viewable in Windows for the SD card.  Right click and create a Notepad file called ssh.txt in the root of that partition.  Just create it.  Don’t edit it.  Create another Notepad file and call it wpa_supplicant.conf.  Open that in Notepad and add the following:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
    ssid="Your SSID Here"
    proto=RSN
    key_mgmt=WPA-PSK
    pairwise=CCMP TKIP
    group=CCMP TKIP
    psk="YourPresharedKeyHere"
}

Modify the SSID and PSK to match your WiFi settings and save it.  Pop the SD card out, pop it into your Zero W and boot it up.  Wait a few minutes, and then you’ll need to find the dhcp address the Zero W received.  For me, I checked the dhcp scope on my firewall and found a new dhcp lease for a device named “raspberrypi”.  Open up Putty and ssh to that IP.  You should be connected at that point.  Probably a good idea to run raspi-config and update the password and host name.

Unifi APs in Grafana using SNMP

This is kind of goofy with how Ubiquiti doesn’t do well at supporting SNMP.  For one thing, they don’t support it through the controller, only directly to each AP.  But, you have to enable it at the controller to have it flip the switch on the APs so they’ll respond.  They really want you to use the API, which is great if you’re a programmer.  I am not.  I’m a router jockey, so I like SNMP.  Anyway, after finding and downloading the MIBs I had a look through them and sorted out a couple of OIDs I was interested in.  Specifically, client count per radio and Eth0 bits in and bits out.  Here’s what I loaded into Telegraf.  You need a separate inputs section for each AP you want to monitor.  Nope, not really an “Enterprise” approach.

[[inputs.snmp]]
agents = [ “192.168.x.x:161” ]  ## The IP of a single AP.
timeout = “5s”
retries = 3
version = 1
community = “RO_Community”
max_repetitions = 10
name = “UnifiWiFiOffice”
[[inputs.snmp.field]]
name = “Bits.Out”
oid = “1.3.6.1.4.1.41112.1.6.2.1.1.12.0”
[[inputs.snmp.field]]
name = “Bits.In”
oid = “1.3.6.1.4.1.41112.1.6.2.1.1.6.0”
[[inputs.snmp.field]]
name = “2.4.Clients”
oid = “1.3.6.1.4.1.41112.1.6.1.2.1.8.0”
[[inputs.snmp.field]]
name = “5.0.Clients”
oid = “1.3.6.1.4.1.41112.1.6.1.2.1.8.3”

Unraid shell script for getting stats into Grafana

Continuing the documentation effort.  This is a shell script you run from Unraid in a cron job to feed stats to InfluxDB.  You can then present them in Grafana.  Note about that, I was having a lot of trouble getting the Grafana graphs to present correctly for anything coming from this script.  I had to change the Fill from “null” to “none” in the graph.  Not sure why that’s happening, but “none” gets it to behave just like everything else.

## Assembled from this post: https://lime-technology.com/forum/index.php?topic=52220.msg512346#msg512346

## add to cron like:

## * * * * * sleep 10; /boot/custom/influxdb.sh > /dev/null 2>&1

## //0,10 * * * * /boot/custom/influxdb.sh > /dev/null 2>&1
#

# Set Vars

#

DBURL=http://192.168.x.x:8086 ## IP address of your InfluxDB server

DBNAME=dashboard ## Easier if you pick an existing DB

DEVICE=”UNRAID”

CURDATE=`date +%s`

# Current array assignment.

# I could pull the automatically from /var/local/emhttp/disks.ini

# Parsing it wouldnt be that easy though.

DISK_ARRAY=( sdn sdl sdf sdc sdj sde sdo sdh sdi sdd sdk sdm sdg sdp sdb )

DESCRIPTION=( parity disk1 disk2 disk3 disk4 disk5 disk6 disk7 disk8 disk9 disk10 disk11 disk12 disk13 cache )

#

# Added -n standby to the check so smartctl is not spinning up my drives

#

i=0

for DISK in “${DISK_ARRAY[@]}”

do

smartctl -n standby -A /dev/$DISK | grep “Temperature_Celsius” | awk ‘{print $10}’ | while read TEMP

do

curl -is -XPOST “$DBURL/write?db=$DBNAME” –data-binary “DiskTempStats,DEVICE=${DEVICE},DISK=${DESCRIPTION[$i]} Temperature=${TEMP} ${CURDATE}000000000” >/dev/null 2>&1

done

((i++))

done
# Had to increase to 10 samples because I was getting a spike each time I read it. This seems to smooth it out more

top -b -n 10 -d.2 | grep “Cpu” | tail -n 1 | awk ‘{print $2,$4,$6,$8,$10,$12,$14,$16}’ | while read CPUusr CPUsys CPUnic CPUidle CPUio CPUirq CPUsirq CPUst

do

top -bn1 | head -3 | awk ‘/load average/ {print $12,$13,$14}’ | sed ‘s/,//g’ | while read LAVG1 LAVG5 LAVG15

do

curl -is -XPOST “$DBURL/write?db=$DBNAME” –data-binary “cpuStats,Device=${DEVICE} CPUusr=${CPUusr},CPUsys=${CPUsys},CPUnic=${CPUnic},CPUidle=${CPUidle},CPUio=${CPUio},CPUirq=${CPUirq},

CPUsirq=${CPUsirq},CPUst=${CPUst},CPULoadAvg1m=${LAVG1},CPULoadAvg5m=${LAVG5},CPULoadAvg15m=${LAVG15} ${CURDATE}000000000” >/dev/null 2>&1

done

done
if [[ -f byteCount.tmp ]] ; then
# Read the last values from the tmpfile – Line “eth0”

grep “eth0” byteCount.tmp | while read dev lastBytesIn lastBytesOut

do

cat /proc/net/dev | grep “eth0” | grep -v “veth” | awk ‘{print $2, $10}’ | while read currentBytesIn currentBytesOut

do

# Write out the current stats to the temp file for the next read

echo “eth0” ${currentBytesIn} ${currentBytesOut} > byteCount.tmp
totalBytesIn=`expr ${currentBytesIn} – ${lastBytesIn}`

totalBytesOut=`expr ${currentBytesOut} – ${lastBytesOut}`
curl -is -XPOST “$DBURL/write?db=$DBNAME” –data-binary “interfaceStats,Interface=eth0,Device=${DEVICE} bytesIn=${totalBytesIn},bytesOut=${totalBytesOut} ${CURDATE}000000000” >/

dev/null 2>&1
done

done
else

# Write out blank file

echo “eth0 0 0” > byteCount.tmp

fi
# Gets the stats for boot, disk#, cache, user

#

df | grep “mnt/\|/boot\|docker” | grep -v “user0\|containers” | sed ‘s/\/mnt\///g’ | sed ‘s/%//g’ | sed ‘s/\/var\/lib\///g’| sed ‘s/\///g’ | while read MOUNT TOTAL USED FREE UTILIZATION DISK

do

if [ “${DISK}” = “user” ]; then

DISK=”array_total”

fi

curl -is -XPOST “$DBURL/write?db=$DBNAME” –data-binary “drive_spaceStats,Device=${DEVICE},Drive=${DISK} Free=${FREE},Used=${USED},Utilization=${UTILIZATION} ${CURDATE}000000000” >/dev/null 2>&

1

done

Telegraf mixed SNMP config

Following my previous post about Grafana, once everything is installed you’ll want to capture some data.  Otherwise, what’s the point.  Telegraf is a data gathering tool made by Influxdata.  It’s stupid simple to get working with InfluxDB.  After following the previous script, go to /etc/telegraf/ and edit telegraf.conf.  Near the top is the Output Plugins section.  Make sure that’s modified for your InfluxDB install.  From there, scroll down to Input Plugins.  There’s a ridiculous number of input plugins available.  We’re focused on SNMP today, but it’s worth looking through the list to see if a “need” can be solved with Telegraf before using some other custom script.

For me, I needed to add SNMP for my Ubiquiti ER-X firewall and my Nutanix CE cluster.  Here’s my SNMP config section with the obvious security bits redacted:

# # Retrieves SNMP values from remote agents
# [[inputs.snmp]]
[[inputs.snmp]]
agents = [ “192.168.x.x:161” ] ##Nutanix CE CVM IP
timeout = “5s”
version = 3

max_repetitions = 50

sec_name = “username”
auth_protocol = “SHA” # Values: “MD5”, “SHA”, “”
auth_password = “password”
sec_level = “authPriv” # Values: “noAuthNoPriv”, “authNoPriv”, “authPriv”

priv_protocol = “AES” # Values: “DES”, “AES”, “”
priv_password = “password”

name = “nutanix”
[[inputs.snmp.field]]
name = “host1CPU”
oid = “1.3.6.1.4.1.41263.9.1.6.1”
[[inputs.snmp.field]]
name = “host2CPU”
oid = “1.3.6.1.4.1.41263.9.1.6.2”
[[inputs.snmp.field]]
name = “host3CPU”
oid = “1.3.6.1.4.1.41263.9.1.6.3”
[[inputs.snmp.field]]
name = “host4CPU”
oid = “1.3.6.1.4.1.41263.9.1.6.4”
[[inputs.snmp.field]]
name = “ClusterIOPS”
oid = “1.3.6.1.4.1.41263.506.0”
[[inputs.snmp.field]]
name = “Host1MEM”
oid = “1.3.6.1.4.1.41263.9.1.8.1”
[[inputs.snmp.field]]
name = “Host2MEM”
oid = “1.3.6.1.4.1.41263.9.1.8.2”
[[inputs.snmp.field]]
name = “Host3MEM”
oid = “1.3.6.1.4.1.41263.9.1.8.3”
[[inputs.snmp.field]]
name = “Host4MEM”
oid = “1.3.6.1.4.1.41263.9.1.8.4”

[[inputs.snmp]]
agents = [ “192.168.0.1:161” ] ##Firewall IP
timeout = “5s”
retries = 3
version = 2
community = “RO_community_string”
max_repetitions = 10

name = “ERX”
[[inputs.snmp.field]]

name = “Bytes.Out”
oid = “1.3.6.1.2.1.2.2.1.10.2”
[[inputs.snmp.field]]
name = “Bytes.In”
oid = “1.3.6.1.2.1.2.2.1.16.2”

You’ll have to get Telegraf to read in the config again.  The sledgehammer method would be a reboot.  I think a Telegraf service restart would also do the trick.  Reboots for me take about 5 seconds (yep, really), so it’s useful to make sure it’s coming up clean on a reboot anyway.

Grafana on Ubuntu 16.04…easy, I think

Just went through setting up Grafana on Ubuntu 16.04 and thought I would grab the steps I went through.  I’m using a combination of Telegraf and some custom remote scripts to get data into InfluxDB.

curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add –
source /etc/lsb-release
echo “deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable” | sudo tee /etc/apt/sources.list.d/influxdb.list
sudo apt-get update && sudo apt-get install influxdb
sudo service influxdb start
echo “deb https://packagecloud.io/grafana/testing/debian/ wheezy main” | sudo tee /etc/apt/sources.list.d/grafana.list
curl https://packagecloud.io/gpg.key | sudo apt-key add –
sudo apt-get update && sudo apt-get install grafana
sudo service grafana-server start
wget https://dl.influxdata.com/telegraf/releases/telegraf_1.2.1_amd64.deb
sudo dpkg -i telegraf_1.2.1_amd64.deb
telegraf -sample-config > telegraf.conf
nano telegraf.conf
telegraf -config telegraf.conf
sudo cp telegraf.conf /etc/telegraf/telegraf.conf
sudo systemctl enable grafana-server.service
sudo systemctl enable telegraf.service
sudo reboot

This gets things installed.  I’ll have another post to describe other configuration that’s required.