Plex, Channels and trying to get the two to work together

For a while now I’ve been running Plex as my primary media server. I’ve been trying to use the Plex DVR, but found it to be very finicky. Mostly, it would fail to record shows without telling me in any way. I’d review my upcoming recordings and then check them a couple of days after and they just weren’t there. No notice, nothing. I’ve also been using the Channels app on the AppleTV for live TV. Part of my frustration with PlexDVR is that the channel tuning was always slow. I think if it worked well otherwise I probably would have stuck with it, but that’s not the case. So, I decided to try Channels DVR. What a difference. All I can say is it just seems to work. Everything appears to be getting recorded, channel changes on live TV are fast, it’s really been an excellent experience. I had one issue that I was able to overcome, which is how to get the Channels recordings into Plex in an automated way. Channels DVR has a means of managing recordings and will tag commercials with chapter markers, but that’s just in the Channels DVR app.

I have Channels DVR running in a Docker in Unraid. Probably not necessary, but it’s handy. It was super simple to install. The problem is that Channels DVR wants to record everything to a “TV” folder within the directory you set it to for the recordings. In my case, I have a couple of Unraid user shares related to TV. I have the main TV shows storage at /mnt/user/TVShows and I have the recordings directory at /mnt/user/LiveTV. This means the Channels DVR recordings will go into /mnt/user/LiveTV/TV/showname.

The fix ended up being pretty simple. I run Plex on a Ubuntu server. Here’s my fstab:

1
2
3
4
5
6
7
//192.168.169.8/DVDs /mnt/DVDs cifs guest,uid=1000,iocharset=utf8 0 0
//192.168.169.8/KidDVDs /mnt/KidDVDs cifs guest,uid=1000,iocharset=utf8 0 0
//192.168.169.8/Movies /mnt/Movies cifs guest,uid=1000,iocharset=utf8 0 0
//192.168.169.8/TVShows /mnt/TVShows cifs guest,uid=1000,iocharset=utf8 0 0
//192.168.169.8/Pictures /mnt/Pictures cifs guest,uid=1000,iocharset=utf8 0 0
//192.168.169.8/Music /mnt/Music cifs guest,uid=1000,iocharset=utf8 0 0
//192.168.169.8/LiveTV/TV /mnt/DVR cifs guest,uid=1000,iocharset=utf8 0 0

Inside Plex I have the TV shows library mapped to both /mnt/TVShows and /mnt/DVR. Plex’s autodiscovery scans both folders just fine and coalesces the shows from both locations. I still need to figure out the comskip, but hitting the jump button is fine for now. In retrospect, I probably could have simply pointed it at TVShows and let it create a new directory in there, but this way keeps the folder structures a little cleaner.

Free Veeam linux backups

I didn’t feel like fighting against Nutanix CE any more so I copied my VMs out using qemu and rebuilt the entire cluster with ESXi 6.7 free. Seems to be ok so far with a few minor issues. Now that I’m back in operation I have a concern about having a single copy of my VMs on standalone hosts with RAID0 single disk configs. A bit of a roll of the dice but it works. So, I need a backup solution. ESXi doesn’t support API connections for backups so you have to use something else. I’m using Veeam Linux Agent free on each of my VMs now. I’m sending the backups to an NFS share on the Unraid server. Here’s my list of commands so I can do this again later and not wonder how I did it.

Make sure we’re updated before starting:

1
2
3
sudo apt-get update
sudo apt-get upgrade
sudo apt-get autoremove

SCP the Veeam download file to the server, which I think might just be a repository and user add script.

1
2
3
sudo dpkg -i veeam-release-deb_1.0.5_amd64.deb
sudo apt-get update
sudo apt-get install veeam

Next we’ll take care of the NFS share mounting.

1
2
3
sudo apt-get install nfs-common
sudo mkdir /mnt/backup
sudo nano /etc/fstab

Here’s my one fstab line for connecting to the Unraid NFS share:

1
192.168.169.8:/mnt/user/vmware /mnt/backup nfs auto 0  0

Mount the new share:

1
sudo mount -a

You should now be able to run the Veeam agent with “sudo veeam” which will launch the ncurses gui. Pretty obvious how to work through it from there, just select local storage and browse to the /mnt/backup location if that’s what you named it.

Nutanix CE, Dell H700 and disk detection

Not sure why I didn’t post this before, but here it is. When installing Nutanix CE on a Dell server (R710 in my case) with a PERC H700 card you can’t do a pass-through on the disks. You have to configure them as a single disk RAID0 in the PERC config.  Ctrl-R at boot for the H700.  That works, but then CE can’t detect the type of drive it is.  It will fail with a message about missing requirements.  You have to tell the install routine which drive is the “rotational” drive.

Exit the install process and login as root:nutanix/4u.

1
dmesg | grep sda

You might need to do this for sda, sdb, sdc until you find the HDD, not the SSD. The drive sizes should tell you which is the right one.

1
echo 0 > /sys/block/sd?/queue/rotational

Where sd? is replaced with the correct sd. Logout and log back in as install. Should be fine from there. This is being done with the img install, not the iso. I did get a similar error on the iso but didn’t try to work around it. I’ll see if it works the same way with that.

Ok, not bailing on Nutanix CE just yet

Since my last post I’ve been fighting to get the cluster back to some semblance of stability.  Doesn’t look like that’s going to happen.  However, there isn’t anything out there that approaches the ease of configuration, fault tolerance and hyperconverged storage I’m looking for.  So…I’m making sure I have a backup copy of my VMs (especially Grafana) that would be problematic to rebuild, and then I’ll blow away the entire cluster and start over from scratch.

First challenge is backups.  I found this link which describes what appears to be the simplest process:  Backup and restore VMs in Nutanix CE

I’ve had to modify this a bit for my particular circumstances, but it appears to be working well.  First, I’m lucky enough to have a Unraid NAS that holds all my media.  Plenty of space and it happens to support NFS.  With some trial and error and the instructions above I managed to work out the following command:

1
qemu-img convert -c nfs://127.0.0.1/default-container-46437110940265/.acropolis/vmdisk/be09f08c-56bf-472c-b8a6-16c022333ca5 -O qcow2 nfs://192.168.169.8/mnt/user/vmware/UnifiVideo.qcow2

Using this command and the previous instructions I was able to send the backup qcow2 image directly to Unraid for backup purposes.  It sure would be nice to have a checkbox option, but this will do for now.  Also, be sure to use the screen option in the other post because it’s likely your session will time out on modestly sized VMs.

Bailing on Nutanix CE

I’ve had a number of problems with CE lately that look like they’ll be difficult to fix.  Not sure I’ll have a better experience with a different platform, but I figure it’s about time to try.  If nothing else, I’ll have some backup copies of my VMs, which is not an obvious thing in CE.

So, here’s what I’ve had to go through to export VMs:

Log into a CVM and then “acli” to find the vmdisk_uuid.  Do a “vm.get [vm name] include_vmdisk_paths=1” to see a list of parameters.  Copy the vmdisk_uuid from about the middle of the output.  Exit the acli.

Run this:

1
qemu-img convert -c nfs://127.0.0.1/PrismCentral-CTR/.acropolis/vmdisk/[vmdisk_uuid from the previous step] -O qcow2 ./data/stargate-storage/disks/drive-scsi0-0-0-1/NameOfVM.qcow2

This command might be different depending on your setup.  I’m running PrismCentral so I had to use this location to find the disk.  The path is listed above the uuid in the acli command.  The output target will need to be adjusted depending on your space requirements.  If you leave out the target destination I believe it will save it in your current directory.  That might be ok or it might be too small for the VM.  I checked mine and decided to save it to the slow tier disk.  Depending on the disk size it might take a very long time.

Once it completes you can use SCP (not SFTP) to copy the file off.  I used WinSCP to connect to the same CVM.  The path for the above command is /home/nutanix/data/stargate-storage/disks/drive-scsi0-0-0-1.  The disk copy is in there for me and I can SCP it to somewhere else.  I tried sending it directly to an NFS share I have running on the network but it failed permissions, despite being whitelisted.  This process is cumbersome, but it works.  I’m sure there are better ways…

Raspberry Pi Zero W headless setup

There seems to be conflicting info out there for how to accomplish this.  Compounded with the Zero’s different micro ports, it’s easier if you can set it up as a headless device.  Unfortunately, I found that if you try to do this locally, with a monitor and keyboard, the order of operations causes the ssh keys to be faulty.  So, let’s make it easy and just do it all from the start.  Download Raspbian Jessie Lite.  I believe the version I got is 03.02.  In Windows I’m using Rufus to write the disk image.  Select the disk image from the folder icon in the lower right.  You need to search for all file types, as it’s not an ISO.  Once you select it, Rufus will automatically determine that it needs to be a DD write for the file.  Fire it off on your micro SD card and let it finish.  It will take a few minutes.

When it’s done you’ll have a single partition viewable in Windows for the SD card.  Right click and create a Notepad file called ssh.txt in the root of that partition.  Just create it.  Don’t edit it.  Create another Notepad file and call it wpa_supplicant.conf.  Open that in Notepad and add the following:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
    ssid="Your SSID Here"
    proto=RSN
    key_mgmt=WPA-PSK
    pairwise=CCMP TKIP
    group=CCMP TKIP
    psk="YourPresharedKeyHere"
}

Modify the SSID and PSK to match your WiFi settings and save it.  Pop the SD card out, pop it into your Zero W and boot it up.  Wait a few minutes, and then you’ll need to find the dhcp address the Zero W received.  For me, I checked the dhcp scope on my firewall and found a new dhcp lease for a device named “raspberrypi”.  Open up Putty and ssh to that IP.  You should be connected at that point.  Probably a good idea to run raspi-config and update the password and host name.

Unifi APs in Grafana using SNMP

This is kind of goofy with how Ubiquiti doesn’t do well at supporting SNMP.  For one thing, they don’t support it through the controller, only directly to each AP.  But, you have to enable it at the controller to have it flip the switch on the APs so they’ll respond.  They really want you to use the API, which is great if you’re a programmer.  I am not.  I’m a router jockey, so I like SNMP.  Anyway, after finding and downloading the MIBs I had a look through them and sorted out a couple of OIDs I was interested in.  Specifically, client count per radio and Eth0 bits in and bits out.  Here’s what I loaded into Telegraf.  You need a separate inputs section for each AP you want to monitor.  Nope, not really an “Enterprise” approach.

[[inputs.snmp]]
agents = [ “192.168.x.x:161” ]  ## The IP of a single AP.
timeout = “5s”
retries = 3
version = 1
community = “RO_Community”
max_repetitions = 10
name = “UnifiWiFiOffice”
[[inputs.snmp.field]]
name = “Bits.Out”
oid = “1.3.6.1.4.1.41112.1.6.2.1.1.12.0”
[[inputs.snmp.field]]
name = “Bits.In”
oid = “1.3.6.1.4.1.41112.1.6.2.1.1.6.0”
[[inputs.snmp.field]]
name = “2.4.Clients”
oid = “1.3.6.1.4.1.41112.1.6.1.2.1.8.0”
[[inputs.snmp.field]]
name = “5.0.Clients”
oid = “1.3.6.1.4.1.41112.1.6.1.2.1.8.3”

Unraid shell script for getting stats into Grafana

Continuing the documentation effort.  This is a shell script you run from Unraid in a cron job to feed stats to InfluxDB.  You can then present them in Grafana.  Note about that, I was having a lot of trouble getting the Grafana graphs to present correctly for anything coming from this script.  I had to change the Fill from “null” to “none” in the graph.  Not sure why that’s happening, but “none” gets it to behave just like everything else.

## Assembled from this post: https://lime-technology.com/forum/index.php?topic=52220.msg512346#msg512346

## add to cron like:

## * * * * * sleep 10; /boot/custom/influxdb.sh > /dev/null 2>&1

## //0,10 * * * * /boot/custom/influxdb.sh > /dev/null 2>&1
#

# Set Vars

#

DBURL=http://192.168.x.x:8086 ## IP address of your InfluxDB server

DBNAME=dashboard ## Easier if you pick an existing DB

DEVICE=”UNRAID”

CURDATE=`date +%s`

# Current array assignment.

# I could pull the automatically from /var/local/emhttp/disks.ini

# Parsing it wouldnt be that easy though.

DISK_ARRAY=( sdn sdl sdf sdc sdj sde sdo sdh sdi sdd sdk sdm sdg sdp sdb )

DESCRIPTION=( parity disk1 disk2 disk3 disk4 disk5 disk6 disk7 disk8 disk9 disk10 disk11 disk12 disk13 cache )

#

# Added -n standby to the check so smartctl is not spinning up my drives

#

i=0

for DISK in “${DISK_ARRAY[@]}”

do

smartctl -n standby -A /dev/$DISK | grep “Temperature_Celsius” | awk ‘{print $10}’ | while read TEMP

do

curl -is -XPOST “$DBURL/write?db=$DBNAME” –data-binary “DiskTempStats,DEVICE=${DEVICE},DISK=${DESCRIPTION[$i]} Temperature=${TEMP} ${CURDATE}000000000” >/dev/null 2>&1

done

((i++))

done
# Had to increase to 10 samples because I was getting a spike each time I read it. This seems to smooth it out more

top -b -n 10 -d.2 | grep “Cpu” | tail -n 1 | awk ‘{print $2,$4,$6,$8,$10,$12,$14,$16}’ | while read CPUusr CPUsys CPUnic CPUidle CPUio CPUirq CPUsirq CPUst

do

top -bn1 | head -3 | awk ‘/load average/ {print $12,$13,$14}’ | sed ‘s/,//g’ | while read LAVG1 LAVG5 LAVG15

do

curl -is -XPOST “$DBURL/write?db=$DBNAME” –data-binary “cpuStats,Device=${DEVICE} CPUusr=${CPUusr},CPUsys=${CPUsys},CPUnic=${CPUnic},CPUidle=${CPUidle},CPUio=${CPUio},CPUirq=${CPUirq},

CPUsirq=${CPUsirq},CPUst=${CPUst},CPULoadAvg1m=${LAVG1},CPULoadAvg5m=${LAVG5},CPULoadAvg15m=${LAVG15} ${CURDATE}000000000” >/dev/null 2>&1

done

done
if [[ -f byteCount.tmp ]] ; then
# Read the last values from the tmpfile – Line “eth0”

grep “eth0” byteCount.tmp | while read dev lastBytesIn lastBytesOut

do

cat /proc/net/dev | grep “eth0” | grep -v “veth” | awk ‘{print $2, $10}’ | while read currentBytesIn currentBytesOut

do

# Write out the current stats to the temp file for the next read

echo “eth0” ${currentBytesIn} ${currentBytesOut} > byteCount.tmp
totalBytesIn=`expr ${currentBytesIn} – ${lastBytesIn}`

totalBytesOut=`expr ${currentBytesOut} – ${lastBytesOut}`
curl -is -XPOST “$DBURL/write?db=$DBNAME” –data-binary “interfaceStats,Interface=eth0,Device=${DEVICE} bytesIn=${totalBytesIn},bytesOut=${totalBytesOut} ${CURDATE}000000000” >/

dev/null 2>&1
done

done
else

# Write out blank file

echo “eth0 0 0” > byteCount.tmp

fi
# Gets the stats for boot, disk#, cache, user

#

df | grep “mnt/\|/boot\|docker” | grep -v “user0\|containers” | sed ‘s/\/mnt\///g’ | sed ‘s/%//g’ | sed ‘s/\/var\/lib\///g’| sed ‘s/\///g’ | while read MOUNT TOTAL USED FREE UTILIZATION DISK

do

if [ “${DISK}” = “user” ]; then

DISK=”array_total”

fi

curl -is -XPOST “$DBURL/write?db=$DBNAME” –data-binary “drive_spaceStats,Device=${DEVICE},Drive=${DISK} Free=${FREE},Used=${USED},Utilization=${UTILIZATION} ${CURDATE}000000000” >/dev/null 2>&

1

done

Telegraf mixed SNMP config

Following my previous post about Grafana, once everything is installed you’ll want to capture some data.  Otherwise, what’s the point.  Telegraf is a data gathering tool made by Influxdata.  It’s stupid simple to get working with InfluxDB.  After following the previous script, go to /etc/telegraf/ and edit telegraf.conf.  Near the top is the Output Plugins section.  Make sure that’s modified for your InfluxDB install.  From there, scroll down to Input Plugins.  There’s a ridiculous number of input plugins available.  We’re focused on SNMP today, but it’s worth looking through the list to see if a “need” can be solved with Telegraf before using some other custom script.

For me, I needed to add SNMP for my Ubiquiti ER-X firewall and my Nutanix CE cluster.  Here’s my SNMP config section with the obvious security bits redacted:

# # Retrieves SNMP values from remote agents
# [[inputs.snmp]]
[[inputs.snmp]]
agents = [ “192.168.x.x:161” ] ##Nutanix CE CVM IP
timeout = “5s”
version = 3

max_repetitions = 50

sec_name = “username”
auth_protocol = “SHA” # Values: “MD5”, “SHA”, “”
auth_password = “password”
sec_level = “authPriv” # Values: “noAuthNoPriv”, “authNoPriv”, “authPriv”

priv_protocol = “AES” # Values: “DES”, “AES”, “”
priv_password = “password”

name = “nutanix”
[[inputs.snmp.field]]
name = “host1CPU”
oid = “1.3.6.1.4.1.41263.9.1.6.1”
[[inputs.snmp.field]]
name = “host2CPU”
oid = “1.3.6.1.4.1.41263.9.1.6.2”
[[inputs.snmp.field]]
name = “host3CPU”
oid = “1.3.6.1.4.1.41263.9.1.6.3”
[[inputs.snmp.field]]
name = “host4CPU”
oid = “1.3.6.1.4.1.41263.9.1.6.4”
[[inputs.snmp.field]]
name = “ClusterIOPS”
oid = “1.3.6.1.4.1.41263.506.0”
[[inputs.snmp.field]]
name = “Host1MEM”
oid = “1.3.6.1.4.1.41263.9.1.8.1”
[[inputs.snmp.field]]
name = “Host2MEM”
oid = “1.3.6.1.4.1.41263.9.1.8.2”
[[inputs.snmp.field]]
name = “Host3MEM”
oid = “1.3.6.1.4.1.41263.9.1.8.3”
[[inputs.snmp.field]]
name = “Host4MEM”
oid = “1.3.6.1.4.1.41263.9.1.8.4”

[[inputs.snmp]]
agents = [ “192.168.0.1:161” ] ##Firewall IP
timeout = “5s”
retries = 3
version = 2
community = “RO_community_string”
max_repetitions = 10

name = “ERX”
[[inputs.snmp.field]]

name = “Bytes.Out”
oid = “1.3.6.1.2.1.2.2.1.10.2”
[[inputs.snmp.field]]
name = “Bytes.In”
oid = “1.3.6.1.2.1.2.2.1.16.2”

You’ll have to get Telegraf to read in the config again.  The sledgehammer method would be a reboot.  I think a Telegraf service restart would also do the trick.  Reboots for me take about 5 seconds (yep, really), so it’s useful to make sure it’s coming up clean on a reboot anyway.