Trans: Latin prefix implying "across" or "Beyond", often used in gender nonconforming situations – Scend: Archaic word describing a strong "surge" or "wave", originating with 15th century english sailors – Survival: 15th century english compound word describing an existence only worth transcending.

Author: Jess (Page 2 of 5)

QEMU for Raspian ARM!

Updated 11/6/19

Visit the repo on Github here-

tested on Mac OSX 10.14.6

Emulates a variety of Raspian releases on proper ARM hardware with QEMU.


QEMU and wget (OSX homebrew)

brew install qemu wget

Get the Python3 CLI in this repo:



After the first launch, it will launch from the persistent .qcow2 image.

With no arguments & in a new folder, Raspian "stretch-lite" (no desktop environment) will be:

  • downloaded as a zip archive with a release kernel
  • unarchived --> to img
  • converted to a Qcow2 with 8gb allocated as disk
  • launched from Qcow2 as administrator
sudo python3 

Optional Arguments:

  • -h prints CLI usage help
  • -rm removes ALL files added in dir with
  • stretch uses standard graphical stretch release with GUI
  • stretchlite for stretchlite release [default!]
  • buster for standard graphical buster release [YMMV]
  • busterlite for busterlite release [YMMV]
# examples:
sudo python3 busterlite
python3 QEMU_Raspian -h  # print help

Burn as .img:

qemu-img convert -f qcow2 -O raw file.qcow2 file.img

Four Pi Emulations ala QEMU

Small Updates to GDAL Notes

GIS Shortcuts

GDAL on OSX, Ubuntu & Windows WSL

Bash is great: It is a shell scripting language that is baked into every new Apple computer and many, many Linux distributions too. It is fun to learn and easy to find your “insert repetitive task here” to Bash translation on Stack Exchange, just a search away. As you uses the command prompt more and more for increasingly cool GIS tasks- be it for Python QGIS OR ESRI, R language for data cleaning and analysis, or just because you noticed you can get “mad amounts of work done” at increasingly faster rates- your vocabulary in the UNIX shell and bash will naturally grow.

I wanted to make a GIS post for Mac OS because it is both under-represented (for great reasons) in GIS and arguably the number 1 choice for any discerning consumers of computer hardware.

Many Linux OS options are faster for much of this “UNIX for GIS”, as quite a few of the things we need are already included with many Debian / Ubuntu distros, and come forms that have been stable for a long, long time.

If you are looking to setup a system primarily for GIS / data science (disregarding ESRI of course), See my initial notes on the Ubuntu variant Pop_OS by System76. If you like the ChromeOS vibe for multitasking and simplicity and the familiarity of Mac OS, it is a keeper (and also a sleeper, seeing how many folks are still on Windows for GIS…….).

Open Terminal
(The less you have in your way on your screen, the faster you can go!) xD

Note: in my opinion, homebrew and macPorts are good ideas- try them! If you don’t have it, get it now:

/usr/bin/ruby -e "$(curl -fsSL"

(….However, port or brew installing QGIS and GDAL (primarily surrounding the delicate links between QGIS / GDAL / Python 2 & 3 / OSX local paths) can cause baffling issues. If possible, don’t do that. Use QGIS installers from the official site and build from source!)

if you need to resolve issues with your GDAL packages via removal:
on MacPorts, try similar:

sudo port uninstall qgis py37-gdal

# on homebrew, list then remove (follow its instructions):

brew list
brew uninstall gdal geos gdal2  

!!! NOTE: I am investigating more reliable built-from-source solutions for gdal on mac.


There are numerous issues with brew-installed gdal. Those I have run into include:

  • linking issues with the crucial directory “gdal-data” (libraries)
  • linking issues Python bindings and python 2 vs. 3 getting confused
  • internal raster library conflicts against the gdal requirements
  • Proj.4 inconsistencies (see source notes below)
  • OSX Framework conflicts with source / brew / port (
  • Linking conflicts with old, qgis default / LTR libraries against new ones
  • Major KML discrepancies: expat standard vs libkml.
    brew install gdal
    # brew install qgis can work well too.  At least you can unbrew it!

Next, assuming your GDAL is not broken (on Mac OS this is rare and considered a miracle):

# double check CLI is working:
gdalinfo --version
# “GDAL 2.4.0, released 2018/12/14”
# list of args

Using Ubuntu GDAL on Windows w/ WSL

LINK: get the WSL shell from Microsoft

# In the WSL shell:

sudo apt-get install python3.6-dev -y
sudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update
sudo apt-get install libgdal-dev -y
sudo apt-get install gdal-bin -y

# See here for more notes including Python bindings:

In a new Shell:

# Double check the shell does indeed have GDAL in $PATH:
gdalinfo --version

To begin- try a recent GIS assignment that relies on the ESRI mosaic system and lots of GUI and clicking but use GDAL instead.

Data source:

!! Warning! These files are not projected in a way ESRI or GDAL understands. They WILL NOT HAVE A LOCATION IN QGIS. They will, however, satisfy the needs of the assignment.

# wget on mac is great.  This tool (default on linux) lets us grab GIS data from
# most providers, via FTP and similar protocols.

brew install wget

make some folders

mkdir GIS_Projects && cd GIS_Projects

use wget to download every .dem file (-A .dem) from the specified folder and sub-folders (-r)

wget -r -A .dem


make an index file of only .dem files.
(If we needed to download other files and keep them from our wget (more common)
this way we can still sort the various files for .dem)

ls -1 *.dem > dem_list.txt

use gdal to make state-plane referenced “Output_merged.tif” from the list of files
in the index we made.
it will use a single generic "0 0 255" band to show gradient. -init "0 0 255" -o Output_Merged.tif --optfile dem_list.txt

copy the resulting file to desktop, then return home

cp Output_Merged.tif ~/desktop && cd

if you want (recommended):

rm -rf GIS_Projects  # remove .dem files.  Some are huge!

In Finder at in ~/desktop, open the new file with QGIS. A normal photo viewer will NOT show any detail.

Need to make something like this a reusable script? In Terminal, just a few extra steps:

mkdir GIS_Scripts && cd GIS_Scripts

open an editor + filename. Nano is generally pre-installed on OSX.



  • ctrl+X , then Y for yes

make your file runnable:

chmod u+x

run with ./


You can now copy + paste your script anywhere you want and run it there. scripts like this should not be exported to your global path / bashrc and will only work if they are in the directory you are calling them: If you need a global script, there are plenty of ways to do that too.

See /Notes_GDAL/ for notes on building GDAL from source on OSX

Click here to visit this repo


Decentralized Pi Video Monitoring w/ motioneye & BATMAN

Visit the me here on Github
Added parabolic musings 10/16/19, see below

...On using motioneye video clients on Pi Zeros & Raspbian over a BATMAN-adv Ad-Hoc network

link: motioneyeos
link: motioneye Daemon
link: Pi Zero W Tx/Rx data sheet:
link: BATMAN Open Mesh

This implementation of motioneye is running on Raspbian Buster (opposed to motioneyeos).

Calculating Mesh Effectiveness w/ Python:
Please take a look at the idea here is one should be able to estimate the maximum plausible distance between mesh nodes before setting anything up. It can be run with no arguments-


...with no arguments, it should use default values (Tx = 20 dBm, Rx = |-40| dBm) to print this:

you can add (default) Rx Tx arguments using the following syntax:
                 python3 20 40
                 python3 <Rx> <Tx>                 

 57.74559999999994 ft = max. mesh node spacing, @
 Rx = 40
 Tx = 20

Regarding the Pi:
The Pi Zero uses an onboard BCM43143 wifi module. See above for the data sheet. We can expect around a ~19 dBm Tx signal from a BCM43143 if we are optimistic. Unfortunately, "usable" Rx gain is unclear in the context of the Pi.

Added 10/16/19:
Notes on generating an accurate parabolic antenna shape with FreeCAD’s Python CLI:

For whatever reason, (likely my own ignorance) I have been having trouble generating an accurate parabolic dish shape in Fusion 360 (AFAICT, Autodesk is literally drenching Fusion 360 in funds right now, I feel obligated to at least try). Bezier, spline, etc curves are not suitable!
If you are not familiar with FreeCAD, the general approach- geometry is formed through fully constraining sketches and objects- is quite different from Sketchup / Tinkercad / Inventor / etc, as most proprietary 3d software does the “constraining” of your drawings behind the scenes. From this perspective, you can see how the following script never actually defines or changes the curve / depth of the parabola; all we need to do is change how much curve to include. A wide, shallow dish can be made by only using the very bottom of the curve, or a deep / narrow dish by including more of the ever steepening parabolic shape.

import Part, math

# musings derived from:

# thinking about units here:
tu = FreeCAD.Units.parseQuantity

def mm(value):
    return tu('{} mm'.format(value))

rs = mm(1.9)
thicken = -(rs / mm(15)) 

# defer to scale during fitting / fillet elsewhere 
# create a parabola with the symmetry axis (0,0,1)

# get only the right part of the curve

# make a solid

# apply a thickness


# Fill screen:
# Remove Part in default env:

FWIW, here is my Python implimentation of a Tx/Rx "Free Space" distance calulator-

from math import log10
from sys import argv
# estimate free space dBm attenuation:
# ...using wfi module BCM43143:

Tx = 19~20 dBm
Rx = not clear how low we can go here

d = distance Tx --> Rx
f = frequency
c = attenuation constant: meters / MHz = -27.55; see here for more info:

f = 2400  # MHz
c = 27.55 # RF attenuation constant (in meters / MHz)

def_Tx = 20  # expected dBm transmit
def_Rx = 40  # (absolute value) of negative dBm thesh

def logdBm(num):
    return 20 * log10(num)

def maxDist(Rx, Tx):
    dBm = 0
    d = .1  # meters!
    while dBm < Tx + Rx:
        dBm = logdBm(d) + logdBm(f) - Tx - Rx + c
        d += .1  # meters!
    return d

# Why not use this with arguments Tx + Rx from shell if we want:
def useargs():
    use = bool
        if len(argv) == 3:
            use = True
        elif len(argv) == 1:
            print('\n\nyou can add (default) Rx Tx arguments using the following syntax: \n \
                python3 20 40 \n \
                python3 <Rx> <Tx> \
            use = False
            print('you must use both Rx & Tx arguments or no arguments')
            raise SystemExit
        print('you must use both Rx & Tx arguments or no arguments')
        raise SystemExit
    return use

def main():

    if useargs() == True:
        arg = [int(argv[1]), int(argv[2])]
        arg = [def_Rx, def_Tx]

    print(str('\n ' + str(maxDist(arg[0], arg[1])*3.281) + \
        ' ft = max. mesh node spacing, @ \n' + \
        ' Rx = ' + str(arg[0]) + '\n' + \
        ' Tx = ' + str(arg[1])))


Google Calendar API with Chapel & Python

Mirroring the repo here:

Google Calendar API with Chapel & Python

Despite Chapel's many quirks and annoyances surrounding string handling, its efficiency and ease of running in parallel are always welcome. The idea here is a Chapel script that may need to weed through enormous numbers of files while looking for a date tag ($D + other tags currently) is probably a better choice overall than a pure Python version. (the intent is to test this properly later.)

As of 9/19/19, there is still a laundry list of things to add- control flow (for instance, “don't add the event over and over”) less brittle syntax, annotations, actual error handling, etc. It does find and upload calendar entries though!

I am using Python with the Google Calendar API (see here: in a looping Daemon thread. All the sifting for tags is managed with the Chapel binary, which dumps anything it finds into a csv from which the daemon will push calendar entries with proper formatting. FWIW, Google’s dates (datetime.datetime) adhere to RFC3339 ( which conveniently is the default of the datetime.isoformat() method.

Some pesky things to keep in mind:

This script uses a sync$ variable to lock other threads out of an evaluation during concurrency. So far I think the easiest way to manage the resulting domain is from within a module like so:

module charMatches {
  var dates : domain(string);

Here, domain charMatches.dates will need to accessed as a reference variable from any procedures that need it.

proc dateCheck(aFile, ref choice) {
coforall folder in walkdirs('check/') {
    for file in findfiles(folder) {
        dateCheck(file, charMatches.dates);

errors like:

error: unresolved call '_ir_split__ref_string.size'

unresolved call 'norm(promoted expression)'

...or other variants of:
string.split().size  (length, etc)

...Tie into a Chapel Specification issue.

The short solution is do not use .split; instead, I have been chopping strings with .partition().

// like so:
if choice.contains(line.partition(hSep)[3].partition(hTerminate)[1]) == false {
    ...process string...

Canon: Conquered

Over 14 years ago, Lim Jeong-hyun posted a cover of Pachelbel's Canon in South Korea, where it eventually filtered through the inter-webnet of old into Youtube.

I saw this video in its infancy, and soon began to learn guitar too.

5 years later I recorded my first cover of Pachelbel's Canon.

Another 5 years passed (2016 at this point) before I began recording a second cover, posting this muse and this percolation on the subject.   While most attempts were futile....

...I made it through the whole song by take # 1,394  (2017 at this point):

Some more years go by and here we are today:


Persistent, Live Ubuntu for College

Below is are live mirrors of my "PSU Hacking Club" Ubuntu repos.

Simple File Hosting

README yet to be updated with Makerspace-specific formatting

Static site built quickly with Hugo CLI

# on OSX
# get hugo

brew install hugo

# clone site

git clone
cd static-site

# Compile and compress public directory

zip -r public

# upload and host with sftp & ssh

> cd
> put

# new terminal window

# check your remote filesystem- the idea is:
> unzip
> rm -rf

visit us

Also, check out the evolving PSU Hacking Club wiki here!

xD  - Jess

Summer 2019 Update!

GIS Updates:

Newish Raster / DEM image → STL tool in the Shiny-Apps repo:

See the (non-load balanced!) live example on the Heroku page:

Summarized for a forum member here too:

CAD / CAM Updates:

Been revamping my CNC thoughts- 

Basically, the next move is a complete rebuild (primarily for 6061 aluminum).

I am aiming for:

  • Marlin 2.x.x around either a full-Rambo or 32 bit Archim 1.0 (
  • Dual endstop configuration, CNC only (no hotend support)
  • 500mm2 work area / swappable spoiler boards (~700mm exterior MPCNC conduit length)
  • Continuous compressed air chip clearing, shop vac / cyclone chip removal
  • Two chamber, full acoustic enclosure (cutting space + air I/O for vac and compressor)
  • Full octoprint networking via GPIO relays

FWIW: Sketchup MPCNC:

Also TinkerCAD version:

Electric Drivetrain Development:

BORGI / Axial Flux stuff:

Designed some rough coil winders for motor design here:


Also, an itty-bitty, skate bearing-scale axial flux / 3-phase motor to hack upon:


- Jess

Warbler Trillers of the Charles

The first ones to arrive in MA, brush up!


Palm warbler



This is usually the first one to arrive.  Gold bird, medium sized warbler, rufus hat.  When they arrive in MA they are often found lower than usual / on the ground looking for anything they can munch on.  Song is a rapid trill. More “musical / pleasant” than a fast chipping sparrow, faster than many Junco trills.


Pine warbler



Slimmer than palm, no hat, very slim beak, has streaks on the breast usually.  Also a triller. They remain higher in the trees on arrival.


Yellow-rumped warbler



Spectacular bird, if it has arrived you can’t miss it- also they will arrive by the dozen so worth waiting for a good visual.  These also trill, which is another reason it is good to get a visual. The trill is slow, very “sing-song”, and has a downward inflection at the end.  If there are a bunch sticking around for the summer, try to watch some sing- soon enough you will be able to pick out this trill from the others.


-- Yellow warbler says “sweet sweet sweet, I’m so Sweet!” and can get a bit confusing with Yellow-rumped warbler

-- Chestnut-sided warbler says “very very pleased to meet ya!” and can get a bit confusing with Yellow warbler


Black-and-white warbler



Looks like a zebra - always acts like a nuthatch (clings to trunk and branches).  This one trills like a rusty wheel. It can easily be distinguished after a bit of birding with some around.  


American Redstart



Adult males look like a late 50’s hot-rodded American muscle car: long, low, two tone paint job.  Matte/luster black with flame accents. Can’t miss it. The females and young males are buff (chrome, to keep in style I guess) with yellow accents.  Look for behavior- if a “female” is getting beaten up while trying to sing a song in the same area, it is actually a first year male failing to establish a territory due to obviously being a youth.



- Jess

Notes on a Free and Open Source Notes App:  Joplin

Joplin for all your Operating Systems and devices

As a lifelong IOS + OSX user (Apple products), I have used many, many notes apps over the years.  From big name apps like OmniFocus, Things 3, Notes+, to all the usual suspects like Trello, Notability, Notemaster, RTM, and others, I always eventually migrate back to Apple notes, simply because it is always available and always up to date.  There are zero “features” besides this convenience, which is why I am perpetually willing to give a new app a spin.

Joplin is free, open source, and works on OSX, Windows, Linux operating systems and IOS and Android phones.  

Find it here:

brew install joplin 

The most important thing this project has nailed is cloud support and syncing.  I have my iPhone and computers syncing via Dropbox, which is easy to setup and works….  really well. Joplin folks have added many cloud options, so this is unlikely to be a sticking point for users.

Here are some of the key features:

  • Markdown is totally supported for straightforward and easy formatting
  • External editor support for emacs / atom / etc folks
  • Layout is clean, uncluttered, and just makes sense
  • Built-in markdown text editor and viewer is great
  • Notebook, todo, note, and tags work great across platforms
  • Browser integration, E2EE security, file attachments, and geolocation included

Hopefully this will be helpful.


- Jess

Mac OSX: Fixing GPT and PMBR Tables

My computer recently crashed very, very hard, while I was removing an small empty alternative OS partition I no longer needed.  This is a fairly mundane operation that I do now and again, and is a ongoing fight to keep at least a few gigs of space free for actual work on precious 250gb Mac SSD.  

The crash results?  Toasted GPT tables all around.   My 2015 computer’s next move was to reboot- only to find essentially no partitions of memory… at all.  What it did show was (wait for it) Clover bootloader of all things, with a single windows boot camp icon (nothing in there either).  That is so wrong…. On all levels!

I brought the machine to the local university repair.  They declared this machine bricked and offered to wipe it.  Back to me it came…

I scheduled an Apple support session with a phone rep, which after around 45 minutes of actually productive troubleshooting ideas (none helping though) was forwarded to a senior supervisor.  She was interested in this problem, and we scheduled a larger block of time. But, in the meantime, I still wanted to try again….

How to recover a garbled GPT table for Mac OSX:

Start with clean SMC and PRAM / NVRAM.

Clearing these actually made accessing internet recovery (how we get to a stand-in OS with a terminal) dozens of times faster.  2.5 hours to 7 minutes. I actually waited 2.5 hours twice on separate attempts before I cleared these.

Follow these Apple links to perform these operations:

Get the computer with a text editor open.

Restart the computer into internet recovery.  Command + R or Command + Shift + R.


Open a Terminal.  The graphical disk utility is useless because the disk / partition we want is unreachable(so it will say everything is great).


diskutil list

For me, I see disk0s2 is 180.6 gb.  That’s my stuff!

I also found /dev/disk2 → /dev/disk14 to be tiny partitions- don’t worry about those.

The syntax you are looking for is:

Name: “untitled” Identifier: disk#

(NOT disk#s#)

Write down ALL of the above information for the disk you are after.  That is probably disk0.


gpt -r show disk0

Copy the following readout in your terminal for all entries bigger than “32”.  The critical fields here are Start, Size, Index, and Contents. Each field is supremely important.

Here is mine (formatted for web):

# Disk0, with contents > “32” :

# First Table:

Start: 40  

Size: 409600

Index:  1

Contents: C12A7328-F81F-11D2-BA4B-00A0C93EC93B

# Second table, the one with my data:

Start: 409640

Size: 352637568

Index: 2


Note, this is the initial Contents.  I rewrote this once with the correct Apple Index 2 data but did not create a new table (leaving the rest of the broken bits broken).  We are replacing / destroying a table here, but not the data.     


# unmount the disk.  From here we are doing tables, not disks / data.

diskutil unmountDisk disk0

# Get rid of the GPT on the disk we are recovering.  We are not touching the data.

gpt destroy disk0

# Make a new one to start with some fresh values.

gpt create -f disk0

# perform magic trick


# we must add that first small partition at index 1.  Verbatim.

gpt add -i 1 -b 40 -s 409600 -t C12A7328-F81F-11D2-BA4B-00A0C93EC93B disk0

# index two (for me) is my data.  We are going to use the default OSX / Mac HD partition values.

# the Length of “372637568” is not as sure fire as the GPT Contents.  

# YMMV, but YOLO.

gpt add -i 2 -b 409640 -s 372637568 -t 7C3457EF-0000-11AA-AA11-00306543ECAC disk0

Again, that Contents value is 7C3457EF-0000-11AA-AA11-00306543ECAC.

- Jess

written in the recovered computer xD

Musings On Chapel Language and Parallel Processing

View below the readme mirror from my Github repo. Scroll down for my Python3 evaluation script.

....Or visit the page directly: 


Investigating modern concurrent programming ideas with Chapel Language and Python 3

See here for dupe detection: /FileChecking-with-Chapel

Iterating through all files for custom tags / syntax: /GenericTagIterator

added 9/14/19:

The thinking here is one could write a global, shorthand / tag-based note manager making use of an efficient tag gathering tool like the example here. Gone are the days of actually needing a note manager- when the need presents itself, one could just add a calendar item, todo, etc with a global tag syntax.

The test uses $D for date: $D 09/14/19

//  Chapel-Language  //

// non-annotated file @ /GenericTagIterator/nScan.chpl //

use FileSystem;
use IO;
use Time;

config const V : bool=true;  // verbose logging, currently default!

module charMatches {
  var dates = {("")};  

// var sync1$ : sync bool=true;  not used in example- TODO: add sync$ var back in!!

proc charCheck(aFile, ref choice, sep, sepRange) {

    // note, reference argument (ref choice) is needed if using Chapel structure "module.domain"

    try {
        var line : string;
        var tmp = openreader(aFile);
        while(tmp.readline(line)) {
            if line.find(sep) > 0 {
                choice += line.split(sep)[sepRange];
                if V then writeln('adding '+ sep + ' ' + line.split(sep)[sepRange]);
    } catch {
      if V then writeln("caught err");

coforall folder in walkdirs('check/') {
    for file in findfiles(folder) {
        charCheck(file, charMatches.dates, '$D ', 1..8);

Get some Chapel:

In a (bash) shell, install Chapel:
Mac or Linux here, others refer to:

# For Linux bash:
git clone
tar xzf chapel-1.18.0.tar.gz
cd chapel-1.18.0
source util/setchplenv.bash
make check

#For Mac OSX bash:
# Just use homebrew
brew install chapel # :)

Get atom editor for Chapel Language support:

#Linux bash:
sudo apt-get install atom
apm install language-chapel
# atom [yourfile.chpl]  # open/make a file with atom

# Mac OSX (download):
# bash for Chapel language support
apm install language-chapel
# atom [yourfile.chpl]  # open/make a file with atom

Using the Chapel compiler

To compile with Chapel:

chpl MyFile.chpl # chpl command is self sufficient

# chpl one file class into another:

chpl -M classFile runFile.chpl

# to run a Chapel file:

Now Some Python3 Evaluation:

# Ajacent to compiled FileCheck.chpl binary:

python3 will loop FileCheck and find the average times it takes to complete, with a variety of additional arguments to toggle parallel and serial operation. The iterations are:

ListOptions = [Default, Serial_SE, Serial_SP, Serial_SE_SP]
  • Default - full parallel

  • Serial evaluation (--SE) but parallel domain creation

  • Serial domain creation (--SP) but parallel evaluation

  • Full serial (--SE --SP)

Output is saved as Time_FileCheck_Results.txt

  • Output is also logged after each of the (default 10) loops.

The idea is to evaluate a "--flag" -in this case, Serial or Parallel in FileCheck.chpl- to see of there are time benefits to parallel processing. In this case, there really are not any, because that program relies mostly on disk speed.

Evaluation Test:

# A WIP by Jess Sullivan
# evaluate average run speed of both serial and parallel versions
# of FileCheck.chpl  --  NOTE: coforall is used in both BY DEFAULT.
# This is to bypass the slow findfiles() method by dividing file searches
# by number of directories.

import subprocess
import time

File = "./FileCheck" # chapel to run

# default false, use for evaluation
SE = "--SE=true"

# default false, use for evaluation
SP = "--SP=true" # no coforall looping anywhere

# default true, make it false:
R = "--R=false"  #  do not let chapel compile a report per run

# default true, make it false:
T = "--T=false" # no internal chapel timers

# default true, make it false:
V = "--V=false"  #  use verbose logging?

# default is false
bug = "--debug=false"

Default = (File, R, T, V, bug) # default parallel operation
Serial_SE = (File, R, T, V, bug, SE)
Serial_SP = (File, R, T, V, bug, SP)
Serial_SE_SP = (File, R, T, V, bug, SP, SE)

ListOptions = [Default, Serial_SE, Serial_SP, Serial_SE_SP]

loopNum = 10 # iterations of each runTime for an average speed.

# setup output file
file = open("Time_FileCheck_Results.txt", "w")

file.write(str('eval ' + str(loopNum) + ' loops for ' + str(len(ListOptions)) + ' FileCheck Options' + "\n\\"))

def iterateWithArgs(loops, args, runTime):
    for l in range(loops):
        start = time.time()
        end = time.time()

for option in ListOptions:
    runTime = []
    iterateWithArgs(loopNum, option, runTime)
    file.write("average runTime for FileCheck with "+ str(option) + "options is " + "\n\\")
    file.write(str(sum(runTime) / loopNum) +"\n\\")
    print("average runTime for FileCheck with " + str(option) + " options is " + "\n\\")
    print(str(sum(runTime) / loopNum) +"\n\\")


Evaluating Ubuntu Pop OS: Dual Boot Setup

Dual OS on a 2015 MacBook pro

As the costs of Apple computers continue to skyrocket and the price of useable amounts of storage zoom past a neighboring galaxy (for a college student at least), I am always on on the hunt for cost effective solutions to house and process big projects and large data.

Pop OS (a neatly wrapped Ubuntu) is the in-house OS from System76.  After looking through their catalog of incredible computers and servers, I thought it would be a good time to see how far I can go with an Ubuntu daily driver.  Of course, there are many major and do-not-pass-go downsides- see the below list:

  • Logic Pro X → There is no replacement 🙁   A killer DAW with fantastic AU libraries. I am versed with Reaper and Bitwig, but neither is as complete as Logic Pro.  I will be evaluating POP with an installation of Reaper, but with so few plugins (I own very few third party sets) this is not a fair replacement.
  • Adobe PS and LR:  I do not like Adobe, but these programs are... ...kind of crucial for most project of mine that involve 2d, raster graphics.  I continue to use Inkscape for many tasks, but it is irrelevant when it comes to pixel-based work and photo management / bulk operations.
  • AutoCAD / Fusion 360 / Sketchup:  I like FreeCAD a lot, but it is not at all like the other programs.  Not worse or better, but these are all very different animals for different uses.
  • Apple notes and other apple-y things:  OSX is extremely refined. Inter-device solutions are superb.  I have gotten myself used to Google Keep, but it is not quite at the in-house Apple level.
  • XCode and IOS Simulator environments:  I do use Expo, but frankly to make products for Apple you need a Mac.

Dual Boot (OSX and Pop Ubuntu) Installation on a 2015 MBP:

This process is quite simple, and only calls for a small handful of post-installation tweaks.  My intent is to create a small sandbox with minimal use of “extras” (no extra boot managers or anything like that)


Partition separate “boot”, “home”, and other drives

  • I am using a 256gb micro sd partitioned in half for OSX and Pop_OS (Sandisk extreme, “v3” speed rating version card via a BaseQi slot adapter)

Use the partition tool in Mac disk utility.  Be sure to set these new partitions as FAT 32- we will be using ext4 and other more linux-y filesystems upon installation, so these need to be as generic as possible.

Get a copy of Pop_OS from System76.

Use Etcher (recommended) or any other image burning tool to create a boot key for Pop.  

The USB key only has one small job, in which Pop_os will be burned into a better location in your boot partition made in the previous step.  If you are coming from a hackintosh experience, fear not: everything will stay in the Macbook Pro, not extra USB safety dongles or Kexts, or Plist mods…!


Restart your computer and hold down the alt-option Key.  THIS IS HOW TO SWITCH from Pop_os, OSX, Bootcamp, and anything else you have in there.  You should see an “efi” option next to the default OSX. (note- at least in my case, the built-in bootloader defaults to the last used OS at each restart.)

Once you are in the Pop_OS installer, click through and select the appropriate partitions when prompted.  After this installation, you may remove the USB key and continue to select
“efi” in the bootloader.


You are now in Pop_OS!  Using the alt/option key will become second nature… but some Pop key mappings may not.  Continue for a list of Macbook Pro - specific tweaks and notes.

First moves:

Go to the Pop Shop and get the “Tweaks” tool.  I made one or two small keymap changes, but this is likely personal preference.  

Default, important Key Mappings:

Command will act as a “control center-ish” thing.  It will not copy or paste anything for you.

Control does what Command did on OSX.  

Terminal uses Control+Shift for copy and paste, but only in Terminal:  if you pull a Control+Shift+C in Chrome, you will get the Dev tool GUI...  The Shift key thing is needed unless you are inclined to root around and change it.

Custom Boot Scripts and Services:

In an effort to make things simple, I made a shell script to house the processes I want running when I turn on the computer- this is to streamline the “.service” making process.  While it may only take marginally more time to make a new service, this way I can keep track of what is doing what from a file in my documents folder.

In terminal, go to where your services live if you want to look:

cd /etc/systemd/system

Or, cut to the chase:

sudo nano /etc/systemd/system/

Paste the following into this new file:

_____________Begin _After_This_Line____________________


Description=Start at Open plz




_____________End _Above_This_Line____________________

Exit nano (saving as you go) and cd back to “/”.


sudo nano /Documents/

Paste the following (and any scripts you may want, see the one I have commented out for odrive CLI) into this new file:

_____________Begin _After_This_Line____________________


# Uncomment the following if you want 24/7 odrive in your system

# otherwise do whatever you want

#nohup "$HOME/.odrive-agent/bin/odriveagent" > /dev/null 2>&1 &

# end

_____________End _Above_This_Line____________________

After exiting the shell script, start it all up with the following:

sudo systemctl start

sudo systemctl enable

Cloud file management with Odrive CLI and Odrive Utilities:

Visit one of the two Odrive CLI pages- this one has linux in it:

Please visit this repo to get going with --recursive and other odrive utilities

These are the two commands I ended up putting in a markdown file on my desktop for easy access.  Nope, not nearly as cool as it is on OSX. But it works…

Odrive sync: [-h] for help


python "$HOME/.odrive-agent/bin/" sync


Odrive utilities:


python "$HOME/odrive-utilities/" sync --recursive


Next, Get Some Apps:

Download Chrome.  Sign into Chrome to get your chrome OS apps loaded into the launcher- in my case, I needed Chrome remote desktop.  DO NOT DOWNLOAD ADDITIONAL PACKAGES for Chrome Remote Desktop, if that is your thing. They will halt all system tools (disk utils, Gnome terminal, graphical file viewer…   !!See this thread, it happened to me!! )

Stock up!  

Get Atom editor:

...Or my favorites:



MySQL Workbench:

If you get stuck:  make sure you have tried installing as root ($ sudo su -) and verified passwords with ($ sudo mysql_secure_installation)  

See here to start “rooting around” MySQL issues:

Get some GIS tools:


sudo apt-get install qgis python-qgis qgis-plugin-grass

uGet for bulk USGS data download!

sudo add-apt-repository ppa:plushuang-tw/uget-stable

sudo apt install uget

That's all for now- Cheers!


Deploy Shiny R apps along Node.JS

Find the tools in action on Heroku as a node.js app!

See the code on GitHub:

After many iterations of ideas regarding deployment for a few research Shiny R apps, I am glad to say the current web-only setup is 100% free and simple to adapt.   I thought I'd go through some of the Node.JS bits I have been fussing with. 

The Current one:  

Heroku has a free tier for node.js apps.  See the pricing and limitations here: as far as I can tell, there is little reason to read too far into a free plan; they don’t have my credit card, and thy seem to convert enough folks to paid customers to be nice enough to offer a free something to everyone.  

Shiny apps- works straight from RStudio.  They have a free plan. Similar to Heroku, I can care too much about limitations as it is completely free.  

The reasons to use Node.JS (even if it just a jade/html wrapper) are numerous, though may not be completely obvious.  If nothing else, Heroku will serve it for free….

Using node is nice because you get all the web-layout-ux-ui stacks of stuff if you need them.  Clearly, I have not gone to many lengths to do that, but it is there.

Another big one is using node.js with Electron. The idea is a desktop app framework serves up your node app to itself, via the chromium.  I had a bit of a foray with Electron- the node execa npm install execa package let me launch a shiny server from electron, wait a moment, then load a node/browser app that acts as a interface to the shiny process.  While this mostly worked, it is definitely overkill for my shiny stuff.  Good to have as a tool though.


Recycled Personal “Cloud Computing” under NAT

As many may intuit, I like the AWS ecosystem; it is easy to navigate and usually just works.  

...However- more than 1000 dollars later, I no longer use AWS for most things....


My goals: 

Selective sync:  I need a unsync function for projects and files due to the tiny 256 SSD on my laptop (odrive is great, just not perfect for cloud computing.

Shared file system:  access files from Windows and OSX, locally and remote

Server must be headless, rebootable, and work remotely from under a heavy enterprise NAT (College)

Needs more than 8gb ram

Runs windows desktop remotely for gis applications, (OSX on my laptop)


Have as much shared file space as possible: 12TB+


Server:  recycled, remote, works-under-enterprise-NAT:

Recycled Dell 3010 with i5:

- Cost: $75 (+ ~$200 in windows 10 pro, inevitable license expense) 

free spare 16gb ram laying around, local SSD and 2TB HDD upgrades

- Does Microsoft-specific GIS bidding, can leave running without hampering productivity

Resilio (bittorrent) Selective sync:

- Cost: $60

- p2p Data management for remote storage + desktop

- Manages school NAT and port restrictions well (remote access via relay server)

Drobo 5c:

Attached and syncs to 10TB additional drobo raid storage, repurposed for NTFS

  • Instead of EBS (or S3)


What I see:  front end-

Jump VNC Fluid service:

- Cost: ~$30

- Super efficient Fluid protocol, clients include chrome OS and IOS,  (with mouse support!)

- Manages heavy NAT and port restrictions well

- GUI for everything, no tunneling around a CLI

  • Instead of Workspaces, EC2

Jetbrains development suite: (OSX)

- Cost:  FREE as a verified GitHub student user.

- PyCharm IDE, Webstorm IDE

  • Instead of Cloud 9


Total (extra) spent: ~$165

(Example:  my AWS bill for only October was $262)



Quick fix: 254 character limit in ESRI Story Map?

If you happened to be working with....  KML data (or any data with large description strings) and transitioning it into the ESRI Story Map toolset, there is a very good chance  you hit the the dBase 254 character length limit with the ESRI Shapefile upload.  Shapefiles are always a terrible idea.


the solution:  with GDAL or QGIS (alright, even in ArcMap), one can use GeoJSON as an output format AND import into the story map system- with complete long description strings!



Merge vector layers -> save to file -> GeoJSON

import arcpy

import os

arcpy.env.workspace = "/desktop/arcmapstuff"

arcpy.FeaturesToJSON_conversion(os.path.join("outgdb.gdb", "myfeatures"), "output.json")

ogr2ogr -f GeoJSON output.json input.kml

New App:  KML Search and Convert

Written in R; using GDAL/EXPAT libraries on Ubuntu and hosted with AWS EC2.

New App:  KML Search and Convert

Here is an simple (beta) app of mine that converts KML files into Excel-friendly CSV documents.  It also has a search function, so you can download a subset of data that contains keywords.   🙂

The files will soon be available in Github.

I'm still working on a progress indicator; it currently lets you download before it is done processing.   Know a completely processed file is titled with "kml2csv_<yourfile>.csv".

...YMMV.  xD

GDAL for R Server on Ubuntu – KML Spatial Libraries and More

GDAL for R Server on Red Hat Xenial Ubuntu - KML Spatial Libraries and More

If you made the (possible mistake) of running with a barebones Red Hat Linux instance, you will find it is missing many things you may want in R.   I rely on GDAL (the definitive Geospatial Data Abstraction Library) on my local OSX R setup, and want it on my server too.  GDAL contains many libraries you need to work with KML, RGDAL, and other spatial packages.  It is massive and usually take a long time to sort out on any machine.

These notes assume you are already involved with a R server (usually port 8787 in a browser).  I am running mine from an EC2 instance with AWS.

! Note this is a fresh server install, using Ubuntu; I messed up my original ones while trying to configure GDAL against conflicting packages. If you are creating a new one, opt for at least a T2 medium (or go bigger) and find the latest Ubuntu server AMI.  For these instructions, you want an OS that is as generic as possible.

On Github:

From Bash:

# SSH into the EC2 instance: (here is the syntax just in case)

#ssh -i "/Users/YourSSHKey.pem"

sudo su -

apt-get update

apt-get upgrade

nano /etc/apt/sources.list

#enter as a new line at the bottom of the doc:

deb xenial/

#exit nano


chmod 777



From SSH:

# SSH into the EC2 instance: (here is the syntax just in case)

ssh -i "/Users/YourSSHKey.pem"

# if you can, become root and make some global users- these will be your access to

# RStudio Server and shiny too!

sudo su –

adduser <Jess>

# Follow the following prompts carefully to create the user

apt-get update

nano /etc/apt/sources.list

# enter as a new line at the bottom of the doc:

deb xenial/

# exit nano

# Start, or try bash:

apt-get install r-base

apt-get install r-base-dev

apt-get update

apt-get upgrade


tar xvf gdal-2.3.1.tar.gz

cd  gdal-2.3.1

# begin making GDAL: this all takes a while

./configure  [if your need proper kml support (like me), search on configuring with expat or libkml.   There are many more options for configuration based on other packages that can go here, and this is the step to get them in order...]

sudo make

sudo make install

cd # Try entering R now and check the version!

# Start installing RStudio server and Shiny

apt-get update

apt-get upgrade
sudo apt-get install gdebi-core
sudo gdebi rstudio-server-1.1.456-amd64.deb

# Enter R or go to the graphical R Studio installation in your browser


# Authenticate if using the graphical interface using the usr:pwd you defined earlier

# this will take a long time


# Note any errors carefully!



install.packages(c("data.table", "tidyverse”, “shiny”)  # etc

Well, there you have it!



##Later, ONLY IF you NEED Anaconda, FYI:

# Get Anaconda: this is a large package manager, and is could be used for patching up missing # dependencies:

#Use  "ls" followed by rm -r <anaconda> (fill in with ls results) to remove conflicting conda

# installers if you have any issue there, I am starting fresh:

mkdir binconda

# *making a weak attempt at sandboxing the massive new package manager installation*

cd binconda
# install and follow the prompts

# Close the terminal window completely and start a new one, and ssh back to where you left

# off.  Conda install requires this.

# open and SSH back into your instance.  You should now have either additional flexibility in

# either patching holes in dependencies, or created some large holes in your server.  YMMV.

### Done

Red Hat stuff:

Follow these AWS instructions if you are doing something else:

See my notes on this here:

and notes on Shiny server:

GDAL on Red Hat:- Existing threads on this:

This is a nice short thread about building from source:

neat RPM package finding tool, just in case:

Info on the LIBKML driver if you end up with issues there:


I hope this is useful- GDAL is important and best to set it up early.  It will be a pain, but so is losing work while trying to patch it in later.  xD




INFO: Deploy a Shiny web app in R using AWS (EC2 Red Hat)

Info on deploying a Shiny web app in R using AWS (EC2 Redhat)

As a follow-up to my post on how to create an AWS RStudio server, the next logical step is to host some useful apps you created in R for people to use.  A common way to do this is the R-specific tool Shiny, which is built in to RStudio.  Learning the syntax to convert R code into a Shiny app is rather subtle, and can be hard.  I plan to do a more thorough demo on this- particularly the use of the $ symbol, as in “input$output”- later. 🙂


It turns out hosting a Shiny Web app provides a large number of opportunities for things to go wrong….  I will share what worked for me.  All of this info is accessed via SSH, to the server running Shiny and RStudio.


I am using the AWS “Linux 2” AMI, which is based on the Red Hat OS.  For reference, here is some extremely important Red Hat CLI language worth being familiar with and debugging:


sudo yum install” and “wget” are for fetching and installing things like shiny.  Don’t bother with instructions that include “apt-get install”, as they are for a different Linux OS!


sudo chmod -R 777” is how you change your directory permissions for read, write, and execute (all of those enabled).  This is handy if your server disconnecting when the app tries to run something- it is a simple fix to a problem not always evident in the logs.  The default root folder from which shiny apps are hosted and run is “/srv/shiny-server” (or just “/srv” to be safe).


nano /var/log/shiny-server.log” is the location of current shiny logs.


sudo stop shiny-server” followed by “sudo start shiny-server” is the best way to restart the server- “sudo restart shiny-server” is not a sure bet on any other process.  It is true, other tools like a node.js server or nginx could impact the success of Shiny- If you think nginx is a problem, “cd /ect/nginx” followed by “ls” will get you in the right direction.  Others have cited problems with Red Hat not including the directories and files at “/etc/nginx/sites-available”.  You do not need these directories.  (though they are probably important for other things).


sudo rm -r” is a good way to destroy things, like a mangled R studio installation.  Remember, it is easy enough to start again fresh!  🙂


sudo nano /etc/shiny-server/shiny-server.conf” is how to access the config file for Shiny.  The fresh install version I used did not work!  There will be lots of excess in that file, much of which can causes issues in a bare-bones setup like mine.  One important key is to ensure Shiny is using a root user- see my example file below.  I am the root user here (jess)- change that to mirror- at least for the beginning- the user defined as root in your AWS installation.  See my notes HERE on that- that is defined in the advanced settings of the EC2 instance.


BEGIN CONFIG FILE:   (or click to download) *Download is properly indented

# Define user: this should be the same user as the AWS root user!
run_as jess;
# Define port and where the home (/) directory is
# Define site_dir/log_dir - these are the defaults
listen 3838;
location / {
site_dir /srv/shiny-server;
log_dir /var/log/shiny-server;
directory_index on;


Well, the proof is in the pudding.   At least for now, you can access a basic app I made that cleans csv field data files that where entered into excel by hand.  They start full of missing fields and have a weird two-column setup for distance- the app cleans all these issues and returns a 4 column (from 5 column) csv.

Download the test file here:   2012_dirt_PCD-git

And access the app here:  Basic Shiny app on AWS!

Below is an iFrame into the app, just to show how very basic it is.  Give it a go!


Off-Grid File Sharing with SAMBA / GL.iNet

Note:  SMB / SharePoint is surely better with a proper server/computer.  A Raspberry Pi running OpenMediaVault (Debian) is a more common and robust option (still 5v low power).

If you are actually in an "it must done in OpenWRT" scenario, Click Here for my Samba config file: OpenWRT_Samba-config and see below.  Also, please use a NTFS or EX4 format.  🙂


...While my sharing method wasn't actually adopted by others, I still think it is good to know!


How to Query KML point data as CSV using QGIS and R

How to Query KML point data as CSV using QGIS and R

Here you can see more than 800 points, each describing an observation of an individual bird.  This data is in the form of KML, a sort of XML document from Google for spatial data.


I want to know which points have “pair” or “female” in the description text nodes using R.  This way, I can quickly make and update a .csv in Excel of only the paired birds (based on color bands).



Even if there was a description string search function in Google Earth Pro (or other organization-centric GIS/waypoint software), this method is more

robust, as I can work immediately with the output as a data frame in R, rather than a list of results.


First, open an instance of QGIS.  I am running ~2.8 on OSX.  Add a vector layer of your KML.

“Command-A” in the point dialog to select all before import!

Next, under “Vector”, select “Merge vector layers” via Data Management Tools.


Select CSV and elect to save the file instead of use a temporary/scratch file (this is a common error).

Open your csv in Excel for verification! 







The R bit:

# query for paired birds

#EDIT:  Libraries

data <- data.frame(fread("Bird_CSV.csv"))

pair_rows <- contains("pair", vars = data$description)

fem_rows <- contains("fem", vars = data$description)

result <- combine(pair_rows, fem_rows)

result <- data[result,]

write_csv(result, "Paired_Birds.csv")











Visual NH Research Update :)

Olive-sided Flycatcher (Is Says, "Three Beers!! [please]"

Walking to work in the morning


Song Sparrow

Common Yellowthroat Warbler

I'll let the photos to the talking:  welcome to my world! 🙂 !!!!





Solar upgrades!

Solar upgrades!

Incredibly, the hut we are working from actually had another solar panel just laying around.  🙂
This 50w square panel had a junction box with MC4 connectors, the standard for small scale solar installations.  As I was unsure how to know when we are running low on electricity reserves, I decided to make some adjustments.

Additional 50w solar panel

(Everything is still solder, hot glue, alligator clips, and zip-ties I’m afraid…)
I traded my NEMA / USA two-prong connection with two MC4 splitters, such that both panels can run in parallel (into a standard USA 110v extension cord that goes into our hut).  This way we should make well over one of the two 35ah batteries-worth of electricity a day.

Dual MC4 splitters to extension cord

I also added a cheap 12v battery level indicator.  It is not very accurate (as it fluctuates with solar input) but it does give us some insight about how much "juice" we have available.  (I also wired and glued the remote-on switch to the back of the input for stability.)

Added battery indicator and button


Research Year Two: Three Photos

Male Common Yellowthroat Warbler

The field season has officially started in Northern NH!

Male Common Yellowthroat warbler (COYE):   This fellow is defending a small territory in a patch of open thicket.   These warblers rely on early succession forest- patches of substrate that haven't  really grown in yet- to build cryptic, ground-level nests.  They develop complex systems to divert/confuse predators away from their nests.


Female Black-throated Blue Warbler (BTBW):  I was lucky to see this female.   She is paired with a male who defends a large mature forest territory.   They have quite a few BTBW neighbors, which makes for a lot of skirmishes among the males over land.  The females are often silent and move very fast...

Male Mourning Warbler (MOWA):  This is a rare bird here.   Even more amazing, it is defending a territory in our research site- and trying to chase out a male COYE while doing so.  The two species "share" resources, which means thy can't stand each other.   🙂   Each time the male COYE sings near the MOWA, it gets berated and chased away- and vice versa.   It appears the COYE isn't budging either, probably because it hasn't had this domestic, neighborly problem before.


Female Black-throated Blue Warbler

Male Mourning Warbler

Gathering point data using Compass 55 on Apple iOS

Keeping track of birds is tricky!

Attached is our team's workflow with Compass 55.    From the Kml, we go into Google Earth Pro - ArcGIS Desktop (arcmap).   QGIS is sometimes used too.





« Older posts Newer posts »