Trans: Latin prefix implying "across" or "Beyond", often used in gender nonconforming situations – Scend: Archaic word describing a strong "surge" or "wave", originating with 15th century english sailors – Survival: 15th century english compound word describing an existence only worth transcending.
View the tools here: http://kml.jessdev.org
Three of my KML tools are now stable and in Github. These are actually displayed via the static site generator Hugo (read about the Hugo CLI here), which is sitting in the shiny server (port 3838) next to the apps. Messy, but it will do for now.
https://github.com/Jesssullivan/Shiny-Apps
-Jess
New Shiny App specific Repo now live…
https://github.com/Jesssullivan/Shiny-Apps
With KML Search and Convert now fully functional (along with the tiny app “clean”) , live shiny apps of mine now have a repo of their own. Check it out!
-Jess
Written in R; using GDAL/EXPAT libraries on Ubuntu and hosted with AWS EC2.
New App: KML Search and Convert
Here is an simple (beta) app of mine that converts KML files into Excel-friendly CSV documents. It also has a search function, so you can download a subset of data that contains keywords. 🙂
The files will soon be available in Github.
I’m still working on a progress indicator; it currently lets you download before it is done processing. Know a completely processed file is titled with “kml2csv_<yourfile>.csv”.
…YMMV. xD
GDAL for R Server on Red Hat Xenial Ubuntu – KML Spatial Libraries and More
If you made the (possible mistake) of running with a barebones Red Hat Linux instance, you will find it is missing many things you may want in R. I rely on GDAL (the definitive Geospatial Data Abstraction Library) on my local OSX R setup, and want it on my server too. GDAL contains many libraries you need to work with KML, RGDAL, and other spatial packages. It is massive and usually take a long time to sort out on any machine.
These notes assume you are already involved with a R server (usually port 8787 in a browser). I am running mine from an EC2 instance with AWS.
! Note this is a fresh server install, using Ubuntu; I messed up my original ones while trying to configure GDAL against conflicting packages. If you are creating a new one, opt for at least a T2 medium (or go bigger) and find the latest Ubuntu server AMI. For these instructions, you want an OS that is as generic as possible.
On Github:
https://github.com/Jesssullivan/rhel-bits
From Bash:
# SSH into the EC2 instance: (here is the syntax just in case)
#ssh -i “/Users/YourSSHKey.pem” ec2-user@yourAWSinstance.amazonaws.com
sudo su –
apt-get update
apt-get upgrade
nano /etc/apt/sources.list
#enter as a new line at the bottom of the doc:
deb https://cloud.r-project.org/bin/linux/ubuntu xenial/
#exit nano
wget https://raw.githubusercontent.com/Jesssullivan/rhel-bits/master/xen-conf.sh
chmod 777 xen-conf.sh
./xen-conf.sh
Or…
From SSH:
# SSH into the EC2 instance: (here is the syntax just in case)
ssh -i “/Users/YourSSHKey.pem” ec2-user@yourAWSinstance.amazonaws.com
# if you can, become root and make some global users- these will be your access to
# RStudio Server and shiny too!
sudo su –
adduser <Jess>
# Follow the following prompts carefully to create the user
apt-get update
nano /etc/apt/sources.list
# enter as a new line at the bottom of the doc:
deb https://cloud.r-project.org/bin/linux/ubuntu xenial/
# exit nano
# Start, or try bash:
apt-get install r-base
apt-get install r-base-dev
apt-get update
apt-get upgrade
wget http://download.osgeo.org/gdal/2.3.1/gdal-2.3.1.tar.gz
tar xvf gdal-2.3.1.tar.gz
cd gdal-2.3.1
# begin making GDAL: this all takes a while
./configure [if your need proper kml support (like me), search on configuring with expat or libkml. There are many more options for configuration based on other packages that can go here, and this is the step to get them in order…]
sudo make
sudo make install
cd # Try entering R now and check the version!
# Start installing RStudio server and Shiny
apt-get update
apt-get upgrade
sudo apt-get install gdebi-core
wget https://download2.rstudio.org/rstudio-server-1.1.456-amd64.deb
sudo gdebi rstudio-server-1.1.456-amd64.deb
# Enter R or go to the graphical R Studio installation in your browser
R
# Authenticate if using the graphical interface using the usr:pwd you defined earlier
# this will take a long time
install.packages(“rgdal”)
# Note any errors carefully!
Then:
install.packages(“dplyr”)
install.packages(c(“data.table”, “tidyverse”, “shiny”) # etc
-Jess
Extras:
##Later, ONLY IF you NEED Anaconda, FYI:
# Get Anaconda: this is a large package manager, and is could be used for patching up missing # dependencies:
#Use “ls” followed by rm -r <anaconda> (fill in with ls results) to remove conflicting conda
# installers if you have any issue there, I am starting fresh:
mkdir binconda
# *making a weak attempt at sandboxing the massive new package manager installation*
cd binconda
wget http://repo.continuum.io/archive/Anaconda2-4.3.0-Linux-x86_64.sh
# install and follow the prompts
bash Anaconda2-5.2.0-Linux-x86_64.sh
# Close the terminal window completely and start a new one, and ssh back to where you left
# off. Conda install requires this.
# open and SSH back into your instance. You should now have either additional flexibility in
# either patching holes in dependencies, or created some large holes in your server. YMMV.
### Done
Red Hat stuff:
Follow these AWS instructions if you are doing something else:
https://aws.amazon.com/blogs/big-data/running-r-on-aws/
See my notes on this here:
https://www.transscendsurvival.org/2018/03/08/how-to-make-a-aws-r-server/
and notes on Shiny server:
https://www.transscendsurvival.org/2018/07/16/deploy-a-shiny-web-app-in-r-using-aws-ec2-red-hat/
GDAL on Red Hat:- Existing threads on this:
https://gis.stackexchange.com/questions/120101/building-gdal-with-libkml-support/120103#120103
This is a nice short thread about building from source:
https://gis.stackexchange.com/questions/263495/how-to-install-gdal-on-centos-7-4
neat RPM package finding tool, just in case:
https://rpmfind.net/linux/rpm2html/
Info on the LIBKML driver if you end up with issues there:
http://www.gdal.org/drv_libkml.html
I hope this is useful- GDAL is important and best to set it up early. It will be a pain, but so is losing work while trying to patch it in later. xD
-Jess
Info on deploying a Shiny web app in R using AWS (EC2 Redhat)
As a follow-up to my post on how to create an AWS RStudio server, the next logical step is to host some useful apps you created in R for people to use. A common way to do this is the R-specific tool Shiny, which is built in to RStudio. Learning the syntax to convert R code into a Shiny app is rather subtle, and can be hard. I plan to do a more thorough demo on this- particularly the use of the $ symbol, as in “input$output”- later. 🙂
It turns out hosting a Shiny Web app provides a large number of opportunities for things to go wrong…. I will share what worked for me. All of this info is accessed via SSH, to the server running Shiny and RStudio.
I am using the AWS “Linux 2” AMI, which is based on the Red Hat OS. For reference, here is some extremely important Red Hat CLI language worth being familiar with and debugging:
“sudo yum install” and “wget” are for fetching and installing things like shiny. Don’t bother with instructions that include “apt-get install”, as they are for a different Linux OS!
“sudo chmod -R 777” is how you change your directory permissions for read, write, and execute (all of those enabled). This is handy if your server disconnecting when the app tries to run something- it is a simple fix to a problem not always evident in the logs. The default root folder from which shiny apps are hosted and run is “/srv/shiny-server” (or just “/srv” to be safe).
“nano /var/log/shiny-server.log” is the location of current shiny logs.
“sudo stop shiny-server” followed by “sudo start shiny-server” is the best way to restart the server- “sudo restart shiny-server” is not a sure bet on any other process. It is true, other tools like a node.js server or nginx could impact the success of Shiny- If you think nginx is a problem, “cd /ect/nginx” followed by “ls” will get you in the right direction. Others have cited problems with Red Hat not including the directories and files at “/etc/nginx/sites-available”. You do not need these directories. (though they are probably important for other things).
“sudo rm -r” is a good way to destroy things, like a mangled R studio installation. Remember, it is easy enough to start again fresh! 🙂
“sudo nano /etc/shiny-server/shiny-server.conf” is how to access the config file for Shiny. The fresh install version I used did not work! There will be lots of excess in that file, much of which can causes issues in a bare-bones setup like mine. One important key is to ensure Shiny is using a root user- see my example file below. I am the root user here (jess)- change that to mirror- at least for the beginning- the user defined as root in your AWS installation. See my notes HERE on that- that is defined in the advanced settings of the EC2 instance.
BEGIN CONFIG FILE: (or click to download) *Download is properly indented
# Define user: this should be the same user as the AWS root user!
#
run_as jess;
#
# Define port and where the home (/) directory is
# Define site_dir/log_dir - these are the defaults
#
server{
listen 3838;
location / {
site_dir /srv/shiny-server;
log_dir /var/log/shiny-server;
directory_index on;
}
}
END CONFIG FILE
Well, the proof is in the pudding. At least for now, you can access a basic app I made that cleans csv field data files that where entered into excel by hand. They start full of missing fields and have a weird two-column setup for distance- the app cleans all these issues and returns a 4 column (from 5 column) csv.
Download the test file here: 2012_dirt_PCD-git
And access the app here: Basic Shiny app on AWS!
Below is an iFrame into the app, just to show how very basic it is. Give it a go!
-Jess
Note: SMB / SharePoint is surely better with a proper server/computer. A Raspberry Pi running OpenMediaVault (Debian) is a more common and robust option (still 5v low power).
If you are actually in an “it must done in OpenWRT” scenario, Click Here for my Samba config file: OpenWRT_Samba-config and see below. Also, please use a NTFS or EX4 format. 🙂
…While my sharing method wasn’t actually adopted by others, I still think it is good to know!
-Jess
How to Query KML point data as CSV using QGIS and R
Here you can see more than 800 points, each describing an observation of an individual bird. This data is in the form of KML, a sort of XML document from Google for spatial data.
I want to know which points have “pair” or “female” in the description text nodes using R. This way, I can quickly make and update a .csv in Excel of only the paired birds (based on color bands).
Even if there was a description string search function in Google Earth Pro (or other organization-centric GIS/waypoint software), this method is more
robust, as I can work immediately with the output as a data frame in R, rather than a list of results.
First, open an instance of QGIS. I am running ~2.8 on OSX. Add a vector layer of your KML.
“Command-A” in the point dialog to select all before import!
Next, under “Vector”, select “Merge vector layers” via Data Management Tools.
Select CSV and elect to save the file instead of use a temporary/scratch file (this is a common error).
Open your csv in Excel for verification! 
The R bit:
# query for paired birds
#EDIT: Libraries
library(data.table)
library(tidyverse)
data <- data.frame(fread("Bird_CSV.csv"))
pair_rows <- contains("pair", vars = data$description)
fem_rows <- contains("fem", vars = data$description)
result <- combine(pair_rows, fem_rows)
result <- data[result,]
write_csv(result, "Paired_Birds.csv")
Tada!
-Jess
Olive-sided Flycatcher (Is Says, “Three Beers!! [please]”
Walking to work in the morning
Work
Song Sparrow
Common Yellowthroat Warbler
I’ll let the photos to the talking: welcome to my world! 🙂 !!!!
-Jess
Solar upgrades!
Additional 50w solar panel
Dual MC4 splitters to extension cord
Added battery indicator and button
Male Common Yellowthroat Warbler
The field season has officially started in Northern NH!
Male Common Yellowthroat warbler (COYE): This fellow is defending a small territory in a patch of open thicket. These warblers rely on early succession forest- patches of substrate that haven’t really grown in yet- to build cryptic, ground-level nests. They develop complex systems to divert/confuse predators away from their nests.
Female Black-throated Blue Warbler (BTBW): I was lucky to see this female. She is paired with a male who defends a large mature forest territory. They have quite a few BTBW neighbors, which makes for a lot of skirmishes among the males over land. The females are often silent and move very fast…
Male Mourning Warbler (MOWA): This is a rare bird here. Even more amazing, it is defending a territory in our research site- and trying to chase out a male COYE while doing so. The two species “share” resources, which means thy can’t stand each other. 🙂 Each time the male COYE sings near the MOWA, it gets berated and chased away- and vice versa. It appears the COYE isn’t budging either, probably because it hasn’t had this domestic, neighborly problem before.
-Jess
Female Black-throated Blue Warbler
Male Mourning Warbler
Keeping track of birds is tricky!
Attached is our team’s workflow with Compass 55. From the Kml, we go into Google Earth Pro – ArcGIS Desktop (arcmap). QGIS is sometimes used too.
Cheers,
-Jess
Equipment used:
Inverter/PWM Controller: http://a.co/fdl9YzI
2x 35ah Batteries: http://a.co/5JBIxTC
100w solar panel: http://a.co/5JBIxTC
We need power! While doing bird research in the wilds of northern NH, it became evident we needed electricity to power computers, big cameras, and phones/GPS units.
Below is a table of the system and our expected electricity needs:
| System | Solar 100w | 35ah universal (x2) | ||
| Ah per day: | 33.33333333 | 35 | TOTAL Ah Reserve: | 70 |
| V | 12 | 12 | Parallel wiring: | 12v |
| Wh in: | 400 | 420 | TOTAL Wh Reserve: | 840 |
| W | 100 | |||
| Cost | $105.00 | $64.00 | ||
| ah/$ | 2 | |||
| Sun Hour / Multiplier | 4 | 2 | ||
| Need/Day | Wh | multiplier | consump. in Wh = | 259.36 |
| Computer | 100 | 2.5 | 250 | |
| iPhone | 1.7 | 2 | 3.4 | |
| AAs | 11.2 | 0.3 | 3.36 | |
| Camera | 2.6 | 1 | 2.6 |
*The milk crate system below can charge a 100 watt MacBook Pro around 8-9 times from being completely empty.
**Remember: V*A=W, W/V=A, and Watts over time is Wh.
-Jess
+/- relates to size of standard prongs
Parallel maintains 12v but doubles Ah. (Series would go to 24v at 35ah)
While somewhat of a historical entry:
Below are some of the few photos I took while birding on may 5, 2018- GLOBAL BIG DAY! Be sure to read about it here: https://ebird.org/news/global-big-day-5-may-2018
Myself and my father contributed 64 species, including the below Northern waterthrush, Female Rose-breasted Grosbeak, Northern Parula, and Common Yellowthroat warbler.
-Jess
Spring is (possibly) coming people! 🙂
Scituate, MA – Osprey
Scituate, MA – Killdeer
Plymouth, NH – (FOY photo for me) Hermit Thrush
Scituate, MA – Brant
Plymouth, NH – Savannah Sparrow
© 2024 Trans Scend Survival
α wιρ Σ ♥ by Jess Sullivan — Up ↑