Trans Scend Survival

Trans: Latin prefix implying "across" or "Beyond", often used in gender nonconforming situations – Scend: Archaic word describing a strong "surge" or "wave", originating with 15th century english sailors – Survival: 15th century english compound word describing an existence only worth transcending.

Category: Featured (page 1 of 9)

Warbler Trillers of the Charles

The first ones to arrive in MA, brush up!

 

Palm warbler

songs

 

This is usually the first one to arrive.  Gold bird, medium sized warbler, rufus hat.  When they arrive in MA they are often found lower than usual / on the ground looking for anything they can munch on.  Song is a rapid trill. More “musical / pleasant” than a fast chipping sparrow, faster than many Junco trills.

 

Pine warbler

songs

 

Slimmer than palm, no hat, very slim beak, has streaks on the breast usually.  Also a triller. They remain higher in the trees on arrival.

 

Yellow-rumped warbler

songs

 

Spectacular bird, if it has arrived you can’t miss it- also they will arrive by the dozen so worth waiting for a good visual.  These also trill, which is another reason it is good to get a visual. The trill is slow, very “sing-song”, and has a downward inflection at the end.  If there are a bunch sticking around for the summer, try to watch some sing- soon enough you will be able to pick out this trill from the others.

 

Yellow warbler says “sweet sweet sweet, I’m so Sweet!” and can get a bit confusing with Yellow-rumped warbler

Chestnut-sided warbler says “very very pleased to meet ya!” and can get a bit confusing with Yellow warbler

 

Black-and-white warbler

songs

 

Looks like a zebra – always acts like a nuthatch (clings to trunk and branches).  This one trills like a rusty wheel. It can easily be distinguished after a bit of birding with some around.  

 

American Redstart

songs

 

Adult males look like a late 50’s hot-rodded American muscle car: long, low, two tone paint job.  Matte/luster black with flame accents. Can’t miss it. The females and young males are buff (chrome, to keep in style I guess) with yellow accents.  Look for behavior- if a “female” is getting beaten up while trying to sing a song in the same area, it is actually a first year male failing to establish a territory due to obviously being a youth.

 

Cheers,

– Jess

Soundcloud Stream

Notes on a Free and Open Source Notes App:  Joplin

Joplin for all your Operating Systems and devices

As a lifelong IOS + OSX user (Apple products), I have used many, many notes apps over the years.  From big name apps like OmniFocus, Things 3, Notes+, to all the usual suspects like Trello, Notability, Notemaster, RTM, and others, I always eventually migrate back to Apple notes, simply because it is always available and always up to date.  There are zero “features” besides this convenience, which is why I am perpetually willing to give a new app a spin.

Joplin is free, open source, and works on OSX, Windows, Linux operating systems and IOS and Android phones.  

Find it here:

https://joplin.cozic.net/

brew install joplin 

The most important thing this project has nailed is cloud support and syncing.  I have my iPhone and computers syncing via Dropbox, which is easy to setup and works….  really well. Joplin folks have added many cloud options, so this is unlikely to be a sticking point for users.

Here are some of the key features:

  • Markdown is totally supported for straightforward and easy formatting
  • External editor support for emacs / atom / etc folks
  • Layout is clean, uncluttered, and just makes sense
  • Built-in markdown text editor and viewer is great
  • Notebook, todo, note, and tags work great across platforms
  • Browser integration, E2EE security, file attachments, and geolocation included

Hopefully this will be helpful.

Cheers,

– Jess

Musings On Chapel Language and Parallel Processing

View below the readme mirror from my Github repo. Scroll down for my Python3 evaluation script.

….Or visit the page directly: https://github.com/Jesssullivan/ChapelTests 

ChapelTests

Investigating modern concurrent programming ideas with Chapel Language and Python 3

Repo in light of PSU OS course 🙂

Test FileCheck.chpl from this repo:


git clone https://github.com/Jesssullivan/ChapelTests cd ChapelTests/FileChecking-with-Chapel # compile fastest / most up to date script: chpl FileCheck2.chpl # not annotated / no extra --args # compile all options (old sync method): chpl FileCheck.chpl # evaluate 5 different run times: python3 Timer_FileCheck.py

These two FileCheck scripts provide both parallel and serial methods for recursive duplicate file finding in Cray’s Chapel Language. All solutions will be “slow”, as they are fundamentally limited by disk speed.

Revision 2 uses standard sync$ variable form.

Use Timer_FileCheck.py to evaluate completion times for all Serial and parallel options. Go to /ChapelTesting-Python3/ for more information on these tests.

To run:

# In Parallel:
chpl FileCheck.chpl && ./FileCheck
# or:
chpl FileCheck2.chpl && ./FileCheck2

Dealing with Dupes in Chapel

Generate three text docs:

  • Same size, same file another
  • Same size, different file
  • Same size, less than 8 bytes

Please see the python3 evaluation scripts to run these options in a loop.

--Flags:

example:

./FileCheck --V --T --debug

...Will run FileCheck with internal timers(--T), which will be displayed with the verbose logs(--V) and all extra debug logging(--debug) from within each loop.

All config --Flags:

// serial options:
config const SE : bool=false; // use serial evaluation?
config const SP : bool=false; // use findfiles() as mastserDom method?

// logging options
config const V : bool=true; // Vebose output of actions?
config const debug : bool=false;  // enable verbose logging from within loops?
config const T : bool=true; // use internal Chapel timers?
config const R : bool=true; // compile report file?

// file options
config const dir = "."; // start here?
config const ext = ".txt";  // use alternative ext?
config const SAME = "SAME";  // default name ID?
config const DIFF = "DIFF"; // default name ID?

General notes:

From inside FileCheck2.chpl on updated sync$ syntax:

//  Chapel-Language  //

module Fs {
  var MasterDom = {("", "")};  // contains same size files as (a,b).
  var same = {("", "")};  // identical files
  var diff = {("", "")};  // sorted files flagged as same size but are not identical
  var sizeZero = {("", "")}; // sort files that are < 8 bytes
}

var sync1$ : sync bool;
sync1$ = true;

proc ParallelRun(a,b) {
  if exists(a) && exists(b) && a != b {
    if isFile(a) && isFile(b) {
      if getFileSize(a) == getFileSize(b) {
        sync1$;
        Fs.MasterDom += (a,b);
        sync1$ = true;
        }
        if getFileSize(a) < 8 && getFileSize(b) < 8 {
          sync1$;
          Fs.sizeZero += (a,b);
          sync1$ = true;
      }
    }
  }
}
coforall folder in walkdirs(".") {
  for a in findfiles(folder, recursive=false) {
    for b in findfiles(folder, recursive=false) {
      ParallelRun(a,b);
    }
  }
}

Get some Chapel:

In a (bash) shell, install Chapel:
Mac or Linux here, others refer to:

https://chapel-lang.org/docs/usingchapel/QUICKSTART.html

# For Linux bash:
git clone https://github.com/chapel-lang/chapel
tar xzf chapel-1.18.0.tar.gz
cd chapel-1.18.0
source util/setchplenv.bash
make
make check

#For Mac OSX bash:
# Just use homebrew
brew install chapel # :)

Get atom editor for Chapel Language support:

#Linux bash:
cd
sudo apt-get install atom
apm install language-chapel
# atom [yourfile.chpl]  # open/make a file with atom

# Mac OSX (download):
# https://github.com/atom/atom
# bash for Chapel language support
apm install language-chapel
# atom [yourfile.chpl]  # open/make a file with atom

Using the Chapel compiler

To compile with Chapel:

chpl MyFile.chpl # chpl command is self sufficient

# chpl one file class into another:

chpl -M classFile runFile.chpl

# to run a Chapel file:
./runFile

Now Some Python3 Evaluation:

# Ajacent to compiled FileCheck.chpl binary:

python3 Timer_FileCheck.py

Timer_FileCheck.py will loop FileCheck and find the average times it takes to complete, with a variety of additional arguments to toggle parallel and serial operation. The iterations are:

ListOptions = [Default, Serial_SE, Serial_SP, Serial_SE_SP]
  • Default – full parallel

  • Serial evaluation (–SE) but parallel domain creation

  • Serial domain creation (–SP) but parallel evaluation

  • Full serial (–SE –SP)

Output is saved as Time_FileCheck_Results.txt

  • Output is also logged after each of the (default 10) loops.

The idea is to evaluate a “–flag” -in this case, Serial or Parallel in FileCheck.chpl- to see of there are time benefits to parallel processing. In this case, there really are not any, because that program relies mostly on disk speed.

Evaluation Test:

# Time_FileCheck.py
#
# A WIP by Jess Sullivan
#
# evaluate average run speed of both serial and parallel versions
# of FileCheck.chpl  --  NOTE: coforall is used in both BY DEFAULT.
# This is to bypass the slow findfiles() method by dividing file searches
# by number of directories.

import subprocess
import time

File = "./FileCheck" # chapel to run

# default false, use for evaluation
SE = "--SE=true"

# default false, use for evaluation
SP = "--SP=true" # no coforall looping anywhere

# default true, make it false:
R = "--R=false"  #  do not let chapel compile a report per run

# default true, make it false:
T = "--T=false" # no internal chapel timers

# default true, make it false:
V = "--V=false"  #  use verbose logging?

# default is false
bug = "--debug=false"

Default = (File, R, T, V, bug) # default parallel operation
Serial_SE = (File, R, T, V, bug, SE)
Serial_SP = (File, R, T, V, bug, SP)
Serial_SE_SP = (File, R, T, V, bug, SP, SE)


ListOptions = [Default, Serial_SE, Serial_SP, Serial_SE_SP]

loopNum = 10 # iterations of each runTime for an average speed.

# setup output file
file = open("Time_FileCheck_Results.txt", "w")

file.write(str('eval ' + str(loopNum) + ' loops for ' + str(len(ListOptions)) + ' FileCheck Options' + "\n\\"))

def iterateWithArgs(loops, args, runTime):
    for l in range(loops):
        start = time.time()
        subprocess.run(args)
        end = time.time()
        runTime.append(end-start)

for option in ListOptions:
    runTime = []
    iterateWithArgs(loopNum, option, runTime)
    file.write("average runTime for FileCheck with "+ str(option) + "options is " + "\n\\")
    file.write(str(sum(runTime) / loopNum) +"\n\\")
    print("average runTime for FileCheck with " + str(option) + " options is " + "\n\\")
    print(str(sum(runTime) / loopNum) +"\n\\")

file.close()

GDAL for GIS on UNIX: Using a Mac (or better, Linux)

https://github.com/Jesssullivan/GIS_Shortcuts

Bash is great: It is a shell scripting language that is baked into every new Apple computer and many, many Linux distributions too. It is fun to learn and easy to find your “insert repetitive task here” to Bash translation on Stack Exchange, just a search away. As you uses the command prompt more and more for increasingly cool GIS tasks- be it for Python QGIS OR ESRI, R language for data cleaning and analysis, or just because you noticed you can get “mad amounts of work done” at increasingly faster rates- your vocabulary in the UNIX shell and bash will naturally grow.

I wanted to make a GIS post for Mac OS because it is both under-represented (for great reasons) in GIS and arguably the number 1 choice for any discerning consumers of computer hardware.

Many Linux OS options are faster for much of this “UNIX for GIS”, as quite a few of the things we need are already included with many Debian / Ubuntu distros, and come forms that have been stable for a long, long time.

If you are looking to setup a system primarily for GIS / data science (disregarding ESRI of course), See my initial notes on the Ubuntu variant Pop_OS by System76. If you like the ChromeOS vibe for multitasking and simplicity and the familiarity of Mac OS, it is a keeper (and also a sleeper, seeing how many folks are still on Windows for GIS…….).

Open Terminal: Bash for GIS

(The less you have in your way on your screen, the faster you can go!) xD

Note: in my opinion, homebrew and macPorts are good ideas- try them! If you don’t have it, get it now:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

(….However, port or brew installing QGIS and GDAL (primarily surrounding the delicate links between QGIS / GDAL / Python 2 & 3 / OSX local paths) can cause baffling issues. If possible, don’t do that. Use QGIS installers from the official site and build from source!)

if you need to resolve issues with your GDAL packages via removal: on MacPorts, try similar:

sudo port uninstall qgis py37-gdal

# on homebrew, list then remove (follow its instructions):

brew list
brew uninstall gdal geos gdal2  

!!! NOTE: I am investigating more reliable built-from-source solutions for gdal on mac.

Really!

There are numerous issues with brew-installed gdal. Those I have run into include:

  • linking issues with the crucial directory “gdal-data” (libraries)
  • linking issues Python bindings and python 2 vs. 3 getting confused
  • internal raster library conflicts against the gdal requirements
  • Proj.4 inconsistencies (see source notes below)
  • OSX Framework conflicts with source / brew / port (http://www.kyngchaos.com/software/frameworks/)
  • Linking conflicts with old, qgis default / LTR libraries against new ones
  • Major KML discrepancies: expat standard vs libkml.
brew install gdal
#
# brew install qgis can work in a pinch too.  At least you can unbrew it!
#

Next, assuming your GDAL is not broken (on Mac OS this is rare and considered a miracle)

# double check CLI is working:
gdalinfo --version
# “GDAL 2.4.0, released 2018/12/14”
gdal_merge.py
# list of args

To begin: Lets try a recent assignment that relies on the ESRI mosaic system and lots of GUI and clicking but use GDAL instead.

Data source: ftp://ftp.granit.sr.unh.edu/pub/GRANIT_Data/Vector_Data/Elevation_and_Derived_Products/d-elevationdem/d-10m/

!! Warning! These files are not projected in a way ESRI or GDAL understands. They WILL NOT HAVE A LOCATION IN QGIS. They will, however, satisfy the needs of my assignment.

In Terminal:

# wget on mac is great.  This tool (default on linux) lets us grab GIS data from
# most providers, via FTP and similar protocols.
brew install wget

make some folders

mkdir GIS_Projects && cd GIS_Projects

use wget to download every .dem file (-A .dem) from the specified folder and sub-folders (-r)

wget -r -A .dem ftp://ftp.granit.sr.unh.edu/pub/GRANIT_Data/Vector_Data/Elevation_and_Derived_Products/d-elevationdem/

cd ftp.granit.sr.unh.edu/pub/GRANIT_Data/Vector_Data/Elevation_and_Derived_Products/d-elevationdem

make an index file of only .dem files.
(If we needed to download other files and keep them from our wget (more common) this way we can still sort the various files for .dem)

ls -1 *.dem > dem_list.txt

use gdal to make state-plane referenced “Output_merged.tif” from the list of files in the index we made. it will use a single generic “0 0 255” band to show gradient.

gdal_merge.py -init "0 0 255" -o Output_Merged.tif --optfile dem_list.txt

copy the resulting file to desktop, then return home

cp Output_Merged.tif ~/desktop && cd

if you want (recommended):

rm -rf GIS_Projects  # remove .dem files.  Some are huge!

In Finder at in ~/desktop, open the new file with QGIS. A normal photo viewer will NOT show any detail.

Need to make something like this a reusable script? In Terminal, just a few extra steps:

mkdir GIS_Scripts && cd GIS_Scripts

open an editor + filename. Nano is generally pre-installed on OSX.

nano GDAL_LiveMerge.sh

COPY + PASTE THE SCRIPT FROM ABOVE INTO THE WINDOW

  • ctrl+X , then Y for yes

make your file runnable:

chmod u+x GDAL_LiveMerge.sh

run with ./

./GDAL_LiveMerge.sh

open GIS_Projects you can now copy + paste your script anywhere you want and run it there. scripts like this should not be exported to your global path / bashrc and will only work if they are in the directory you are calling them: If you need a global script, there are plenty of ways to do that too.

Notes on source-

BUILD FROM SOURCE:

Please understand this is probably far from the last post you’ll need before you flex your GDAL powers.

// If it works in less the 6 tries you win //

Back in Terminal:

get some tools we need, mostly for the step configuring proj.4 from github each of these are used in this method of installation, but much of each package will be ignored

brew install autoconf automake wget postgresql libiconv
brew link --force libiconv
# this *may* need to be linked as well
brew link libxml2 --force  #

Setup your home directory- go home

cd
# get proj.4 from the gdal people first.
git clone https://github.com/OSGeo/proj.4
#
./autogen.sh
./configure
#
# Make takes ~1-10 min on
#
sudo make
sudo make install
cd
#

get gdal, repeat steps # ./autogen.sh is usually not needed for gdal but to take no chances, run it anyway as they also include in in the repo….?

git clone https://github.com/OSGeo/gdal
cd gdal/gdal
#
./autogen.sh
#
./configure

some configure ideas:

 --with-python --with-pcraster=internal --with-libjson-c=internal --with-jpeg=internal --with-geotiff=internal --with-libtiff=internal
#
# Use a prefix to make an attempt at sandboxing the gdal install.  This may be required.
#
# More ideas:
#
--with-curl=/usr/local/bin/curl LDFLAGS="-L/usr/local/opt/libiconv/lib" LDFLAGS="-L/usr/local/opt/libxml2/lib" CPPFLAGS="-I/usr/local/opt/libxml2/include" LDFLAGS="-L/usr/local/opt/libiconv/lib" LDFLAGS="-L/usr/local/Cellar//openssl/1.0.2q/lib/"
sudo make
sudo make install

🙂

« Older posts