Trans Scend Survival

Trans: Latin prefix implying "across" or "Beyond", often used in gender nonconforming situations – Scend: Archaic word describing a strong "surge" or "wave", originating with 15th century english sailors – Survival: 15th century english compound word describing an existence only worth transcending.

Page 2 of 21

Small Updates to GDAL Notes

GIS Shortcuts


GDAL on OSX, Ubuntu & Windows WSL

Bash is great: It is a shell scripting language that is baked into every new Apple computer and many, many Linux distributions too. It is fun to learn and easy to find your “insert repetitive task here” to Bash translation on Stack Exchange, just a search away. As you uses the command prompt more and more for increasingly cool GIS tasks- be it for Python QGIS OR ESRI, R language for data cleaning and analysis, or just because you noticed you can get “mad amounts of work done” at increasingly faster rates- your vocabulary in the UNIX shell and bash will naturally grow.

I wanted to make a GIS post for Mac OS because it is both under-represented (for great reasons) in GIS and arguably the number 1 choice for any discerning consumers of computer hardware.

Many Linux OS options are faster for much of this “UNIX for GIS”, as quite a few of the things we need are already included with many Debian / Ubuntu distros, and come forms that have been stable for a long, long time.

If you are looking to setup a system primarily for GIS / data science (disregarding ESRI of course), See my initial notes on the Ubuntu variant Pop_OS by System76. If you like the ChromeOS vibe for multitasking and simplicity and the familiarity of Mac OS, it is a keeper (and also a sleeper, seeing how many folks are still on Windows for GIS…….).

Open Terminal
(The less you have in your way on your screen, the faster you can go!) xD

Note: in my opinion, homebrew and macPorts are good ideas- try them! If you don’t have it, get it now:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

(….However, port or brew installing QGIS and GDAL (primarily surrounding the delicate links between QGIS / GDAL / Python 2 & 3 / OSX local paths) can cause baffling issues. If possible, don’t do that. Use QGIS installers from the official site and build from source!)

if you need to resolve issues with your GDAL packages via removal:
on MacPorts, try similar:

sudo port uninstall qgis py37-gdal

# on homebrew, list then remove (follow its instructions):

brew list
brew uninstall gdal geos gdal2  

!!! NOTE: I am investigating more reliable built-from-source solutions for gdal on mac.

Really!

There are numerous issues with brew-installed gdal. Those I have run into include:

  • linking issues with the crucial directory “gdal-data” (libraries)
  • linking issues Python bindings and python 2 vs. 3 getting confused
  • internal raster library conflicts against the gdal requirements
  • Proj.4 inconsistencies (see source notes below)
  • OSX Framework conflicts with source / brew / port (http://www.kyngchaos.com/software/frameworks/)
  • Linking conflicts with old, qgis default / LTR libraries against new ones
  • Major KML discrepancies: expat standard vs libkml.
    brew install gdal
    #
    # brew install qgis can work well too.  At least you can unbrew it!
    #

Next, assuming your GDAL is not broken (on Mac OS this is rare and considered a miracle):

# double check CLI is working:
gdalinfo --version
# “GDAL 2.4.0, released 2018/12/14”
gdal_merge.py
# list of args

Using Ubuntu GDAL on Windows w/ WSL

LINK: get the WSL shell from Microsoft

# In the WSL shell:

sudo apt-get install python3.6-dev -y
sudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update
sudo apt-get install libgdal-dev -y
sudo apt-get install gdal-bin -y

# See here for more notes including Python bindings:
# https://mothergeo-py.readthedocs.io/en/latest/development/how-to/gdal-ubuntu-pkg.html

In a new Shell:

# Double check the shell does indeed have GDAL in $PATH:
gdalinfo --version

To begin- try a recent GIS assignment that relies on the ESRI mosaic system and lots of GUI and clicking but use GDAL instead.

Data source: ftp://ftp.granit.sr.unh.edu/pub/GRANIT_Data/Vector_Data/Elevation_and_Derived_Products/d-elevationdem/d-10m/

!! Warning! These files are not projected in a way ESRI or GDAL understands. They WILL NOT HAVE A LOCATION IN QGIS. They will, however, satisfy the needs of the assignment.

# wget on mac is great.  This tool (default on linux) lets us grab GIS data from
# most providers, via FTP and similar protocols.

brew install wget

make some folders

mkdir GIS_Projects && cd GIS_Projects

use wget to download every .dem file (-A .dem) from the specified folder and sub-folders (-r)

wget -r -A .dem ftp://ftp.granit.sr.unh.edu/pub/GRANIT_Data/Vector_Data/Elevation_and_Derived_Products/d-elevationdem/

cd ftp.granit.sr.unh.edu/pub/GRANIT_Data/Vector_Data/Elevation_and_Derived_Products/d-elevationdem

make an index file of only .dem files.
(If we needed to download other files and keep them from our wget (more common)
this way we can still sort the various files for .dem)

ls -1 *.dem > dem_list.txt

use gdal to make state-plane referenced “Output_merged.tif” from the list of files
in the index we made.
it will use a single generic "0 0 255" band to show gradient.

gdal_merge.py -init "0 0 255" -o Output_Merged.tif --optfile dem_list.txt

copy the resulting file to desktop, then return home

cp Output_Merged.tif ~/desktop && cd

if you want (recommended):

rm -rf GIS_Projects  # remove .dem files.  Some are huge!

In Finder at in ~/desktop, open the new file with QGIS. A normal photo viewer will NOT show any detail.

Need to make something like this a reusable script? In Terminal, just a few extra steps:

mkdir GIS_Scripts && cd GIS_Scripts

open an editor + filename. Nano is generally pre-installed on OSX.

nano GDAL_LiveMerge.sh

COPY + PASTE THE SCRIPT FROM ABOVE INTO THE WINDOW

  • ctrl+X , then Y for yes

make your file runnable:

chmod u+x GDAL_LiveMerge.sh

run with ./

./GDAL_LiveMerge.sh

You can now copy + paste your script anywhere you want and run it there. scripts like this should not be exported to your global path / bashrc and will only work if they are in the directory you are calling them: If you need a global script, there are plenty of ways to do that too.

See /Notes_GDAL/README.md for notes on building GDAL from source on OSX

Click here to visit this repo

🙂

Decentralized Pi Video Monitoring w/ motioneye & BATMAN

Visit the me here on Github
Added parabolic musings 10/16/19, see below

...On using motioneye video clients on Pi Zeros & Raspbian over a BATMAN-adv Ad-Hoc network

link: motioneyeos
link: motioneye Daemon
link: Pi Zero W Tx/Rx data sheet:
link: BATMAN Open Mesh

This implementation of motioneye is running on Raspbian Buster (opposed to motioneyeos).

Calculating Mesh Effectiveness w/ Python:
Please take a look at dBmLoss.py- the idea here is one should be able to estimate the maximum plausible distance between mesh nodes before setting anything up. It can be run with no arguments-

python3 dBmLoss.py

...with no arguments, it should use default values (Tx = 20 dBm, Rx = |-40| dBm) to print this:

you can add (default) Rx Tx arguments using the following syntax:
                 python3 dBmLoss.py 20 40
                 python3 dBmLoss.py <Rx> <Tx>                 

 57.74559999999994 ft = max. mesh node spacing, @
 Rx = 40
 Tx = 20

Regarding the Pi:
The Pi Zero uses an onboard BCM43143 wifi module. See above for the data sheet. We can expect around a ~19 dBm Tx signal from a BCM43143 if we are optimistic. Unfortunately, "usable" Rx gain is unclear in the context of the Pi.

Added 10/16/19:
Notes on generating an accurate parabolic antenna shape with FreeCAD’s Python CLI:

For whatever reason, (likely my own ignorance) I have been having trouble generating an accurate parabolic dish shape in Fusion 360 (AFAICT, Autodesk is literally drenching Fusion 360 in funds right now, I feel obligated to at least try). Bezier, spline, etc curves are not suitable!
If you are not familiar with FreeCAD, the general approach- geometry is formed through fully constraining sketches and objects- is quite different from Sketchup / Tinkercad / Inventor / etc, as most proprietary 3d software does the “constraining” of your drawings behind the scenes. From this perspective, you can see how the following script never actually defines or changes the curve / depth of the parabola; all we need to do is change how much curve to include. A wide, shallow dish can be made by only using the very bottom of the curve, or a deep / narrow dish by including more of the ever steepening parabolic shape.

import Part, math

# musings derived from:
# https://forum.freecadweb.org/viewtopic.php?t=4430

# thinking about units here:
tu = FreeCAD.Units.parseQuantity

def mm(value):
    return tu('{} mm'.format(value))

rs = mm(1.9)
thicken = -(rs / mm(15)) 

# defer to scale during fitting / fillet elsewhere 
m=App.Matrix()
m.rotateY(math.radians(-90))
# create a parabola with the symmetry axis (0,0,1)
parabola=Part.Parabola()
parabola.transform(m)

# get only the right part of the curve
edge=parabola.toShape(0,rs)
pt=parabola.value(rs)
line=Part.makeLine(pt,App.Vector(0,0,pt.z))
wire=Part.Wire([edge,line])
shell=wire.revolve(App.Vector(0,0,0),App.Vector(0,0,1),360)

# make a solid
solid=Part.Solid(shell)

# apply a thickness
thick=solid.makeThickness([solid.Faces[1]],thicken,0.001)
Part.show(thick)

Gui.SendMsgToActiveView("ViewFit")

"""
# Fill screen:
Gui.SendMsgToActiveView("ViewFit")
# Remove Part in default env:
App.getDocument("Unnamed1").removeObject("Shape")
"""

FWIW, here is my Python implimentation of a Tx/Rx "Free Space" distance calulator-
dBmLoss.py:

from math import log10
from sys import argv
'''
# estimate free space dBm attenuation:
# ...using wfi module BCM43143:

Tx = 19~20 dBm
Rx = not clear how low we can go here

d = distance Tx --> Rx
f = frequency
c = attenuation constant: meters / MHz = -27.55; see here for more info:
https://en.wikipedia.org/wiki/Free-space_path_loss
'''

f = 2400  # MHz
c = 27.55 # RF attenuation constant (in meters / MHz)

def_Tx = 20  # expected dBm transmit
def_Rx = 40  # (absolute value) of negative dBm thesh

def logdBm(num):
    return 20 * log10(num)

def maxDist(Rx, Tx):
    dBm = 0
    d = .1  # meters!
    while dBm < Tx + Rx:
        dBm = logdBm(d) + logdBm(f) - Tx - Rx + c
        d += .1  # meters!
    return d

# Why not use this with arguments Tx + Rx from shell if we want:
def useargs():
    use = bool
    try:
        if len(argv) == 3:
            use = True
        elif len(argv) == 1:
            print('\n\nyou can add (default) Rx Tx arguments using the following syntax: \n \
                python3 dBmLoss.py 20 40 \n \
                python3 dBmLoss.py <Rx> <Tx> \
                \n')
            use = False
        else:
            print('you must use both Rx & Tx arguments or no arguments')
            raise SystemExit
    except:
        print('you must use both Rx & Tx arguments or no arguments')
        raise SystemExit
    return use

def main():

    if useargs() == True:
        arg = [int(argv[1]), int(argv[2])]
    else:
        arg = [def_Rx, def_Tx]

    print(str('\n ' + str(maxDist(arg[0], arg[1])*3.281) + \
        ' ft = max. mesh node spacing, @ \n' + \
        ' Rx = ' + str(arg[0]) + '\n' + \
        ' Tx = ' + str(arg[1])))

main()

Google Calendar API with Chapel & Python

Mirroring the repo here:

Google Calendar API with Chapel & Python

Despite Chapel's many quirks and annoyances surrounding string handling, its efficiency and ease of running in parallel are always welcome. The idea here is a Chapel script that may need to weed through enormous numbers of files while looking for a date tag ($D + other tags currently) is probably a better choice overall than a pure Python version. (the intent is to test this properly later.)

As of 9/19/19, there is still a laundry list of things to add- control flow (for instance, “don't add the event over and over”) less brittle syntax, annotations, actual error handling, etc. It does find and upload calendar entries though!

I am using Python with the Google Calendar API (see here: https://developers.google.com/calendar/v3/reference/) in a looping Daemon thread. All the sifting for tags is managed with the Chapel binary, which dumps anything it finds into a csv from which the daemon will push calendar entries with proper formatting. FWIW, Google’s dates (datetime.datetime) adhere to RFC3339 (https://tools.ietf.org/html/rfc3339) which conveniently is the default of the datetime.isoformat() method.

Some pesky things to keep in mind:

This script uses a sync$ variable to lock other threads out of an evaluation during concurrency. So far I think the easiest way to manage the resulting domain is from within a module like so:

module charMatches {
  var dates : domain(string);
}

Here, domain charMatches.dates will need to accessed as a reference variable from any procedures that need it.

proc dateCheck(aFile, ref choice) {
    ...
}
... 
coforall folder in walkdirs('check/') {
    for file in findfiles(folder) {
        dateCheck(file, charMatches.dates);
    }
}

errors like:

error: unresolved call '_ir_split__ref_string.size'

unresolved call 'norm(promoted expression)'

...or other variants of:
string.split().size  (length, etc)

...Tie into a Chapel Specification issue.

https://github.com/chapel-lang/chapel/issues/7982

The short solution is do not use .split; instead, I have been chopping strings with .partition().

// like so:
...
if choice.contains(line.partition(hSep)[3].partition(hTerminate)[1]) == false {
    ...process string...
    ...
}

Canon: Conquered

Over 14 years ago, Lim Jeong-hyun posted a cover of Pachelbel's Canon in South Korea, where it eventually filtered through the inter-webnet of old into Youtube.

I saw this video in its infancy, and soon began to learn guitar too.

5 years later I recorded my first cover of Pachelbel's Canon.

Another 5 years passed (2016 at this point) before I began recording a second cover, posting this muse and this percolation on the subject.   While most attempts were futile....

...I made it through the whole song by take # 1,394  (2017 at this point):

Some more years go by and here we are today:
xD

-Jess

Persistent, Live Ubuntu for College

Below is are live mirrors of my "PSU Hacking Club" Ubuntu repos.


https://github.com/psu-hacking/iso-gen


https://psuhacking.club/

https://github.com/psu-hacking/static-site

Simple File Hosting


README yet to be updated with Makerspace-specific formatting


Static site built quickly with Hugo CLI

# on OSX
# get hugo

brew install hugo

# clone site

git clone https://github.com/psu-hacking/static-site
cd static-site

# Compile and compress public directory

hugo
zip -r site-archive.zip public

# upload and host with sftp & ssh

sftp user@yoursite.net
> cd yoursite.net
> put site-archive.zip

# new terminal window

ssh user@yoursite.net
# check your remote filesystem- the idea is:
> unzip site-archive.zip
> rm -rf yoursite.net/site-archive.zip

visit us


Also, check out the evolving PSU Hacking Club wiki here!

xD  - Jess

« Older posts Newer posts »