Trans Scend Survival

Trans: Latin prefix implying "across" or "Beyond", often used in gender nonconforming situations – Scend: Archaic word describing a strong "surge" or "wave", originating with 15th century english sailors – Survival: 15th century english compound word describing an existence only worth transcending.

Category: Featured (page 1 of 10)

Some GDAL shell macros from R instead of rgdal

also here on github

it's not R sacrilege if nobody knows

Even the little stuff benefits from some organizational scripting, even if it’s just to catalog one’s actions. Here are some examples for common tasks.

Get all the source data into a R-friendly format like csv. ogr2ogr has a nifty option -lco GEOMETRY=AS_WKT (Well-Known-Text) to keep track of spatial data throughout abstractions- we can add the WKT as a cell until it is time to write the data out again.

# define a shapefile conversion to csv from system's shell:
sys_SHP2CSV <- function(shp) {
  csvfile <- paste0(shp, '.csv')
  shpfile <-paste0(shp, '.shp')
  if (!file.exists(csvfile)) {
    # use -lco GEOMETRY to maintain location
    # for reference, shp --> geojson would look like:
    # system('ogr2ogr -f geojson output.geojson input.shp')
    # keeps geometry as WKT:
    cmd <- paste('ogr2ogr -f CSV', csvfile, shpfile, '-lco GEOMETRY=AS_WKT')
    system(cmd)  # executes command
  } else {
    print(paste('output file already exists, please delete', csvfile, 'before converting again'))
  }
  return(csvfile)
}

Read the new csv into R:

# for file 'foo.shp':
foo_raw <- read.csv(sys_SHP2CSV(shp='foo'), sep = ',')

One might do any number of things now, some here lets snag some columns and rename them:

# rename the subset of data "foo" we want in a data.frame:
foo <- data.frame(foo_raw[1:5])
colnames(foo) <- c('bar', 'eggs', 'ham', 'hello', 'world')

We could do some more careful parsing too, here a semicolon in cell strings can be converted to a comma:

# replace ` ; ` to ` , ` in col "bar":
foo$bar <- gsub(pattern=";", replacement=",", foo$bar)

Do whatever you do for an output directory:

# make a output file directory if you're into that
# my preference is to only keep one set of output files per run
# here, we'd reset the directory before adding any new output files
redir <- function(outdir) {
  if (dir.exists(outdir)) {
    system(paste('rm -rf', outdir))
  }
  dir.create(outdir)
}

Of course, once your data is in R there are countless "R things" one could do...

# iterate to fill empty cells with preceding values
for (i in 1:length(foo[,1])) {
  if (nchar(foo$bar[i]) < 1) {
    foo$bar[i] <- foo$bar[i-1]
  }
  # fill incomplete rows with NA values:
  if (nchar(foo$bar[i]) < 1) {
    foo[i,] <- NA  
  }
}

# remove NA rows if there is nothing better to do:
newfoo <- na.omit(foo)

Even though this is totally adding a level of complexity to what could be a single ogr2ogr command, I've decided it is still worth it- I'd definitely rather keep track of everything I do over forget what I did.... xD

# make some methods to write out various kinds of files via gdal:
to_GEO <- function(target) {
  print(paste('converting', target, 'to geojson .... '))
  system(paste('ogr2ogr -f', " geojson ",  paste0(target, '.geojson'), paste0(target, '.csv')))
}

to_SHP <- function(target) {
  print(paste('converting ', target, ' to ESRI Shapefile .... '))
  system(paste('ogr2ogr -f', " 'ESRI Shapefile' ",  paste0(target, '.shp'), paste0(target, '.csv')))
}

# name files:
foo_name <- 'output_foo'

# for table data 'foo', first:
write.csv(foo, paste0(foo_name, '.csv'))

# convert with the above csv:
to_SHP(foo_name)

Cheers!
-Jess

Ubuntu on Captive Portal WiFi

wget https://raw.githubusercontent.com/Jesssullivan/misc/master/sel/portal.py

Some distros (Ubuntu and its derivatives on my Macbook for instance) struggle to find a route to the captive portal on public networks (read: Starbucks). Assuming you want to connect the way they intend (via the gateway through the captive portal) because you are a rule abiding patron, all you need to do is visit the address of the gateway. There is no need to fiddle with your network drivers, disable SSL stuff or engage in plebian network tomfoolery.

"""
find and open wifi captive portal (such as Starbucks)
written by Jess Sullivan
"""
from netifaces import gateways
from sys import argv
from time import sleep
import subprocess

help_str = str("\n " +
               'usage: \n ' +
               ' -h : print this message again \n' +
               ' -i : `pip3 install netifaces`  \n' +
               'You may specify a browser argument to complete the portal, such as \n' +
               'google-chrome')

def argtype():
    try:
        if len(argv) > 1:
            use = True
        elif len(argv) == 1:
            use = False
            print(help_str)
        else:
            print('command takes 0 or 1 args, use -h for help')
            raise SystemExit
    except:
        print('arg error... \n command takes 0 or 1 args, use -h for help')
        raise SystemExit
    return use

def main():

    all_gates = gateways()
    target = all_gates['default'][2][0]

    if argtype():

        if argv[1] == '-h':
            print(help_str)
            quit()

        if argv[1] == '-i':
            try:
                subprocess.Popen('pip3 install netifaces',
                                 shell=True,
                                 executable='/bin/bash')
                sleep(1)
                quit()
            except:
                print('err running pip3 install netifaces')
                quit()

        else:

            print(str('opening portal in ' + argv[1]))
            subprocess.Popen(str(argv[1] + ' ' + str(target)),
                             shell=True,
                             executable='/bin/bash')
            sleep(1)
            quit()

    else:

        print(str('\n please visit address ' + target +
                  ' in a browser to complete portal setup \n'))
        quit()

main()

More Morning Metal

I haven't posted a Morning Metal on Soundcloud for a while- glad to break the lull. xD
-Jess

JDK Management in R

Quickly & forcefully manage extra JDKs in base R
Simplify rJava woes

# get this script:
wget https://raw.githubusercontent.com/Jesssullivan/rJDKmanager/master/JDKmanager.R

rJava is depended upon by lots of libraries- XLConnect, OpenStreetMap, many db connectors and is often needed while scripting with GDAL.

library(XLConnect)   # YMMV

Errors while importing a library with depending on a JDK are many, but can (usually) be resolved by reconfiguring the version listed somewhere in the error.

On mac OSX (on Mojave at least), check what you have installed here- (as admin, this is a system path) :

sudo ls  "/Library/Java/JavaVirtualMachines/ 

I seem to usually have at least half a dozen or more versions in there, between Oracle and openJDK. Being Java, these are basically sandboxed as JVMs and are will not get in each others way.

However...

Unlike JDK configuration for just about everything else, aliasing or exporting a specific release to $PATH will not cut it in R. The shell command to reconfigure for R-

sudo R CMD javareconf

...seems to always choose the wrong JDK. Renaming, hiding, otherwise trying to explain to R the one I want (lib XLConnect currently wants none other than Oracle 11.0.1) is futile.
The end-all solution for me is usually to temporarily move other JDKs elsewhere.
This is not difficult to do now and again, but keeping a CLI totally in R for moving / replacing JDKs makes for organized scripting.

 JDKmanager help: 
 (args are not case sensitive) 
 (usage: `sudo rscript JDKmanager.R help`) 

 list    :: prints contents of default JDK path and removed JDK path 
 reset   :: move all JDKs in removed JDK path back to default JDK path 
 config ::  configure rJava.  equivalent to `R CMD javareconf` in shell 

 specific JDK, such as 11.0.1, 1.8,openjdk-12.0.2, etc: 
    searches through both default and removed pathes for the specific JDK.  
    if found in the default path, any other JDKs will be moved to the `removed JDKs` directory. 
    the specified JDK will be configured for rJava.

QEMU for Raspian ARM!

Updated 11/6/19

Visit the repo on Github here-
https://github.com/Jesssullivan/QEMU-Raspian

tested on Mac OSX 10.14.6

Emulates a variety of Raspian releases on proper ARM hardware with QEMU.

Prerequisites:

QEMU and wget (OSX homebrew)

brew install qemu wget

Get the Python3 CLI in this repo:

wget https://raw.githubusercontent.com/Jesssullivan/USBoN/master/QEMU_Raspian.py

Usage:

After the first launch, it will launch from the persistent .qcow2 image.

With no arguments & in a new folder, Raspian "stretch-lite" (no desktop environment) will be:

  • downloaded as a zip archive with a release kernel
  • unarchived --> to img
  • converted to a Qcow2 with 8gb allocated as disk
  • launched from Qcow2 as administrator
sudo python3 QEMU_Raspian.py 

Optional Arguments:

  • -h prints CLI usage help
  • -rm removes ALL files added in dir with QEMU_Raspian.py
  • stretch uses standard graphical stretch release with GUI
  • stretchlite for stretchlite release [default!]
  • buster for standard graphical buster release [YMMV]
  • busterlite for busterlite release [YMMV]
# examples:
sudo python3 QEMU_Raspian.py busterlite
python3 QEMU_Raspian -h  # print help

Burn as .img:

qemu-img convert -f qcow2 -O raw file.qcow2 file.img

Four Pi Emulations ala QEMU

« Older posts