Below are examples deemed worthy of the front page…
Bash is great: It is a shell scripting language that is baked into every new Apple computer and many, many Linux distributions too. It is fun to learn and easy to find your “insert repetitive task here” to Bash translation on Stack Exchange, just a search away. As you uses the command prompt more and more for increasingly cool GIS tasks- be it for Python QGIS OR ESRI, R language for data cleaning and analysis, or just because you noticed you can get “mad amounts of work done” at increasingly faster rates- your vocabulary in the UNIX shell and bash will naturally grow.
I wanted to make a GIS post for Mac OS because it is both under-represented (for great reasons) in GIS and arguably the number 1 choice for any discerning consumers of computer hardware.
Many Linux OS options are faster for much of this “UNIX for GIS”, as quite a few of the things we need are already included with many Debian / Ubuntu distros, and come forms that have been stable for a long, long time.
If you are looking to setup a system primarily for GIS / data science (disregarding ESRI of course), See my initial notes on the Ubuntu variant Pop_OS by System76. If you like the ChromeOS vibe for multitasking and simplicity and the familiarity of Mac OS, it is a keeper (and also a sleeper, seeing how many folks are still on Windows for GIS…….).
(The less you have in your way on your screen, the faster you can go!) xD
Note: in my opinion, homebrew and macPorts are good ideas- try them! If you don’t have it, get it now:
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
(….However, port or brew installing QGIS and GDAL (primarily surrounding the delicate links between QGIS / GDAL / Python 2 & 3 / OSX local paths) can cause baffling issues. If possible, don’t do that. Use QGIS installers from the official site and build from source!)
if you need to resolve issues with your GDAL packages via removal: on MacPorts, try similar:
sudo port uninstall qgis py37-gdal # on homebrew, list then remove (follow its instructions): brew list brew uninstall gdal geos gdal2
There are numerous issues with brew-installed gdal. Those I have run into include:
brew install gdal # # brew install qgis can work in a pinch too. At least you can unbrew it! #
# double check CLI is working: gdalinfo --version # “GDAL 2.4.0, released 2018/12/14” gdal_merge.py # list of args
To begin: Lets try a recent assignment that relies on the ESRI mosaic system and lots of GUI and clicking but use GDAL instead.
Data source: ftp://ftp.granit.sr.unh.edu/pub/GRANIT_Data/Vector_Data/Elevation_and_Derived_Products/d-elevationdem/d-10m/
!! Warning! These files are not projected in a way ESRI or GDAL understands. They WILL NOT HAVE A LOCATION IN QGIS. They will, however, satisfy the needs of my assignment.
# wget on mac is great. This tool (default on linux) lets us grab GIS data from # most providers, via FTP and similar protocols. brew install wget
make some folders
mkdir GIS_Projects && cd GIS_Projects
use wget to download every .dem file (-A .dem) from the specified folder and sub-folders (-r)
wget -r -A .dem ftp://ftp.granit.sr.unh.edu/pub/GRANIT_Data/Vector_Data/Elevation_and_Derived_Products/d-elevationdem/ cd ftp.granit.sr.unh.edu/pub/GRANIT_Data/Vector_Data/Elevation_and_Derived_Products/d-elevationdem
make an index file of only .dem files.
(If we needed to download other files and keep them from our wget (more common) this way we can still sort the various files for .dem)
ls -1 *.dem > dem_list.txt
use gdal to make state-plane referenced “Output_merged.tif” from the list of files in the index we made. it will use a single generic “0 0 255” band to show gradient.
gdal_merge.py -init "0 0 255" -o Output_Merged.tif --optfile dem_list.txt
copy the resulting file to desktop, then return home
cp Output_Merged.tif ~/desktop && cd
if you want (recommended):
rm -rf GIS_Projects # remove .dem files. Some are huge!
In Finder at in ~/desktop, open the new file with QGIS. A normal photo viewer will NOT show any detail.
Need to make something like this a reusable script? In Terminal, just a few extra steps:
mkdir GIS_Scripts && cd GIS_Scripts
open an editor + filename. Nano is generally pre-installed on OSX.
COPY + PASTE THE SCRIPT FROM ABOVE INTO THE WINDOW
make your file runnable:
chmod u+x GDAL_LiveMerge.sh
run with ./
open GIS_Projects you can now copy + paste your script anywhere you want and run it there. scripts like this should not be exported to your global path / bashrc and will only work if they are in the directory you are calling them: If you need a global script, there are plenty of ways to do that too.
Notes on source-
Please understand this is probably far from the last post you’ll need before you flex your GDAL powers.
// If it works in less the 6 tries you win //
Back in Terminal:
get some tools we need, mostly for the step configuring proj.4 from github each of these are used in this method of installation, but much of each package will be ignored
brew install autoconf automake wget postgresql libiconv brew link --force libiconv # this *may* need to be linked as well brew link libxml2 --force #
Setup your home directory- go home
cd # get proj.4 from the gdal people first. git clone https://github.com/OSGeo/proj.4 # ./autogen.sh ./configure # # Make takes ~1-10 min on # sudo make sudo make install cd #
get gdal, repeat steps # ./autogen.sh is usually not needed for gdal but to take no chances, run it anyway as they also include in in the repo….?
git clone https://github.com/OSGeo/gdal cd gdal/gdal # ./autogen.sh # ./configure
some configure ideas:
--with-python --with-pcraster=internal --with-libjson-c=internal --with-jpeg=internal --with-geotiff=internal --with-libtiff=internal # # Use a prefix to make an attempt at sandboxing the gdal install. This may be required. # # More ideas: # --with-curl=/usr/local/bin/curl LDFLAGS="-L/usr/local/opt/libiconv/lib" LDFLAGS="-L/usr/local/opt/libxml2/lib" CPPFLAGS="-I/usr/local/opt/libxml2/include" LDFLAGS="-L/usr/local/opt/libiconv/lib" LDFLAGS="-L/usr/local/Cellar//openssl/1.0.2q/lib/"
sudo make sudo make install
View below the readme mirror from my Github repo for Chapel Installation!
….Or visit the page directly: https://github.com/Jesssullivan/ChapelTests
Repo in light of PSU OS course
git clone https://github.com/Jesssullivan/ChapelTests cd chapeltests/ChapelTesting-Python3/ chpl ../FileChecking-with-Chapel/FileCheck.chpl # For same parallel domain read but parallel vs. serial evaluation: python3 Timer_FileCheck.py # for (questionable but) entirely serial evaluation: python3 Timer_PURE_Serial.py
FileCheck.chpl provides both parallel and serial methods for recursive duplicate file finding in Cray’s Chapel Language. Both solutions will be “slow”, as they are fundamentally limited by disk speed. Go to /FileChecking-with-Chapel/ for more information on this script. Timer_FileCheck.py and other tests evaluate completion times for both Serial and parallel options. Go to /ChapelTesting-Python3/ for more information on these tests.
In a (bash) shell, install Chapel:
Mac or Linux here, others refer to:
# For Linux bash: git clone https://github.com/chapel-lang/chapel tar xzf chapel-1.18.0.tar.gz cd chapel-1.18.0 source util/setchplenv.bash make make check #For Mac OSX bash: # Just use homebrew brew install chapel # :)
#Linux bash: cd sudo apt-get install atom apm install language-chapel # atom [yourfile.chpl] # open/make a file with atom # Mac OSX (download): # https://github.com/atom/atom # bash for Chapel language support apm install language-chapel # atom [yourfile.chpl] # open/make a file with atom
To compile with Chapel:
chpl MyFile.chpl # chpl command is self sufficient # chpl one file class into another: chpl -M classFile runFile.chpl # to run a Chapel file: ./runFile
Dual OS on a 2015 MacBook pro
As the costs of Apple computers continue to skyrocket and the price of useable amounts of storage zoom past a neighboring galaxy (for a college student at least), I am always on on the hunt for cost effective solutions to house and process big projects and large data.
Pop OS (a neatly wrapped Ubuntu) is the in-house OS from System76. After looking through their catalog of incredible computers and servers, I thought it would be a good time to see how far I can go with an Ubuntu daily driver. Of course, there are many major and do-not-pass-go downsides- see the below list:
Dual Boot (OSX and Pop Ubuntu) Installation on a 2015 MBP:
This process is quite simple, and only calls for a small handful of post-installation tweaks. My intent is to create a small sandbox with minimal use of “extras” (no extra boot managers or anything like that)
Partition separate “boot”, “home”, and other drives
Use the partition tool in Mac disk utility. Be sure to set these new partitions as FAT 32- we will be using ext4 and other more linux-y filesystems upon installation, so these need to be as generic as possible.
Use Etcher (recommended) or any other image burning tool to create a boot key for Pop.
The USB key only has one small job, in which Pop_os will be burned into a better location in your boot partition made in the previous step. If you are coming from a hackintosh experience, fear not: everything will stay in the Macbook Pro, not extra USB safety dongles or Kexts, or Plist mods…!
BOOT INTO POP_OS:
Restart your computer and hold down the alt-option Key. THIS IS HOW TO SWITCH from Pop_os, OSX, Bootcamp, and anything else you have in there. You should see an “efi” option next to the default OSX. (note- at least in my case, the built-in bootloader defaults to the last used OS at each restart.)
Once you are in the Pop_OS installer, click through and select the appropriate partitions when prompted. After this installation, you may remove the USB key and continue to select
“efi” in the bootloader.
ASSUMING ALL GOES WELL:
You are now in Pop_OS! Using the alt/option key will become second nature… but some Pop key mappings may not. Continue for a list of Macbook Pro – specific tweaks and notes.
Go to the Pop Shop and get the “Tweaks” tool. I made one or two small keymap changes, but this is likely personal preference.
Default, important Key Mappings:
Command will act as a “control center-ish” thing. It will not copy or paste anything for you.
Control does what Command did on OSX.
Terminal uses Control+Shift for copy and paste, but only in Terminal: if you pull a Control+Shift+C in Chrome, you will get the Dev tool GUI… The Shift key thing is needed unless you are inclined to root around and change it.
Custom Boot Scripts and Services:
In an effort to make things simple, I made a shell script to house the processes I want running when I turn on the computer- this is to streamline the “.service” making process. While it may only take marginally more time to make a new service, this way I can keep track of what is doing what from a file in my documents folder.
In terminal, go to where your services live if you want to look:
Or, cut to the chase:
sudo nano /etc/systemd/system/startsh.sh.service
Paste the following into this new file:
Description=Start at Open plz
Exit nano (saving as you go) and cd back to “/”.
sudo nano /Documents/startsh.sh
Paste the following (and any scripts you may want, see the one I have commented out for odrive CLI) into this new file:
# Uncomment the following if you want 24/7 odrive in your system
# otherwise do whatever you want
#nohup “$HOME/.odrive-agent/bin/odriveagent” > /dev/null 2>&1 &
After exiting the shell script, start it all up with the following:
sudo systemctl start startsh.sh
sudo systemctl enable startsh.sh
Cloud file management with Odrive CLI and Odrive Utilities:
Visit one of the two Odrive CLI pages- this one has linux in it:
Please visit this repo to get going with –recursive and other odrive utilities
These are the two commands I ended up putting in a markdown file on my desktop for easy access. Nope, not nearly as cool as it is on OSX. But it works…
Odrive sync: [-h] for help
python “$HOME/.odrive-agent/bin/odrive.py” sync
python “$HOME/odrive-utilities/odrivecli.py” sync –recursive
Next, Get Some Apps:
Download Chrome. Sign into Chrome to get your chrome OS apps loaded into the launcher- in my case, I needed Chrome remote desktop. DO NOT DOWNLOAD ADDITIONAL PACKAGES for Chrome Remote Desktop, if that is your thing. They will halt all system tools (disk utils, Gnome terminal, graphical file viewer… !!See this thread, it happened to me!! )
Get Atom editor: https://atom.io/
…Or my favorites: https://www.jetbrains.com/toolbox/app/
MySQL Workbench: https://dev.mysql.com/downloads/workbench/
If you get stuck: make sure you have tried installing as root ($ sudo su -) and verified passwords with ($ sudo mysql_secure_installation)
See here to start “rooting around” MySQL issues: https://stackoverflow.com/questions/50132282/problems-installing-mysql-in-ubuntu-18-04/50746032#50746032
Get some GIS tools:
sudo apt-get install qgis python-qgis qgis-plugin-grass
uGet for bulk USGS data download!
sudo add-apt-repository ppa:plushuang-tw/uget-stable
sudo apt install uget
That’s all for now- Cheers!