TRANSFORM 2020

Tutorial: Open Source Spatial Data Analytics in Python with GeostatsPy
 
'Build from zero, learn some spatial data analytics / geostatistics tools in Python.  All welcome!'

Unsure if You Want to Attend?

I model uncertainty as part of my job, but I actually don't like uncertainty in my life. Let me remove all uncertainty from this choice!

I have posted all of the Tutorial on GitHub @ https://github.com/GeostatsGuy/GeostatsPy_Intro_Course:

1. Lectures as PDFs with a fast treatment of fundamental spatial data analytics concepts
2. Interactive workflows in Python Jupyter Notebooks with GeostatPy, ipywidget and matplotlib packages
3. Well-documented workflows in Python Jupyter with GeostatsPy, demonstrated complete workflows to help get you started
4. Realistic, synthetic subsurface/spatial datasets for the demonstrations

Check it out, try it out, then decide if this is a good investment of your valuable time. Hope to see you there!

About the Lecturer

  Howdy Folks, I'm Michael Pyrcz, an associate professor at The University of Texas at Austin, appointed in the Cockrell School of Engineering and the Jackson School of Geosciences. I teach and conduct research on spatial data analytics, geostatistics and machine learning and I record all my lectures on my YouTube channel at https://www.youtube.com/GeostatsGuyLectures and post my well-documented demonstrations/exercises at: https://github.com/GeostatsGuy. I work in C++, Python, Fortran, R etc.

GeostatsPy Python Package Background

 Geostatistics and spatial data analytics are essential tools for data-drive spatiotemporal modeling workflows that are commonly applied to support decision making for subsurface resource development (ground water, minerals and hydrocarbons).  As an applied branch of statistics they add:

1. spatiotmeporal context,
2. geoscience information integration, 
3. accounting for spatial data and model scales
4. uncertainty models

When I started teaching my graduate level 'Stochastic Subsurface Modeling' course, I failed to find the Python solution to support the students' experiential learning and completion of the term-long subsurface modeling project. So I spent the weekends writing the GeostatsPy package (just days ahead of the students).  I'm excited to see wide use and contributions to open source package.

Tutorial Description

Goals

1. Introduce the Package: Expand familarity with the functionality of the GeostatsPy package
2. Add it to Your Toolbox: Provide experience with a variety of spatial data analytics workflows
3. You're Welcome: Invite community contributions to the GeostatsPy open source project

How Will We Accomplish All of That?

1. lectures: a limited amount time for overview lectures to provide basic theory
2. demonstrations: walk throughs of well-documented workflows
3. exercises: hands-on challenges for experiential learning

Topics Covered

  We will cover common topics/workflows for spatial data analytics, including:

1. Variogram Calculation
2. Variogram Modeling
3. Spatial Estimation

 Prerequisites

Please have the following installed locally:

1. Anaconda / Jupyter Notebooks
2. GeostatsPy (available at PyPi, use 'pip install GeostatsPy' to install)

More Information

• Times are in UTC and the location is virtual.
• You can reach me at mpyrcz@austin.utexas.edu.

I'm happy to discuss,

Michael

MICHAEL J. PYRCZ, Ph.D., P.Eng.,  Associate Professor
H.B. Harkings, Jr. Professor of Petroleum Engineering
Hildebrand Department of Petroleum and Geosystems Engineering
Jackson School of Geosciences and Bureau of Economic Geology
The University of Texas at Austin  |  512-471-3252  |  www.pge.utexas.edu
Twitter (@GeostatsGuy) | YouTube | LinkedIn | Webpage | Book | GoogleScholar

The objective of this course is to teach geoscientists, engineers and researchers all the basics about 3D structural geological modeling using our open-source Python library GemPy.

They will learn about the most important theoretical principles behind its implicit modeling engine, its advantages, and how to use GemPy most efficiently.

We will start by building a relatively simple geological model from scratch and introduce functionalities as we add features step by step. Subsequently, participants will learn more about adequate data management and how to import data and models into GemPy, including from other modeling software. Towards the end, we will give an outlook into advanced approaches such as the quantification of geological uncertainty.

 In this tutorial, we will provide a hand-on overview of using SimPEG to simulate and invert geophysical data. The examples we plan to work through use Direct Current (DC) Resistivity and Induced Polarization (IP) data from the Century Zinc Deposit in Australia.

Starting from field data in a text file we will learn how to

• load those data into SimPEG 
• construct a survey object that contains the geometry of the sources and receivers
• set up and run a forward simulation 
• define the inverse problem consisting of a data misfit and regularization
• run an inversion and discuss inversion strategies

Here are the links to the materials for this session

• Intro Slides
• Course Notebooks

And some additional resources for getting involved in the SimPEG community

• Resources on Geophysics and Inversions

  • Geophysics for Practicing Geoscientists

  • EM GeoSci

  • Lectures from EOSC 350: Exploration & Environmental Geophysics (2017, 2018)

  • DISC 2017 lectures (Mexico City)

• Resources on SimPEG

  • Docs

  • Discourse

  • Slack

  • This Tutorial

This tutorial will be a hands-on tour of Verde, a Python package for processing and gridding geophysical/geospatial data with a twist of machine learning.
BEFORE THE TUTORIAL: Please go to github.com/fatiando/transform2020 and follow the setup instructions. There is also information on where to ask questions and what is going to happen in the tutorial.

We'll start with a real dataset and work our way towards producing one or more gridded products. The way there will take us through:

• Loading some data
• Generating and handling coordinates and projections (using pyproj)
• Splitting training and testing data for validation
• Data decimation with blocked means/medians to avoid aliasing
• 2D trend estimation
• Gridding with bi-harmonic splines
• Combining everything into a data processing pipeline
• Cross-validation of data distrubuted spatially on the Earth (including parallel execution)

UPDATE: Given the size of the tutorial, this will mainly be some live coding to showcase what Verde can do with participants following along. There will be some short exercises that can be done individually after the tutorial is finished. We will finish a bit earlier than the scheduled time to leave room for more questions/chat at the end.  

We'll cover the image processing operations you need to know to solve common geologic and geophysical problems across scales using Python. Images are more than just photographs, and we'll focus on methods that can be applied to any regularly sampled dataset, be it a thin section, an elevation map, a seismic volume, or a satellite image.  If you've ever wondered how to chain Python libraries together to accomplish tasks you do in ImageJ, ArcGIS Spatial Analyist, or Photoshop, this tutorial is right for you.

We'll focus primarly on using basic numpy operations, scipy.ndimage, and scikit-image, but we'll do some work with more specalized libraries as well.

Materials are in the git repository here: https://github.com/joferkington/geo_image_processing_tutorial   The material is still under active development at the moment, but feel free to browse around.  If you're not familiar with how to work with this repository, you can run the tutorial in binder.

While the details may change, here's the current outline:

• Overview / Introduction
• Seamount Detection Example
  • Thresholding
  • Filtering
  • Segmentation
  • Simplification
• Slope and Hillshade of Topographic Data
  • Gradients
• Grain Detection in Thin Sections
  • Using gradients for more than slope
• Lineament Analysis from Aerial Photography
  • Edge detection
  • Hough Transform
  • Structure Tensor
• What is color?
  • Satellite data examples

Welcome to your new assignment!

You are a Cloud Consultant who is beginning an assignment at Unicorn Reservoir Charachterization (URC) * a new tool for geological analysis that "will change how much your rocks rock!"

In this session, you will meet with the URC team, and design a scalable architecture that meets their requirements. What are their requirements? What will this system do? How does the data go from a rock to a data science notebook?

Well, to answer those questions, you simply have to show up and ask them.

In this session, you will be an Architect responsible to translating business requirements to technical requirements, white boarding out an architectural diagram, and then figuring out how to take your great ideas and translate them into actionable next steps.

In this session, you'll meet some of the key stakeholders at URC, learn about their goals, and design a system to meet their vision of "Being the Rock Stars of the Reservoir Characterization World"

• Note that Unicorn Reservoir Characterization (URC) is a fictitious company and any overlap with real companies and products is purely coincidental

Learning Objectives

• Understand the basics of translating business requirements to technical requirements
• Knowledge about key AWS Services and where to learn more
• Techniques for whiteboarding successfully
• What the AWS Well-Architected Framework is and how to use it
• How to take those great ideas and architecture and do something with them

Background

Data engineering and systems architecture is a necessary precursor to delivering data-driven subsurface insights. As a geologist or geophysicist, systems design typically isn’t something you learn in school, yet it can be a necessary skill set to progress your career and drive meaningful change.

Session Overview

This session will present an overview of the key concepts and design principles for building a scalable and cloud-optimized infrastructure in line with the AWS Well-Architected Framework, with a specific focus on Energy-Industry use cases and accompanying challenges for subsurface professionals. Attendees will leave this session with techniques for outlining technical requirements, knowledge of key AWS services and solutions, an overview of the AWS Well-Architected Framework, and an understanding of how to design for scalable and resilient solutions that can be implemented to improve subsurface workflows.

Agenda

• Working backwards: User stories to drive technical requirements
• Services, solutions, and scalability
• Whiteboarding as a superpower
• The AWS Well-Architected Framework
• Now what? How to communicate and drive architectural improvements

Important note: This talk is being rescheduled as a result of the #ShutDownSTEM initiative. Further details are available here: https://softwareunderground.org/blog/2020/6/7/shutdownstem-at-transform

I will update this once it has been rescheduled, but for now, please consider taking the time that would have been spent on this tutorial to read the links in the blogpost and consider what else can be done to aid and support marginalised voices in STEM in general and the geosciences in particular.

---

This tutorial is intended to give an overview of some vector geospatial capabilities within Python, predominantly using geopandas. This should be accessible to most people who have some familiarity with Python. Some basic knowledge of pandas would help, but is not required. Matplotlib will be used for some of the plotting; knowing how to make a simple plot is going to make parts of this tutorial simpler.
Topics to be covered:

• File I/O and GeoDataFrames
  
• Plotting Data
• Spatial Selections and subsetting
  
• Neighbours and distances
• Basemaps (Demo, more than a tutorial)
  
• Spatial Joins (time permitting)

Please make sure that your environment has geopandas, mapclassify, rasterio, contextily, matplotlib, numpy, and jupyter, along with their dependencies installed.
The majority of this tutorial will be done in jupyter notebooks, as a mix of live coding and hands-on exercises for participants, if they wish to engage.
The course materials, an environment file for installing everything on your own PC are available in this GitHub repo.There is also an option to use a Binder to run it remotely, with no install, by clicking this finely crafted link.

As a starting point I assume you are familiar with basic sklearn model pipelines and the “shape” of NN training code. We’ll be using pytorch for that bit.

UPDATED

Then the tutorial will focus on Ray and its libraries. Exploring some of the training code patterns used, and the Ray/Raytune API. Once your code can be run by Ray then it’s easy to use ray features and train models on a multi core machine though to a multinode gpu cluster without much change to that core code. Ray does the heavy lifting.

So we’ll step through various ray features taking a more verbose and relaxed path then some of the ray docs, and (hopefully) with some examples relevant to the subsurface domain.

We’ll spend most of our time looking at SOTA Hyperparameter optimisation with Raytune, Ray itself and [hopefully] plpaying with a Ray Cluster.

But we’ll also at least pay homage to RayRLib (Reinforcement Learning) RaySGD (Distributed Training) and look at the very recent addition Ray Serve.

Some parts of the tutorial you will be able to run on a local machine. More if you have a GPU available. Some parts may involve launching AWS clusters and that part will probably be “follow along” only. 

We'll show you how to get from an idea in your head to a Minimum Viable Product on the web. This session will help you transform from mild-mannered subsurface professional to digital solution superhero!

The specific use case we are looking at is putting a machine learning model on the web. We'll primarily be using scikit-learn for our (very simple) machine learning model, and flask for the web server. We will mention some other use cases in passing: dashboards, calculators, database queries, etc.

We will also spend a little time asking you to think about some of your own ideas, and helping you map those from your head into a solid plan for a minimum viable product of your own.