Lesson 13: Google Earth Engine

WUR Geoscripting WUR logo

Week 3, Lesson 13: Google Earth Engine

Good morning! Today we will start working with Google Earth Engine.

Time Activity
Morning Self-study: go through the following tutorial
14:00 to 14:40 Presentation and discussion
Rest of the afternoon Do/finalise the exercise.

If you do not have an activated account yet, register an account now and hopefully your account will be activated soon. If it doesn't work out, ask someone to share a repository with you, that should grant you an account.

Today's schedule

We will have a look at the purpose of Earth Engine, what Earth Engine can do for you and how to use Google Earth Engine with JavaScript and Python.

Purpose Earth Engine

The purpose of Earth Engine is:

  • Perform highly-interactive algorithm development at global scale
  • Push the edge of the envelope for big data in remote sensing
  • Enable high-impact, data-driven science
  • Make substantive progress on global challenges that involve large geospatial datasets

The platform for global analysis is there, but it needs specialists, students and researchers like you, to use the Earth Engine platform to do high-impact data-driven science.

What does Google Earth Engine do?

Google Earth Engine is a cloud-based platform for planetary-scale environmental data analysis. The main components of Earth Engine are:

  • Datasets: A petabyte-scale archive of publicly available remotely sensed imagery (such as Landsat, Sentinel and Modis) and other data (e.g. elevation data). Explore categories of data in the archive or browse and search datasets.
  • Compute power: Google’s computational infrastructure optimized for parallel processing of geospatial data. Learn more from Gorelick et al. (2017).
  • APIs: APIs for JavaScript and Python (see GitHub) for making requests to the Earth Engine servers. These docs focus on JavaScript; (Guide for JavaScript API). For Python, see the Python install guide and the Python examples in the Earth Engine GitHub repository.
  • Code Editor: An online Integrated Development Environment (IDE) for rapid prototyping and visualization of complex spatial analyses using the JavaScript API Code Editor docs.

Google Earth Engine: Code Editor in JavaScript

Today you will learn about Google Earth Engine by going through 9 tutorials made by Google. In the morning you will work on these tutorials. Have fun working in the cloud!

  1. Get familiar with basic JavaScript syntax
  2. Get familiar with advanced JavaScript syntax
  3. Get familiar with the Code Editor
  4. Visualizing Images and Image Bands
  5. Computations using Images
  6. Image Collections
  7. Compositing, Masking, and Mosaicking
  8. NDVI, Mapping a Function over a Collection, Quality Mosaicking
  9. Exporting Charts and Images
ProTip: at almost any point you can use .aside(print) for debugging purposes. Come back to this Debugging guide when in need

Google Earth Engine is a nice platform for processing and visualization. Even though raw data can be downloaded, such an attempt would defeat its intended goal of uploading the code next to the data instead of downloading data next to the code.

Question 1: Is the code on the online editor reproducible for other users?

If you finished these tutorials and want to do more advanced stuff with the Code Editor you can have a look at other tutorials made by universities or at more advanced guides of Earth Engine (you can find the guides in the left pane) about the Image (e.g. gradients, edge detection, object detection), ImageCollection, Geometry & Features, Reducer, Join, Chart (e.g. time series analysis), Array, User Interfaces, Specialized Algorithms, Asset Management, Custom Applications and Earth Engine setup.

Google Earth Engine: Python API

Although Earth Engine focuses on using JavaScript in the online Earth Engine Code Editor, an Earth Engine Python module was developed. The module allows you to connect via an API to to the Earth Engine servers.

This presentation on the Earth Engine user summit explains more about the Python API.

Earth Engine normally uses Python 2.7. However Earth Engine is trying to make code compatible for both Python 2 and Python 3. Since Python 3 is the future, we will use Python 3. Your Google account needs to be accepted to authenticate your local Earth Engine Python module. So we need to setup the Python environment and get authenticated:

# Create Conda environment
conda create -n ee_py3 -c conda-forge python=3 google-api-python-client pyCrypto spyder jupyter

source activate ee_py3

# Ensure that a crypto library is available
python -c "from oauth2client import crypt"

# Install earthengine-api
pip install earthengine-api

# Authenticate earth engine
earthengine authenticate
# Follow procedure to authenticate and paste the access token in your terminal

# Check if earth engine has been installed
python -c "import ee; ee.Initialize()"
# If you don't get an error, you are good to go

After setting up the Python environment, you can start a jupyter notebook or spyder and run some Python code.

# Import the Earth Engine Python Package
import ee
# Initialize the Earth Engine object, using the authentication credentials.
ee.Initialize()
# Print the information for an image asset.
image = ee.Image('srtm90_v4')
print(image.getInfo())

Now we want a Python script that shows a time series of Landsat 8 imagery.

import ee
from ee import batch
## Initialize connection to server
ee.Initialize()
## Define your image collection 
collection = ee.ImageCollection('LANDSAT/LC8_L1T_TOA')
## Define time range
collection_time = collection.filterDate('2013-04-11', '2018-01-01') #YYYY-MM-DD
## Select location based on location of tile
path = collection_time.filter(ee.Filter.eq('WRS_PATH', 198))
pathrow = path.filter(ee.Filter.eq('WRS_ROW', 24))
# or via geographical location:
#point_geom = ee.Geometry.Point(5, 52) #longitude, latitude
#pathrow = collection_time.filterBounds(point_geom)
## Select imagery with less then 5% of image covered by clouds
clouds = pathrow.filter(ee.Filter.lt('CLOUD_COVER', 5))
## Select bands
bands = clouds.select(['B4', 'B3', 'B2'])
## Make 8 bit data
def convertBit(image):
    return image.multiply(512).uint8()  
## Convert bands to output video  
outputVideo = bands.map(convertBit)
print("Starting to create a video")
## Export video to Google Drive
out = batch.Export.video.toDrive(outputVideo, description='Netherlands_video_region_L8_time', dimensions = 720, framesPerSecond = 2, region=([5.588144,51.993435], [5.727906, 51.993435],[5.727906, 51.944356],[5.588144, 51.944356]), maxFrames=10000)
## Process the image
process = batch.Task.start(out)
print("Process sent to cloud")

To explain what is going on in the script: an ImageCollection is created and queried by filtering on date, location, clouds, bands. The ImageCollection object holds the information of the query and is sent to the Google Earth Engine server. The Google Earth Engine server then performs the task of gathering the data, making the imagery into a video and then exporting it to your Google Drive.

There are some more advanced Python scripts made by the Google Earth Engine developers that work with bqplot and ipyleaflet. If you want you can have a look at those too.

# Install bqplot from conda-forge channel and ipyleaflet with pip
source activate ee_py3
conda install -c conda-forge bqplot
pip install ipyleaflet
jupyter nbextension enable --py --sys-prefix ipyleaflet

Exercise 13 : Blue skies

In remote sensing clouds are our arch enemies. You probably heard this a lot, and you'll hear it again from everyone working with satellite imagery.

But how big of a problem are clouds, cloud shadows, or other obstructions? Are they as bad as people say?

Have fun figuring it out using the online code editor of Google Earth Engine.

A bit of organization first

  1. Create a new repository, named like your team, using the red new button on your left in the GEE code editor

  2. Create a new file in that repository and name it Assignment13. At the beginning of this file write your names as comments.

  3. Either now, or after you're done with your assignment, share your newly created repository with andrei.mirt@gmail.com

Requirements

Display a map for an area of your choice in which each pixel represents the number of clear Landsat 8 (not clouded) observations at that location for the year 2018. The final map should look somewhat similar to the picture below:

white for clear skyes and black for cloudy

Hints

The quality mask is stored in the pixel_qa band as a bitmask. You can use the Landsat 8 Surface Reflectance Code Product Guide to remap the clear pixel_qa values to 1. 386 is a good value for clear observations.

Question 2: Can you find a better value for clear observations than 386, could you use these together?

More hints

-Are there any interesting patterns on the total number of observations? Can you explain all these patterns? The answer to this question on StackExchange might come in handy.

SuperHint
You should probably first count all valid observations and then mask your way through each band towards the total number of clear observations.

Bonus

  • Do the same for sentinel 2. Plot the map in parallel linked panels.

Still have some time left? Work on the visualization parameters.

Submission

Ask the team checking your script for their Google Earth Engine account (gmail address), and don't forget to share your repository with them and with andrei.mirt@gmail.com

In order to assess the exercise, consider the following tasks:

  • Task 1: The script displays a map somehow related to the input data
  • Task 2: The script displays the map that shows the number of clear observations, but some values are unnecessarily repeated
  • Task 3: The script displays the correct map that shows the number of clear observations, without repeating unnecessary values

Further you can explore how you can use Google Earth Engine for your project or try to do some of the more advanced guides(scroll through the left pane).

Sunny places make good holidays. After all this work, you can use the map you just created to search for your next holiday destination, while listening to an Irving Berlin classic.

References and more info

Most of the information on Google Earth Engine in this tutorial comes directly from Google and from universities, whom developed tutorials for students to work with Google Earth Engine.