Introduction

Good afternoon! Today we will start working with Python for geoscripting and do a refresher of functions in Python. If you are unfamiliar with Python and/or feel that you need more training, follow one of the Datacamp courses as introduction into Python before today:

• Introduction to Python | recommended to follow if you haven’t any scripting experience so far
• Python for R users | recommended if you have experience already in R

Today’s Learning objectives

• Know how to work with virtual environments: Conda + Mamba
• Get introduced to Python editors and IDEs
• Familiarize yourself with Python objects and inheritance
• Be able to visualize data using Python and matplotlib

Python package management with Conda

A set of tools exist for installing and managing Python packages. Although it is possible to install packages on your base Python interpreter, sooner or later you will get conflicts between Python packages since packages may have varying dependencies. This can even break your system Python interpreter.

Instead, we recommend to use a Python package manager that can make use of virtual environments, such as Conda or Mamba. That way, you can create a Conda environment on your machine for each project. In Conda environments, basically anything, such as software, C libraries or R packages can be installed. Here we use them for installing Python packages. Packages installed in one environment do not interfere with your base Python or with other Conda environments. Additionally, it is possible to export and share the requirements for your (open source) project with collaborators or users of your code.

Mamba installation

For this course, we will make use of Mamba, a fast drop-in re-implementation of the Conda package manager. It has its core parts implemented in C++ for maximum efficiency, makes use of parallel downloading of repository data and package files using multi-threading, and uses libsolv for (much) faster dependency solving. To install Mamba in your Linux environment, we have prepared a short Bash script for you. Just run the following lines of code, line by line, in a new terminal window.

git clone https://github.com/GeoScripting-WUR/InstallLinuxScript.git
cd InstallLinuxScript/user
chmod u+x ./install.sh
./install.sh

This will install Mamba into ~/mamba. Finally, restart your terminal to be able to use Mamba and Conda in the terminal. Next, let’s see how to use Mamba in case you want make new virtual environments by yourself, or install packages after creating the environment.

Mamba usage

Mamba creates isolated Conda environments with sets of packages, that do not interfere with your base Python or with other Conda environments. To create an environment:

mamba create --name geotest python numpy

This would create a new environment called geotest with Python, NumPy and Spyder installed into the Conda environment. Another option is to create an environment from a .yaml file, in which all required modules are listed. You will see an example of this later on in this tutorial. To create an environment from such a file, you can use the argument --file (or --f in short).

Let’s first list the currently available environments:

mamba info --envs

Mamba puts an asterisk (*) in front of the active environment. Now we activate the environment. While Mamba replaces Conda for most commands, this is not the case for (de)activating environments:

# Cross-platform (but not always working, like in our VM, so we use the next option)
conda activate geotest

# Linux, macOS
source activate geotest

# Windows
activate geotest

After this, the current environment is shown in parentheses in front of your prompt ((geotest)$). Note that the activated environment is only valid for the shell in which you activated it. For instance, if you close the shell window and open a new one you will have to activate it again.

After creating a Conda environment, (additional) Python packages can be installed. There are three possible ways to install packages, which we list below.

• Using Mamba to install and manage Conda packages. This downloads Conda packages using Conda channels, which are URLs to directories containing the Conda packages. Generally, installing Conda packages using Mamba is the preferred method.
• Using pip to install packages and Mamba to manage these packages. pip is available for Windows, macOS and Linux. pip can also install binary wheels on Windows. You should generally not install packages from pip in a Conda environment unless it’s the last resort. This is because after you use pip to modify an environment, you can no longer use conda/mamba to do so (trying that will break your environment, because pip does not communicate its changes to Conda). Hence install packages with mamba that you can first, and only then use pip, and then never touch the environment with mamba again (delete and start fresh if you need to).
• Using the distribution’s package manager (only on Ubuntu, that is sudo apt-get install python-*).

The mamba search command searches a set of channels. By default, packages are automatically downloaded and updated from the default channel. To search for a package, type:

mamba search pandas

This gives a list of all packages that have “pandas” in the name and lists all available versions. To install:

mamba install pandas

This installs the latest compatible version of Pandas. Note that this would install it into your currently activated environment.

Note that you can also install multiple packages at the same time:

mamba install geopandas matplotlib

As you saw with Spyder (which is an IDE, more on that later), Mamba is also able to install some non-Python packages that have Python bindings. This is useful for making sure your Python and binary versions match and do not interfere with the system-wide ones.

Some additional helpful utilities for package management in this context are:

• mamba list to check which packages are installed in root or in the active environment;
• python --version or gdal-config --version to check which Python or GDAL version is used in the environment;
• which spyder or type spyder to find out which Spyder executable is used either from system or Conda environment.

Removing packages is just as simple:

mamba remove geopandas pandas folium

Now, we deactivate the environment and return to base environment.

# Cross-platform
conda deactivate

# Linux, macOS
source deactivate

# Windows
deactivate

When we are finished, and do not need the environment for next time, we can remove the environment geotest.

mamba remove --name geotest --all

Running a Python script in the terminal

Within a Conda environment, Python can be started directly, or can be called to run a script file. To start Python directly:

python

Now, you can type Python expressions that will be executed one by one:

import sys
print('Good morning, you are running Python:', sys.version)

To go back, type:

exit()
# or 
quit()

Usually, we do not want to run expressions one by one, but build scripts instead, to ensure transferability and reproducibilty. Create a new text file and (re)name it (to) test.py. Open it, for example with a text editor, paste in the code you used above (import sys etc.), and save the script. Navigate in the terminal to the location where this script is stored, using cd. Finally, run the script with:

python test.py

The output is printed to the terminal. Running a script from the terminal is less error-prone than running it from an IDE (see the next section), such as Spyder, as IDEs often keep variables in memory after the script has finished running. Therefore, running a script from the terminal is a good final test before submitting an exercise or assignment.

Python editors and IDEs

There are many Integrated Development Environments [IDE] for Python, and every programmer has their own preference. An IDE is a software application that provides facilities for software development.

• Jupyter Notebook integrates visualization with code and is suitable to make tutorials, simple dashboards, quick visualizations, and do prototype testing. Jupyter Notebooks run in your browser on a localhost server or on a web server. They allow for various programming languages, e.g. Python, R, Julia, Spark or PySpark.
• Spyder is a lightweight IDE. In this course, Spyder is the recommended Python IDE.
• PyCharm Community Edition is a free professional Python IDE with a lot of advanced functionality, such as integrated GIT version control, code completion, code checking, debugging and navigation. This IDE can optionally be used by more advanced scripters during this course instead of Spyder, but do know that you will not be assisted for solving IDE-related issues.

jupyter notebook

import folium

m = folium.Map(location=[51.9700000, 5.6666700], zoom_start=13)
m

conda deactivate

Setting up the environment

First, make a directory structure for this tutorial:

cd ~/Documents/
mkdir PythonRefresher #or give the directory a name to your liking
cd ./PythonRefresher
mkdir output

We only make a directory for output, because no input data or separate scripts are created in this tutorial. Next, we will create a Conda environment from a file. First create a text file in your preferred text editor, e.g. gedit. Then, (re)name it (to) refresher.yaml, and copy the following content into the file:

name: refresher
dependencies:
  - python
  - numpy
  - matplotlib
  - geopandas
  - spyder

Now, create a new Conda environment based on this file:

mamba env create --file refresher.yaml

Important to note: for compatibility, it is best to install packages from the same channel as much as possible. Given that packages in the file refresher.yaml are installed from the conda-forge channel, it is wise to use this same channel when you want to install additional packages in your environment.

Once everything is installed, activate the environment and start Spyder:

source activate refresher
spyder

In Spyder you should see an editor, a file explorer and a console. Have a look at the toolbar. Some important shortcuts are:

• F5 to run your script
• CTRL + S to save your script
• CTRL + 1 to comment/uncomment your code
• TAB to indent your code
• SHIFT + TAB to unindent your code

Open a new file and save it in the project directory as main.py (File – > New File –> Save As). Set the working directory to the source file location. This can be done by clicking on the three lines icon on the top right of the editor pane, and selecting Set console working directory.

Setting working directory in Spyder

Quick refresher

In the tutorial about R and Python we have gone over the differences and similarities of Python and R. This tutorial also contains some basic Python syntax, in this tutorial we assume you know this content, but we will go over a few basics here as well. The example below are mostly meant for reference purposes, we assume you understand most of this refresher already.

You can run the following code in the main.py file you just created. All of the variables which you create will appear in the Variable Explorer tab, and any visualizations should be rendered in the Plots tab, of the top right pane in Spyder.

# Integer
age = 25

# Float
height = 1.75

# String
name = "John Doe"

# Boolean
is_student = True

# Print a name
print(name)

# String formatting and printing 
print(f'{name} is {age} years old and is {height} meters tall.)

a = 10
b = 5

addition = a + b
subtraction = a - b
multiplication = a * b
division = a / b
modulo = a % b
exponentiation = a ** b

print(addition, subtraction, multiplication, division, modulo, exponentiation)

x = 15

if x > 10:
    print("x is greater than 10")
elif x == 10:
    print("x is equal to 10")
else:
    print("x is less than 10")

# For loop
for i in range(5):
    print(i)

# While loop
count = 0
while count < 5:
    print(count)
    count += 1

# Creating a list
fruits = ["apple", "banana", "orange"]

# Accessing elements
print(fruits[0])  # Output: "apple"

# Adding elements
fruits.append("grape")

# Removing elements
fruits.remove("banana")

# Length of the list
print(len(fruits))  # Output: 3

# Function to add two numbers and return the result
def add_numbers(a, b):
    return a + b

result = add_numbers(5, 3)
print(result)  # Output: 8

# Creating a dictionary
person = {
    "name": "Alice",
    "age": 30,
    "is_student": False
}

# Accessing values
print(person["name"])  # Output: "Alice"

# Adding a new key-value pair
person["occupation"] = "Engineer"

# Removing a key-value pair
del person["is_student"]

import pandas as pd

data = {
    'Name': ['Alice', 'Bob', 'Charlie'],
    'Age': [25, 30, 22],
    'City': ['New York', 'San Francisco', 'Chicago']
}

df = pd.DataFrame(data)
print(df)

# Display the first few rows of the DataFrame
print(df.head())

# Get statistical information about the DataFrame
print(df.describe())

# Access a specific column
print(df['Age'])

import geopandas as gpd

# Dummy data for the GeoDataFrame
data = {
    'Name': ['Location A', 'Location B', 'Location C'],
    'Latitude': [40.7128, 34.0522, 41.8781],
    'Longitude': [-74.0060, -118.2437, -87.6298]
}

# Create the GeoDataFrame with a single line of code
gdf = gpd.GeoDataFrame(data, geometry=gpd.points_from_xy(data['Longitude'], data['Latitude']))

# Display the GeoDataFrame
print(gdf)

Object-Oriented Programming in Python

We have now gone over most of the more basic basic functionality of Python, a lot of similar things you have used in R. A concept we have not used before is the concept of objects. They have shortly been introduced in R, but we will elaborate on them in Python. Up until now, in this course we have looked at R mainly as a scripting language, we call this way of programming Procedural Programming. Both Python and R can be used in another programming paradigm: Object Oriented Programming. Object Oriented Programming (OOP) is a way of programming where functionality and information is encapsulated in objects. Instead of assigning variables and functions, objects are used where both values (properties) and calculations (methods) can be stored together. This offers several advantages. OOP promotes modularity and re-usability by breaking down complex problems into smaller, manageable units, these are the objects. These objects can be reused in various parts of the program or even in other projects, leading to more efficient, scalable and organized programming. Especially when working on projects containing lots of code OOP will make your work a lot easier to understand for you and others and it is easier to re-use parts of the code.

class Person:
    def __init__(self, name, age):
        self.name = name  # < this is a property
        self.age = age    # < this is also a property
    
    def greet(self):  # < This "function" is a method
        print(f"Hello, my name is {self.name} and I'm {self.age} years old.")

person1 = Person("Alice", 25)
person2 = Person("Bob", 30)

print(person1.name)  # Output: Alice
print(person2.age)   # Output: 30
person1.greet()      # Output: Hello, my name is Alice and I'm 25 years old.
person2.greet()      # Output: Hello, my name is Bob and I'm 30 years old.

class GeoSeries(GeoPandasBase, Series):

class GeoSeries:

class Student(Person):
    def __init__(self, name, age, student_id):
        super().__init__(name, age)
        self.student_id = student_id
        self.is_studying = False
    
    def study(self):
        self.is_studying = True
        print(f"{self.name} is studying.")

student = Student("Eve", 22, "123456")
print(student.name)         # Output: Eve
print(student.student_id)   # Output: 123456
student.greet()             # Output: Hello, my name is Eve and I'm 22 years old.
student.study()             # Output: Eve is studying.

Visualization

Visualization is essential to make concepts understandable and patterns recognizable. In Python, Matplotlib is a general plotting package. It is used as a base for many other, more tailored, packages. One of the core advantages of Matplotlib is that the representation of figure is separated from the act of rendering it. This enables building increasingly sophisticated features and logic into the figure, a bit like making a map in a GIS.

In Matplotlib, a figure object contains axes, or subplots. These axes contain an x-axis, a y-axis and can contain lines and text. This hierarchy is important to understand when you want to edit a plot:

Let’s create a simple line figure, based on data in NumPy.arrays.

import numpy as np
from matplotlib import pyplot as plt

# Create some data
x = np.arange(-np.pi, np.pi, 0.2)
y = np.sin(x)

# Plot x against y
plt.plot(x, y)

# Show the plot
plt.show()

Note that the behavior of plt.show() depends on how you run the script. If you are using Spyder and want plt.show() to work:

1. Go to Tools.
2. Go to Preferences.
3. Select IPython console.
4. Go to Graphics tab.
5. In the Graphics backend section, select Automatic as the backend type.
6. Restart your kernel.

Instead of a single plot, the figure can have subplots (here two). Using the sharex and/or sharey argument, the axes can be aligned automatically, for easy comparison between the subplots. Also, we can add a title and axis labels. Finally, we can change the positions of the tick marks on an axis to something meaningful in the context of trigonometric functions and customize the labels with regular text or lateX. Check if you understand what object in the hierarchy is edited and why.

import numpy as np
from matplotlib import pyplot as plt

# Create some data
x1 = np.arange(-np.pi, np.pi, 0.2)
y1 = np.sin(x1)
x2 = np.arange(-np.pi, 0, 0.2)
y2 = np.cos(x2)

# Initiate a figure with two subplots 
f, axarr = plt.subplots(2, sharex=True)

# Subplots are stored in an array
line = axarr[0].plot(x1, y1)
axarr[0].set_title('plot 1')
axarr[1].plot(x2, y2)
axarr[1].set_title('plot 2')
f.suptitle('super title', fontsize=16)

# Axis label
axarr[1].set_xlabel('x')
axarr[0].set_ylabel('sin(x)')
axarr[1].set_ylabel('cos(x)')

# Axis ticks and tick labels
xticks = axarr[1].get_xticks()
print(xticks)
new_ticks = np.arange(-np.pi, np.pi + 0.1, 0.25 * np.pi)
new_labels = [r"$-\pi$", r"$-\frac{3}{4}\pi$",
              r"$-\frac{1}{2}\pi$", r"$-\frac{1}{4}\pi$",
              "$0$", r"$\frac{1}{4}\pi$",
              r"$\frac{1}{2}\pi$", r"$\frac{3}{4}\pi$",
              r"$2\pi$"]
axarr[1].set_xticks(new_ticks)
axarr[1].set_xticklabels(new_labels)
plt.show()

One can also create multiple subplots (axes) and use different plotting styles, changing e.g. the marker style, line style, marker size, and colors. Furthermore, for the upper left subplot it is demonstrated how to add a legend; adding a label to the plotted line is essential for this.

from matplotlib import pyplot as plt

x = [1, 2, 3, 4, 5]
y = [6, 7, 8, 9, 10]

# New: define number of rows and columns of subplots and unpack them directly 
# into variables that then each contain one axes object
f, ((ax0, ax1), (ax2, ax3)) = plt.subplots(2, 2)

# Dashed line, label for legend, and show the legend on the subplot
ax0.plot(x, y, 'r--', label='red dashed line')
ax0.legend(loc='lower right')

# Scatter plot, using a colormap based on the y-value, changing the marker size to 35
ax1.scatter(x, y, c=y, cmap='bwr', s=35)

# Bar chart, changing the bar color to black
ax2.bar(x, y, color='k')

# Horizontal bar chart, changing the bar color to yellow
ax3.barh(x, y, color='y')
plt.show()

When working with spatial data, it is crucial that the spatial scale (aspect) in the x and y direction is the same. Because Matplotlib can be used for plotting any kind of data, not necessarily spatial, it does not automatically do this. We can use plt.axis('equal'), or ax.set_aspect('equal') on the axis, to ensure equal scales in both directions. Let’s test this on the non-spatial data in the plots above, see what happens (make sure to adjust the code above properly, and replace plt.show() with the code below). Adjust the Spyder working directory by clicking on the directory icon in the top right pane if necessary.

ax0.set_aspect('equal')
ax1.set_aspect('equal')
ax2.set_aspect('equal')
ax3.set_aspect('equal')
plt.savefig('output/equal_scale.png')