Introduction

Good afternoon and welcome to the python part of this course! Today we will introduce how we will work with Python during this course and show some alternative methods. If you are unfamiliar with Python and/or feel that you need more training, follow one of the Datacamp courses as introduction into Python before today:

• Introduction to Python | recommended to follow if you haven’t any scripting experience so far
• Python for R users | recommended if you have experience already in R

Today’s Learning objectives

• Know how to work with virtual environments: Conda + Mamba
• Know how to run a Python script from the terminal
• Get introduced to Python editors and IDEs
• Refresh Python programming knowledge
• Familiarize yourself with some visualization techniques

Python package management with Conda

The high availability of packages is also a threat sometimes. If a piece of software is developed depending on a package, but this packages changes later on, the code might not work anymore. Also, different packages require different dependencies that they are built upon. It is important to make sure all these dependencies are working together and that the right versions are used. Luckily, a set of tools exist for installing and managing Python packages. It is possible to install packages on your main Python installation (called the base python interpreter), but sooner or later you will get conflicting Python packages since packages have varying dependencies and you might have installed several versions of the same package. It can even break your system Python interpreter.

Therefor, we recommend to use a Python package manager that uses of virtual environments, such as Conda or Mamba. That way, you can create a separate environment on your machine for each project. In these environments, any dependencie of the project, such as software, C libraries or R packages can be installed. We will use them here for installing Python packages. Packages installed in one environment do not interfere with your base Python or with other environments. Additionally, it is possible to export and share the requirements for your (open source) project with collaborators or users of your code.

Mamba installation

For this course, we will make use of Mamba, a fast drop-in reimplementation of the Conda package manager. It has its core parts implemented in C++ for maximum efficiency, makes use of parallel downloading of repository data and package files using multi-threading, and uses libsolv for (much) faster dependency solving. To install Mamba in your Linux environment, we have prepared a short Bash script for you. Just run the following lines of code, line by line, in a new terminal window.

git clone https://github.com/GeoScripting-WUR/InstallLinuxScript.git
cd InstallLinuxScript/user
chmod u+x ./install.sh
./install.sh

This will install Mamba into ~/mamba. Finally, restart your terminal to be able to use Mamba and Conda in the terminal. Next, let’s see how to use Mamba in case you want make new virtual environments by yourself, or install packages after creating the environment.

Mamba usage

Mamba creates isolated conda environments with sets of packages, that do not interfere with your base Python or with other conda environments. To create an environment:

mamba create --name geotest python numpy spyder

This creates a new environment called geotest with Python, NumPy and Spyder installed into the conda environment. Another option is to create an environment from a file, a YAML file. In this file all required packages are listed and if required which version should be used. An example of a YAML file is the following:

name: geotest
dependencies:
  - python
  - numpy
  - spyder

The first line defines what the evnironment will be called (geotest in this example) and what packages should be installed (python, numpy and spyder). As you can see, this definition of the geotest environment is the exact same as the geotest environment as defined before. To create the geotest environment from such a file, save this yaml to a new file named env.yaml, or however you want to call it and use the argument --file (or --f in short):

mamba env create -f env.yaml

Let’s list the currently available environments:

mamba info --envs

Mamba puts an asterisk (*) in front of the active environment. Now we activate the environment. While Mamba replaces Conda for most commands, this is not the case for (de)activating environments:

# Cross-platform (but not always working, like in our VM, so we use the next option)
conda activate geotest

# Linux, macOS
source activate geotest

# Windows
activate geotest

After this, the current environment is shown in parentheses in front of your prompt ((geotest)$). Note that the activated environment is only valid for the shell in which you activated it. For instance, if you close the shell window and open a new one you will have to activate it again.

After creating a conda environment, (additional) Python packages can be installed. There are three possible ways to install packages, which we list below.

• Using Mamba to install and manage conda packages. This downloads conda packages using conda channels, which are URLs to directories containing the conda packages. Generally, installing conda packages using Mamba is the preferred method.
• Using pip to install packages and Mamba to manage these packages. pip is available for Windows, macOS and Linux. pip can also install binary wheels on Windows. You should generally not install packages from pip in a conda environment unless it’s the last resort. This is because after you use pip to modify an environment, you can no longer use conda/mamba to do so (trying that will break your environment, because pip does not communicate its changes to Conda). Hence install packages with mamba that you can first, and only then use pip, and then never touch the environment with mamba again (delete and start fresh if you need to).
• Using the distribution’s package manager (only on Ubuntu, that is sudo apt-get install python-*).

The mamba search command searches a set of channels. By default, packages are automatically downloaded and updated from the default channel. To search for a package, type:

mamba search pandas

This gives a list of all packages that have “pandas” in the name and lists all available versions. To install:

mamba install pandas

This installs the latest compatible version of Pandas. Note that this would install it into your currently activated environment.

Note that you can also install multiple packages at the same time:

mamba install geopandas matplotlib

As you saw with Spyder (which is an IDE, more on that later), Mamba is also able to install some non-Python packages that have Python bindings. This is useful for making sure your Python and binary versions match and do not interfere with the system-wide ones.

Some additional helpful utilities for package management in this context are:

• mamba list to check which packages are installed in root or in the active environment;
• python --version or gdal-config --version to check which Python or GDAL version is used in the environment;
• which spyder or type spyder to find out which Spyder executable is used either from system or conda environment.

Removing packages is just as simple:

mamba remove geopandas pandas folium

Now, we deactivate the environment and return to base environment.

# Cross-platform
conda deactivate

# Linux, macOS
source deactivate

# Windows
deactivate

When we are finished, and do not need the environment for next time, we can remove the environment geotest.

mamba remove --name geotest --all

Running a Python script in the terminal

Within a conda environment, Python can be started directly, or can be called to run a script file. To start Python directly:

python

Now, you can type Python expressions that will be executed one by one:

import sys
print('Good morning, you are running Python:', sys.version)

To go back, type:

exit()
# or 
quit()

Usually, we do not want to run expressions one by one, but build scripts instead, to ensure transferability and reproducibilty. Create a new text file and (re)name it (to) test.py. Open it, for example with a text editor, paste in the code you used above (import sys etc.), and save the script. Navigate in the terminal to the location where this script is stored, using cd. Finally, run the script with:

python test.py

The output is printed to the terminal. Running a script from the terminal is less error-prone than running it from an IDE (see the next section), such as Spyder, as IDEs often keep variables in memory after the script has finished running. Therefore, running a script from the terminal is a good final test before submitting an exercise or assignment.

Python editors and IDEs

There are many Integrated Development Environments (IDEs) for Python, and every programmer has their own preference. An IDE is a software application that provides facilities for software development.

• Spyder is a lightweight IDE. In this course, Spyder is the recommended Python IDE.
• Jupyter notebook integrates visualization with code and is suitable to make tutorials, simple dashboards, quick visualizations, and do prototype testing. Jupyter Notebook run in your browser on a localhost server or on a web server. They allow for various programming languages, e.g. Python, R, Julia, Spark or PySpark.
• PyCharm Community Edition is a free professional Python IDE with a lot of advanced functionality, such as integrated GIT version control, code completion, code checking, debugging and navigation. This IDE can optionally be used by more advanced scripters during this course instead of Spyder, but do know that you will not be assisted for solving IDE-related issues.

spyder

jupyter notebook

import folium

m = folium.Map(location=[51.9700000, 5.6666700], zoom_start=13)
m

conda deactivate

Setting up the environment

Now that we know how to set up an environment and run code, lets use this new knowledge and run some Python code. Again, During this course advise you to code in Spyder, as this IDE is the recommended IDE for the Python part of this course. To practice you might also want to try out Jupyter locally and Google Colab to run the same code.

First, make a directory structure for this tutorial:

cd ~/Documents/
mkdir PythonRefresher #or give the directory a name to your liking
cd ./PythonRefresher
mkdir output

We only make a directory for output, because no input data or separate scripts are created in this tutorial. Next, we will create a conda environment from a file. First create a text file in your preferred text editor, e.g. gedit. Then, (re)name it (to) refresher.yaml, and copy the following content into the file:

name: refresher
dependencies:
  - python
  - numpy
  - matplotlib
  - geopandas
  - spyder

Now, create a new conda environment based on this file:

mamba env create --file refresher.yaml

Once everything is installed, activate the environment and start Spyder:

source activate refresher
spyder

Create a new Python script and save it.

Important to note: for compatibility, it is best to install packages from the same channel as much as possible. Given that packages in the file refresher.yaml are installed from the conda-forge channel, it is wise to use this same channel when you want to install additional packages in your environment.

Quick refresher

In the tutorial about R and Python we have gone over the differences and similarities of python and R. This tutorial also contains some basic python syntax, in this tutorial we assume you know this content, but we will go over a few basics here as well. The examples below are mostly meant for reference purposes, we assume you understand most of this refresher already.

# Integer
age = 25

# Float
height = 1.75

# String
name = "John Doe"

# Boolean
is_student = True

# Print a name
print(name)

# String formatting and printing 
print(f'{name} is {age} years old and is {height} meters tall.')

a = 10
b = 5

addition = a + b
subtraction = a - b
multiplication = a * b
division = a / b
modulo = a % b
exponentiation = a ** b

print(addition, subtraction, multiplication, division, modulo, exponentiation)

x = 15

if x > 10:
    print("x is greater than 10")
elif x == 10:
    print("x is equal to 10")
else:
    print("x is less than 10")

# For loop
for i in range(5):
    print(i)

# While loop
count = 0
while count < 5:
    print(count)
    count += 1

# Creating a list
fruits = ["apple", "banana", "orange"]

# Accessing elements
print(fruits[0])  # Output: "apple"

# Adding elements
fruits.append("grape")

# Removing elements
fruits.remove("banana")

# Length of the list
print(len(fruits))  # Output: 3

# Function to add two numbers and return the result
def add_numbers(a, b):
    return a + b

result = add_numbers(5, 3)
print(result)  # Output: 8

# Creating a dictionary
person = {
    "name": "Alice",
    "age": 30,
    "is_student": False
}

# Accessing values
print(person["name"])  # Output: "Alice"

# Adding a new key-value pair
person["occupation"] = "Engineer"

# Removing a key-value pair
del person["is_student"]

import pandas as pd

data = {
    'Name': ['Alice', 'Bob', 'Charlie'],
    'Age': [25, 30, 22],
    'City': ['New York', 'San Francisco', 'Chicago']
}

df = pd.DataFrame(data)
print(df)

# Display the first few rows of the DataFrame
print(df.head())

# Get statistical information about the DataFrame
print(df.describe())

# Access a specific column
print(df['Age'])

import geopandas as gpd

# Dummy data for the GeoDataFrame
data = {
    'Name': ['Location A', 'Location B', 'Location C'],
    'Latitude': [40.7128, 34.0522, 41.8781],
    'Longitude': [-74.0060, -118.2437, -87.6298]
}

# Create the GeoDataFrame with a single line of code
gdf = gpd.GeoDataFrame(data, geometry=gpd.points_from_xy(data['Longitude'], data['Latitude']))

# Display the GeoDataFrame
print(gdf)