They produce items one at a time and only when needed, which makes them the best choice for working with large datasets or streams of data where it would be inefficient and impractical to load everything into memory at once.

How to Define and Use Generators

To define a generator, you can use the def keyword like you would with a normal function. However, instead of returning a value with return, we use yield.

Here, the yield keyword is used to produce a value and pause the execution of the generator function. When the function is resumed, it continues execution immediately after the yield statement.

Example: Simple Generator

Here is a simple generator that yields the first n numbers:

def simple_generator(n):
    i = 0
    while i < n:
        yield i
        i += 1

# Using the generator
gen = simple_generator(5)
for number in gen:
    print(number)

Output:

0
1
2
3
4

When the simple_generator() function is called, it doesn't execute its code. Instead, it returns a generator object that contains an internal method named __next__(), which is created when the generator function is called.

The generator object implicitly uses this method as an iterator protocol when we iterate over the generator.

Benefits of Using Generators

Python generators offer several advantages that significantly enhance code efficiency and readability. By efficiently producing items on-the-fly, generators optimize memory usage and enhance performance compared to traditional iterable methods.

Let's explore some these benefits in detail, highlighting how generators streamline Python development and improve code quality.

Memory Optimization

Compared to lists that load all elements into memory at once, generators are memory usage optimizers. They produce items one at a time and only when required.

Here is an example that considers a scenario where we need to generate a large list of numbers:

# Using a list
numbers = [i for i in range(1000000)]

# Using a generator
def number_generator():
    for i in range(1000000):
        yield i

gen_numbers = number_generator()

With the list, all 1000000 numbers are stored in memory at once, but with the generator, numbers are produced one at a time, reducing memory usage.

Enhanced Performance

Since generators produce items on-the-fly, they enhance performance, particularly in terms of speed and efficiency. They can start yielding results immediately without waiting to process an entire dataset.

In this example, let's assume that we need to process each number in a sequence:

# Using a list
numbers = [i for i in range(1000000)]
squared_numbers = [x**2 for x in numbers]

# Using a generator
def number_generator():
    for i in range(1000000):
        yield i

def squared_gen(gen):
    for num in gen:
        yield num**2

gen_numbers = number_generator()
squared_gen_numbers = squared_gen(gen_numbers)

When we use the list, we generated all the numbers and then process them, which takes more time. However, with the generator, each number is processed as soon as it is generated, making the process more efficient.

Code Simplicity and Readability

Generators help in writing clean and readable code. They allow us to define an iterative algorithm in a simple way, without the need for boilerplate code to manage the state of the iteration. Let's consider a scenario where we need to read lines from a large file:

# Using a list
def read_large_file(file_name):
    with open(file_name, 'r') as file:
        lines = file.readlines()
    return lines

lines = read_large_file('large_file.txt')

# Using a generator
def read_large_file(file_name):
    with open(file_name, 'r') as file:
        for line in file:
            yield line

lines_gen = read_large_file('large_file.txt')

With the list approach, we read all lines into memory at once. With the generator, we read and yielded one line at a time, which makes the code simpler and more readable while contributing to saving memory.

Practical Use Cases

This section explores some practical use cases where Python generators excel, discovering how generators simplify complex tasks while optimizing performance and memory usage.

Stream Processing

Generators are excellent at handling continuous streams of data, like real-time sensor data, log streams, or live feeds from APIs. They provide efficient processing of data as it becomes available, without the need to store large amounts of data in memory.

import time

def data_stream():
    """Simulate a data stream."""
    for i in range(10):
        time.sleep(1) # Simulate data arriving every second
        yield 1


def stream_processor(data_stream):
    """Process data from the stream."""
    for data in data_stream:
        processed_data = data * 2 # Example processing step
        yield processed_data


# Usage
stream = data_stream()
processed_stream = stream_processor(stream)
for data in processed_stream:
    print(data)

In this example, the data_stream() method generates data at intervals, simulating a continuous data stream. The stream_processor() processes each piece of data as it arrives, demonstrating how generators can handle streaming data efficiently without the need to load all data into memory at once.

Iterative Algorithms

Generators provide a straightforward way to define and execute iterative algorithms that involve repetitive calculations and simulations. They allow us to maintain the state of the iteration without manually managing loop variables, which can enhance code clarity and maintainability.

def fibonacci_generator():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b


# Usage
fib_gen = fibonacci_generator()
for i in range(10):
    print(next(fib_gen))

In the example above, the fibonacci_generator() method defines a generator that produces Fibonacci numbers indefinitely. It returns each Fibonacci number one at a time, starting from 0 and 1.

Here, the for loop iterates 10 times to print the first 10 Fibonacci numbers, demonstrating how generators can efficiently generate and manage sequences without preloading them into memory.

Real-Time Simulator

In this example, we will simulate real-time updates of a stock price. The generator will produce a new stock price at each step, based on the previous price and some random fluctuation.

import random
import time

def stock_price_generator(initial_price, volatility, steps):
    """Generates stock prices starting from initial_price with given volatility."""

    price = initial_price
    for _ in range(steps):
        # Simulate price change
        change_percent = random.uniform(-volatility, volatility)
        price += price * change_percent
        yield price
        time.sleep(1) # Simulate real-time delay

# Create the stock price generator
initial_price = 100.0 # Starting stock price
volatility = 0.02 # Volatility as a percentage
steps = 10 # Number of steps (updates) to simulate

stock_prices = stock_price_generator(initial_price, volatility, steps)

# Simulate recieving and processing real-time stock prices
for price in stock_prices:
    print(f"New stock price: {price:.2f}")

This example generates each stock price on-the-fly based on the previous price and doesn't store all generated prices in memory, making it efficient for long-running simulations.

The generator provides a new stock price at each step with minimal computation. The time.sleep(1) simulates a real-time delay, allowing the system to handle other tasks concurrently if needed.

Summary

In summary, Python generators offer efficient memory management and enhanced performance, simplifying code while tackling diverse tasks like stream processing, iterative algorithms, and real-time simulation.

Their ability to optimize resources makes them a valuable tool for modern Python developers seeking elegant and scalable solutions.

Hopefully, this exploration of Python generators provides you with the insights needed to leverage their full potential. If you have any questions or want to discuss further, feel free to reach out to me on LinkedIn. Additionally, you can subscribe to my YouTube channel where I share videos on coding techniques and projects I'm working on.

This small difference can cause issues if you are working on a project and you want other developers who come from different operating systems to expand your code.

Fortunately, if you're coding in Python, the Pathlib module does the heavy lifting by letting you make sure that your file paths work the same in different operating systems. Also, it provides functionalities and operations to help you save time while handling and manipulating paths.

Prerequisites

Pathlib comes as default with Python >= 3.4. However, if you are using a version of Python lower than 3.4, you won't have access to this module.

How Does Pathlib Work?

To understand how you can construct a basic path using Pathlib, let's create a new Python file called example.py and put it inside a particular directory.

Open the file, and type the following content:

import pathlib

p = pathlib.Path(__file__)
print(p)

In this example, we import the Pathlib module. Then, we create a new variable called p to store the path. Here, we use the Path object from Pathlib with a built-in variable in Python called file to refer to the file path we are currently writing in it example.py.

If we print p, we will get the path to the file we are currently in:

/home/rochdikhalid/dev/src/package/example.py

As shown above, Pathlib creates a path to this file by putting this particular script in a Path object. Pathlib contains many objects such as PosixPath() and PurePath(), which we will learn more about in the following sections.

Before we jump into this, Pathlib divides the filesystem paths into two different classes that represent two types of path objects: Pure Path and Concrete Path.

PurePath() classes

The pure path provides utilities to handle and manipulate your file path without making writing operations, while the concrete path allows you to manipulate and do writing operations on your file path.

In other words, a concrete path is a subclass of a Pure path. It inherits manipulation from the parent class and adds input/output operations that do system calls.

Pure paths in Python

Pure paths manipulate a file path on your machine even if it belongs to a different operating system.

For example, let's say you are on Linux and you want to use a Windows file path. Here, Pure path class objects will help you get the path working on your machine with some basic operations like creating child paths or accessing individual parts of a path.

But pure paths won't be able to mimic some other operations like creating a directory or a file because you are not actually in that operating system.

How to use pure paths

As you can see in the diagram above, pure paths consist of three classes that handle any file system path on your machine:

PurePath() is the root node that provides handling operations to every path object in Pathlib.

When you instantiate PurePath(), it creates two classes to handle Windows paths and non-Windows paths. PurePath() creates a generic path object "agnostic path", regardless of the operating system you are running on.

In [*]: pathlib.PurePath('setup.py')                                            
Out[*]: PurePosixPath('setup.py')

PurePath() in the example above creates a PurePosixPath() because we assumed we are running on a Linux machine. But if you instantiate it on Windows you will get something like PureWindowsPath('setup.py').

PurePosixPath() is the child node of PurePath() implemented for non-Windows file system paths.

In [*]: pathlib.PurePosixPath('setup.py')                                            
Out[*]: PurePosixPath('setup.py')

You will not get any error if you instantiate PurePosixPath() on Windows because too simply this class doesn't make system calls.

PureWindowsPath() is the child node of PurePath() implemented for Windows file system paths.

In [*]: pathlib.PureWindowsPath('setup.py')                                     
Out[*]: PureWindowsPath('setup.py')

The same applies to PureWindowsPath() since this class doesn't provide system calls, so instantiating it will not raise any error for other operating systems.

Pure path properties

Each subclass in PurePath() provides the following properties:

PurePath().parent outputs the parent of the path:

In [*]: pathlib.PurePath('/src/goo/scripts/main.py').parent                     
Out[*]: PurePosixPath('/src/goo/scripts')

In the example above, we are using the .parent property to get the path of the logical parent of **main.py**.

PurePath().parents[] outputs the ancestors of the path:

In [*]: p = pathlib.PurePath('/src/goo/scripts/main.py')
        p.parents[0]               
Out[*]: PurePosixPath('/src/goo/scripts')

In [*]: p.parents[1]                
Out[*]: PurePosixPath('/src/goo')

You should always specify the ancestor index in the square brackets as seen above. In Python 3.10 and above, you can use slices and negative index values.

PurePath().name provides the name of the last component of your path:

In [*]: pathlib.PurePath('/src/goo/scripts/main.py').name                      
Out[*]: 'main.py'

In this example, the final path component is main.py. So, the .name property outputs the name of the file main.py which is main with its suffix .py.

On the other hand, PurePath().suffix provides the file extension of the last component of your path:

In [*]: pathlib.PurePath('/src/goo/scripts/main.py').suffix                    
Out[*]: '.py'

Compared to the .name property, the .suffix property outputs the file extension and excludes the file name.

PurePath().stem outputs only the name of the final component of your path without the suffix:

In [*]: pathlib.PurePath('/src/goo/scripts/main.py').stem                      
Out[*]: 'main'

As seen above, the .stem property excludes the suffix of the final component main.py and provides only the name of the file.

Pure path methods

Each subclass of PurePath() provides the following methods:

PurePath().is_absolute() checks whether your path is absolute or not:

In [*]: p = pathlib.PurePath('/src/goo/scripts/main.py')
        p.is_absolute()

Out[*]: True

In [*]: o = pathlib.PurePath('scripts/main.py')
        o.is_absolute()

Out[*]: False

Note that the absolute path consists of a root and drive name. In this case, PurePath() doesn't allow us to know the drive's name.

If you use PureWindowsPath(), you can represent an absolute path that contains a drive name like PureWindowsPath('c:/Program Files').

PurePath().is_relative() checks whether the path belongs to the other given path or not:

In [*]: p = pathlib.PurePath('/src/goo/scripts/main.py')
        p.is_relative_to('/src')

Out[*]: True

In [*]: p.is_relative_to('/data')

Out[*]: False

In this example, the given path /src is a part of or belongs to the path p, while the other given path /data raises False because it has no relative relationship with the path p.

PurePath().joinpath() concatenates the path with the given arguments (child paths):

In [*]: p = pathlib.PurePath('/src/goo')
        p.joinpath('scripts', 'main.py')

Out[*]: PurePosixPath('/src/goo/scripts/main.py')

Note that there is no need to add slashes in your given arguments, as the .joinpath() method handles this for you.

PurePath().match() checks whether the path matches a given pattern:

In [*]: pathlib.PurePath('/src/goo/scripts/main.py').match('*.py')
Out[*]: True

In [*]: pathlib.PurePath('/src/goo/scripts/main.py').match('goo/*.py')
Out[*]: True

In [*]: pathlib.PurePath('src/goo/scripts/main.py').match('/*.py')
Out[*]: False

Based on the examples above, the pattern should match the path. If the given pattern is absolute, the path must be absolute too.

PurePath().with_name() changes the name of the final component with its suffix:

In [*]: p = pathlib.PurePath('/src/goo/scripts/main.py')
        p.with_name('app.js')
Out[*]: PurePosixPath('/src/goo/scripts/app.js')

In [*]: p
Out[*]: PurePosixPath('/src/goo/scripts/main.py')

The .with_name() method doesn't change the name of the last component permanently. Also, if the given path doesn't contain a name, an error occurs as mentioned in the official documentation.

PurePath().with_stem() changes only the name of the final component of the path:

In [*]: p = pathlib.PurePath('/src/goo/scripts/main.py')
        p.with_stem('app.py')
Out[*]: PurePosixPath('/src/goo/scripts/app.py')

In [*]: p
Out[*]: PurePosixPath('/src/goo/scripts/main.py')

This is similar to the .with_name() method. The .with_stem() changes the name of the last component temporarily. Also, if the given path doesn't contain a name, an error will occur.

PurePath().with_suffix() temporarily changes the suffix or the extension of the final component of your path:

In [*]: p = pathlib.PurePath('/src/goo/scripts/main.py')
        p.with_suffix('.js')
Out[*]: PurePosixPath('/src/goo/scripts/main.js')

If the name of the given path contains no suffix, the .with_suffix() method adds the suffix for you:

In [*]: p = pathlib.PurePath('/src/goo/scripts/main')
        p.with_suffix('.py')
Out[*]: PurePosixPath('/src/goo/scripts/main.py')

But, if we don't include the suffix and we keep the argument empty '', the current suffix will be removed.

In [*]: p = pathlib.PurePath('/src/goo/scripts/main')
        p.with_suffix('')
Out[*]: PurePosixPath('/src/goo/scripts/main')

Some methods like .with_stem(), and .is_relative_to() have been added recently to Python 3.9 and above. So, if you call these methods using Python 3.8 or lower, an attribute error is raised.

Concrete Paths in Python

Concrete Paths allows you to handle, manipulate, and do writing operations on different filesystem paths.

In the other words, this type of path object helps you to create for example a new file, a new directory, and do other input/output operations while not being in that operating system.

How to use concrete paths

Concrete paths handle any file system path and make system calls on your machine. Those path objects are the child paths of the pure paths and consist of three subclasses like the pure ones:

Path() is the child node of PurePath(), it provides handling operations with the ability to do writing operations on your path.

When you instantiate Path(), it creates two classes to handle Windows paths and non-Windows paths. Like PurePath(), Path() also creates a generic path object "agnostic path", regardless of the operating system you are running on.

In [*]: pathlib.Path('setup.py')                                            
Out[*]: PosixPath('setup.py')

Path() in the example above creates a PosixPath() because we assume we are running on a Linux machine. But if you instantiate it on Windows you will get something like WindowsPath('setup.py')

PosixPath() is the child node of Path() and PurePosixPath(), implemented to handle and manipulate non-Windows file system paths.

In [*]: pathlib.PosixPath('setup.py')                                            
Out[*]: PosixPath('setup.py')

You will get an error if you instantiate PosixPath() on a Windows machine because you cannot make system calls while running on a different operating system.

WindowsPath() is the child node of Path() and PureWindowsPath() implemented for Windows file system paths.

In [*]: pathlib.WindowsPath('setup.py')                                     
Out[*]: WindowsPath('setup.py')

The same applies to WindowsPath() since you are running on a different operating system – so instantiating it will raise an error.

Properties of concrete paths

Since the concrete path is the subclass of the pure path, you can do everything with concrete paths using the PurePath() properties. This means that we can use, for example, the .with_suffix property to add a suffix to a concrete path:

In [*]: p = pathlib.Path('/src/goo/scripts/main')
        p.with_suffix('.py')
Out[*]: PosixPath('/src/goo/scripts/main.py')

Or, you can check if a given path is relative to the original path:

In [*]: p = pathlib.Path('/src/goo/scripts/main.py')
        p.is_relative_to('/src')

Out[*]: True

Always remember that concrete paths inherit handling operations from the pure paths and add writing operations that do system calls and input/output configurations.

Methods of concrete path

Each subclass of Path() provides the following methods to handle paths and do system calls:

Path().iterdir() returns the content of a directory. Let's say we have the following folder that contains the following files:

data
    population.json
    density.json
    temperature.yml
    stats.md
    details.txt

To return the content of /data directory, you can use .iterdir() method here:

In [*]: p = pathlib.Path('/data')

        for child in p.iterdir():
            print(child)

Out[*]: PosixPath('/data/population.json')
         PosixPath('/data/density.json')
         PosixPath('/data/temprature.yml')
         PosixPath('/data/stats.md')
         PosixPath('/data/details.txt')

The .iterdir() method creates an iterator that lists the files randomly.

Path().exists() checks whether the file/directory exists in a current path. Let's use the directory of the previous example (our current directory is /data):

In [*]: p = pathlib.Path('density.json').exists()
        p
Out[*]: True

The .exists() method returns True because the given file exists in the data directory. The method returns False if the file doesn't exist.

In [*]: p = pathlib.Path('aliens.py').exists()
        p
Out[*]: False

The same applies to directories, the method returns True if the given directory exists and returns False if it does not.

Path().mkdir() creates a new directory at a given path:

In [*]: p = pathlib.Path('data')
        directory = pathlib.Path('data/secrets')
        directory.exists()
Out[*]: False

In [*]: directory.mkdir(parents = False, exist_ok = False)
        directory.exists()
Out[*]: True

According to the official documentation, the .mkdir() method takes three arguments. We will focus only at the moment on parents and exist_ok.

Both arguments are set to False as default. The parent raises a FileNotFound error in case of a missing parent, while the exist_ok raises a FileExists error if the given directory already exists.

In the example above, you can set the arguments to True to ignore the mentioned errors and update the directory.

We can also create a new file at a given path using the Path().touch() method:

In [*]: file = pathlib.Path('data/secrets/secret_one.md')
        file.exists()
Out[*]: False

In [*]: file.touch(exist_ok = False)
        file.exists()
Out[*]: True

The same logic applies to the .touch() method. Here, the exist_ok can be set to True to ignore the FileExists error and update the file.

Path().rename() renames the file/directory at a given path. Let's take an example using our directory /data:

In [*]: p = pathlib.Path('density.json')
        n = pathlib.Path('density_2100.json')
        p.rename(n)
Out[*]: PosixPath('density_2100.json')

If you assign a non existing file to the method, it raises a FileNotFound error. The same applies to directories.

Path().read_text() returns the content of a file in string format:

In [*]: p = pathlib.Path('info.txt')
        p.read_text()

Out[*]: 'some text added'

Also, you can use the **write_text()** method to write a content in a file:

In [*]: p = pathlib.Path('file.txt')
        p.write_text('we are building an empire')

Out[*]: 'we are building an empire'

Note that the .write_text() method has been added to Python 3.5 and was updated recently in Python 3.10 to have some additional parameters.

Important note

You might ask yourself why you need to use Windows file system paths – because every package should be compatible with other operating systems, not Windows only.

You're right if the goal is to make an OS-agnostic path. But, sometimes we can't do this due to some settings that are unique to Windows or Posix systems. That's why those objects are available to help developers deal with those use cases.

Some packages target problems only present in the Windows ecosystem, and Python accommodates those use cases in this library.

What next?

Hopefully, this tutorial helps you learn how and why to use Pathlib and how it is useful to handle and manipulate filesystem paths.

It would be great to play around with what you learned and turn things into a real project. If you have any questions, feel free to connect and hit me up at any time on LinkedIn.

Also, you can take a look at my channel on YouTube where I share videos on what I'm learning and building with code.

See you in the next tutorial, and keep moving forward!

References

There is a lot to know. In this blog post, I covered the basics you need to use Pathlib in your project.

The official documentation highlights more methods and properties that you can apply to your filesystem paths:

• Pathlib — Object-oriented filesystem paths

Well, I got confused and a bit stressed trying to find a simple and easy tutorial I could use to get started. For this reason, I decided to write this tutorial documenting how I built my first Python package.

What is a package in Python?

Before we get started, we should know a package means in Python.

A Python package is a directory that contains a bunch of modules with a dependency file called __init__.py. This file can be completely empty and you use it to mark the directory on a disk as a Python package.

The following shows an example of a package directory:

package
    __init__.py
    module_a.py
    module_b.py
    module_c.py

The __init__.py is a dependency file that helps Python look for the available modules in our package directory. If we remove this file, Python will fail to import our modules.

Packages vs modules in Python

You should now understand that Python packages create a structured directory with several modules, but what about modules? Let's make sure we understand the difference between a package and a module:

A Module always corresponds to a single Python file turtle.py. It contains logic like classes, functions, and constants.

A Package is basically a module that could contain many modules or sub-packages.

Package structure

Packages don't only contain modules, though. They consist of top-level scripts, documentation, and tests, as well. The following example shows how a basic Python package can be structured:

package_name/
    docs/
    scripts/
    src/
        package_a
            __init__.py
            module_a.py
        package_b
            __init__.py
            module_b.py
    tests/
        __init__.py
        test_module_a.py
        test_module_b.py
    LICENSE.txt
    CHANGES.txt
    MANIFEST.in
    README.txt
    pyproject.toml
    setup.py
    setup.cfg

Let's understand what each file in the tree above is used for:

• package_name: represents the main package.
• docs: includes documentation files on how to use the package.
• scripts/: your top-level scripts.
• src: where your code goes. It contains packages, modules, sub-packages, and so on.
• tests: where you can put unit tests.
• LICENSE.txt: contains the text of the license (for example, MIT).
• CHANGES.txt: reports the changes of each release.
• MANIFEST.in: where you put instructions on what extra files you want to include (non-code files).
• README.txt: contains the package description (markdown format).
• pyproject.toml: to register your build tools.
• setup.py: contains the build script for your build tools.
• setup.cfg: the configuration file of your build tools.

Note that there are two options for how to include our test files in our main package. We can keep it at the top level as we did above or put it inside the package like the following:

package_name/
      __init__.py
      module_a.py
      module_b.py
      test/
          __init__.py
          test_module_a.py
          test_module_b.py

In my opinion, I think keeping tests at the top level can help a lot especially when our tests require reading and writing other external files.

Should you use setup.cfg or setup.py?

The setup.py and setup.cfg are the default packaging tools within PyPI, setuptools, pip, and the standard python library.

Here, they represent the configuration and build scripts for setuptools. They both tell the setuptools how the package can be built and installed.

The mentioned file contains information like the version, packages, and files to include, along with any requirements.

The following shows an example of setup.py that uses some setup() arguments. You can find more arguments here:

import setuptools

with open("README.md", "r", encoding = "utf-8") as fh:
    long_description = fh.read()

setuptools.setup(
    name = "package-name",
    version = "0.0.1",
    author = "author",
    author_email = "author@example.com",
    description = "short package description",
    long_description = long_description,
    long_description_content_type = "text/markdown",
    url = "package URL",
    project_urls = {
        "Bug Tracker": "package issues URL",
    },
    classifiers = [
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    package_dir = {"": "src"},
    packages = setuptools.find_packages(where="src"),
    python_requires = ">=3.6"
)

The setup.cfg is written in a different format compared to setup.py, and contains basically two essential keys, command and options.

The command key represents one of the distutils commands, while the options key defines the options the command can support.

[command]
option = value

The following shows an example of setup.cfg that uses some metadata and options. You can find a variety of metadata and options here:

[metadata]
name = package-name
version = 0.0.1
author = name of the author
author_email = author@example.com
description = short package description
long_description = file: README.md
long_description_content_type = text/markdown
url = package url
project_urls =
    Bug Tracker = package issues url
classifiers =
    Programming Language :: Python :: 3
    License :: OSI Approved :: MIT License
    Operating System :: OS Independent

[options]
package_dir =
    = src
packages = find:
python_requires = >=3.6

[options.packages.find]
where = src

The setup.py and setup.cfg are both specific to setuptools. Also, the setup.cfg can safely be moved to pyproject.toml.

Here, the idea is that maybe one day we'll want to switch to other build systems like flit or poetry. In that case, all we'll need to do is change the build-system entry (setuptools for example) in pyproject.toml to something like flit or poetry rather than starting over from scratch.

Here you can find information about other tools that build and distribute Python packages.

No matter which configuration file we chose, we are "locked-in" to maintaining that particular configuration file, either pyproject.toml, setup.cfg, or setup.py.

According to the Python Packaging User Guide, setup.cfg is preferred because it's static, clean, easier to read, and avoids encoding errors.

How to build your first Python package

Now, it's time to start building a simple Python package. We will use setuptools as a build system and we will configure our project using setup.cfg and pyproject.toml.

Set up the package files

For this simple package, we need to structure our directory by adding the dependency files needed to get the package ready for distribution. This is how to structure our package:

basicpkg/
    src/
        divide
            __init__.py
            divide_by_three.py
        multiply
            __init__.py
            multiply_by_three.py
    tests/
        __init__.py
        test_divide_by_three.py
        test_multiply_by_three.py
    LICENSE.txt
    README.txt
    pyproject.toml
    setup.cfg

Our main package consists of two packages: the first one to divide numbers by three, and the other to multiply numbers by three.

Also, we ignore some files like CONTEXT.txt, MANIFEST.in, and the docs/ directory to keep things simple at the moment. But we need the test/ directory to include our unit tests to test the package's behaviors.

Add some logic to our modules

As always, we will keep the __init__.py empty. Then, we need to put some logic in our modules to perform our operations.

For the divide package, let's add the following content into divide_by_three.py to divide any number by three:

def divide_by_three(num):
    return num / 3

The same logic applies to multiply_by_three.py inside the multiply package. But, this time is to multiply any number by three:

def multiply_by_three(num):
    return num * 3

Feel free to add more packages and modules to perform other kinds of operations. For example, you can add packages to do addition and subtraction tasks.

Test our modules

It's good to practice adding automated tests to any program we create. We will use unittest to test our modules and the package's behavior.

Inside the test/ directory, let's add the following code to test_divide_by_three.py:

import unittest
from divide.by_three import divide_by_three 

class TestDivideByThree(unittest.TestCase):

    def test_divide_by_three(self):
        self.assertEqual(divide_by_three(12), 4)

unittest.main()

We imported TestCase from unittest to perform our automated testing. Then, we imported our division method divide_by_three() from by_three module that is located inside the divide package.

If we remove the __init__.py here, Python will no longer be able to find our modules.

The .assertEqual() is used to check the equality of the two values above (divide_by_three(12) and 4). The unittest.main() is instantiated to run all our tests.

The same logic applies to test_multiply_by_three.py:

import unittest
from multiply.by_three import multiply_by_three

class TestMultiplyByThree(unittest.TestCase):

    def test_multiply_by_three(self):
        self.assertEqual(multiply_by_three(3), 9)

unittest.main()

To run the tests, type the following in your terminal/command:

For Linux:

python3 tests/test_divide_by_three.py

For Windows:

py tests/test_divide_by_three.py

Do the same to test the multiply module. If our tests run successfully, you should get the following:

.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

If you add extra packages and modules, try to add some unittest methods to them. This is going to be a good challenge for you.

Configure metadata using setup.cfg

Next, we need to add a configuration file for setuptools. As mentioned before, this config file will tell setuptools how our package can be built and installed.

So, we need to add the following metadata and options to our setup.cfg. Then, don't forget to choose a different name because I already uploaded this package with the name below to TestPyPi. Also, change other information like author, email, and project URLs to distribute the package with your info.

[metadata]
name = basicpkg
version = 1.0.0
author = your name
author_email = your email
description = A simple Python package
long_description = file: README.md, LICENSE.txt
long_description_content_type = text/markdown
url = https://gitlab.com/codasteroid/basicpkg
project_urls =
    Bug Tracker = https://gitlab.com/codasteroid/basicpkg/-/issues
    repository = https://gitlab.com/codasteroid/basicpkg
classifiers =
    Programming Language :: Python :: 3
    License :: OSI Approved :: MIT License
    Operating System :: OS Independent

[options]
package_dir =
    = src
packages = find:
python_requires = >=3.6

[options.packages.find]
where = src

You should just keep everything as default in the options category. The package_dir locates the root package where your packages, modules, and all your Python source files are located.

In the packages key, we can list our packages manually like this [divide, multiply]. But if we want to get all the packages, we can use find: and specify where we need to find these packages by using [options.packages.find] with the where key assigned to the name of the root package.

Always make sure to include the classifiers key in your configuration file to add some important information like the version of Python and the operating system our package is suitable for. You can find the complete list of classifiers here.

Create pyproject.toml

We will be using setuptools as a build system. To tell pip or other build tools about our build system, we need two variables, as seen below.

We use build-system.require to include what we need to build our package, while build-system.build-back-end defines the object that will perform the build.

So, let's enter the following content in pyproject.toml:

[build-system]
requires = ['setuptools>=42']
build-backend = 'setuptools.build_meta'

Note that you can safely move all configuration settings in setup.cfg to pyproject.toml if you decide to change the build system to flit or poetry, for example. This will save you time.

Create the README.md

Creating a good README.md is important. Let's add a description to our package, and include some instructions on how to install it:

# `basicpkg`

The `basicpkg` is a simple testing example to understand the basics of developing your first Python package.

We can also add how to use our package like this:

from multiply.by_three import multiply_by_three
from divide.by_three import divide_by_three

multiply_by_three(9)
divide_by_three(21)

Feel free to add any information that can help other devs understand what your package is used for and some instructions on how to install it and work properly with it.

Note that our configuration file will load the README.md and will be included when we distribute our package.

Add a license

It's very important to add a LICENSE to your project to let users know how they can use your code. Let's choose an MIT license for our Python package and add the following content to LICENSE.txt:

MIT License

Copyright (c) [year] [fullname]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Don't forget to replace [year] with the current year and [full name] with your name or the names of the copyright holders.

Generate the distribution archives

One more step left to get our package ready for distribution: generating the distribution archives for our Python package. To do so, first we need to update our PyPA's build and then generate the source and built archives.

In the terminal/cmd, run the following commands from the same directory where the pyproject.toml file is located:

For Linux:

python3 -m pip install --upgrade build
python3 -m build

For Windows:

py -m pip install --upgrade build
py -m build

Once the process above is completed, a new directory is generated called dist/ with two files in it. The .tag.tz file is the source archive and the .whl* file is the built archive. These files represent the distribution archives of our Python package which will be uploaded to the Python Package Index and installed by pip in the following sections.

How to upload a package in Python

Python Package Index is where we should upload our project. Since our Python package is in test and we might add other functionalities to experiment with it, we should use a separate instance of PyPI called TestPyPI. This instance is specifically implemented for testing and experimentation. Then, once your package is ready and meets your expectations, you can upload it to PyPI as a real package.

Let's follow the instructions below to get our TestPyPI ready to upload our package:

1. Go to TestPyPI and create an account.
2. Verify your email address so you can upload packages.
3. Update your profile settings (add your picture and so on).
4. Go to api-tokens and create your API token to securely upload your packages.
5. On the same page, set the scope to "entire account".
6. Copy and save your token in a safe place on your disk.

Next, we need to upload our distribution archives. To do so, we have to use an upload tool to upload our package. The official PyPI upload tool is twine, so let's install twine and upload our distribution archives under the dist/ directory.

In the terminal/cmd, run the following commands from the same directory where the pyproject.toml file is located:

For Linux:

python3 -m pip install --upgrade twine
python3 -m twine upload --repository testpypi dist/*

For Windows:

py -m pip install --upgrade twine
py -m twine upload --repository testpypi dist/*

Then, enter __token__. as username, and the token (pypi- prefix included) you saved as a password. Press Enter to upload the distributions.

How to install the uploaded Python package

Now, it's time to install our package. You can create a new virtual environment and use pip to install it from TestPyPI:

For Linux:

python3 -m venv env
source env/bin/activate
(env) python3 -m pip install --index-url https://test.pypi.org/simple/ --no-deps basicpkg

For Windows:

py -m venv env
.\env\Scripts\activate
(env) py -m pip install --index-url https://test.pypi.org/simple/ --no-deps basicpkg

Make sure your virtual environment is activated before you verify if our package works properly.

In the terminal/command, run python3 for Linux users or run py for Windows users. Then, type the following code to make sure that the multiply and divide packages work as expected:

from multiply.by_three import multiply_by_three
from divide.by_three import divide_by_three

multiply_by_three(9)
divide_by_three(21)

# Output: 27
# Output: 7

Remember that we need to import the methods from our modules that we need to perform the desired operations, as we did above.

Hooray! Our package works as expected.

So, once you test and experiment with your package, follow the instructions below to upload your package to the real PyPI:

1. Go to PyPI and create an account.
2. Run twine upload dist/* in the terminal/command line.
3. Enter the account credentials you registered for on the actual PyPI.
4. Then, run pip install [package_name] to install your package.

Congratulations! Your package was installed from the real PyPI.

What's next?

It would be great if you come up with a simple idea and build with it your first real Python package. In this blog post, I focused only on the basics you need to get started, but there is a lot to learn in the world of packaging.

Hopefully, my first experience with developing Python packages helps you learn what you need to get building. If you have any questions, feel free to connect and hit me up at any time on LinkedIn. Also, you can subscribe to my channel on YouTube where I share videos on what I'm learning and building with code.

See you in the next post and keep moving forward!

References

Here are some references that helped me develop my first Python package:

• Packaging Python Projects
• Configuring metadata
• PEP 621 – Storing project metadata in pyproject.toml
• Glossary - Python Packaging User Guide