This is a hands-on tutorial, which will help you form your early GUI projects. It's meant for anyone who wants to start building visual projects in Python.

The Tkinter library is a standard built-in Python library which helps us make Graphical User Interfaces in Python. Since it's a built-in library, we don't have to separately install it. So, once you have Python installed on your computer, you just have to set it up and you're good to follow along here.

But keep in mind that Tkinter may not be installed with your Python from the distributor end. To check if it's installed or not, open your command prompt and type:

python -m tkinter

This will open up a Tkinter specimen window if Tkinter is installed and working on your computer.

Table of Contents

• Prerequisites

• What Do We Want to See in Our Project?

• How to Set Up the Window

• How to Name the Window

• How to Create Frames in the Window

• How to Add Buttons to the Window

• How to Add the Output Screen of the Calculator

• How to Make the Numbers Visible on the Output Screen

• How to Add a Scrollbar to the Output Screen

• How to Add the Equal To Button

• How to Add the AC Button

• Wrapping Up

Prerequisites

Before starting, here are some prerequisites for this tutorial which will help you get the most out of it:

• Basic Python Syntax

• Understanding of how to import and use libraries and its different functions

• Understanding of how to use different attributes of the module

Now that we know what we need to proceed in this tutorial, let's actually dive-in the process!

The first step for building any project is to create a clear-cut idea of what you want to build. Let's look at what we're going to make.

What Do We Want to See in Our Project?

We're going to build a simple arithmetic calculator. The calculator works as follows:

• It has all the numerals (0, 1, 2, ...., 9) in a keyboard.

• It has basic arithmetic (+, -, /, *, =) operators lining the keyboard.

• The calculator is non-resizable, that is the user can't extend the width or the height of the application window.

• The calculator has a screen above the keyboard which shows the user input and the final answer.

• Finally, the calculator has an 'AC' button which stands for 'All Clear' which erases everything on the output screen of the window and allows the user to use it again.

With this, we have a clear idea about what we're going to build.

Also, you can create the UI beforehand and place the widgets accordingly on the window. Here's an image of the UI we'll create in this tutorial:

How to Set Up the Window

To set up our main window where we'll later add our widgets, first we need to import the Tkinter library into our program. Then we'll initialize the window using the tk.Tk() function. To display the window on the screen continuously until we quit manually, we'll use the mainloop() function. Here's what the code looks like:

import tkinter as tk

# screen initialization
root = tk.Tk()

# This keeps the window active
root.mainloop()

Theroot variable represents our window. So, from now on, we'll be adding the widgets to this window.

When a user hits "Run", you'll see a blank window on your screen as shown in the image below. Congrats! This is your first GUI.

How to Name the Window

The 'tk' written on the Title Bar is the default title of the window. To set our own window title, we can use the title() function. The following code shows how you can do that:

import tkinter as tk

root = tk.Tk()

# Naming the window
root.title("Calculator")

root.mainloop()

On executing the program, we get the following window:

Now you should be able to see that the title of the window changed successfully.

How to Create Frames in the Window

After setting up the window, now we have to place the buttons on it. For placing the buttons, we need to create a container in which we'll put them.

The container could be the main window, but we'll avoid that for this project. This is because we want to place some buttons to the side of and below others to create our keyboard. To make it easier, we'll create Frame containers.

A Frame container represents a vertical column of the window. The initial dimension of the frame is 0 x 0. The frame resizes accordingly when we place a widget in it.

We'll create four frames in our window. The first frame will contain the buttons 1, 4, 7, and AC. The second frame will contain the buttons 2, 5, 8, and 0, the third frame will contain the buttons 3, 6, 9, and =, and the last frame will contain the buttons +, -, x, and / (just like in the UI shown above).

We can create frames in Tkinter using tk.Frame(). We'll pass the parent container for the Frame – that is, the main window in its argument. The following code should make it clear:

import tkinter as tk

# screen initialization
root = tk.Tk()

# Naming the window
root.title("Calculator")

# Creating Frames
frame1 = tk.Frame(root)
frame1.pack(side='left', anchor='n')
frame2 = tk.Frame(root)
frame2.pack(side='left', anchor='n')
frame3 = tk.Frame(root)
frame3.pack(side='left', anchor='n')
frame4 = tk.Frame(root)
frame4.pack(side='left', anchor='n')

# This keeps the window active
root.mainloop()

The pack() function embeds the Frame geometrically on the window. The side='left' parameter embeds the Frames to the extreme left of the screen. By default, this is set to the center. anchor='n' tells us that the widgets should be placed starting from the very top of the frame. By default, the widgets start adding from the center of the Frame. The 'n' in the anchor='n' stands for 'North'.

An important thing to note is that, since we defined frame1 early in the program, it will occupy the extreme left portion of the window. But even though frame2 is also set to occupy extreme left, the two frames frame1 and frame2 won't overlap. Instead frame2 will take a position so that it goes as far left as it can go on the window without overlapping frame1. So frames frame1, frame2, frame3 and frame4 are side by side on the left side of the window.

How to Add Buttons to the Window

We can create a button widget in Tkinter by using the tk.Button() function. The tk.Button() function consists of various parameters:

• master: This allows us to provide the parent container in which we have to place our button. This expects a container object.

• text: In this parameter, we have to pass the text which we want to display on our button. This expects a string.

• font: This expects a tuple with the first element providing the name of the font and the next element providing the font size.

• image: This allows us to put an image over our button.

• bg: This allows us to set the background colour for our button.

• fg: This allows us to set the foreground colour for our button.

• activebackground: When the button is clicked, the colour passed in this parameter becomes visible.

• command: This allows us to link a command to the button.

Now that we know the basics of creating a button, lets actually create the keyboard of our calculator.

To create the keyboard, we have to put quite a few buttons on the window. To make our work easier, we'll define a function to create our buttons, just with different text. Let's look at the code below:

import tkinter as tk

# screen initialization
root = tk.Tk()

# Naming the window
root.title("Calculator")

frame1 = tk.Frame(root)
frame1.pack(side='left', anchor='n')
frame2 = tk.Frame(root)
frame2.pack(side='left', anchor='n')
frame3 = tk.Frame(root)
frame3.pack(side='left', anchor='n')
frame4 = tk.Frame(root)
frame4.pack(side='left', anchor='n')

pixel = tk.PhotoImage(width=55, height=55)

def buttons(text, frame):
    button = tk.Button(frame, text=text, font=('Arial', 20), image=pixel, bg="#333300", fg="white", compound="center")
    return button


def buttons_ops(text, frame, bg, fg):
    button = tk.Button(frame, text=text, font=('Arial', 20), image=pixel, bg=bg, fg=fg, activebackground="black",
                        compound="center")
    return button

btn1 = buttons('1',frame1).pack()
btn4 = buttons('4', frame1).pack()
btn7 = buttons('7', frame1).pack()

btn2 = buttons('2', frame2).pack()
btn5 = buttons('5', frame2).pack()
btn8 = buttons('8', frame2).pack()
btn0 = buttons_ops('0', frame2, '#333300', 'white').pack()

plus = buttons_ops('+', frame4, 'black', 'white').pack()
minus= buttons_ops('-', frame4,  'black', 'white').pack()
mul = buttons_ops('x', frame4, 'black', 'white').pack()
div = buttons_ops('/', frame4, 'black', 'white').pack()

btn3 = buttons('3', frame3).pack()
btn6 = buttons('6', frame3).pack()
btn9 = buttons('9', frame3).pack()

# This keeps the window active
root.mainloop()

Now let's break it down:

First, we created an Tkinter image object via tk.PhotoImage(). This is a transparent image. The purpose behind creating this image is to set a perfect width and height of the button pixel-wise. The compound='center' ensures that the button text is aligned at the center of the transparent image.

You can change the size of the button by changing the width and height parameters of the pixel object.

Secondly, we created a function which takes the 'text' and the 'container frame' as the argument. Inside the function, we created a button object and returned it. For the numerical buttons, we've created the function buttons whereas for operator buttons, we've created the function buttons_ops. This was done only to ensure different style of buttons (in terms of background and foreground, and so on).

You can change the colours of the buttons by making changes in the bg and fg parameters of the tk.Button() function.

Then we created all the buttons with these two functions. The pack() function puts the buttons in their respective places. Remember that we haven't created the = and AC buttons.

When we execute the program, the following window will pop up:

You can try clicking the buttons to make sure that everything is working great up to this point.

How to Add the Output Screen of the Calculator

For the output screen of the calculator, we'll be using the Entry object in Tkinter. The Entry object will be the best match in this case because we want a single line screen to showcase the user input. We could also use a Text object, but it provides a multiline area. So here, we'll just be using the Entry object.

Also, since we want the output screen to be on the top of the keyboard, we need to define and embed this object before embedding the frames.

The Entry object is created using the tk.Entry() function. This has similar parameters to the tk.Button() function. The following code creates an entry box:

import tkinter as tk

# screen initialization
root = tk.Tk()

# Naming the window
root.title("Calculator")

# creating the output screen
entry = tk.Entry(root, width=9, font=('Arial', 38, 'bold'), state='readonly')
entry.pack(pady=(30, 10))


frame1 = tk.Frame(root)
frame1.pack(side='left', anchor='n')
frame2 = tk.Frame(root)
frame2.pack(side='left', anchor='n')
frame3 = tk.Frame(root)
frame3.pack(side='left', anchor='n')
frame4 = tk.Frame(root)
frame4.pack(side='left', anchor='n')

pixel = tk.PhotoImage(width=55, height=55)

def buttons(text, frame):
    button = tk.Button(frame, text=text, font=('Arial', 20), image=pixel, bg="#333300", fg="white", compound="center")
    return button


def buttons_ops(text, frame, bg, fg):
    button = tk.Button(frame, text=text, font=('Arial', 20), image=pixel, bg=bg, fg=fg, activebackground="black",
                        compound="center")
    return button

btn1 = buttons('1',frame1).pack()
btn4 = buttons('4', frame1).pack()
btn7 = buttons('7', frame1).pack()

btn2 = buttons('2', frame2).pack()
btn5 = buttons('5', frame2).pack()
btn8 = buttons('8', frame2).pack()
btn0 = buttons_ops('0', frame2, '#333300', 'white').pack()

plus = buttons_ops('+', frame4, 'black', 'white').pack()
minus= buttons_ops('-', frame4,  'black', 'white').pack()
mul = buttons_ops('x', frame4, 'black', 'white').pack()
div = buttons_ops('/', frame4, 'black', 'white').pack()

btn3 = buttons('3', frame3).pack()
btn6 = buttons('6', frame3).pack()
btn9 = buttons('9', frame3).pack()

# This keeps the window active
root.mainloop()

In the code above, we put the parent container of the entry object as the main window root. I set the width parameter to 9 as it fit well with the dimensions of the window and the keyboard. You can try it out with different values for width and set a perfectly sized output screen.

You may have noticed that we didn't use the pack() on the same line as object definition. This is because using pack() on the same line as object definition is a bad practice as it limits certain functionality.

So, why did we use the pack() function on the same line while creating buttons? This is because we didn't work heavily with the buttons in this project, so we attempted to reduce the lines of code.

In the tk.Entry() function, we set state='readonly'. This prohibits any direct text input into the the output screen. That means, we can only use the buttons to show the characters on the output screen. By default, this is set to state='normal', which allows direct input from the keyboard into the entry box.

The pady parameter inside the pack() function leaves the given amount of pixels above and below the object. To perform such an operation, let's say to pad 10 pixels on both sides of the object, we can write pady=10 .

Here, we didn't want the same amount of padding above and below the object. So we used a tuple with first element representing the pixels to pad above the output screen, and the second element representing the pixels to pad below the output screen.

Up until now, our GUI looks as shown below:

We can now see that the output screen is set perfectly.

How to Make the Numbers Visible on the Output Screen

Next step is to make characters visible on the output screen. Every button that we click should render on the output screen. For this, we have to link commands to each button. Let's first look at the code and then see how it works:

import tkinter as tk

# screen initialization
root = tk.Tk()

# Naming the window
root.title("Calculator")

entry = tk.Entry(root, width=9, font=('Arial', 38, 'bold'),state='readonly')
entry.pack(pady=(30, 10))


frame1 = tk.Frame(root)
frame1.pack(side='left', anchor='n')
frame2 = tk.Frame(root)
frame2.pack(side='left', anchor='n')
frame3 = tk.Frame(root)
frame3.pack(side='left', anchor='n')
frame4 = tk.Frame(root)
frame4.pack(side='left', anchor='n')

pixel = tk.PhotoImage(width=55, height=55)

def command(text):
    entry.config(state='normal')
    entry.insert(tk.END, text) 
    entry.config(state='readonly')  


def buttons(text, frame):
    button = tk.Button(frame, text=text, font=('Arial', 20), image=pixel, bg="#333300", fg="white", compound="center",
                       command=lambda :command(text))
    return button


def buttons_ops(text, frame, bg, fg):
    button = tk.Button(frame, text=text, font=('Arial', 20), image=pixel, bg=bg, fg=fg, activebackground="black",
                        compound="center", command=lambda:command(text))
    return button

btn1 = buttons('1',frame1).pack()
btn4 = buttons('4', frame1).pack()
btn7 = buttons('7', frame1).pack()

btn2 = buttons('2', frame2).pack()
btn5 = buttons('5', frame2).pack()
btn8 = buttons('8', frame2).pack()
btn0 = buttons_ops('0', frame2, '#333300', 'white').pack()

plus = buttons_ops('+', frame4, 'black', 'white').pack()
minus= buttons_ops('-', frame4,  'black', 'white').pack()
mul = buttons_ops('x', frame4, 'black', 'white').pack()
div = buttons_ops('/', frame4, 'black', 'white').pack()

btn3 = buttons('3', frame3).pack()
btn6 = buttons('6', frame3).pack()
btn9 = buttons('9', frame3).pack()

# This keeps the window active
root.mainloop()

In the code above, we defined a new function called command(). This function takes one argument text. Inside the function, we changed the state of the entry object to normal via config. By doing this, we can now make changes in the text of the entry object.

Then we used the insert() function for the entry object. The insert() function appends the text argument to the existing set of characters.

The first argument of the insert() function takes the index where the text will be inserted. tk.END represents the last character of the text in the object. The second argument of the insert() function takes the text that is to be inserted.

Finally, we change the state of the object again to readonly to prohibit any outside input other than our defined calculator keyboard.

Now let's look at the buttons and the buttons_ops functions. You may have noticed that we've added the command parameter to the tk.Button() function. The lambda tells the program to perform the command only when the button is clicked.

Collectively, command=lambda:command(text) means that, on clicking the buttons which we have defined up until now, it executes the command() function and shows the pressed button character on the output screen.

Now try clicking some buttons on your window. They should appear on the output screen as shown below:

How to Add a Scrollbar to the Output Screen

Now, you might have encountered a problem: when you input a large number of characters, you were able to see only the first few characters. The rest were invisible.

To tackle this, we'll add a scrollbar to the output screen.

First, we'll create a scrollbar object via tk.Scrollbar() before the entry object. The following code shows how:

import tkinter as tk

# screen initialization
root = tk.Tk()

# Naming the window
root.title("Calculator")

scrollbar = tk.Scrollbar(root, orient='horizontal')

entry = tk.Entry(root, width=9, font=('Arial', 38, 'bold'), state='readonly', xscrollcommand=scrollbar.set)
entry.pack(pady=(30, 10))

scrollbar.config(command=entry.xview)
scrollbar.pack()

frame1 = tk.Frame(root)
frame1.pack(side='left', anchor='n')
frame2 = tk.Frame(root)
frame2.pack(side='left', anchor='n')
frame3 = tk.Frame(root)
frame3.pack(side='left', anchor='n')
frame4 = tk.Frame(root)
frame4.pack(side='left', anchor='n')

pixel = tk.PhotoImage(width=55, height=55)

def command(text):
    entry.config(state='normal')
    entry.insert(tk.END, text)
    entry.config(state='readonly')


def buttons(text, frame):
    button = tk.Button(frame, text=text, font=('Arial', 20), image=pixel, bg="#333300", fg="white", compound="center",
                       command=lambda :command(text))
    return button


def buttons_ops(text, frame, bg, fg):
    button = tk.Button(frame, text=text, font=('Arial', 20), image=pixel, bg=bg, fg=fg, activebackground="black",
                        compound="center", command=lambda:command(text))
    return button

btn1 = buttons('1',frame1).pack()
btn4 = buttons('4', frame1).pack()
btn7 = buttons('7', frame1).pack()

btn2 = buttons('2', frame2).pack()
btn5 = buttons('5', frame2).pack()
btn8 = buttons('8', frame2).pack()
btn0 = buttons_ops('0', frame2, '#333300', 'white').pack()

plus = buttons_ops('+', frame4, 'black', 'white').pack()
minus= buttons_ops('-', frame4,  'black', 'white').pack()
mul = buttons_ops('x', frame4, 'black', 'white').pack()
div = buttons_ops('/', frame4, 'black', 'white').pack()

btn3 = buttons('3', frame3).pack()
btn6 = buttons('6', frame3).pack()
btn9 = buttons('9', frame3).pack()

# This keeps the window active
root.mainloop()

The orient parameter in the tk.Scrollbar() object determines the nature of the scrollbar. Here, we've aligned it with the X-axis. We also added a parameter in the original entry object. The xscrollcommand sets the scrollbar to the output screen.

Then we connected the scrollbar to the entry object by setting command=entry.xview and embedded the scrollbar in the output screen.

The following image shows the scrollbar. You can use the arrow signs to navigate forward or backward through the text:

How to Add the Equal To Button

We haven't yet made the equal to button – so let's do that now. To start, we'll define a function called cmd_equal(). In this function, we'll first change the state of the entry to normal. Then we'll extract the text in the output screen using the entry.get() function and replace 'x' by '*'. We do this because multiplication is represented by '*' and not 'x'.

Then we'll add a try-except block. We'll try to evaluate the mathematical expression that we extracted using Python's built-in eval() function. If that's invalid, instead of throwing an error, we'll output 'Invalid' onto our screen.

import tkinter as tk

# screen initialization
root = tk.Tk()

# Naming the window
root.title("Calculator")

scrollbar = tk.Scrollbar(root, orient='horizontal')

entry = tk.Entry(root, width=9, font=('Arial', 38, 'bold'), state='readonly', xscrollcommand=scrollbar.set)
entry.pack(pady=(30, 10))

scrollbar.config(command=entry.xview)
scrollbar.pack()

frame1 = tk.Frame(root)
frame1.pack(side='left', anchor='n')
frame2 = tk.Frame(root)
frame2.pack(side='left', anchor='n')
frame3 = tk.Frame(root)
frame3.pack(side='left', anchor='n')
frame4 = tk.Frame(root)
frame4.pack(side='left', anchor='n')

pixel = tk.PhotoImage(width=55, height=55)

def command(text):
    entry.config(state='normal')
    entry.insert(tk.END, text)
    entry.config(state='readonly')

def cmd_equal():
    entry.config(state='normal')
    txt = entry.get().replace('x', '*')

    try:
        result = eval(txt)

    except:
        result = 'INVALID'
    entry.delete(0, tk.END)
    entry.insert(tk.END, result)
    entry.config(state='readonly')   


def buttons(text, frame):
    button = tk.Button(frame, text=text, font=('Arial', 20), image=pixel, bg="#333300", fg="white", compound="center",
                       command=lambda :command(text))
    return button


def buttons_ops(text, frame, bg, fg):
    button = tk.Button(frame, text=text, font=('Arial', 20), image=pixel, bg=bg, fg=fg, activebackground="black",
                        compound="center", command=lambda:command(text))
    return button

btn1 = buttons('1',frame1).pack()
btn4 = buttons('4', frame1).pack()
btn7 = buttons('7', frame1).pack()

btn2 = buttons('2', frame2).pack()
btn5 = buttons('5', frame2).pack()
btn8 = buttons('8', frame2).pack()
btn0 = buttons_ops('0', frame2, '#333300', 'white').pack()

plus = buttons_ops('+', frame4, 'black', 'white').pack()
minus= buttons_ops('-', frame4,  'black', 'white').pack()
mul = buttons_ops('x', frame4, 'black', 'white').pack()
div = buttons_ops('/', frame4, 'black', 'white').pack()

btn3 = buttons('3', frame3).pack()
btn6 = buttons('6', frame3).pack()
btn9 = buttons('9', frame3).pack()
equal= tk.Button(frame3, text='=', font=('Arial', 20), image=pixel, bg='white', fg='black', activebackground="black",
                        compound="center", command=lambda: cmd_equal()).pack()

# This keeps the window active
root.mainloop()

Here, we've also used entry.delete(). This function will delete all the text on the output screen from the first argument's index (that is from the 0th index) to the last argument's index, that is to the end of the text (represented by tk.END).

Then we inserted our result onto the output screen using entry.insert(). An important thing to note is that we've embedded the equal to button below the definition of btn9 in the same frame. This puts our equal to button in just the right place.

The following images show the initial and final screens, respectively.

On clicking the equal to button:

How to Add the AC Button

Now finally, we'll define our last function: cmd_ac(). This function will delete everything on the output screen. We'll do this by first changing the state to normal, then using entry.delete(), and lastly changing the state back to readonly. Then we'll put this function in the command() parameter of the ac button.

To keep the UI from dismantling when we expand the window, we'll use the resizable() function. This functions takes two arguments: one corresponds to the permission to expand the width and the other to the height. To prohibit expansion of the window, we'll set both the parameters to False.

So the final code will be:

import tkinter as tk

# screen initialization
root = tk.Tk()

# Naming the window
root.title("Calculator")

scrollbar = tk.Scrollbar(root, orient='horizontal')

entry = tk.Entry(root, width=9, font=('Arial', 38, 'bold'), state='readonly', xscrollcommand=scrollbar.set)
entry.pack(pady=(30, 10))

scrollbar.config(command=entry.xview)
scrollbar.pack()

frame1 = tk.Frame(root)
frame1.pack(side='left', anchor='n')
frame2 = tk.Frame(root)
frame2.pack(side='left', anchor='n')
frame3 = tk.Frame(root)
frame3.pack(side='left', anchor='n')
frame4 = tk.Frame(root)
frame4.pack(side='left', anchor='n')

pixel = tk.PhotoImage(width=55, height=55)

def command(text):
    entry.config(state='normal')
    entry.insert(tk.END, text)
    entry.config(state='readonly')

def cmd_ac():
    entry.config(state='normal')
    entry.delete(0, tk.END)
    entry.config(state='readonly')

def cmd_equal():
    entry.config(state='normal')
    txt = entry.get().replace('x', '*')

    try:
        result = eval(txt)

    except:
        result = 'INVALID'
    entry.delete(0, tk.END)
    entry.insert(tk.END, result)
    entry.config(state='readonly')


def buttons(text, frame):
    button = tk.Button(frame, text=text, font=('Arial', 20), image=pixel, bg="#333300", fg="white", compound="center",
                       command=lambda :command(text))
    return button


def buttons_ops(text, frame, bg, fg):
    button = tk.Button(frame, text=text, font=('Arial', 20), image=pixel, bg=bg, fg=fg, activebackground="black",
                        compound="center", command=lambda:command(text))
    return button

btn1 = buttons('1',frame1).pack()
btn4 = buttons('4', frame1).pack()
btn7 = buttons('7', frame1).pack()
ac = tk.Button(frame1, text="AC", font=('Arial', 20), image=pixel, bg="#666699", fg="white", compound="center",
                        command=lambda: cmd_ac()).pack()

btn2 = buttons('2', frame2).pack()
btn5 = buttons('5', frame2).pack()
btn8 = buttons('8', frame2).pack()
btn0 = buttons_ops('0', frame2, '#333300', 'white').pack()

plus = buttons_ops('+', frame4, 'black', 'white').pack()
minus= buttons_ops('-', frame4,  'black', 'white').pack()
mul = buttons_ops('x', frame4, 'black', 'white').pack()
div = buttons_ops('/', frame4, 'black', 'white').pack()

btn3 = buttons('3', frame3).pack()
btn6 = buttons('6', frame3).pack()
btn9 = buttons('9', frame3).pack()
equal= tk.Button(frame3, text='=', font=('Arial', 20), image=pixel, bg='white', fg='black', activebackground="black",
                        compound="center", command=lambda: cmd_equal()).pack()


root.resizable(0,0) 
# This keeps the window active
root.mainloop()

When we hit run, this should display our final project.

Wrapping Up

So now you know how to build a simple arithmetic calculator. To strengthen and build upon the concepts that you learned here, you can try to add some more functionality to this calculator. Here are some ideas for you to practice the things learnt here:

• Adding a decimal point button to the calculator to allow users work with fractional numbers.

• Adding percentage button to the calculator to allow users calculate percentages.

• Adding a delete button to the calculator which, instead of clearing entire screen, deletes one character at a time.

• Making the calculator 'computer keyboard interactive', that is, allowing input directly from the computer keyboard. (Hint for this task: changing the state of the entry object to normal, and adding conditions for 'invalid' expressions).

Thanks for reading!

Polars is an open-source library, originally written in Rust, which makes data wrangling easier in Python. The syntax of Polars is very similar to Pandas, so if you’ve worked with Pandas or the PySpark library before, using Polars should be a breeze.

Polars excels at giving fast results. It’s also memory efficient and helps you optimize your code using parallelism. It also lets you convert data from and to various libraries like NumPy, Pandas, and others.

In this tutorial, we’ll be learning about the Polars Library from absolute scratch, from installing and importing the library on the system, to manipulating data in a dataset with the help of this library.

First, we’ll look at Polars basic functions. We’ll be also writing some practical code, which will help you apply what you’ve learned. Finally, we’ll be working with an example dataset to solidify some more key Polars concepts. Let’s dive in.

Table of Contents

• Prerequisites

• Installing and Importing the Polars Library

• What is a Series?

• What is a DataFrame?

• How to Read CSV Files with Polars

• Some other Important Functions

• Summary

Prerequisites

Even though this tutorial is beginner-friendly, having some basic knowledge of the following areas will help you understand this article better:

• Basic Python syntax

• Data structures

• Ability to import libraries and knowledge of using functions and methods

• Basics of NumPy and Pandas will come in handy (not necessary).

Now, that you’re aware of the prior requirements to follow along, let’s get started with our tutorial.

Installing and Importing the Polars Library

To install the Polars library, you can use the following command in your terminal:

pip install polars

Now, this works if you already have the pip package manager on your system. If you’re on a conda environment, you can work with this:

conda install -c conda-forge polars

But I strongly recommend using the pip package manager to avoid various inconveniences.

Let’s import Polars in our program. We’ll follow the same process as we use for importing other libraries in Python:

import polars as pl # pl is a conventional alias

While creating a Polars object with the data, it’s important to know the size of our data. Polars has the capacity to have 2³² rows in the DataFrame. To load more data, use the following command to install the Polars library:

pip install polars[rt64]

If you want to use the Polars library right away without actually installing it on your system, using a Google Colab notebook is the best option. When using a Google Colab Notebook, you can directly import and start using Polars in your program. I’ll be using Google Colab Notebook for this tutorial.

What is a Series?

A series is a fundamental element of a DataFrame. It’s a 1-dimensional data-structure that you can correlate with a ‘list’ in Python or a ‘1-D array’ in NumPy. But the difference between a series and a 1-D array is that the former is labeled while the later is not. Many series come together to form a DataFrame.

We can create a series with homogenous data as well as heterogenous data.

Creating a Series with Homogenous Data

In a series, the datatype of all the elements should be the same. If it’s not, an error is thrown.

The syntax to define a Polars series is as follows:

var_name = pl.Series(“column_name”, [values])

The following code shows an example of a homogenous series definition in Python:

import polars as pl
series_homo = pl.Series("Numbers", ['One', 'Two', 'Three', 'Four', 'Five'])
print(series_homo)

Output:

shape: (5,)
Series: 'Numbers' [str]
[
    "One"
    "Two"
    "Three"
    "Four"
    "Five"
]

In the above code, we first imported the Polars library using the pl alias to start using it throughout the code. Using aliases is a matter of choice, but pl is a conventional one (like np for NumPy and pd for Pandas). The benefit of using conventional aliases is that when you hand over the code to someone else, it’s easy for them to follow along.

Next, we used the pl.Series() function to create a Polars series object. As its first parameter, we passed the label for our series (Numbers in this case). Then we passed the values to be stores in the form of a list. Remember that the list of values that we pass acts as a single argument. Finally, we printed our series.

We can see that the output tells us about the dimensions of the the Polars object as well as the datatype of the series. The shape (rows, columns) tells us about the the number of rows and columns present in the Polars object.

We can find the data-type of a homogenous series explicitly by using the dtype method.

print(series_homo.dtype)

Output:

String

Creating a Series with Heterogenous Data

Heterogenous data means that the data-type of all the elements is not the same. The syntax to define a series with heterogenous data is as follows:

var_name = pl.Series(“Column_name”, [values], strict=False)

So you’re probably wondering, based on what I said above: how can we have a series with heterogenous data? Well, one thing to note is that a series is always homogenous irrespective of the data that is fed to it. I’ll explain below - first let’s look at this code:

import polars as pl

series_hetero = pl.Series("Numbers", [1, "Two", 3, "Four"], strict=False)
print(series_hetero)

Output:

shape: (4,)
Series: 'Numbers' [str]
[
    "1"
    "Two"
    "3"
    "Four"
]

Here, we created a series object using the pl.Series() function, labelled it, and passed the values that we want in our series.

But you’ll notice that we have provided heterogenous data (data that doesn’t have the same datatype) to the function. Usually, this throws an error. But as we have set the strict parameter as False, the function now becomes lenient with the schema of the series. (The schema is just the expected data-type of the values that are to be recorded in the series.)

If no particular schema is defined for a series that’s fed heterogenous data, pl.Series() sets the schema to pl.Utf8 (string datatype). You can see this automatic fixing of the schema in the above example. This prevents the program from bugging, as a string datatype can comprehend characters – numbers as well as symbols.

Also, we can see that datatype of all elements is the same (pl.Utf8). This means that the series is homogenous, even though we put heterogenous data in it.

If we define a schema for the series, then the Polars library converts all the records – which show a different datatype than the defined schema – to null objects. This should be clear in the following example:

import polars as pl
# defined the schema as Integer bit 32
series = pl.Series("ints", [1, -2, 3, 4, 5, 'Thirteen', 'Fourteen'], dtype=pl.Int32, strict=False)
print(series)

Output:

shape: (7,)
Series: 'ints' [i32]
[
    1
    -2
    3
    4
    5
    null
    null
]

Here, we can see that the last two entities were ‘String’, but since we set the schema as ‘Integer’, they were reflected as null records.

So as you can see, the leniency of the program depends on whether you set the strict parameter to True of False. If we set it as True, we enforce the schema to the data strictly. Upon failing to obey the schema, the program raises an exception. On the other hand, if we set the strict parameter as False, the series still preserves its homogenous nature by turning schema-disobeying elements to null.

Now that you understand how series work, we’re ready to move on to DataFrames.

What is a DataFrame?

A DataFrame is a two-dimensional data structure that you can use to store large numbers of related parameters of the collected data. It’s also useful for analyzing that data. A DataFrame is nothing more than the collection of many series, each labelled differently to store different aspects of data.

Here’s the syntax to create a Polars DataFrame object:

var_name = pl.DataFrame({key: value pairs}, schema)

The following example shows you how to define a DataFrame object in Python:

import polars as pl
import numpy as np

schema = {"Number": pl.UInt32, "Natural Log": None, "Log Base 10": None}

df = pl.DataFrame(
    {
        "Number" : np.arange(1, 11),
        "Natural Log" : [np.log(x) for x in range(1,11)],
        'Log Base 10' : [np.log10(x) for x in range(1,11)]
        },
    schema=schema
    )
print(df)

Output:

shape: (10, 3)
┌────────┬─────────────┬─────────────┐
│ Number ┆ Natural Log ┆ Log Base 10 │
│ ---    ┆ ---         ┆ ---         │
│ u32    ┆ f64         ┆ f64         │
╞════════╪═════════════╪═════════════╡
│ 1      ┆ 0.0         ┆ 0.0         │
│ 2      ┆ 0.693147    ┆ 0.30103     │
│ 3      ┆ 1.098612    ┆ 0.477121    │
│ 4      ┆ 1.386294    ┆ 0.60206     │
│ 5      ┆ 1.609438    ┆ 0.69897     │
│ 6      ┆ 1.791759    ┆ 0.778151    │
│ 7      ┆ 1.94591     ┆ 0.845098    │
│ 8      ┆ 2.079442    ┆ 0.90309     │
│ 9      ┆ 2.197225    ┆ 0.954243    │
│ 10     ┆ 2.302585    ┆ 1.0         │
└────────┴─────────────┴─────────────┘

Above, we created a Polars DataFrame object with the pl.DataFrame() function. In the function, we created a dictionary as an argument for passing the values of the DataFrame.

In the dictionary, each key-value pair represents a series. Each key represents the label of the series, whereas its value represent the values of the series. The values are passed in the form of a list as each key can map to only one value.

Then we defined the schema for the DataFrame. Again, the schema is a dictionary, where each key-value pair corresponds to the schema of the series. In the schema, every key represents the label of the series (to map the schema to the correct series) and its value represents the schema.

In the output, we can see that we got a nice table representing our data. The labels are neatly separated from the data and below them, their schema is also represented.

What is a Schema?

A schema refers to the definition of the datatype of the series. We fix a particular datatype to the homogenous series to avoid getting in mixed-data.

For example, in the above code, we set the datatype of the column Number to Unsigned Integer - 32 bit (pl.UInt32) as we don’t want to put negative integers in our NumPy logarithm function.

Now, if we want to hide the datatype (that’s written below each label), we can use the following function:

pl.Config.set_tbl_hide_column_data_types(active=True)

The Head, Tail, and Glimpse Functions

The head(), tail() and glimpse() functions are used to have a quick look at the data by reviewing certain records (rows). These are useful especially for large datasets for taking a look at the data, for example to see which columns are present, what type of data is present in each column, and so on.

The head() function prints the given number of rows (passed as the argument of the head() function) from the top of the DataFrame. If no argument is passed, it prints the first five rows of the DataFrame.

import polars as pl
import numpy as np

schema = {"Number": pl.UInt32, "Natural Log": None, "Log Base 10": None}

df = pl.DataFrame(
    {
        "Number" : np.arange(1, 11),
        "Natural Log" : [np.log(x) for x in range(1,11)],
        'Log Base 10' : [np.log10(x) for x in range(1,11)]
        },
    schema=schema
    )
pl.Config.set_tbl_hide_column_data_types(active=True)
print(df.head(3))

Output:

shape: (3, 3)
┌────────┬─────────────┬─────────────┐
│ Number ┆ Natural Log ┆ Log Base 10 │
╞════════╪═════════════╪═════════════╡
│ 1      ┆ 0.0         ┆ 0.0         │
│ 2      ┆ 0.693147    ┆ 0.30103     │
│ 3      ┆ 1.098612    ┆ 0.477121    │
└────────┴─────────────┴─────────────┘

In this example, we have the used the same DataFrame that we just created. Then we used the head() function to output the first three rows of the DataFrame. Also, you may now notice that the schema representation under column names has disappeared. This is because we used pl.Config.set_tbl_hide_column_data_types(active=True).

The glimpse() function presents the data briefly and in a horizontal manner (rows are represented as columns and columns are represented as rows) for better readability.

import polars as pl
import numpy as np

schema = {"Number": pl.UInt32, "Natural Log": None, "Log Base 10": None}

df = pl.DataFrame(
    {
        "Number" : np.arange(1, 11),
        "Natural Log" : [np.log(x) for x in range(1,11)],
        'Log Base 10' : [np.log10(x) for x in range(1,11)]
        },
    schema=schema
    )
pl.Config.set_tbl_hide_column_data_types(active=True)
print(df.glimpse())

Output:

Rows: 10
Columns: 3
$ Number      <u32> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
$ Natural Log <f64> 0.0, 0.6931471805599453, 1.0986122886681098, 1.3862943611198906, 1.6094379124341003, 1.791759469228055, 1.9459101490553132, 2.0794415416798357, 2.1972245773362196, 2.302585092994046
$ Log Base 10 <f64> 0.0, 0.3010299956639812, 0.47712125471966244, 0.6020599913279624, 0.6989700043360189, 0.7781512503836436, 0.8450980400142568, 0.9030899869919435, 0.9542425094393249, 1.0

None

Here, we used the glimpse() function on our previously created DataFrame df. We can see the output as our transposed DataFrame. Also, None is returned. This is because, by default, glimpse() sets its return_as_string parameter to None. To change it to string, we can set the return_as_string parameter to True. The following example shows how to do it:

import polars as pl
import numpy as np

schema = {"Number": pl.UInt32, "Natural Log": None, "Log Base 10": None}

df = pl.DataFrame(
    {
        "Number" : np.arange(1, 11),
        "Natural Log" : [np.log(x) for x in range(1,11)],
        'Log Base 10' : [np.log10(x) for x in range(1,11)]
        },
    schema=schema
    )
pl.Config.set_tbl_hide_column_data_types(active=True)
print(f'Returned as String: \n{df.glimpse(return_as_string=True)}')

Output:

Returned as String: 
Rows: 10
Columns: 3
$ Number      <u32> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
$ Natural Log <f64> 0.0, 0.6931471805599453, 1.0986122886681098, 1.3862943611198906, 1.6094379124341003, 1.791759469228055, 1.9459101490553132, 2.0794415416798357, 2.1972245773362196, 2.302585092994046
$ Log Base 10 <f64> 0.0, 0.3010299956639812, 0.47712125471966244, 0.6020599913279624, 0.6989700043360189, 0.7781512503836436, 0.8450980400142568, 0.9030899869919435, 0.9542425094393249, 1.0

In the above code, we can see that the DataFrame is returned as a string and None is not returned.

Finally, the tail() function outputs the given number of rows (passed as the argument of the tail() function) from the bottom of the dataset. When no argument is passed, it outputs the last 5 rows by default.

This is useful for checking if our data was completely loaded. Checking the first few records using the head() function and the last few records with the tail() function ensures that the data is correctly and totally loaded.

Also, we can check if there are any empty records at the end of the dataset. Having empty records at the end of the dataset can be fatal in some cases. For example, if you have to train an ML model on a dataset and you split the dataset statically into testing and training datasets, the empty rows at the end are going to cause an issue. So, checking our data beforehand is a best practice, and these functions help us do it.

import polars as pl
import numpy as np

schema = {"Number": pl.UInt32, "Natural Log": None, "Log Base 10": None}

df = pl.DataFrame(
    {
        "Number" : np.arange(1, 11),
        "Natural Log" : [np.log(x) for x in range(1,11)],
        'Log Base 10' : [np.log10(x) for x in range(1,11)]
        },
    schema=schema
    )
pl.Config.set_tbl_hide_column_data_types(active=True)
print(df.tail(3))

Output:

shape: (3, 3)
┌────────┬─────────────┬─────────────┐
│ Number ┆ Natural Log ┆ Log Base 10 │
╞════════╪═════════════╪═════════════╡
│ 8      ┆ 2.079442    ┆ 0.90309     │
│ 9      ┆ 2.197225    ┆ 0.954243    │
│ 10     ┆ 2.302585    ┆ 1.0         │
└────────┴─────────────┴─────────────┘

In the above code, we used the tail() function on the dataset (that we created earlier) and passed ‘3’ as our argument. Thus our program returned the last three rows of the dataset.

The Sample Function

The sample() function returns a given number of random rows in random order based on their occurrence in the DataFrame. This helps to avoid biased sampling of data.

import polars as pl
import numpy as np

schema = {"Number": pl.UInt32, "Natural Log": None, "Log Base 10": None}

df = pl.DataFrame(
    {
        "Number" : np.arange(1, 11),
        "Natural Log" : [np.log(x) for x in range(1,11)],
        'Log Base 10' : [np.log10(x) for x in range(1,11)]
        },
    schema=schema
    )
pl.Config.set_tbl_hide_column_data_types(active=True)
print(df.sample(3))

Output:

shape: (3, 3)
┌────────┬─────────────┬─────────────┐
│ Number ┆ Natural Log ┆ Log Base 10 │
╞════════╪═════════════╪═════════════╡
│ 6      ┆ 1.791759    ┆ 0.778151    │
│ 5      ┆ 1.609438    ┆ 0.69897     │
│ 10     ┆ 2.302585    ┆ 1.0         │
└────────┴─────────────┴─────────────┘

We can see in the output that we got random rows of the data in a random order of their occurrence in the dataset (row 5 comes before row 6 in the DataFrame, yet by sampling we got row 5 after row 6.) Sampling is a good practice as it helps avoid overfitting in ML in some cases and gives us a general idea about the entire dataset.

Concatenating Two DataFrames

In a nutshell, ‘concatenating’ simply means ‘linking’. Adding or linking one dataset to another – basically, stacking one on top of another – is concatenating the two datasets.

For example, in the previous DataFrame, we had numbers from 1 to 10 and their logarithms. Now, if we want to make it 1 to 20, we have to concatenate a different dataset containing numbers 11 to 20 to the former dataset.

The following code shows how this works:

import polars as pl
import numpy as np

schema = {"Number": pl.UInt32, "Natural Log": None, "Log Base 10": None}

df = pl.DataFrame(
    {
        "Number" : np.arange(1, 11),
        "Natural Log" : [np.log(x) for x in range(1,11)],
        'Log Base 10' : [np.log10(x) for x in range(1,11)]
        },
    schema=schema
    )
pl.Config.set_tbl_hide_column_data_types(active=True)

# new dataset created for concatenation
df1 = pl.DataFrame({
    "Number" : [x for x in range(11, 21)],
    "Log Base 10" : [np.log10(x) for x in range(11,21)],
    "Natural Log" : [np.log(x) for x in range(11, 21)]
}, schema=schema)

print(pl.concat([df, df1], how='vertical')) # concatenating the two datasets

Output:

shape: (20, 3)
┌────────┬─────────────┬─────────────┐
│ Number ┆ Natural Log ┆ Log Base 10 │
╞════════╪═════════════╪═════════════╡
│ 1      ┆ 0.0         ┆ 0.0         │
│ 2      ┆ 0.693147    ┆ 0.30103     │
│ 3      ┆ 1.098612    ┆ 0.477121    │
│ 4      ┆ 1.386294    ┆ 0.60206     │
│ 5      ┆ 1.609438    ┆ 0.69897     │
│ …      ┆ …           ┆ …           │
│ 16     ┆ 2.772589    ┆ 1.20412     │
│ 17     ┆ 2.833213    ┆ 1.230449    │
│ 18     ┆ 2.890372    ┆ 1.255273    │
│ 19     ┆ 2.944439    ┆ 1.278754    │
│ 20     ┆ 2.995732    ┆ 1.30103     │
└────────┴─────────────┴─────────────┘

In this code, we first created the DataFrame df. Then we created another DataFrame df1. Next, we used pl.concat() to concatenate the DataFrames.

The first argument that we passed is the list of the DataFrames that are to be linked. The how parameter defines the manner of concatenation. ‘Vertical’ in this context means that we are linking DataFrames vertically (adding more rows).

The important thing to note here is that schema incompatibility may raise an exception. If the DataFrames that are to be concatenated have different schemas, there will be a schema incompatibility problem. So it’s better to keep the schemas of both the datasets (that are to be concatenated) the same.

Here, we introduced a variable named schema containing the schema parameter of the DataFrame and we applied it to both the DataFrames to avoid schema incompatibility.

Also, concatenation occurs in the order of the passed arguments. For example, in the above code, df appears prior to df1, thus in the linked DataFrame, df appears first and then df1. If we had changed the sequence of values, the concatenated DataFrame would start from df1 and then df.

The following code explains that:

import polars as pl
import numpy as np

schema = {"Number": pl.UInt32, "Natural Log": None, "Log Base 10": None}

df = pl.DataFrame(
    {
        "Number" : np.arange(1, 11),
        "Natural Log" : [np.log(x) for x in range(1,11)],
        'Log Base 10' : [np.log10(x) for x in range(1,11)]
        },
    schema=schema
    )
pl.Config.set_tbl_hide_column_data_types(active=True)

# new dataset created for concatenation
df1 = pl.DataFrame({
    "Number" : [x for x in range(11, 21)],
    "Log Base 10" : [np.log10(x) for x in range(11,21)],
    "Natural Log" : [np.log(x) for x in range(11, 21)]
}, schema=schema)

print(pl.concat([df1, df], how='vertical')) # sequence changed from [df,df1] to [df1, df]

Output:

shape: (20, 3)
┌────────┬─────────────┬─────────────┐
│ Number ┆ Natural Log ┆ Log Base 10 │
╞════════╪═════════════╪═════════════╡
│ 11     ┆ 2.397895    ┆ 1.041393    │
│ 12     ┆ 2.484907    ┆ 1.079181    │
│ 13     ┆ 2.564949    ┆ 1.113943    │
│ 14     ┆ 2.639057    ┆ 1.146128    │
│ 15     ┆ 2.70805     ┆ 1.176091    │
│ …      ┆ …           ┆ …           │
│ 6      ┆ 1.791759    ┆ 0.778151    │
│ 7      ┆ 1.94591     ┆ 0.845098    │
│ 8      ┆ 2.079442    ┆ 0.90309     │
│ 9      ┆ 2.197225    ┆ 0.954243    │
│ 10     ┆ 2.302585    ┆ 1.0         │
└────────┴─────────────┴─────────────┘

Here, we can see that the df1 appears first and then df appears (unlike the previous example). Thus, the sequence of the values matters.

How to Join Two DataFrames

Joining datasets and concatenating datasets are two different concepts. While concatenating means ‘linking’ two separate datasets, joining refers to combining datasets based on a shared column (a key).
The computer matches rows from both datasets where the key values are the same.

In the above dataset ‘df’, we’ll add a new column by joining the dataset ‘df’ with another DataFrame.

# new dataframe
new_col = pl.DataFrame({
    "Number" : [x for x in range(1, 11)],
    "Log Base 2" : [np.log2(x) for x in range(1, 11)]
})

new_data = df.join(new_col, on="Number", how="left") # Both have one column same to map values

print(new_data.head())

Output:

shape: (5, 4)
┌────────┬─────────────┬─────────────┬────────────┐
│ Number ┆ Natural Log ┆ Log Base 10 ┆ Log Base 2 │
╞════════╪═════════════╪═════════════╪════════════╡
│ 1      ┆ 0.0         ┆ 0.0         ┆ 0.0        │
│ 2      ┆ 0.693147    ┆ 0.30103     ┆ 1.0        │
│ 3      ┆ 1.098612    ┆ 0.477121    ┆ 1.584963   │
│ 4      ┆ 1.386294    ┆ 0.60206     ┆ 2.0        │
│ 5      ┆ 1.609438    ┆ 0.69897     ┆ 2.321928   │
└────────┴─────────────┴─────────────┴────────────┘

In this example, we used the join function on df and passed new_col as its argument. This is why the columns of the df function occur prior to the column of the new_col dataset. The parameter on should be given a column name on the basis of which the two datasets are to be joined.

Here, we first mapped the elements of the column Number and its corresponding rows and joined the DataFrames accordingly.

If we used the join() function on the new_col DataFrame, the columns of df would appear later than the column in new_col. The following code will make it clear:

# new dataframe
new_col = pl.DataFrame({
    "Number" : [x for x in range(1, 11)],
    "Log Base 2" : [np.log2(x) for x in range(1, 11)]
})

new_data = new_col.join(df, on="Number", how="left") # passed df as argument

print(new_data.head())

Output:

shape: (5, 4)
┌────────┬────────────┬─────────────┬─────────────┐
│ Number ┆ Log Base 2 ┆ Natural Log ┆ Log Base 10 │
╞════════╪════════════╪═════════════╪═════════════╡
│ 1      ┆ 0.0        ┆ 0.0         ┆ 0.0         │
│ 2      ┆ 1.0        ┆ 0.693147    ┆ 0.30103     │
│ 3      ┆ 1.584963   ┆ 1.098612    ┆ 0.477121    │
│ 4      ┆ 2.0        ┆ 1.386294    ┆ 0.60206     │
│ 5      ┆ 2.321928   ┆ 1.609438    ┆ 0.69897     │
└────────┴────────────┴─────────────┴─────────────┘

You can notice that the column ‘Log Base 2’ appears prior to other columns (unlike in the previous example). Thus this change is significant.

How to Use the with_columns() Function

The with_columns() function enables us to make changes to the column and print it as a new column with existing columns from the original dataset. This is similar to the join() function.

The following example will make it clear:

import polars as pl
import numpy as np

df = pl.DataFrame(
    {
        "Number" : np.arange(1, 11),
        "Natural Log" : [np.log(x) for x in range(1,11)],
        'Log Base 10' : [np.log10(x) for x in range(1,11)]
        },
    schema=schema
    )
new_data = df.with_columns((np.log2(pl.col("Number"))).alias("Log Base 2"))

print(new_data.head())

Output:

shape: (5, 4)
┌────────┬─────────────┬─────────────┬────────────┐
│ Number ┆ Natural Log ┆ Log Base 10 ┆ Log Base 2 │
╞════════╪═════════════╪═════════════╪════════════╡
│ 1      ┆ 0.0         ┆ 0.0         ┆ 0.0        │
│ 2      ┆ 0.693147    ┆ 0.30103     ┆ 1.0        │
│ 3      ┆ 1.098612    ┆ 0.477121    ┆ 1.584963   │
│ 4      ┆ 1.386294    ┆ 0.60206     ┆ 2.0        │
│ 5      ┆ 1.609438    ┆ 0.69897     ┆ 2.321928   │
└────────┴─────────────┴─────────────┴────────────┘

In this example, we have a DataFrame df. To add a column to it , we use the with_columns() function. In this function, we selected column named ‘Number’ using the pl.col() function and put it inside the np.log2() to get the log base 2 value for every record. Finally, to label the new column, we used the alias() function, with the label passed to it as an argument.

Now that we know about the basics of DataFrames, let’s look at how we can work with CSV files.

How to Read CSV Files with Polars

Reading CSV files with Polars is extremely similar to how it works in Pandas. For this tutorial, I’ll be using the Titanic Dataset. Here’s the link to the dataset so you can download it. In this part of the tutorial, we’ll be mainly talking about column selection (useful in feature selection) and filtering the data.

Here’s the syntax for reading a CSV file:

var_name = pl.read_csv(“path_dataset“)

Example code:

import polars as pl

data = pl.read_csv("/titanic_dataset.csv")
print(data.head())

Output:

shape: (5, 12)
┌─────────────┬──────────┬────────┬─────────────────────┬───┬─────────┬─────────┬───────┬──────────┐
│ PassengerId ┆ Survived ┆ Pclass ┆ Name                ┆ … ┆ Ticket  ┆ Fare    ┆ Cabin ┆ Embarked │
╞═════════════╪══════════╪════════╪═════════════════════╪═══╪═════════╪═════════╪═══════╪══════════╡
│ 892         ┆ 0        ┆ 3      ┆ Kelly, Mr. James    ┆ … ┆ 330911  ┆ 7.8292  ┆ null  ┆ Q        │
│ 893         ┆ 1        ┆ 3      ┆ Wilkes, Mrs. James  ┆ … ┆ 363272  ┆ 7.0     ┆ null  ┆ S        │
│             ┆          ┆        ┆ (Ellen Need…        ┆   ┆         ┆         ┆       ┆          │
│ 894         ┆ 0        ┆ 2      ┆ Myles, Mr. Thomas   ┆ … ┆ 240276  ┆ 9.6875  ┆ null  ┆ Q        │
│             ┆          ┆        ┆ Francis             ┆   ┆         ┆         ┆       ┆          │
│ 895         ┆ 0        ┆ 3      ┆ Wirz, Mr. Albert    ┆ … ┆ 315154  ┆ 8.6625  ┆ null  ┆ S        │
│ 896         ┆ 1        ┆ 3      ┆ Hirvonen, Mrs.      ┆ … ┆ 3101298 ┆ 12.2875 ┆ null  ┆ S        │
│             ┆          ┆        ┆ Alexander (Helg…    ┆   ┆         ┆         ┆       ┆          │
└─────────────┴──────────┴────────┴─────────────────────┴───┴─────────┴─────────┴───────┴──────────┘

We can get the statistical analysis of the data by using the describe() function.

print(data.describe())

Output:

shape: (9, 13)
┌────────────┬─────────────┬──────────┬──────────┬───┬─────────────┬───────────┬───────┬──────────┐
│ statistic  ┆ PassengerId ┆ Survived ┆ Pclass   ┆ … ┆ Ticket      ┆ Fare      ┆ Cabin ┆ Embarked │
╞════════════╪═════════════╪══════════╪══════════╪═══╪═════════════╪═══════════╪═══════╪══════════╡
│ count      ┆ 418.0       ┆ 418.0    ┆ 418.0    ┆ … ┆ 418         ┆ 417.0     ┆ 91    ┆ 418      │
│ null_count ┆ 0.0         ┆ 0.0      ┆ 0.0      ┆ … ┆ 0           ┆ 1.0       ┆ 327   ┆ 0        │
│ mean       ┆ 1100.5      ┆ 0.363636 ┆ 2.26555  ┆ … ┆ null        ┆ 35.627188 ┆ null  ┆ null     │
│ std        ┆ 120.810458  ┆ 0.481622 ┆ 0.841838 ┆ … ┆ null        ┆ 55.907576 ┆ null  ┆ null     │
│ min        ┆ 892.0       ┆ 0.0      ┆ 1.0      ┆ … ┆ 110469      ┆ 0.0       ┆ A11   ┆ C        │
│ 25%        ┆ 996.0       ┆ 0.0      ┆ 1.0      ┆ … ┆ null        ┆ 7.8958    ┆ null  ┆ null     │
│ 50%        ┆ 1101.0      ┆ 0.0      ┆ 3.0      ┆ … ┆ null        ┆ 14.4542   ┆ null  ┆ null     │
│ 75%        ┆ 1205.0      ┆ 1.0      ┆ 3.0      ┆ … ┆ null        ┆ 31.5      ┆ null  ┆ null     │
│ max        ┆ 1309.0      ┆ 1.0      ┆ 3.0      ┆ … ┆ W.E.P. 5734 ┆ 512.3292  ┆ G6    ┆ S        │
└────────────┴─────────────┴──────────┴──────────┴───┴─────────────┴───────────┴───────┴──────────┘

How to Select Columns from the Dataset

Now we’re going to learn how to select certain columns from the dataset and transform those columns into a new DataFrame. This can be useful if we want to train an ML model based on only certain columns and not the entire dataset (that is, using feature selection).

Let’s first look at the code below:

new_df = data.select(
    pl.col("Survived"),
    pl.col("Name"),
    pl.col("Age"),
    pl.col("Sex")
)

print(new_df.head())

Output:

shape: (5, 4)
┌──────────┬─────────────────────────────────┬──────┬────────┐
│ Survived ┆ Name                            ┆ Age  ┆ Sex    │
╞══════════╪═════════════════════════════════╪══════╪════════╡
│ 0        ┆ Kelly, Mr. James                ┆ 34.5 ┆ male   │
│ 1        ┆ Wilkes, Mrs. James (Ellen Need… ┆ 47.0 ┆ female │
│ 0        ┆ Myles, Mr. Thomas Francis       ┆ 62.0 ┆ male   │
│ 0        ┆ Wirz, Mr. Albert                ┆ 27.0 ┆ male   │
│ 1        ┆ Hirvonen, Mrs. Alexander (Helg… ┆ 22.0 ┆ female │
└──────────┴─────────────────────────────────┴──────┴────────┘

In the code above, we selected four columns using the select() and pl.col() functions from the Titanic Dataset and transformed them into a new DataFrame called new_df.

Now, we can filter this data however we want. Let’s make a new DataFrame by filtering out only surviving passengers from the dataset:

survived_data = data.select(
    pl.col("Survived"),
    pl.col("Name"),
    pl.col("Age"),
    pl.col("Sex")
).filter(pl.col("Survived")==1)

print(survived_data.head())

Output:

shape: (5, 4)
┌──────────┬─────────────────────────────────┬──────┬────────┐
│ Survived ┆ Name                            ┆ Age  ┆ Sex    │
╞══════════╪═════════════════════════════════╪══════╪════════╡
│ 1        ┆ Wilkes, Mrs. James (Ellen Need… ┆ 47.0 ┆ female │
│ 1        ┆ Hirvonen, Mrs. Alexander (Helg… ┆ 22.0 ┆ female │
│ 1        ┆ Connolly, Miss. Kate            ┆ 30.0 ┆ female │
│ 1        ┆ Abrahim, Mrs. Joseph (Sophie H… ┆ 18.0 ┆ female │
│ 1        ┆ Snyder, Mrs. John Pillsbury (N… ┆ 23.0 ┆ female │
└──────────┴─────────────────────────────────┴──────┴────────┘

In the above code, we used the filter() function. This function helps us gather data that applies to our given condition. In the above example, we added the condition that, “Every element in the column named ‘Survived’ should be equal to 1”. Hence, we got our required data.

Some Other Important Functions

How to Print the Names of the Columns of a Dataset

You can print the names of a column using the columns method. The following code shows how to use the columns method:

print(data.columns) # data --> Titanic Dataset

Output:

['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp', 'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked']

How to Index a Dataset

Indexing a dataset means adding an index column to the existing dataset. It can prove useful in keeping track of the rows of the dataset.

We can index the dataset using the with_row_index() function. Inside this function, we can pass the argument to name this new index column. If we don’t pass any argument, the index column name is set as ‘index’ by default.

data = pl.read_csv("/titanic_dataset.csv").with_row_index('#') # naming the index column as '#'
print(data.head())

Output:

shape: (5, 13)
┌─────┬─────────────┬──────────┬────────┬───┬─────────┬─────────┬───────┬──────────┐
│ #   ┆ PassengerId ┆ Survived ┆ Pclass ┆ … ┆ Ticket  ┆ Fare    ┆ Cabin ┆ Embarked │
│ --- ┆ ---         ┆ ---      ┆ ---    ┆   ┆ ---     ┆ ---     ┆ ---   ┆ ---      │
│ u32 ┆ i64         ┆ i64      ┆ i64    ┆   ┆ str     ┆ f64     ┆ str   ┆ str      │
╞═════╪═════════════╪══════════╪════════╪═══╪═════════╪═════════╪═══════╪══════════╡
│ 0   ┆ 892         ┆ 0        ┆ 3      ┆ … ┆ 330911  ┆ 7.8292  ┆ null  ┆ Q        │
│ 1   ┆ 893         ┆ 1        ┆ 3      ┆ … ┆ 363272  ┆ 7.0     ┆ null  ┆ S        │
│ 2   ┆ 894         ┆ 0        ┆ 2      ┆ … ┆ 240276  ┆ 9.6875  ┆ null  ┆ Q        │
│ 3   ┆ 895         ┆ 0        ┆ 3      ┆ … ┆ 315154  ┆ 8.6625  ┆ null  ┆ S        │
│ 4   ┆ 896         ┆ 1        ┆ 3      ┆ … ┆ 3101298 ┆ 12.2875 ┆ null  ┆ S        │
└─────┴─────────────┴──────────┴────────┴───┴─────────┴─────────┴───────┴──────────┘

How to Rename Columns in the Dataset

Lastly, to rename columns in the Dataset, we use the rename() function.

data = pl.read_csv("/titanic_dataset.csv").with_row_index('#').rename({'PassengerId':'renamed_col'})
print(data.head())

Output:

shape: (5, 13)
┌─────┬─────────────┬──────────┬────────┬───┬─────────┬─────────┬───────┬──────────┐
│ #   ┆ renamed_col ┆ Survived ┆ Pclass ┆ … ┆ Ticket  ┆ Fare    ┆ Cabin ┆ Embarked │
│ --- ┆ ---         ┆ ---      ┆ ---    ┆   ┆ ---     ┆ ---     ┆ ---   ┆ ---      │
│ u32 ┆ i64         ┆ i64      ┆ i64    ┆   ┆ str     ┆ f64     ┆ str   ┆ str      │
╞═════╪═════════════╪══════════╪════════╪═══╪═════════╪═════════╪═══════╪══════════╡
│ 0   ┆ 892         ┆ 0        ┆ 3      ┆ … ┆ 330911  ┆ 7.8292  ┆ null  ┆ Q        │
│ 1   ┆ 893         ┆ 1        ┆ 3      ┆ … ┆ 363272  ┆ 7.0     ┆ null  ┆ S        │
│ 2   ┆ 894         ┆ 0        ┆ 2      ┆ … ┆ 240276  ┆ 9.6875  ┆ null  ┆ Q        │
│ 3   ┆ 895         ┆ 0        ┆ 3      ┆ … ┆ 315154  ┆ 8.6625  ┆ null  ┆ S        │
│ 4   ┆ 896         ┆ 1        ┆ 3      ┆ … ┆ 3101298 ┆ 12.2875 ┆ null  ┆ S        │
└─────┴─────────────┴──────────┴────────┴───┴─────────┴─────────┴───────┴──────────┘

In the above example, we renamed the column named ‘PassengerId’ to ‘renamed_col’.

Summary

Now you know how to work with the Polars Python library to analyze your data more effectively.

In this article, you learned:

• What Polars is and how to install it

• How to define series and DataFrames in Polars

• Different functions to deal with DataFrames.

• How to read and work with CSV files in Polars

Thanks for Reading, and happy data wrangling!

By the end of this tutorial, you’ll understand:

• What is the copy module?

• The difference between copying and referencing.

• The difference between a deep copy and a shallow copy.

• How to actually shallow copy and deep copy objects in Python.

• The difference in referencing for immutable objects and mutable objects.

Prerequisites

To get the most out of this tutorial, you need to have a basic understanding of the following:

1. Fundamental knowledge of programming and its terminology (such as objects, memory addresses, and so on)

2. Basic knowledge of Python programming, especially (for this tutorial),

   • Function id(): Outputs the memory address of the object passed as argument.

   • Data structures: Dictionaries and Lists.

   • Modules: importing and using them in the program. Basic understanding of methods and functions.

Table of Contents:

1. What is the Copy Module in Python?

   • Why Can’t We Just Use the Assignment Operator?

2. How to Properly Copy Objects in Python

3. More About Copying Objects in Python

4. Summary

What is the copy Module?

The copy module is an in-built module in Python which is primarily used for copying objects in Python. it lets you make changes to a mutable object and save it as a different copy in memory. So basically, it makes a copy of the original object and stores it in a different memory location.

Why can’t we just use the assignment operator (=) for copying objects?

If we use the assignment operator for the purpose of copying objects, it doesn’t actually copy the object – rather, it creates a binding between the object and the identifier. This means that if the original object points at memory location x, then the identifier in which we attempted to copy the object by using the = operator will also point at the same memory location, that is location x.

Now, this may create problems while manipulating various aspects of the data, as the changes that we make in the object will reflect in its binding as well.

Before jumping into the code, let’s first look at the difference between copying and referencing:

• Copying: Creating a copy refers to replicating the target object and storing it separately into the memory, making it an independent object with same data.

• Referencing: Referencing an object refers to pointing to the same memory address where the target object is stored. The referenced object is just another name (which we call as an ‘alias’ in programming) to call out the original object.

Let’s understand this with an example:

# creating a dictionary object.
d1 = {
    'A' : 1,
    'B' : 2,
    'C' : 3
}

# using assignment operator to copy d1 in d2.
d2=d1

# printing both the dictionaries.
print(f'd1 = {d1} \nd2 = {d2}')

Output:

d1 = {'A': 1, 'B': 2, 'C': 3}

d2 = {'A': 1, 'B': 2, 'C': 3}

From above example, it may seem that the dictionary got copied in variable d2 – but in reality, it’s just pointing to the object stored in variable d1. In this case, variable d2 is just an alias or reference to the same object d1. We can prove this as follows:

d1 = {
    'A' : 1,
    'B' : 2,
    'C' : 3
}

d2=d1

d1['D'] = 4 # added a key-value pair in d1

print(f'd1 = {d1} \nd2 = {d2}')

Output:

d1 = {'A': 1, 'B': 2, 'C': 3, 'D': 4}

d2 = {'A': 1, 'B': 2, 'C': 3, 'D': 4}

Now, in the above code, we appended a key-value pair in dictionary d1 only – but the change is seen in dictionary d2, too. From this, it is evident that both the identifiers were referencing the same object.

From this we understand that, assignment operator = can be used for referencing the objects and we cannot use it for copying objects in true sense.

How to Properly Copy Objects in Python

Since you now understand the difference between copying and referencing, let’s see how you can actually copy objects in Python. For this, we will make use of the copy module (mentioned earlier).

Now, before using this module, you should understand the difference between a deep copy and a shallow copy.

• Deep copy: While working with compound objects (also known as container objects or composite objects), deep copying means to create a copy of the inner objects as well as the outer object.

• Shallow copy: While working with compound objects, shallow copying refers to copying only the outer object and referencing the inner objects.

Note: Compound objects are objects that contain other objects inside them.

Let’s better understand the difference between deep and shallow copies by actually implementing them in a program:

import copy # importing copy module

# creating a composite object
categories = {
    'Fruits' : ['Apple', 'Banana', 'Mango'],
    'Flowers' : ['Rose', 'Sunflower', 'Tulip'],
}

# copying the object by using the copy() function of the copy module
categories_copy = copy.copy(categories)

print(f'Categories = {categories}\nCategories (Copied) = {categories_copy}')

Output:

Categories = {'Fruits': ['Apple', 'Banana', 'Mango'], 'Flowers': ['Rose', 'Sunflower', 'Tulip']}

Categories (Copied) = {'Fruits': ['Apple', 'Banana', 'Mango'], 'Flowers': ['Rose', 'Sunflower', 'Tulip']}

In the above example, we made a composite object called categories that contains lists as the inner objects. Then, we used the copy() function of the copy module to shallow copy the original object. Also, since the copy module is in-built, there is no need to install it manually! Now, both the objects appear similar.

Next, let’s modify the original object to see if the object is really copied or it is just referenced:

import copy

categories = {
    'Fruits' : ['Apple', 'Banana', 'Mango'],
    'Flowers' : ['Rose', 'Sunflower', 'Tulip'],
}

categories_copy = copy.copy(categories)

# added a key-value pair in the original dictionary.
categories['Color'] = ['Red', 'Yellow', 'Blue']

print(f'Categories = {categories}\nCategories (Copied) = {categories_copy}')

Output:

Categories = {'Fruits': ['Apple', 'Banana', 'Mango'], 'Flowers': ['Rose', 'Sunflower', 'Tulip'], 'Color': ['Red', 'Yellow', 'Blue']}

Categories (Copied) = {'Fruits': ['Apple', 'Banana', 'Mango'], 'Flowers': ['Rose', 'Sunflower', 'Tulip']}

Here, we can see that even when we changed the original dictionary, the copied dictionary (stored in variable categories_copy) remained the same. This means that we successfully copied the dictionary to a different memory location.

But we shallow copied the dictionary. We know that, for a shallow copied composite object, the inner objects point at same memory location as that of the original composite object. You can see this in the following example:

import copy

categories = {
    'Fruits' : ['Apple', 'Banana', 'Mango'],
    'Flowers' : ['Rose', 'Sunflower', 'Tulip'],
}

categories_copy = copy.copy(categories)

# checking if the inner object list 'Fruits' of both the dictionaries point to same memory address.
print(f"""
Do 'categories' and 'categories_copy' inner object share same memory address? 
--> {id(categories_copy['Fruits']) == id(categories['Fruits'])}
""")

# checking if the outer objects (dictionaries) point to same memory address.
print(f"""
Do 'categories' and 'categories_copy' outer object share same memory address? 
--> {id(categories_copy) == id(categories)}
""")

Output:

Do 'categories' and 'categories_copy' inner object share same memory address?

--> True

Do 'categories' and 'categories_copy' outer object share same memory address?

--> False

In the above code, we used the same example as earlier. Then, we made use of the in-built function id() to extract and compare the memory addresses of both the dictionaries.

For inner objects, the memory address is the same. But outer objects are located at different locations in the memory. Thus, we can say that while shallow copying objects, the inner objects are only referenced, while the outer objects are copied to a separate memory address.

On the other hand, the deepcopy() function of the copy module copies the object completely (both inner and outer objects are stored at different memory locations). The following code shows how we can deep copy the objects within our code:

import copy

categories = {
    'Fruits' : ['Apple', 'Banana', 'Mango'],
    'Flowers' : ['Rose', 'Sunflower', 'Tulip'],
}

# deep copying the dictionary
categories_copy = copy.deepcopy(categories)

print(f"""
Do 'categories' and 'categories_copy' inner object share same memory address? 
--> {id(categories_copy['Fruits']) == id(categories['Fruits'])}
""")

print(f"""
Do 'categories' and 'categories_copy' outer object share same memory address? 
--> {id(categories_copy) == id(categories)}
""")

Output:

Do 'categories' and 'categories_copy' inner object share same memory address?

--> False

Do 'categories' and 'categories_copy' outer object share same memory address?

--> False

In the code, we deep copied the dictionary by using the deepcopy() function. When we compared the memory addresses of the inner and the outer objects stored in both the identifiers, we can see that they are separately stored in the memory.

So you’ll use a shallow copy or a deep copy depending upon the situation.

For example, if you just want to copy the outer object and keep the nested object same for all, you should opt for a shallow copy. If you have defined a class to create students’ ID of grade X, then you might need to keep the self.grade = X for all the students. In such cases, you can just reference the nested object.

Also, for non-nested objects, the shallow copy method fulfills the purpose, as there are no nested objects and shallow copying completely copies the outer object to a different memory location.

On the other hand, if you want a complete, independent copy of the object, you should deep copy the object.

More About Copying Objects in Python

You can use the copy module for both immutable and mutable objects. But for immutable objects, you can also use assignment operator = for copying objects.

Now, as I mentioned earlier, in this case too, the object is referenced when you use the = operator. But, when you mutate immutable objects, the mutated objects get stored at a different memory location. This makes the alias of the original object an independent object, pointing at the same memory address as earlier.

Let’s understand this with an example:

str1 = "String" # created a string object

str2 = str1 # using '=' to reference the string stored in 'str1'

print(str2, str1, sep='\n') # printing the strings

Output:

String

String

Above, the variable str2 referenced the string stored in variable str1. Basically, str2 and str1 point at same memory address and str2 is just an alias of str1.

But if we go further and modify the string in str1, then str1 starts to point at a new memory location (since a string is immutable, if it’s modified it gets stored at a different memory address). But str2 will still point at the previous memory address, commonly fulfilling the purpose of copying the objects.

str1 = "String"

str2 = str1

# printing memory addresses of both the variables before mutation.
print(f"""
Memory address of str1: {id(str1)}
Memory address of str2: {id(str2)}
""")

str1+='***' # concatenated the string '***' with str1.

print(str2, str1, sep='\n')

# printing memory addresses of both the variables after mutation.
print(f"""
Memory address of str1: {id(str1)}
Memory address of str2: {id(str2)}
""")

Output:

Memory address of str1: 2652367074480

Memory address of str2: 2652367074480

String

String***

Memory address of str1: 2652367370736

Memory address of str2: 2652367074480

Note: Memory addresses may vary on your device from the ones shown above.

Now, in the above example, we first created a string object and referenced it in another variable, and printed the memory addresses of both the variables. Then, we modified the original string, and again, printed the memory addresses of both the variables.

In the output, we can see that the memory addresses of both the variables before mutation were the same. So you can see that both the variables were pointing at the same memory location. But after mutation, the variable str1 started pointing at a different memory location, thus making the alias str2 an independent object, which still points at the previous memory address.

To sum up, you can use the = operator for storing a copy of the original object if you plan to modify it further in the program.

Summary

In this tutorial, you learned about copying objects in Python. Specifically, we talked about:

• How the assignment operator = is used for referencing and not copying.

• The built-in copy module, which provides functions that allow us to shallow copy and deep copy the objects in our program.

• The concept of shallow copy and deep copy, which are essential when copying compound objects.

• How a shallow copy copies the outer object and references the inner objects.

• How a deep copy copies both the outer object and the inner objects.

• How for immutable objects, the assignment operator works fine for copying the objects most of the time.

Thanks for reading!