This tutorial demonstrates how to clean messy JSON and export the results into a new file, based on a predefined schema. The JSON file we’ll be cleaning contains a dataset of 200 synthetic customer records.

In this tutorial, we’ll apply two methods for cleaning the input data:

• With pure Python

• With pandas

You can apply either of these in your code. But the pandas method is better for large, complex data sets. Let’s jump right into the process.

Here’s what we’ll cover:

• Prerequisites

• Add and Inspect the JSON File

• Define the Target Schema

• How to Clean JSON Data with Pure Python

• How to Clean JSON Data with Pandas

• How to Validate the Cleaned JSON

• Pandas vs Pure Python for Data Cleaning

Prerequisites

To follow along with this tutorial, you should have a basic understanding of:

• Python dictionaries, lists, and loops

• JSON data structure (keys, values, and nesting)

• How to read and write JSON files with Python’s json module

Add and Inspect the JSON File

Before you begin writing any code, make sure that the .json file you intend to clean is in your project directory. This makes it easy to load in your script using the file name alone.

You can now inspect the data structure by viewing the file locally or loading it in your script, with Python’s built-in json module.

Here’s how (assuming the file name is “old_customers.json”):

This shows you whether the JSON file is structured as a dictionary or a list. It also prints out the entire file in your terminal. Mine is a dictionary that maps to a list of 200 customer entries. You should always open up the raw JSON file in your IDE to get a closer look at its structure and schema.

Define the Target Schema

If someone asks for JSON data to be cleaned, it probably means that the current schema is unsuitable for its intended purpose. At this point, you want to be clear on what the final JSON export should look like.

JSON schema is essentially a blueprint that describes:

• required fields

• field names

• data type for each field

• standardized formats (for example, lowercase emails, trimmed whitespace, etc.)

Here’s what the old schema versus the target schema looks like:

As you can see, the goal is to delete the ”customer_id” and ”address” fields in each entry and rename the rest from:

• ”name” to ”full_name”

• ”email” to ”email_address”

• ”phone” to ”mobile”

• ”membership_level” to ”tier”

The output should contain 4 response fields instead of 6, all renamed to fit the project requirements.

How to Clean JSON Data with Pure Python

Let’s explore using Python’s built-in json module to align the raw data with the predefined schema.

Step 1: Import json and time modules

Importing json is necessary because we’re working with JSON files. But we’ll use the time module to track how long the data cleaning process takes.

import json
import time

Step 2: Load the file with json.load()

start_time = time.time()
with open('old_customers.json') as file:
    crm_data = json.load(file)

Step 3: Write a function to loop through and clean each customer entry in the dictionary

def clean_data(records):
    transformed_records = []
    for customer in records["customers"]:
        transformed_records.append({
                "full_name": customer["name"],
                "email_address": customer["email"],
                "mobile": customer["phone"],
                "tier": customer["membership_level"],

                })
    return {"customers": transformed_records}

new_data = clean_data(crm_data)

clean_data() takes in the original data (temporarily) stored in the records variable, transforming it to match our target schema.

Since the JSON file we loaded is a dictionary containing a ”customers” key, which maps to a list of customer entries, we access this key and loop through each entry in the list.

In the for loop, we rename the relevant fields and store the cleaned entries in a new list called ”transformed_records”.

Then, we return the dictionary, with the ”customers” key intact.

Step 4: Save the output in a .json file

Decide on a name for your cleaned JSON data and assign that to an output_file variable, like so:

output_file = "transformed_data.json"
with open(output_file, "w") as f:
    json.dump(new_data, f, indent=4)

You can also add a print() statement below this block to confirm that the file has been saved in your project directory.

Step 5: Time the data cleaning process

At the beginning of this process, we imported the time module to measure how long it takes to clean up JSON data using pure Python. To track the runtime, we stored the current time in a start_time variable before the cleaning function, and we’ll now include an end_time variable at the end of the script.

The difference between the end_time and start_time values gives you the total runtime in seconds.

end_time = time.time()
elapsed_time = end_time - start_time

print(f"Transformed data saved to {output_file}")
print(f"Processing data took {elapsed_time:.2f} seconds")

Here’s how long the data cleaning process took with the pure Python approach:

How to Clean JSON Data with Pandas

Now we’re going to try achieving the same results as above, using Python and a third-party library called pandas. Pandas is an open-source library used for data manipulation and analysis in Python.

To get started, you need to have the Pandas library installed in your directory. In your terminal, run:

pip install pandas

Then follow these steps:

Step 1: Import the relevant libraries

import json
import time
import pandas as pd

Step 2: Load file and extract customer entries

Unlike the pure Python method, where we simply indexed the key name ”customers” to access the list of customer data, working with pandas requires a slightly different approach.

We must extract the list before loading it into a DataFrame because pandas expects structured data. Extracting the list of customer dictionaries upfront ensures that we isolate and clean the relevant records alone, preventing errors caused by nested or unrelated JSON data.

start_time = time.time()
with open('old_customers.json', 'r') as f:
    crm_data = json.load(f)

#Extract the list of customer entries
clients = crm_data.get("customers", [])

Step 3: Load customer entries into a DataFrame

Once you’ve got a clean list of customer dictionaries, load the list into a DataFrame and assign said list to a variable, like so:

#Load into a dataframe
df = pd.DataFrame(clients)

This creates a tabular or spreadsheet-like structure, where each row represents a customer. Loading the list into a DataFrame also allows you to access pandas’ powerful data cleaning methods like:

• drop_duplicate(): removes duplicate rows or entries from a DataFrame

• dropna(): drops rows with any missing or null data

• fillna(value): replaces all missing or null data with a specified value

• drop(columns): drops unused columns explicitly

Step 4: Write a custom function to rename relevant fields

At this point, we need a function that takes in a single customer entry – a row – and returns a cleaned version that fits the target schema (“full_name”, “email_address”, “mobile” and “tier”).

The function should also handle missing data by setting default values like ”Unknown” or ”N/A” when a field is absent.

P.S: At first, I used drop(columns) to explicitly remove the “address” and “customer_id” fields. But it’s not needed in this case, as the transform_fields() function only selects and renames the required fields. Any extra columns are automatically excluded from the cleaned data.

Step 5: Apply schema transformation to all rows

We’ll use pandas' apply() method to apply our custom function to each row in the DataFrame. This will creates a Series (for example, 0 → {...}, 1 → {...}, 2 → {...}), which is not JSON-friendly.

As json.dump() expects a list, not a Pandas Series, we’ll apply tolist(), converting the Series to a list of dictionaries.

#Apply schema transformation to all rows
transformed_df = df.apply(transform_fields, axis=1)

#Convert series to list of dicts
transformed_data = transformed_df.tolist()

Another way to approach this is with list comprehension. Instead of using apply() at all, you can write:

transformed_data = [transform_fields(row) for row in df.to_dict(orient="records")]

orient=”records” is an argument for df.to_dict that tells pandas to convert the DataFrame to a list of dictionaries, where each dictionary represents a single customer record (that is, one row).

Then the for loop iterates through every customer record on the list, calling the custom function on each row. Finally, the list comprehension ([...]) collects the cleaned rows into a new list.

Step 6: Save the output in a .json file

#Save the cleaned data
output_data = {"customers": transformed_data}
output_file = "applypandas_customer.json"
with open(output_file, "w") as f:
    json.dump(output_data, f, indent=4)

I recommend picking a different file name for your pandas output. You can inspect both files side by side to see if this output matches the result you got from cleaning with pure Python.

Step 7: Track runtime

Once again, check for the difference between start time and end time to determine the program’s execution time.

end_time = time.time()
elapsed_time = end_time - start_time

#print(f"Transformed data saved to {output_file}")
print(f"Transformed data saved to {output_file}")
print(f"Processing data took {elapsed_time:.2f} seconds")

When I used list comprehension to apply the custom function, my script’s runtime was 0.03 seconds, but with pandas’ apply() function, the total runtime dropped to 0.01 seconds.

Final output preview:

If you followed this tutorial closely, your JSON output should look like this – whether you used the pandas method or the pure Python approach:

How to Validate the Cleaned JSON

Validating your output ensures that the cleaned data follows the expected structure before being used or shared. This step helps to catch formatting errors, missing fields, and wrong data types early.

Below are the steps for validating your cleaned JSON file:

Step 1: Install and import jsonschema

jsonschema is a third-party validation library for Python. It helps you define the expected structure of your JSON data and automatically check if your output matches that structure.

In your terminal, run:

pip install jsonschema

Import the required libraries:

import json
from jsonschema import validate, ValidationError

validate() checks whether your JSON data matches the rules defined in your schema. If the data is valid, nothing happens. But if there’s an error – like a missing field or wrong data type – it raises a ValidationError.

Step 2: Define a schema

As you know, JSON schema changes with each file structure. If your JSON data differs from what we’ve been working with so far, learn how to create a schema here. Otherwise, the schema below defines the structure we expect for our cleaned JSON:

schema = {
    "type": "object",
    "properties": {
        "customers": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "full_name": {"type": "string"},
                    "email_address": {"type": "string"},
                    "mobile": {"type": "string"},
                    "tier": {"type": "string"}
                },
                "required": ["full_name", "email_address", "mobile", "tier"]
            }
        }
    },
    "required": ["customers"]
}

• The data is an object that must contain a key: "customers".

• "customers" must be an array (a list), with each object representing one customer entry.

• Each customer entry must have four fields–all strings:

  • "full_name"

  • "email_address"

  • "mobile"

  • "tier"

• The "required" fields ensure that none of the relevant fields are missing in any customer record.

Step 3: Load the cleaned JSON file

with open("transformed_data.json") as f:
    data = json.load(f)

Step 4: Validate the data

For this step, we’ll use a try. . . except block to end the process safely, and display a helpful message if the code raises a ValidationError.

try:
    validate(instance=data, schema=schema)
    print("JSON is valid.")
except ValidationError as e:
    print("JSON is invalid:", e.message)

Pandas vs Pure Python for Data Cleaning

From this tutorial, you can probably tell that using pure Python to clean and restructure JSON is the more straightforward approach. It is fast and ideal for handling small datasets or simple transformations.

But as data grows and becomes more complex, you might need advanced data cleaning methods that Python alone does not provide. In such cases, pandas becomes the better choice. It handles large, complex datasets effectively, providing built-in functions for handling missing data and removing duplicates.

You can study the Pandas cheatsheet to learn more data manipulation methods.

This statement is 100% accurate, as this encompasses a series of steps that ensure data used for data science, machine learning and analysis projects are complete, accurate, unbiased and reliable.

The quality of your dataset plays a pivotal role in the success of your analysis or model. As the saying goes, “garbage in, garbage out”, the quality and reliability of your model and analysis heavily depends on the quality of your data.

Raw data, collected from various sources, are often messy, contain errors, inconsistencies, missing values and outliers. Data cleaning and preprocessing aims to identify and rectify these issues to ensure accurate, reliable and meaningful results during model building and data analysis as wrong conclusions could be costly.

This is where Pandas comes into play, it is a wonderful tool used in the data world to do both data cleaning and preprocessing. In this article, we'll delve into the essential concepts of data cleaning and preprocessing using the powerful Python library, Pandas.

Table of Contents

• Prerequisites

• Introduction

• What is Data Cleaning?

• What is Data Processing?

• How to Import the Necessary Libraries

• How to Load the Dataset

• Exploratory Data Analysis (EDA)

• How to Handle Missing Values

• How to Remove Duplicate Records

• Data Types and Conversion

• How to Encode Categorical Variables

• How to Handle Outliers

• Conclusion

Prerequisites

• A basic understanding of Python.

• Basic understanding of data cleaning.

Introduction

Pandas is a popular open-source data manipulation and analysis library for Python. It provides easy-to-use functions needed to work with structured data seamlessly.

Pandas also integrates seamlessly with other popular Python libraries, such as NumPy for numerical computing and Matplotlib for data visualization. This makes it a powerful asset for data driven tasks.

Pandas excels in handling missing data, reshaping datasets, merging and joining multiple datasets, and performing complex operations on data, making it exceptionally useful for data cleaning and manipulation.

At its core, Pandas introduces two key data structures: Series and DataFrame. A Series is a one-dimensional array-like object that can hold any data type, while a DataFrame is a two-dimensional table with labeled axes (rows and columns). These structures allow users to manipulate, clean, and analyze datasets efficiently.

What is Data Cleaning?

Before we embark on our data adventure with Pandas, let's take a moment to explain the term "data cleaning." Think of it as the digital detox for your dataset, where we tidy up, and and prioritize accuracy above all else.

Data cleaning involves identifying and rectifying errors, inconsistencies, and missing values within a dataset. It's like preparing your ingredients before cooking; you want everything in order to get the perfect analysis or visualization.

Why bother with data cleaning? Well, imagine trying to analyze sales trends when some entries are missing, or working with a dataset that has duplicate records throwing off your calculations. Not ideal, right?

In this digital detox, we use tools like Pandas to get rid of inconsistencies, straighten out errors, and let the true clarity of your data shine through.

What is Data Processing?

You may be wondering, "Does data cleaning and data preprocessing mean the same thing?" The answer is no – they do not.

Picture this: you stumble upon an ancient treasure chest buried in the digital sands of your dataset. Data cleaning is like carefully unearthing that chest, dusting off the cobwebs, and ensuring that what's inside is authentic and reliable.

As for data preprocessing, you can think of it as taking that discovered treasure and preparing its contents for public display. It goes beyond cleaning; it's about transforming and optimizing the data for specific analyses or tasks.

Data cleaning is the initial phase of refining your dataset, making it readable and usable with techniques like removing duplicates, handling missing values and data type conversion while data preprocessing is similar to taking this refined data and scaling with more advanced techniques such as feature engineering, encoding categorical variables and and handling outliers to achieve better and more advanced results.

The goal is to turn your dataset into a refined masterpiece, ready for analysis or modeling.

How to Import the Necessary Libraries

Before we embark on data cleaning and preprocessing, let's import the Pandas library.

To save time and typing, we often import Pandas as pd. This lets us use the shorter pd.read_csv() instead of pandas.read_csv() for reading CSV files, making our code more efficient and readable.

import pandas as pd

How to Load the Dataset

Start by loading your dataset into a Pandas DataFrame.

In this example, we'll use a hypothetical dataset named your_dataset.csv. We will load the dataset into a variable called df.

#Replace 'your_dataset.csv' with the actual dataset name or file path
df = pd.read_csv('your_dataset.csv')

Exploratory Data Analysis (EDA)

EDA helps you understand the structure and characteristics of your dataset. Some Pandas functions help us gain insights into our dataset. We call these functions by calling the dataset variable plus the function.

For example:

• df.head() will call the first 5 rows of the dataset. You can specify the number of rows to be displayed in the parentheses.

• df.describe() gives some statistical data like percentile, mean and standard deviation of the numerical values of the Series or DataFrame.

• df.info() gives the number of columns, column labels, column data types, memory usage, range index, and the number of cells in each column (non-null values).

Here's a code example below:

#Display the first few rows of the dataset
print(df.head())

#Summary statistics
print(df.describe())

#Information about the dataset
print(df.info())

How to Handle Missing Values

As a newbie in this field, missing values pose a significant stress as they come in different formats and can adversely impact your analysis or model.

Machine learning models cannot be trained with data that has missing or "NAN" values as they can alter your end result during analysis. But do not fret, Pandas provides methods to handle this problem.

One way to do this is by removing the missing values altogether. Code snippet below:

#Check for missing values
print(df.isnull().sum())

#Drop rows with missing valiues and place it in a new variable "df_cleaned"
df_cleaned = df.dropna()

#Fill missing values with mean for numerical data and place it ina new variable called df_filled
df_filled = df.fillna(df.mean())

But if the number of rows that have missing values is large, then this method will be inadequate.

For numerical data, you can simply compute the mean and input it into the rows that have missing values. Code snippet below:

#Replace missing values with the mean of each column
df.fillna(df.mean(), inplace=True)

#If you want to replace missing values in a specific column, you can do it this way:
#Replace 'column_name' with the actual column name
df['column_name'].fillna(df['column_name'].mean(), inplace=True)

#Now, df contains no missing values, and NaNs have been replaced with column mean

How to Remove Duplicate Records

Duplicate records can distort your analysis by influencing the results in ways that do not accurately show trends and underlying patterns (by producing outliers).

Pandas helps to identify and remove the duplicate values in an easy way by placing them in new variables.

Code snippet below:

#Identify duplicates
print(df.duplicated().sum())

#Remove duplicates
df_no_duplicates = df.drop_duplicates()

Data Types and Conversion

Data type conversion in Pandas is a crucial aspect of data preprocessing, allowing you to ensure that your data is in the appropriate format for analysis or modeling.

Data from various sources are usually messy and the data types of some values may be in the wrong format, for example some numerical values may come in 'float' or 'string' format instead of 'integer' format and a mix up of these formats leads to errors and wrong results.

You can convert a Column of type int to float with the following code:

#Convert 'Column1' to float
df['Column1'] = df['Column1'].astype(float)

#Display updated data types
print(df.dtypes)

You can use df.dtypes to print column data types.

How to Encode Categorical Variables

For machine learning algorithms, having categorical values in your dataset (non-numerical values) is crucial in ensuring the best model as they are equally as important.

These could be car brand names in a cars dataset for predicting car prices. But machine learning algorithms cannot processes this datatype, therefore it must be converted to numerical data before it can be used.

Pandas provides the get_dummies function which converts categorical values into numerical format(Binary format) such that it is recognized by the algorithm as a placeholder for values and not hierarchical data that can undergo numerical analysis. this just means that the numbers the brand name is converted to is not interpreted as 1 is greater than 0, but it tells the algorithm that both 1 and 0 are placeholders for categorical data. Code snippet is shown below:

#To convert categorical data from the column "Car_Brand" to numerical data
df_encode = pd.get_dummies(df, columns=[Car_Brand])

#The categorical data is converted to binary format of Numerical data

How to Handle Outliers

Outliers are data points significantly different from the majority of the data, they can distort statistical measures and adversely affect the performance of machine learning models.

They may be caused by human error, missing NaN values, or could be accurate data that does not correlate with the rest of the data.

There are several methods to identify and remove outliers, they are:

• Remove NaN values.

• Visualize the data before and after removal.

• Z-score method (for normally distributed data).

• IQR (Interquartile range) method for more robust data.

The IQR is useful for identifying outliers in a dataset. According to the IQR method, values that fall below Q1−1.5× IQR or above Q3+1.5×IQR are considered outliers.

This rule is based on the assumption that most of the data in a normal distribution should fall within this range.

Here's a code snippet for the IQR method:

#Using median calculations and IQR, outliers are identified and these data points should be removed
Q1 = df["column_name"].quantile(0.25)
Q3 = df["column_name"].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
df = df[df["column_name"].between(lower_bound, upper_bound)]

Conclusion

Data cleaning and preprocessing are integral components of any data analysis, science or machine learning project. Pandas, with its versatile functions, facilitates these processes efficiently.

By following the concepts outlined in this article, you can ensure that your data is well-prepared for analysis and modeling, ultimately leading to more accurate and reliable results.

We just published a course on the freeCodeCamp.org YouTube channel that will teach you how to use Pandas through interactive projects. You will develop 7 projects ranging from the basics of Pandas for Data Analysis, to Data Cleaning and Data Wrangling.

This course targets everyone, from data science enthusiasts to professionals, aiming to refine their skills in data analysis, data cleaning, and data wrangling using Pandas and Python.

Santiago Basulto developed this course. His is knowledgeable and has an engaging teaching style, guarantees an enriching learning experience. Santiago is also the creator of datawars.io, a platform that offers a bunch of interactive data science projects.

A Sneak Peek into the Course

The course is designed to provide you with hands-on experience through real-life projects. The projects vary in complexity, catering to learners with different skill levels. It’s advisable to try resolving the projects independently and then compare your solutions with Santiago’s.

Below are the projects you will build.

For Beginners:

DataFrames Practice: Working with English Words: This project is an excellent entry point for beginners. You will get acquainted with the basics of Pandas DataFrames, focusing on understanding and manipulating their structures, all while working with an extensive dictionary of English words.

Filtering and Sorting with Pokemon Data: Dive into the captivating world of Pokemon as you perform fundamental data analysis tasks like filtering and sorting. This project is not only educational but also fun, making it perfect for those just starting.

Intermediate Level:

The Birthday Paradox in the NBA: Have you heard of the Birthday Paradox? Discover the answer to an intriguing question: How many people need to be in a room to have a 50% probability that at least two people share a birthday? Apply these insights to explore shared player birthdays within NBA teams.

Matching Strings by Similarity using Levenshtein Distance: String handling is an integral aspect of data cleaning. This project introduces advanced techniques such as Combinatorics and the Levenshtein distance to detect irregularities in company names.

Data Cleaning with Google Playstore Dataset: This project is a comprehensive guide to data cleaning. Learn how to identify and rectify null values, duplicate values, outliers, and more, using a dataset scraped from the Google Playstore, which is replete with irregularities.

Advanced:

Premier League Match Analysis: This project is designed for advanced learners. It combines data cleaning with analysis based on grouping operations, using data from the Premier League, the top-tier football league in England.

NBA 2017 Season Analysis: Joining and Groupby Practice: This project is a test of your data wrangling skills. Learn how to merge different dataframes, clean them, and perform analysis and question/answering tasks using the 2017 NBA statistics dataset.

Conclusion

Data is the new oil, and learning how to refine it is a skill that is in high demand. The Pandas course on freeCodeCamp.org, developed by Santiago Basulto, is a great opportunity to enhance your data science career. You can watch the full course on the freeCodeCamp.org YouTube channel (5-hour watch).

A dataframe is two-dimensional data structure in Pandas that organizes data in a tabular format with rows and columns.

In this article, you'll learn how to get the number of rows in a dataframe using the following:

• The len() function.
• The shape attribute.
• The index attribute.
• The axes attribuite.

How to Get the Number of Rows in a Dataframe Using the len() Function

You can use the len() function to return the length of an object. With a dataframe, the function returns the number of rows.

Consider the dataframe below:

import pandas as pd

data = {
  "name": ["John", "Jane", "Jade"],
  "age": [2, 10, 3]
}

df = pd.DataFrame(data)
df

In the example above, we created a dataframe with three rows — row 0, 1, and 2.

You can use the len() function to verify the number of rows:

import pandas as pd

data = {
  "name": ["John", "Jane", "Jade"],
  "age": [2, 10, 3]
}

df = pd.DataFrame(data)
df

num_of_rows = len(df)

print(f"The number of rows is {num_of_rows}")
# The number of rows is 3

In the code above, we passed the dataframe as a parameter to the len() function and stored it in a variable called num_of_rows:

num_of_rows = len(df)

When num_of_rows was printed, we got a value of 3 (the number of rows).

How to Get the Number of Rows in a Dataframe Using the shape Attribute

The shape attribute returns a tuple with the number of rows and columns in a dataframe.

Here's an example using the same dataframe as in the last section:

import pandas as pd

data = {
  "name": ["John", "Jane", "Jade"],
  "age": [2, 10, 3]
}

df = pd.DataFrame(data)
df

num_of_rows = df.shape

print(num_of_rows)
# (3, 2)

In the code above, a tuple — (3, 2) — was returned when we used the shape attribute on the dataframe: df.shape.

The first value, 3, is the number of rows in the dataframe while the second value, 2, is the number of columns.

Since we're only interested in the number of rows, we can extract just that value using its index in the tuple (remember that index numbers start at 0). That is:

import pandas as pd

data = {
  "name": ["John", "Jane", "Jade"],
  "age": [2, 10, 3]
}

df = pd.DataFrame(data)
df

num_of_rows = df.shape[0]

print(f"The number of rows is {num_of_rows}")
# The number of rows is 3

Now we're getting just the number of rows using its index in the tuple: df.shape[0].

How to Get the Number of Rows in a Dataframe Using the index Attribute

You can use the index attribute to access the number of elements in a dataframe, which corresponds with the number of rows.

You can do this in two different ways:

• Using the index attribute's size property.
• Passing the index property as a parameter to the len() function.

Here are examples to explain the methods above:

import pandas as pd

data = {
  "name": ["John", "Jane", "Jade"],
  "age": [2, 10, 3]
}

df = pd.DataFrame(data)
df

num_of_rows = df.index.size

print(f"The number of rows is {num_of_rows}")
# The number of rows is 3

In the example above, we accessed the number of rows in the dataframe using df.index.size.

Without the size property, you'd get a result like this: RangeIndex(start=0, stop=3, step=1).

• start denotes the first index number.
• stop denotes the number of rows in the dataframe.
• step denotes the way indexes are incremented (indexes are increased by 1 in our case).

So the size property is way of specifying that you're only interested in the number of elements in the dataframe.

Here's another example that uses the len() function:

import pandas as pd

data = {
  "name": ["John", "Jane", "Jade"],
  "age": [2, 10, 3]
}

df = pd.DataFrame(data)
df

num_of_rows = len(df.index)

print(f"The number of rows is {num_of_rows}")
# The number of rows is 3

In the code above, we passed df.index as a parameter to the len() function. This returns the number of rows in the dataframe.

The difference between this example and the previous one is that we're not attaching the size property to df.index. Instead, we're using df.index as the len() function's parameter.

How to Get the Number of Rows in a Dataframe Using the axes Attribute

The axes attribute returns the value as the index attribute: RangeIndex(start=0, stop=3, step=1).

Similarly, you can return the number of rows using either the size property or the len() function:

import pandas as pd

data = {
  "name": ["John", "Jane", "Jade"],
  "age": [2, 10, 3]
}

df = pd.DataFrame(data)
df

num_of_rows = df.axes[0].size

print(f"The number of rows is {num_of_rows}")
# The number of rows is 3

import pandas as pd

data = {
  "name": ["John", "Jane", "Jade"],
  "age": [2, 10, 3]
}

df = pd.DataFrame(data)
df

num_of_rows = len(df.axes[0])

print(f"The number of rows is {num_of_rows}")
# The number of rows is 3

The logic in the two code blocks above is the same as those in the last section:

• df.index.size returns the number of elements/rows in the dataframe.
• len(df.index) returns the number of rows in the dataframe.

Summary

In this article, we talked about dataframes in Pandas. They are two-dimensional data structures that organize data in rows and columns.

We saw different methods for getting the number of rows in a dataframe. We discussed the following methods along with code examples to show their application:

• The len() function.
• The shape attribute.
• The index attribute.
• The axes attribuite.

Happy coding! You can learn more about Python on my blog.

In this article, you'll learn about the differences between Python and Pandas, and what they are used for.

You'll start by learning what each technology is used for, then you'll see their differences in tabular format.

Let's get started!

What Is Python Used For?

Python is a popular high-level, general-purpose programming language. Python has a simple syntax that is easy to read, write and understand.

It has an active open source community.

Python has a variety of libraries and frameworks that can be used in developing different applications and products.

Here are some of the use cases for Python:

• Web development: Python can be used to create full-stack web applications using frameworks Django, and Flask.
• Machine Learning: You can use Python libraries like TensorFlow, PyTorch, and so on to build machine learning models.
• Game Development: Ren'Py, Pygame, and Panda3D are some of the Python frameworks that can be used to build cross-platform games.
• Data Science and Analysis: There are numerous libraries and frameworks that can be used in data science and analysis like Matplotlib, NumPy, Pandas, and so on. They can also be used for scientific computing.

From the use cases above, you should have an idea of the first difference between Python and Pandas — Python is a programming language while Pandas is a Python library.

What Is Pandas Used For?

Pandas is an open source Python library used for manipulating and analyzing data.

Here are some of the use cases:

• Data manipulation.
• Data analysis.
• Data visualization.
• Machine learning, and so on.

What Are the Differences Between Python and Pandas?

Here are some of the differences between Python and Pandas:

Summary

In this article, we talked about the differences between Python and Pandas. We saw what each of them is used for, and their differences in tabular format.

Python is a general-purpose programming language used in different fields like web development, machine learning, and so on.

Pandas is a Python library used mainly for data manipulation and analysis.

Happy coding!

This article provides a comprehensive guide on how to loop through a Pandas DataFrame in Python.

I'll start by introducing the Pandas library and DataFrame data structure. I'll explain the essential characteristics of Pandas, how to loop through rows in a dataframe, and finally how to loop through columns in a dataframe.

What is Pandas?

Pandas is a popular open-source Python library that's used for data cleaning, analysis, and manipulation. In addition to functions for carrying out operations on those datasets, it offers data structures for effectively storing and handling large and complex datasets.

Some of the essential characteristics of Pandas are:

• DataFrame and Series Objects: Pandas provides two primary data structures, DataFrames and Series. They allow users to store and manipulate tabular data and time series data, respectively. These data structures are highly efficient and can handle large datasets with ease.
• Data Cleaning and Preparation: Pandas provides a wide range of functions and methods for cleaning and preparing data, including handling missing values, removing duplicates, and transforming data.
• Data Analysis and Visualization: Pandas provides powerful functions for performing data analysis, including statistical functions and grouping and aggregation functions. It also integrates well with other data analysis and visualization libraries in Python, such as Matplotlib and Seaborn.
• Data Input and Output: Pandas provides functions for reading and writing data in a variety of formats, including CSV, Excel, SQL databases, and more.

What is a Pandas Dataframe?

In Pandas, a dataframe is a two-dimensional labeled data structure. It is comparable to a spreadsheet or a SQL table, where data is arranged in rows and columns with a variety of data types in each column.

Since dataframes offer an easy way to store, manipulate, and analyze data, they are frequently used in data science and data analysis applications. Dataframes provide a number of features, including pivoting, grouping, indexing, and filtering, that make it simple to carry out complex operations on data.

How to Loop Through Rows in a Dataframe

You can loop through rows in a dataframe using the iterrows() method in Pandas. This method allows us to iterate over each row in a dataframe and access its values.

Here's an example:

import pandas as pd

# create a dataframe
data = {'name': ['Mike', 'Doe', 'James'], 'age': [18, 19, 29]}
df = pd.DataFrame(data)

# loop through the rows using iterrows()
for index, row in df.iterrows():
    print(row['name'], row['age'])

Output:

Mike 18
Doe 19
James 29

In this example, we first create a dataframe with two columns, name and age. We then loop through each row in the dataframe using iterrows(), which returns a tuple containing the index of the row and a Series object that contains the values for that row.

Within the loop, we can access the values for each column by using the column name as an index on the row object. For example, to access the value for the name column, we use row['name'].

How to Loop Through Columns in a Dataframe

Looping through columns in a dataframe is a common task in data analysis and manipulation. It's different from the way we loop through rows, though.

Here's an example:

import pandas as pd

# Create a sample dataframe
df = pd.DataFrame({
    'A': [1, 2, 3],
    'B': [4, 5, 6],
    'C': [7, 8, 9]
})

# Loop through columns using a for loop
for col in df.columns:
    print(col)

Output:

A
B
C

First, we import the Pandas library using the import pandas as pd statement.

Then, we create a sample dataframe using the pd.DataFrame() function, which takes a dictionary of column names and values as an input.

Next, we loop through the columns of the dataframe using a for loop and the df.columns attribute, which returns a list of column names.

Inside the loop, we simply print the name of each column using the print() function.

Use Cases for Looping Through a Dataframe

Looping through a dataframe is an important technique in data analysis and manipulation, as it allows us to perform operations on each row or column of the dataframe.

You'll loop through dataframes in the following activities:

• Data Cleaning and Transformation.
• Data Analysis.
• Data Visualization.
• Feature Engineering.

Conclusion

By looping through the rows in a dataframe, we can perform operations on each row, such as filtering or transforming the data.

But it's important to note that looping through rows in a dataframe can be slow and inefficient for large datasets. In general, it's often better to use vectorized operations or apply() functions to perform operations on dataframes, as these methods are optimized for performance.

Let's connect on Twitter and on LinkedIn. You can also subscribe to my YouTube channel.

Happy Coding!

In Pandas, sometimes you'll need to remove columns from a DataFrame for various reasons, such as cleaning data, reducing memory usage, or simplifying analysis. And in this article, I'll show you how to do it.

I'll start by introducing the .drop() method, which is the primary method for removing columns in Pandas. We'll go through the syntax and parameters of the .drop() method, including how to specify columns to remove and how to control whether the original DataFrame is modified in place or a new DataFrame is returned.

Next, I'll provide an example of how to use the .drop() method to remove columns from a DataFrame.

How to Use the .drop() Method in Pandas

The .drop() method is a built-in function in Pandas that allows you to remove one or more rows or columns from a DataFrame. It returns a new DataFrame with the specified rows or columns removed and does not modify the original DataFrame in place, unless you set the inplace parameter to True.

The syntax for using the .drop() method is as follows:

DataFrame.drop(labels=None, axis=0, index=None, columns=None, level=None, inplace=False, errors='raise')

Here, DataFrame refers to the Pandas DataFrame that you want to remove rows or columns from. The parameters you can use with the .drop() method include:

• labels: This parameter specifies the labels or indices of the rows or columns to be removed. You can pass either a single label or index or a list of labels or indices.
• axis: This parameter specifies whether to remove rows or columns. By default, it is set to 0, which means rows are removed. If you want to remove columns, set it to 1.
• index and columns: These parameters are alternative to the labels parameter and specify the labels or indices of rows or columns to be removed, respectively.
• level: This parameter is used to remove a specific level of a hierarchical index.
• inplace: This parameter is a boolean value that determines whether to modify the original DataFrame in place. By default, it is set to False.
• errors: This parameter specifies how to handle errors if the specified label(s) or index(es) are not found in the DataFrame. By default, it is set to raise, which means that a KeyError is raised. Other options are ignore and warn, which will respectively ignore or display a warning when the label/index is not found.

How to Remove a Single Column from a Dataframe in Pandas

Let's ease into it by first learning how to remove a single column from a Dataframe before we remove multiple columns.

Code sample:

import pandas as pd

# create a sample dataframe
data = {'name': ['Alice', 'Bob', 'Charlie'],
        'age': [25, 30, 35],
        'gender': ['F', 'M', 'M']
        }
df = pd.DataFrame(data)

# display the original dataframe
print('Original DataFrame:\n', df)

# drop the 'gender' column
df = df.drop(columns=['gender'])

# display the modified dataframe
print('Modified DataFrame:\n', df)

Output:

Original DataFrame:
       name  age gender
0    Alice   25      F
1      Bob   30      M
2  Charlie   35      M

Modified DataFrame:
       name  age
0    Alice   25
1      Bob   30
2  Charlie   35

Code explanation:

In the example above, we first created a sample DataFrame with three columns – name, age, and gender. We then used the .drop() method with the columns parameter to remove the gender column. The resulting DataFrame only contains the name and age columns.

It's important to note that the .drop() method does not modify the original DataFrame in place. Instead, it returns a new DataFrame with the specified column(s) removed. If you want to modify the original DataFrame, you need to assign the result of the .drop() method back to the original variable, as we did in the example above.

In addition to the columns parameter, the .drop() method also has a number of other optional parameters you can use to control how columns are removed.

For example, you can use the inplace parameter to modify the original DataFrame in place instead of returning a new DataFrame. You can also use the axis parameter to remove columns by index instead of name.

How to Remove Multiple Columns from a Dataframe in Pandas

In this section we will remove multiple columns from our dataframe. This approach is similar to removing a single column from the dataframe.

To remove two or more columns from a DataFrame using the .drop() method in Pandas, we can pass a list of column names to the columns parameter of the method.

Code sample:

import pandas as pd

# create a sample dataframe
data = {'name': ['John', 'Mary', 'Peter'],
        'age': [30, 25, 35],
        'gender': ['Male', 'Female', 'Male'],
        'city': ['New York', 'London', 'Paris']}
df = pd.DataFrame(data)

# remove the 'gender' and 'city' columns
df.drop(columns=['gender', 'city'], inplace=True)

# print the modified dataframe
print(df)

Output:

    name  age
0   John   30
1   Mary   25
2  Peter   35

Code explanation:

In this example, we first create a sample DataFrame with four columns – name, age, gender, and city. Then, we use the .drop() method to remove the gender and city columns by passing a list of their names to the columns parameter. Finally, we set the inplace parameter to True to modify the original DataFrame and print the modified DataFrame.

Note that you can also remove columns by their indices by passing a list of indices to the columns parameter. For example, to remove the second and third columns, you can use:

df.drop(columns=df.columns[1:3], inplace=True)

This will remove the columns with indices 1 and 2 (that is the age and gender columns in this example).

Conclusion

I hope this article is a useful resource for anyone working with Pandas DataFrames who needs to remove columns efficiently and effectively.

Let's connect on Twitter and on LinkedIn. You can also subscribe to my YouTube channel.

Happy Coding!

Pandas is a widely used open-source library in Python for data manipulation and analysis. It provides a range of data structures and functions for working with data, one of which is the DataFrame.

DataFrames are a powerful tool for storing and analyzing large sets of data, but they can be challenging to work with if they are not saved or exported correctly.

It is common practice in data analysis to export data from Pandas DataFrames into CSV files because it can help conserve time and resources. Due to their portability and ability to be easily read by numerous applications, CSV files are a common file format for storing and distributing tabular data.

Regardless of whether you are a novice or an expert data analyst, this article will walk you through the process of saving Pandas DataFrames into CSV files and give you useful tips on how to do so.

How to Save Pandas DataFrames Using the .to_csv() Method

The .to_csv() method is a built-in function in Pandas that allows you to save a Pandas DataFrame as a CSV file. This method exports the DataFrame into a comma-separated values (CSV) file, which is a simple and widely used format for storing tabular data.

The syntax for using the .to_csv() method is as follows:

DataFrame.to_csv(filename, sep=',', index=False, encoding='utf-8')

Here, DataFrame refers to the Pandas DataFrame that we want to export, and filename refers to the name of the file that you want to save your data to.

The sep parameter specifies the separator that should be used to separate values in the CSV file. By default, it is set to , for comma-separated values. We can also set it to a different separator like \t for tab-separated values.

The index parameter is a boolean value that determines whether to include the index of the DataFrame in the CSV file. By default, it is set to False, which means the index is not included.

The encoding parameter specifies the character encoding to be used for the CSV file. By default, it is set to utf-8, which is a standard encoding for text files.

Code example

import pandas as pd

# Create a sample dataframe
Biodata = {'Name': ['John', 'Emily', 'Mike', 'Lisa'],
        'Age': [28, 23, 35, 31],
        'Gender': ['M', 'F', 'M', 'F']
        }
df = pd.DataFrame(Biodata)

# Save the dataframe to a CSV file
df.to_csv('Biodata.csv', index=False)

Code explanation

Let's break down what each part of this code does:

• import pandas as pd: This imports the Pandas library and assigns it the alias pd, which is a commonly used convention.
• Biodata = {'Name': ['John', 'Emily', 'Mike', 'Lisa'], 'Age': [28, 23, 35, 31], 'Gender': ['M', 'F', 'M', 'F']}: This creates a Python dictionary with the data we want to store in the DataFrame. Each key represents a column in the DataFrame, and its corresponding value is a list of values for that column.
• df = pd.DataFrame(Biodata): This creates a Pandas DataFrame from the Biodata dictionary.
• df.to_csv('Biodata.csv', index=False): This saves the DataFrame to a CSV file named Biodata.csv.

Other Ways to Save Pandas DataFrames

There are several alternative methods to .to_csv() for saving Pandas DataFrames into various file formats, including:

1. to_excel(): This method is used to save a DataFrame as an Excel file.
2. to_json(): This method is used to save a DataFrame as a JSON file.
3. to_hdf(): This method is used to save a DataFrame as an HDF5 file, which is a hierarchical data format commonly used in scientific computing.
4. to_sql(): This method is used to save a DataFrame to a SQL database.
5. to_pickle(): This method is used to save a DataFrame as a pickled object, which is a serialized representation of the DataFrame.

These alternative methods provide flexibility in choosing the file format that best suits your use case and can be particularly useful for advanced data analysis and sharing.

Conclusion

Thanks for reading! I hope you now understand how you can easily convert your Pandas Dataframes by exporting into a CSV file using the build-in to_csv() method.

Let's connect on Twitter and on LinkedIn. You can also subscribe to my YouTube channel.

Happy Coding!

In this article, you'll learn how to sort data in ascending and descending order using Pandas' sort_values() method.

How To Sort Values in Pandas

You can use the sort_values() method to sort values in a data set. By default, the method sorts values in ascending order.

In this section, you'll learn how to sort data in ascending and descending order using the sort_values() method.

How To Sort Values in Ascending Order Using Pandas sort_values() Method

The sort_values() method takes in multiple parameters, as can be seen in the Pandas documentation.

We'll focus on the by and ascending parameters. That is:

Dataframe.sort_values(by, ascending)

• The by parameter denotes the column or index to sort.
• ascending is used to specify what other the values should be sorted in. By default, it is set to True.

Here's an example:

import pandas as pd

# create a sample dataframe
data = {'cost': [50000, 30000, 70000, 60000]}

df = pd.DataFrame(data)

df

In the table above, we have different items along with the cost of each item. To sort the items in ascending order using their cost, you can do this:

import pandas as pd

data = {'item': ['laptop', 'monitor', 'HDMI', 'speaker'],
        'cost': [500, 300, 700, 600]
       }

df = pd.DataFrame(data)

sorted_data = df.sort_values(by='cost', ascending=True)

sorted_data

In the code above, the sort_values() method was used to sort the cost column.

• Using the by parameter, we specified which column was to be sorted: by='cost'
• Using the ascending parameter, we set the order of the data to be sorted: ascending=True.

Note that the default order of the sort_values() method is ascending=True. So if you remove ascending parameter, you'd still have the values sorted in ascending order.

How To Sort Values in Descending Order Using Pandas sort_values() Method

You can sort values in descending order by simply setting the ascending parameter to False.

We'll work with the same code in the last section:

import pandas as pd

# create a sample dataframe
data = {'cost': [50000, 30000, 70000, 60000]}

df = pd.DataFrame(data)

df

Here's the code for sorting the cost column in descending order:

import pandas as pd

data = {'item': ['laptop', 'monitor', 'HDMI', 'speaker'],
        'cost': [500, 300, 700, 600]
       }

df = pd.DataFrame(data)

sorted_data = df.sort_values(by='cost', ascending=False)

sorted_data

By setting the value of the ascending parameter to False, we've sorted the data by cost in descending order.

Summary

In this article, we learned about sorting values in Pandas using the sort_values() method.

We saw two code examples on how to sort data in Pandas in ascending or descending order.

You can use the sort_values() method's ascending parameter to sort data in ascending or descending order.

Happy coding!

In this article, you'll learn how to use the Pandas round() method to round a float value to a specified number of decimal places.

We'll begin by looking the method's syntax, and then see some practical code applications.

Pandas round() Method Example

Here's what the syntax for the round() method looks like:

DataFrame.round(decimals)

The decimals parameter represents the number of decimal places a number should be rounded to.

The number of decimal places to be returned is passed in as a parameter. round(2) return rounds a number to two decimal places.

Here's an example to demonstrate:

import pandas as pd

data = {'cost':[20.5550, 21.03535, 19.67373, 18.233233]}

df = pd.DataFrame(data)

df['rounded_cost'] = df['cost'].round(2)
print(df)

In the code above, we have a list of numbers that fall under the cost column. The column had these values: [20.5550, 21.03535, 19.67373, 18.233233].

Using the round() method, we rounded the values to 2 decimal places: df['cost'].round(2).

The return values were stored in a column called rounded_cost.

Here's the output of the code:

From the table above, you can see that the values in the cost column have been rounded to 2 decimal places in the rounded_cost columns.

Summary

In this article, we have learned about rounding float values with Pandas using the round() method.

We started looking at the syntax for the round() method looks like. We then saw an example using the method to round float values to a specified number of decimal places.

Happy coding!

Pandas is a fast and approachable open-source library in Python built for analyzing and manipulating data.

This library has a lot of functions and methods to expedite the data analysis process. One of my favorites is the groupby method, mainly because it lets you get quick insights into your data by transforming, aggregating, and splitting data into various categories.

In this article, you will learn about the Pandas groupby function, how to aggregate data, and group Pandas DataFrames with multiple columns using the groupby method.

What do I need to install on my computer to follow this article?

For this article, I'll be using a Jupyter notebook. You can install Jupyter notebook and get it up and running on your computer via the official website.

After installing Juypter, create a new notebook and run Import pandas as pd to import pandas and Import numpy as np to import NumPy.

NumPy will let us work with multi-dimensional arrays and high-level mathematical functions. On the other hand, Pandas will allow us to manipulate our data and access the df.groupby(), the groupby method.

Let's get started.

Importing the required libraries

What is groupby in Pandas?

If you're familiar with SQL and its GROUP BY syntax, you already know how powerful it is in summarizing and categorizing data.

The Pandas groupby method in Python does the same thing and is great when splitting and categorizing data into groups to analyze your data better.

Here is the syntax for Pandas groupby:

python DataFrame.groupby(by=None, axis=0, level=None, as_index=True, sort=True, group_keys=_NoDefault.no_default, squeeze=_NoDefault.no_default, observed=False, dropna=True)

Each attribute has a meaning:

• by – List of the columns you want to group by.
• axis – Defaults to 0. It takes 0 or 'index', 1 or 'columns'.
• level – Used with MultiIndex.
• as_index – SQL style grouped otput.
• sort – Defaults to True. Specify whether to sort after grouping.
• group_keys – add group keys or not.
• squeeze – deprecated in new versions.
• observed – Only use if any of the groupers are Categoricals.
• dropna – Defaults to False. Use True to drop None/Nan.

Now let's see how this function works in action.

How to Load the Dataset

For this tutorial, we'll use the supermarket sales dataset from Kaggle, which you can access and download here.

After downloading the dataset, load the data into a pandas dataframe.

A DataFrame is a 2-dimensional data structure made up of rows and columns. This is very similar to your spreadsheet.

You can do that by running this code:

df = pd.read_csv(r"C:\Users\Double Arkad\Downloads\archive\supermarket_sales - Sheet1.csv")

After that, use the df.head() method to show the first few rows of your dataset. After running df.head(), you should get the result below. This indicates that the dataset got loaded successfully.

Dataset has loaded correctly

How to Use the groupby Method in Pandas

Assume your employer asked you to total the number of items ordered and categorize them according to the different payment options. This will let you determine which payment method generates the most revenue.

You can answer this question with the groupby function by simply grouping the data based on the 'Payment'.

df.groupby('Payment')['Quantity'].sum()

Using the sum function with groupby

The first column, 'Payments', is the column you want to group by. The second column, 'Quantity' is the column you'll perform an aggregate function on. Lastly, you have the aggregate function .sum().

The Sum() is one of many functions you can use in a groupby. You could also use other aggregate functions like the Min(), Mean(), Median(), Count(), and Average() to find the minimum, mean, median, count, and average value in a group within your dataset.

But by using the agg() function, you can perform two or more aggregations simultaneously.

Let's see how that works.

How to Aggregate Data Using groupby in Pandas

Pandas groupby and Agg()

Here's how to use agg() in a groupby function to find this supermarket's most used payment method.

df.groupby('Payment')['Quantity'].agg([np.sum, np.mean])

Using the agg function with groupby

There are more cash transactions done. Ewallets and credit card transactions follow in level of use.

Notice here we created a dictionary and passed the aggregate functions to be performed. This simultaneously performed two statistical computations on our data! Of course, you can add more aggregate functions in the dictionary depending on the insights you want to get.

Here is what I mean:

df.groupby(['Payment', 'Customer type'])['Quantity'].agg([np.sum, np.mean, np.max, np.min])

Adding more aggregate functions

In the groupby function, we added more aggregate functions to our statistical computation to gain insight into the maximum and the minimum number of goods ordered in each payment group.

Pandas groupby and count()

Here's how it works:

df.groupby('Payment')['Quantity'].count()

And here's the result you get:

Using the count function with groupby

From the output, we're counting the total number of orders placed in the store and grouping the results by each payment method.

How to Group Pandas DataFrames by Multiple Columns

You can also group multiple columns in the groupby function. For example, we included a column below to our groupby function called 'Customer type'.

df.groupby(['Payment', 'Customer type'])['Quantity'].sum()

Grouping multiple columns

Our output shows that the data was split and categorized into two groups based on the Customer type column. The output is becoming easier to analyze.

How to Aggregate Multiple Columns Using Pandas groupby

You can also perform statistical computations on multiple columns with the groupby function. For example, let's look at the total sales generated and quantity ordered and group our results by the "Payment" and "Customer type" columns.

Run the code:

df.groupby(['Payment', 'Customer type']) [['Quantity','Unit price']].sum()

Using the sum function with multiple columns grouping

We can see from the output that the Payment type "Ewallet" generated the most revenue, and you can move on to determine which type of Customers contributed the most revenue for the Store.

Summary

In this article, you learned about the importance of the Pandas groupby method. You saw how the groupby function allows you to do a lot of operations on your data, from splitting the data to applying a function like Sum() to get more insight and add more functionality.

To learn more about Python and how you can use it for data analysis, I'll recommend this Python for data analysis course on the freeCodeCamp YouTube channel.

If you enjoyed reading this article and/or have questions and want to connect, you can find me on LinkedIn or Twitter.

In this article, you'll learn how to rename columns in a Pandas Dataframe by using:

• The rename() function.
• A List.
• The set_axis() function.

How to Rename a Column in Pandas Using the rename() Function

In this section, you'll see a practical example of renaming a Pandas Dataframe using the rename() function.

Let's begin by passing data into a Dataframe object:

import pandas as pd

students = {
    "firstname": ["John", "Jane", "Jade"], 
    "lastname": ["Doe", "Done", "Do"]
}

# convert student names into a Dataframe
df = pd.DataFrame(students)

print(df)

# Output
  firstname lastname
0      John      Doe
1      Jane     Done
2      Jade       Do

In the example above, we created a Python dictionary which we used to store the firstname and lastname of students.

We then converted the dictionary to a Dataframe by passing it as a parameter to the Pandas Dataframe object: pd.DataFrame(students).

When printed to the console, we had this table printed out:

  firstname lastname
0      John      Doe
1      Jane     Done
2      Jade       Do

The goal here is to rename the columns. We can do that using the rename() function.

Here's what the syntax looks like:

df.rename(columns={"OLD_COLUMN_VALUE": "NEW_COLUMN_VALUE"})

Let's go ahead and change the column names (firstname and lastname) in the table from lowercase to uppercase (FIRSTNAME and LASTNAME).

import pandas as pd

students = {
    "firstname": ["John", "Jane", "Jade"], 
    "lastname": ["Doe", "Done", "Do"]
}

# convert student names into a Dataframe
df = pd.DataFrame(students)

df.rename(columns={"firstname": "FIRSTNAME", "lastname": "LASTNAME"}, inplace=True)

print(df)

# Output
  FIRSTNAME LASTNAME
0      John      Doe
1      Jane     Done
2      Jade       Do

In the code above, we specified that the columns firstname and lastname should be renamed to FIRSTNAME and LASTNAME, respectively: df.rename(columns={"firstname": "FIRSTNAME", "lastname": "LASTNAME"}, inplace=True)

You'll notice that we added the inplace=True parameter. This helps in persisting the new changes in the Dataframe. Delete the parameter and see what happens ;)

You can rename the columns to whatever you want. For instance, we can use SURNAME instead of lastname by doing this:

import pandas as pd

students = {
    "firstname": ["John", "Jane", "Jade"], 
    "lastname": ["Doe", "Done", "Do"]
}

# convert student names into a Dataframe
df = pd.DataFrame(students)
df.rename(columns={"firstname": "FIRSTNAME", "lastname": "SURNAME"}, inplace=True)

print(df)

# Output
  FIRSTNAME SURNAME
0      John     Doe
1      Jane    Done
2      Jade      Do

You can change just one column name, too. You are not required to change all the column names at the same time.

How to Rename a Column in Pandas Using a List

You can access the column names of a Dataframe using df.columns. Consider the table below:

  firstname lastname
0      John      Doe
1      Jane     Done
2      Jade       Do

We can print out the column names with the code below:

print(df.columns)

# Index(['firstname', 'lastname'], dtype='object')

Using that, we can rename the column of a Dataframe. Here's an example:

import pandas as pd

students = {
    "firstname": ["John", "Jane", "Jade"], 
    "lastname": ["Doe", "Done", "Do"]
}

# convert student names into a Dataframe
df = pd.DataFrame(students)
df.columns = ["FIRSTNAME", "SURNAME"]

print(df)

# Output
  FIRSTNAME SURNAME
0      John     Doe
1      Jane    Done
2      Jade      Do

In the example above, we put the new column names in a List and assigned it to the Dataframe columns: df.columns = ["FIRSTNAME", "SURNAME"].

This will override the previous column names.

How to Rename a Column in Pandas Using the set_axis() Function

The syntax for renaming a column with the set_axis() function looks like this:

df.set_axis([NEW_COLUMN_NAME,...], axis="columns")

Here's a code example:

import pandas as pd

students = {
    "firstname": ["John", "Jane", "Jade"], 
    "lastname": ["Doe", "Done", "Do"]
}

# convert student names into a Dataframe
df = pd.DataFrame(students)

df.set_axis(["FIRSTNAME", "SURNAME"], axis="columns", inplace=True) 

print(df)

# Output
  FIRSTNAME SURNAME
0      John     Doe
1      Jane    Done
2      Jade      Do

Note that the inplace=True parameter might raise a warning because it's deprecated for the set_axis() function and will be replaced in the future.

Summary

In this article, we talked about renaming a column in Pandas.

We saw different methods that can be used to rename a Pandas Dataframe column with code examples.

Happy coding!

Data Analysis is the process of exploring, investigating, and gathering insights from data using statistical measures and visualizations.

The objective of data analysis is to develop an understanding of data by uncovering trends, relationships, and patterns.

Data analysis is both a science and an art. On the one hand it requires that you know statistics, visualization techniques, and data analysis tools like Numpy, Pandas, and Seaborn.

On the other hand, it requires that you ask interesting questions to guide the investigation, and then interpret the numbers and figures to generate useful insights.

This tutorial on data analysis covers the following topics:

1. What is Numerical Computation? Python and Numpy for Beginners
2. How to Analyze Tabular Data using Python and Pandas
3. Data Visualization using Python, Matplotlib, and Seaborn

What is Numerical Computation? Python and Numpy for Beginners

_Source: Elegant Scipy_

You can follow along with the tutorial and run the code here: https://jovian.ai/aakashns/python-numerical-computing-with-numpy

This section covers the following topics:

• How to work with numerical data in Python
• How to turn Python lists into Numpy arrays
• Multi-dimensional Numpy arrays and their benefits
• Array operations, broadcasting, indexing, and slicing
• How to work with CSV data files using Numpy

How to Work with Numerical Data in Python

The "data" in Data Analysis typically refers to numerical data, like stock prices, sales figures, sensor measurements, sports scores, database tables, and so on.

The Numpy library provides specialized data structures, functions, and other tools for numerical computing in Python. Let's work through an example to see why and how to use Numpy to work with numerical data.

Suppose we want to use climate data like the temperature, rainfall, and humidity to determine if a region is well suited for growing apples.

A simple approach to do this would be to formulate the relationship between the annual yield of apples (tons per hectare) and the climatic conditions like the average temperature (in degrees Fahrenheit), rainfall (in millimeters), and average relative humidity (in percentage) as a linear equation.

yield_of_apples = w1 * temperature + w2 * rainfall + w3 * humidity

We're expressing the yield of apples as a weighted sum of the temperature, rainfall, and humidity.

This equation is an approximation, since the actual relationship may not necessarily be linear, and there may be other factors involved. But a simple linear model like this often works well in practice.

Based on some statistical analysis of historical data, we might come up with reasonable values for the weights w1, w2, and w3. Here's an example set of values:

w1, w2, w3 = 0.3, 0.2, 0.5

Given some climate data for a region, we can now predict the yield of apples. Here's some sample data:

To begin, we can define some variables to record climate data for a region.

kanto_temp = 73
kanto_rainfall = 67
kanto_humidity = 43

We can now substitute these variables into the linear equation to predict the yield of apples.

kanto_yield_apples = kanto_temp * w1 + kanto_rainfall * w2 + kanto_humidity * w3
kanto_yield_apples
# 56.8

print("The expected yield of apples in Kanto region is {} tons per hectare.".format(kanto_yield_apples))
# The expected yield of apples in Kanto region is 56.8 tons per hectare.

To make it slightly easier to perform the above computation for multiple regions, we can represent the climate data for each region as a vector, that is a list of numbers.

kanto = [73, 67, 43]
johto = [91, 88, 64]
hoenn = [87, 134, 58]
sinnoh = [102, 43, 37]
unova = [69, 96, 70]

The three numbers in each vector represent the temperature, rainfall, and humidity data, respectively.

We can also represent the set of weights used in the formula as a vector.

weights = [w1, w2, w3]

We can now write a function crop_yield to calculate the yield of apples (or any other crop) given the climate data and the respective weights.

def crop_yield(region, weights):
    result = 0
    for x, w in zip(region, weights):
        result += x * w
    return result

crop_yield(kanto, weights)
# 56.8

crop_yield(johto, weights)
# 76.9

crop_yield(unova, weights)
# 74.9

How to Turn Python Lists into Numpy Arrays

The calculation performed by the crop_yield (element-wise multiplication of two vectors and taking a sum of the results) is also called the dot product. Learn more about dot products here.

The Numpy library provides a built-in function to compute the dot product of two vectors. However, we must first convert the lists into Numpy arrays.

Let's install the Numpy library using the pip package manager.

!pip install numpy --upgrade --quiet

Next, let's import the numpy module. It's common practice to import numpy with the alias np.

import numpy as np

We can now use the np.array function to create Numpy arrays.

kanto = np.array([73, 67, 43])

kanto
# array([73, 67, 43])

weights = np.array([w1, w2, w3])

weights
# array([0.3, 0.2, 0.5])

Numpy arrays have the type ndarray.

type(kanto)
# numpy.ndarray

type(weights)
# numpy.ndarray

Just like lists, Numpy arrays support the indexing notation [].

weights[0]
# 0.3

kanto[2]
#43

How to Operate on Numpy arrays

We can now compute the dot product of the two vectors using the np.dot function.

np.dot(kanto, weights)
# 56.8

We can achieve the same result with low-level operations supported by Numpy arrays: performing an element-wise multiplication and calculating the resulting numbers' sum.

(kanto * weights).sum()
# 56.8

The * operator performs an element-wise multiplication of two arrays if they have the same size. The sum method calculates the sum of numbers in an array.

arr1 = np.array([1, 2, 3])
arr2 = np.array([4, 5, 6])

arr1 * arr2
# array([ 4, 10, 18])

arr2.sum()
# 15

What are the Benefits of Using Numpy Arrays?

Numpy arrays offer the following benefits over Python lists for operating on numerical data:

• They're easy to use: You can write small, concise, and intuitive mathematical expressions like (kanto * weights).sum() rather than using loops and custom functions like crop_yield.
• Performance: Numpy operations and functions are implemented internally in C++, which makes them much faster than using Python statements and loops that are interpreted at runtime

Here's a comparison of dot products performed using Python loops vs. Numpy arrays on two vectors with a million elements each.

# Python lists
arr1 = list(range(1000000))
arr2 = list(range(1000000, 2000000))

# Numpy arrays
arr1_np = np.array(arr1)
arr2_np = np.array(arr2)

%%time
result = 0
for x1, x2 in zip(arr1, arr2):
    result += x1*x2
result

# CPU times: user 300 ms, sys: 3.26 ms, total: 303 ms
# Wall time: 302 ms
# 833332333333500000

%%time
np.dot(arr1_np, arr2_np)

# CPU times: user 2.11 ms, sys: 951 µs, total: 3.07 ms
# Wall time: 1.58 ms
# 833332333333500000

As you can see, using np.dot is 100 times faster than using a for loop. This makes Numpy especially useful while working with really large datasets with tens of thousands or millions of data points.

Multi-Dimensional Numpy Arrays

We can now go one step further and represent the climate data for all the regions using a single 2-dimensional Numpy array.

climate_data = np.array([[73, 67, 43],
                         [91, 88, 64],
                         [87, 134, 58],
                         [102, 43, 37],
                         [69, 96, 70]])

climate_data
# array([[ 73,  67,  43],
#        [ 91,  88,  64],
#        [ 87, 134,  58],
#        [102,  43,  37],
#        [ 69,  96,  70]])

If you've taken a linear algebra class in high school, you may recognize the above 2-d array as a matrix with five rows and three columns. Each row represents one region, and the columns represent temperature, rainfall, and humidity, respectively.

Numpy arrays can have any number of dimensions and different lengths along each dimension. We can inspect the length along each dimension using the .shape property of an array.

_Source: Elegant Scipy_

# 2D array (matrix)
climate_data.shape
# (5, 3)

weights
# array([0.3, 0.2, 0.5])

# 1D array (vector)
weights.shape
# (3,)

# 3D array 
arr3 = np.array([
    [[11, 12, 13], 
     [13, 14, 15]], 
    [[15, 16, 17], 
     [17, 18, 19.5]]])

arr3.shape
# (2, 2, 3)

All the elements in a numpy array have the same data type. You can check the data type of an array using the .dtype property.

weights.dtype
# dtype('float64')

climate_data.dtype
# dtype('int64')

If an array contains even a single floating point number, all the other elements are also converted to floats.

arr3.dtype
# dtype('float64')

We can now compute the predicted yields of apples in all the regions, using a single matrix multiplication between climate_data (a 5x3 matrix) and weights (a vector of length 3). Here's what it looks like visually:

You can learn about matrices and matrix multiplication by watching the first 3-4 videos of this YouTube playlist.

We can use the np.matmul function or the @ operator to perform matrix multiplication.

np.matmul(climate_data, weights)
# array([56.8, 76.9, 81.9, 57.7, 74.9])

climate_data @ weights
# array([56.8, 76.9, 81.9, 57.7, 74.9])

How to Work with CSV Data Files

Numpy also provides helper functions reading from and writing to files. Let's download a file climate.txt, which contains 10,000 climate measurements (temperature, rainfall, and humidity) in the following format:

temperature,rainfall,humidity
25.00,76.00,99.00
39.00,65.00,70.00
59.00,45.00,77.00
84.00,63.00,38.00
66.00,50.00,52.00
41.00,94.00,77.00
91.00,57.00,96.00
49.00,96.00,99.00
67.00,20.00,28.00
...

This format of storing data is known as comma-separated values or CSV.

CSVs: A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. A CSV file typically stores tabular data (numbers and text) in plain text, in which case each line will have the same number of fields. (Wikipedia)

To read this file into a numpy array, we can use the genfromtxt function.

import urllib.request

urllib.request.urlretrieve(
    'https://hub.jovian.ml/wp-content/uploads/2020/08/climate.csv', 
    'climate.txt')

climate_data = np.genfromtxt('climate.txt', delimiter=',', skip_header=1)

climate_data
# array([[25., 76., 99.],
#        [39., 65., 70.],
#        [59., 45., 77.],
#        ...,
#        [99., 62., 58.],
#        [70., 71., 91.],
#        [92., 39., 76.]])

climate_data.shape
# (10000, 3)

We can now perform a matrix multiplication using the @ operator to predict the yield of apples for the entire dataset using a given set of weights.

weights = np.array([0.3, 0.2, 0.5])

yields = climate_data @ weights
yields
# array([72.2, 59.7, 65.2, ..., 71.1, 80.7, 73.4])

yields.shape
# (10000,)

Let's add the yields to climate_data as a fourth column using the np.concatenate function.

climate_results = np.concatenate((climate_data, yields.reshape(10000, 1)), axis=1)

climate_results
# array([[25. , 76. , 99. , 72.2],
#        [39. , 65. , 70. , 59.7],
#        [59. , 45. , 77. , 65.2],
#        ...,
#        [99. , 62. , 58. , 71.1],
#        [70. , 71. , 91. , 80.7],
#        [92. , 39. , 76. , 73.4]])

There are a couple of subtleties here:

• Since we wish to add new columns, we pass the argument axis=1 to np.concatenate. The axis argument specifies the dimension for concatenation.
• The arrays should have the same number of dimensions, and the same length along each except the dimension used for concatenation. We use the np.reshape function to change the shape of yields from (10000,) to (10000,1).

Here's a visual explanation of np.concatenate along axis=1 (can you guess what axis=0 results in?):

Source: w3resource.com

The best way to understand what a Numpy function does is to experiment with it and read the documentation to learn about its arguments and return values. Use the cells below to experiment with np.concatenate and np.reshape.

Let's write the final results from our computation above back to a file using the np.savetxt function.

np.savetxt('climate_results.txt', 
           climate_results, 
           fmt='%.2f', 
           delimiter=',',
           header='temperature,rainfall,humidity,yeild_apples', 
           comments='')

The results are written back in the CSV format to the file climate_results.txt.

temperature,rainfall,humidity,yeild_apples
25.00,76.00,99.00,72.20
39.00,65.00,70.00,59.70
59.00,45.00,77.00,65.20
84.00,63.00,38.00,56.80
...

Numpy provides hundreds of functions for performing operations on arrays. Here are some commonly used functions:

• Mathematics: np.sum, np.exp, np.round, arithmetic operators
• Array manipulation: np.reshape, np.stack, np.concatenate, np.split
• Linear Algebra: np.matmul, np.dot, np.transpose, np.eigvals
• Statistics: np.mean, np.median, np.std, np.max

So how do you find the function you need?** The easiest way to find the right function for a specific operation or use-case is to do a web search. For instance, searching for "How to join numpy arrays" leads to this tutorial on array concatenation.

You can find a full list of array functions here.

Numpy Arithmetic Operations, Broadcasting, and Comparison

Numpy arrays support arithmetic operators like +, -, *, etc. You can perform an arithmetic operation with a single number (also called a scalar) or with another array of the same shape.

Operators make it easy to write mathematical expressions with multi-dimensional arrays.

arr2 = np.array([[1, 2, 3, 4], 
                 [5, 6, 7, 8], 
                 [9, 1, 2, 3]])

arr3 = np.array([[11, 12, 13, 14], 
                 [15, 16, 17, 18], 
                 [19, 11, 12, 13]])

# Adding a scalar
arr2 + 3

# array([[ 4,  5,  6,  7],
#        [ 8,  9, 10, 11],
#        [12,  4,  5,  6]])

# Element-wise subtraction
arr3 - arr2

# array([[10, 10, 10, 10],
#        [10, 10, 10, 10],
#        [10, 10, 10, 10]])

# Division by scalar
arr2 / 2

# array([[0.5, 1. , 1.5, 2. ],
#        [2.5, 3. , 3.5, 4. ],
#        [4.5, 0.5, 1. , 1.5]])

# Element-wise multiplication
arr2 * arr3

# array([[ 11,  24,  39,  56],
#        [ 75,  96, 119, 144],
#        [171,  11,  24,  39]])

# Modulus with scalar
arr2 % 4

# array([[1, 2, 3, 0],
#        [1, 2, 3, 0],
#        [1, 1, 2, 3]])

Numpy Array Broadcasting

Numpy arrays also support broadcasting, allowing arithmetic operations between two arrays with different numbers of dimensions but compatible shapes. Let's look at an example to see how it works.

arr2 = np.array([[1, 2, 3, 4], 
                 [5, 6, 7, 8], 
                 [9, 1, 2, 3]])               
arr2.shape
# (3, 4)

arr4 = np.array([4, 5, 6, 7])
arr4.shape
# (4,)

arr2 + arr4
# array([[ 5,  7,  9, 11],
#        [ 9, 11, 13, 15],
#        [13,  6,  8, 10]])

When the expression arr2 + arr4 is evaluated, arr4 (which has the shape (4,)) is replicated three times to match the shape (3, 4) of arr2. Numpy performs the replication without actually creating three copies of the smaller dimension array, thus improving performance and using lower memory.

Source: Python Data Science Handbook

Broadcasting only works if one of the arrays can be replicated to match the other array's shape.

arr5 = np.array([7, 8])
arr5.shape
# (2,)

arr2 + arr5
# ValueError: operands could not be broadcast together with shapes (3,4) (2,)

In the above example, even if arr5 is replicated three times, it will not match the shape of arr2. So arr2 + arr5 cannot be evaluated successfully. Learn more about broadcasting here.

Numpy Array Comparison

Numpy arrays also support comparison operations like ==, !=, > and so on. The result is an array of booleans.

arr1 = np.array([[1, 2, 3], [3, 4, 5]])
arr2 = np.array([[2, 2, 3], [1, 2, 5]])

arr1 == arr2
# array([[False,  True,  True],
#        [False, False,  True]])

arr1 != arr2
# array([[ True, False, False],
#        [ True,  True, False]])

arr1 >= arr2
# array([[False,  True,  True],
#        [ True,  True,  True]])

arr1 < arr2
# array([[ True, False, False],
#        [False, False, False]])

Array comparison is frequently used to count the number of equal elements in two arrays using the sum method. Remember that True evaluates to 1 and False evaluates to 0 when you use booleans in arithmetic operations.

(arr1 == arr2).sum()
# 3

Numpy Array Indexing and Slicing

Numpy extends Python's list indexing notation using [] to multiple dimensions in an intuitive fashion. You can provide a comma-separated list of indices or ranges to select a specific element or a subarray (also called a slice) from a Numpy array.

arr3 = np.array([
    [[11, 12, 13, 14], 
     [13, 14, 15, 19]], 

    [[15, 16, 17, 21], 
     [63, 92, 36, 18]], 

    [[98, 32, 81, 23],      
     [17, 18, 19.5, 43]]])

arr3.shape
# (3, 2, 4)

# Single element
arr3[1, 1, 2]

# 36.0

# Subarray using ranges
arr3[1:, 0:1, :2]

# array([[[15., 16.]],
# 
#        [[98., 32.]]])

# Mixing indices and ranges
arr3[1:, 1, 3]

# array([18., 43.])

arr3[1:, 1, :3]
# array([[63. , 92. , 36. ],
#        [17. , 18. , 19.5]])

# Using fewer indices
arr3[1]

# array([[15., 16., 17., 21.],
#        [63., 92., 36., 18.]])

arr3[:2, 1]
# array([[13., 14., 15., 19.],
#        [63., 92., 36., 18.]])

# Using too many indices
arr3[1,3,2,1]

# IndexError: too many indices for array: array is 3-dimensional, but 4 were indexed

The notation and its results can seem confusing at first, so take your time to experiment and become comfortable with it.

Use the cells below to try out some examples of array indexing and slicing, with different combinations of indices and ranges. Here are some more examples demonstrated visually:

_Source: Scipy Lectures_

How to Create Numpy Arrays – Other Methods

Numpy also provides some handy functions to create arrays of desired shapes with fixed or random values. Check out the official documentation or use the help function to learn more.

# All zeros
np.zeros((3, 2))

# array([[0., 0.],
#        [0., 0.],
#        [0., 0.]])

# All ones
np.ones([2, 2, 3])

# array([[[1., 1., 1.],
#         [1., 1., 1.]],
#
#        [[1., 1., 1.],
#         [1., 1., 1.]]])

# Identity matrix
np.eye(3)

# array([[1., 0., 0.],
#        [0., 1., 0.],
#        [0., 0., 1.]])

# Random vector
np.random.rand(5)

# array([0.92929562, 0.11301864, 0.64213555, 0.8600434 , 0.53738656])

# Random matrix
np.random.randn(2, 3) # rand vs. randn - what's the difference?

# array([[ 0.09906435, -1.64668094,  0.08073528],
#        [ 0.1437016 ,  0.80715712,  1.27285476]])

# Fixed value
np.full([2, 3], 42)

# array([[42, 42, 42],
#        [42, 42, 42]])

# Range with start, end and step
np.arange(10, 90, 3)

# array([10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46, 49, 52, 55, 58,
#        61, 64, 67, 70, 73, 76, 79, 82, 85, 88])

# Equally spaced numbers in a range
np.linspace(3, 27, 9)

# array([ 3.,  6.,  9., 12., 15., 18., 21., 24., 27.])

Exercises

Try the following exercises to become familiar with Numpy arrays and practice your skills:

• Assignment on Numpy array functions: https://jovian.ml/aakashns/numpy-array-operations
• (Optional) 100 numpy exercises: https://jovian.ml/aakashns/100-numpy-exercises

Summary and Further Reading

With this, we complete our discussion of numerical computing with Numpy. We've covered the following topics in this part of the tutorial:

• How to go from Python lists to Numpy arrays
• How to operate on Numpy arrays
• The benefits of using Numpy arrays over lists
• Multi-dimensional Numpy arrays
• How to work with CSV data files
• Arithmetic operations and broadcasting
• Array indexing and slicing
• Other ways of creating Numpy arrays

Check out the following resources for learning more about Numpy:

• Official tutorial
• Numpy course on freeCodeCamp
• Advanced Numpy (exploring the internals)

Review Questions to Check Your Comprehension

Try answering the following questions to test your understanding of the topics covered in this notebook:

1. What is a vector?
2. How do you represent vectors using a Python list? Give an example.
3. What is a dot product of two vectors?
4. Write a function to compute the dot product of two vectors.
5. What is Numpy?
6. How do you install Numpy?
7. How do you import the numpy module?
8. What does it mean to import a module with an alias? Give an example.
9. What is the commonly used alias for numpy?
10. What is a Numpy array?
11. How do you create a Numpy array? Give an example.
12. What is the type of Numpy arrays?
13. How do you access the elements of a Numpy array?
14. How do you compute the dot product of two vectors using Numpy?
15. What happens if you try to compute the dot product of two vectors which have different sizes?
16. How do you compute the element-wise product of two Numpy arrays?
17. How do you compute the sum of all the elements in a Numpy array?
18. What are the benefits of using Numpy arrays over Python lists for operating on numerical data?
19. Why do Numpy array operations have better performance compared to Python functions and loops?
20. Illustrate the performance difference between Numpy array operations and Python loops using an example.
21. What are multi-dimensional Numpy arrays?
22. Illustrate how you'd create Numpy arrays with 2, 3, and 4 dimensions.
23. How do you inspect the number of dimensions and the length along each dimension in a Numpy array?
24. Can the elements of a Numpy array have different data types?
25. How do you check the data types of the elements of a Numpy array?
26. What is the data type of a Numpy array?
27. What is the difference between a matrix and a 2D Numpy array?
28. How do you perform matrix multiplication using Numpy?
29. What is the @ operator used for in Numpy?
30. What is the CSV file format?
31. How do you read data from a CSV file using Numpy?
32. How do you concatenate two Numpy arrays?
33. What is the purpose of the axis argument of np.concatenate?
34. When are two Numpy arrays compatible for concatenation?
35. Give an example of two Numpy arrays that can be concatenated.
36. Give an example of two Numpy arrays that cannot be concatenated.
37. What is the purpose of the np.reshape function?
38. What does it mean to “reshape” a Numpy array?
39. How do you write a numpy array into a CSV file?
40. Give some examples of Numpy functions for performing mathematical operations.
41. Give some examples of Numpy functions for performing array manipulation.
42. Give some examples of Numpy functions for performing linear algebra.
43. Give some examples of Numpy functions for performing statistical operations.
44. How do you find the right Numpy function for a specific operation or use case?
45. Where can you see a list of all the Numpy array functions and operations?
46. What are the arithmetic operators supported by Numpy arrays? Illustrate with examples.
47. What is array broadcasting? How is it useful? Illustrate with an example.
48. Give some examples of arrays that are compatible for broadcasting.
49. Give some examples of arrays that are not compatible for broadcasting.
50. What are the comparison operators supported by Numpy arrays? Illustrate with examples.
51. How do you access a specific subarray or slice from a Numpy array?
52. Illustrate array indexing and slicing in multi-dimensional Numpy arrays with some examples.
53. How do you create a Numpy array with a given shape containing all zeros?
54. How do you create a Numpy array with a given shape containing all ones?
55. How do you create an identity matrix of a given shape?
56. How do you create a random vector of a given length?
57. How do you create a Numpy array with a given shape with a fixed value for each element?
58. How do you create a Numpy array with a given shape containing randomly initialized elements?
59. What is the difference between np.random.rand and np.random.randn? Illustrate with examples.
60. What is the difference between np.arange and np.linspace? Illustrate with examples.

You are ready to move on to the next section of this tutorial.

How to Analyze Tabular Data using Python and Pandas

Follow along and run the code here: https://jovian.ai/aakashns/python-pandas-data-analysis.

This section covers the following topics:

• How to read a CSV file into a Pandas data frame
• How to retrieve data from Pandas data frames
• How to query, sort, and analyze data
• How to merge, group, and aggregate data
• How to extract useful information from dates
• Basic plotting using line and bar charts
• How to write data frames to CSV files

How to Read a CSV File Using Pandas

Pandas is a popular Python library used for working in tabular data (similar to the data stored in a spreadsheet). It provides helper functions to read data from various file formats like CSV, Excel spreadsheets, HTML tables, JSON, SQL, and more.

Let's download a file italy-covid-daywise.txt which contains day-wise Covid-19 data for Italy in the following format:

date,new_cases,new_deaths,new_tests
2020-04-21,2256.0,454.0,28095.0
2020-04-22,2729.0,534.0,44248.0
2020-04-23,3370.0,437.0,37083.0
2020-04-24,2646.0,464.0,95273.0
2020-04-25,3021.0,420.0,38676.0
2020-04-26,2357.0,415.0,24113.0
2020-04-27,2324.0,260.0,26678.0
2020-04-28,1739.0,333.0,37554.0
...

This format of storing data is known as comma-separated values or CSV. Here's a reminder in case you need a definition of what the CSV format is:

CSVs: A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. A CSV file typically stores tabular data (numbers and text) in plain text, in which case each line will have the same number of fields. (Wikipedia)

We'll download this file using the urlretrieve function from the urllib.request module.

from urllib.request import urlretrieve

urlretrieve('https://hub.jovian.ml/wp-content/uploads/2020/09/italy-covid-daywise.csv', 'italy-covid-daywise.csv')

To read the file, we can use the read_csv method from Pandas. First, let's install the Pandas library.

!pip install pandas --upgrade --quiet

We can now import the pandas module. As a convention, it is imported with the alias pd.

import pandas as pd

covid_df = pd.read_csv('italy-covid-daywise.csv')

Data from the file is read and stored in a DataFrame object – one of the core data structures in Pandas for storing and working with tabular data. We typically use the _df suffix in the variable names for dataframes.

type(covid_df)
# pandas.core.frame.DataFrame

covid_df

Here's what we can tell by looking at the dataframe:

• The file provides four day-wise counts for COVID-19 in Italy
• The metrics reported are new cases, deaths, and tests
• Data is provided for 248 days: from Dec 12, 2019, to Sep 3, 2020

Keep in mind that these are officially reported numbers. The actual number of cases and deaths may be higher, as not all cases are diagnosed.

We can view some basic information about the data frame using the .info method.

covid_df.info()

It appears that each column contains values of a specific data type. You can view statistical information for numerical columns (mean, standard deviation, minimum/maximum values, and the number of non-empty values) using the .describe method.

covid_df.describe()

The columns property contains the list of columns within the data frame.

covid_df.columns
# Index(['date', 'new_cases', 'new_deaths', 'new_tests'], dtype='object')

You can also retrieve the number of rows and columns in the data frame using the .shape method.

covid_df.shape
# (248, 4)

Here's a summary of the functions and methods we've looked at so far:

• pd.read_csv – Read data from a CSV file into a Pandas DataFrame object
• .info() – View basic information about rows, columns, and data types
• .describe() – View statistical information about numeric columns
• .columns – Get the list of column names
• .shape – Get the number of rows and columns as a tuple

How to Retrieve Data from a Data Frame in Pandas

The first thing you might want to do is retrieve data from this data frame, like the counts of a specific day or the list of values in a particular column.

To do this, you should understand the internal representation of data in a data frame. Conceptually, you can think of a dataframe as a dictionary of lists: keys are column names, and values are lists/arrays containing data for the respective columns.

# Pandas format is simliar to this
covid_data_dict = {
    'date':       ['2020-08-30', '2020-08-31', '2020-09-01', '2020-09-02', '2020-09-03'],
    'new_cases':  [1444, 1365, 996, 975, 1326],
    'new_deaths': [1, 4, 6, 8, 6],
    'new_tests': [53541, 42583, 54395, None, None]
}

Representing data in the above format has a few benefits:

• All values in a column typically have the same type of value, so it's more efficient to store them in a single array.
• Retrieving the values for a particular row simply requires extracting the elements at a given index from each column array.
• The representation is more compact (column names are recorded only once) compared to other formats that use a dictionary for each row of data (see the example below).

# Pandas format is not similar to this
covid_data_list = [
    {'date': '2020-08-30', 'new_cases': 1444, 'new_deaths': 1, 'new_tests': 53541},
    {'date': '2020-08-31', 'new_cases': 1365, 'new_deaths': 4, 'new_tests': 42583},
    {'date': '2020-09-01', 'new_cases': 996, 'new_deaths': 6, 'new_tests': 54395},
    {'date': '2020-09-02', 'new_cases': 975, 'new_deaths': 8 },
    {'date': '2020-09-03', 'new_cases': 1326, 'new_deaths': 6},
]

With the dictionary of lists analogy in mind, you can now guess how to retrieve data from a data frame. For example, we can get a list of values from a specific column using the [] indexing notation.

covid_data_dict['new_cases']
# [1444, 1365, 996, 975, 1326]

covid_df['new_cases']
# 0         0.0
# 1         0.0
# 2         0.0
# 3         0.0
# 4         0.0
#         ...  
# 243    1444.0
# 244    1365.0
# 245     996.0
# 246     975.0
# 247    1326.0
# Name: new_cases, Length: 248, dtype: float64

Each column is represented using a data structure called Series, which is essentially a numpy array with some extra methods and properties.

type(covid_df['new_cases'])
# pandas.core.series.Series

Like arrays, you can retrieve a specific value with a series using the indexing notation [].

covid_df['new_cases'][246]
# 975.0

covid_df['new_tests'][240]
57640.0

Pandas also provides the .at method to retrieve the element at a specific row & column directly.

covid_df.at[246, 'new_cases']
# 975.0

covid_df.at[240, 'new_tests']
# 57640.0

Instead of using the indexing notation [], Pandas also allows accessing columns as properties of the dataframe using the . notation. However, this method only works for columns whose names do not contain spaces or special characters.

covid_df.new_cases
# 0         0.0
# 1         0.0
# 2         0.0
# 3         0.0
# 4         0.0
#         ...  
# 243    1444.0
# 244    1365.0
# 245     996.0
# 246     975.0
# 247    1326.0
# Name: new_cases, Length: 248, dtype: float64

Further, you can also pass a list of columns within the indexing notation [] to access a subset of the data frame with just the given columns.

cases_df = covid_df[['date', 'new_cases']]
cases_df

The new data frame cases_df is simply a "view" of the original data frame covid_df. Both point to the same data in the computer's memory. Changing any values inside one of them will also change the respective values in the other.

Sharing data between data frames makes data manipulation in Pandas blazing fast. You needn't worry about the overhead of copying thousands or millions of rows every time you want to create a new data frame by operating on an existing one.

Sometimes you might need a full copy of the data frame, in which case you can use the copy method.

covid_df_copy = covid_df.copy()

The data within covid_df_copy is completely separate from covid_df, and changing values inside one of them will not affect the other.

To access a specific row of data, Pandas provides the .loc method.

covid_df

covid_df.loc[243]
# date          2020-08-30
# new_cases         1444.0
# new_deaths           1.0
# new_tests        53541.0
# Name: 243, dtype: object

Each retrieved row is also a Series object.

type(covid_df.loc[243])
# pandas.core.series.Series

We can use the .head and .tail methods to view the first or last few rows of data.

covid_df.head(5)

covid_df.tail(4)

Notice above that while the first few values in the new_cases and new_deaths columns are 0, the corresponding values within the new_tests column are NaN. That is because the CSV file does not contain any data for the new_tests column for specific dates (you can verify this by looking into the file). These values may be missing or unknown.

covid_df.at[0, 'new_tests']
# nan

type(covid_df.at[0, 'new_tests'])
# numpy.float64

The distinction between 0 and NaN is subtle but important. In this dataset, it represents that daily test numbers were not reported on specific dates. Italy started reporting daily tests on Apr 19, 2020. They'd already conducted 935,310 tests before Apr 19.

We can find the first index that doesn't contain a NaN value using a column's first_valid_index method.

covid_df.new_tests.first_valid_index()
# 111

Let's look at a few rows before and after this index to verify that the values change from NaN to actual numbers. We can do this by passing a range to loc.

covid_df.loc[108:113]

We can use the .sample method to retrieve a random sample of rows from the data frame.

covid_df.sample(10)

Notice that even though we have taken a random sample, each row's original index is preserved. This is a useful property of data frames.

Here's a summary of the functions and methods we looked at in this section:

• covid_df['new_cases'] – Retrieving columns as a Series using the column name
• new_cases[243] – Retrieving values from a Series using an index
• covid_df.at[243, 'new_cases'] – Retrieving a single value from a data frame
• covid_df.copy() – Creating a deep copy of a data frame
• covid_df.loc[243] - Retrieving a row or range of rows of data from the data frame
• head, tail, and sample – Retrieving multiple rows of data from the data frame
• covid_df.new_tests.first_valid_index – Finding the first non-empty index in a series

How to Analyze Data from Data Frames in Pandas

Let's try to answer some questions about our data.

Q: What are the total number of reported cases and deaths related to Covid-19 in Italy?

Similar to Numpy arrays, a Pandas series supports the sum method to answer these questions.

total_cases = covid_df.new_cases.sum()
total_deaths = covid_df.new_deaths.sum()

print('The number of reported cases is {} and the number of reported deaths is {}.'.format(int(total_cases), int(total_deaths)))
# The number of reported cases is 271515 and the number of reported deaths is 35497.

Q: What is the overall death rate (ratio of reported deaths to reported cases)?

death_rate = covid_df.new_deaths.sum() / covid_df.new_cases.sum()

print("The overall reported death rate in Italy is {:.2f} %.".format(death_rate*100))
# The overall reported death rate in Italy is 13.07 %.

Q: What is the overall number of tests conducted? A total of 935,310 tests were conducted before daily test numbers were reported.

initial_tests = 935310
total_tests = initial_tests + covid_df.new_tests.sum()

total_tests
# 5214766.0

Q: What fraction of tests returned a positive result?

positive_rate = total_cases / total_tests

print('{:.2f}% of tests in Italy led to a positive diagnosis.'.format(positive_rate*100))
# 5.21% of tests in Italy led to a positive diagnosis.

Try asking and answering some more questions about the data.

How to Query and Sort Rows in Pandas

Let's say we only want to look at the days which had more than 1,000 reported cases. We can use a boolean expression to check which rows satisfy this criterion.

high_new_cases = covid_df.new_cases > 1000

high_new_cases
# 0      False
# 1      False
# 2      False
# 3      False
# 4      False
#        ...  
# 243     True
# 244     True
# 245    False
# 246    False
# 247     True
# Name: new_cases, Length: 248, dtype: bool

The boolean expression returns a series containing True and False boolean values. You can use this series to select a subset of rows from the original dataframe, corresponding to the True values in the series.

covid_df[high_new_cases]

The data frame contains 72 rows, but only the first and last five rows are displayed by default with Jupyter for brevity. We can change some display options to view all the rows.

high_cases_df = covid_df[covid_df.new_cases > 1000]

high_cases_df

The data frame contains 72 rows, but only the first & last five rows are displayed by default with Jupyter for brevity. We can change some display options to view all the rows.

from IPython.display import display
with pd.option_context('display.max_rows', 100):
    display(covid_df[covid_df.new_cases > 1000])

This is just part of the data frame. Check out the rest here.

We can also formulate more complex queries that involve multiple columns. As an example, let's try to determine the days when the ratio of cases reported to tests conducted is higher than the overall positive_rate.

positive_rate
# 0.05206657403227681

high_ratio_df = covid_df[covid_df.new_cases / covid_df.new_tests > positive_rate]

high_ratio_df

The result of performing an operation on two columns is a new series.

covid_df.new_cases / covid_df.new_tests
# 0           NaN
# 1           NaN
# 2           NaN
# 3           NaN
# 4           NaN
#          ...   
# 243    0.026970
# 244    0.032055
# 245    0.018311
# 246         NaN
# 247         NaN
# Length: 248, dtype: float64

We can use this series to add a new column to the data frame.

covid_df['positive_rate'] = covid_df.new_cases / covid_df.new_tests

covid_df

However, keep in mind that sometimes it takes a few days to get the results for a test, so we can't compare the number of new cases with the number of tests conducted on the same day. Any inference based on this positive_rate column is likely to be incorrect.

It's essential to watch out for such subtle relationships that are often not conveyed within the CSV file and require some external context. It's always a good idea to read through the documentation provided with the dataset or ask for more information.

For now, let's remove the positive_rate column using the drop method.

covid_df.drop(columns=['positive_rate'], inplace=True)

Can you figure the purpose of the inplace argument?

How to Sort Rows Using Column Values in Pandas

You can also sort the rows by a specific column using .sort_values. Let's sort to identify the days with the highest number of cases, then chain it with the head method to list just the first ten results.

covid_df.sort_values('new_cases', ascending=False).head(10)

It looks like the last two weeks of March had the highest number of daily cases. Let's compare this to the days where the highest number of deaths were recorded.

covid_df.sort_values('new_deaths', ascending=False).head(10)

It appears that daily deaths hit a peak just about a week after the peak in daily new cases.

Let's also look at the days with the smallest number of cases. We might expect to see the first few days of the year on this list.

covid_df.sort_values('new_cases').head(10)

It seems like the count of new cases on Jun 20, 2020, was -148, a negative number! Not something we might have expected, but that's the nature of real-world data. It could be a data entry error, or the government may have issued a correction to account for miscounting in the past.

Can you dig through news articles online and figure out why the number was negative?

Let's look at some days before and after Jun 20, 2020.

covid_df.loc[169:175]

For now, let's assume this was indeed a data entry error. We can use one of the following approaches for dealing with the missing or faulty value:

1. Replace it with 0.
2. Replace it with the average of the entire column
3. Replace it with the average of the values on the previous and next date
4. Discard the row entirely

Which approach you pick requires some context about the data and the problem. In this case, since we are dealing with data ordered by date, we can go ahead with the third approach.

You can use the .at method to modify a specific value within the dataframe.

covid_df.at[172, 'new_cases'] = (covid_df.at[171, 'new_cases'] + covid_df.at[173, 'new_cases'])/2

Here's a summary of the functions and methods we looked at in this section:

• covid_df.new_cases.sum() – Computing the sum of values in a column or series
• covid_df[covid_df.new_cases > 1000] – Querying a subset of rows satisfying the chosen criteria using boolean expressions
• df['pos_rate'] = df.new_cases/df.new_tests – Adding new columns by combining data from existing columns
• covid_df.drop('positive_rate') – Removing one or more columns from the data frame
• sort_values – Sorting the rows of a data frame using column values
• covid_df.at[172, 'new_cases'] = ... – Replacing a value within the data frame

How to Work with Dates in Pandas

While we've looked at overall numbers for the cases, tests, positive rate, and more, it would also be useful to study these numbers on a month-by-month basis.

The date column might come in handy here, as Pandas provides many utilities for working with dates.

covid_df.date
# 0      2019-12-31
# 1      2020-01-01
# 2      2020-01-02
# 3      2020-01-03
# 4      2020-01-04
#           ...    
# 243    2020-08-30
# 244    2020-08-31
# 245    2020-09-01
# 246    2020-09-02
# 247    2020-09-03
# Name: date, Length: 248, dtype: object

The data type of date is currently object, so Pandas does not know that this column is a date. We can convert it into a datetime column using the pd.to_datetime method.

covid_df['date'] = pd.to_datetime(covid_df.date)

covid_df['date']
# 0     2019-12-31
# 1     2020-01-01
# 2     2020-01-02
# 3     2020-01-03
# 4     2020-01-04
#          ...    
# 243   2020-08-30
# 244   2020-08-31
# 245   2020-09-01
# 246   2020-09-02
# 247   2020-09-03
# Name: date, Length: 248, dtype: datetime64[ns]

You can see that it now has the datatype datetime64. We can now extract different parts of the data into separate columns, using the DatetimeIndex class (view docs).

covid_df['year'] = pd.DatetimeIndex(covid_df.date).year
covid_df['month'] = pd.DatetimeIndex(covid_df.date).month
covid_df['day'] = pd.DatetimeIndex(covid_df.date).day
covid_df['weekday'] = pd.DatetimeIndex(covid_df.date).weekday

covid_df

Let's check the overall metrics for May. We can query the rows for May, choose a subset of columns, and use the sum method to aggregate each selected column's values.

# Query the rows for May
covid_df_may = covid_df[covid_df.month == 5]

# Extract the subset of columns to be aggregated
covid_df_may_metrics = covid_df_may[['new_cases', 'new_deaths', 'new_tests']]

# Get the column-wise sum
covid_may_totals = covid_df_may_metrics.sum()

covid_may_totals
# new_cases       29073.0
# new_deaths       5658.0
# new_tests     1078720.0
# dtype: float64

type(covid_may_totals)
# pandas.core.series.Series

We can also combine the above operations into a single statement.

covid_df[covid_df.month == 5][['new_cases', 'new_deaths', 'new_tests']].sum()
# new_cases       29073.0
# new_deaths       5658.0
# new_tests     1078720.0
# dtype: float64

As another example, let's check if the number of cases reported on Sundays is higher than the average number of cases reported every day. This time, we might want to aggregate columns using the .mean method.

# Overall average
covid_df.new_cases.mean()

# 1096.6149193548388

# Average for Sundays
covid_df[covid_df.weekday == 6].new_cases.mean()

# 1247.2571428571428

It seems like more cases were reported on Sundays compared to other days.

Try asking and answering some more date-related questions about the data.

How to Group and Aggregate Data in Pandas

As a next step, we might want to summarize the day-wise data and create a new dataframe with month-wise data. We can use the groupby function to create a group for each month, select the columns we wish to aggregate, and aggregate them using the sum method.

covid_month_df = covid_df.groupby('month')[['new_cases', 'new_deaths', 'new_tests']].sum()

covid_month_df

The result is a new data frame that uses unique values from the column passed to groupby as the index. Grouping and aggregation is a powerful method for progressively summarizing data into smaller data frames.

Instead of aggregating by sum, you can also aggregate by other measures like mean. Let's compute the average number of daily new cases, deaths, and tests for each month.

covid_month_mean_df = covid_df.groupby('month')[['new_cases', 'new_deaths', 'new_tests']].mean()

covid_month_mean_df

Apart from grouping, another form of aggregation is the running or cumulative sum of cases, tests, or deaths up to each row's date. We can use the cumsum method to compute the cumulative sum of a column as a new series.

Let's add three new columns: total_cases, total_deaths, and total_tests.

covid_df['total_cases'] = covid_df.new_cases.cumsum()
covid_df['total_deaths'] = covid_df.new_deaths.cumsum()
covid_df['total_tests'] = covid_df.new_tests.cumsum() + initial_tests

We've also included the initial test count in total_test to account for tests conducted before daily reporting was started.

covid_df

Notice how the NaN values in the total_tests column remain unaffected.

How to Merge Data from Multiple Sources in Pandas

To determine other metrics like test per million, cases per million, and so on, we require some more information about the country, namely its population.

Let's download another file locations.csv that contains health-related information for many countries, including Italy.

urlretrieve('https://gist.githubusercontent.com/aakashns/8684589ef4f266116cdce023377fc9c8/raw/99ce3826b2a9d1e6d0bde7e9e559fc8b6e9ac88b/locations.csv', 'locations.csv')

locations_df = pd.read_csv('locations.csv')
locations_df

locations_df[locations_df.location == "Italy"]

We can merge this data into our existing data frame by adding more columns. However, to merge two data frames, we need at least one common column. Let's insert a location column in the covid_df dataframe with all values set to "Italy".

covid_df['location'] = "Italy"

covid_df

We can now add the columns from locations_df into covid_df using the .merge method.

merged_df = covid_df.merge(locations_df, on="location")

merged_df

Check out the full data frame here.

The location data for Italy is appended to each row within covid_df. If the covid_df data frame contained data for multiple locations, then the respective country's location data would be appended for each row.

We can now calculate metrics like cases per million, deaths per million, and tests per million.

merged_df['cases_per_million'] = merged_df.total_cases * 1e6 / merged_df.population
merged_df['deaths_per_million'] = merged_df.total_deaths * 1e6 / merged_df.population
merged_df['tests_per_million'] = merged_df.total_tests * 1e6 / merged_df.population

merged_df

Check out the full data frame here.

How to Write Data Back to Files in Pandas

After completing your analysis and adding new columns, you should write the results back to a file. Otherwise, the data will be lost when the Jupyter notebook shuts down.

Before writing to file, let's first create a data frame containing just the columns we wish to record.

result_df = merged_df[['date',
                       'new_cases', 
                       'total_cases', 
                       'new_deaths', 
                       'total_deaths', 
                       'new_tests', 
                       'total_tests', 
                       'cases_per_million', 
                       'deaths_per_million', 
                       'tests_per_million']]

result_df

To write the data from the data frame into a file, we can use the to_csv function.

result_df.to_csv('results.csv', index=None)

The to_csv function also includes an additional column for storing the index of the dataframe by default. We pass index=None to turn off this behavior. You can now verify that the results.csv is created and contains data from the data frame in CSV format:

date,new_cases,total_cases,new_deaths,total_deaths,new_tests,total_tests,cases_per_million,deaths_per_million,tests_per_million
2020-02-27,78.0,400.0,1.0,12.0,,,6.61574439992122,0.1984723319976366,
2020-02-28,250.0,650.0,5.0,17.0,,,10.750584649871982,0.28116913699665186,
2020-02-29,238.0,888.0,4.0,21.0,,,14.686952567825108,0.34732658099586405,
2020-03-01,240.0,1128.0,8.0,29.0,,,18.656399207777838,0.47964146899428844,
2020-03-02,561.0,1689.0,6.0,35.0,,,27.93498072866735,0.5788776349931067,
2020-03-03,347.0,2036.0,17.0,52.0,,,33.67413899559901,0.8600467719897585,
...

Bonus: Basic Plotting with Pandas

We generally use a library like matplotlib or seaborn to plot graphs within a Jupyter notebook. However, Pandas dataframes and series provide a handy .plot method for quick and easy plotting.

Let's plot a line graph showing how the number of daily cases varies over time.

result_df.new_cases.plot();

While this plot shows the overall trend, it's hard to tell where the peak occurred, as there are no dates on the X-axis. We can use the date column as the index for the data frame to address this issue.

result_df.set_index('date', inplace=True)

result_df

Notice that the index of a data frame doesn't have to be numeric. Using the date as the index also allows us to get the data for a specific data using .loc.

result_df.loc['2020-09-01']
# new_cases             9.960000e+02
# total_cases           2.696595e+05
# new_deaths            6.000000e+00
# total_deaths          3.548300e+04
# new_tests             5.439500e+04
# total_tests           5.214766e+06
# cases_per_million     4.459996e+03
# deaths_per_million    5.868661e+02
# tests_per_million     8.624890e+04
# Name: 2020-09-01 00:00:00, dtype: float64

Let's plot the new cases and new deaths per day as line graphs.

result_df.new_cases.plot()
result_df.new_deaths.plot();

We can also compare the total cases vs. total deaths.

result_df.total_cases.plot()
result_df.total_deaths.plot();

Let's see how the death rate and positive testing rates vary over time.

death_rate = result_df.total_deaths / result_df.total_cases

death_rate.plot(title='Death Rate');

positive_rates = result_df.total_cases / result_df.total_tests

positive_rates.plot(title='Positive Rate');

Finally, let's plot some month-wise data using a bar chart to visualize the trend at a higher level.

covid_month_df.new_cases.plot(kind='bar');

covid_month_df.new_tests.plot(kind='bar')

Pandas Exercises

Try the following exercises to become familiar with Pandas dataframes and practice your skills:

• Assignment on Pandas dataframes
• Additional exercises on Pandas
• Try downloading and analyzing some data from Kaggle

Summary and Further Reading

We've covered the following topics in this tutorial:

• How to read a CSV file into a Pandas data frame
• How to retrieve data from Pandas data frames
• How to query, sort, and analyze data
• How to merge, group, and aggregate data
• How to extract useful information from dates
• Basic plotting using line and bar charts
• How to write data frames to CSV files

Check out the following resources to learn more about Pandas:

• User guide for Pandas
• Python for Data Analysis (book by Wes McKinney - creator of Pandas)

Review Questions to Check Your Comprehension

Try answering the following questions to test your understanding of the topics covered in this notebook:

1. What is Pandas? What makes it useful?
2. How do you install the Pandas library?
3. How do you import the pandas module?
4. What is the common alias used while importing the pandas module?
5. How do you read a CSV file using Pandas? Give an example.
6. What are some other file formats you can read using Pandas? Illustrate with examples.
7. What are Pandas dataframes?
8. How are Pandas dataframes different from Numpy arrays?
9. How do you find the number of rows and columns in a dataframe?
10. How do you get the list of columns in a dataframe?
11. What is the purpose of the describe method of a dataframe?
12. How are the info and describe dataframe methods different?
13. Is a Pandas dataframe conceptually similar to a list of dictionaries or a dictionary of lists? Explain with an example.
14. What is a Pandas Series? How is it different from a Numpy array?
15. How do you access a column from a dataframe?
16. How do you access a row from a dataframe?
17. How do you access an element at a specific row and column of a dataframe?
18. How do you create a subset of a dataframe with a specific set of columns?
19. How do you create a subset of a dataframe with a specific range of rows?
20. Does changing a value within a dataframe affect other dataframes created using a subset of the rows or columns? Why is it so?
21. How do you create a copy of a dataframe?
22. Why should you avoid creating too many copies of a dataframe?
23. How do you view the first few rows of a dataframe?
24. How do you view the last few rows of a dataframe?
25. How do you view a random selection of rows of a dataframe?
26. What is the "index" in a dataframe? How is it useful?
27. What does a NaN value in a Pandas dataframe represent?
28. How is Nan different from 0?
29. How do you identify the first non-empty row in a Pandas series or column?
30. What is the difference between df.loc and df.at?
31. Where can you find a full list of methods supported by Pandas DataFrame and Series objects?
32. How do you find the sum of numbers in a column of a dataframe?
33. How do you find the mean of numbers in a column of a dataframe?
34. How do you find the number of non-empty numbers in a column of a dataframe?
35. What is the result obtained by using a Pandas column in a boolean expression? Illustrate with an example.
36. How do you select a subset of rows where a specific column's value meets a given condition? Illustrate with an example.
37. What is the result of the expression df[df.new_cases > 100] ?
38. How do you display all the rows of a pandas dataframe in a Jupyter cell output?
39. What is the result obtained when you perform an arithmetic operation between two columns of a dataframe? Illustrate with an example.
40. How do you add a new column to a dataframe by combining values from two existing columns? Illustrate with an example.
41. How do you remove a column from a dataframe? Illustrate with an example.
42. What is the purpose of the inplace argument in dataframe methods?
43. How do you sort the rows of a dataframe based on the values in a particular column?
44. How do you sort a pandas dataframe using values from multiple columns?
45. How do you specify whether to sort by ascending or descending order while sorting a Pandas dataframe?
46. How do you change a specific value within a dataframe?
47. How do you convert a dataframe column to the datetime data type?
48. What are the benefits of using the datetime data type instead of object?
49. How do you extract different parts of a date column like the month, year, month, weekday, and so on into separate columns? Illustrate with an example.
50. How do you aggregate multiple columns of a dataframe together?
51. What is the purpose of the groupby method of a dataframe? Illustrate with an example.
52. What are the different ways in which you can aggregate the groups created by groupby?
53. What do you mean by a running or cumulative sum?
54. How do you create a new column containing the running or cumulative sum of another column?
55. What are other cumulative measures supported by Pandas dataframes?
56. What does it mean to merge two dataframes? Give an example.
57. How do you specify the columns that should be used for merging two dataframes?
58. How do you write data from a Pandas dataframe into a CSV file? Give an example.
59. What are some other file formats you can write to from a Pandas dataframe? Illustrate with examples.
60. How do you create a line plot showing the values within a column of a dataframe?
61. How do you convert a column of a dataframe into its index?
62. Can the index of a dataframe be non-numeric?
63. What are the benefits of using a non-numeric dataframe? Illustrate with an example.
64. How you create a bar plot showing the values within a column of a dataframe?
65. What are some other types of plots supported by Pandas dataframes and series?

You are ready to move on to the next section of the tutorial.

Data Visualization using Python, Matplotlib, and Seaborn

Notebook link: https://jovian.ai/aakashns/python-matplotlib-data-visualization

Data visualization is the graphic representation of data. It involves producing images that communicate relationships among the represented data to viewers.

Visualizing data is an essential part of data analysis and machine learning. We'll use Python libraries Matplotlib and Seaborn to learn and apply some popular data visualization techniques. We'll use the words chart, plot, and graph interchangeably in this tutorial.

To begin, let's install and import the libraries. We'll use the matplotlib.pyplot module for basic plots like line and bar charts. It is often imported with the alias plt. We'll use the seaborn module for more advanced plots. It is commonly imported with the alias sns.

!pip install matplotlib seaborn --upgrade --quiet

import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

Notice this we also include the special command %matplotlib inline to ensure that our plots are shown and embedded within the Jupyter notebook itself. Without this command, sometimes plots may show up in pop-up windows.

How to Create a Line Chart in Python

The line chart is one of the simplest and most widely used data visualization techniques. A line chart displays information as a series of data points or markers connected by straight lines.

You can customize the shape, size, color, and other aesthetic elements of the lines and markers for better visual clarity.

Here's a Python list showing the yield of apples (tons per hectare) over six years in an imaginary country called Kanto.

yield_apples = [0.895, 0.91, 0.919, 0.926, 0.929, 0.931]

We can visualize how the yield of apples changes over time using a line chart. To draw a line chart, we can use the plt.plot function.

plt.plot(yield_apples)

Calling the plt.plot function draws the line chart as expected. It also returns a list of plots drawn [<matplotlib.lines.Line2D at 0x7ff70aa20760>], shown within the output. We can include a semicolon (;) at the end of the last statement in the cell to avoiding showing the output and display just the graph.

plt.plot(yield_apples);

Let's enhance this plot step-by-step to make it more informative and beautiful.

How to Customize the X-axis in MatPlotLib

The X-axis of the plot currently shows list element indices 0 to 5. The plot would be more informative if we could display the year for which we're plotting the data. We can do this by two arguments plt.plot.

years = [2010, 2011, 2012, 2013, 2014, 2015]
yield_apples = [0.895, 0.91, 0.919, 0.926, 0.929, 0.931]

plt.plot(years, yield_apples)

Axis Labels in MatPlotLib

We can add labels to the axes to show what each axis represents using the plt.xlabel and plt.ylabel methods.

plt.plot(years, yield_apples)
plt.xlabel('Year')
plt.ylabel('Yield (tons per hectare)');

How to Plot Multiple Lines in MatPlotLib

You can invoke the plt.plot function once for each line to plot multiple lines in the same graph. Let's compare the yields of apples vs. oranges in Kanto.

years = range(2000, 2012)
apples = [0.895, 0.91, 0.919, 0.926, 0.929, 0.931, 0.934, 0.936, 0.937, 0.9375, 0.9372, 0.939]
oranges = [0.962, 0.941, 0.930, 0.923, 0.918, 0.908, 0.907, 0.904, 0.901, 0.898, 0.9, 0.896, ]

plt.plot(years, apples)
plt.plot(years, oranges)
plt.xlabel('Year')
plt.ylabel('Yield (tons per hectare)');

Chart Title and Legend in MatPlotLib

To differentiate between multiple lines, we can include a legend within the graph using the plt.legend function. We can also set a title for the chart using the plt.title function.

plt.plot(years, apples)
plt.plot(years, oranges)

plt.xlabel('Year')
plt.ylabel('Yield (tons per hectare)')

plt.title("Crop Yields in Kanto")
plt.legend(['Apples', 'Oranges']);

How to Use Line Markers in MatPlotLib

We can also show markers for the data points on each line using the marker argument of plt.plot.

Matplotlib provides many different markers like a circle, cross, square, diamond, and more. You can find the full list of marker types here: https://matplotlib.org/3.1.1/api/markers_api.html .

plt.plot(years, apples, marker='o')
plt.plot(years, oranges, marker='x')

plt.xlabel('Year')
plt.ylabel('Yield (tons per hectare)')

plt.title("Crop Yields in Kanto")
plt.legend(['Apples', 'Oranges']);

How to Style Lines and Markers in MatPlotLib

The plt.plot function supports many arguments for styling lines and markers:

• color or c – Set the color of the line (supported colors)
• linestyle or ls – Choose between a solid or dashed line
• linewidth or lw – Set the width of a line
• markersize or ms – Set the size of markers
• markeredgecolor or mec – Set the edge color for markers
• markeredgewidth or mew – Set the edge width for markers
• markerfacecolor or mfc – Set the fill color for markers
• alpha – Opacity of the plot

Check out the documentation for plt.plot to learn more: https://matplotlib.org/api/_as_gen/matplotlib.pyplot.plot.html#matplotlib.pyplot.plot .

plt.plot(years, apples, marker='s', c='b', ls='-', lw=2, ms=8, mew=2, mec='navy')
plt.plot(years, oranges, marker='o', c='r', ls='--', lw=3, ms=10, alpha=.5)

plt.xlabel('Year')
plt.ylabel('Yield (tons per hectare)')

plt.title("Crop Yields in Kanto")
plt.legend(['Apples', 'Oranges']);

The fmt argument provides a shorthand for specifying the marker shape, line style, and line color. You can provide it as the third argument to plt.plot.

fmt = '[marker][line][color]'

plt.plot(years, apples, 's-b')
plt.plot(years, oranges, 'o--r')

plt.xlabel('Year')
plt.ylabel('Yield (tons per hectare)')

plt.title("Crop Yields in Kanto")
plt.legend(['Apples', 'Oranges']);

You can use the plt.figure function to change the size of the figure.

plt.plot(years, oranges, 'or')
plt.title("Yield of Oranges (tons per hectare)");

How to Change the Figure Size in MatPlotLib

You can use the plt.figure function to change the size of the figure.

plt.figure(figsize=(12, 6))

plt.plot(years, oranges, 'or')
plt.title("Yield of Oranges (tons per hectare)");

How to Improve Default Styles using Seaborn

An easy way to make your charts look beautiful is to use some default styles from the Seaborn library. You can apply them globally using the sns.set_style function. You can see a full list of predefined styles here: https://seaborn.pydata.org/generated/seaborn.set_style.html .

sns.set_style("whitegrid")
plt.plot(years, apples, 's-b')
plt.plot(years, oranges, 'o--r')

plt.xlabel('Year')
plt.ylabel('Yield (tons per hectare)')

plt.title("Crop Yields in Kanto")
plt.legend(['Apples', 'Oranges']);

sns.set_style("darkgrid")

plt.plot(years, apples, 's-b')
plt.plot(years, oranges, 'o--r')

plt.xlabel('Year')
plt.ylabel('Yield (tons per hectare)')

plt.title("Crop Yields in Kanto")
plt.legend(['Apples', 'Oranges']);

plt.plot(years, oranges, 'or')
plt.title("Yield of Oranges (tons per hectare)");

You can also edit default styles directly by modifying the matplotlib.rcParams dictionary. Learn more: https://matplotlib.org/3.2.1/tutorials/introductory/customizing.html#matplotlib-rcparams .

import matplotlib

matplotlib.rcParams['font.size'] = 14
matplotlib.rcParams['figure.figsize'] = (9, 5)
matplotlib.rcParams['figure.facecolor'] = '#00000000'

Scatter Plots in MatPlotLib

In a scatter plot, the values of 2 variables are plotted as points on a 2-dimensional grid. Additionally, you can also use a third variable to determine the size or color of the points. Let's try out an example.

The Iris flower dataset provides sample measurements of sepals and petals for three species of flowers. The Iris dataset is included with the Seaborn library and you can load it as a Pandas data frame.

# Load data into a Pandas dataframe
flowers_df = sns.load_dataset("iris")

flowers_df

flowers_df.species.unique()
# array(['setosa', 'versicolor', 'virginica'], dtype=object)

Let's try to visualize the relationship between sepal length and sepal width. Our first instinct might be to create a line chart using plt.plot.

plt.plot(flowers_df.sepal_length, flowers_df.sepal_width);

The output is not very informative as there are too many combinations of the two properties within the dataset. There doesn't seem to be simple relationship between them.

We can use a scatter plot to visualize how sepal length and sepal width vary using the scatterplot function from the seaborn module (imported as sns).

sns.scatterplot(x=flowers_df.sepal_length, y=flowers_df.sepal_width);

How to Add Hues in MatPlotLib

Notice how the points in the above plot seem to form distinct clusters with some outliers. We can color the dots using the flower species as a hue. We can also make the points larger using the s argument.

sns.scatterplot(x=flowers_df.sepal_length, y=flowers_df.sepal_width, hue=flowers_df.species, s=100);

Adding hues makes the plot more informative. We can immediately tell that Setosa irises have a smaller sepal length but higher sepal widths. In contrast, the opposite is true for Virginica irises.

How to Customize Seaborn Figures**

Since Seaborn uses Matplotlib's plotting functions internally, we can use functions like plt.figure and plt.title to modify the figure.

plt.figure(figsize=(12, 6))
plt.title('Sepal Dimensions')

sns.scatterplot(x=flowers_df.sepal_length, 
                y=flowers_df.sepal_width, 
                hue=flowers_df.species,
                s=100);

How to Plot Data using Pandas Data Frames with Seaborn

Seaborn has built-in support for Pandas data frames. Instead of passing each column as a series, you can provide column names and use the data argument to specify a data frame.

plt.title('Sepal Dimensions')
sns.scatterplot(x='sepal_length', 
                y='sepal_width', 
                hue='species',
                s=100,
                data=flowers_df);

Histograms in MatPlotLib

A histogram represents the distribution of a variable by creating bins (intervals) along the range of values and showing vertical bars to indicate the number of observations in each bin.

For example, let's visualize the distribution of values of sepal width in the Iris dataset. We can use the plt.hist function to create a histogram.

# Load data into a Pandas dataframe
flowers_df = sns.load_dataset("iris")

flowers_df.sepal_width
# 0      3.5
# 1      3.0
# 2      3.2
# 3      3.1
# 4      3.6
#       ... 
# 145    3.0
# 146    2.5
# 147    3.0
# 148    3.4
# 149    3.0
# Name: sepal_width, Length: 150, dtype: float64

plt.title("Distribution of Sepal Width")
plt.hist(flowers_df.sepal_width);

We can immediately see that the sepal widths lie in the range 2.0 - 4.5, and around 35 values are in the range 2.9 - 3.1, which seems to be the most populous bin.

How to Control the Size and Number of Bins**

We can control the number of bins or the size of each one using the bins argument.

# Specifying the number of bins
plt.hist(flowers_df.sepal_width, bins=5);

import numpy as np

# Specifying the boundaries of each bin
plt.hist(flowers_df.sepal_width, bins=np.arange(2, 5, 0.25));

# Bins of unequal sizes
plt.hist(flowers_df.sepal_width, bins=[1, 3, 4, 4.5]);

How to Manage Multiple Histograms in MatPlotLib

Similar to line charts, we can draw multiple histograms in a single chart. We can reduce each histogram's opacity so that one histogram's bars don't hide the others'.

Let's draw separate histograms for each species of flowers.

setosa_df = flowers_df[flowers_df.species == 'setosa']
versicolor_df = flowers_df[flowers_df.species == 'versicolor']
virginica_df = flowers_df[flowers_df.species == 'virginica']

plt.hist(setosa_df.sepal_width, alpha=0.4, bins=np.arange(2, 5, 0.25));
plt.hist(versicolor_df.sepal_width, alpha=0.4, bins=np.arange(2, 5, 0.25));

We can also stack multiple histograms on top of one another.

plt.title('Distribution of Sepal Width')

plt.hist([setosa_df.sepal_width, versicolor_df.sepal_width, virginica_df.sepal_width], 
         bins=np.arange(2, 5, 0.25), 
         stacked=True);

plt.legend(['Setosa', 'Versicolor', 'Virginica']);

Bar Charts in MatPlotLib

Bar charts are quite similar to line charts, that is they show a sequence of values. However, a bar is shown for each value, rather than points connected by lines. We can use the plt.bar function to draw a bar chart.

years = range(2000, 2006)
apples = [0.35, 0.6, 0.9, 0.8, 0.65, 0.8]
oranges = [0.4, 0.8, 0.9, 0.7, 0.6, 0.8]

plt.bar(years, oranges);

Like histograms, we can stack bars on top of one another. We use the bottom argument of plt.bar to achieve this.

plt.bar(years, apples)
plt.bar(years, oranges, bottom=apples);

Bar Plots with Averages in Seaborn

Let's look at another sample dataset included with Seaborn called tips. The dataset contains information about the sex, time of day, total bill, and tip amount for customers visiting a restaurant over a week.

tips_df = sns.load_dataset("tips");

tips_df

We might want to draw a bar chart to visualize how the average bill amount varies across different days of the week. One way to do this would be to compute the day-wise averages and then use plt.bar (try it as an exercise).

However, since this is a very common use case, the Seaborn library provides a barplot function which can automatically compute averages.

sns.barplot(x='day', y='total_bill', data=tips_df);

The lines cutting each bar represent the amount of variation in the values. For instance, it seems like the variation in the total bill is relatively high on Fridays and low on Saturdays.

We can also specify a hue argument to compare bar plots side-by-side based on a third feature, for example sex.

sns.barplot(x='day', y='total_bill', hue='sex', data=tips_df);

You can make the bars horizontal simply by switching the axes.

sns.barplot(x='total_bill', y='day', hue='sex', data=tips_df);

Heatmaps in Seaborn

A heatmap is used to visualize 2-dimensional data like a matrix or a table using colors. The best way to understand it is by looking at an example.

We'll use another sample dataset from Seaborn, called flights, to visualize monthly passenger footfall at an airport over 12 years.

flights_df = sns.load_dataset("flights").pivot("month", "year", "passengers")

flights_df

flights_df is a matrix with one row for each month and one column for each year. The values show the number of passengers (in thousands) that visited the airport in a specific month of a year. We can use the sns.heatmap function to visualize the footfall at the airport.

plt.title("No. of Passengers (1000s)")
sns.heatmap(flights_df);

The brighter colors indicate a higher footfall at the airport. By looking at the graph, we can infer two things:

• The footfall at the airport in any given year tends to be the highest around July and August.
• The footfall at the airport in any given month tends to grow year by year.

We can also display the actual values in each block by specifying annot=True and using the cmap argument to change the color palette.

plt.title("No. of Passengers (1000s)")
sns.heatmap(flights_df, fmt="d", annot=True, cmap='Blues');

Images in MatPlotLib

We can also use Matplotlib to display images. Let's download an image from the internet.

from urllib.request import urlretrieve

urlretrieve('https://i.imgur.com/SkPbq.jpg', 'chart.jpg');

Before displaying an image, it has to be read into memory using the PIL module.

from PIL import Image

img = Image.open('chart.jpg')

An image loaded using PIL is simply a 3-dimensional numpy array containing pixel intensities for the red, green & blue (RGB) channels of the image. We can convert the image into an array using np.array.

img_array = np.array(img)

img_array.shape
# (481, 640, 3)

We can display the PIL image using plt.imshow.

plt.imshow(img);

We can turn off the axes & grid lines and show a title using the relevant functions.

plt.grid(False)
plt.title('A data science meme')
plt.axis('off')
plt.imshow(img);

To display a part of the image, we can simply select a slice from the numpy array.

plt.grid(False)
plt.axis('off')
plt.imshow(img_array[125:325,105:305]);

How to Plot Multiple Charts in a Grid in MatPlotLib and Seaborn

Matplotlib and Seaborn also support plotting multiple charts in a grid, using plt.subplots, which returns a set of axes for plotting.

Here's a single grid showing the different types of charts we've covered in this tutorial.

fig, axes = plt.subplots(2, 3, figsize=(16, 8))

# Use the axes for plotting
axes[0,0].plot(years, apples, 's-b')
axes[0,0].plot(years, oranges, 'o--r')
axes[0,0].set_xlabel('Year')
axes[0,0].set_ylabel('Yield (tons per hectare)')
axes[0,0].legend(['Apples', 'Oranges']);
axes[0,0].set_title('Crop Yields in Kanto')


# Pass the axes into seaborn
axes[0,1].set_title('Sepal Length vs. Sepal Width')
sns.scatterplot(x=flowers_df.sepal_length, 
                y=flowers_df.sepal_width, 
                hue=flowers_df.species, 
                s=100, 
                ax=axes[0,1]);

# Use the axes for plotting
axes[0,2].set_title('Distribution of Sepal Width')
axes[0,2].hist([setosa_df.sepal_width, versicolor_df.sepal_width, virginica_df.sepal_width], 
         bins=np.arange(2, 5, 0.25), 
         stacked=True);

axes[0,2].legend(['Setosa', 'Versicolor', 'Virginica']);

# Pass the axes into seaborn
axes[1,0].set_title('Restaurant bills')
sns.barplot(x='day', y='total_bill', hue='sex', data=tips_df, ax=axes[1,0]);

# Pass the axes into seaborn
axes[1,1].set_title('Flight traffic')
sns.heatmap(flights_df, cmap='Blues', ax=axes[1,1]);

# Plot an image using the axes
axes[1,2].set_title('Data Science Meme')
axes[1,2].imshow(img)
axes[1,2].grid(False)
axes[1,2].set_xticks([])
axes[1,2].set_yticks([])

plt.tight_layout(pad=2);

See this page for a full list of supported functions: https://matplotlib.org/3.3.1/api/axes_api.html#the-axes-class .

Pair Plots with Seaborn

Seaborn also provides a helper function sns.pairplot to automatically plot several different charts for pairs of features within a dataframe.

sns.pairplot(flowers_df, hue='species');

See the full output here.

sns.pairplot(tips_df, hue='sex');

Summary and Further Reading

We have covered the following topics in this tutorial:

• How to create and customize line charts using Matplotlib
• How to visualize relationships between two or more variables using scatter plots
• How to study distributions of variables using histograms and bar charts
• How to visualize two-dimensional data using heatmaps
• How to display images using Matplotlib's plt.imshow
• How to plot multiple Matplotlib and Seaborn charts in a grid

In this tutorial we've covered some of the fundamental concepts and popular techniques for data visualization using Matplotlib and Seaborn. Data visualization is a vast field and we've barely scratched the surface here. Check out these references to learn and discover more:

• Data Visualization cheat sheet: https://jovian.ml/aakashns/dataviz-cheatsheet
• Seaborn gallery: https://seaborn.pydata.org/examples/index.html
• Matplotlib gallery: https://matplotlib.org/3.1.1/gallery/index.html
• Matplotlib tutorial: https://github.com/rougier/matplotlib-tutorial

Review Questions to Check Your Comprehension

Try answering the following questions to test your understanding of the topics covered in this notebook:

1. What is data visualization?
2. What is Matplotlib?
3. What is Seaborn?
4. How do you install Matplotlib and Seaborn?
5. How you import Matplotlib and Seaborn? What are the common aliases used while importing these modules?
6. What is the purpose of the magic command %matplotlib inline?
7. What is a line chart?
8. How do you plot a line chart in Python? Illustrate with an example.
9. How do you specify values for the X-axis of a line chart?
10. How do you specify labels for the axes of a chart?
11. How do you plot multiple line charts on the same axes?
12. How do you show a legend for a line chart with multiple lines?
13. How you set a title for a chart?
14. How do you show markers on a line chart?
15. What are the different options for styling lines and markers in line charts? Illustrate with examples.
16. What is the purpose of the fmt argument to plt.plot?
17. Where can you see a list of all the arguments accepted by plt.plot?
18. How do you change the size of the figure using Matplotlib?
19. How do you apply the default styles from Seaborn globally for all charts?
20. What are the predefined styles available in Seaborn? Illustrate with examples.
21. What is a scatter plot?
22. How is a scatter plot different from a line chart?
23. How do you draw a scatter plot using Seaborn? Illustrate with an example.
24. How do you decide when to use a scatter plot vs a line chart?
25. How do you specify the colors for dots on a scatter plot using a categorical variable?
26. How do you customize the title, figure size, legend, and son on for Seaborn plots?
27. How do you use a Pandas dataframe with sns.scatterplot?
28. What is a histogram?
29. When should you use a histogram vs a line chart?
30. How do you draw a histogram using Matplotlib? Illustrate with an example.
31. What are "bins" in a histogram?
32. How do you change the sizes of bins in a histogram?
33. How do you change the number of bins in a histogram?
34. How do you show multiple histograms on the same axes?
35. How do you stack multiple histograms on top of one another?
36. What is a bar chart?
37. How do you draw a bar chart using Matplotlib? Illustrate with an example.
38. What is the difference between a bar chart and a histogram?
39. What is the difference between a bar chart and a line chart?
40. How do you stack bars on top of one another?
41. What is the difference between plt.bar and sns.barplot?
42. What do the lines cutting the bars in a Seaborn bar plot represent?
43. How do you show bar plots side-by-side?
44. How do you draw a horizontal bar plot?
45. What is a heat map?
46. What type of data is best visualized with a heat map?
47. What does the pivot method of a Pandas dataframe do?
48. How do you draw a heat map using Seaborn? Illustrate with an example.
49. How do you change the color scheme of a heat map?
50. How do you show the original values from the dataset on a heat map?
51. How do you download images from a URL in Python?
52. How do you open an image for processing in Python?
53. What is the purpose of the PIL module in Python?
54. How do you convert an image loaded using PIL into a Numpy array?
55. How many dimensions does a Numpy array for an image have? What does each dimension represent?
56. What are "color channels" in an image?
57. What is RGB?
58. How do you display an image using Matplotlib?
59. How do you turn off the axes and gridlines in a chart?
60. How do you display a portion of an image using Matplotlib?
61. How do you plot multiple charts in a grid using Matplotlib and Seaborn? Illustrate with examples.
62. What is the purpose of the plt.subplots function?
63. What are pair plots in Seaborn? Illustrate with an example.
64. How do you export a plot into a PNG image file using Matplotlib?
65. Where can you learn about the different types of charts you can create using Matplotlib and Seaborn?

Congratulations on making it to the end of this tutorial! You can now apply these skills to analyze real world datasets from sources like Kaggle.

If you're pursuing a career in data science and machine learning, consider joining the Zero to Data Science Bootcamp by Jovian. It's a 20-week part-time program where you'll complete 7 courses, 12 coding assignments and 4-real world projects. You will also receive 6 months of career support to help you find your first data science job.

When you have a list of data records in a dataframe, you may need to drop a specific list of rows depending on the needs of your model and your goals when studying your analytics.

In this tutorial, you'll learn how to drop a list of rows from a Pandas dataframe.

To learn how to drop columns, you can read here about How to Drop Columns in Pandas.

How to Drop a Row or Column in a Pandas Dataframe

To drop a row or column in a dataframe, you need to use the drop() method available in the dataframe. You can read more about the drop() method in the docs here.

Dataframe Axis

• Rows are denoted using axis=0
• Columns are denoted using axis=1

Dataframe Labels

• Rows are labelled using the index number starting with 0, by default.
• Columns are labelled using names.

Drop() Method Parameters

• index - the list of rows to be deleted
• axis=0 - Marks the rows in the dataframe to be deleted
• inplace=True - Performs the drop operation in the same dataframe, rather than creating a new dataframe object during the delete operation.

Sample Pandas DataFrame

Our sample dataframe contains the columns product_name, Unit_Price, No_Of_Units, Available_Quantity, and Available_Since_Date columns. It also has rows with NaN values which are used to denote missing values.

import pandas as pd

data = {"product_name":["Keyboard","Mouse", "Monitor", "CPU","CPU", "Speakers",pd.NaT],
        "Unit_Price":[500,200, 5000.235, 10000.550, 10000.550, 250.50,None],
        "No_Of_Units":[5,5, 10, 20, 20, 8,pd.NaT],
        "Available_Quantity":[5,6,10,"Not Available","Not Available", pd.NaT,pd.NaT],
        "Available_Since_Date":['11/5/2021', '4/23/2021', '08/21/2021','09/18/2021','09/18/2021','01/05/2021',pd.NaT]
       }

df = pd.DataFrame(data)

df

The dataframe will look like this:

And just like that we've created our sample dataframe.

After each drop operation, you'll print the dataframe by using df which will print the dataframe in a regular HTML table format.

You can read here about how to Pretty Print a Dataframe to print the dataframe in different visual formats.

Next, you'll learn how to drop a list of rows in different use cases.

How to Drop a List of Rows by Index in Pandas

You can delete a list of rows from Pandas by passing the list of indices to the drop() method.

df.drop([5,6], axis=0, inplace=True)

df

In this code,

• [5,6] is the index of the rows you want to delete
• axis=0 denotes that rows should be deleted from the dataframe
• inplace=True performs the drop operation in the same dataframe

After dropping rows with the index 5 and 6, you'll have the below data in the dataframe:

This is how you can delete rows with a specific index.

Next, you'll learn about dropping a range of indices.

How to Drop Rows by Index Range in Pandas

You can also drop a list of rows within a specific range.

A range is a set of values with a lower limit and an upper limit.

This may be useful in cases where you want to create a sample dataset exlcuding specific ranges of data.

You can create a range of rows in a dataframe by using the df.index() method. Then you can pass this range to the drop() method to drop the rows as shown below.

df.drop(df.index[2:4], inplace=True)

df

Here's what this code is doing:

• df.index[2:4] generates a range of rows from 2 to 4. The lower limit of the range is inclusive and the upper limit of the range is exclusive. This means that rows 2 and 3 will be deleted and row 4 will not be deleted.
• inplace=True performs the drop operation in the same dataframe

After dropping rows within the range 2-4, you'll have the below data in the dataframe:

This is how you can drop the list of rows in the dataframe using its range.

How to Drop All Rows after an Index in Pandas

You can drop all rows after a specific index by using iloc[].

You can use iloc[] to select rows by using its position index. You can specify the start and end position separated by a :. For example, you'd use 2:3 to select rows from 2 to 3. If you want to select all the rows, you can just use : in iloc[].

This may be useful in cases where you want to split the dataset for training and testing purposes.

Use the below snippet to select rows from 0 to the index 2. This results in dropping the rows after the index 2.

df = df.iloc[:2]

df

In this code, :2 selects the rows until the index 2.

This is how you can drop all rows after a specific index.

After dropping rows after the index 2, you'll have the below data in the dataframe:

This is how you can drop rows after a specific index.

Next, you'll learn how to drop rows with conditions.

How to Drop Rows with Multiple Conditions in Pandas

You can drop rows in the dataframe based on specific conditions.

For example, you can drop rows where the column value is greater than X and less than Y.

This may be useful in cases where you want to create a dataset that ignores columns with specific values.

To drop rows based on certain conditions, select the index of the rows which pass the specific condition and pass that index to the drop() method.

df.drop(df[(df['Unit_Price'] >400) & (df['Unit_Price'] < 600)].index, inplace=True)

df

In this code,

• (df['Unit_Price'] >400) & (df['Unit_Price'] < 600) is the condition to drop the rows.
• df[].index selects the index of rows which passes the condition.
• inplace=True performs the drop operation in the same dataframe rather than creating a new one.

After dropping the rows with the condition which has the unit_price greater than 400 and less than 600, you'll have the below data in the dataframe:

This is how you can drop rows in the dataframe using certain conditions.

Conclusion

To summarize, in this article you've learnt what the drop() method is in a Pandas dataframe. You've also seen how dataframe rows and columns are labelled. And finally you've learnt how to drop rows using indices, a range of indices, and based on conditions.

If you liked this article, feel free to share it.

You May Also Like

• How to Add a Column to a Dataframe in Pandas
  • How to Rename a Column in Pandas

The Pandas package in Python gives you a bunch of cool functions and features that help you manipulate data more efficiently. It also lets you perform numerous data cleaning and data preprocessing steps with very little hassle.

That's great isn't it? Here's a list of some of the most frequently used Pandas functions and tricks to help you enjoy your data science journey.

How to Remove Missing Values in DataFrame

Getting rid of missing values is one of the most common tasks in data cleaning. Missing values could be just across one row or column or across multiple rows and columns.

Depending on your application and problem domain, you can use different approaches to handle missing data – like interpolation, substituting with the mean, or simply removing the rows with missing values.

Pandas offers the dropna function which removes all rows (for axis=0) or all columns (for axis=1) where missing values are present. Some of the arguments for the dropna function are as follows:

• axis which specifies if rows are to be dropped (axis=0) or if columns are to be dropped (axis=1)
• subset which specifies a list of columns to consider for missing values when axis=0
• inplace which specifies if changes are to be made in the existing DataFrame itself

Check out the docs linked here for more in-depth coverage.

In the example below, we're creating a small DataFrame with missing values and then discarding rows with missing values in any column.

Drop missing values in Pandas

How to Remove Duplicates in DataFrame

Another common data cleaning task is removing duplicate rows. The drop_duplicates function performs this with arguments similar to dropna such as:

• subset, which specifies a subset of columns to consider for duplicate value when axis=0
• inplace
• keep, which specifies which duplicated values to keep. Keep can be equal to first, last, or False to drop all duplicates.

Check out the docs linked here for more detailed info.

Let's duplicate a few rows and remove them from our dataset:

Drop duplicate values in Pandas

How to Remove Rows with Column-specific Values

Suppose we want to keep only those rows where project type is Web or where the number of hours worked is equal to 12. Here's how we can do it.

Using this method, we can filter out rows based on certain specific column values:

Remove rows with column specific values

How to Convert DataFrames to JSON

DataFrames are super cool optimized structures that are great to work with. And JSON is one of the most popular data formats for seamless data exchange.

Let's convert our DataFrame to JSON using to_json which requires arguments like:

• orient, which specifies what should be the key and value pairs. Default is columns, so column name is the key and each column is the value.
• date_format which specifies the format of the date. The default is epoch.

Look at the example below:

Convert DataFrame to JSON

We can see that to_json has returned a string with the following schema:

column_0 :
{ row_index_0: column_value_0, row_index_1:column_value_1, ...}, 
column_1:
{ row_index_0: column_value_0, row_index_1:column_value_1, ...}, 
...
column_N:
{ row_index_0: column_value_0, row_index_1:column_value_1, ...}

If we want to convert each row to a dictionary, we need to specify that orient=records and parse it using the JSON module.

Convert DataFrame to JSON with orient=records

How to Count the Number of Unique Values in a Column

Let's say we want to know how many different project types exist. We can get that information using the nunique function.

Count number of unique values in a column

How to Save DataFrame as .csv File

Just one line of code is required to save the DataFrame as a csv file:

dataset.to_csv("save_as_csv.csv")

How to Save Multiple Lists as One .csv File

Suppose we have three separate lists as our data source and we want to save them together in one csv file. This just involves two steps:

• converting it to a number of tuples using zip,
• and then converting it to a list.

In the example below, we follow this approach to convert the three lists into one DataFrame which we can now save as a .csv file.

Save multiple lists as one csv file

How to Read DataFrames in a Memory Efficient Way

Often we need to read files which are so large that they can't fit into memory. For such mammoth datasets, we use a different approach.

First, we create a TextFileReader object. Next we specify a parameter called chunksize which specifies how many rows of the file we want to read at a time, let's say 4 rows. So we read 4 rows at a time, perform some tasks on that chunk, and move on to the next 4 rows.

Small chunks are more likely to fit into memory than the entire file of thousands of rows. The following example shows how chunking works.

Read DataFrame in a memory efficient manner

Here we read the california dataset 1000 rows at a time, remove all rows where median_income is less than or equal to 3, and append these reduced chunks together to make a smaller dataset.

You can save more memory by reading only those columns which you need and specifying smaller datatypes for columns as described in detail in the docs linked here.

How to Change All Values in a DataFrame Using apply

Let's go back to our example of a projects DataFrame to illustrate this. We focus on the Hours_Worked column, increasing the count by 1 if it's an even number and by 2 if it's an odd number. We use a lambda function for this purpose.

Change all values in a DataFrame using apply

Conclusion

Pandas is a powerful package which can seem daunting sometimes due to its vastness. This is why I tried to list out some of the most useful functions I've come across.

These Pandas functions will help you accelerate your data analysis endeavors. Thank you for your time and I hope you enjoyed reading this article.

###

###