Why Learn Data Analytics with Google Cloud?

Turning raw numbers into actionable insights is a skill that sets you apart in the job market. As a Cloud Data Analyst, you’ll be able to unlock the stories hidden within massive datasets, helping organizations make smarter, faster decisions.

This course will help you enter that world. It offers a blend of expert instruction, hands-on labs, and real-world projects that will help you build a portfolio to impress employers.

What’s In the Course?

This course is a carefully curated program developed by Google Cloud, designed to take you from beginner to job-ready. You’ll learn technical skills like SQL, data visualization, and cloud storage. And you’ll also learn how to communicate your findings and drive business impact.

The course is structured to help you build a portfolio of industry-relevant projects, so you can showcase your expertise to potential employers.

Here’s what you’ll learn:

1. Introduction to Data Analytics in Google Cloud
Get a solid foundation in cloud data analysis. You’ll define the field, explore the roles and responsibilities of a cloud data analyst, and understand how data analytics drives business value. This module sets the stage for your journey, introducing you to the tools and concepts you’ll use throughout the program.

2. Data Management and Storage in the Cloud
Dive into how data is structured, organized, and stored in the cloud. You’ll get hands-on with data lakehouse architecture and tools like BigQuery, learning how to manage data efficiently and securely. This section is crucial for understanding how to handle large-scale datasets in a modern cloud environment.

3. Data Transformation in the Cloud
Follow the journey of data from collection to insight. You’ll learn to use SQL and other tools to clean, transform, and prepare data for analysis. By mastering data transformation, you’ll be able to turn messy, raw data into clean, usable information.

4. The Power of Storytelling: How to Visualize Data in the Cloud
Master the art of data visualization. This course teaches you how to turn complex data into clear, compelling stories using cloud-based visualization tools, making your insights accessible to any audience. You’ll learn the five key stages of visualizing data in the cloud, from planning to building impactful dashboards.

5. Put It All Together: Prepare for a Cloud Data Analyst Job
Apply everything you’ve learned in a capstone project that simulates real-world challenges. You’ll combine your skills in analysis, visualization, and communication to solve a comprehensive data problem, building a portfolio piece that demonstrates your ability to analyze, visualize, and communicate data insights.

Learn by Doing

The course is designed to be watched alongside the Google Cloud Skills Boost platform, where you’ll find interactive labs and practice environments. You get 35 free credits per month for labs, and if you want to move faster, you can pay for unlimited access. These labs give you the chance to practice your skills in real cloud environments, reinforcing your learning with practical experience. Complete the program and you’ll earn a Google Cloud Data Analytics Certificate.

Start Your Journey Today

Whether you’re just starting out or looking to advance your career, this course is for anyone who wants to harness the power of data. If you’re interested in technology, business, or problem-solving, this course will give you the tools you need to succeed in the fast-growing field of data analytics.

Watch the full course on the freeCodeCamp.org YouTube channel and begin your journey into cloud data analytics (10-hour watch).

Whether you're trying to control a rocket or forecast the stock market, the ability to get good data from an uncertain environment is important.

This is exactly the problem Kalman filters help solve. Kalman filters offer a solution that help you deal with noisy data in many fields.

In this article, we'll discuss:

• Driving Through Fog: Kalman Filters as Your Headlights
• What are Kalman Filters?
• Kalman Filters in Action: A Step-by-Step Code Example
• Conclusion: Navigating Nonlinear Data with Advanced Techniques

Driving Through Fog: Kalman Filters as Your Headlights

Photo by eberhard grossgasteiger: https://www.pexels.com/photo/forest-under-clouds-1287075/

Imagine you are driving through a dense fog with limited visibility.

To reach the destination, you rely on your senses and your car's navigation system that combines real-time data with a predetermined map.

As you move, the car navigation system is always constantly adjusting to get the destination, and you are always relying on your senses to drive the car well.

This process is very similar to how a Kalman Filter works.

It is constantly updating, and it refines estimates based on incoming data. Even though that data is full of noise and uncertainty.

By integrating past information with current information, a Kalman Filter gives you a clear picture of where you are and where you're headed.

What are Kalman Filters?

Photo by Mike Bird on Pexels

A Kalman filter is a math algorithm used to find the state of a dynamic system from many noisy measurements.

It is often used for systems that change over time – like tracking the position of a moving object.

How Does a Kalman Filter Work?

The Kalman filter predicts your current state based on past data, like the map and your previous location.

When new data appears, like new GPS signals, the filter compares the new data with its prediction and adjusts its estimate.

Even if the data is noisy, the Kalman filter uses a smart averaging process to improve the estimation. Like how you balance what your navigation system tells you and what you see on the road.

By always integrating new data with past data, Kalman filters help you know where you are and where you are going. This way, it is possible to predict things even in uncertain conditions.

Why are Kalman Filters used in engineering?

Since Kalman filters are able to handle incomplete data, they are widely used to make good predictions even when the measurements are not certain.

This makes them very useful for:

• Navigation Systems: Estimating the position and velocity of vehicles.
• Robotics: Helping robots understand their environment and position.
• Finance: Filtering out noise from stock price data to predict trends.

This way, they are very adaptive and can process real-time information

What problem did Kalman Filters solve?

Kalman filters were developed by Rudolf Kalman in the early 1960s to solve the problem of managing uncertainty and noise in data

Nowadays, they are great for extracting meaningful information from noisy data.

Mathematically, Kalman Filters are called linear quadratic estimators.

This is because, in the process of estimating the future based on current and past data, Kalman filters use:

• Linear algebra: The study of vectors and matrices used to solve linear equations.
• Quadratic optimization: Finding the optimal solution for problems with squared terms

Kalman Filters in Action: A Step-by-Step Code Tutorial

Photo by capt.sopon: https://www.pexels.com/photo/gray-airplane-control-panel-3402846/

Kalman Filters were created to handle linear systems – that is, systems that follow predictable patterns.

In this code example, we will implement an Extended Kalman Filter. This is a variant that was created to handle non-linear data (in other words, systems that have unpredictable or changing patterns).

Here's the full code (which we'll break down below):

import numpy as np
from filterpy.kalman import ExtendedKalmanFilter as EKF
from filterpy.common import Q_discrete_white_noise

def fx(x, dt):
    """ State transition function for the nonlinear system. """
    # Example: x' = [x[0] + x[1]*dt, x[1]]
    F = np.array([x[0] + x[1]*dt, x[1]])
    return F

def hx(x):
    """ Measurement function for the nonlinear system. """
    # Example: z = [x[0]]
    return np.array([x[0]])

def jacobian_F(x, dt):
    """ Jacobian of the state transition function. """
    return np.array([[1, dt],
                     [0, 1]])

def jacobian_H(x):
    """ Jacobian of the measurement function. """
    return np.array([[1, 0]])

# Initialize EKF
ekf = EKF(dim_x=2, dim_z=1)

# Initial state
ekf.x = np.array([0, 1])

# Initial state covariance
ekf.P = np.eye(2)

# Process noise covariance
ekf.Q = Q_discrete_white_noise(dim=2, dt=1, var=0.1)

# Measurement noise covariance
ekf.R = np.array([[0.1]])

# Define the state transition and measurement functions
ekf.F = jacobian_F
ekf.H = jacobian_H

# Control input
dt = 1.0  # time step

# Simulated measurements
measurements = [1, 2, 3, 4, 5]

for z in measurements:
    # Predict step
    ekf.predict_update(z, HJacobian=jacobian_H, Hx=hx, Fx=fx, args=(dt,), hx_args=())

    # Print the current state estimate
    print("Estimated state:", ekf.x)

Full code

Let's see the code block by block.

Import the Libraries

import numpy as np
from filterpy.kalman import ExtendedKalmanFilter as EKF
from filterpy.common import Q_discrete_white_noise

Importing Libraries

In this part of the code we import the Python libraries we need:

• import numpy as np: This imports a tool called NumPy, which helps us work with numbers and lists of numbers (like a spreadsheet).
• from [filterpy](https://filterpy.readthedocs.io/en/latest/).kalman import ExtendedKalmanFilter as EKF: This brings in a tool called ExtendedKalmanFilter from the filterpy library. We will use this tool, named EKF here, to track things that change over time in a way that's not straight-line simple.
• from [filterpy](https://filterpy.readthedocs.io/en/latest/).common import Q_discrete_white_noise: This imports a function that helps us set up noise, which is like the natural "fuzziness" or uncertainty in our system.

Define How the System Works

def fx(x, dt):
    """ State transition function for the nonlinear system. """
    # Example: x' = [x[0] + x[1]*dt, x[1]]
    return np.array([x[0] + x[1]*dt, x[1]])

def hx(x):
    """ Measurement function for the nonlinear system. """
    # Example: z = [x[0]]
    return np.array([x[0]])

Define How the System Works

In this code we define how the system will work:

• fx(x, dt): This function describes how our system changes over time. It says the new position is the old position plus speed times time (x[0] + x[1]*dt). The speed (x[1]) stays the same.
• hx(x): This function tells us what we can measure from the system. Here, it says we can measure the position (x[0]).

Define How Changes Affect the System

def jacobian_F(x, dt):
    """ Jacobian of the state transition function. """
    return np.array([[1, dt],
                     [0, 1]])

def jacobian_H(x):
    """ Jacobian of the measurement function. """
    return np.array([[1, 0]])

Define How the System Works

In this code we define how changes affect the system:

• jacobian_F(x, dt): This function shows us how sensitive the system is to changes in time and position. It helps the filter predict changes more accurately by considering these sensitivities.
• jacobian_H(x): This function tells us how sensitive our measurement is to changes in position. It helps the filter adjust the prediction based on new measurements.

Set Up the Kalman Filter

# Initialize EKF
ekf = EKF(dim_x=2, dim_z=1)

# Initial state
ekf.x = np.array([0, 1])
print("Initial state:", ekf.x)

# Initial state covariance
ekf.P = np.eye(2)
print("Initial state covariance:\n", ekf.P)

Set Up the Kalman Filter

In this part of the code, we create a very simple Kalman filter:

• ekf = EKF(dim_x=2, dim_z=1): This creates an Extended Kalman Filter that tracks two things (position and speed) and one measurement (position).
• ekf.x = np.array([0, 1]): This sets the starting position to 0 and speed to 1.

It prints out:

Initial state: [0 1]

• ekf.P = np.eye(2): This is a way of saying we aren't very sure about our starting guesses. It's like saying "let's start from here, but we are open to changes."

It prints out:

Initial state covariance:
 [[1. 0.]
 [0. 1.]]

Describe Uncertainty in the System

# Process noise covariance
ekf.Q = Q_discrete_white_noise(dim=2, dt=1, var=0.1)
print("Process noise covariance:\n", ekf.Q)

# Measurement noise covariance
ekf.R = np.array([[0.1]])
print("Measurement noise covariance:\n", ekf.R)

Describe Uncertainty in the System

• ekf.Q = Q_discrete_white_noise(dim=2, dt=1, var=0.1): This sets how much randomness or unpredictability we expect in the system itself. It's like saying, "things might not move exactly as we think."

It prints out:

Process noise covariance:
 [[0.025 0.05 ]
 [0.05  0.1  ]]

• ekf.R = np.array([[0.1]]): This sets how much we trust our measurements. A smaller number means we trust them more.

Measurement noise covariance:
 [[0.1]]

Simulate Data and Initial State

# Control input
dt = 1.0  # time step

# Simulated measurements
measurements = [1, 2, 3, 4, 5]

# True initial state for comparison (not used in the EKF)
true_state = np.array([0, 1])
print("\nTrue initial state:", true_state)

Simulate Data and Initial State

• dt = 1.0: This is the time between each step of our simulation.
• measurements = [1, 2, 3, 4, 5]: These are the pretend measurements we will use to test the filter.
• true_state = np.array([0, 1]): This is the real starting position and speed of our system, used for comparison.

It gives:

True initial state: [0 1]

Simulate Real System Changes

# Simulate the true state evolution (for comparison)
true_states = [true_state[0]]
for _ in range(len(measurements) - 1):
    true_state = fx(true_state, dt)
    true_states.append(true_state[0])

print("\nSimulated true states (for reference):", true_states)

Simulate Real System Changes

• Simulating True States: This part calculates what the real position should be over time using the way the system works (fx). It's like having a perfect GPS to check against our estimates.

Simulated true states (for reference): [0, 1.0, 2.0, 3.0, 4.0]

Filter Steps to Estimate the State

for i, z in enumerate(measurements):
    print(f"\nStep {i+1}:")
    print("Measurement:", z)

    # Predict step
    ekf.predict(u=0)  # Use predict_x if you need to customize the prediction
    print("Predicted state before update:", ekf.x)

    # Update step
    ekf.update(z, HJacobian=jacobian_H, Hx=hx, args=(), hx_args=())
    print("Updated state after measurement:", ekf.x)
    print("State covariance after update:\n", ekf.P)

Filter Steps to Estimate the State

Loop Through Measurements: This loop goes through each fake measurement one by one.

• Predict Step (ekf.predict(u=0)): Before looking at the new measurement, the filter makes a guess about where the position and speed are now.
• Update Step (ekf.update): After the guess, the filter sees the new measurement and adjusts its guess to be closer to this measurement, balancing the new information with what it previously predicted.

Here are the results:

Step 1:
Measurement: 1
Predicted state before update: [0. 1.]
Updated state after measurement: [0.91111111 1.04444444]
State covariance after update:
 [[0.09111111 0.00444444]
 [0.00444444 1.09777778]]

Step 2:
Measurement: 2
Predicted state before update: [0.91111111 1.04444444]
Updated state after measurement: [1.49614396 1.31876607]
State covariance after update:
 [[0.05372751 0.0251928 ]
 [0.0251928  1.1840617 ]]

Step 3:
Measurement: 3
Predicted state before update: [1.49614396 1.31876607]
Updated state after measurement: [2.15857605 1.95145631]
State covariance after update:
 [[0.0440489  0.0420712 ]
 [0.0420712  1.25242718]]

Step 4:
Measurement: 4
Predicted state before update: [2.15857605 1.95145631]
Updated state after measurement: [2.91071524 2.95437384]
State covariance after update:
 [[0.04084552 0.05446424]
 [0.05446424 1.30228131]]

Step 5:
Measurement: 5
Predicted state before update: [2.91071524 2.95437384]
Updated state after measurement: [3.74022237 4.27039095]
State covariance after update:
 [[0.03970292 0.06298888]
 [0.06298888 1.33648045]]

Conclusion: Navigating Nonlinear Data with Advanced Techniques

Photo by Noelle Otto on Pexels

Kalman Filters are a powerful tool for extracting accurate estimates from noisy and incomplete data.

Variants like the Extended Kalman Filter (EKF) and Unscented Kalman Filter (UKF) have been developed to address non-linearities in data.

However, these variants can still face challenges related to stability and accuracy when applied to complex non-linear systems.

This is due to their reliance on linear approximations, which may not capture the full dynamics of highly non-linear processes.

To overcome these limitations, alternative methods such Neural Network-based approaches have gained attention.

Neural Networks can learn complex patterns directly from data, offering a robust solution for highly non-linear scenarios.

Despite these advancements, Kalman Filters remain an important tool in various fields of science and economics due to their simplicity, efficiency, and effectiveness in a wide range of applications.

As technology continues to evolve, the integration of Kalman Filters with other advanced techniques will likely enhance their capability to navigate the challenges of non-linear data more effectively.

Here is the full code:

We just posted a course on the freeCodeCamp.org YouTube channel that will teach you about data analytics through hands-on projects, leveraging a suite of free Google services. This course from Analyst Adithya offers a practical approach to understanding and applying data analytics tools.

Data Analysis with Google Sheets

The course starts with Google Sheets, a versatile tool that goes beyond basic spreadsheet functionalities. You will learn how to perform sophisticated data analysis tasks, manipulate data sets, and employ formulas and functions to uncover insights. Google Sheets' accessibility and user-friendly interface make it an excellent starting point for those new to data analytics, providing a solid foundation in data manipulation and analysis.

SQL with Google BigQuery

As you progress, you are introduced to Google BigQuery, a powerful tool for querying massive datasets using SQL (Structured Query Language). This segment of the course is designed to bridge the gap between traditional spreadsheet analysis and more advanced database management. By exploring BigQuery, you will gain proficiency in SQL, learning how to execute complex queries, join tables, and extract meaningful information from large data sets.

Python with Google Colab

The course then teaches Python programming with Google Colab, an environment that allows users to write and execute Python code through their browsers. This section is particularly exciting for those looking to harness the power of Python for data analysis, offering a hands-on experience with coding, data manipulation, and the application of libraries like Pandas and NumPy. Google Colab's collaborative features also introduce learners to the concept of shared, interactive computing.

Data Visualization with Google Looker Studio

Finally, the course ends with data visualization using Google Looker Studio (formerly known as Google Data Studio), where learners will discover how to transform their analyzed data into compelling, interactive reports and dashboards. This module emphasizes the importance of data storytelling, teaching participants how to present their findings visually to communicate insights effectively and influence decision-making.

Conclusion

By covering a range of Google services, the course provides a broad perspective on data analytics, ensuring that you are well-versed in various tools and methodologies.

Watch the full course on the freeCodeCamp.org YouTube channel (3-hour watch).

In this article, I'll introduce you to the Google Certifications and detail what's new. I'll focus on the Data Analytics pathway, but will include resources to their other programs.

Video Walkthrough

Here's a video walkthrough of what we'll discuss below.

Grow With Google Overview

Career Certifications from Google are a part of their "Grow with Google" initiative. A portion of their mission page is in the picture below 👇.

Grow with Google is an assortment of training and learning resources for small business owners, job seekers, veterans, educators, developers and startups to learn vital digital skills.

From Grow with Google's mission: https://grow.google/our-mission/

Part of Grow with Google are the Career Certificates pathways which offer flexible online training programs in Digital Marketing & E-commerce, IT Support, Data Analytics, Project Management, and UX Design.👇

picture of Grow with Google's Career Certificates page

The proposition is attractive: no experience necessary to begin the programs, learn at your own pace, have a well-regarded certification from Google at the end of it and be qualified for a new entry-level job in careers with a median salary of $72,000 across the certificate fields.

The median salaries range from $57,000 for IT Support to $112,000 for UX Design.

The entire certificate program is a partnership between Google, who has crafted the courses, and Coursera, which is one of the OG MOOCS that hosts the coursework.

This of course begs the question: how much does it cost? 🤔

gif of man saying, how much money?

The good news is that you can nab the knowledge for free. And if you fork over a little bit, you can have the official certificate to display proudly on that portion of your LinkedIn page rarely ever scrolled to. 😀

pic of my LinkedIn licenses & certifications

The Free Version:

Yes, the knowledge is free. You can audit any of these courses at Coursera for free. So, while there's not an option to audit the whole certificate, you can click into the individual courses and audit them one by one.

picture of audit option in coursera courser

I've clicked into the Foundations of Data Science course and selected audit. In the screenshot below, you can see that I now have access to all the coursework.

screenshot of Foundations of Data Science coursework on coursera

What don't you get with the free version?

Well, you don't get that nice certificate "proving" you did the work. But if you've got the knowledge, you can muster together projects that will do a better job of proving that anyway.

And you also may not be able to turn in certain assignments for grades, depending on the course.

The Not Free Version:

Coursera costs $39 a month for any of the individual Google certificates, including the new advanced ones. And since it's at your own pace, if you work fast, you can finish in one, two, or three months instead of the projected six...saving you some $$$. 💵💰

pic of Coursera's individual course pricing

As is now common, Coursera also offers a more expensive Plus program for $59 a month or $399 a year. This gives unlimited access to courses and could be a better deal if you plan on plowing through many courses during a year.

But, you'd have to be flying through them for this to be valuable considering you could by 10 individual courses for $390.

pic of coursera plus costs

My opinion: if you're strapped for cash, start knocking out the free audit versions. But, it's pretty cheap to grab a certificate if that's important to you. Particularly if you work hard and fast 👉 it could cost as little as one month's $39 fee.

My Experience with the Data Analytics Cert

I took the Data Analytics Certification a while back, and can attest that it was a fantastic overview of the field. I'd been using Excel heavily before taking the course, and was familiar with programming and database concepts, so I was able to move at a brisk pace.

I spent 2-3 months working through the "6 month" course, and certainly could have gone faster. Six months is conservative, even for a novice, in my opinion.

screenshot of my data analytics certificate

This certificate took some flak because it used R instead of Python for the programming portion. But I thought it was well laid out and offered a good introduction to data analytics.

Would it make someone hirable?

Well, it's just one piece in that puzzle. Relying on any of these certifications alone to land a job is perhaps unwise. It will take a combination of the skills learned, the networking you're willing to do, and your continued work through projects to prove your skills and land a job.

On the whole, I recommend it, especially if you're new to Excel, SQL, or programming.

And if you're wondering, yes, I ponied up the cash and enrolled in the program. It took me a little over two months and so I paid $117 +tax.

My payment history for Data Analytics coursework

Advanced Certificates

My foray into the data analytics was one of curiosity more than anything. I was not actively seeking new employment – I simply wanted to sharpen and expand my skills. And that's exactly what piqued my interest in the new, advanced certificates that are now available.

pic of banner advertising new advanced Certificates

In addition to Industry Specializations (added in October 2022), there are now three advanced certificates (as of April 2023) with more likely in the future: Advanced Data Analytics, Business Intelligence, and IT Automation with Python.

pic of advanced certificate listings from Grow with Google

According to Google's announcement post, the Data Analytics Certificate program was the most popular professional certificate on Coursera, so it's unsurprising that this field has received the first advanced follow-up courses.

Excerpt from announcement post

And the field is well paid, making it an attractive aspiration for career transitioners and students alike.

_pic of advanced data analytics certificate details from https://grow.google/certificates/data-analytics/?advanced#?modalactive=none

Of particular note, Google has wisely opted to go with Python and Jupyter Notebook (industry standard tools) as detailed in the above card for the Advanced Data Analytics Certificate.

Below, you can see similar details for the Business Intelligence Certificate which focuses more heavily on modeling, dashboards, and engaging with stakeholders via SQL, Tableau, and BigQuery.

_pic of details from business intelligence certificate from https://grow.google/certificates/data-analytics/?advanced#?modalactive=none

I'm excited to try out both courses and see how deep they go after the introductory, pretty surface level course. And right now, it has that same fresh, new, adventurous feel as when a new WoW server launches 🧙‍♂️. Only 3 enrolled in the Data Analytics Advanced degree!😲😅

pic of advanced data analytics certificate page on Coursera

And there are only 4 in the Business Intelligence program! This is sure to change quickly, and indeed, there are a few hundred in the individual courses already. Looks like many will indeed opt to audit the courses before jumping into the paid program.

Coursework Overview

Here are the respective courses in the two new advanced certificates. I encourage you to go check them out yourselves for further details, or watch my video walkthrough, but this will give you an idea of what to expect.

Data Analytics Courses

Go here to check out the overview of each data analytics course.

Click into each course below to explore in more detail and see all the modules within each course. This program is estimated by Google to take six months to complete.

1. Foundations of Data Science
2. Get Started with Python
3. Go Beyond the Numbers: Translate Data into Insights
4. The Power of Statistics
5. Regression Analysis: Simplify Complex Data Relationships
6. The Nuts and Bolts of Machine Learning
7. Google Advanced Data Analytics Capstone

You can see from the screenshot below the estimated hours to complete each module as well as the higher $118,000 median salary for entry-level roles:

screenshot of coursework for Advanced Data Analytics

Business Intelligence Courses

Go here to check out the overview of each business intelligence course.

Click into each course below to explore in more detail and see all the modules within each course. This shorter program is estimated by Google to take two months to complete.

1. Foundations of Business Intelligence
2. The Path to Insights: Data Models and Pipelines
3. Decisions, Decisions: Dashboards and Reports

screenshot of Business Intelligence certificate coursework

After the Certificate

Google has also developed an employer consortium of over 150 companies which will consider recent graduates of Google's programs for hiring. While this is certainly no guarantee of employment, if you're in the market it is a nice touch, and will be another helpful tool in your search.

Summary

I hope this has been a helpful overview for you!

I make weekly spreadsheet and coding videos on my YouTube channel. Come check it out if you find this helpful! Here's a link to my Google Sheets playlist where I walk through basic and advanced topics alike:

You can also find me on LinkedIn. Have a great one!👋

Let's face it, we're all drowning in data these days. From social media posts to financial transactions to medical records, there's no shortage of information to sift through.

That's where data analysts come in. They're the ones who help us make sense of all that data and turn it into valuable insights.

And those insights can be game-changing. They can help businesses improve their products and services, governments make more informed policy decisions, and individuals make better choices in their personal and professional lives.

But it's not just about the impact. Data analysis can also be quite lucrative. According to recent studies, the median salary for a data analyst in the US is around $70,000 per year, and that number can climb even higher with experience and expertise.

Of course, like any profession, data analysis has its challenges. There's the occasional headache-inducing data set, the ever-present threat of imposter syndrome, and the endless debates over the best programming language or data visualization tool. But hey, if you're up for a challenge, this could be the field for you.

So if you're trying to be a Data Analyst, this article is for you. Hopefully, it saves you a lot of time and effort and you don't have to waste your time learning a whole lot of irrelevant things like I once did.

Step 1: Get to Know the Role of a Data Analyst

Are you considering a career as a data analyst? That's fantastic! Let's take a moment to grasp the gist of it before moving forward.

You see, a data analyst's job involves more than just crunching figures and creating charts (although those things can be pretty cool, too). It involves leveraging data to find insights and address issues. This means coming up with the correct questions, organizing and evaluating the data, and explaining your conclusions to others.

Some core skills and activities that a data analyst typically performs include:

• Collecting and analyzing large data sets to identify patterns, trends, and insights that can inform business decisions.

• Using statistical tools and techniques to draw insights from data.

• Developing and implementing data collection systems and other strategies that optimize efficiency and data quality.

• Collaborating with other teams to identify business needs and develop data solutions that address them.

• Communicating findings and insights to stakeholders in a clear and actionable way.

So if you're not a math prodigy or a computer whiz, don't worry. Statistics, computer science, and business are just a few of the numerous disciplines that are incorporated into the multidisciplinary topic of data analysis. Everyone who enjoys learning, solving problems, and making a difference can really enjoy it.

So how can you find out more about what a data analyst does? You can start by looking through the numerous free resources that are accessible online. For example, here an article that discusses what data analysts actually do. And here's one that discusses data analyst vs data science roles.

Many blogs, podcasts, and YouTube channels offer entertaining and informative content on the subject of data analysis. I will be dropping some YouTube channels that have helped me over the years below.

To gain a feel of the skills and qualities needed, you can also network with other data analysts, go to meetings or seminars, and study job descriptions.

Don't forget to consider whether dealing with data is something you enjoy. Do you enjoy finding patterns and solving puzzles? Do you want to change the world for the better? Data analysis may be the ideal career choice for you if the answer is yes.

The first step in your path is to understand what a data analyst does. Enjoy yourself, take your time, and don't be hesitant to ask questions.

Step 2: Explore Job Requirements for Data Analyst Roles

Now that you have a better understanding of the role of a data analyst, it's time to start looking at what employers are looking for. After all, you want to make sure that your skills and knowledge match up with what's required in the job market.

But before you start panicking about not having enough experience, remember that every company is different. Some may prioritize programming skills, while others may value communication and business acumen. That's why it's important to do your research and find out what specific skills and qualifications are most in demand in your desired industry or company.

So how do you go about finding this information? Well, one great place to start is by checking out job listings and descriptions on job boards like LinkedIn, Indeed, or Glassdoor. This can give you a good sense of the key requirements and qualifications for different data analyst roles.

Some examples of what job listings might ask for include:

• Proficiency in SQL and experience working with large datasets

• Familiarity with Python and data visualization tools like Tableau or Power BI

• Strong analytical skills and the ability to draw insights from complex data

• Experience with statistical analysis and modeling techniques

• Excellent communication skills and the ability to explain complex findings to both technical and non-technical audiences

But don't stop there! You can also reach out to people who work in the field or who have job titles that interest you. Ask them about their experience and what skills they think are most important for success in their role. You might even want to consider setting up informational interviews to learn more about the field and get advice on how to get started.

And speaking of getting started, it's important to remember that there's no substitute for hands-on experience. As tempting as it may be to spend all your time watching tutorials, you'll learn much more quickly and effectively by actually building things and working on real data analysis projects.

So take some time to explore job requirements, but don't forget to keep building your skills and gaining practical experience. With a little effort and a lot of curiosity, you'll be well on your way to becoming a successful data analyst.

Step 3: Get Comfortable with Math and Statistics

Okay, I know what you're thinking. Math and statistics? Yikes!

But hear me out before you run for the hills. For a data analyst to be able to make sense of data and derive valuable insights from it, having a fundamental understanding of these concepts is essential.

So what fundamental statistical concepts and formulas should you be familiar with?

Now, to start, there are measures of central tendency known as mean, median, and mode, which can give you an idea of the typical value in a dataset. You should know how to calculate them.

The standard deviation is a measure of how widely distributed the data are from the average, and you should be able to calculate it.

And in order to find relationships between variables and generate predictions based on those associations, you should also be familiar with correlation and regression.

But it's not just about numbers. You'll also need a rudimentary understanding of linear algebra, which is employed in many data analysis approaches. Here's an in-depth course (and textbook) to get you started.

You may need to employ matrices to modify and manipulate data, or you may need to use linear regression to forecast future values based on historical trends.

If you don't have a solid math background, this may seem difficult. But don't be concerned! There are numerous resources available to assist you in your learning.

For example, Khan Academy offers lessons and practice tasks in math and statistics. If you prefer books, "Data Science for Beginners" by Andrew Park is an excellent resource that covers both statistical and mathematical principles in an accessible manner.

freeCodeCamp is developing a math curriculum which you can read about here.

And here's a guide on the statistics you need to know to get into data science and pursue fields like Machine Learning.

The key is to start small and build up your knowledge gradually. Don't be afraid to ask questions or seek help when you need it. With a little practice and persistence, you'll soon find that math and statistics are actually kind of fun (no, seriously!).

Step 4: Master Excel for Data Analysis

Excel is a vital tool in a data analyst's arsenal. It's used by virtually every organization out there, and mastering it will help you clean, manipulate, and analyze data with ease.

With Excel, you can create formulas and functions to perform calculations, pivot tables and charts to visualize data, and use data analysis tools to make predictions and identify patterns. Excel is particularly useful for regression analysis, forecasting, and scenario analysis.

If you're serious about becoming a data analyst, it's essential to master Excel. Fortunately, there are plenty of online resources available to help you learn. Check out ExcelIsFun, Excel Chandoo, Tutorials Point, Ashutosh Kumar , and MyOnlineTrainingHub for tutorials on Youtube. Also, the following courses will guide you on how to get the most out of Excel.

1. Introduction to Data Analysis Using Excel by Coursera

2. Data Analysis with Python for Excel Users on freeCodeCamp's YouTube channel

3. Excel Skills for Business Specialization by Coursera

4. Analyzing and Visualizing Data with Excel by EdX

Remember, Excel is just one tool in your data analysis toolkit. But it's a crucial one that you'll use daily as a data analyst. By mastering Excel, you'll be well-equipped to handle any data-related task that comes your way.

Now let's move on to the next skill and also one of the most important skills you'll need as a data analyst.

Step 5: Master SQL for Data Extraction

SQL (Structured Query Language) is a critical tool in data analysis. As a data analyst, one of your primary responsibilities is to extract data from databases, and SQL is the language used to do so.

SQL is more than just running basic queries like SELECT, FROM, and WHERE. It's a complex language that allows you to manipulate and transform data in countless ways. SQL is used for joining data from multiple tables, filtering and aggregating data, and creating new tables and views.

To be an effective data analyst, it's essential to master SQL. You should be comfortable with writing queries, creating tables, and understanding how to optimize your queries for performance.

Fortunately, there are many resources available to help you learn SQL. Some great places to start are Khan Academy SQL, W3Schools, SQLZoo, SQLbolt, Luke Barousse, Alex the Analyst, Microsoft Power Tools, and finally some SQL games like SQL island and SQL Murder.

Additionally, there are many online courses and books available that cover SQL in-depth. Here are a few to get you started:

1. Learn SQL and Databases – Full Course for Beginners

2. Relational Database curriculum from freeCodeCamp

3. Collection of free relational database courses

By mastering SQL, you'll be able to extract valuable insights from databases and manipulate data in ways that provide meaningful business insights.

Step 6: Learn Python for Data Analysis

I know there's a lot of speculation as to whether or not a data analyst needs Python – some say they do while some say they don't.

For me, I'd like to say it depends on the company you're working for – but it's nice to have an edge by learning Python as it's one of the most widely used programming languages in the world of data analysis. Python is known for its simplicity, readability, and versatility, making it a popular choice for data analysts.

Python has a vast array of libraries and tools that can make data analysis easier, such as Pandas for data manipulation and analysis, NumPy for scientific computing, and Matplotlib for data visualization. It also has the ability to automate tasks, making data analysis more efficient and effective.

Learning Python for data analysis is a great investment in your career as a data analyst. Not only will it allow you to work with powerful libraries, but it will also open up many opportunities to work with larger datasets and more complex analyses.

There are many resources available to help you learn Python, from free online courses to paid online programs and textbooks. Some resources include freeCodeCamp, DataCamp, CodeBasis, Programming with Mosh and Microsoft Learn.

By learning Python, you'll be able to perform more complex data analysis, automate tasks, and work with a broader range of datasets, making you a valuable asset in any data-focused organization.

Step 7: Master a Data Visualization Tool

As a data analyst, it's essential to be able to communicate your findings in a clear and concise manner. One way to do this is through data visualization. Data visualization tools like PowerBI and Tableau can help you create interactive charts, graphs, and dashboards that make it easy for others to understand your findings. We'll talk about them more in a minute.

Here's a Tableau for Data Science and Data Visualization course you can check out.

While SQL is great for querying and manipulating data, it can't fully bring your data to life. This is where a data visualization tool comes in. These tools allow you to transform your data into insightful and easy-to-understand visualizations that can be shared with stakeholders.

You can learn data visualization basics in this in-depth free course on freeCodeCamp's YouTube channel.

PowerBI is a great choice for data visualization as it is easy to learn and integrate with other Microsoft products. This makes it an ideal tool for organizations that use Microsoft Office. Tableau is also a popular choice and has a strong community of users and a wide range of features.

Learning a data visualization tool like PowerBI or Tableau will enable you to create compelling visualizations that help you better understand your data and communicate your findings to others. There are many online courses and tutorials available to help you learn these tools, such as the official Microsoft PowerBI training and Tableau's own training courses.

By mastering a data visualization tool, you'll be able to create interactive and engaging visualizations that will help you better understand your data and communicate your findings to others, making you an invaluable asset to any data-focused organization.

You can also dive into other popular data viz tools like D3.js - here's a course on it to get you started.

Step 8: Network with Other Data Analysts and Developers

Networking is an essential part of any profession, and data analytics is no exception. By networking with other data analysts and developers, you can learn from their experiences, get insights on the latest industry trends and technologies, and potentially find job opportunities.

Here are a few ways to network with others in the field:

1. Attend industry events: Look for conferences, meetups, and other events related to data analytics and attend them. This is a great way to meet others in the field and learn about new developments and technologies.

2. Join online communities: There are many online communities for data analysts and developers, such as forums, LinkedIn groups, and social media groups. Join these communities and participate in discussions to connect with others in the field.

3. Reach out to others: Don't be afraid to reach out to other data analysts and developers, whether through social media, email, or in person. Introduce yourself, ask for advice, and build relationships.

Remember, networking is a two-way street. Be willing to offer help and advice to others in the field as well. By building a strong network of contacts in the data analytics field, you can enhance your career opportunities and stay up to date on the latest industry trends and technologies.

Step 9: Don't Forget about "Soft Skills"

One final skill that I think needs to be worked on before you can be a great DA is your soft skills which involves your ability to communicate, solve problems etc

After all is said and done, practice and practice and build projects.

Conclusion.

Becoming a data analyst requires dedication, hard work, and a passion for data analysis. Following the steps outlined in this roadmap will help you gain the necessary skills and knowledge to become a successful data analyst.

From understanding the role of a data analyst, to mastering SQL and Python, to networking with other developers, each step is crucial to achieving success in this field.

Remember to stay curious, never stop learning, and always be willing to adapt to new technologies and methodologies. With determination and persistence, you can achieve your goal of becoming a proficient data analyst and unlock a world of exciting career opportunities.

Data analytics mostly involves studying data trends over a given period, and then extracting useful information from these trends.

Why Is Data Analytics Important?

More precise decision making process: Data analytics helps organizations make more accurate decisions based on the insights gotten from data trends over time.

For example, a company selling different products can figure out what time of the year different products sell higher. This will enable them boost production of such products at the required time.

A better decision making process will eliminate the need for guess work, and minimize losses and avoidable risks.

Improved customer satisfaction: When you're able to serve customers, you retain them and keep business going. Insights gotten from data analytics can help you understand exactly what your customers want and when to act.

Data analytics also enables businesses to identify their target audience easily.

Improved business strategy: Data analytics helps organizations channel their resources towards the most efficient strategies.

Performance evaluation: Data analytics can help organizations evaluate how well or badly they've performed over a specified period. This will enable them make important decisions for the future of the organization.

Although the points listed above seem to be from the business point of view, that's not the only industry where data analytics is important.

You can see data analytics being used in healthcare, education, agriculture, and so on.

Types of Data Analytics

There are mainly four different types of data analytics:

• Descriptive analytics: This type of analytics has to do with what happened with analyzed data over a specified period of time.
• Diagnostic analytics: Diagnostic data analytics shows the "why" in a data trend. This involves having a deeper look into why certain patterns were present in the data.
• Predictive analytics: The goal here is to foretell what is expected to happen in the future based on the outcomes of analyzed data over time.
• Prescriptive analytics: In prescriptive analytics, the results from data analysis is used to make recommendations on what to do next.

What Is the Difference Between Data Analysis and Data Analytics?

You'll come across different definitions of data analytics and data analysis.

Some sources would define data analytics and data analysis as the same. Others would use them interchangeably.

Although, they are closely related, these terms have slightly different meanings. They are similar because they aid in the decision making process.

What Is Data Analysis?

Data analysis is the process of studying what has happened in the past in a dataset. There is no need to extend this definition.

Data analysis studies the why and how of data trends. Yes, it involves data collection, organization, and "analysis".

"How did the users respond to a new feature?".

"Why did the rate of purchase of a product fall during a particular period?".

Data analysts can make use of programming languages when analyzing data or data visualization tools.

What Is Data Analytics?

Data analytics is the process of taking insights gained from the analysis of past data trends, and making predictions or decisions for the future.

In the beginning of the article, we defined data analytics to include both analysis and analytics. This is mainly as a convention.

Analytics is used to proffer solutions or make recommendations.

Summary

There is data everywhere. We create them on a daily basis. But data in its raw form has no real meaning.

In order to understand the behavior of data over time, we have to group the data together, study them, and derive useful insights.

This article explained what data analytics is, the importance of data analytics, and the types of data analytics.

We also explained the difference between data analysis and data analytics.

Thank you for reading!

The basic difference between the two is that a data scientist works to capture data while a data analyst tries to gain insights from that data.

This article is for you if you’re interested in a career in big data and you don’t know whether you'd want to be a data analyst or data scientist. It will also help you if you just want to know the differences between a data analyst and a data scientist.

What We'll Cover

• What is Data Analytics and Who is a Data Analyst?
  • What does a Data Analyst Do?
  • How to Become a Data Analyst
• What is Data Science and Who is a Data Scientist?
  • What does a Data Scientist Do?
  • How to Become a Data Scientist
• What are the Differences between Data Analyst and Data Scientist?
• Conclusion

What is Data Analytics and Who is a Data Analyst?

Data analytics bridges the gap between data science and business analytics. It is the systematic approach of processing raw data and subsequently extracting meaningful information from it.

The information extracted from the raw data is the focus of data analysis. The professional who does this analysis is a data analyst.

What does a Data Analyst Do?

Data analysts make use of statistical and logical techniques to evaluate data. They use tools such as SQL to query databases and extract the needed information that can help companies make better decisions.

To dig into and assess the information from this data, a data analyst uses programming languages like R, SAS, and Python, and tools like D3, Tableau, and Power BI.

In addition, a data analyst cleans up the database by getting rid of redundant and unusable data.

How to Become a Data Analyst

To become a data analyst, you can earn a relevant degree from an accredited college or university, attend a bootcamp, or learn it yourself.

You can learn to become a data analyst yourself because building a career in a certain field in tech is all about skills. Once you have those skills and you can put them into practical use, then you can become a data analyst.

Some job requirements for data analysts include degrees and some don’t. So there’s room for anyone who doesn’t have a degree but has the skills.

As a data analyst, the skills you need are:

• Soft skills (critical thinking, communication, and others)
• Data visualization (D3, Tableau, Power BI)
• SQL and (probably) NoSQL
• Statistics
• Spreadsheets (Excel, Google Sheets, and others)
• A few programming languages like Python, R, SAS, and JavaScript for D3
• Machine learning

It doesn’t end there. You should try to work on projects that make you appear employable to recruiters. You should also try to get an entry-level job that can help you put those skills into real-world practice. And if you can’t find an entry-level job, then you can consider volunteering.

Here are a few resources you can use to get started:

1. Learn Data Analysis with Python
2. What is Data Analysis? Full Handbook
3. Data Analysis with Python for Excel Users
4. What does a Data Analyst Do?

What is Data Science and Who is a Data Scientist?

Data science is the development of strategies for capturing data and preparing it for analysis. It also involves processing and developing data models with programming languages like R and Python, then deploying those models into applications. The professional who develops these strategies is called a data scientist.

What does a Data Scientist Do?

A data scientist is more focused on developing and implementing tools that help data analysts analyze the data and extract the needed information from it.

This means data scientists spend their time developing models and preparing algorithms. And if the organization needs to deploy a model, data scientists are in charge of that.

How to Become a Data Scientist

Most data science job openings require a relevant degree such as Statistics and Computer Science. But on a personal note, I’ve seen data science openings that don’t require degrees.

Towards the end of this article, I will link an article that shows you where to see those data science job openings.

Once again, what matters is the skills. Once you have those skills and can put them into use, then you can get a job as a data scientist.

Some of the skills you need to become a data scientist are:

• Mathematics
• Programming (Python, R, SAS)
• Statistics
• Linear algebra
• Machine learning
• Cloud computing
• SQL and NoSQL (Most openings won’t require NoSQL but it’s a good skill to learn)
• Apache Hadoop
• Calculus

Here are some resources to get you started:

1. Learn the Basics of Data Science - Hands-On Course
2. Python for Data Science Course
3. Top Statistics Concepts to Know Before Getting Into Data Science
4. Data Science Interview Questions for Beginners
5. Programming, Math, and Science Concepts to Know for Data Science

What are the Differences between Data Analyst and Data Scientist?

Conclusion

Data scientist and data analyst are both in-demand career paths you can follow in big data. If you’re confused about which to take get into between the two, here are some things to consider:

• if you’re well-versed in Mathematics, Statistics, and computer science, either of the two is good for you
• if you want to create advanced machine learning models, you should consider getting into data science
• if you are interested in analytics, you’d probably make a great data analyst.

There’s no black-and-white guide to help you choose between becoming a data scientist and a data analyst. And it's not helpful to say one is better than the other.

In the end, what matters is solving problems and helping humanity learn and improve, not how much a data analyst makes or how much a data scientist makes.

More General Readings

• freeCodeCamp article on how to find remote jobs
• What is Data Science
• What is Data Analytics

Thank you for reading.

Databricks helps you create data apps more quickly. This in turn brings to light valuable insights from your data and helps you create robust Artificial Intelligence solutions.

Azure Databricks also combines the strength of Databricks as an end-to-end Apache Spark platform with the scalability and security of Microsoft's Azure platform.

In this tutorial, you will learn how to get started with the platform in Microsoft Azure and see how to perform data interactions including reading, writing, and analyzing datasets. By the end of this tutorial, you will be able to use Azure Databricks to read multiple file types, both with and without a schema.

Prerequisites

You will need a valid and active Microsoft Azure account.

• Free Azure Trial: With this option, you will start with $100 Azure credit and will have 30 days to use it in addition to free services.
• Azure for Students: This offer is available for students only. With this option, you will start with $100 Azure credit with no credit card required. You'll get access to popular services for free whilst you have your credit.

How to Create Your Databricks Workspace

You must create an Azure Databricks workspace in your Azure subscription before you can utilize Azure Databricks. Go to the Azure portal to do this. As long as you've created a valid and active Microsoft Azure account, this will function.

The Microsoft Azure Home Page

Once there, click the Create a resource button.

On the search prompt in the Create a resource page, search for Azure Databricks and select the Azure Databricks option.

The Microsoft Azure page showing the list of popular resources

Open the Azure Databricks tab and create an instance.

The Azure Databricks pane.

Click the blue Create button (arrow pointed at it) to create an instance.

Then enter the project details before clicking the Review + create button.

The Azure Databricks configuration page

It is important to note that the Subscription option shown above will differ from yours. It will depend on the Azure subscription you have available on your account.

Fill the Workspace name field with a globally unique name. Mine is named salim-freeCodeCamp-databricks1.

Enter the location closest to where you are in the Region option. A region is a set of physical data centers that serve as servers. Since I am based in Lagos, Nigeria, I selected South Africa North.

Select the Standard option which includes Apache Spark with Azure AD in the Pricing Tier option.

With all the configurations set, click the Review + create button. The validation process usually takes about two minutes.

With the validation and deployment processes completed for the workspace, launch the workspace using the Launch Workspace button that appears.

The home page for the created instance of Azure databricks - salim-freeCodeCamp-databricks

Click on the button and you will automatically be signed in using the Azure Directory Single Sign On.

Signing into the workspace of the integration of Microsoft Azure and Databricks

The Microsoft Azure Databricks home page will come up in a new tab as shown below:

The Microsoft Azure Databricks home page

With the workspace launch, create a cluster using the Create a cluster option on the left of the page.

After you have clicked the button and you have created any prior, you will pick one and build on it. Else, you will have to create a new cluster using the Create Cluster button.

Set the configurations for the Azure Databricks cluster

To create the cluster, you have to set the configurations. Choose the Single node option, changing from the Multi node default option, and maintain the other options as default.

Click the Create Cluster button at the bottom of the page. Note that this will take a few minutes and that if the dataset is large, you can explore the Multi node option.

Having created the cluster, import some ready-to-use notebooks by navigating to Workspace > Users > your_account on the left taskbar.

Right-click and select the Import option on the dropdown menu.

With the cluster created, you will then have to import some ready to use notebooks.

To do this, using the left taskbar, you will navigate through Workspace > Users > your_account . Then right-click to see the dropdown menu. You will then select the Import option on the dropdown menu.

The import button will be used to import the dataset to be used

Once you click on the Import button, you will then select the URL option and paste the following URL:

https://github.com/salimcodes/microsoft-learning-paths-databricks-notebooks/blob/master/data-engineering/DBC/03-Reading-and-writing-data-in-Azure-Databricks.dbc

The database folder named 03-Reading-and-writing-data-in-Azure-Databricks.dbc will be used,

You will see he list of files in the 03-Reading-and-writing-data-in-Azure-Databricks.dbc database folder

The image above is what the workspace will like after downloading the file. As such, you have created a Databricks workspace.

How to Read the Data in CSV Format

Open the file named Reading Data - CSV.

Upon opening the file, you will see the notebook shown below:

You will see that the cluster created earlier has not been attached.

On the top left corner, you will change the dropdown which initially shows Detached to your cluster's name. Mine is named Salim Oyinlola's freeCodeCamp Cluster.

The cluster initially created is now attahed to the python notebook

With your cluster attached, you will then run all the cells one after the other.

Running the first cell of the python notebook will initialize the classroom variables & function, mount the dataset and create user-specific database

At its core, the notebook simply reads the data in csv format. Then it adds an option that tells the reader that the data contains a header and to use that header to determine our column names.

You can also add an option that tells the reader to infer each column's data types (also known as a schema).

It is important to note that data can be read in different formats such as JSON (with or without schemas), parquet, and table and views. To achieve this, you can simply run the respective notebooks for each format.

How to Write Data into a Parquet File

Just as there are many ways to read data, there are many ways to write data. But in this notebook, we'll get a quick peek of how to write data back out to Parquet files.

Apache Parquet is a column storage file format that Hadoop systems (such as Spark and Hive) use. The file format is cross-platform, language independent, and it stores data in a column layout using a binary representation.

Parquet files, which effectively store large datasets, have the extension .parquet.

Like what you did when reading data, you will also run the cells one after the other.

The cell to write data into a parquet file

Integral to writing into the parquet file is creating a DataFrame. You will be creating one by running this cell.

This cell shows that the existing files are being overwritten

The .mode"overwrite" method shown below implies that by writing DataFrame to parquet files, you are replacing existing files.

The file has been written and saved in an output location.

At its core, the notebook reads a .tsv file (the same used to read for the .csv file) and writes it back out as a Parquet file.

How to Delete the Azure Databricks Instance (Optional)

Finally, the Azure resources that you created in this tutorial can incur ongoing costs. To avoid such costs, it is important to delete the resource or resource group that contains all those resources. You can do that by using the Azure portal.

• Navigate to the Azure portal.
• Navigate to the resource group that contains your Azure Databricks instance.
• Select Delete resource group.
• Type the name of the resource group in the confirmation text box.
• Select Delete.

Conclusion

In this tutorial, you have learned the basics about reading and writing data in Azure Databricks.

You now understand the basics of Azure Databricks, including what it is, how to install it, how to read CSV and parquet files, and how to read parquet files into the Databricks file system (DBFS) using compression options.

Finally, I share my writings on Twitter if you enjoyed this article and want to see more.

Thank you for reading :)

In machine learning, you need to have a large amount of data in order to achieve strong model performance.

Using a method known as data augmentation, you can create more data for your machine learning project. Data augmentation is a collection of techniques that manage the process of automatically generating high-quality data on top of existing data.

In computer vision applications, augmenting approaches are quite common. If you are working on a computer vision project (like image classification), for instance, you can apply dozens of techniques to each image: shift, modify color intensities, scale, rotate, crop, and more.

If you have a tiny dataset for your ML project or wish to reduce overfitting in your machine learning models, it may be a good idea to apply data augmentation approaches.

“We don’t have better algorithms. We just have more data.”- Peter Norvig

In the field of Natural Language Processing (NLP), the tremendous complexity of language makes it difficult to augment the text. The process of augmenting text data is more challenging and not as straightforward as you might expect.

In this article, you will learn how to use a library called TextAttack to improve data for natural language processing.

What is TextAttack?

TextAttack is a Python framework that was built by the QData team for the purpose of conducting adversarial attacks, adversarial training, and data augmentation in natural language processing.

TextAttack has components that can be utilized independently for a variety of basic natural language processing tasks, including sentence encoding, grammar checking, and word substitution.

TextAttack excels in performing the following three functions:

1. Adversarial attacks (Python: **textattack.Attack**, Bash: **textattack attack**).
2. Data augmentation (Python: **textattack.augmentation.Augmenter**, Bash: **textattack augment**).
3. Model training (Python: **textattack.Trainer**, Bash: **textattack train**).

For this article, we will focus on how to use the TextAttack library for data augmentation.

How to Install TexAttack

To use this library, make sure you have Python 3.6 or above in your environment.

Run the following command to install textAttack:

pip install textattack

Note: Once you have installed TexAttack, you can run it via the Python module or via the command line.

Data Augmentation Techniques for Text Data

The TextAttack library has various augmentation techniques that you can use in your NLP project to add more text data.

Here are some of the techniques that you can apply:

CharSwapAugmenter technique

This technique augments words by swapping characters out for other characters.

from textattack.augmentation import CharSwapAugmenter

text = "I have enjoyed watching that movie, it was amazing."

charswap_aug = CharSwapAugmenter()

charswap_aug.augment(text)

[‘I have enjoyed watching that omvie, it was amazing.’]

The Augmenter has swapped the word “movie” for “omvie”.

DeletionAugmenter technique

This one augments the text by deleting some parts of the text to make new text.

from textattack.augmentation import DeletionAugmenter

text = "I have enjoyed watching that movie, it was amazing."

deletion_aug = DeletionAugmenter()

deletion_aug.augment(text)

[‘I have watching that, it was amazing.’]

This method has removed the word “enjoyed” to create a new augmented text.

EasyDataAugmenter technique

This augments the text with a combination of different methods, such as:

• Randomly swapping the positions of the words in the sentence.
• Randomly removing words from the sentence.
• Randomly inserting a random synonym of a random word at a random location.
• Randomly replacing words with their synonyms.

from textattack.augmentation import EasyDataAugmenter

text = "I was billed twice for the service and this is the second time it has happened"

eda_aug = EasyDataAugmenter()

eda_aug.augment(text)

[‘I was billed twice for the service and this is the second time it has happen’, ‘I was billed twice for the one service and this is the second time it has happened’, ‘I billed twice for the service and this is the second time it has happened’,
‘I was billed twice for the this and service is the second time it has happened’]

As you can see from the augmented texts, it shows different results based on the methods applied. For example in the first augmented text, the last word has been modified from “happened” to “happen”.

WordNetAugmenter technique

This technique can augment the text by replacing it with synonyms from the WordNet thesaurus.

from textattack.augmentation import WordNetAugmenter

text = "I was billed twice for the service and this is the second time it has happened"

wordnet_aug = WordNetAugmenter()

wordnet_aug.augment(text)

[‘I was billed twice for the service and this is the second time it has pass’]

This method has changed the word “happened” to “pass” in order to create a new augmented text.

How to Create Your Own Augmenter

Importing transformations and constraints from textattack.transformations and textattack.constraints allows you to build your own augmenter from the ground up.

The following is an illustration of the use of the WordSwapRandomCharacterDeletion algorithm to produce augmentations of a string:

from textattack.transformations import WordSwapRandomCharacterDeletion
from textattack.transformations import CompositeTransformation
from textattack.augmentation import Augmenter

my_transformation = CompositeTransformation([WordSwapRandomCharacterDeletion()])
augmenter = Augmenter(transformation=my_transformation, transformations_per_example=3)

text = 'Siri became confused when we reused to follow her directions.'

augmenter.augment(text)

[‘Siri became cnfused when we reused to follow her directions.’, ‘Siri became confused when e reused to follow her directions.’, ‘Siri became confused when we reused to follow hr directions.’]

The output shows different augmented texts after implementing the WordSwapRandomCharacterDeletion method. For example, in the first augmented text, the method randomly removes the character “o” in the word “confused”.

Conclusion

In this article, you have learned the significance of data augmentation for your Machine Learning projects. You've also learned how to execute data augmentation for textual data using the TextAttack library.

To the best of my knowledge, these techniques are the most effective approaches available to do the task for your NLP project. Hopefully they’ll be of use to you in your work.

You can also try to use other available augmentation techniques from the TextAttack library such as:

• EmbeddingAugmenter
• CheckListAugmenter
• CLAREAugmenter

If you learned something new or enjoyed reading this article, please share it so that others can see it. Until then, see you in the next post!

You can also find me on Twitter @Davis_McDavid.

And you can read more articles like this here.

That amount of data is generated by activities like browsing the web, using social media sites and streaming platforms, and communicating via instant messaging applications with friends and family.

These are just a few of the online activities we take part in on a daily basis.

And that means data is everywhere.

Because of how present data is in our lives and how it affects and powers most of the things we do nowadays, companies have found ways and strategies to use data to process information, predict future trends, and accelerate their growth and profit.

An example of this is that a company will use data to understand the appropriate audience for its product/service.

Then, it will use data to target that audience with content and advertisements to drive sales.

Behind those strategies are data analysts. Their job is to solve problems using data.

They gather the available data and clean it. Then they process and interpret it and make correlations and relationships between the different data points.

Finally, they make decisions based on the analysis results, which will help solve the problem at hand.

In this article, I will first explain what data analysis is, what data analysts do on a day-to-day basis on their job, as well as the qualifications and technical skills you need to become a data analyst yourself.

Here is what we will cover:

1. What is data analysis?
2. What does a data analyst do on a day to day basis?
   1. Understand business needs
   2. Gather data
   3. Work with relational databases
   4. Organize and clean data
   5. Analyze and identify patterns in data
   6. Visualize and Present Data
   7. Conduct Reports On Data Analysis Results
3. What skills and qualifications do you need to become a data analyst?

What Is Data Analysis?

Data analysis is the process of analyzing and turning raw data into practical insights. By utilizing those insights, businesses and organizations can make more informed decisions.

In more detail, data analysis includes:

• Identifying a problem and then identifying the kind and type of data needed for the analysis. This stage also involves creating various hypotheses.
• Extracting and gathering information from multiple sources.
• Cleaning the extracted data. Data is usually raw and messy, and is not usable in that form. This stage involves fixing typos, correcting errors, filling in missing values, and standardizing the data.
• Analyzing data to confirm the initial hypotheses.
• Interpreting the results to understand the problem at hand and what is going on. This stage involves presenting the findings and results of the analysis to business executives, key stakeholders, and decision-makers clearly and concisely. Data analysts will use the data to tell a compelling story that will determine the future and next steps for the business.

To learn more about data analysis, the data analysis process, and why data analysis is beneficial for businesses, read this article which covers these topics in more detail.

What Does A Data Analyst Do On A Day to Day Basis?

In the following sections, I've listed some of the daily tasks and responsibilities of a data analyst.

These will depend on the company (size and sector) and the team the data analyst is on.

Understand Business Needs

Data analysts need to understand how the business is currently performing and what the current business needs are.

They need to understand what problem the business is trying to solve and what direction it wants to head in the future.

Trying to figure out all those needs will involve working with management and different departments in the company.

It will also involve asking questions about why the data analyst is conducting the analysis in the first place and turning all the questions into hypotheses and actionable tasks.

Gather Data

Data analysts typically find data themselves and collect it from multiple sources.

The method of gathering data will depend on the type of data they are using. The way of extracting data will depend on whether the data is quantitative (numerical) or qualitative (non-numerical).

The data that needs to be collected should be relevant to the problem at hand.

Some of the ways of gathering information are:

• Conducting surveys on user satisfaction
• Viewing customer feedback
• Tracking website visits and social media analytics
• Querying for the most searched keywords
• Checking which ads get clicked on the most

A data analyst is also responsible for improving the existing processes for gathering and collecting data and looking for ways that the process can be streamlined and more effective.

Work With Relational Databases

When looking for, pulling, and storing information, data analysts will mostly query and work with relational databases using SQL (Structured Query Language) - the language used for communicating with relational databases.

In relational databases, data is stored in rows and columns, and all data points have defined and pre-established relationships with one another.

Data analysts will also be involved in developing and improving the systems and structure of the relational database by modeling the data and defining the schema, which is the behind-the-scenes logic in relational databases.

Organize and Clean Data

When data analysts first gather the required data, most often than not, it will not be in a usable state.

Data analysts will improve the quality and format of the data by filling in missing values, removing duplicates, and identifying outliers and errors.

Cleaning is one of the most necessary tasks in the data analyst’s job since the analysis results will depend on it.

If data is not cleaned, there is a very high chance that the results of the analysis will be skewed.

Analyze and Identify Patterns In Data

A big part of a data analyst’s job is to interpret complex data and find patterns.

Data analysts connect the dots between a problem and the data available.

Their job is to predict future trends and translate those predictions into something useful, such as answering questions and coming up with actionable next steps for the company.

Visualize and Present Data

A big part of the job of a data analyst is to visualize data with graphs and charts that will answer the initial questions and problems defined in the early stages of the analysis.

Visualizing data is achieved by developing and maintaining dashboards and using data visualization tools such as Tableau.

Visualizing data is a way of communicating findings and presenting data and valuable business insights to the company, which includes stakeholders, leaders, and executives, in a way that is clear and easily understood by non-technical folks.

The presentation will inform business decisions and play a big part in the company's strategy.

Conduct Reports On Data Analysis Results

Once the data analysis process has come to an end, the data analyst needs to build and write reports that summarize the key findings and insights gathered.

What Skills and Qualifications Do You Need To Become A Data Analyst?

One of the first questions you may have about starting a career in data analysis is whether you need a university degree to enter the field.

Although it is not necessary per se, the majority of companies list a university degree as one of the minimum requirements for the job. And that degree could be in computer science, math, statistics, finance, or business.

So, although you can learn the hard technical skills needed for a data analyst job on your own, having a university degree in one of those disciplines definitely wouldn't hurt.

The next question you may have is, what are the hard skills necessary for the job?

Some of them include:

• Microsoft Excel skills and Google Sheets skills. Excel and Google Sheets are two spreadsheet tools used for extracting insights and figuring out patterns from data.
• Knowledge of SQL. As mentioned earlier, SQL is a database query language and is used for interacting with data stored in relational databases. As a data analyst, you need to know how to use SQL to perform CRUD (Create Read Update Delete) operations on the data stored in relational databases.
• Knowledge of a data visualization tool. A data visualization tool such as Tableau is used for presenting data insights by creating interactive dashboards.
• Knowledge of statistcs. As a data analyst, a big part of the job is working with numbers, so you need to know at least the basics of statistics by taking an intro to statistics course. And some foundational knowledge in math can be of great help too.
• Knowledge of a programming language. Knowing a programming language will help data analysts pull data from various sources, automate repetitive data analysis tasks, and analyze and visualize data to spot patterns to extract meaning.Python is a beginner-friendly programming language and is popular for data analysis. Another language commonly used is R, which was specifically designed for performing statistical analysis. That said, it has a steep learning curve.

Conclusion

Hopefully, you found this guide helpful and got some insight into what the role of a data analyst entails and how you can get started with data analysis.

To learn more, check out freeCodeCamp's data analysis with Python certification. You will learn data analysis concepts using Python and Python libraries such as NumPy and Pandas. In the end, you will also build 5 projects to claim your certification.

Thank you for reading!

There is a large amount of data available, but what do we do with all that data? How is it all used? And what does all that data mean?

It’s not much use if we just collect and store data in a spreadsheet or database and don't look at it, explor it, or research it.

Data analysts use tools and processes to derive meaning from data. They are responsible for collecting, manipulating, investigating, analyzing, gathering insights, and gaining knowledge from it.

This is one of the reasons data analysts are very high in demand: they play an integral role in business and science.

In this article, I will first go over what data analysis means as a term and explain why it is so important.

I will also break down the data analysis process and list some of the necessary skills required for conducting data analysis.

Here is an overview of what we will cover:

1. What is data?
2. What is data analysis?
3. Why is data analysis important?
   1. Effective customer targeting
   2. Measure success and performance
   3. Problem solving
4. An overview of the data analysis process
   1. Step 1: recognising and identifying the questions that need answering
   2. Step 2: collecting raw data
   3. Step 3: cleaning the data
   4. Step 4: analyzing the data
   5. Step 5: sharing the results
5. What skills are required for data analysis?
   1. A good grasp of maths and statistics
   2. Knowledge of SQL and Relational Databases
   3. Knowledge of a programming language
   4. Knowledge of data visualization tools
   5. Knowledge of Excel

What Is Data? Meaning and Definition of Data

Data refers to collections of facts and individual pieces of information.

Data is vital for decision-making, planning, and even telling a story.

There are two broad and general types of data:

• Qualitative data
• Quantitative data

Qualitative data is data expressed in non-numerical characters.

It is expressed as images, videos, text documents, or audio.

This type of data can’t be measured or counted.

It is used to determine how people feel about something – it’s about people's feelings, motivations, opinions, perceptions and involves bias.

It is descriptive and aims to answer questions such as ‘Why’, ‘How’, and ‘What’.

Qualitative data is gathered from observations, surveys, or user interviews.

Quantitative data is expressed in numerical characters.

This type of data is countable, measurable, and comparable.

It is about amounts of numbers and involves things such as quantity and the average of numbers.

It aims to answer questions such as ‘How much, ‘How many’, ‘How often’, ‘and 'How long’.

The act of collecting, analyzing, and interpreting quantitative data is known as performing statistical analysis.

Statistical analysis helps uncover underlying patterns and trends in data.

What Is Data Analysis? A Definition For Beginners

Data analysis is the act of turning raw, messy data into useful insights by cleaning the data up, transforming it, manipulating it, and inspecting it.

The insights gathered from the data are then presented visually in the form of charts, graphs, or dashboards.

The insights discovered can help aid the company’s or organization’s growth. Decision-makers will be able to come to an actionable conclusion and make the right business decisions.

Extracting knowledge from raw data will help the company/organization take steps towards achieving greater customer reach, improving performance, and increasing profit.

At its core, data analysis is about identifying and predicting trends and figuring out patterns, correlations, and relationships in the available data, and finding solutions to complex problems.

Why Is Data Analysis Important?

Data equals knowledge.

This means that data analysis is integral for every business.

It can be useful and greatly beneficial for every department, whether it's administration, accounting, logistics, marketing, design, or engineering, to name a few.

Below I will explain why exploring data and giving data context and meaning is really important.

Data Analysis Improves Customer Targeting

By analyzing data, you understand your competitors, and you will be able to match your product/service to the current market needs.

It also helps you determine the appropriate audience and demographic best suited to your product or service.

This way, you will be able to come up with an effective pricing strategy to make sure that your product/service will be profitable.

You will also be able to create more targeted campaigns and know what methods and forms of advertising and content to use to reach your audience directly and effectively.

Knowing the right audience for your product or service will transform your whole strategy. It will become more customer-oriented and customized to fit customers' needs.

Essentially, with the appropriate information and tools, you will be able to figure out how your product or service can be of value and high quality.

You'll also be able to make sure that your product or service helps solve a problem for your customers.

This is especially important in the product development phases since it cuts down on expenses and saves time.

Data Analysis Measures Success and Performance

By analyzing data, you can measure how well your product/service performs in the market compared to others.

You are able to identify the stronger areas that have seen the most success and desired results. And you will be able to identify weaker areas that are facing problems.

Additionally, you can predict what areas could possibly face problems before the problem actually occurs. This way, you can take action and prevent the problem from happening.

Analyzing data will give you a better idea of what you should focus more on and what you should focus less on going forward.

By creating performance maps, you can then go on to set goals and identify potential opportunities.

Data Analysis Can Aid Problem Solving

By performing data analysis on relevant, correct, and accurate data, you will have a better understanding of the right choices you need to make and how to make more informed and wiser decisions.

Data analysis means having better insights, which helps improve decision-making and leads to solving problems.

All the above will help a business grow.

Not analyzing data, or having insufficient data, could be one of the reasons why your business is not growing.

If that is the case, performing data analysis will help you come up with a more effective strategy for the future.

And if your business is growing, analyzing data will help it grow even further.

It will help reach its full potential and meet different goals – such as boosting customer retention, finding new customers, or providing a smoother and more pleasant customer experience.

An Overview Of The Data Analysis Process

Step 1: Recognising and Identifying The Questions That Need Answering

The first step in the data analysis process is setting a clear objective.

Before setting out to gather a large amount of data, it is important to think of why you are actually performing the data analysis in the first place.

What problem are you trying to solve?

What is the purpose of this data analysis?

What are you trying to do?

What do you want to achieve?

What is the end goal?

What do you want to gain from the analysis?

Why do you even need data analysis?

At this stage, it is paramount to have an insight and understanding of your business goals.

Start by defining the right questions you want to answer and the immediate and long-term business goals.

Identify what is needed for the analysis, what kind of data you would need, what data you want to track and measure, and think of a specific problem you want to solve.

Step 2: Collecting Raw Data

The next step is to identify what type of data you want to collect – whether it will be qualitative (non-numerical, descriptive ) or quantitative (numerical).

The way you go about collecting the data and the sources you gather from will depend on whether it is qualitative or quantitative.

Some of the ways you could collect relevant and suitable data are:

• By viewing the results of user groups, surveys, forms, questionnaires, internal documents, and interviews that have already been conducted in the business.
• By viewing customer reviews and feedback on customer satisfaction.
• By viewing transactions and purchase history records, as well as sales and financial figure reports created by the finance or marketing department of the business.
• By using a customer relationship management system (CRM) in the company.
• By monitoring website and social media activity and monthly visitors.
• By monitoring social media engagement.
• By tracking commonly searched keywords and search queries.
• By checking which ads are regularly clicked on.
• By checking customer conversion rates.
• By checking email open rates.
• By comparing the company’s data to competitors using third-party services.
• By querying a database.
• By gathering data through open data sets using web scraping. Web scraping is the act of extracting and collecting data and content from websites.

Step 3: Cleaning The Data

Once you have gathered the data from multiple sources, it is important to understand the structure of that data.

It is also important to check if you have gathered all the data you needed and if any crucial data is missing.

If you used multiple sources for the data collection, your data will likely be unstructured.

Raw, unstructured data is not usable. Not all data is necessarily good data.

Cleaning data is the most important part of the data analysis process and one on which data analysts spend most of their time.

Data needs to be cleaned, which means correcting errors, polishing, and sorting through the data.

This could include:

• Looking for outliers (values that are unusually big or small).
• Fixing typos.
• Removing errors.
• Removing duplicate data.
• Managing inconsistencies in the format.
• Checking for missing values or correcting incorrect data.
• Checking for inconsistencies
• Getting rid of irrelevant data and data that is not useful or needed for the analysis.

This step will ensure that you are focusing on and analyzing the correct and appropriate data and that your data is high-quality.

If you analyze irrelevant or incorrect data, it will affect the results of your analysis and have a negative impact overall.

So, the accuracy of your end analysis will depend on this step.

Step 4: Analyzing The Data

The next step is to analyze the data based on the questions and objectives from step 1.

There are four different data analysis techniques used, and they depend on the goals and aims of the business:

• Descriptive Analysis: This step is the initial and fundamental step in the analysis process. It provides a summary of the collected data and aims to answer the question: “What happened?”. It goes over the key points in the data and emphasizes what has already taken place.
• Diagnostic Analysis: This step is about using the collected data and trying to understand the cause behind the issue at hand and identify patterns. It aims to answer the question: “Why has this happened?”.
• Predictive Analysis: This step is about detecting and predicting future trends and is important for the future growth of the business. It aims to answer the question: “What is likely to happen in the future?
• Prescriptive Analysis: This step is about gathering all the insights from the three previous steps, making recommendations for the future, and creating an actionable plan. It aims to answer the question: “What needs to be done?”

Step 5: Sharing The Results

The last step is to interpret your findings.

This is usually done by creating reports, charts, graphs, or interactive dashboards using data visualization tools.

All the above will help support the presentation of your findings and the results of your analysis to stakeholders, business executives, and decision-makers.

Data analysts are storytellers, which means having strong communication skills is important.

They need to showcase the findings and present the results in a clear, concise, and straightforward way by taking the data and creating a narrative.

This step will influence decision-making and the future steps of the business.

What Skills Are Required For Data Analysis?

A Good Grasp Of Maths And Statistics

The amount of maths you will use as a data analyst will vary depending on the job. Some jobs may require working with maths more than others.

You don’t necessarily need to be a math wizard, but with that said, having at least a fundamental understanding of math basics can be of great help.

Here are some math courses to get you started:

• College Algebra – Learn College Math Prerequisites with this Free 7-Hour Course
• Precalculus – Learn College Math Prerequisites with this Free 5-Hour Course
• Math for Programmers Course

Data analysts need to have good knowledge of statistics and probability for gathering and analyzing data, figuring out patterns, and drawing conclusions from the data.

To get started, take an intro to statistics course, and then you can move on to more advanced topics:

• Learn College-level Statistics in this free 8-hour course
• If you want to learn Data Science, take a few of these statistics classes

Knowledge of SQL and Relational Databases

Data analysts need to know how to interact with relational databases to extract data.

A database is an electronic storage localization for data. Data can be easily retrieved and searched through.

A relational database is structured in format and all data items stored have pre-defined relationships with each other.

SQL stands for Structured Query Language and is the language used for querying and interacting with relational databases.

By writing SQL queries you can perform CRUD (Create, Read, Update, and Delete) operations on data.

To learn SQL, check out the following resources:

• SQL Commands Cheat Sheet – How to Learn SQL in 10 Minutes
• Learn SQL – Free Relational Database Courses for Beginners
• Relational Database Certification

Knowledge Of A Programming Language

To further organize and manipulate databases, data analysts benefit from knowing a programming language.

Two of the most popular ones used in the data analysis field are Python and R.

Python is a general-purpose programming language, and it is very beginner-friendly thanks to its syntax that resembles the English language. It is also one of the most used technical tools for data analysis.

Python offers a wealth of packages and libraries for data manipulation, such as Pandas and NumPy, as well as for data visualization, such as Matplotlib.

To get started, first see how to go about learning Python as a complete beginner.

Once you understand the fundamentals, you can move on to learning about Pandas, NumPy, and Matplotlib.

Here are some resources to get you started:

• How to Get Started with Pandas in Python – a Beginner's Guide
• The Ultimate Guide to the Pandas Library for Data Science in Python
• The Ultimate Guide to the NumPy Package for Scientific Computing in Python
• Learn NumPy and start doing scientific computing in Python
• How to Analyze Data with Python, Pandas & Numpy - 10 Hour Course
• Matplotlib Course – Learn Python Data Visualization
• Python Data Science – A Free 12-Hour Course for Beginners. Learn Pandas, NumPy, Matplotlib, and More.

R is a language used for statistical analysis and data analysis. That said, it is not as beginner-friendly as Python.

To get started learning it, check out the following courses:

• R Programming Language Explained
• Learn R programming language basics in just 2 hours with this free course on statistical programming

Knowledge of data visualization tools

Data visualization is the graphical interpretation and presentation of data.

This includes creating graphs, charts, interactive dashboards, or maps that can be easily shared with other team members and important stakeholders.

Data visualization tools are essentially used to tell a story with data and drive decision-making.

One of the most popular data visualization tools used is Tableau.

To learn Tableau, check out the following course:

• Tableau for Data Science and Data Visualization - Crash Course

Knowledge of Excel

Excel is one of the most essential tools used in Data analysis.

It is used for storing, structuring, and formatting data, performing calculations, summarizing data and identifying trends, sorting data into categories, and creating reports.

You can also use Excel to create charts and graphs.

To learn how to use Excel, check out the following courses:

• Learn Microsoft Excel - Full Video Course
• Excel Classes Online – 11 Free Excel Training Courses
• Data Analysis with Python for Excel Users Course

Conclusion

This marks the end of the article – thank you so much for making it to the end!

Hopefully this guide was helpful, and it gave you some insight into what data analysis is, why it is important, and what skills you need to enter the field.

Thank you for reading!

Then there are more complex analyses that you'll need to do. At times relationships between two seemingly unrelated data sets can provide surprising insights and unveil important opportunities for the organization.

Data scientists and engineers are continuing to improve how they break down and work on data. Experimentation entails discovering the right correlations among data points.

This means they also need to do some sort of parallelization of such data and resulting models. Parallelization simply means that the same data set is being operated upon in many different ways without damaging the integrity of the original data.

In this article we are going to talk about how you can make sure you're doing such experimentation and parallel processing efficiently and that it provides the maximum insights. We will be tackling different concepts related to data storage and data versioning.

Block Storage vs Object Storage

For the uninitiated, we first must understand the difference between block and object storage and why the latter is the better option when dealing with data experimentation.

_Image source_

What is Block Storage?

It is called “block storage” (also known as SAN) because each dataset (in the form of files) is grouped into blocks stored in disks.

A classic example of block storage is the file system on your personal computer. For enterprise-level use-cases, it is scaled through a network of hard drives connected through fiber optic cables.

There are a few disadvantages to using block storage. First, if a sector (or a block) becomes corrupted, it can damage the files. Another problem is the lack of scalability (expanding the network of fiber optic cables is costly).

What is Object Storage?

In object storage, data is stored as objects. Each object contains the actual data, called the blob, a unique identifier (UUID), and metadata, which contains information about the object (such as timestamp, version, and author).

Object storage makes it cost-effective to scale your data store—you don’t need complex hardware for this. It also makes data retrieval faster as each object can be retrieved through its UUID.

This is in contrast to block storage, where each data location needs to be identified before the actual information can be retrieved.

One disadvantage of using object storage is that data can only be written once and cannot be updated. But this isn’t really a disadvantage as we will see further on in this article.

What Problems Does Object Storage Solve?

As we have already seen, data retrieval can be incredibly fast with object storage (no matter the size of the data store). But when it comes to data experimentation and data parallelization, object storage shines the brightest.

As mentioned before, you can't overwrite any data already stored as an object. This ensures object storage is protected from unwanted (or unauthorized) data destruction or updating. That’s great to know if you do a lot of data processing where accidental corruption of information could happen.

One other problem that object storage can solve is that it doesn’t require data to be structured. As companies produce and consume tremendous amounts of information every moment, often non-structured data (such as PDFs, videos, images) are not so easily processed into useful forms (such as for analytics or dashboards).

With object storage, this is now possible. You can now use non-structured data to develop machine learning models.

With data storage, it’s possible to have different versions of the same blob (with different metadata). As there is Git for code version control, we can have similar ways of managing different versions of the same data.

This brings us to the concept of data lakes.

What are Data Lakes?

Data lakes are central repositories of data that don’t care which format such data is in.

Companies produce and consume tremendous amounts of data. Such data traditionally sits in silos because they belong to different departments or are in different forms (for example, videos aren’t stored in the same directory as the data in the MySQL database).

With data lakes, any department in the enterprise can store information without the need to pre-process it. Likewise, any data can be retrieved and analyzed by anybody from any department.

Data lakes are important because they make data analytics extremely fast and convenient.

How Data Experimentation and Parallelization Work with Object Storage

As with developing software, working with data requires us to utilize tools that can aid us in our workflow. A powerful open source tool for experimenting with data and performing parallelization (that is working on the same data to create different sets of machine learning models) is LakeFS.

LakeFS is an open source platform that provides Git-like capabilities when working with data. This means you can create branches (allowing you to experiment with data) and commit versions of data (and data models).

Why is this Git-like feature important?

First, you need to make sure that your data lake is ACID compliant. This means that your data changes can happen in isolation (in branches). Thus, the integrity of the data is maintained in the master branch (until such changes are ready to be merged).

Another important feature of LakeFS is continuous integration of data (again, much like in software development). Enterprises need to incorporate new data quickly and without being disrupted. Therefore, this ability to have a CI/CD workflow is invaluable.

So, let’s see how we can get started with using LakeFS with our object storage experimentation and parallelization.

How to Install LakeFS

Locally you can install LakeFS by running the following command in your terminal:*

Code source

_*This is assuming you have Docker and Docker-Compose installed in your system. If you don’t have Docker and Docker-Compose, you may try other installation methods here._

Now visit http://127.0.0.1:8000/setup in your browser to verify you have installed it correctly.

How to Create a Repository in LakeFS

Once you’ve verified that LakeFS is installed correctly, go ahead and create an admin user.

Image source

_Image source_

Click on the login link and log in as an administrator.

On the page to which you get redirected, click on Create Repository. A popup will appear:

_Image source_

Congratulations! You now have your first repository. This is the main “bucket” in which you are going to store your data.

Next, we’ll start adding some data.

How to Add Data to your LakeFS Repository

Visit here to install AWS CLI.

With the credentials created during the admin-user creation phase, configure a new connection profile:

Code source

To test if the connection is working, run the following:

Code source

Now, to copy files into the main branch:

Code source

Just note that we need to prefix the path with the name of the branch we want to use.

Now, we will see the file we’ve added in the UI:

_Image source_

Next, we will need to know how to commit and create branches. To do that, we will need to install the LakeFS CLI.

How to Install the LakeFS CLI

You need to first download the binary file here.

Again, we need to use the earlier created admin credentials:

Code source

Here are some of the commands we can run to try it out:

Code source

You can find all the other commands, such as branch creation, and so on, online.

There you have it! Now, you can work with your data any way you like. Experiment without guilt and create multiple versions of your data models.

In Closing

In this article, we covered a bit of ground. We learned the different kinds of data storage mechanisms and why object storage has a lot of edge when dealing with data experimentations and parallelism.

Next, we looked into data lakes and LakeFS, which is a powerful tool for working with data.

At first, it might seem a daunting task. But, as we’ve shown here, with the right set of tools and knowledge, there’s a lot you can accomplish.

Gone are the days of storing your data in a CSV file or an Excel spreadsheet. If you want to quickly analyze millions of data rows in seconds, BigQuery is the way to go.

In this getting started guide, we'll learn about BigQuery and how we can use it to query and analyze data.

What is BigQuery?

BigQuery is an enterprise data warehouse that many companies use who need a fully-managed cloud based solution for their massive datasets.

BigQuery's serverless architecture allows you to quickly execute standard SQL queries and analyze millions of data rows in seconds. You can then store your data both in Google Cloud Storage in files and buckets or in BigQuery storage.

BigQuery also has excellent integrations with other GCP products, like Data Flow and Data Studio that makes it a great choice for data analytics tasks.

Before You Begin:

We are going to query tables in a public dataset that Google has provided to try out BigQuery using the Google Cloud Platform. Therefore, this guide assumes that:

• You have an access on Google Cloud Platform.
• You have already created a Google Cloud project.
• Google sandbox environment is up and running.

How to Access a Public Dataset

A public dataset is available to the general public through the Google Cloud Public Dataset Program. We'll use a Hacker News dataset that contains all stories and comments from Hacker News from its launch in 2006 to present. Let's get started.

Navigate to Hacker News dataset and click the VIEW DATASET button. It will take you to the Google Cloud Platform login screen. Login to the account and it will open the BigQuery Editor window with the dataset.

How the BigQuery Interface Is Organized

BigQuery is structured as a hierarchy with 4 levels:

• Projects: Top-level containers that store the data
• Datasets: Within projects, datasets allow you to organize your data and hold one or more tables of data
• Tables: Within datasets, tables hold actual data.
• Jobs: task performed on data such as running queries, loading data, and exporting data.

Note: Please note that while working with tables, you'll also notice that:

• Tables are broken out by day meaning that you will need to use a wildcard, or * to pull a larger date range.
• There is also an “intraday” table that will give you data for the last 24 hours.

How to Check the Table Schema

Click on the table name. This will allow you to see what columns are in the table, as well as some buttons to perform various operations on the table.

How to Preview the Data

Use the preview button to get a sample of some rows in the table. Don’t do a SELECT * in BigQuery:

How to Query Big Data

SQL statements are used to perform various database tasks, such as querying data, creating tables, and updating databases.

Basic Queries

Basic queries contain the following components:

• SELECT (required): identifies the columns to be included in the query
• FROM (required): the table that contains the columns in the SELECT statement
• WHERE: a condition for filtering records
• ORDER BY: Used to sort the result-set in ascending or descending order.
• GROUP BY: how to aggregate data in the result set

How to Compose a Query in BigQuery

For our first query, let’s find out what are the top 5 domains shared in Hacker News in year 2021 so far (query executed on July 9th 2021).

Click the Compose New query button. It will open the editor tab.

Write your first query as below:

SELECT REGEXP_EXTRACT(url, '//([^/]*)/?') domain, COUNT(*) total
FROM `bigquery-public-data.hacker_news.full`
WHERE url!='' AND EXTRACT(YEAR FROM timestamp)=2021
GROUP BY domain ORDER BY total DESC LIMIT 5

You'll notice that BigQuery debugs your code as you construct it. If the query is valid, then a check mark appears along with the amount of data that the query will process. This helps you determine the cost of running the query.

If the query is invalid, then an exclamation point appears along with an error message.

To run this query, click on the Run button. In a few seconds, you should see results returned from the query:

You can click on the JSON tab if you want the results in JSON format. You'll also find interesting details under the 'Execution details' column.

How to Query Multiple Tables Using a Wildcard Table

Wildcard tables enable you to query multiple tables using concise SQL statements. A wildcard table represents a union of all the tables that match the wildcard expression:

FROMtablename.stories_*``

_TABLE_SUFFIX Pseudo Column

Queries with wildcard tables support the _TABLE_SUFFIX pseudo column in the WHERE clause. To restrict a query so that it scans only a specified set of tables, use the _TABLE_SUFFIX pseudo column in a WHERE clause with a condition that is a constant expression.

Using _TABLE_SUFFIX can greatly reduce the number of bytes scanned, which helps reduce the cost of running your queries.

How to Get Data by Providing a Date Range

WHERE _TABLE_SUFFIX BETWEEN
    FORMAT_DATE(‘%Y%m%d’,DATE_SUB(CURRENT_DATE(), INTERVAL 36 MONTH))
    AND
    FORMAT_DATE(‘%Y%m%d’,DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY))

How to Use UNNEST to Flatten the Date

To convert an ARRAY into a set of rows, also known as "flattening," use the UNNEST operator. UNNEST takes an ARRAY and returns a table with a single row for each element in the ARRAY:

SELECT * FROM UNNEST (['Ambreen', 'Abdul', 'Adam', 'David']) AS names;

How to Save and Share Queries

You can save your queries for later use. There are 3 types of saved queries:

• Private: Private saved queries are visible only to the user who creates them.
• Project-level: Project-level saved queries are visible to members of the predefined BigQuery IAM roles with the required permissions.
• Public: Public saved queries are visible to anyone with a link to the query.

Summary

BigQuery is much more sophisticated than what we explored in this simple tutorial. You can also export Firebase Analytics data to BigQuery, which will let you run sophisticated ad hoc queries against your analytics data.

And with BigQuery ML, you can create and execute machine learning models using standard SQL queries.

If you’re feeling excited and want to learn more about BigQuery, check out the links below.

Resources:

• BigQuery cookbook
• Filtering selected tables using _TABLE_SUFFIX
• BigQuery Tip: The UNNEST Function
• BigQuery UNNEST: How to work with nested data in BigQuery

This isn't just about your career: data has grown far beyond the world of IT professionals. It's getting harder to imagine any significant life events that analytics can't help secure and enhance.

And the good news is that acquiring those basic data skills doesn't have to cost you any money or take much time. There's a lot of sophisticated number crunching going on using plain old spreadsheets, for instance.

But even getting up to speed with the far more versatile world of Python and Jupyter Notebooks can happen faster than you might think.

freeCodeCamp's YouTube channel is host to a quick and dirty course of my own that'll get you started. The rest of the course curriculum that's available (for free) on my website can finish the job.

But why should you bother?

Who needs data analytics?

Operations and business decision making of all kinds depend heavily on understanding data. Professionals in the security, administration, business intelligence, scientific research, insurance, and engineering fields would accomplish precious little without their data insights.

There's a constant torrent of data being automatically generated daily by billions of events executed through millions of devices. Whether the data comes from credit card transactions, environmental sensors installed in cars, or system activity on your laptop, there's no end to the stories it could be telling us.

It's the job of an analyst to capture and manipulate data until it yields useful interpretations. That might mean:

• Identifying and blocking suspicious behavior on distributed IT system infrastructure
• Understanding and fixing performance choke points within complex multi-level business processes
• Discovering drug interactions with biological pathogens
• Detecting patterns and predicting events within financial markets
• Integrating online data with real-world operations (through technologies like augmented reality)

...Or countless other applications, most of which haven't yet been imagined.

How will data fit into your life?

Smart use of publicly-accessible data can help you with important decisions. Searching for the right career? Try crunching government data, like that of the US Bureau of Labor Statistics – something I partially demonstrate here.

Facing some salary negotiation with your boss? You'd be surprised how much helpful geo-specific salary information there is available.

Not sure which way housing availability and prices in your city are likely to move in the next couple of years? You guessed it, there's almost certainly a public data source to answer all your questions.

Sometimes, of course, the data you're after won't exist in exactly the format you need. Rather than give up, you can crowd source the problem.

That, for example, is what's behind my online "Consumer Product Durability" survey. If I'm successful in that particular project, I'll be able to convert the actual experiences of thousands of living, breathing consumers into practical guidance, to help people know when to pay the big bucks for name brands, and when "no-name" is just as good.

The bottom line is that endless streams of data are sitting there, waiting for you to capitalize on them. Whether your goals are personal or professional, if you don't use your data, rest assured, the competition (other people fighting for your job or other organizations fighting for your company's profits) will.

One more thought. You shouldn't think that data analytics is an all or nothing thing: perhaps because – on a basic level – they're so accessible, learning to use some analytics tools can be a powerful addition to your technology portfolio. Think how much more you'll be worth to your employer or business once you can tame their data.

We just released a Data Analytics course on the freeCodeCamp.org YouTube channel. The course includes a 40-minute video, as well as a website and Jupyter notebooks. If you follow the plan laid out in these course resources, you can learn data analytics in 30 days.

David Clinton developed this course. David has written many popular technical books and created many helpful video courses.

The course aims to be a quick and dirty introduction to Python-based data
analytics. The goal is to get users with some basic understanding of the workings of Python to the point where they can confidently find and manipulate data sources and use a Jupyter environment to derive insights from their data. The course will demonstrate effective analytics methods, but does not try to be exhaustive.

The only prerequisite for the course is a basic understanding of Python programming, or at least how programming works in general.

Here are the main topics covered in this course:

• Installing Python and Jupyter
• Working with the Jupyter environment
• Finding data sources and using APIs
• Working with data
• Plotting data
• Understanding data

You can watch the full course below or on the freeCodeCamp.org YouTube channel.

Make sure to also check out the accompanying website: https://stories.thedataproject.net/

Full Transcript

(note: autogenerated)

David Clinton has written and created many popular technical books and video courses.

This data analytics course, along with the accompanying website, and Jupyter Notebooks will help you learn data analytics in 30 days.

Welcome to my course, I'm really glad to have you here.

And I'm even happier that you've decided to join the data analytics party.

Who am I, I'm the author of more than a dozen books on Linux and AWS administration, digital security, and dozens of courses on Pluralsight.

I've also got a fistful of articles right here on the Free Code Camp news site.

But I just write about stuff.

Hopefully, when you're done with this content, you'll be out using data to change the world.

Since you've already seen my claim that this will only take you 30 days, I should explain what this actually is going to show you the tools you'll need to find and manipulate raw data, and use various graphing tools to help you understand and interpret it.

But don't expect us to cover a full data science curriculum here complete with single and multivariable calculus, algorithmic problem solving, or even machine learning.

That would require a whole lot more time and effort.

If that's what you're after, check out the new data science content that Free Code Camp is in the process of bringing online, there's something else you won't get from these videos experience.

Once you've watched the entire course, you probably still won't be able to do much on your own.

The value of actual hands on experience is the mistakes you make, you know, Miss typing syntax, not properly understanding what your code is doing, or failing to account for Environment specific restrictions.

diagnosing and working through those mistakes is where you'll really begin to take charge and accomplish great things.

So where will you get that experience? If you're ambitious, and you've got exciting project ideas of your own, by all means, dive in and try it out.

But if you think you still need some guidance, then I've got everything you should need on my stories dot the data project dotnet site, you can work through the exercises in each of eight data stories that you'll find over there.

If there's a specific skill you're looking for the learning objectives index down here will point you directly to the chapter where you'll find it all that's available for anyone and for free.

If you happen to prefer working with a real book, you can purchase the same content in that format.

But don't think anything's missing from the free website. But right now,let's talk about data analytics tools.

There are many ways to consume data, the one you choose will reflect your specific needs, and your comfort with various skills.

spreadsheets, as you probably already know, are much more than just fancy calculators or places to keep your household budget numbers.

They also come with powerful functions, external integrations and graphing capabilities.

Enterprise strength tools like Tableau Splunk, or Microsoft's Power BI, are also great for crunching numbers and visualizing insights, which you can then share with your team members.

So then what's the big deal with Python? Well, the Python ecosystem is much, much broader than those purpose build tools.

And the Python community makes all kinds of useful data specific libraries and modules available.

When you let Python loose against your data.

You've got all the resources of a full bore industrial strength programming language at your fingertips.

It's not what you can do with it.

That's the challenge.

It's finding something that you can't do.

OK, but what about Jupiter? Jupiter is an open source platform within which you can load your data and execute your Python code.

It's a lot like a programming Id like Microsoft Visual Studio.

And while Jupyter Notebooks can be used with a growing number of languages, and for as many tasks as you can imagine, it's best known and loved as a host for Python data heroics.

Once upon a time, the lines of code you'd write for your programs would be saved to a single text file whose name ended with a dot p y suffix.

When you want it to run your code to see how things went.

You do it from the command line or a powerful and complicated source code editor like Visual Studio.

But for anything to work, it all had to work.

That would make it harder to troubleshoot when something didn't go according to spec.

But it would also make it a lot harder to play around with specific details just to see what happens.

And it also made it tough to share live versions of your code across the internet.

As we'll soon see, Jupyter Notebooks let you run your code a single line at a time, or altogether.

That flexibility makes it easier to understand your code and when things go wrong to troubleshoot it.

notebooks, by the way, are JSON based files that effectively move the processing environment for just about any data oriented programming code from your server or workstation to your web browser.

You can download Jupiter to your PC or a private server and access the interface through any browser with network access.

Or you can run notebooks on third party hosting services like Google's core laboratory, or for a cost cloud providers like Amazon's Sage maker studio notebooks, or Microsoft's as your notebook.

Jupiter comes in three flavors.

The two you're most likely to encounter are classic notebooks and the newer Jupiter lab.

Both run nicely within your browser.

But Jupiter lab comes with more extensions and lets you work with multiple notebook files and terminal access within a single browser tab.

I'll be using the classic notebook environment for the demos in this course, but there's usually no problem transferring notebooks between versions.

The third labor just to be complete is Jupiter hub.

A server version built to provide authenticated notebook access to multiple users, you can serve up to 100 or so users from a single cloud server using the littlest Jupiter hub.

For larger deployments involving clusters of servers, it probably be better off with a Kubernetes version known as zero to Jupiter hub with Kubernetes.

But all that's way beyond the scope of this course.

Our next job is to build our work environment.

Assuming you've decided to host Jupiter on your own machine, you'll need Python installed.

The good news is that most operating systems come with Python pre installed, you can confirm that you've got an up to date version of Python by opening a command prompt and typing Python dash dash version or sometimes Python three dash dash version, a pythons installed, you'll probably see something like this.

Just make sure you've got Python three installed and not the deprecated and insecure Python two.

If you do need to install Python manually, you're best off using pythons official documentation that will be the most complete and up to date source available that will work with whatever operating system you're on.

It's important to note that not all Python versions even those from three point x will necessarily behave quite the way you expect.

You may for instance, find that you need a library written for version 3.9.

But that there's no way to get it working on your 3.7 system.

Upgrading your system version to 3.9 might work out well for you.

But it could also cause some unexpected and unpleasant consequences.

It's hard to know when a particular Python library might also be needed by your core operating system.

If you pull the original version of the library, he might end up disabling the OS itself.

And don't think it won't happen.

I crippled my own O's that way just a few months ago.

One solution is to run Python for your project within a special virtual environment that's isolated from your larger o 's.

That way, you can install all the libraries and versions you like without having to worry about damaging your work system.

You can do this using a full scale virtual container running a Docker or as I prefer LX D image or on a standalone AWS cloud instance.

But you can also use pythons own v n module, you want to read the official documentation for the virtual environment instructions specific to your host OS.

Whichever version of Jupiter you choose, if you decide to install and run it locally, the Jupiter project officially recommends doing it through the Python Anaconda distribution and its binary package manager conda.

Various guides to doing that are available for various OS hosts, but this official page is a good place to start.

As you can see though, the Python PIP package manager is also an option.

Once all that's done, you should be able to open a notebook right in your browser and get right down to work.

For me, a notebooks most powerful feature is the way you can run subsets of your code within individual cells.

This makes it easier to break down long and complex programs into easily readable and executable snippets.

With a cell selected.

Clicking the Run button will execute just that cells code.

Note how the box on the left gets a number representing the sequential position of the execution.

As you become more familiar with Jupiter, you'll probably get in the habit of executing cells using Ctrl and enter rather than a mouse click, you can insert a new cell right after the one that's currently selected by clicking the plus button.

the up and down arrows move cells as you might expect, up and down.

cells, by default are formatted to handle code, Python three in my case, but they can also be set for markdown which can be handy for documenting your notebooks, or making new sections easier to find.

A single hashtag in markdown, for instance, represents a top level selection title, executing the cell will print the text to match your formatting instruction.

The precise locations and appearance of the buttons you'll use to get this stuff done will vary between the different Jupiter versions, but all the basic functions are universally available.

Whatever values your code creates, will remain in the kernel memory until the output from a particular cell or for the entire kernel are cleared.

This lets you rerun previous or subsequent cells to see what impact the change might have.

It also means resetting your environment for a complete start over is as easy as selecting restart kernel and clear all outputs.

Not all Python functionality will be available out of the box.

Sometimes, as you saw just before, you'll need to tell Python to load one or more modules through declarations within your code.

But some modules need to be installed manually from the host command line before they can even be imported.

For such cases, Python recommends their installer program Pip, or in some cases, the conda tool from the Anaconda distribution.

You can read more about using PIP for the proper care and feeding of your Python system within the helpful Python documentation site.

Okay, here's where we get down to real work.

We're going to head out to the internet to find reliable data that will help us answer a real world question.

And we'll use a public API to get the data.

Then we'll examine the data to get a feel for its current formatting, and what it'll take to fix it.

After applying the necessary formatting, so our code can happily read it will merge multiple data sets together so we can look for correlations, and then experiment with graphing tools to find the one that represents our data in the most intelligible way.

What's the problem we're trying to solve? I'm curious to see whether wages paid to us workers over the past 20 years have on average, gone up, assuming they have increased, I'd also like to know whether the extra money in their pockets has also increased their actual standard of living? To answer those questions, we're going to access two data sets collected and maintained by the US government's Bureau of Labor Statistics.

One of the many nice things about the Bureau of Labor Statistics usually referred to as BLS, is that they provide an API for access from within our Python scripts.

To make that work, you'll need to know the BLS endpoint address matching the specific data series you need the Python code to initiate the request, and for higher volume requests a BLS API key.

Getting the series endpoint addresses you need may take some digging around in the BLS website.

However, the most popular data sets are accessible through a single page.

This shows you what that looks like, including the endpoint codes like lns, one one and a whole lot of zeros for the civilian labor force said, You can also search for data sets on this page.

Searching for a computer for instance, will take you to a list that includes the deeply tempting average hourly wage for level 11 computer and mathematical occupations in Austin, Round Rock, Texas.

The information you'll discover by expanding that selection will include its series ID, which is the end point because I know you can barely contain your curiosity.

I'll click through and show you that it turns out that level 11 computer in mathematical professions in Austin, Round Rock Texas, could expect to earn $51.76 an hour back in 2019.

So how do you turn those series IDs into Python friendly data? manually writing get and put requests can be very picky, and it'll take a lot of tries before you get it exactly right.

To avoid all that, I decided to go with a third party Python library called BLS.

that's available through all of our share roses GitHub repo, you install the library on your host machine, using pip install BLS.

That's all it'll take.

While we're here, we might as well activate our BLS API key.

You register for the API From this page, and they'll send you an email with your key and a validation URL that you'll need to click.

Once you've got your key, you export it to your system environment on Linux, or Mac OS, that would mean running something like this, where your key is substituted for the fake one I'm using here, I'm going to use the API to request us consumer price index, CPI, and wage and salary statistics between 2002 and 2020.

The CPI is a measure of the price of a basket of essential consumer goods.

It's an important proxy for changes in the cost of living, which in turn, is an indicator of the general health of the economy.

Our wages data will come from the BLS employment cost index, covering wages and salaries for private industry workers in all industries and occupations.

A growing employment index would at first glance, suggests that things are getting better for most people.

However, seeing the average employment wage trends in isolation isn't all that useful.

After all, the highest salary won't do you much good if your basic expenses are higher still.

So the goal is to pull both the CPI and wages data sets, and then correlate them looking for patterns.

This will show us how wages have been changing in relation to costs.

Now, let me show you how it actually works.

With valid endpoints for the two data sets we are going to be using, we're all set to start digging for CPI, and employment gold.

Importing these four libraries, including BLS will give us all the tools we'll need.

pandas stands for Python for data analysis, which is a library for working with data as data frames.

Data frames are perhaps the most important structure you'll use as you learn to process large data sets.

NumPy is a library for executing mathematical functions against large arrays of data.

And map plot live is a library for plotting data in visual graphs of various kinds.

when importing a library, you assign it the name you'll use to invoke it.

You can choose just about anything but pandas is often represented by PD NumPy as NP and matplotlib as PLT, I'm also importing the BLS library that we installed a bit earlier, that will be invoked using its actual name, BLS.

I'll execute that cell.

Now I pass the BLS endpoint for the CPI data series to the BLS get series command from the BLS library.

The endpoint code itself was of course, copied from the popular data sets page on the BLS website, I'll assign the data series that comes back to a data frame using the variable CPI, and then save the data frame to a local CSV file.

This isn't necessary, but you might find it easier to work with the data when it's saved locally.

Next, I'll load the data from the new CSV file using the pan das PD read CSV command against the file name.

I'll assign the variable name CPI data to the new data frame that comes out the other end.

Running just CPI data will print out the first and last five lines of the data frame.

The date column contains month and year values.

And the second column contains our actual data.

I'd like to simplify the headers to make them easier to work with.

So I'll use the pan das columns attribute I definitely prefer this way.

However, we'll need to also see the wages data to know whether the formatted uses is compatible with our CPI set.

So I'll pull the wages data series using the BLS library and assign it to the wages data frame.

Once again, I'll save it to a local CSV file and read that data into a new data frame called df.

I'll clean up my column headers and use head to print only the first five lines of data, you should notice two things.

This data isn't delivered in monthly increments, but quarterly, but only one entry for every three months.

And the date format is different.

Instead of a month number, there's q1 or q2.

If you want Python to sync between our two data sets, we'll need to do some editing.

I'll do that by replacing every March data point meaning any date entry containing the string dash 03 within its date value with the string q one June that is a string including dash 06 we'll get q2 and so on.

As you can see now when I print just the date column, some values have been updated to the new format.

But the rest of them are from this point unnecessary and will cause us trouble So we'll have to get rid of them altogether.

I'll do that by creating a new data frame called New CPI and reading into it the contents of the old CPI data data frame.

But I'll use the pan das string dot contains function to identify all the rows in the data frame that contain a dash.

And by specifying false, dropping them, we'll be left with only properly formatted quarterly data points.

And I said, I'll save this data frame to a CSV file to notice how we've dropped from 232 rows to just 77.

Just because I'm paranoid, I'll create a new data frame called New df.

So the old df data frame will still be available to me, should I accidentally make a mess with what we're about to do next, with our data all neatened up, we're ready to begin our analysis.

We've got a big problem here, the data in the CPI set comes in absolute point values.

While the wages are reported in percentages measuring growth, as is there's no way to accurately compare them.

For one thing, each row of our wages data is the percentage by which wages whatever is in that quarter had the current rate continued for a full 12 months.

So not only do those values not correspond to the absolute CPI price data, they're not even technically true of their own timeframe.

So when we're told that the rate for the first quarter of 2002 was 3.5%.

That means that if wages continued to rise at the current first quarter rate, for a full 12 months, the annual average growth would have been 3.5%, but not 14%, which means the numbers we're going to work with will have to be adjusted.

That's because the actual growth during say the three months of 2002, quarter one wasn't 3.5%, but only a quarter of that, or 0.8 75%.

If I don't make this adjustment, but continue to map quarterly growth numbers to quarterly CPI prices that are calculated output would lead us to think that wages are growing so fast, that they become detached from reality.

Now, I should warn you that solving this compatibility problem will require some fake math, I'm going to divide each quarterly growth rate by four or in other words, I'll pretend that the real changes to wages during those three months were exactly one quarter of the reported year over year rate.

I'm sure that's almost certainly not true.

And it's a gross simplification.

However, for the big historical picture I'm trying to draw here, it's probably close enough.

Now that will still leave us with a number that's a percentage, but the corresponding CPI number we're comparing it to is again a point figure.

To solve this problem.

I'll apply one more piece of fakery.

To convert those percentages to match the CPI values, I'm going to create a function, I'll feed the function the starting 2002 first quarter CPI value of 170 7.1.

That'll be my baseline.

I'll give that variable the name new num.

For each iteration, the function will make through the rows of my wages data series, I'll divide the current wage value x by 400.

Where'd I get that number 100 simply converts the percentage to 3.5, etc, to a decimal 0.035.

And the four will reduce the annual or 12 month rate to a quarterly rate covering three months to convert that to a usable number a multiply it by the current value of new num, and then add new num to the product.

That should give us an approximation of the original CPI value adjusted by the related wage growth percentage.

But of course, this won't be a number that has any direct equivalent in the real world.

Instead, it is, as I said, an arbitrary approximation of what that number might have been.

But again, I think it'll be close enough for our purposes.

Take a moment to read through the function.

Global new num declares a variable as global.

This makes it possible for me to replace the original value of new num with the functions output, so the percentage in the next row will be adjusted by the updated value.

Note also how any strings will be ignored.

And finally, Note how the updated data series will populate the new wages data variable.

Let's check that new data looks great.

Our next task will be to merge our two data frames and then plot their data.

Don't go away.

What's left, we need to merge our two data series so Python can compare them.

But since we've already done all the cleaning up and manipulation, this will go smoothly.

I'll create a new data frame called merge data and feed it with the URL Put up this PD merge function, I simply supply the names of my two data frames and specify that the date column should be the index.

That wasn't hard.

Let's take a look.

Our data is all there, we can visually scan through the CPI and wages columns and look for any unusual relationships.

But that defeats the point.

Python data analytics is all about letting our code do that for us.

Let's plot the thing.

Here, we'll tell plot to take our merge data frame merge data and create a bar chart.

Because there's an awful lot of data here, I'll extend the size of the chart with a manual fixed size value, I set the x axis to use values in the date column.

And again, because there are so many of them, I'll rotate the labels by 45 degrees to make them more readable.

Finally, I'll set the labels for the x and y axes.

This is what comes out the other end, because of the crowding, it's not especially easy to read.

But you can see that the orange wages bars are for the most part higher than the blue CPI bars.

That means that wages are experiencing a higher growth rate than the CPI, we'll have a stab at analyzing some of this a bit later, is there an easier way to display all this data, you bet there is I can change the value of kind from bar to line, and things will instantly improve.

Here's how the new code will work as a line plot and with the grid.

Python, along with its associated libraries, gives us the ability to use a much wider variety of plotting tools than just bar and line graphs.

We're going to explore just two of them here, scatter plots, and histograms.

We'll also talk a bit about how regression lines work, and what kinds of insights they can show us.

We'll begin with scatter plots.

This code is from the property rights and economic development chapter on my Teach Yourself data analytics website, you can catch up on the background over there.

But the code you're looking at comes from two data sources, the World Bank's measure of per capita gross domestic product by country and the index of economic freedom data from the heritage.o

rg site, I merged data from the two data frames into this one called merge data, I'll create a simple scatterplot.

With this one line command, we can clearly see a pattern, the higher the per capita gross domestic product, meaning the more economic activity a country generates, the further to the right on the x axis, a country's dot is likely to fall.

And the further to the right, the higher is the economic freedom score.

Of course, there are anomalies in our data.

There are countries whose position appears way out of range of all the others, it'd be nice if we could somehow see which countries those are, and would also be nice if we could quantify the precise statistical relationship between our two values, rather than having to visually guess.

We'll begin by visualizing those anomalies in our data.

To make this happen.

all important another couple of libraries that are part of the plotly family of tools, you may need to manually install them on your host using pip install plotly.

Before those will work.

From there, we can run p x scatter, and point it to our merged data data frame associating the score column with the x axis and value with the y axis.

So we'll be able to hover over a dot and see the data it represents.

We'll add the hover data argument and tell it to include country and score data.

This time when you run the code, you get the same nice plot.

But if you hover your mouse over any dot, you'll also see its data values.

In this example, you can see that the tiny but rich country of Luxembourg has an economic freedom score of 75.9

and a per capita GDP of more than $121,000.

You can similarly pick out other countries at either end of the chart, we can learn more about the statistical relationship between our values by adding a regression line, a measure of the data is R squared value.

We already saw how our plot showed a visible trend up and to the right.

But we also saw there were outliers.

Can we be confident that the outliers are the exceptions, and that the overall relationship between our two data sources is sound.

There's only so much we can assume based on visually viewing the graph at some point, we'll need hard numbers to describe what we're looking at.

A simple linear regression analysis can give us a measure of the strength of the relationship between a dependent variable and the data model.

r squared is a number between zero and 100%.

Where 100% would indicate a perfect fit.

Of course, in the real world, a 100% fit is next to impossible.

You'll judge The accuracy of your model or assumption within the context of the data you're working with, how can you add a regression line to a panda's chart? There are, as always, many ways to go about it, I like simple and the O LS trend line approach is about as simple as it gets, just add a trendline argument to the code we've already been using.

That's it.

Oh LS By the way, stands for ordinary least squares, which is a, which is a type of linear regression.

And here's how it looked with our regression line.

When I hover over the line, I'm showing an R squared value of 0.550451.

Or in other words around 35%.

For our purposes, I consider that a pretty strong correlation.

A histogram is a plotting tool that breaks your data down into bins. A bin is actually an approximation of a statistically appropriate interval between sets of your data bins attempt to guess at the probability density function PDF, that will best represent the values you're actually using.

But they may not display exactly the way you think, especially when you use a default value.

I'll illustrate how this works or actually how it doesn't work.

Using data from the do birthdays make elite athletes chapter on the website.

As you can see over there, I'd scraped the semi official NHL API for the birth dates of around 1100 current NHL players.

My goal was to visualize the distribution of their birth dates across all 12 months to see if their births were concentrated within a specific yearly season.

When I display the data using a histogram, we didn't see the pattern we'd expected.

In fact, the pattern wasn't truly representative of the real world.

That's because histograms are great for showing frequency distributions by grouping data points together into bins.

This can help us quickly visualize the state of a very large data set where granular precision will get in the way, but it can be misleading for use cases like ours.

Since we were looking for a literal mapping of events to calendar dates.

Even setting the bin amount to 12 to match the number of months won't help, because a histogram won't necessarily stick to those exact borders.

What we really need here is a plain old bar graph that incorporates our value counts numbers, I'll pipe the results of value counts to a data frame called df one, and then plot that as a simple bar graph.

In the next module, we're going to talk about understanding our data visualizations, and integrating what we see in our Jupyter Notebooks with stuff that happens out there.

In the real world.

Stay tuned.

We're supposed to be doing data analytics here, such as staring at pretty graphs probably isn't the whole point.

The CPI and wages data sets we plotted in the previous chapter, for instance, showed us a clear general correlation.

But there were some visually recognizable anomalies.

Unless we can connect those anomalies with historical events, and explain them in a historical context, we won't be getting the full value from our data.

But even before going there, we should confirm that our plots actually make sense in the context of their data sources.

Working with our BLS examples, let's look at graphs to compare CPI and wages data from both before and after our manipulation.

That way, we can be sure that our math and particularly our fake math didn't skew things too badly.

Here's what our CPI data look like when plotted using the raw data.

It's certainly a busy graph, but you can clearly see the gentle upward slope punctuated by a handful of sudden jumps.

Next, we'll see that same data after removing three out of every four months data points, the same ups and downs are still visible.

Given our overall goals.

I'd categorize our transformation as a success.

Now, how about the wages data here, because we move from percentages to currency, the transformation was more intrusive, and the risks of misrepresentation were greater.

We'll also need to take into account the way a percentage will display differently from an absolute value.

Here's the original data.

Note how there's no consistent curve, either upwards or downwards.

That's because we're measuring the rate of growth as it took place within each individual quarter, not the growth itself.

Now compare that with this line graph of that wage data now converted to currency based values.

The gentle curve, you see makes some sense, it's about real growth after all, not growth rates, but it's also possible to recognize a few spots where the curve steepens and others where it smooths out a bit more.

Why are the slopes so smooth in comparison with the percentage based data? Look at the y axis labels.

The index graph is measured in points between 180 and 280.

While the percentage graph goes from zero to 3.5.

It's the scale that's different.

All in all, I believe we're safe concluding that what we produced is a good match with our source data.

establishing some kind of historical context for your data will require looking for anomalies and associating them with known historical events.

That's something I do at length in the wages and CPI Reality Check chapter on the website.

If you're interested, I'm sure you'll work through that material on your own.

But I think you've seen enough here to get a picture of how plotting the right visualization can be helpful.

But that brings us to the end of this particular course, as I've mentioned, a number of times already, the full curriculum is available on my the data project dotnet site, and you're more than welcome to join all the cool kids over there and be in touch if you've got something to add to the conversation.

The main thing is to realize that the end of this course isn't anywhere near the end of your data analytics education.

watching me calmly execute nice, clean code samples isn't really learning.

Unless you're a very special breed of genius, you won't begin to understand how all this really works.

Until you dive in and work things through yourself.

I say worth things through.

But what I really mean is not worth things through because it's mistakes and frustration that are the best teachers.

Don't imagine that my Python code just came to exist on a quiet afternoon.

While I was sipping nice hot coffee.

First of all, I don't drink coffee.

But more to the point, there was nothing quiet about it.

There were humiliating failures, reformulations start overs and countless trips to stack overflow before things began to take shape.

But the more problems I faced and overcame, the deeper the process sank into my mind, and the better I got at it.

And so will you just be prepared for tough times ahead.

Before you all run off and get on with your day, let's spend a moment or two reviewing everything we saw here.

We spoke about the many ways you can work with Jupyter notebooks, including online platforms like Google's Collaboratory, and locally hosting either Jupiter lab or plastic notebooks within introduced ourselves to the Jupiter environment, learning about cells kernels and the operating environment.

We saw how we can find data through public API's and how to integrate API credentials into our Python environment.

Python libraries and modules were our next focus, including how to import appropriate libraries to allow us to effectively clean and manipulate our data.

And finally, turning to some actual data analytics.

We learned some basics of plotting, including working with scatter plots, regression lines and histograms.

And we closed out the chorus with a quick discussion of how to use our data visualizations to integrate our insights with the real world.

I hope this has been helpful for you and I invite you to check out some of my other content on my main website.

Take care.