If you’re looking for a simpler, open-source, and privacy-friendly solution, Umami is a great alternative. It’s lightweight, easy to deploy, and doesn’t track personal data, making it compliant with modern privacy laws like GDPR.

In this article, you’ll learn what Umami is, why it’s an excellent Google Analytics alternative, and how to set it up on your own server from scratch using Sevalla.

What We’ll Cover:

1. Understanding Umami

2. Why Choose Umami Over Google Analytics

3. How to Install Umami

   • Step 1: Get the Source Code

   • Step 2: Configure the Database

   • Step 3: Build the Application

   • Step 4: Start the Server

   • Step 5: Keeping Umami Updated

   • Step 6: Adding Tracking to Your Website

   • Step 7: Exploring the Dashboard

4. Hosting Umami on the Cloud using Sevalla

5. Privacy and Compliance

6. Conclusion

Understanding Umami

Umami is an open-source web analytics platform designed to be fast, simple, and privacy-focused.

It collects essential website data like page views, referrals, and device information without storing personally identifiable details. Unlike Google Analytics, Umami doesn’t use cookies or share data with third parties.

The project is actively maintained by the open-source community and has grown into one of the most trusted tools for developers and businesses who want full control over their analytics. It provides a clean dashboard that shows all the key metrics in real time and works across any website or application.

You can find the project on GitHub at github.com/umami-software/umami and even try a live demo here.

Why Choose Umami Over Google Analytics

Google Analytics is powerful but often overwhelming for simple websites. It’s also tied to Google’s data collection ecosystem, which can conflict with privacy-focused organizations.

Umami takes a different approach. It collects only the information you need to make decisions, such as traffic sources and popular pages, and it stores everything on your own infrastructure.

There are no third-party cookies, no user tracking, and no hidden integrations. You get complete ownership of your data and peace of mind knowing that it’s not leaving your server.

Plus, Umami is free under the MIT license, making it suitable for both personal projects and commercial deployments.

How to Install Umami

Before you begin, make sure you have a few basic tools and requirements ready.

You’ll need a server with Node.js version 18.18 or newer installed. Umami also requires a database to store analytics data. It supports PostgreSQL (version 12.14 or higher), MySQL (version 8.0 or higher), and MariaDB (version 10.5 or higher).

Step 1: Get the Source Code

The first step is to download the Umami source code from GitHub. Open your terminal and run:

git clone https://github.com/umami-software/umami.git

cd umami

pnpm install

The pnpm install command installs all the necessary dependencies for the application. Make sure you have pnpm installed globally before running this command. You can install it by running npm install -g pnpm.

Step 2: Configure the Database

Next, you need to configure a database connection. Create a new .env file in the root directory of the Umami project. Inside this file, add the following line:

DATABASE_URL=connection-url

Replace connection-url with your actual database connection string. Here are two examples depending on your database type:

For PostgreSQL:

postgresql://username:password@localhost:5432/umami

For MySQL:

mysql://username:password@localhost:3306/umami

This connection string allows Umami to connect to your database and automatically create the necessary tables during the setup.

Step 3: Build the Application

Once your configuration is complete, you can build the application by running:

pnpm run build

This step compiles the code and prepares it for production. It will also initialize your database with the required tables and create a default admin account.

You can log in with the username admin and password umami after setup. It’s a good idea to change this password immediately once you log in for the first time.

Step 4: Start the Server

Now it’s time to start the application. Run the following command:

pnpm run start

By default, Umami will start on http://localhost:3000. You can open this address in your browser to access the analytics dashboard. If you want to make it accessible publicly, you’ll need to configure a reverse proxy using a web server like nginx.

Step 5: Keeping Umami Updated

Like any software, Umami receives regular updates that include new features, security patches, and performance improvements. Keeping your installation up to date is simple.

If you installed from source, navigate to your Umami folder and run:

git pull

pnpm install

pnpm run build

This command updates the source code, installs new dependencies, and rebuilds the app. If you are using Docker, you can update by pulling the latest images and restarting the containers:

docker compose pull

docker compose up — force-recreate -d

Regularly updating ensures you have access to the latest analytics features and bug fixes.

Step 6: Adding Tracking to Your Website

After you log in to the dashboard, you’ll see an option to add a new website. Once you create it, Umami will generate a small tracking script.

Copy the script tag and paste it into the <head> section of your website’s HTML pages.

This script is lightweight and won’t slow down your website. Once added, you’ll start seeing traffic data in your dashboard almost instantly.

You can track multiple websites from the same Umami installation, making it ideal for developers managing several projects.

Step 7: Exploring the Dashboard

The Umami dashboard is clean, modern, and easy to understand. It shows metrics such as page views, referrers, operating systems, and devices. You can filter by date, view live visitors, and export data for reporting.

There are no complicated configuration options or hidden features  –  just the information you need to make informed decisions about your website traffic. Everything runs fast, even on modest servers.

Hosting Umami on the Cloud using Sevalla

When you are ready to move beyond testing, Umami gives you two options. You can self-host it using your own infrastructure or use their managed cloud version at Umami.is.

Self-hosting gives you full control and is usually preferred by technical teams who want to keep sensitive data in-house.

You can choose any cloud provider, like AWS, DigitalOcean, or others to set up Umami. But I will be using Sevalla.

Sevalla is a PaaS provider designed for developers and dev teams shipping features and updates constantly in the most efficient way. It offers application hosting, database, object storage, and static site hosting for your projects.

I am using Sevalla for two reasons:

• Every platform will charge you for creating a cloud resource. Sevalla comes with a $50 credit for us to use, so we won’t incur any costs for this example.

• Sevalla has a template for Umami, so it simplifies the manual installation and setup for each resource you will need for installation.

Log in to Sevalla and click on Templates. You can see Umami as one of the templates.

Click on the “Umami” template. You will see the resources needed to provision the application like PostgreSQL and Redis. Click on “Deploy Template”

You can see the resource being provisioned. Once the resources are provisioned, go to your Umami application and click on “Visit app”

You will get a cloud URL with a login page. Use the default login credentials admin for username and umami for password. You will see the empty dashboard.

You now have a production-grade Umami server running on the cloud. You can use this to setup analytics for your website by clicking on “Settings” and then “Add website”.

You can then click “Edit” to get the tracking code for your website.

Once you add the tracking code for your website, you can start monitoring your traffic and other analytics in your new dashboard.

Privacy and Compliance

One of the best reasons to use Umami is its commitment to privacy. It doesn’t use cookies, doesn’t track individual users, and doesn’t share data with any third-party service.

All information stays on your server. This makes it a great choice for websites that need to comply with privacy laws like GDPR, CCPA, or PECR.

Since you own the data, you can decide how long to keep it, how to analyze it, and who has access.

Conclusion

Setting up your own analytics system might sound complex, but with Umami, it’s surprisingly easy. It gives you everything you need to understand your website traffic without compromising user privacy. You control the data, the infrastructure, and the configuration.

By following these steps, you can deploy Umami on your own server in less than an hour and start monitoring your website visitors right away. Whether you run a personal blog, a SaaS platform, or a client project, Umami offers a transparent, fast, and privacy-friendly alternative to Google Analytics.

Hope you enjoyed this article. Find me on Linkedin or visit my website

Sharing data through charts is a great way to provide others with a clearer understanding of this information. Pairing it with a pivot table is also a smart approach, allowing you to view your data from different perspectives. And what if you could also aggregate data and customize charts to suit your needs?

In this guide, I’ve consolidated my knowledge on creating an analytics dashboard in Next.js using Flexmonster and Highcharts. To make it engaging, we’ll explore some interesting survey results about passenger flying etiquette. I hope you find it helpful. Let’s roll!

Table of Contents

• Prerequisites

• The Subject Area

• The Final Dashboard

• How to Configure Highcharts and Flexmonster for Our Next.js App

• How to Set Up the Dashboard

• How to Set Up the Chart Configuration

• Filling Up the Dashboard with Other Charts

• Connect between Flexmonster and Highcharts

• Full demo link

• Wrapping up

Prerequisites

1. Install Next.js (installation guide here)

2. In this project we’ll use the flexmonster, react-flexmonster, and highcharts libraries.

npm i react flexmonster react-flexmonster highcharts highcharts-react-official
npm i -g flexmonster-cli

These are paid tools for sharing analytics but offer more options for working with data and customization. You can also use free/open-source alternatives like:

Data/Analytics Libraries:

• PivotTable.js — basic open-source pivot functionality

• Orb — clean pivot UI with neat design

• WebDataRocks — modern, free pivot table solution

Charts:

1. Nivo — awesome free charts with D3 power

2. Chart.js — simple yet powerful canvas charts

3. Recharts — React components for D3 charts

The Subject Area

Have you ever been curious about why people get so angry on planes with others' behavior? Or maybe you’re that person who’s… misbehaving? Have you wondered if it’s okay to bother your neighbors onboard, whether it’s for stepping out of your seat or asking for something?

Imagine: you board a plane with your friends, and you are seated apart, not all together. You want to change places with someone to sit near each other. How likely is it that the person next to you will consider this to be rude or unacceptable?

We can explore these questions through a survey on passenger etiquette conducted by ABC News. Our goal is to examine passengers' attitudes toward in-flight interactions and present our findings with charts and a pivot table. The complete survey data is sourced from here.

In this guide, I’ll highlight some intriguing correlations between different social groups and their perspectives on certain questions, such as:

• The impact of age and income on overall dissatisfaction.

• What behaviors do passengers consider the rudest?

• How does traveling with children affect the flight experience?

• Which social groups tend to leave the flight most satisfied or dissatisfied?

• Who is more likely to break flight rules?

The Final Dashboard

In this tutorial, we’ll create an interactive dashboard using Next.js, featuring a pivot table and several charts on the final page.

Tools

I was looking for the tools that offer customizable and functional UI, wide range of chart types and pivot table features. So, I focused on Flexmonster for creating tables and Highcharts for the charts. These tools offer user-friendly interfaces, extensive customization options, handle large datasets, and integrate with each other.

Now, let's delve into the process of integration.

How to Configure Highcharts and Flexmonster for Our Next.js App

In order to create the analytical dashboard, you’ll need to install the libraries and configure your project.

1. Define the Next.js project:

npx create-next-app flexmonster-project --ts --app
cd flexmonster-project

2. Get the Flexmonster wrapper for React:

flexmonster add react-flexmonster

Got it, libraries are now actually installed! Now, let’s move further, embedding them into the project.

3. Import Flexmonster styles to global.css:

@import "flexmonster/flexmonster.css";

4. Create the wrapper

Now you’ll create the wrapper for your future pivot table that integrates Flexmonster and Highcharts. First, let’s create a PivotWrapper.tsx file.

'use client'
import * as React from 'react';
import * as FlexmonsterReact from "react-flexmonster";
import Flexmonster from 'flexmonster';
import "flexmonster/lib/flexmonster.highcharts.js";


// take general Flexmonster parameters and some special for Next.js
type PivotProps = Flexmonster.Params & {
    pivotRef?: React.ForwardedRef<FlexmonsterReact.Pivot>;
  }

// pivotRef provides a reference to the Flexmonster instance for accessing the Flexmonster API.
const PivotWrapper: React.FC<PivotProps> = ({ pivotRef, ...params}) => {
    return (
        <FlexmonsterReact.Pivot
            {...params}
            ref={pivotRef}
        />
    )
}

export default PivotWrapper;

5. Import the wrapper

Now, import the wrapper in the analytical-dashboard/page.tsx file. You can change the route name analytical-dashboard to your liking:

"use client"
import * as React from "react";
import type { Pivot } from "react-flexmonster";
import dynamic from "next/dynamic";
import * as Highcharts from 'highcharts';
import HighchartsReact from 'highcharts-react-official';


// Load the wrapper dynamically
const PivotWrap = dynamic(() => import('@/app/PivotWrapper'), {
    ssr: false,
    loading: () => <h1>Loading Flexmonster...</h1>
});


const ForwardRefPivot = React.forwardRef<Pivot, Flexmonster.Params>((props, ref?: React.ForwardedRef<Pivot>) =>
    <PivotWrap {...props} pivotRef={ref} />

)

ForwardRefPivot.displayName = 'ForwardRefPivot';

By the way, Flexmonster supports dynamic loading for its tables, which provides a smooth experience when extracting large amounts of data.

6. Place your pivot wrapper and charts into the dashboard.

Let’s define a WithHighcharts() function to render our dashboard and work with its essential parts. First, we’ll initialize the ref object for the pivot table to access it and its events. The charts will be created in the createChart() function, which runs when pivot data loads (in the reportComplete event). Finally, we return the layout function.

Add the code below to the analytical-dashboard/page.tsx file:

export default function WithHighcharts() {
    const pivotRef: React.RefObject<Pivot> = React.useRef<Pivot>(null);

    const reportComplete = () => {
        pivotRef.current!.flexmonster.off("reportComplete", reportComplete);
        createChart();
    }

    // define and create charts on the page. Called when data loading is successfully completed
    const createChart = () => {
        // here we will place charts rendering later
    }

    return (
        <div className="App">
            <div id="pivot-container" className="">
              <ForwardRefPivot
                ref={pivotRef}
                toolbar={true}
                beforetoolbarcreated={toolbar => {
                  toolbar.showShareReportTab = true;
                }}
                shareReportConnection={{
                  url: "https://olap.flexmonster.com:9500"
                }}
                width="100%"
                height={600}
                report = {{
                  dataSource: {
                    type: "csv",
                    // connect to our dataset
                    filename: "https://query.data.world/s/vvjzn4x5anbdunavdn6lpu6tp2sq3m?dws=00000"
                  }
                }}
                reportcomplete={reportComplete}
                // your license key
                licenseKey="XXXX-XXXX-XXXX-XXXX-XXXX"
              />
          </div>

          // here we will place chart layouts
    )
}

7. Run the application:

npm run build
npm start

If you are interested in a detailed integration of Flexmonster with Next.js and Highcharts, you can check the complete documentation that I’ve linked here.

For now, we’ve set up a blank dashboard that’s ready to be populated with data. In the next section, I’ll walk you through the overall chart definition process and highlight some key considerations.

First, you’ll need to ensure that your charts update based on the filters applied to the grid. For that, you need to add the following snippet of code:

 React.useEffect(() => {
    if (pivotRef.current) {
        const pivot = pivotRef.current.flexmonster;

        // Trigger the chart update when data changes
        pivot.on('dataChanged', createChart);
        pivot.on('filterclose', createChart);


        return () => {
          pivot.off('dataChanged', createChart);
          pivot.off('filterclose', createChart);
        }
    }
}, [pivotRef]);

In this case, Next.js tracks the filterClose event and triggers the createChart() function, which updates the charts. Unfortunately, there is no event for filter changes, only for filter pop-up window opening/closing. So, I used the event that triggers on filter pop-up window closing, as I usually use it for applying filters. You can read more about Flexmonster events here.

How to Set Up the Dashboard

To get started, let's get acquainted with our pivot table. The upcoming charts will draw information directly from it, so it's a good idea to first understand how the dashboard works.

In the previous step, we already created a default grid. If you run it now, it will look like this:

It simply displays some default data from the dataset. Not bad, but we want to configure more representative data and explore the full potential of Flexmonster.

1. Configure the pivot table

The main buttons that allow you to configure what and how the data is displayed in the table are Format, Settings, and Fields. We’ll start with Fields.

By opening the Fields tab, you can select which columns from your dataset you want to display in the table. What shall we choose?

Hmm… It seems interesting to see the percentage distribution of respondents of different ages across various regions. Some polled regions likely have more older respondents, some younger, and others may have a balanced mix. This is more about analyzing the composition of the survey itself, but it’s a good start.

To achieve this, select the necessary fields in Rows and Columns. Next, choose the function to calculate the final values. There are quite a few options, but we’ll go with Percentage of column. As a result, you’ll get the percentage distribution of respondents of specific age groups in each region.

Here we go:

Alright, let’s tidy up this table a bit. We’ll keep just one decimal place, which you can do in the Format tab like this:

It’s quite simple: select the required field (in this case, there will only be one) in the Choose Value field and set Decimal Places to 1.

We’ll also remove the Grand Totals since they are unnecessary for our purposes. You can do this in the Settings tab:

Finally, we’ll want to remove blank responses. This is a feature of the selected dataset – some respondents chose not to answer certain questions, leaving the response field blank. For our analysis, this data is unnecessary, so we’ll filter it out.

And here’s how the table looks now. Quite neat!

2. Conditional Formatting

Now we want to set up conditional formatting to highlight standout data. The table is already configured, but adding visual cues will make the information easier to interpret. Fortunately, Flexmonster supports this feature, so let’s use it.

Here’s my idea: we want to identify areas where the distribution of respondents by age is uneven. We have four age ranges: youngsters (18-29), millennials (30-44), Gen X (45-59), and seniors (60+). In a balanced distribution, each age group should account for 100/4 = 25% (with a margin of ±2÷3%).

Here’s the plan:

• 22-27%: Average distribution (within range).

• 17-22% or 27-32%: slightly underrepresented or overrepresented.

• <17% or >32%: strongly underrepresented or overrepresented.

In the Format tab, select Conditional Formatting and add conditions. Define the range with the value it applies to, and the highlight color. As a result, you’ll get a table that looks like this:

According to the table, most respondents are middle-aged. In some cases, they make up the majority of polled people of their region. The most unbalanced distribution is around the East South Central and Middle-Atlantic states. There are overrepresented in one age group, and underrepresented in others.

Awesome! We’ve explored the Flexmonster table. Beyond displaying data, it can also filter data for all our upcoming charts. Let’s move on to setting up the charts, and I’ll show you how the table can influence its data.

How to Set Up the Chart Configuration

In this section, we’ll create pie, bar, column, area, and line charts. It sounds like a lot, but Highcharts provides a wide range of chart types. You can also customize them to your liking.

First, we’ll create a pie chart showing location distribution of passengers. While a basic pie chart would typically be a circle, I’m opting for a donut chart instead. This example demonstrates the common chart configuration process. For other charts, simply modify the chart type, rows, and measures. You can also play with dataset fields and chart types to your liking.

1. Insert Highcharts into the page.

First, move down to the return() section. at the bottom, leave a free place for the chart layouts. So, here we’ll insert the first <div> block to store the future pie chart. As it’s about the etiquette of passengers onboard, let’s assign it an id of chart-location-distribution:

<div className="chart-item">
  <h2>Location Distribution</h2>
  <div id="chart-location-distribution"></div>
</div>

2. Define the chart options.

Let’s move on to the createChart() function. At the moment, it’s empty. To define the chart, paste pivotRef.current!.flexmonster.highcharts?.getData() into it. It gets three parameters inside: chart options, callbackHandler, and updateHandler.

pivotRef.current!.flexmonster.highcharts?.getData(
    // getting the current slice of data from the grid to apply filters to the chart
    const gridSlice = pivotRef.current!.flexmonster.getReport()?.slice as Flexmonster.Slice
        {
            type: 'pie',
            slice: {
                rows: [{ uniqueName: 'Location (Census Region)', }],
                measures: [
                    {
                        uniqueName: 'RespondentID',
                        aggregation: 'count',
                    },
                ],
                // apply current grid filters to the chart
                reportFilters: gridSlice.reportFilters
            },
        },
    // ...
    // in the next code section, I describe last two parameters
    // to be continued
)

Since we pass chart options as Flexmonster.Slice from getReport(), the data updates dynamically from the grid. The Slice object specifies which data sample to display on the chart by selecting from the dataset. You can read more about the properties of grid slices here.

You can customize the chart by changing data properties inside the callbackHandler parameter. Down below, we’ll create a custom donut chart by doing this:

pivotRef.current!.flexmonster.highcharts?.getData(
  // ...
  // continuing of configuring chart
  // chart options parameter defined in previous code section

  (data: any) => {  
        // Define the chart configuration within the data object
        data.chart = {
            type: 'pie',
        };

        data.title = {
            text: 'Aggregated Survey Responses'
        };

        data.legend = {
            layout: 'horizontal',
            align: 'center',
            verticalAlign: 'bottom', 
            x: 0,
            y: 10, 
        }

        data.plotOptions = {
            pie: {
                innerSize: '50%',
                dataLabels: {
                    enabled: true,
                    format: '<b>{point.name}</b>: {point.percentage:.1f} %',
                },
            },
        };

        Highcharts.chart('chart-location-distribution', data);
    },
    (data: any) => {
        Highcharts.chart('chart-location-distribution', data);
    }
)

Note! You should change data before it passes to the Highcharts.chart() function as a parameter.

3. Completed Location Distribution Pie Chart

The chart shows that the most common passengers on this airline are locals from the Pacific, and the least common passengers are from the East South Central states. This is quite interesting. It might be because the Pacific region includes large cities such as San Francisco, Los Angeles, Seattle, and Portland, while the East South Central region includes states like Mississippi and Alabama, which have smaller populations and population density.

Filling Up the Dashboard with Other Charts

Now, let’s create the remaining charts following the same process – adjusting the chart type, slice options, and callbackHandler functions for each diagram. You can find the full chart configuration of each chart at the end of the guide on my GitHub.

1. Frequency of using airlines distributed by locations

Here we can see with a stacked column chart how often people from different regions fly.

The average passenger from regions across the board tend to fly infrequently, with most flying once a year. Overall, the trend suggests that flying habits are influenced by regional factors, with occasional travel being more common than frequent flying. An interesting fact is that many of the polled people here have never flown! (How did they get into this poll?!)

2. Who breaks the rules on board?

Tricky question, isn’t it? We all know that smoking and using electronics are prohibited on board. But not everyone follows these rules, and fewer will admit to breaking them.

Still, pollsters managed to gather some insights about rule violations. The bar chart above reveals a trend: frequent flyers tend to neglect the rules, while those who fly less often are more likely to follow them.

From my own experience, your first flights are stressful, and you typically try not to disrupt the delicate balance of the plane. Every rule feels crucial, almost sacred. But regular fliers don’t seem to think this way.

3. Ease of communication depending on age

The line chart below shows that age influences your ease of communication with strangers, with middle-aged people often feeling more comfortable initiating conversations.

Personally, I always enjoy befriending or communicating with strangers, as it brings a sense of connection. But for many people, especially on board a flight, approaching strangers can be difficult, with older passengers often feeling more hesitant to engage.

People aged 30-44 are most comfortable communicating with strangers during flights. In contrast, both younger passengers (under 30) and older passengers (60+) view excessive communication as rude and prefer to keep to themselves during the journey.

4. Do men consider bringing children onboard ruder than women?

I thought that men might be more affected by unruly children in different spaces, so I decided to check that. The question asked was, ‘Is it rude to bring unruly children on board?’ The options were ‘No, not at all,’ ‘Yes, somewhat rude,’ and ‘Yes, it’s rude.’ So, I built another pie chart to see the relationship.

Almost two-thirds of men chose the ‘Yes, rude’ option, while fewer women selected that answer.

5. People's demands of others and themselves.

People often hold double standards, considering certain behaviors rude when done by others but not when they engage in those behaviors themselves. One of the most painful issues onboard is the mess created by reclining seats, which can affect the comfort of other passengers. This led me to research how people feel about reclining seats and how frequently they engage in the practice.

An interesting correlation emerged in two key points: the most outraged passengers are often those who do not recline their seats and are usually the ones who can't tolerate when others do so. Also, there is always a small group of people who recline their seats regularly but are willing to argue with others about the same action.

6. Which age group tends to be outraged?

The area chart covers all the questions about what passengers consider rude, allowing us to observe how different age groups respond to various situations.

Interestingly, the most outraged responses are regarding babies onboard and people waking up to walk around the plane. Youngsters are most outraged by reclining seats, while passengers aged 30-44 tend to be more upset about people walking to the bathroom. Older passengers are also concerned about reclining seats.

The least outraged responses are about communicating with others and moving to unsold seats, indicating that gadgets haven’t completely stolen the value of live communication.

I attached links to the Highcharts documentation for each chart, but additional charts are available. You can inspect them here.

How to Sync Data Between Flexmonster and Highcharts

Let’s recall the earlier section on configuring chart options. Here, adding the reportFilters field to the slice object enables data synchronization across all dashboard elements, from pivot table to charts.

To filter dashboard data, add fields to Report Filters in the Fields tab and observe chart updates. For example, filtering Location values affects the pie chart and column chart since they use Location data.

You can try it out by adding other rows to Report Filters and directly influence the appearance of your charts.

Full Demo Link

You can check out the full Next.js demo app and the dashboard yourself here.

Wrapping up

In this tutorial, we built an interactive statistical dashboard, analyzed the subject matter, and processed the dataset. Along the way, we explored key configuration specifics for the libraries we used: Highcharts and Flexmonster. We then populated the dashboard with charts and demonstrated how to interact with it effectively.

I aimed to highlight a diverse and engaging dataset to make the process more interesting. I hope this guide serves as a helpful resource for building analytical dashboards in Next.js. Best of luck with your projects!

The reason I'd missed the live version was because I had no time (I was, if you must know, rushing to finish my Manning book, The Complete Obsolete Guide to Generative AI – from which this article is excerpted).

Well, a half a dozen hours later I still had no time for the video. And, inexplicably, the book was still not finished.

So here's how I resolved the conflict the GPT way:

• I used OpenAI Whisper to generate a transcript based on the audio from the recording

• I exported the transcript to a PDF file

• I uploaded the PDF to ChatPDF

• I prompted ChatPDF for summaries connected to the specific topics that interested me

Total time to "download" the key moments from the 90 minute call: 10 minutes. That's 10 minutes to convert a dataset made up of around 15,000 spoken words to a machine-readable format, and to then digest, analyze, and summarize it.

How to Use GPT for Business Analytics

But all that's old news by now. The next-level level will solve the problem of business analytics.

Ok. So what is the "problem with business analytics"? It's the hard work of building sophisticated code that parses large datasets to make them consistently machine readable (also known as "data wrangling"). It then applies complex algorithms to tease out useful insights. The figure below broadly outlines the process.

A lot of the code that fits that description is incredibly complicated, not to mention clever. Inspiring clever data engineers to write that clever code can, of course, cost organizations many, many fortunes. The "problem" then, is the cost.

So solving that problem could involve leveraging a few hundred dollars worth of large language model (LLM) API charges. Here's how I plan to illustrate that.

I'll need a busy spreadsheet to work with, right? The best place I know for good data is the Kaggle website.

Kaggle is an online platform for hosting datasets (and data science competitions). It's become in important resource for data scientists, machine learning practitioners, and researchers, allowing them to showcase their skills, learn from others, and collaborate on projects. The platform offers a wide range of public and private datasets, as well as tools and features to support data exploration and modeling.

How to Prepare a Dataset

The "Investing Program Type Prediction" dataset associated with this code should work perfectly. From what I can tell, this was data aggregated by a bank somewhere in the world that represents its customers' behavior.

Everything has been anonymized, of course, so there's no way for us to know which bank we're talking about, who the customers were, or even where in the world all this was happening. In fact, I'm not even 100% sure what each column of data represents.

What is clear is that each customer's age and neighborhood are there. Although the locations have been anonymized as C1, C2, C3 and so on, some of the remaining columns clearly contain financial information.

Based on those assumptions, my ultimate goal is to search for statistically valid relationships between columns. For instance, are there specific demographic features (income, neighborhood, age) that predict a greater likelihood of a customer purchasing additional banking products? For this specific example I'll see if I can identify the geographic regions within the data whose average household wealth is the highest.

For normal uses, such vaguely described data would be worthless. But since we're just looking to demonstrate the process it'll do just fine. I'll make up column headers that more or less fit the shape of their data. Here's how I named them:

• Customer ID

• Customer age

• Geographic location

• Branch visits per year

• Total household assets

• Total household debt

• Total investments with bank

The column names need to be very descriptive because those will be the only clues I'll give GPT to help it understand the data. I did have to add my own customer IDs to that first column (they didn't originally exist).

The fastest way I could think of to do that was to insert the =(RAND()) formula into the top data cell in that column (with the file loaded into spreadsheet software like Excel, Google Sheets, or LibreOffice Calc) and then apply the formula to the rest of the rows of data. When that's done, all the 1,000 data rows will have unique IDs, albeit IDs between 0 and 1 with many decimal places.

How to Apply LlamaIndex to the Problem

With my data prepared, I'll use LlamaIndex to get to work analyzing the numbers. As before, the code I'm going to execute will:

• Import the necessary functionality

• Add my OpenAI API key

• Read the data file that's in the directory called data

• Build the nodes from which we'll populate our index

import os
import openai
from llama_index import SimpleDirectoryReader
from llama_index.node_parser import SimpleNodeParser
from llama_index import GPTVectorStoreIndex

os.environ['OPENAI_API_KEY'] = "sk-XXXX"

documents = SimpleDirectoryReader('data').load_data()
parser = SimpleNodeParser()
nodes = parser.get_nodes_from_documents(documents)
index = GPTVectorStoreIndex.from_documents(documents)

Finally, I'll send my prompt:

response = index.query(
    "Based on the data, which 5 geographic regions had the highest average household net wealth? Show me nothing more than the region codes"
)
print(response)

Here it is again in a format that's easier on the eyes:

Based on the data, which 5 geographic regions had the highest household net wealth?

I asked this question primarily to confirm that GPT understood the data. It's always good to test your model just to see if the responses you're getting seem to reasonably reflect what you already know about the data.

To answer properly, GPT would need to figure out what each of the column headers means and the relationships between columns. In other words, it would need to know how to calculate net worth for each row (account ID) from the values in the Total household assets, Total household debt, and Total investments with bank columns. It would then need to aggregate all the net worth numbers that it generated by Geographic location, calculate averages for each location and, finally, compare all the averages and rank them.

The result? I think GPT nailed it. After a minute or two of deep and profound thought (and around $0.25 in API charges), I was shown five location codes (G0, G90, G96, G97, G84, in case you're curious). This tells me that GPT understands the location column the same way I did and is at least attempting to infer relationships between location and demographic features.

What did I mean "I think"? Well I never actually checked to confirm that the numbers made sense. For one thing, this isn't real data anyway and, for all I know, I guessed the contents of each column incorrectly.

But also because every data analysis needs checking against the real world so, in that sense, GPT-generated analysis is no different. In other words, whenever you're working with data that's supposed to represent the real world, you should always find a way to calibrate your data using known values to confirm that the whole thing isn't a happy fantasy.

I then asked a second question that reflects a real-world query that would interest any bank:

Based on their age, geographic location, number of annual visits to bank branch, and total current investments, who are the ten customers most likely to invest in a new product offering? Show me only the value of the customer ID columns for those ten customers.

Once again GPT spat back a response that at least seemed to make sense. This question was also designed to test GPT on its ability to correlate multiple metrics and submit them to a complex assessment ("...most likely to invest in a new product offering").

I'll rate that as another successful experiment.

Wrapping Up

GPT – and other LLMs – are capable of independently parsing, analyzing, and deriving insights from large data sets.

There will be limits to the magic, of course. GPT and its cousins can still hallucinate – especially when your prompts give it too much room to be "creative" or, sometimes, when you've been gone too deep into a single prompt thread. And there are also some hard limits to how much data OpenAI will allow you to upload.

But, overall, you can accomplish more and faster than you can probably imagine right now.

While all that greatly simplifies the data analytics process, success still depends on understanding the real-world context of your data and coming up with specific and clever prompts. That'll be your job.

This article is excerpted from my Manning book, The Complete Obsolete Guide to Generative AI. There's plenty more technology goodness available through my website.

Ken hosts the excellent Ken's Nearest Neighbors podcast and the Exponential Athelete podcast.

We talk about:

• How an injury pushed Ken out of pro sports and into data science

• How Ken explains his statistical insights to coaches and players to help them improve their performance

• Why Ken doesn't think building projects is all that useful anymore. "Data Scientists should instead build products."

• How Ken starts and ends each day with meditation, and writes down all the ideas that pop into his head after each session.

• Ken's observation that: "Who is the best suited to excel in a world where AI tools are prominent? Probably the people who are building them. People in the data science domain, people who are coding – they're the most prepared to use these tools for other things."

Can you guess what song I'm playing on my bass during the intro? It's from a 2006 dance song, and it was originally played on a synth.

Also, I want to thank the 9,779 kind people who support our charity each month, and who make this podcast possible. You can join them and support our mission at: https://www.freecodecamp.org/donate

You can watch the interview on YouTube:

Or you can listen to the podcast in Apple Podcasts, Spotify, or your favorite podcast app. Be sure to follow the freeCodeCamp Podcast there so you'll get new episodes each Friday.

Links we talk about during our conversation:

Ken's Nearest Neighbors Podcast: https://www.youtube.com/channel/UCpEJMMRoTIHJ8vG8q_EwqCg

The Exponential Athelete Podcast, also hosted by Ken: https://www.youtube.com/playlist?list=PLAkSd12rP282takuFJKsAsYlHdpdEDhuE

The Founders podcast, which both Ken and Quincy listen to. James Dyson episode: https://www.founderspodcast.com/episodes/88384801/senra-james-dyson-against-the-odds-an-autobiography

Anna Wintour episode: https://www.founderspodcast.com/episodes/58741411/senra-326-anna-wintour

San Antonio caves that Quincy visited: https://naturalbridgecaverns.com/

Grafana is an open-source analytics and monitoring platform. It has gained widespread recognition among developers and enterprises looking to extract more insights from the data produced by their systems.

Grafana has many powerful visualization features, and when combined with Amazon EC2's scalability and flexibility, it creates a stable environment for efficient monitoring.

This article will walk you through setting up Grafana on Amazon EC2 and creating informative dashboards out of raw data.

For Whom is this Intended?

This tutorial is intended for both novices to the cloud and experts in DevOps. The goal of this post is to make the installation process easier so you can use Grafana on AWS to its fullest. Now let's get going.

How to Configure Your EC2 Instance

You need to configure the inbound rule for your EC2 instance to access port 3000, as Grafana operates on this port. But first, you need to establish an EC2 instance. You can follow this guide on how to set up your AWS EC2 instance. It takes less than 5 minutes.

Once you have created your EC2 instance, you'll need to configure the network inbound rules. So head to your instance page and click on it. On the button widget, click on the security tab and click on the security group link (it should look like this: “sg-547**”).

Once you open the page in the inbound rules section, click on ‘Edit inbound rules’. Click on Add a new rule and add 3000 to the port range field, and on the source field, select 0.0.0.0/0. Then save.

Inbound rules

How to Create an IAM Role

Now you need to construct an IAM (Identity Access Management) role. You're developing an identity role so that you can generate credentials that you'll subsequently use to log in to your Grafana service.

IAM Dashboard

So, in the search field, type "IAM service" and click it. Click 'Create role', and select the AWS service as the trusted entity type.

IAM Trusted Entity

On the use case section, select EC2, then click next.

IAM role use case

On the Add Permissions page, click on the AdministratorAccess policy, then click next. Enter a role name – in this case, I used Grafana-Server-Role.

Role creation

How to Download Grafana

Now that you've configured your EC2 inbound rule and also configured the IAM role, let's set up Grafana on your EC2 instance.

So head over to Grafana's download page. Since we'll be downloading the version for Amazon Linux in this tutorial, you need to type in the following command on your Linux command line. Note: You need to connect to your VM instance through SSH (Secure Shell). In this case, I am using the EC2 Instance Connect.

sudo yum install -y https://dl.grafana.com/enterprise/release/grafana-enterprise-11.1.0-1.x86_64.rpm

Now you'll enable the Grafana service on your terminal by typing the following command:

systemctl enable grafana-server.service

Then start the service:

systemctl start grafana-server.service

Check the status of the Grafana service on the EC2 instance by running this command:

systemctl status grafana-server.service

Grafana Service Status

Now that you've confirmed that the service is currently active, you'll also need to check if the Grafana service is active on port 3000, as you've already created an inbound rule to cater for this.

You can do this by typing the following command:

netstat -tunpl | grep grafana

Port 3000 confirmation

Now that you've confirmed that the service runs on port 3000, you can go ahead and set up your Grafana dashboard.

You can access the Grafana dashboard by typing the Public IP of your EC2 instance and adding port 3000 on your web browser, something like this: 34.239.101.172:3000.

Grafana Login

The default username and password for Grafana are admin, but you'll be given the option to change your password after you sign in with the default credentials. You can also skip the password change process if you like.

Change password on Grafana

After this step, go to the home page. The next thing to do is to start connecting your Grafana dashboard to a data source. In this case, you're going to connect it to the AWS cloud watch service.

Grafana

How to Connect Data Sources to the Grafana Dashboard

Click on the connections tab on the side menu and click on data sources. Search for the CloudWatch service.

Cloudwatch configuration

Now you'll be prompted to input your access key ID and secret access key. You will need to create this on your AWS IAM service.

So go back to your IAM management dashboard and go to the user's tab. If you haven’t created an IAM user, you can do so by checking out this IAM user creation tutorial.

In the user IAM dashboard, scroll down to the access keys section and click on Create access key.

Access key

Select the Command Line Interface use case.

Access key use case

Set the description tag. This step is optional. Then click on the Create access key.

Access keys

Now copy the Access Key ID and Secret access key and paste them into the CloudWatch Datasource configuration page on Grafana. Set your default cloud region – in this case, mine is us-east-1

Additional settings

When you’re done, click on the save and test buttons. Grafana will query the Cloudwatch logs, and if it works fine it will save the configuration.

How to Create a Dashboard on Grafana

Now that you have successfully configured your grafana service, let’s start creating dashboards.

Click on the dashboard tab on the side menu click on New and select new dashboard. You should see the screen below:

Create a new dashboard

Then select Import dashboard.

Import a dashboard

For this case, you'll be importing an already-made dashboard from Grafana. Grafana has a lot of dashboards for a lot of use cases and services. But in this case, you'll be importing an EC2 dashboard (Grafana EC2 dashboard).

If you want to import it, you can easily copy the ID of the dashboard that you want to import. It is always accompanied by the dashboard.

So now copy the ID – in this case, it's 11265. Then paste it into the import field on the import dashboard, and click on the load button.

Grafana Dashboard

Now you have successfully created a dashboard in Grafana. This dashboard lets you monitor the performance of your EC2 instance. You can monitor metrics such as CPU Utilization, CPU Credit, Disk Ops, Disk Bytes, Network, Network Packets, Status check, and so on.

Wrapping Up

Thank you for reading! I hope this step by step guide has helped you learn how to create and set up efficient dashboards using Grafana.

Google Analytics is an effective tool for tracking and analyzing traffic and events on your websites and mobile apps. In this article, you'll learn how to add Google Analytics to your Astro website.

To follow along, you'll need the following:

• A Google account.
• A deployed Astro project.

Let's get started!

How to Add Google Analytics to an Astro Website

Before adding Google analytics, make sure you've already deployed your project. Here's one that I've deployed using Netlify: https://astro-article.netlify.app/. This is an Astro blog template created with the npm create astro@latest -- --template blog command.

This section will be divided into two sub-sections. In the first sub-section, you'll learn how to create an Analytics account and how to set up your account for tracking and monitoring your website.

In the second sub-section, you'll learn how to configure your code to sync with Google Analytics.

How to Set Up Google Analytics

You can follow these steps to set up Google Analytics:

Step #1 – Account Creation Page

The first step is to create a Google Analytics account. You can do that by visiting the Google Analytics website.

When the page loads, you should see something similar to this:

Google Analytics home page

Click on the "Start measuring" button. This will take you to the account creation page where you'll fill in some info about your website/app. That is:

"Create an Account" page on Google Analytics

After inputting an account name, scroll down the page and click on the "Next" button.

Step #2 – Property Creation Page

On the property creation page, you can create a property name, time zone, and currency.

A property acts as a unique identifier for your websites and apps. So it is like a container for all the data related to a specified website or app.

"Create a Property" page on Google Analytics

Step #3 – Business Details Page

On the business details page, you can specify your industry category and the business size.

"Describe your business" page on Google Analytics

Step #4 – Business Objectives Page

The business objectives helps Analytics generate personalized reports for your business. You can select the options that are important to your website, app, or product category.

"Business objectives" page on Google Analytics

After selecting the options, click on the "Create" button.

Step #5 – Data Collection Page

On the data collection page, you can choose between different platforms to collect data from. Since we're working with a website, we'll go with the Web option.

"Start collecting data" page on Google Analytics

Next, you'll have to create a data stream by inputting your website URL and your stream name (this can be whatever you want, but I'll recommend using a name related to your website).

Page to setup a data stream on Google Analytics

In my data stream information, I used https://astro-article.netlify.app/ as the URL and "Astro Blog Template" as the stream name.

Make sure you remove the "https://" part of the link in the input box so you don't get the "Valid website URL is required" error.

So instead of https://astro-article.netlify.app/, it should be astro-article.netlify.app/.

This is what mine looks like:

Example of information entered properly in data stream setup page

Go ahead and click on the "Create stream" button.

On the next page/prompt, you'll see your stream name, URL, ID, and measurement ID values.

The measurement ID will be important in our code integration so you can copy and paste it somewhere.

Here's what the page looks like:

"Web stream details" page on Google Analytics

If you click on "View tag instructions", you'll see a code snippet for manual integration with code. Copy the code and paste somewhere because we'll make use of it soon.

Installation instructions

Here's the code snippet if you missed it:

<!-- Google tag (gtag.js) -->
<script async src="https://www.googletagmanager.com/gtag/js?id=MEASUREMENT_ID"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'MEASUREMENT_ID');
</script>

If you're copying the code above, make sure you replace the two MEASUREMENT_ID values with the actual value of your measurement ID.

How to Integrate Google Analytics and Astro

At this point, we've created a Google Analytics account and generated our project's measurement ID. The next thing to do is to sync our code to Google Analytics.

But first, we have to install a library called Partytown. When using third party integrations, you may run into performance issues because you're making use of third-party code.

Partytown allows these integrations to run as expected without reducing your site's performance. You can read more about that here.

Step #1 – Install Partytown

Head to your Astro project terminal and run this command to install Partytown:

npm install @astrojs/partytown

Step #2 – Configure Partytown

After the installation is complete, you'll need to add the integration to your astro.config.mjs file:

import { defineConfig } from 'astro/config';
import mdx from '@astrojs/mdx';
import partytown from '@astrojs/partytown'

import sitemap from '@astrojs/sitemap';

// https://astro.build/config
export default defineConfig({
    site: 'https://astro-article.netlify.app/',
    integrations: [
        mdx(), 
        sitemap(),
        partytown({
            config: {
              forward: ["dataLayer.push"],
            },
        }),
    ],
});

In the code above, we imported the Partytown library: import partytown from '@astrojs/partytown'.

We then added this piece of code to the integrations object:

partytown({
      config: {
        forward: ["dataLayer.push"],
      },
})

Everything else in the code came with the Astro project.

Step #3 – Add Google Tag to your Pages

Remember the code snippet from Google Analytics? This is where we'll use it.

<!-- Google tag (gtag.js) -->
<script type="text/partytown" async src="https://www.googletagmanager.com/gtag/js?id=MEASUREMENT_ID"></script>
<script type="text/partytown">
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'MEASUREMENT_ID');
</script>

Note that we added the type="text/partytown" attribute to both <script> tags.

Remember to change the MEASUREMENT_ID values to the value of your measurement ID. Mine looks like this:

<!-- Google tag (gtag.js) -->
<script type="text/partytown" async src="https://www.googletagmanager.com/gtag/js?id=G-KM36S70L8Y"></script>
<script type="text/partytown">
    window.dataLayer = window.dataLayer || [];
    function gtag(){dataLayer.push(arguments);}
    gtag('js', new Date());

    gtag('config', 'G-KM36S70L8Y');
</script>

Do not use the snippet above as it has my measurement ID so it's already associated with my website (not my actual website :)). Copy the first code snippet and change the MEASUREMENT_ID values to the value of your measurement ID.

To track and monitor a page, you need to paste the code snippet in that page. Copy the code and paste it in the <head> section of every page file (index.astro, BlogPost.astro, and so on) you want to track and analyze using Google Analytics.

Here's an example using the index.astro file in the pages directory:

---
import BaseHead from '../components/BaseHead.astro';
import Header from '../components/Header.astro';
import Footer from '../components/Footer.astro';
import { SITE_TITLE, SITE_DESCRIPTION } from '../consts';
---

<!doctype html>
<html lang="en">
    <head>
        <!-- Google tag (gtag.js) -->
        <script type="text/partytown" async src="https://www.googletagmanager.com/gtag/js?id=MEASUREMENT_ID"></script>
        <script type="text/partytown">
            window.dataLayer = window.dataLayer || [];
            function gtag(){dataLayer.push(arguments);}
            gtag('js', new Date());

            gtag('config', 'MEASUREMENT_ID');
        </script>
        <BaseHead title={SITE_TITLE} description={SITE_DESCRIPTION} />
    </head>
... <!-- The rest of the HTML code -->

In the code above, we put the code from Google Analytics in the <head> tag. You can do this for all the pages you want to track and monitor.

Remember to add the type="text/partytown" attribute to the <script> tags, and to change MEASUREMENT_ID (used in two places in the snippet) to the value of your measurement ID.

Step #4 – Build and Deploy your Project

Deploy your project using your preferred process.

In my case, I use npm run build to build into the dist folder, and then push the code to GitHub. This automatically triggers a Netlify deployment.

You don't have to use my method, but make sure your project builds before deployment.

Once your website has been deployed, you may have to wait up to 48 hours to start seeing your analytics in the Google Analytics dashboard.

Pending page

You'll see this page when you click on the "Continue to Home" button:

Analytics dashboard

This is your Analytics dashboard. You can view and analyze your data here when there's data available.

And there you have it! You've successfully integrated your project with Google Analytics.

Summary

In this article, we saw how to add Google Analytics to an Astro project. This can be used to track and analyze traffic and events on websites and apps.

We saw how to create and set up an Analytics account.

We then saw how to integrate Google Analytics to an Astro project using code.

Happy coding! You can read more Astro tutorials on my blog.

We'll answer these questions and more. You'll get a manager-level crash course in augmented analytics. By the end of this article, you should have a solid understanding of what augmented analytics is, and how you can use it to improve your decision making.

What is augmented analytics?

Augmented analytics is a term used to describe the use of machine learning to automatically analyze data, then provide insights that humans can act on.

It’s sometimes also referred to as “cognitive analytics” or “analytics 2.0.”

How do augmented analytics work?

Augmented analytics rely on machine learning algorithms.

Machine learning is a type of Artificial Intelligence (AI) that allows computers to learn directly from data – without the need for a human to programming them. (That is, there's no need for a human to explicitly tell the computers what to do.)

Machine learning algorithms can automatically detect patterns in data, then use that to make predictions about future events.

What are the benefits of augmented analytics?

There are several benefits of using augmented analytics, including:

1. Automation of tedious and time-consuming tasks. Augmented analytics can automate the more rote aspects of data analysis, freeing managers to focus on the higher-level considerations, including a lot of domain-specific considerations.

2. Improved accuracy. By using machine learning algorithms, augmented analytics can provide more accurate insights than humans alone could. Computers are much better at crunching numbers than humans are. And humans are better at interpreting the underlying meaning of those numbers.

3. Increased speed. Augmented analytics can help managers make faster decisions by providing real-time insights. Imagine automatically refreshing dashboards that you can reference whenever you need to make a decision.

4. Increased scalability. You can easily scale up your augmented analytics to support larger organizations, or incorporate a broader array of data sources.

Which Industries Use Augmented Analytics?

Here are a few examples of industries that have embraced augmented analytics:

Augmented Analytics in Retail

In the retail industry, you can use augmented analytics to automatically analyze customer data and identify trends. If you can anticipate what customers are likely to buy, you can make better decisions about product assortment, pricing, and promotions.

Augmented Analytics in Manufacturing

In the manufacturing industry, managers use augmented analytics to automatically analyze data from sensors and machine data to identify issues and optimize production.

Imagine a factory where you know the exact state of every assembly line, the output of every human worker, or even the power usage of individual machines. Imagine being able to identify bottlenecks or waste through real-time dashboards.

There is a lot of room for applying Augmented Analytics tools to save money. So we see manufacturers like Toyota, Apple, and Airbus all making heavy use of these tools.

Automated Analytics in Healthcare

In the healthcare industry, doctors and managers can use augmented analytics to automatically analyze patient data, identify vital sign trends, and improve patient care.

Healthcare in particular has a lot of room for improvement in understanding medical records, and making better decisions based on available data. Augmented Analytics can give hospitals a big leg up.

Automated Analytics in Financial Services

There is perhaps no more obvious use of automated analytics than in finance. Numbers are an intrinsic part of moving money around between investors and companies deploying that money.

Because finance involves money, it is a magnet for fraudulent activity. Automated Analytics can help banks and other parties prevent fraud. They can also help governments prevent money laundering.

Automated Analytics in Marketing

Marketers can use augmented analytics to automatically analyze customer data to identify trends and improve marketing campaigns.

When you are spending money on advertising, even marginal refinements can improve your bottom line.

In short: Automated Analytics can help managers focus on what they do best, while offloading the grunt work to computers

Thanks for taking time to read my article. I hope you have fun exploring the world of Automated Analytics. If you're looking to harness technology tools, you can learn some new programming and computer science skills here on freeCodeCamp.org.

OpenTelemetry offers a single set of APIs and libraries that standardize how you collect and transfer telemetry data. OpenTelemetry provides a specification for instrumentation so that you can send data to distinct backends of your choice.

We just released a full OpenTelemetry course on the freeCodeCamp.org YouTube channel.

This course was made possible by a grant from New Relic, an observability platform that helps developers monitor and debug their applications.

Ania Kubów created the course. Ania is a popular software educator and she will help you get a full understanding of how to use OpenTelemetry in your projects.

In this course, you will learn how to use OpenTelemetry to get full stack observability on the performance and behavior of your software projects.

Here are the topics covered in this course:

• What is OpenTelemetry?
• What are Microservices?
• What is Observability?
• M.E.L.T
• History
• Setting up our Project
• What is Tracing?
• Context and Propagation
• Setting up our Tracing
• What are Metrics?
• Use cases for OpenTelemetry
• Setting up Distributed Tracing
• Using other Analysis Tools - New Relic
• Where to go next

Watch the course below or on the freeCodeCamp.org YouTube channel (1-hour watch).

Video Transcript

(Autogenerated)

Hello, internet, and welcome to this course all about open telemetry.

In this course, I'm going to be showing you how you can get full stack observability on the performance of the behavior of your software project by using open telemetry with your analysis tool of choice.

If you want to be able to tell why your project or app is running too slow, is broken, or you just want to improve its code quality.

This is the video for you.

But first, what exactly do we mean by over telemetry? Well, let's start off with the name itself, we have open so like open source, and telemetry, which is an institute collection of measurements or other data at remote points.

And that automatic transmission to receiving equipment for monitoring.

The word telemetry actually comes from the Greek root word Teller, or remote, and metron, for measure.

And that's exactly what we are going to be doing, we are going to be facilitating a way to measure the performance of everything we use in our app remotely.

With any app, when you want to start looking at this kind of data, you have two parts that need to come together.

The first is figuring out how to generate and transmit that data.

And then the second part is deciding what you are going to do with that data.

So in other words, how are you going to analyze it? Open telemetry deals with that bus part.

Up until now there has been no real standardized way of describing what your system is doing.

This is down to the fact that we all like to think differently, use different programming languages, different machines, and a combination of different ways.

This was a problem, especially for those wanting to build observability tools.

At the heart of open telemetry project is exactly that, a standardization for describing what distributed systems are doing, no matter what programming language or computer systems you are using.

Today, the open telemetry project can be described as a collection of tools, API's, and SDKs, use it instrument, generate, collect and export telemetry data, so that we can analyze it later on with whatever platform we wish by standardizing our data and means we are not tied to anything in the long run.

And it makes moving from one analysis tool to the next super easy, I will not affect your historic data.

As I mentioned at the beginning, this is an open source project.

It is made up of many, many developers and their inputs.

If we have a look at GitHub here for the project, you will see it is super transparent.

You can see the Governance Committee, you can see the technical committee.

If you are interested in getting involved, please do join their mailing list and attend the community meetings.

You can even see all the meetings that you are welcome to join.

Before we get started, though, I just want to go into a little bit of what we're going to learn on this course.

So let's break it down.

Okay, so in this course, first we're going to look at what are microservices followed by what is observability then we're going to look at melt, followed by the history of open telemetry, then we're going to actually start by setting up a project.

We are then going to talk about tracing, distributed tracing and context and propagation before adding tracing into our project.

We will then move on to looking at metrics before finally ending on our distributed projects.

I will end this course with where to go next.

We expect our websites, apps and online services to load almost instantaneously right? Think of the frustration some of us feel when websites take more than two seconds to load.

on the backend a feat of engineering is needed to keep a global system running to make sure that you have access to Netflix or Instagram.

Only a tap away Let's have a look at Netflix for a minute.

Netflix at its peak consumes 37% of internet bandwidth in the US, there are 1000s of people clicking play at the same time, with activity peaking in the evening.

Though, as a global platform, it's a constant pick.

The challenge is how to run a service with zero loss while processing over 400 billion events daily, and 17 gigabyte per second during peak.

In today's video, we will dive deeper into what types of systems power these amazing services and how we can use data to get deeper visibility into the insights of these complex systems.

Historically, developers about applications in monoliths with large complex code bases, a single monolith will contain all the code for all the business activities and application performed.

This is all fine and dandy if you have a small app.

But what happens if your application turns out to be successful, users will like it and begin to depend on it traffic increases dramatically, and always inevitably use requests improvements and additional features.

So more developers are roped into work on the growing application.

before too long, your application becomes a big ball of mud.

a situation where no single developer understands the entirety of the application.

Your once simple application has now become larger complex.

multiple independent development teams are simulataneously.

Working on the same codebase and simultaneously changing the same sections of code, then it becomes virtually impossible to know who is working on what poetry does collide, code quality suffers, it becomes harder and harder for individual development teams to make changes without having to calculate what the impact will be to other teams and teams can lose sight of how that code might be incompatible with others, among other issues.

This generally results in slower, less reliable applications, not to mention longer and longer development schedules.

In come microservices, the main principle behind a microservice architecture is that the applications are simpler to build and maintain when broken down into smaller pieces.

When using microservices, you isolate software functionality into multiple independent modules that are individually responsible for performing precisely defined standalone tasks.

These modules communicate with each other through simple API's.

microservice architectures let you split applications into distinct independent services, each managed by different teams.

It leads to naturally delegating the responsibilities for building highly scaled applications, allowing work to be done independently on individual services without impacting the work of other developers in other groups working on the same overall application.

However, trying to get visibility into this system can be very difficult.

When you have hundreds of services and applications your request traveled through debugging and troubleshooting can be a nightmare.

This is where open telemetry comes in.

As we already touched on telemetry is defined as the science or process of collecting information about objects that are far away and sending the information somewhere electronically.

Now.

observability means how well you can understand what is going on internally in a system based on its outputs.

Especially as systems become more distributed and complex.

It's hard to see what's going on inside your application and why things may be going wrong.

When talking about observability, we need to define the data types necessary to understand the performance and health of our application broadly, metrics, events, logs and traces.

Met, metrics are measurements collected at regular intervals must have a timestamp and name, one or more numeric values and a count of how many events are represented.

These include error rate, response time or output.

An event is a discrete action happening at any moment in time.

Take a vending machine.

For instance, an event could be the moment when a user makes a purchase from the machine.

Adding metadata to events makes them much more powerful.

With the vending machine example we could add additional attributes such as item category, and payment type.

This allows questions to be asked such as how much money was made from each item category, or what is the most common payment type use.

logs come directly from your app, exporting detailed data and detailed context around an event.

So engineers can recreate what happened millisecond by millisecond.

You have probably logged something when you use things like system out print, or console log traces follow a request from the initial request to the returned output.

It requires the casual chain of events to determine relationships between different entities.

traces are very valuable for highlighting inefficiencies, bottlenecks and roadblocks in the user experience as they can be used to show the end to end latency of individual cores and a distributed architecture.

However, getting that data is very difficult.

You would have to manually instrument every single service one by one layer by layer.

This will take as much time as writing the code itself, which is annoying.

Luckily there are some awesome open source projects as well as companies that make this a lot easier in today's 16 open tracing was released as a cn CF project focused only around distributed tracing.

Because the libraries were lightweight and simple, it could be used to fit any use case.

While it made it easy to instrument data, it made it hard to instrument software that was shipped as binaries without a lot of manual engineering work.

In 2018, a similar project called Open census was open source out of Google, which supported both the capturing retracing permission and metrics.

While it made it easier to get telemetry data from software that was shipped as binaries like Kubernetes, and databases, it made it hard to use the API to instrument custom implementations, not part of the default use case.

Both projects were able to make observability easy for modern applications and expedite wide adoption of distributed tracing by the software industry.

However, developers had to choose between two options with pros and cons.

It turns out that the approaches of the two projects were complimentary rather than contradictory.

There was a no reason why we couldn't have both the abstract vendor neutral API and a well supported default implementation.

In late 2019, the two projects merged to form opentelemetry.

This brought forward the idea of having a single standard for observability instead of two competing standards.

Okay, so first things first, let's go a bit meta and understand exactly what is going to happen.

In order for us to view what is happening in our app.

Here we have our project.

Think about project as a project we have made with our code editors of choice.

For example, if we run it, it runs locally on our machines.

Let's also say this project is built to listen out for requests.

So like listening out for a get request, for example, we have decided that we want to measure our app's performance based on the requests that it makes.

To do this, the first step as we mentioned at the start, would be to implement open telemetry into the project.

We are doing this to help us standardize the data.

Once we have implemented open telemetry and standardize the data, we need to think about what we are going to do with the data, how we're going to view it, and so on.

For this, we can use an analysis tool.

By analysis tool, I mean, any type of tool that gives you observability, we are going to look at a few of these tools, one that focuses specifically on tracing, one that focuses specifically on metrics, and one that looks at everything in one platform.

We are then going to send our data to our analysis tool of choice.

Here we have an example of what our data can look like in a tracing app such as zipkin, a metrics app such as permittees, and an observability tool such as New Relic that will give us an overview of everything as well as other more bespoke insights in one platform.

We will go into each of these in their dedicated sections.

Let's start with implementing open telemetry first.

Okay, so before we start, the only prerequisite I am going to ask of you is that you have Docker downloaded onto your machine.

Docker is a container platform for rapid up microservices development, and delivery.

And if you don't know much about containers and micro services, just stick with me for now.

I have a section dedicated to both of these topics coming up after this section.

For those of you that don't have it, please navigate to the Docker website and follow the instructions.

In order to get set up.

I personally would choose to download the Docker desktop.

As I am working on a Mac, I choose the Mac option.

Once you are done, make sure your Docker desktop is running on a Mac, I would simply hover over the icon like so.

You will see if you open up the platform I currently have no containers running.

Having a container is going to be important for when we get to using the analysis tools, such as a tracing back end like zipkin.

Once we have all got Docker up and running, let's get going.

Let's get our terminal up.

Now I'm going to navigate to a folder where I like to store all my projects.

It's called development.

Please go ahead and choose any directory that you would like to work in.

Now once here using the Mk dir command, I'm going to make a folder for the new project that I'm about to start.

I'm going to call it open telemetry starting out.

Please be aware that if you are using Different terminal, other commands may need to be used.

Now, let's go into the project using the command cd.

The first thing I'm going to do is start our container.

So it's ready for the next section.

So, as mentioned, the first thing that you need before you can start collecting traces is a tracing back end like zipkin, that you can export traces to, to set up zipkin as quickly as possible, run the latest Docker zipkin container, exposing Port 9411.

If you can't, or don't want to run a Docker container, you could also choose to download zipkin directly.

So just so you know, that is an option too.

If you want to explore that option more, I would suggest visiting the zipkin website.

So this is the command to run our container.

And there we go, we now have this container ID, it has worked.

Next up, we need to get a package JSON file in our project, we can do so by typing NPM.

In it, having a package JSON file will make it easier for others to manage and install all the packages we need for the project.

If you are getting errors, it could be because you don't have no js and NPM installed.

If that is the case, please visit node j s.org.

In order to get set up with that by following the download instructions.

Okay, so we have initialized the utility, I am just going to press enter for all these fields, it's prompting me to answer.

So enter and, and, and, and enter.

OK, we are done creating our package JSON file for now.

If we list out all the files on our project, using the ls command, you will see the file right there.

Now finally, I need to create an app js file.

So a JavaScript file and put that in the project to some of you might have a different approach to adding files to a project.

So that is totally up to you, however you'd like to create that file.

But once we are done, we now need to open up our project.

As I am using VS code, I'm going to use the command code.in order to open up our project.

And there we go.

There is a folder with an app js file and a package JSON file.

You will see the package JSON file has all the prompts we were asked, as I skipped all the prompts is just the standard default entries are no entries at all.

If you go to the app js file, you will also see that it's currently empty, it has nothing in it.

Okay, so the first thing I need to do is change this to point to the app js file we created not the index j s as there is no index js file.

And just for fun, this is not necessary, I'm going to fill out the description on my project.

So open telemetry costs.

The next thing I want to do is just add a start script.

So for this script, I'm going to run node app j s.

Okay, now for the fun part, let's get to adding some packages.

To start using open telemetry, we are going to need to install some of its packages.

Using NPM II or install for short.

I'm going to store the open telemetry core package, the open telemetry node package opentelemetry plugin http opentelemetry.

plugin HTTP s, open telemetry exporters zipkin to get us ready for the next section on open telemetry tracing for the next section to an express the only non open telemetry one.

Okay, so we need all of those to install.

Okay, and great.

So these are looking good.

So if we look back here, here are the packages that we just installed.

They have automatically populated our package JSON file.

So this is looking good.

We just missed one.

So let's go back and use NPM II or install for short and install Open telemetry plug in Express.

And we are done.

Okay, now let's move on to our app js file.

The first thing I'm going to do is define a port for us.

So const port, process and V port or a string of Port 8080 Next up, we are going to need Express.

So const Express.

And Express is one of the packages we installed.

So this one right here.

So we need to tell our file that this const requires Express.

We are then going to call Express and stored as the const app.

The first thing I want to do is just get a message in our console log to let us know all is good, and that we are listening out and on which port.

So just as a recap, what I am doing here is getting the app to start a server.

And then with this code, I'm getting it to listen out to our defined port for any requests.

Okay, next up, I'm going to paste this super basic piece of code.

This code is an example of a very basic route.

routing refers to how an applications end points respond to client requests.

We have defined the route using methods of the Express app object that corresponds to http methods.

For example, app get handles GET requests.

These routing methods specify a callback function called when the application receives a request to the specified route an HTTP method.

In other words, it listens out for requests that much specified routes and methods.

In this case, the root is our homepage, we essentially want to trace every single time a get request is made to the homepage.

We will do this in the next section.

Now that we have the basic installation done, it is now time for us to talk about tracing.

So as you know, open telemetry allows us to essentially standardize our data.

The next part is actually viewing the data in a way that we can analyze what is happening behind the scenes, we will do this with a tracing system.

in software engineering tracing involves a specialized use of logging to record information about a program's execution.

This information is typically used by programmers to debug by using the information contained in a trace log to diagnose any problems that might arise with a particular software or app.

Distributed tracing, however, also called a distributed request tracing is a method used to debug and monitor applications built using a micro service architecture.

Distributed tracing helps pinpoint where failures occur, and what causes poor performance.

So as we can now see, being able to get tracing data telemetry is pretty important to the overall performance of an app.

However, as we discussed in the introduction, due to systems using all types of different languages, frameworks and infrastructures, it's a hard thing to do without some sort of common approach.

That's why Evans limitary can help so much with distributed tracing.

By providing a common set of API's, SDKs, and wire protocols.

It gives organizations a single, well supported integration service for end to end distributed tracing telemetry.

For this course, the tracer we're going to be using is called zipkin.

zipkin is a distributed tracing system that helps them gather timing data needed to troubleshoot latency problems and service architectures.

Circuit was originally created by Twitter.

And it's currently run by the open zipkin volunteer organization.

I am using zipkin for no other reason than I had to pick one.

But please do feel free to choose any back end tracing system that you wish, the choice is completely up to you.

In general, once the traces are implemented into applications, they essentially record timing and metadata about operations that take place.

An example of this is when a web server records exactly when it receives a request.

And when it sends a response.

This data is usually represented by a bar, just like this, and goes by the official name of spam.

So in this example, we have two services and a bunch of spans.

To explain this, imagine this represents your favorite food delivery app.

So imagine you make an order right.

Now a few things will happen, each represented by a span.

You send information back and forth from services in order to make the payment, find a delivery driver closest to you, and notify that driver with your order.

Each of these operations generates a spam showing you the work being done to make this happen.

In this case, the spans have implicit relationships, so parent and child but also from individual services and the trace.

As you can see, each of the spans starts at a different point and takes a different amount of time.

We call this latency and network latency.

In a nutshell, latency is delay between an action and a response to that action.

network latency refers to specific delays that take place within a network.

latency is generally measured in milliseconds and is unavoidable due to the way networks communicate with each other.

It depends on several aspects of a network and can vary if any of them are changed.

errors on most systems are usually quite easy to spot.

If your bar ends in red or similar, for example, you know that an error has occurred.

Now it's time for us to look at context and propagation.

These two concepts will allow us to understand the topic of tracing a lot better.

So as we know distributed tracing allows us to correlate events across service boundaries.

But how do we find these correlations for this components in our distributed system need to be able to collect, store and transfer metadata? We refer to this metadata as context.

Context is divided into two types, span context and correlation context.

span context represents the data required for moving trace information across boundaries.

It contains the following metadata.

We have a trace ID, a span ID, the trace flags and the trace state of this bond context.

And then we have the correlation context.

A correlation contacts carries user defined properties.

This is usually things such as a customer ID, providers, hearse name, data region and other telemetry that gives you application specific performance insights.

Correlation context is not required and components may choose to not carry or store this information.

A context will usually have information so we can identify the current span and trace and propagation is the mechanism we use to bundle up our context and transfer across services so that we have it context and propagation.

Together.

These two concepts represent the engine behind distributed tracing.

If you would like to learn more about these two topics and do a deep dive at please visit the urban telemetry website.

For the purpose of this tutorial, however, a basic knowledge above will suffice.

So just as a bit of a heads up in this next section, we will go through how to first initialize a global tracer and after that initialize and register a trace exporter.

Okay, so here we are where we left off.

Now in the last section, we ran the latest Docker zipkin container exposing Port 9411.

If we actually visit localhost 9411, we will see the zipkin UI that comes as part of doing that.

So here we are, this is what we are going to use to view our traces.

Okay, let's carry on.

Next, let's create a file named tracing and j s and add the following code.

This is just the sample code provided to us by opentelemetry.

If you visit their website, you can see I am just copying this code right here and pasting it into my project.

You will also see that this file uses two of the packages we installed in the initial setup.

Once we have pasted that we need to initialize and register a trace exporter.

We have already done part of this as we install two packages necessary for this part and the initial setup section.

And that is the open telemetry tracing and open telemetry export to zipkin packages.

So first off, let's make a new zipkin exporter.

So provide add span processor, new simple span processor.

This is from the open telemetry tracing package, as you can see that has shown up at the top news of can export out.

And that also comes from another package.

So the open telemetry exporters, it can package as you can see appearing at the top, this might not automatically show up for you.

It's just the code editor I'm using.

So you might have to type those two out, and then the right service name and choose a service now.

I'm going to put Getting Started but you can replace it with your own service name.

Okay, that is looking good.

And so is the app js file.

Okay, great.

All tracing in a civilization should happen before your application code runs.

The easiest way to do this is to initialize tracing and a separate file that required using the node r option before your application code runs, I will show you what I mean by this.

So, now, if you run your application with node, our tracing j s and app j s, your application will create and propagate traces over HTTP.

So let's run that.

And now let's send requests to application over HTTP.

We can do so simply by refreshing the localhost 8080 page, you'll see traces exported to our tracing back end, they look like this.

So he will see we are making a get request to the homepage, which is responding with hello world.

And here we have getting started.

As that is what we called our service name.

We also get a start time and a duration as a span.

Now, as you use this more and more often, some spans might appear to be duplicated, but they are not.

This is because some applications can be both the client and the server for these requests.

If this is the case, you will see one span which is the client side request timing, and one span which is the server side request timing, anywhere that they don't overlap is network time.

Okay, now I'm going to show you one more example.

So just to differentiate, I'm going to change my service name to get date.

Now I am going to go into my app js file.

I'm just going to copy this piece of code right here.

And paste.

Now I want to essentially listen out to any time a get request is made to the park date.

So in other words, if someone right now went to localhost 8080 forward slash date, that is us making a get request, we will also be able to see respond with the actual date.

So let's go ahead, let's go back to our localhost 8080 and type forward slash date.

Oops, I stopped my app running, let's make sure it's running.

So I'm just going to start this up again.

Okay, and refresh our page.

And great, there is our object with today's date.

Amazing.

So now if we visit localhost 9411, so the port we exposed, and click run a query, we will see all the requests that have been made.

So there we go, we can now see our get date service.

And at the moment, the only request we have that it's listening out for is the get request.

Now, I have actually renamed the service, remember so if I visit the homepage, you will see that request is also being stored under the get date service name, you can tell which one it is by the timestamp.

Okay, let's move on.

Okay, we are now done with a basic implementation of tracing.

However, we are literally just touching the surface.

In the project portion of our course I will show you how to use open telemetry to instrument a distributed system.

By this I mean I will show you how to trace multiple services and their interactions with each other if those exist.

In this next section, we're going to learn how to collect metrics with open telemetry and remediate Prometheus as a monitoring platform that collects metrics from monitor targets by scraping metrics HTTP endpoints on these targets.

In this section, I will show you how to install, configure and monitor our fast app with Prometheus and open telemetry.

We will download, install and run Prometheus to expose time series data on hosts and services.

Unlike tracing which works in spans metrics are a numeric representation of data measured over intervals of time.

metrics can harness the power of mathematical modeling and prediction to derive knowledge of the behavior of a system over intervals of time in the present and future.

Since numbers are optimized for storage, processing, compression and retrieval, metrics enable longer retention of data as well as easier querying.

This makes metrics perfectly suited to building dashboards that reflect historical trends.

metrics also allow for gradual reduction of data resolution.

After a certain period of time, data can be aggregated into daily or weekly frequency.

Let's have a look at this in action.

In this section, I'm going to be using promethium as my metrics backend.

Now that we have set up end to end traces, we can collect and export some basic metrics.

First, I'm going to stop this from running.

Let's go to the Prometheus download page and download the latest release.

permittees for your operating system.

As I'm using a Mac, I'm going to click on this one right here.

Once that has downloaded open a command line and use CD or the command cd to go into the directory, where you downloaded the promethease tarball.

In my case, it will be the Downloads directory.

Now I need to unretire it into the newly created directory, make sure that you replace the file name with your downloaded harbor.

So don't necessarily use this one.

And now let's go into the directory.

If I list out all the files and folders, you will see a file named permit this yamo this is the file used to configure permit Yes, for now, just make sure permit is start by running the dot four slash promethease binary and the folder and browse to localhost 9090.

Okay, great.

And that is our promethease user interface.

So you will see this as server is ready to receive web requests.

So that should be all good.

I am just going to go ahead and open up a new tab right here so I can keep that running.

And I'm going to open up our directory using the VS code shortcut.

Once we have confirmed that permit the US has started, we need to replace the contents of the primitives yamo file with the following.

So literally just delete everything and put in this much shorter piece of code.

This will set the scrape interval to every 15 seconds.

We are now ready to monitor our Node JS application.

In this next section, we need to initialize the required open telemetry metrics library, initialize a meter and collect metrics and initialize and register a metrics exporter.

To do this, we are going to have to install some libraries, we are going to need the open telemetry metrics package.

So let's go ahead and install that.

Let's navigate back to our project first.

So don't do this in the directory we just downloaded.

And in here, type NPM II or install Open telemetry metrics.

Great, we are now ready to initialize a meter and collect metrics.

We first need a meter to create and monitor metrics a meter and open telemetry is the mechanism used to create and manage metrics, labels, unmetric exporters, create a file named monitoring j s.

So a JavaScript file and the root of your folder and add the following code.

So we're going to need the open telemetry metrics package for this Kant's.

And we're gonna get the meter provider from it and buy it I mean, the open telemetry metrics package, we are then going to make a new console constant meter.

And we're going to use the meter provider we're gonna make a new meter provider and use get meter, as well as I'm just gonna put your meter name for now, we can change this whenever we want.

Now we can require this file from your application code and use the meter to create and manage metrics.

The simplest of these metrics is a counter.

In this next part, we're going to create an export from our monitoring js file, a middleware function that express can use to count all requests made by route.

So first off, we need to modify IO monitoring j s file.

So once again, I'm just going to copy this sample code from open telemetry open source projects in order to help us count the requests and paste that into my monitoring j s file.

Next, we need to import and use this middleware in our application code.

So our app js file.

So we need to get count all requests from our monitoring js file.

So the module export, we do so by typing const count all requests require monitoring j s.

So literally, we are using this right here.

Now let's get to using it.

Type app use count all requests and call it now when you make requests your service you will meter will count all the requests.

Perfect.

Next up, let's look at initializing and registering a metrics exporter counter metrics are only useful if you can export them somewhere where you can see that.

For this we're going to use Prometheus is creating and registering a metrics exporter is much like the tracing exporter above fast, you need to install the Prometheus exporter by running the following command.

So I'm just going to use npm install and open telemetry exporter permit this next step, we need to add some more code to our monitoring js file.

So once again, I'm going to copy the code given to us by opentelemetry To get started, and paste it into my monitoring js file.

Don't worry, I will share this repo with all of you so that if you do get stuck, you can refer to my finished project.

Now in a separate tab, so just leave this running and show Prometheus is running by running the Prometheus binary from earlier and start your application.

We do so by using the script we wrote.

So NPM start, you should see promethease scrape endpoint and the HTTP localhost 94644 slash metrics, as well as listening for requests on localhost 8080.

Now, each time you browse to localhost 8080, you should see Hello, in your browser and your metrics, and Prometheus should update, you can verify the current metrics by browsing to localhost 9464, forward slash metrics, which should look like this, you should also be able to see the gathered metrics in your Prometheus web UI, we can also add more routes in our app js file.

Let's go ahead and do that to see what that would look like.

So I'm just going to add some pre written code here.

And that is a middle tier route, and another route that has the pot back end.

So now we have our date, homepage from the previous section, our back end route now, and a middle tier route, as well as a new homepage route.

I'm also going to need axios for this.

So another package that will help me in making these requests.

So let's go ahead and import that into my project.

And nice that is done.

Let's run NPM start.

Okay, now let's check everything is working as expected.

The home page now responds with Hello backend.

This is actually because we have two homepage routes.

So I'm just going to get rid of the other one in a bit.

The backend route responds with Hello back end, the date route responds with today's date.

So that looks good.

So I'm just going to delete the initial homepage route that we had that responds with hello world and keep the new one.

Okay, and now let's visit middle tier.

And we get response of Hello backend.

And finally, let's visit matrix where we get a request counter of all the routes we have visited.

Okay.

So this looks like it's all working, we visited all the routes, and I counter seems to be working fine.

Now let's go ahead and see that in the permittees UI.

So we're going to have to pick something to execute.

And there we go.

Now, before we move on to our project, I thought let's take a little bit of time to understand exactly what issues can be detected.

With open telemetry.

Here's just a list that I'm going to go through with you.

Starting with the backend.

In the backend with open telemetry you can pick up bad logic or user input, leading to exceptions being thrown, poorly implemented downstream calls.

So for example to infrastructures like databases or downstream API's, leading to exceptionally long response times.

Or you can pick up poorly performant code on a single API, leading to exceptional response times.

On the front end, with open telemetry, you can detect bad logic or user input leading to JavaScript errors.

You can also use it to find poorly implemented JavaScript, making your UI prohibitively slow, despite performant API's.

And you can even use it to locate geo specific slowness requiring geo distribution.

And finally, for infrastructure, you can use it to identify noisy neighbors running on a host sapping resource from other apps, configuration changes, leading to performance degeneration version audit.

So zero day vulnerability checks, ensuring convict changes went through, or just miss figuration with your DNS making your apps inaccessible.

So that is a list of you thinking of issues that you can detect with open telemetry.

Now that we have that covered, let's move on to our project.

And this part of the course, I want to show you what happens when you are building out an app with a more complicated back end that deals with two services.

It is a hypothetical project that you can adapt to anything that you wish, it is an app that gets movies for your database.

By the end of the project, you will be able to trace exactly how we got the movies, and how long each step took.

Okay, so here is a project I have pre made, thanks to the open source community and inspired by an open telemetry contributor, Alan storm.

It is a project that has two services with one service relying on the other, but not the other way around.

In this project, I have a main dashboard service, as well as the movies service, which will return all the movies for our app.

The layout of this project is similar to what we did in the tracing setup.

However, instead of having a separate tracing js file to trace each service, the code is directly in each file.

So as you can see, each service is in the root of our project.

And just going to minimize this so we can see the code a little better now.

So from the beginning of our course, this should look familiar.

So as a reminder, open telemetry requires that we instantiate a trace provider, configure that trace provider with an exporter and install Open telemetry plugins to instrument specific node modules.

Let's talk through the code.

So we can see here as a refresher, especially as it's organized in a different way than what we saw in the basic implementation.

So first, we are going to get the node trace provider from the open telemetry node package.

A trace provider is what will help us create traces on no Jess.

Next, we will get the console span exporter and the simple span processor from the open telemetry tracing package.

And then we need to get the zipkin exporter from the open telemetry exporter zipkin package.

Now that we have what is necessary, let's move on.

So at the moment, we have made a tracing program.

And to actually generate the spans if you remember, we installed a plugin called Open telemetry plugin dash HTTP, the node a trace provider object is smart enough to notice the presence of a plugin and load it.

This code creates a trace provider and adds a span of processor to it.

The trace processor requires an exporter.

So we instantiate that as well.

Both are responsible for getting the telemetry data out of your service and into another system.

With this code, we create an exporter then use that exporter when creating a span processor, and then added that spam processor to our trace provider.

Okay, so that is what that code does.

Let's actually name our service.

Now, I have left this blank.

So let's go ahead and fill that in.

As this service is going to deal with the main service, I'm just going to call it dashboard service.

Here we're instantiating, a zipkin.

exporter, and then adding it to the trace provider.

We of course need to get Express from the package Express.

And so a server and listen out on port 3000 on one four connections.

The app is currently not going to respond with anything for requests any part as we haven't written anything yet.

I wanted to respond with the dashboard itself and the movies from the movie service.

But before we do that, we need to build out our movies js file.

So this file is exactly the same as the other file just perhaps with some code in different places.

uses a different port.

In this file, I want to deal with the movies.

So I'm just going to rename this service name to movies service.

If I ran this service, we would be listening out to Port 3000.

Now, I'm going to determine how our app responds to a get request to the movies endpoint.

So I am just going to write up, get, and then I'm simply going to put the movies path.

So I'm making this up.

This is an async function, and I'm going to pass through a request and response.

Okay, so there we go.

And then I'm simply going to type rest type Jason as a string.

And response, send Jason stringify.

And I'm just going to send a movie object with, let's say, an array of movies.

So let's put some objects in our array, I'm going to make the array have movie objects.

And each movie object is going to have a name.

So a name like Jaws, a genre.

So for example, jaws is a thriller as a string, and that is my first object.

And then I'm just going to make a quick other object.

This time, let's put a different type of film.

So I'm just going to put the string of Ani, and once again, let's put a genre.

So I'm just making the same movie object just with a different name and genre.

I'm going to put family and make another object.

I'm going to stop after this one, because this is just for illustration purposes.

And let's put Jurassic Park as the name and let's put action as the genre.

Okay, that is it.

That's our array of three movie objects.

Okay, now let's run our app.

So let's actually go to localhost 3000, and put the movies path.

So what I'm doing is I am requesting the URL.

And of course, our app is going to listen out for it and make a trace.

So now let's go over to the zipkin UI and search for recent traces, we will see we recorded a single service trace.

However, if we look through into the trace details, we'll see these traces do not look like the traces we've previously seen.

In our previous examples, one span equaled one service.

However, a span is just a span of time that's related to other spans of time, we can use a span to measure any individual part of our service as well.

The Express auto instrumentation plugin creates spans that measure actions within the Express framework, we can use it to find out how long each middleware took to execute, how long your Express handler took to execute, and so on.

It gives you insights not just into what's going on with the service as a whole, but also individual parts of your Express system.

This is the role most of the contract plugins play in the open telemetry project.

The core plugins are concerned with ensuring each request is bound to a single trace.

But the contract plugins will create spans specific to the behavior of a particular framework.

Okay, great.

Let's carry on.

So now that we've done that, I want to show you how to use open telemetry to instrument a distributed system.

This is what our dashboard js file is for is essentially want my dashboard js file to call the movies service as well.

So let's get to writing that code.

The first thing I am going to do is actually use the node fetch library which we haven't installed yet.

So this service, use the node fetch library to call our movies service.

Let's go ahead and install that.

So I'm just going to get my terminal and type NPM I for install and node batch.

Okay, now, once again, I'm going to have to type apt get, and then use the route of dashboard and then write in async function, so a sick function and pass through a request and response.

Now I am going to write here I need to fetch data from a second service.

And that's the movie service.

I'm just gonna write some pseudocode to remind us of that, fetch data running from second service and My second service is the movies service.

Okay, now, I am going to write a function that will essentially help us get all the URL contact from the movies service.

So essentially our object with the three movies in an array or movie objects in an array.

So let's get to writing this function, I'm going to write this function and pass through two parameters.

So whenever I pass through a URL into this function and a fetch, so the fetch is actually going to use our node fetch library, then I'm going to use these two parameters to essentially get the body of that URL.

So I'm going to use a promise to do this new promise, I'm going to pass through resolve and reject.

And then I'm going to use fetch to fetch the URL.

And then whatever is in the body, I'm going to use so fetch URL, resolve reject, then Russ, Russ text, then body.

So that is my function for getting the URL content.

Let's get to using that.

Okay, I'm just going to actually change this.

I don't like the way this is written.

And I want to make it consistent with the bottom.

So I'm just going to change this up.

So it looks a bit neater is just a different way of writing functions.

So just so it's consistent to that.

Okay.

So once again, let's go down to the code I've pre written.

And in here, I'm going to fetch data running from the second service.

So the movies service, I am going to actually save the contents of the URL as movies.

So const movies, await and use the function that we pre wrote to pass through the URL.

So the URL we want to get the content of is HTTP localhost 3004 slash movies.

So there we go, that is ju L.

It is the same URL that I have written here.

So URL U, r, l, that's what we have done.

And then I'm going to need to require node fetch.

So I'm going to look through right require and the package node fetch, I'm going to put rest type Jason, and rez send Jason stringify.

And now I'm going to write the word dashboard.

Okay, so I'm making an object, and I'm gonna write dashboard, and then whatever we've saved as movies should show up here.

So essentially, the contents of the URL will show up here.

Okay, so now I cannot run this file.

So I could run this file, but we will see an error.

This is because our file relies on the movies service being up and running.

So I'm just going to show you now I'm going to type node dashboard, j.

s.

So it's listening at localhost 3001.

However, if I visit localhost 3001 for slash dashboard, I get an error.

This is because we need our movie service to be running.

So let's go ahead and make that true.

I'm just going to open up a new tab and type node movies j s.

Okay, that is not running and listening at Port localhost 3000.

So now, here's our movies.

And let's refresh or let's rerun the dashboard.

So once again, node dashboard, j s.

And then refresh our page.

Amazing, we will see our dashboard object with the contents of the URL from the Ford slash movies path.

Amazing.

So this is working fine.

Our code is working as it should.

Now, let's see how this looks in our zipkin UI.

So I'm just going to rerun the query and look at our latest.

And there we go.

So once again, to reiterate that, in a nutshell, the dashboard service is dependent on a movie service to populate it.

This is true for many apps you interact with today, and which is why this example is the one I wanted to show you for our project.

Now, we can see here that each span from the services are linked together.

The opentelemetry HTTP plugin took care of this for us, the node fetch plugin use the underlying functionality of node j, s, HTTP and HTTPS to make requests.

So that's how to instrument an application using open telemetry.

Like this is pretty cool, as obviously, you can see our dashboard service, and then you can see exactly what time it's going to the get movies, service, and then coming back.

Okay, that brings us to the end of our project.

I hope you've enjoyed this section.

This is, of course, just the surface of what you can do the opentelemetry.

There is a lot more to it.

But until you get the fundamentals, I sincerely hope you can go through this course again and again until you feel more comfortable building your own projects.

So far, in this project, we have directly gotten data from an app and send the data to zipkin.

But what happens if we want to try sending it to another back end to process our telemetry data? Does that mean we would have to re instrument our whole app? Well, the amazing contributors to open telemetry have come up with a solution to fix this.

The open telemetry collector is a way for developers to receive process and export telemetry data to multiple backends.

It supports open source observability data formats like zipkin, Jaeger permittees, or flume bit, sending it to one or more open source or commercial backends.

In this next section, I'm going to show you how to use it.

Okay, so for this section, and using the open telemetry collector, we're going to be using New Relic as our observability tool of choice.

All I'm going to do is head over to New Relic and sign up.

Next, you'll see some questions, please answer these to the best of your ability.

So for example, where you store your data, and just click save, your account will then be set up.

Once you get to this page right here, I'm just going to ask you to not interact with anything for now, and head over to one dot new relic.com.

Once here, I'm going to ask you to go on the drop down of your profile and click API keys.

Once here, I'm going to ask you to gravitate to creating a new queue and just select ingest license.

I'm going to name this otol example.

And I'm just gonna give it some notes just so we can keep track of our API keys.

And great, our API key is now created.

Let's copy it and move on.

The next thing we're going to do is actually get our open telemetry collector.

For this, I'm going to head over to New Relic GitHub account in which I can get the open telemetry examples.

So I'm just going to ask you to clone this repo into your local machine.

I have already done this.

So I'm just going to go ahead and head over to that repo now.

And here it is.

Once here, I'm going to ask you to navigate to the collectors and our exporter Docker otol config yamo file, because we're gonna have to change this up a little bit.

So please do head over here now.

And I'm just going to ask you to add a little line of code.

This line of code will add zipkin as a receiver, a receiver is aware that data gets into the open telemetry collector.

So because we already have our app configured to use zipkin, we will be telling the open telemetry collector Hey, we will be sending you data in the form of zipkin.

So that is what is happening, we are adding zipkin as a receiver and then giving it an endpoint, which in this case is 0.0 dot 09411.

Now because zipkin report tracing data, we are going to add zipkin as a tracing receiver underneath service.

And that's it.

The next thing we need to do is go over to the Docker compose yamo file, and make sure that the Docker container that runs this open telemetry collector is actually able to receive the data through the same port that it would have if it was zipkin.

So we're going to add the Port 9411, just like so and save it.

Now, according to the readme to run this, we need to use the API key we just created.

So in my terminal, I'm just going To export the New Relic API key I just created with this command.

Next, we need to spin up the Docker container with this command, making sure of course that we are in the correct directory.

So that is my fault.

Let's go into that directory.

So the nr exporter Docker.

And once again, we run the Docker container.

and wonderful.

Now let's go back to our movies dashboard project that we have been working for in this course.

So now I just need to modify our app ever since Thirdly, to work with the open telemetry collector.

For this to work, we need to change two things, we need to change the URL that is reporting at.

So this one right here, I'm simply going to use it like so.

So we need to do is for the dashboard and also for the movies service.

And that's it.

Now, I'm just going to reinstall all the dependencies for those who are just joining us here and I've taken this project from the description below.

And then let's run the two services, like we have been doing previously in this tutorial.

Wonderful.

Okay, now I'm just going to call the services by visiting the dashboard service.

As a reminder, the dashboard service relies on the movies service, I'm going to call it multiple times, so we can get lots and lots of data to work with.

So maybe just a few more.

Okay, done.

Let's move on.

Now that we have our app and have successfully instrumented the open telemetry collector that is forwarding our data to New Relic, we should now be able to visualize our data.

For this, we need to go to the Explorer tab on New Relic.

And once here, we will see the two services, the dashboard service and the movies service, just like in the previous distributed tracing section.

Let's deep dive further.

As you can see in the dashboard service, that was a spike.

Let's find out what was happening.

So it looks like there were 18 traces with nine spans and two entities for the dashboard service.

That sounds right.

Great.

And if we dig deeper, you will see the micro service that our dashboard service was communicating with to actually solve the final result.

Wonderful.

As you can see, we can get a lot of awesome data to do things such as get to the root cause analysis of what could be going wrong in your app, check how your microservices are performing, and so, so much more.

Okay, and there we have it.

That was our open telemetry course, before we finish, I just want to take a moment to recap what we have learned in this course.

So to recap, in this course, we learnt how to set up the back end for a project.

Then we learnt how to implement tracing into our project, as well as metrics if we need to.

And then we also looked at two services and how they communicate.

Thanks to distributed tracing.

I hope you now feel comfortable knowing the benefits of using open telemetry, as well as have a good understanding of how you would go about implementing it into your Node JS projects.

If you are wondering where to go next, with your newfound knowledge, I would suggest learning about infrastructure monitoring and digital experience monitoring.

These are two other ways to visualize, analyze and troubleshoot your entire software stack.

I will leave you with this and a link to New Relic To find out more and get a free account with no expiration date.

Thanks so much again for watching and I'll see you soon

If you are building products, you want to know how people are using the products, how they're hearing about what you are building, and ultimately how they reach you. The Search Console plays an important role in helping you track and measure these operational metrics.

The Console is free to use. You just have to prove to Google that you own a website. Then Google will give you access to this treasure trove of search data.

I am going to walk you through the Google Search Console’s core features and describe each in plain English. I hope this helps you leverage it so you can measure the data points that matter the most to your product.

Google Search Console: Features and Core Functionality

The Google Search Console is a web service by Google which allows webmasters to check indexing status and optimize visibility of their websites. Below is a screenshot of the main part of the Search Console so that you can see a visual layout.

The main tabs on the Google Search Console interface

There are five different areas of search performance that the Console enables you to to quickly see and dive deeper into. These 5 areas are:

1. Queries
2. Pages
3. Countries
4. Search Appearance
5. Dates

I want to provide an outline of each. I will then explain how the performance and rendering of your website in each field will tell you a story about your website’s traffic and user experience.

How to Use Queries in Google Search Console

Builders operate at all levels, stay connected to the details, and audit data frequently. Before diving deeper into the Console, a builder will want to know what people are typing into Google.com to reach the landing page or website.

In other words, what are the specific phrases, words, or queries that people are typing in when looking for content pertaining to your page? What is the search intent?

The Console enables builders to see the exact queries that users are entering in order to reach your website. Information pertaining to queries is hugely important as it enables you to see what people are actually typing into their search engine before reaching your page.

Knowing what people are searching for is a great guide for allocating resources or optimizing your pages for what people want to learn about.

How to Use Pages in Google Search Console

Pages is the section of the Console that enables you to see which of your pages are ranking, the number of clicks each page gets, and the total impressions that page has earned. By default, Google lists your most trafficked pages first.

If you don’t know which pages are receiving the most traffic you won’t know how people are finding out about your website. Pages in particular can be revealing as you might think that your website is about topic X but most people are finding out about you by landing on page Y.

When I first built a work from home website, I wanted to build an all purpose site for helping people work from home.

Much to my surprise, the pages that were ranking best, including quad monitor stands and red office chair, struck me as somewhat esoteric.

Nevertheless, that is what people wanted to read about and where they were spending their time. Without the pages section of the Console I could not have gleaned that insight.

You can also use The Console to compare your site’s performance across different devices and operating systems. If you wanted to learn whether or not specific pages are more relevant to users on their mobile or on their desktop, there is data for that.

It would be interesting to discover if you have a lot of users coming from Gmail via their desktop app, for example, and then to see if they spend more time on the page than gmail users who are on their phones.

How to Use Countries in Google Search Console

This section will matter most to people who expect and want to drive traffic from different geographies. If you are an education-based website, for example, you might want users from all over who can access your content.

Where people are coming from can influence the language of the pages you show or help you better interact with your users. If you are a website that sells a physical product, you might value traffic from places you can actually ship to.

The countries tab is useful to confirm your hypothesis that your traffic comes from certain locales.

How to Use Search Appearance on Google Search Console

Some of your traffic will come from standard queries, other sources of traffic will come from video, or Google's "Accelerated Mobile Pages," more commonly known as AMP. The appearance tab captures how Google classifies the appearance of your content.

This tab is the least important in my perspective, because you likely won’t change your behavior as a result of what you see here.

However, if you are practicing agile product development and want to iterate and see how each change impacts search results, this tab will be a helpful guide.

How to Use Dates in Google Search Console

The last tab is the most self explanatory and pertains to time.

You can track your organic search results by day or see trends by zooming out to monthly, quarterly, or annual performance. In general, it is helpful to take a long term perspective on search performance, as anything you do to improve your website will most likely take many days or weeks for Google to index.

Google is looking for websites that are performant, useful to users, fast, responsive, and well structured. If your website affords users these benefits – and drives user engagement over time – your traffic will likely increase and the dates tab will help you see this visually.

Conclusion: Leverage the Search Console to Build Better Products

The Google Search Console is a powerful and necessary tool to review, leverage, and explore core operating metrics for your website. The Console will help you understand how users hear about your site in organic search and which pages they visit on your domain.

By measuring these data points you can improve your website by enhancing individual pages, improving the quality of your search appearance, and seeing how you are improving over time.

Google provides access to the console at no additional cost so that you can build and scale without paying per query.

The console is very powerful and it's a useful tool to leverage when building and deploying any site on the public internet.

Google Search Console is a web service by Google that lets you see the indexing and performance of your websites and webpages on Google search.

At a high level, the search console is a powerful tool to confirm that your website is ranking and that Google can access your website.

You can check and set the crawl rate and view statistics about where your traffic is landing and where you are acquiring traffic from.

The console, like many other web dev tools, is robust and powerful. If you are building your first website you should take time to understand it, play around with it, and learn to leverage it.

This tutorial walks you through the basics of this tool, and can be a useful guide to help you save time when learning which metrics matter for your site’s performance and how you can leverage this data to build better products.

I have lived in the search console for the past six months because I built a website and I care deeply about how people find the content that I write.

Prior to my experience building this site, I was not a regular user of the Search Console. I had to learn about this tool, including what it could do to help me better understand my users and their intent when landing on my website.

I have learned three important lessons about the Search Console’s Performance section and want to pass those lessons on to you.

Lesson #1: Know the source of your traffic: Search vs Discover

Google gives you two ways to see how people are reaching your website through Search: via Search results or via Discover.

“Discover” (previously known as Google Feed) is a personalized content feed created by Google that proactively serves relevant content to users. You can’t nominate your content for Discover. If Google thinks that your website is relevant to users, Google will show it to people.

The other type of performance metric, and one that is seemingly far larger, is “Search results”. Search results shows a webmaster four vital pieces of information:

1. Total clicks
2. Total impressions
3. Average CTR (Click Thru Rate)
4. Average Position

Discover provides the same information with the exception of Average Position because there are no relative ranking positions within this metric.

When learning to use the Search Console, spend time looking at traffic through the lens of Search and Discover. If you do not see Discover traffic, that means your content has not yet been shown via Discover to users.

When starting a website or blog, you will want users to find your content. Search and discover are the two organic ways that people will do so. Know how many people find you via each.

Lesson #2: Study your average position

When I first started using the Search Console I was amazed (and happy) that I was getting any traffic at all! Over time, your clicks will increase if you are publishing relevant and useful content.

As more people find your website and the site gets more impressions, clicks will go up.

You might be tempted to pat yourself on the back and call it a day. Not so fast!

Each page that you publish on your website is eligible to appear on Google search if you want that page to be crawled and indexed.

Some of your pages will perform very well (that is, they'll land on the first page of Google) and others will land further down (like on the 12th page where nobody will see them).

Over time you will want to monitor and track your average position and find ways to increase your average position by producing higher quality and more relevant content that benefits readers.

Here is a simple strategy to see your average position over time: create articles of similar length, quality, and usefulness to readers.

Then, come back over the ensuing weeks and months and see which ones are starting to rank and what their relative positions are. This strategy is deployed by review sites, thought leadership posts, and even online coding schools.

When I started my first website my average position was around 60. That means that 59 other websites showed up before mine.

Today, that average position is 26.8. Clearly I have a long way to go but it's a step in the right direction. Be aware of this metric and spend time tracking it.

Source: https://wfhadviser.com/

Lesson #3: Impressions are the top of the funnel

Regardless of the type of website you have - blog, educational, e-commerce, recreational - you will want traffic.

Traffic is just another way of saying unique visitors. Depending on what your website has to offer, you might value certain types of traffic more than others: by region, country, age of users, device operating system, and so on.

But as you focus in on clicks and performance, it all starts with impressions. An impression is counted each time your webpage is shown on a search result page.

In other words, impressions are the top of the funnel. You will need to grow impressions first in order to grow all other metrics.

A word of caution that I have discovered the hard way. If your website is on the lower part of the search results, an impression might count even if the user doesn’t scroll all the way down.

In other words, the way Google counts an impression is seemingly slightly different from how a person might count it. In layperson's terms (that is, not technology) an impression is when you actually see something.

In Google’s terms, an impression is when your website is on a results page and the user sees the results page. In short, scrolling doesn’t seem to impact impression counts.

Bringing It All Together: Know How Your Users Reach Your Website To Build Better Products

If you want to build a product, you always need to know how your users learned about your product and what they value most. Builders start with the customer and work backwards.

Google’s Search Console is a terrific tool to understand one aspect of the user journey to better align your product, your messaging, and your value-prop with the needs of your clients.

You can use the Performance section of the Search Console to obsess over customers and how they reach you.

Google makes it easy to test this in real time so that your feedback loop is minimal: you can review the crawled page automatically or promagically.

Growing SaaS, a company that notes that SEO is one of the core user acquisition strategies for companies, also convincingly argues that if you don't know where, what, how, or why to measure traffic, you will be missing guidance as to what to build and for whom.

If you build any product - a website, a mobile app, or a tool for Enterprises - you need to know how people hear about your product and reach you.

Imagine if you built a newsletter but had no idea how your email list grew over time or what your subscribers valued most? Sounds crazy, right? That is akin to building a site and not mastering the Search Console.

Certainly this would be a suboptimal situation for any coder or creator.

Builders are never done learning.

The Search Console gives you data and visual guidance on ways to improve your website and customer acquisition. By being curious about new possibilities and acting to explore them, you can do a better job building.

And Google reinforces this by helping guide you along the way. These green circles and checks confirm that the site has live pages and that are properly indexed. It's always a nice plus when the visual design helps the builders :)

Builders not only index on what is going well - and how to improve these trends - but what is not working so that they can course correct. Google’s search console, like other online tools, helps users see broken links and error pages, pages that need improvements, and site speed.

If you were looking at hundreds of pieces of paper in a book - or bulk emails - or thousands of indexed web pages, you need to leverage a tool to understand the data in front of you.

The Search Console is just that.

It is free to use and fast to deploy. It is important to use and critical to master - your product and users will be better served if you leverage it.

The story began while I was optimizing a landing page on my Bootstrap IT website for my new book, Keeping Up: Backgrounders to all the big technology trends you can't afford to miss.

I wanted to provide access to the PDF file of a sample chapter from the book. But I also wanted some way to know how many people actually downloaded it.

Now let's take a step back. Google Analytics is a free service that uses code snippets inserted into your HTML files to collect and display data on how often your files were accessed.

The magic - and problem - of Google Analytics is in just how much information about your users can be revealed. I discussed some of the privacy concerns involved with the service in the Keeping Up book. I also mentioned how I feel at least a bit guilty for using the service myself on my own sites.

At any rate, all by itself, Google Analytics isn't able to tell you much about how your web-based PDFs are being used. Of course, there are tricks for getting around the problem.

Traditional approaches include setting up the Google Tag Manager, customizing the syntax of the request URLs you use or, if your site uses WordPress software, working with the Monster Insights plugin. Each of those can work, but will require a fairly steep learning curve.

But I'm a Linux sysadmin. And, as I never fail to remind the people around me, the best sysadmins are lazy. Learning curve? That sounds suspiciously like work. Not gonna happen on my watch.

So here's the deal. My web server, obviously, runs Linux. And, under the hood, HTTP traffic is handled by Apache. That means that everything happening to and on my websites is going to be logged by Apache.

Everything. All it'll take to give me what I need to know about what my PDF sample chapter has been up to, is a single line of Bash run from my local workstation:

echo "cd /var/log/apache2 && grep -nr KeepingUpSampleChapter" \
   | ssh -i PrivateKey.pem LoginName@bootstrap-it.com

Let's break that down. The first of the two commands in quotation marks (cd /var/log/apache2) will move us to the /var/log/apache2/ directory on the Linux server where Apache writes its logs. That's not rocket science.

There are going to be multiple files of interest in that directory. That's because messages relevant to regular access and errors are saved to different files, and because file rotation policies mean that there could be more than one version of either of those files, too. So I'll use grep to search all the uncompressed files for the KeepingUpSampleChapter string. KeepingUpSampleChapter is, of course, part of the filename of the PDF.

I then pipe that command to SSH, which will connect to my remote server and execute the command. Here's what a single entry would look like from a successful run (I've removed the requester's IP address out of privacy concerns):

other_vhosts_access.log.1:12200:bootstrap-it.com:443 <requester's IP Address> - - [01/Dec/2020:16:39:36 -0500] "GET /davidclinton/KeepingUpSampleChapter.pdf?pdf=SamplePDF HTTP/1.1" 200 65146 "https://bootstrap-it.com/davidclinton/keeping-up/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"

We can see:

• The log file where the entry appeared (other_vhosts_access.log.1)
• The requester's IP address (redacted)
• The timestamp telling us exactly when the file was accessed
• The relative location of the file on the server file system (/davidclinton/KeepingUpSampleChapter.pdf)
• The URL from which the request was made (https://bootstrap-it.com/davidclinton/keeping-up/)
• And the browser the user was running

That's a lot of information. If we're just curious about how many times the file was downloaded, we can simply pipe the output to the wc command that will tell us three things about the output: the number of lines, words, and characters it contained. That command would look like this:

echo "cd /var/log/apache2 && grep -nr KeepingUpSampleChapter | wc" \
   | ssh -i PrivateKey.pem LoginName@bootstrap-it.com

There is one possible limitation with this method. If your website is busy, the log files will roll over frequently, often more than once a day. By default, after the first rollover, the files are compressed using the gz algorithm, which can't be read by grep.

The zgrep command won't have any trouble handling such files, but the process could end up taking a very long time. You might consider writing a simple custom script to decompress each gz file and then run regular grep against its contents. That'll be your project.

There's much more administration goodness in the form of books, courses, and articles available at my bootstrap-it.com.

As a content creator, I like to know the page analytics of my website.

Overall, I am curious to know how many people are reading my content, where they came from (referrer and countries), and what the most popular pages are.

20 years ago, tools like Webalizer were all that we could count on. Tools like this parsed Apache logs and created static pages with the processed data.

Another way to get page analytics was to insert an image - often invisible - on your site. By using the request headers sent to the server, people could count visitors and get a little bit more information such as origin IP, browser, and operating system. This technique is old, but services like statcounter still provide this functionality.

In 2005 Google launched Google Analytics after acquiring Urchin, a company which also analyzed server logs. Its presence has been growing since the early days and it has far more usage than any of its competitors. But, there are a few reasons why you should stop using Google Analytics on your website:

1) It is owned by Google, which uses analytics for their own benefit 2) It affects site speed by adding 45KB to page requests 3) It is too invasive, collecting lots of personal data that you don't need 4) It is blocked by many plugins and browsers, creating inaccurate data

With all this in mind, I want to share a few open source alternatives I have been looking at for the last few months.

Fathom

Fathom (demo) is a light Golang app to collect analytics.

They have different paid plans which start at $14/month. They also have a lite version that you can install on your server or Heroku for free.

The lite version uses cookies and it gives you information about unique visitors, page views, average time on site, bounce rate, top pages, and top referrers. Fathom stores data in SQLite, MySQL, or Postgresql databases.

umami

umami (demo) is a solution created with Next.js, making it very easy to deploy. In my case, used Vercel.

On top of unique visitors, page views, average time on site, bounce rate, top pages, and top referrers, umami also shows you information about countries, browsers, OS and device data.

Plausible

I think I first heard about Plausible (demo) in the "De-Google-ing your website analytics" Changelog podcast. From a product perspective, it is nice to see a public roadmap out in the wild so customers can learn what is coming next.

Their plans start at $6/month and go up according to your page views - like Fathom. They also have an alpha self-hosted option, but I didn't have a chance to test it.

Conclusion

There are alternatives out there, and you don't need to worry about a big corp looking at you or your users with these options. Each service's setup time is very similar, and once you're done, you can add multiple sites just like you would with Google Analytics.

I don't have a favorite. Feature-wise, umami provides all the basic information you would want to know for free. It's also very easy to set up on services like Vercel or Netlify.

Both Fathom and Plausible offer free trials so you can easily test their solutions before deciding.

Do you know another minimalist, open source alternative to Google Analytics? Let me know in the comments section of my blog.

If you liked this article, follow me on Twitter and GitHub.

Cover photo by Markus Winkler/Unsplash

Good news - we now have a brand new set of standards by which to judge our search engine's worthiness! ?

If you're like me, you may not have been jumping for joy when you read Google's announcement of its upcoming search algorithm change. But after taking some time to breathe, I believe it's a positive change.

The announcement emphasizes web page experience and its role in the future of search indexing. By following this new direction, we can not only provide a better experience to website users, but also establish effective strategies to improve SEO.

What are Web Vitals?

The following metrics encompass Web Vitals as defined at the time of this writing.

• First Contentful Paint (FCP) measures the time from when the page starts loading to when any part of the page's content is rendered on the screen.
• First Input Delay (FID) measures the time from when a user first interacts with a page to the time when the browser is able to respond to that interaction.
• Largest Contentful Paint (LCP) metric reports the render time of the largest content element visible within the viewport.
• Time to First Byte (TTFB) is the time that it takes for a user's browser to receive the first byte of page content.
• Cumulative Layout Shift (CLS) measures the sum total of all individual layout shift scores for every unexpected layout shift that occurs during the entire lifespan of a page. To calculate the layout shift score, the browser looks at the viewport size and the movement of unstable elements in the viewport between two rendered frames.

Why are Web Vitals Important?

In recent years, Lighthouse, an open-source automated tool for improving the quality of web pages, became widely adopted as an industry standard.

Now another Google project called Web Vitals has emerged, deriving metrics from real users in a way that accurately matches how they're measured by Chrome and reported to other Google tools.

With it, we can establish page experience perspective from an SEO point of view, analyze, and adjust accordingly. ?

Core Web Vitals are the subset of Web Vitals that apply to all web pages, should be measured by all site owners, and will be surfaced across all Google tools. Each of the Core Web Vitals represents a distinct facet of the user experience, is measurable in the field, and reflects the real-world experience of a critical user-centric outcome.

Web Vitals in Google Search Console

Search Console provides reporting of how real users are accessing a website and a variety of data about these users.

Core Web Vitals are reported as a summary showing the total number of URLs that are good, need improvement, or are just poor. ?

Google Search Console Core Web Vitals

Sending Web Vitals to Google Analytics and Visualizing in Data Studio

Search Console provides a summary of results in the grand scheme, but in order to get detailed reporting we can take this a step further. The Web Vitals GitHub project documents a way of capturing metrics as analytics events that can be visualized as charts in Google's Data Studio.

Disclaimer: I haven't personally been able to wire analytics Web Vitals events to Data Studio, and the documentation is lacking at this time. But I'll update this post once I can put together an example.

Visualizing and Analyzing Web Vitals in Real Time with Automated Lighthouse Check

Automated Lighthouse Check Web Vitals Demo

Google Analytics and Data Studio are powerful tools that provide great insight. And best of all, they are free!

Automated Lighthouse Check is a website that monitors websites with Lighthouse and now offers a Web Vitals implementation. You can embed a JS snippet on your website and start collecting Web Vitals metrics in real time.

One advantage of this tool is its simple setup process and easy filtering. You can filter data by URL as well as browser, OS, and device.

Conclusion

The road to SEO success is a winding one, but fortunately we now have a more concrete set of guidelines. If your goal is to achieve high ranking on Google's search engine, it's a good idea to utilize the tools and projects Google recommends including Lighthouse and Web Vitals.

• Why is SEO important?
• How is SEO important to YouTube?
• And what is Google Analytics?
• How do I connect my channel?
• What will I be able to see?
• What won’t I be able to see?
• What else can I do with YouTube and Google Analytics?

Why is SEO important?

SEO, or Search Engine Optimization, is the practice of writing and organizing content in a way that search engines like Google can crawl and ultimately understand what your website or YouTube channel is about.

Using this information, Google and others make decisions with their algorithms to determine which content is of higher quality, more relevant, and more likely to answer the question you’re looking for on their search engine in the first place. With that information, the search engines rank this content and display their results ordered by those rankings.

How is SEO important to YouTube?

Just like any other website, YouTube gets crawled by Google and other search engines. Additionally, YouTube has its own internal search that will take these same things into consideration when deciding how to display results on a YouTube search.

Searching for "code" channels on YouTube

This means, depending on how well you create your descriptions, manage your keywords, or name your videos, it could impact how well your videos rank in the results. And this can impact how many views your videos get.

This also applies to your channel. You have opportunities to experiment with effectiveness through the content you feature, your channel description, and the name of your channel.

And what is Google Analytics?

Google Analytics is a free analytics tool from Google that will allow you to gain better insights into your traffic. I previously wrote about what is Google Analytics and how you can make sense of it which provides a more in depth view. So if you want to learn a little more before diving in, I highly recommend starting there.

How do I connect my channel?

Setting up a new tracking code

To start, we’ll need a tracking code from Google Analytics. Google has some great up to date resources on how to do this, so I'm not going to try to re-explain here:

• Setting up a new property
• Getting your Tracking ID

Though some say you can use your website’s property and create a filtered View, I recommend starting with a separate property. That way you don’t have to worry about any data crossover or setting up complicated filters.

Tracking ID in Google Analytics

Your tracking ID will be in the following format: UA-######-#. Once you have that, we're ready to go.

Adding your tracking code to YouTube

There are a few steps we have to navigate through to find where we can set up our Google Analytics account. If you want to skip to the right place, you can visit youtube.com/advanced_settings.

To take the long route, which will also help you get a little more familiar with your YouTube account, first head over to the Settings section from within your YouTube Studio page.

Finding Settings on your YouTube Studio dashboard

Once selected, find the Advanced channel settings link by visiting Channel, Advanced Settings, and then scrolling down to the bottom.

Advanced channel settings on YouTube

Finally, scroll down to the bottom of the page again, find the Google Analytics property tracking ID field, enter the tracking ID you created, and hit save.

Setting the Google Analytics property tracking ID for your YouTube channel

Sit back and wait

Google Analytics will only show your website traffic from the point it was set up and through the future. Unfortunately we can’t check out that weekend your video first went viral if you didn’t have Google Analytics set up then, but at least we’re prepared for the next time!

Joey and Chandler reclining their chairs

That said, now's the time to continue working hard on your channel since you have the ability to track how that hard work is paying off as people visit your channel.

Optional: Setting up Site Search

Setting up Google Analytic’s Site Search feature gives us an easy way to separate out search usage to make it easier to gain insight into how people are searching our channel.

To enable Site Search, we want to go to the Admin section of our Google Analytics property and then navigate over to View Settings. Once there, under the Site Search settings at the bottom, first click the button to toggle on Site search Tracking, then type “query” into the Query parameter input.

Site search Tracking in Google Analytics

Optionally, though recommended, you can select to strip query parameters out of your URL. This means that in your main content view, you will see all traffic as /search instead of many instances of /search?query=[keyword], which can be more cumbersome to analyze.

Note: before you set this up, it’s generally recommended to have more than one view for your property. I would recommend having at least 2 views, a Raw Data view and Main view. You would only apply the Site Search feature to your Main view. This will help to make sure you can always see the unfiltered Raw Data view if you want.

What will I be able to see?

How many people visited my channel?

The first thing we get immediately with our new data when we open up our Google Analytics property is how many people visited our site.

Google Analytics Home

The default here is in the past 7 days, but you can change the time range in the bottom left corner of the panel.

What this also provides is a quick insight into how the number of people has changed since the previous period (the 7 days before in this example). As we can see here, the number of people this week has increased by 13.9% which is awesome news for freeCodeCamp’s YouTube channel, proving whatever they did is working.

How are people finding our channel?

So how do we figure out if the strategies we’re using (like SEO) to get people to our channel are effective? By analyzing our organic search traffic.

By navigating to the Source/Medium report by visiting Acquisition, All Traffic, then Source/Medium, we can see what sources are providing our YouTube channel the most traffic.

Source/Medium report in Google Analytics

By clicking in to google / organic, we can also see how this has changed over time.

Organic Google traffic report in Google Analytics

While analyzing a single week isn’t the most effective, being able to tell how your organic traffic has changed over multiple weeks will be able to tell you if your strategy is working.

What websites and pages are people coming from?

Navigating to the Referrals report by going to Acquisition, All Traffic, and then Referrals, we can see that most of the referral traffic for the freeCodeCamp YouTube is from freeCodeCamp.org itself.

Referral traffic showing freecodecamp.org as highest referrer in Google Analytics

But say we want to see what pages those referrals are coming from. We can find this out by clicking on the freecodecamp.org link in the view above where we can see a full breakdown of which pages are giving the channel the most traffic.

freecodecamp.org referral pages on Google Analytics

What are people searching for on my channel?

After setting up Site Search on your Google Analytics account, you’ll be able to get better insight into how people are actually searching your site.

Search Terms report in Google Analytics

Here we can see what keywords people want to see the most, meaning we can tailor our content and future videos to those keywords, making our channel more effective.

More insights

By default, you’ll get a lot of other cool insights from Google Analytics that are baked in like where your visitors are physically located and whether they’re visiting on a desktop or mobile device.

To learn more about what you can see, check out my article on making sense of Google Analytics.

What won’t I be able to see?

While the information you’ll discover through Google Analytics is important, it’s not all inclusive. There are many points you’ll need to dive into YouTube’s own Analytics tool to see.

Analytics dashboard in YouTube Studio

Video analytics

Video states and actions aren’t going to be visible in Google Analytics, which includes things like Play, Pause, and time watched.

However, by using the Engagement tab in the YouTube Studio Analytics section, we can see how long people are watching our videos and a graph of the Audience retention. This will help us determine how the content of our videos is performing.

Subscribers

You’re not going to be able to gain insights into how the visitors on your channel are subscribing.

The good news is you can find this by visiting the the Analytics section in your YouTube Studio page, then clicking the Audience tab at the top.

Dig in to YouTube Studio Analytics

There’s a whole lot you can find out if you dig around YouTube Studio Analytics. Take the time to poke around both Analytics report solutions and learn what information is most useful to providing an impactful experience for your channel.

What else can I do with YouTube and Google Analytics?

Track links from YouTube to your website

If you have a website outside of your YouTube channel and have Google Analytics set up on it, you can build custom URLs that will allow you to see your YouTube traffic as a campaign. This is useful not only for YouTube, but any other source you’re directing traffic to your website from.

Building campaign URLs Campaign URL Builder

Google Analytics provides this capability using URL parameters attached to the links. You can learn more about the setup and what you need to do with Google’s Analytics Help site.

It should also be noted that you don’t really need to set up your YouTube channel with Google Analytics to make use of this feature.

Track how videos are watched when embedded on your website

YouTube provides an API that developers can use to write custom JavaScript and track usage of embedded videos on a given website.

Using this, we can send custom events based on time references or video actions (like play and pause) to get a better idea of how the videos on our site are being used.

To be clear – this is only for videos embedded on your website and will probably track usage with your website's Google Analytics property unless you configure it otherwise.

Check out YouTube iFrame Player API for more info.

Pretty much anything Google Analytics provides by default

There’s a whole lot you can do with Google Analytics, whether it’s getting better visibility into where people are coming from or where they’re physically located. And by connecting your YouTube channel you automatically get those insights.

The more resources, the more insight you can gain

Though there are benefits to both YouTube Analytics and Google Analytics, having more information will ultimately help you make better judgement calls about how you manage your channel and content. Use these tools to help launch yourself to inevitable YouTube stardom!

Sandy wrangling

What is Google Tag Manager?

If you’ve ever worked with any kind of analytics software, or you have worked with a marketing team before, you’ve probably heard the word pixel thrown around. A pixel is literally what it sounds like: a 1x1 image that sends information to a server through an image request.

Google Analytics pixel request

Though pixels are still common, many teams have moved towards small javascript snippets that sit right along side the rest of your HTML. They will allow software like Google Analytics to run their own scripts on your page, sometimes even with a pixel backup, in the event that a browser doesn't run JavaScript.

Crazy Egg snippet

These pixels and snippets work great. But when you’re dealing with a bunch of them and they all use the same data, it might seem like you’re just adding to an unmanageable mess of single use code snippets that never feel like they’re in the right place.

Google Tag Manager, or GTM, is a software solution to manage these pixels and snippets for you. To start, GTM works pretty much like any of those other code snippets, as it’s a code snippet itself. But where it shines is that you get to manage the rest of those pixels and snippets along with the flow of data inside GTM, leaving it to be the only snippet to manage within your code.

Why do I want to use it?

Fewer code changes, Fewer deploys

Most of the time, if you’re managing these snippets in your code, each change will require another merge request and another deploy to get the changes out. Not only does this add more risk, as you need to make yet another change to the code, but this is additional time spent dealing with your deploy pipeline and making sure everything is working as it should be.

GTM allows you to break outside of that flow, giving you more flexibility to get changes out that could give some needed insight into fixing some UI errors or could add a few bucks to the bottom line.

More manageable data flow

Wrangling all the different variables throughout your application, making sure they’re all available in the right spot, and keeping up with avoiding breaking any changes can prove to be challenging. This also makes your code more fragile and prone to breaking.

Google Tag Manager utilizes what they call a dataLayer, which essentially functions like an array of events that it listens to. This allows you to push or seed new data to make available within GTM itself. And it means you’re funneling all of your variables to one spot in the code. This lets whoever manages GTM do the rest – they can use that data with peace of mind, knowing that your data flow won’t break after forgetting to update 1 out of 10 spots.

Ability to give marketers a little more access

This is an opportunity to free up some of your time by letting the marketing team deal with tag changes themselves rather than you being the middle man for them. Perhaps they need to adjust the ID for your analytics software or maybe they want to add Crazy Egg. With the right permissions (explained later), they can do all of the work and send the changes to you for review before publishing them.

How does this relate to Google Analytics?

There is no direct relation. But along with Google Analytics, GTM is another tool in the Google Marketing Platform and gives you the immediate ability to more easily manage your Google Analytics installation. GA comes with prebuilt tags inside GTM that make it a breeze to set up.

What else can I use this for?

Google Analytics is just one of many code snippets this can be used for. Ideally, you’re not going to write your entire site here. But you have the ability to pretty much do whatever you want on your own site as long as Google doesn’t deem it malware.

Some other use cases include:

• Traffic visualization with Crazy Egg or Hot Jar
• Conversion and remarketing pixels with Google Ads
• A/B Testing with Google Optimize or AB Tasty
• Client side error tracking (and logging them)
• GDPR compliance management

There are many baked-in like the examples above, meaning you don’t even need to mess with code. Just add the IDs or settings and go. But for whatever custom solution or tag you need, you can always set up the HTML manually.

Anything I need to know before jumping in?

Before jumping right in, let’s familiarize ourselves with a few key terms that will make this ride a little easier.

Tags

Tags in GTM are your pixels or code snippets. A tag includes a single instance of a contained piece of code that is used for one function.

Example: your Google Analytics snippet will be one tag and if you add Crazy Egg, that will be another tag.

Variables

A variable is a name that you give a predetermined or dynamic value. The variable can be based on a variety of different things, hence the name “variable”. But it will function as a single name that you can give and reference that will never change when using it.

Example: we’ll set up your Google Analytics ID as a variable, meaning you’ll use that variable within the GA tag itself. If you ever need to update that ID, you won’t need to modify the tag or any other tags that use the variable – you’ll only need to update the variable itself.

Triggers

Triggers are the event or action that make your tag fire or load. This can happen in a variety of ways, such as when all or a specific page loads, when something is clicked, or when you have a completely custom event you fire with Javascript.

Example: when we set up Google Analytics, our trigger will be when any page loads.

Container

GTM allows you to manage multiple “containers” or groups of tags within your organization. This is helpful when you have one company with a few different website properties.

Example: you’re a small company under a parent company. The parent company wants to maintain one organization, but each smaller company gets their own container, as they have their individual needs for tags and code snippets.

How do I get set up?

Getting your account set up

The first thing you’ll need to do is get your account set up. After getting to the tag manager homepage, you’ll probably want to use your preexisting Google account. If you’re in an organization setting it up for the organization, you most likely want to use your business email, otherwise your personal account works okay too.

Next, create a new GTM account:

Google Tag Manager new account

Once there, fill out the form appropriately. The Account Name should represent the top level of your hierarchy and the container name should represent the specific installation instance.

Google Tag Manager new account container

As for the target platform, choose whichever makes sense. If it’s a website or web app, Web makes the most sense here.

Finding and installing your snippet

As soon as you accept the terms and hit create, you’ll be presented with a snippet in a little modal. You'll want to do exactly what it says and install the snippets per the instructions.

Google Tag Manager install snippet

Didn’t get the screen or accidentally exited out? You can find your snippet again by navigating to Admin in the top navbar and then click Install Google Tag Manager under Container to the right.

Testing that it’s working

To make sure it’s working, let’s verify a few things:

• We created our account
• We created our container
• We installed the Google Tag Manager snippets on our page and the change are live

Once all of those things are true, go ahead and click the Preview button in the top right of the page.

Google Tag Manager preview

If successful, you’ll now see an orange banner at the top of the page that says you’re in Preview Mode:

Google Tag Manager preview mode

Now wander over to the website that you installed GTM on and you should now see a banner at the bottom of the page. This will serve as your Debugger for working with GTM’s Preview.

Google Tag Manager debugger

You should see something similar to that, which means it worked! ?

Note: if you have an ad blocker on, similar to GA, you may need to disable it on the page you’re installing on in order to see it working.

Cool, it’s working, what about Google Analytics?

Now that we have a basic installation of GTM working, let’s set up Google Analytics. First we'll want to do some prep, so let's jump back over to the Tag Manager dashboard.

Create a settings variable

Navigate to Variables in the left sidebar and then click New beside User-Defined Variables.

Google Tag Manager new variable

For the name, enter “GA Settings” and under Variable Type, click through and select Google Analytics Settings. Enter your Google Analytics Tracking ID (or Property ID) in the appropriate field, and finally click Save, at which point you have your new settings variable.

Create a new GA tag

Navigate to Tags in the left sidebar and then click New beside Tags.

Google Tag Manager new tag

For the name, enter “GA - All Pages”. Under Tag Type, click and select Google Analytics: Universal Analytics in the panel that pops out from the right.

After selecting the tag type, under Google Analytics Settings, select your variable from the previous step, which if you followed along will be called “GA Settings”.

Next, click in the middle of the Triggering box, which should open a new UI to select a trigger.

Google Tag Manager trigger

Select All Pages, which should be the only Trigger there if you’re in a new account. It will take you back to the New Tag UI with your newly selected trigger.

Once the above is done, click Save in the top right of the UI which will then save and create your new Google Analytics tag.

Test that GTM is working

Similar to when we installed GTM to begin with, let’s go back to the Workspace home by clicking Overview in the left sidebar. Then click Preview at the top right.

Google Tag Manager debugger with Google Analytics

Open up your page GTM is installed on and you should once again see your Debugger, but this time with the Google Analytics Tag loading.

Are we done yet?

Not quite. Although we have GA working, we need to tweak the configuration to make sure GA and GTM work correctly together.

Updating the page snippet

Let’s dig into the code one more time to update our Google Analytics and Google Tag Manager snippets.

The issue with the installation that we set up here is that we installed GA through the GA instructions and GA inside GTM. This means we have GA installed twice. In practice, GA may send 1 pageview from your default installation and then another via GTM. To avoid this, we want to remove the pageview from the default GA snippet.

If you just have a super basic default installation, I did the work for you and you can go ahead and copy the snippet below. Replace [YOUR GA PROPERTY ID] with your GA Property ID and [YOUR GTM CONTAINER ID] with your GTM Container ID, and swap it out for your existing GA and GTM installation on your page.

<script>
// Set up an initial dataLayer configuration
window.dataLayer = window.dataLayer || [{
  "gaPropertyId": "[YOUR GA PROPERTY ID]"
}];

// Configure gtag and your GA ID
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', '[YOUR GA PROPERTY ID]');

// Google Tag Manager snippet
(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
'https://www.googletagmanager.com/gtm.js?id='+i+dl+'&gtm_cookies_win=x';f.parentNode.insertBefore(j,f);
})(window,document,'script','dataLayer', '[YOUR GTM CONTAINER ID]');
</script>

This should REPLACE your current Google Analytics snippet if you have one. If you don't, this should go as far in the top of your <head> tag as you can, but under the charset meta tag.

For more advanced users, just make sure you’re not sending duplicate pageviews or duplicate events between the page snippet and any GTM tag you create.

Test that GA is working

Follow the steps above to make sure GA is showing up with the debugger.

The final thing you’ll want to do is make sure your installation is working using Google’s Tag Assistant extension. Once you install the extension, go back to your page and click Enable in the Extension’s UI.

Google Tag Assistant enable recording

Refresh the page and you should see 2 tags, GA and GTM.

Google Tag Assistant results

If you notice, my GA tag is blue and yours may be too. Click in to Google Analytics for more info.

Google Tag Assistant non-standard implementation

As you can see, we have a healthy 1 Pageview Request and a note that says Non-standard implementation (hence the blue) This is because we installed it with GTM instead of the default on page.

Publish and Deploy!

We’re there! Last step is to Publish your Container and make the changes live.

Google Tag Manager submit version

Click Submit in the Overview UI, enter a name and description.

Google Tag Manager version configuration

Finally click Publish and your changes will be live on your site!

What else do I need to know?

There’s plenty more to talk through, but that’s for a different day. There are a ton of resources online to get you started with adding your own new tags and exploring the capabilities of Google Analytics, but here are a few points to keep in mind with GTM.

Permissions

You most likely don’t want every marketer on your 1,000 person team to have publish access to your GTM container. It’s best to have one or a few people responsible for managing the container, reviewing, and publishing while keeping everyone else to a lower access role. This will help prevent accidental changes or abuse within a bigger team.

Security

Permissions are your first step towards maintaining a secure installation. But Google takes this an extra step and performs some actions for you, such as checking for code errors and scanning for malware. While this is great, it’s still important to keep an active review cycle of tags getting published, just like any other pieces of code, to make sure the site integrity and health remain in tact.

Conventions

I highly recommend starting early with naming conventions and best practices when configuring your tags, variables, triggers, and really anything within GTM. The different ways to name things can spiral out of control really quickly, depending on how many people are working in there. This will make your work difficult to find, so it’s best to agree with the team on a convention to use early and get off on the right foot.

Single Page Apps

This doesn’t account for the impact single page apps have on a GTM and GA setup. The trigger we set up here will fire when any page loads, which is defined by a new page load by the browser. Single page apps don’t actually load a new page – rather, the JavaScript within the app makes it appear that a new page loads, so the trigger here will only fire the first pageview. Though it doesn’t include information directly usable in GTM, the GA documentation has some good information about tracking these types of apps.

GDPR

While GTM alone does not (currently) violate GDPR, you still need to be cognizant of how any of the tags you use impact your standing. This guide doesn’t touch this, so please make sure to do your homework.

Exiting out of Debugger

Stuck? One way is to clear your cookies, but you should always be able to go to the GTM Overview page and hit "Leave Preview Mode” right under the orange Preview Mode banner at the top of the page.

What's next?

From here, you can start managing your tags and pixels through GTM, which allows you to have a single source of consistent data flow into those tags. Experiment and use the debugger to play around, as it opens the door for a lot of new ways to work with the marketing side of things.

Originally published at https://www.colbyfayock.com/2019/11/how-to-use-google-tag-manager-to-maintain-google-analytics-and-other-marketing-tags