That promise shaped an entire generation of platform-as-a-service products. In 2026, that landscape is changing.

Heroku has clearly stated that it’s moving into a sustaining engineering model. The platform remains stable, secure, and supported, but active innovation is no longer the focus.

For many teams, this is acceptable. For others, especially startups and product teams planning three to five years ahead, it raises an important question: where should new applications live?

In this article, we’ll look at five strong Heroku alternatives that are well-positioned for 2026. Each platform approaches deployment differently, but all aim to preserve what developers loved about Heroku while improving on cost, flexibility, or modern workflows.

What We’ll Cover

• Why Teams Are Looking Beyond Heroku

• Sevalla: The Closest Successor to Classic Heroku

• Render: A Broad Platform for Growing Teams

• Fly.io: Global-First Deployment for Latency-Sensitive Apps

• Upsun: Enterprise-Grade Control Without Losing Structure

• Vercel: The Frontend-Native Deployment Platform

• Choosing the Right Heroku Alternative in 2026

• Conclusion

Why Teams Are Looking Beyond Heroku

Heroku’s shift toward maintenance over expansion signals maturity, not failure. But modern teams expect faster iteration, deeper infrastructure control, and tighter integration with cloud-native tooling.

AI workloads, edge computing, and global latency expectations are also reshaping deployment needs.

As a result, teams may want platforms that feel simple on day one but don’t become limiting as scale and complexity grow.

The alternatives discussed here are not identical replacements. Each represents a different philosophy about how applications should be built and operated in 2026.

Sevalla: The Closest Successor to Classic Heroku

Sevalla has quietly positioned itself as one of the most Heroku-like platforms available today. The core idea is familiar. You deploy applications without managing servers, environments are predictable, and the platform stays out of your way.

What makes Sevalla compelling in 2026 is its balance between simplicity and control. It keeps the developer experience tight while avoiding the opaque pricing and rigid abstractions that frustrated many Heroku users over time. Deployments are fast, logs are easy to access, and scaling feels intuitive rather than magical.

Sevalla is particularly attractive for mid-sized teams as well as enterprises that want a clean path from prototype to production. It supports modern application stacks without forcing you into complex infrastructure decisions too early. For teams migrating directly from Heroku, Sevalla often feels like the least disruptive transition.

The platform’s biggest strength is restraint. It doesn’t try to be everything at once. Instead, it focuses on being a reliable home for long-running services, APIs, and background workers. In 2026, that clarity is refreshing.

Built for the enterprise, Sevalla meets the highest standards of security. They are fully compliant with SOC2, ISO 27017, and ISO 27001:2022, ensuring your data stays protected and your requirements are met.

Render: A Broad Platform for Growing Teams

Render takes a more expansive approach. While it’s often compared to Heroku, Render aims to cover a wider range of use cases, from simple web services to complex microservice architectures.

Render stands out because it blends platform simplicity with infrastructure transparency. You still get managed databases, background jobs, and zero-downtime deploys, but you also gain more visibility into how resources are allocated. This makes it easier to reason about cost and performance as systems grow.

For teams that expect to scale steadily, Render offers a comfortable middle ground. It removes much of the operational burden while allowing deeper configuration when needed. Many engineering teams appreciate that Render feels less restrictive than Heroku without pushing them into full DevOps territory.

In 2026, Render is especially popular with SaaS companies that have outgrown entry-level platforms but are not ready to manage Kubernetes clusters themselves. It supports modern CI/CD workflows and integrates well with common developer tools.

Fly.io: Global-First Deployment for Latency-Sensitive Apps

Fly.io represents a different philosophy entirely. Instead of abstracting infrastructure away, Fly.io embraces it, but makes it programmable and developer-friendly.

Fly.io allows applications to run close to users by deploying workloads across multiple regions by default. This makes it ideal for applications where latency matters, such as real-time collaboration tools, gaming backends, or global APIs.

Unlike Heroku, Fly.io expects developers to understand a bit more about how their application runs. You interact with virtual machines rather than dynos, and configuration is more explicit. However, this added complexity comes with real power.

In 2026, Fly.io appeals strongly to experienced teams that want performance and control without adopting heavy orchestration systems. It’s not always the easiest option, but it is one of the most flexible. For teams willing to invest in understanding the platform, Fly.io can outperform traditional PaaS solutions in both speed and cost efficiency.

Upsun: Enterprise-Grade Control Without Losing Structure

Upsun, previously known as Platform.sh, brings a more opinionated, enterprise-oriented model to application deployment. It’s designed for teams that care deeply about environment parity, reproducibility, and long-term maintainability.

Upsun treats infrastructure as part of the application. Environments are versioned alongside code, and deployments are deterministic. This approach reduces surprises and makes complex systems easier to reason about over time.

For organizations with compliance requirements or multi-environment workflows, Upsun offers a level of rigor that Heroku never aimed to provide. At the same time, it abstracts away much of the operational burden that typically comes with such control.

In 2026, Upsun is particularly well-suited for regulated industries, large content platforms, and teams with multiple long-lived environments. It’s less about rapid experimentation and more about predictable, repeatable delivery at scale.

Vercel: The Frontend-Native Deployment Platform

Vercel is often discussed in a different category, but it deserves inclusion in any modern deployment conversation. Vercel is optimized for frontend applications, serverless functions, and edge workloads.

If Heroku excelled at hosting monolithic web apps, Vercel excels at composable, frontend-driven architectures. It integrates deeply with modern frameworks and makes global deployment nearly effortless.

In 2026, many applications are frontend-heavy, with APIs split into smaller services or serverless functions. For these use cases, Vercel offers a developer experience that feels faster and more modern than traditional PaaS platforms.

However, Vercel is not a full replacement for Heroku in every scenario. Long-running background jobs and stateful services often live elsewhere. Still, for teams building modern web products, Vercel frequently becomes the centerpiece of their deployment strategy.

Choosing the Right Heroku Alternative in 2026

There is no single “best” Heroku replacement. The right choice depends on how your application behaves, how your team works, and how much control you want over infrastructure.

Sevalla is ideal for teams that want familiarity and minimal friction. Render suits growing teams that need flexibility without chaos. Fly.io is powerful for global, performance-sensitive systems. Upsun excels in structured, enterprise environments. Vercel dominates frontend-centric architectures.

The common thread is that deployment in 2026 is no longer one-size-fits-all. Heroku set the standard, but the ecosystem has evolved. Today’s platforms offer sharper trade-offs, clearer philosophies, and better alignment with modern development patterns.

For teams starting new projects, the opportunity is clear. You can choose a platform that matches your future, not just your present.

Conclusion

Heroku is not disappearing, and for many existing workloads, it will continue to run reliably for years. However, its shift toward a sustaining engineering model makes one thing clear: teams building new products in 2026 should think carefully about where they place their long-term bets.

Deployment platforms are no longer just hosting choices. They shape how fast teams move, how systems scale, and how painful future migrations become.

In 2026, the strongest deployment strategy is intentional, not inherited. Heroku showed the industry what was possible. Its successors are now defining what comes next.

Hope you enjoyed this article. Learn more about me by visiting my website.

If you've been using GitHub for a while, you've probably heard of or used GitHub actions.

If you haven't heard of Github Actions or used them before, you can use them for automating your build, test, or deployment pipelines. You can create workflows that will be triggered upon certain actions such as opening a pull request or pushing to a branch.

These actions are useful to create build pipelines that automate deployments. They also help maintain the integrity of branches by running tests on all pushes/pull-requests.

Here is a simple workflow that would run tests whenever you push branches or open pull-requests in Github:

---
name: Run tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Setup and Run Tests
        run:  |
          docker-compose build
          docker-compose run web rake db:setup
          docker-compose run web rspec

The trigger is listed after the "on" tag. This workflow is triggered on pushes to the remote. Below you will see examples where there are multiple triggers for the workflow. If you have multiple triggers you can list the triggers in brackets like an array.

Next, you have your jobs tag. Under this tag you can list the various jobs you want to run. You may have a few different test jobs for unit tests, integration tests, and then perhaps a build job to build an image that gets pushed to a remote repository.

Within the job you have various steps. The first step is usually the checkout step. GitHub actions will spin up a virtual machine runner to run your jobs in, so you will want to include all the steps you need to set up this virtual machine for your application.

This means the first thing you'll want to do is get the code onto the virtual machine. This happens with the checkout step above.

Then your job needs give GitHub instructions on how to run things like the tests. The workflow above is running everything through Docker. The GitHub runner can see the docker-compose file when it checks out the project. Then it can run the three Docker steps listed above to spin up a container and then run the unit tests inside that container.

With this workflow if you were to open a pull-request in GitHub and the branch had failing tests, you'd get an alert telling you that your introducing breaking changes with an output like this:

Sample GitHub Actions alert

These workflows can sometimes become complex and very involved. In this tutorial, I will explain some simple ways to cleanup your workflows to avoid copy-pasting yaml configurations.

I will then go on to explain how you can create a simple deployment pipeline that enforces that one job passes before the other one runs.

Where You Might Find Duplicate Jobs

Say you have a job that runs your tests, and that job needs to be run in multiple different workflows. You want to run tests on all pull-requests. You also want to run them on merges to main in a way that blocks the production build/deploy step if the tests don’t pass.

Perhaps your first idea here is to build two workflows, one named test and the other named deploy. The test workflow would have one job: to set up and run all unit tests. In the other workflow you can copy the yaml from the test job in the test workflow and paste it as the first job in your deploy workflow.

Your test workflow would look something like this:

---
name: Run tests
on: pull-request

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Setup and Run Tests
        run:  |
          docker-compose build
          docker-compose run web rake db:setup
          docker-compose run web rspec

And your deployment workflow would look something like this:

---
name: Deploy
on:
  push:
    branches: [main]

jobs:
  tests:
      runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Setup and Run Tests
        run:  |
          docker-compose build
          docker-compose run web rake db:setup
          docker-compose run web rspec
  deploy:
    name: deploy
    runs-on: ubuntu-latest
    steps:
      - run: echo 'The triggering workflow succeeded deploying now'
      - uses: actions/checkout@v2
      - uses: akhileshns/heroku-deploy@v3.12.13 # This is the action
        with:
          usedocker: true

The deploy workflow allows both jobs (test and deploy) to run on merges to main, but it actually won’t block the deploy job from running if the tests fail.

But we can make this better in two ways: the first is by using reusable GitHub actions so that we can eliminate the copy pasting of the job yaml. Second, is using the GitHub job “needs” keyword so that we can make our deploy job depend on our test job succeeding.

I built this out here and will be going through that example.

How to Create Reusable Github Actions

Github has a blog post about reusable actions that I recommend. It goes a bit further than I will go here. But the important information for us is the explanation of what a reusable action is. A reusable action is one where you create a job in one place and then call it a separate workflow.

If I wanted my test workflow to be re-useable, I'd need to add a trigger labeled "workflow_call". I also want my workflow triggered on push and pull-requests. So my triggers would look something like this:

---
name: Run CI Process for the app
on: [workflow_call, push, pull_request, workflow_dispatch]

And the full workflow would look like this:

---
name: Run tests
on: [workflow_call, push, pull_request, workflow_dispatch]


jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Setup and Run Tests
        run:  |
          docker-compose build
          docker-compose run web rake db:setup
          docker-compose run web rspec

To reuse the test workflow in a deploy workflow (where I want to run tests and deploy my application) I could do something like the following:

jobs:
    test:
        uses:./.github/workflows/test.yml

This would allow the test job to be run in a separate workflow without having to copy the yaml associated with the test job from one workflow to another.

Dependent Jobs

That’s cool, but we don’t just want our test job to be reused in our deploy workflow. If the test job happens to fail in the deploy workflow, we actually want that to block deployment.

Now, we can look at the “needs” keyword which is documented here. Using “needs” we can require that the deploy job won’t even run unless the test job is successful.

Before I jump into using the "needs" tag, let me briefly explain what the deployment job is doing.

I have deployed the example app using Heroku. Instead of using the Heroku Git remote, I open pull-requests against a main branch. On merges to main I use this open source Github action to deploy the main branch to Heroku.

My deploy job will look something like this:

deploy:
    name: deploy
    runs-on: ubuntu-latest
    steps:
      - run: echo 'The triggering workflow succeeded deploying now'
      - uses: actions/checkout@v2
      - uses: akhileshns/heroku-deploy@v3.12.13 # This is the action
        with: 
            usedocker: true

The "usedocker" tag above specifies that I want to deploy this application to Heroku by building a pushing a docker image to the Heroku Container Registry. If you look through the source code for the Heroku deploy action that is referenced above, you’ll see that when I set "usedocker" to "true" it will run this command:

heroku container:push

That can be seen here.

If we want this job to require the a successful test run before we run the deploy job, we can add a test job to our workflow that references our reusable test job that we created:

---
name: Deploy
on:
  push:
    branches: [main]

jobs:
  tests:
    uses: ./.github/workflows/test.yml

Now we can add the needs tag to our deployment action and our full workflow yaml will look something like this:

---
name: Deploy
on:
  push:
    branches: [main]

jobs:
  tests:
    uses: ./.github/workflows/test.yml
  deploy:
    name: deploy
    needs: [ tests ]
    runs-on: ubuntu-latest
    steps:
      - run: echo 'The triggering workflow succeeded deploying now'
      - uses: actions/checkout@v2
      - uses: akhileshns/heroku-deploy@v3.12.13 # This is the action
        with:
          usedocker: true

And that's it! Now when we merge branches to main, GitHub gives us a visual of our dependent jobs that looks like this:

If we had a more complex workflow we could see exactly which job it is failing on.

Wrapping Up

With these steps, we can clean up our workflow yamls to reference existing jobs/workflows when possible. We can also build a simple pipeline of dependent actions where one steps depends on the success of a previous step.

These are the beginnings of CI/CD pipeline that can allow for frequent deploys. In turn it will allow us to get new features and fixes to users faster.

In this tutorial, we will be learning how to build and deploy an image management application backend.

It will be able to store a record of an image in the database, get the image's record back from the database, update the record, and even delete the record completely as the case may be.

To achieve all of this, we will be using Express (a Node.js framework), Postgres (a database), Cloudinary (a cloud based image storage), GitHub (for version control/storage) and Heroku (a hosting platform).

These tools are all free. So you don’t have to bother about how to go about paying for them. Thanks to these great innovators.

Prerequisites

If you are new to most of these technologies, I would advise you go through my other tutorial on how to create a server and upload images to Cloudinary.

If you are totally to Postgres, then check out this tutorial.

Whenever you are ready, let’s get to work!

How to Store and Retrieve an Image Record

Create Database and Table

So you'll want to start by cloning this project if you don't already have it.

In your pgAdmin:

• Create a database and name it tutorial
• Create a table and name it tutorial
• Create a Login/Group Role and name it tutorial. (Do not forget to give it all privileges.)

Back in your project directory, install the node-postgres (npm i pg) and make-runnnable (npm i make-runnable) packages.

In your package.json file, replace the contents of the "scripts" with "create": "node ./services/dbConnect createTables". We will use this to execute the dbConnect file we are about to create.

Create a services/dbConnect file to contain the following code:

const pg = require("pg");

const config = {
  user: "tutorial",
  database: "tutorial",
  password: "tutorial",
  port: 5432,
  max: 10, // max number of clients in the pool
  idleTimeoutMillis: 30000,
};

const pool = new pg.Pool(config);

pool.on("connect", () => {
  console.log("connected to the Database");
});

const createTables = () => {
  const imageTable = `CREATE TABLE IF NOT EXISTS
    images(
      id SERIAL PRIMARY KEY,
      title VARCHAR(128) NOT NULL,
      cloudinary_id VARCHAR(128) NOT NULL,
      image_url VARCHAR(128) NOT NULL
    )`;
  pool
    .query(imageTable)
    .then((res) => {
      console.log(res);
      pool.end();
    })
    .catch((err) => {
      console.log(err);
      pool.end();
    });
};

pool.on("remove", () => {
  console.log("client removed");
  process.exit(0);
});

//export pool and createTables to be accessible from anywhere within the application
module.exports = {
  createTables,
  pool,
};

require("make-runnable");

Now we are all set to create the table in our database. If you are ready, let's rock and roll!

Execute the following code in your terminal:

  npm run create

If the image below is your result, then you are good to go:

Check your pgAdmin, and you should have your table seated properly in your database like in the image below:

Alright, it's been a long road. It's time to unite Node, Postgres, and Cloudinary.

How to Create Endpoints to Store and Retrieve Image Records

Endpoint 1: Persist Image

First, require the dbConnect.js file on the top of the app.js file like so:

  const db = require('services/dbConnect.js');

Then in the app.js file, make a new endpoint (persist-image) with the following code:

// persist image
app.post("/persist-image", (request, response) => {
  // collected image from a user
  const data = {
    title: request.body.title,
    image: request.body.image,
  }

  // upload image here
  cloudinary.uploader.upload(data.image)
  .then().catch((error) => {
    response.status(500).send({
      message: "failure",
      error,
    });
  });
})

Replace the then block with the following code:

.then((image) => {
    db.pool.connect((err, client) => {
      // inset query to run if the upload to cloudinary is successful
      const insertQuery = 'INSERT INTO images (title, cloudinary_id, image_url) 
         VALUES($1,$2,$3) RETURNING *';
      const values = [data.title, image.public_id, image.secure_url];
    })
  })

The image.public_id and image.secure_url are gotten as part of the details returned for an image after the image has been successfully uploaded to Cloudinary.

We are now keeping a record of the image.public_id and image.secure_url (as you can see in the code above) in order to use it to retrieve, update, or delete the image record when we see fit.

Alright, let's move forward!

Still in the then block, add the following code under the query we created:

// execute query
client.query(insertQuery, values)
      .then((result) => {
        result = result.rows[0];

        // send success response
        response.status(201).send({
          status: "success",
          data: {
            message: "Image Uploaded Successfully",
            title: result.title,
            cloudinary_id: result.cloudinary_id,
            image_url: result.image_url,
          },
        })
      }).catch((e) => {
        response.status(500).send({
          message: "failure",
          e,
        });
      })

So our persist-image endpoint now looks like this:

// persist image
app.post("/persist-image", (request, response) => {
  // collected image from a user
  const data = {
    title: request.body.title,
    image: request.body.image
  }

  // upload image here
  cloudinary.uploader.upload(data.image)
  .then((image) => {
    db.pool.connect((err, client) => {
      // inset query to run if the upload to cloudinary is successful
      const insertQuery = 'INSERT INTO images (title, cloudinary_id, image_url) 
         VALUES($1,$2,$3) RETURNING *';
      const values = [data.title, image.public_id, image.secure_url];

      // execute query
      client.query(insertQuery, values)
      .then((result) => {
        result = result.rows[0];

        // send success response
        response.status(201).send({
          status: "success",
          data: {
            message: "Image Uploaded Successfully",
            title: result.title,
            cloudinary_id: result.cloudinary_id,
            image_url: result.image_url,
          },
        })
      }).catch((e) => {
        response.status(500).send({
          message: "failure",
          e,
        });
      })
    })  
  }).catch((error) => {
    response.status(500).send({
      message: "failure",
      error,
    });
  });
});

Now let's test out all our hard work:

Open your postman and test out your endpoint like the image below. Mine was successful. Hope yours had no errors too?

Open your Cloudinary console/dashboard and check your media Library. Your new image should be sitting there comfortably like mine below:

And now to the main reason why we are here: check the images table in your pgAdmin. Mine is what you see below:

Oohlala! We made it this far. Please take a break if you need one. I will be here waiting when you return. :)

If you are ready, then let's retrieve the image we persisted a moment ago.

Endpoint 2: Retrieve Image

Start with this code:

app.get("/retrieve-image/:cloudinary_id", (request, response) => {

});

Next, we will need to collect a unique ID from the user to retrieve a particular image. So add const { id } = request.params; to the code above like so:

app.get("/retrieve-image/:cloudinary_id", (request, response) => {
  // data from user
  const { cloudinary_id } = request.params;

});

Add the following code just below the code above:

db.pool.connect((err, client) => {
      // query to find image
    const query = "SELECT * FROM images WHERE cloudinary_id = $1";
    const value = [cloudinary_id];
    });

Under the query, execute the query with the following code:

// execute query
    client
      .query(query, value)
      .then((output) => {
        response.status(200).send({
          status: "success",
          data: {
            id: output.rows[0].cloudinary_id,
            title: output.rows[0].title,
            url: output.rows[0].image_url,
          },
        });
      })
      .catch((error) => {
        response.status(401).send({
          status: "failure",
          data: {
            message: "could not retrieve record!",
            error,
          },
        });
      });

Now our retrieve-image API looks like this:

app.get("/retrieve-image/:cloudinary_id", (request, response) => {
  // data from user
  const { cloudinary_id } = request.params;

  db.pool.connect((err, client) => {
    // query to find image
    const query = "SELECT * FROM images WHERE cloudinary_id = $1";
    const value = [cloudinary_id];

    // execute query
    client
      .query(query, value)
      .then((output) => {
        response.status(200).send({
          status: "success",
          data: {
            id: output.rows[0].cloudinary_id,
            title: output.rows[0].title,
            url: output.rows[0].image_url,
          },
        });
      })
      .catch((error) => {
        response.status(401).send({
          status: "failure",
          data: {
            message: "could not retrieve record!",
            error,
          },
        });
      });
  });
});

Let's see how well we did:

In your postman, copy the cloudinary_id and add it to the URL like in the image below:

YEEESSS! We can also retrieve our image.

If you are here, then you deserve a round of applause and a standing ovation for your industriousness.

Congratulations! You just reached a great milestone.

The code for storing and retrieving image records is here.

How to Update and Delete an Image Record

We will now see how to delete and update an image record as the case maybe. Let's begin with the delete endpoint.

Delete Endpoint

In the app.js file, start with the following code:

// delete image
app.delete("delete-image/:cloudinary_id", (request, response) => {

});

Next, we want to get the unique ID of the image we want to delete from the URL, that is cloudinary_id. So inside the code above add:

const { cloudinary_id } = request.params;

We now start the deleting process.

First, we delete from Cloudinary. Add the following code to delete the image from Cloudinary:

cloudinary.uploader
    .destroy(cloudinary_id)
    .then((result) => {
      response.status(200).send({
        message: "success",
        result,
      });
    })
    .catch((error) => {
      response.status(500).send({
        message: "Failure",
        error,
      });
    });

At this point, our API can delete the image from Cloudinary only (you can check it out in postman). But we also want to get rid of the record we have in our Postgres database.

Second, we delete from our Postgres database. To do so, replace the code in the then block with the following query:

db.pool.connect((err, client) => {

      // delete query
      const deleteQuery = "DELETE FROM images WHERE cloudinary_id = $1";
      const deleteValue = [cloudinary_id];

})

Execute the query with the following code underneath it:

// execute delete query
      client.query(deleteQuery, deleteValue)
      .then((deleteResult) => {
        response.status(200).send({
          message: "Image Deleted Successfully!",
          deleteResult
        });
      }).catch((e) => {
        response.status(500).send({
          message: "Image Couldn't be Deleted!",
          e
        });
      });

So our Endpoint should look like this:

// delete image
app.delete("/delete-image/:cloudinary_id", (request, response) => {
  // unique ID
  const { cloudinary_id } = request.params;

  // delete image from cloudinary first
  cloudinary.uploader
    .destroy(cloudinary_id)

    // delete image record from postgres also
    .then(() => {
      db.pool.connect((err, client) => {

      // delete query
      const deleteQuery = "DELETE FROM images WHERE cloudinary_id = $1";
      const deleteValue = [cloudinary_id];

      // execute delete query
      client
        .query(deleteQuery, deleteValue)
        .then((deleteResult) => {
          response.status(200).send({
            message: "Image Deleted Successfully!",
            deleteResult,
          });
        })
        .catch((e) => {
          response.status(500).send({
            message: "Image Couldn't be Deleted!",
            e,
          });
        });
      })
    })
    .catch((error) => {
      response.status(500).send({
        message: "Failure",
        error,
      });
    });
});

The time has arrived for us to put our Endpoint to the test.

The following is my Cloudinary media library with two images I uploaded already. Take note of their unique ID (public_id).

If you don't already have that, please use the persist-image endpoint to upload some images.

Now let's proceed to postman:

Notice, the unique ID as it matches one of the image in my Cloudinary media library.

From the output, we executed the DELETE command and that deleted one ROW from our image TABLE in our database.

Now this is my media library with one of the images remaining:

Walahhhh... We are now able to get rid of an image.

Do take a break if you need one. ✌🏾

If you are ready, I am ready to update images.

Update Image API

Below the delete-image API, let's start creating the update-image API with the following code:

// update image
app.put("/update-image/:cloudinary_id", (request, response) => {

});

All codes will live in there.

Collect the unique Cloudinary ID and new image details from the user with the following code:

// unique ID
  const { cloudinary_id } = request.params;

// collected image from a user
  const data = {
    title: request.body.title,
    image: request.body.image,
  };

Delete the image from Cloudinary with the following code:

// delete image from cloudinary first
  cloudinary.uploader
    .destroy(cloudinary_id)
      // upload image here
    .then()
    .catch((error) => {
      response.status(500).send({
        message: "failed",
        error,
      });
    });

Next, upload another image to Cloudinary. To do that, enter the following code into the then block:

() => {
      cloudinary.uploader
        .upload(data.image)
        .then()
        .catch((err) => {
          response.status(500).send({
            message: "failed",
            err,
          });
        });
    }

Now let's replace our initial record with the new image details. Replace the content of the then block with the following:

(result) => {
          db.pool.connect((err, client) => {

            // update query
            const updateQuery =
              "UPDATE images SET title = $1, cloudinary_id = $2, image_url = $3 WHERE cloudinary_id = $4";
            const value = [
              data.title,
              result.public_id,
              result.secure_url,
              cloudinary_id,
            ];
          });
        }

We execute the query using the following code just beneath the query declaration:

// execute query
            client
              .query(updateQuery, value)
              .then(() => {

                // send success response
                response.status(201).send({
                  status: "success",
                  data: {
                    message: "Image Updated Successfully"
                  },
                });
              })
              .catch((e) => {
                response.status(500).send({
                  message: "Update Failed",
                  e,
                });
              });

At this point, this is what I have:

// update image
app.put("/update-image/:cloudinary_id", (request, response) => {
  // unique ID
  const { cloudinary_id } = request.params;

  // collected image from a user
  const data = {
    title: request.body.title,
    image: request.body.image,
  };

    // delete image from cloudinary first
    cloudinary.uploader
      .destroy(cloudinary_id)

      // upload image here
      .then(() => {
        cloudinary.uploader
          .upload(data.image)

          // update the database here
          .then((result) => {
            db.pool.connect((err, client) => {
            // update query
            const updateQuery =
              "UPDATE images SET title = $1, cloudinary_id = $2, image_url = $3 WHERE cloudinary_id = $4";
            const value = [
              data.title,
              result.public_id,
              result.secure_url,
              cloudinary_id,
            ];

            // execute query
            client
              .query(updateQuery, value)
              .then(() => {

                // send success response
                response.status(201).send({
                  status: "success",
                  data: {
                    message: "Image Updated Successfully"
                  },
                });
              })
              .catch((e) => {
                response.status(500).send({
                  message: "Update Failed",
                  e,
                });
              });
            });
          })
          .catch((err) => {
            response.status(500).send({
              message: "failed",
              err,
            });
          });
      })
      .catch((error) => {
        response.status(500).send({
          message: "failed",
          error,
        });
      });

});

It's testing time!

This is my postman in the image below:

Take note of the unique cloudinary ID which matches the image left in my Cloudinary media library.

Now take a look at my Cloudinary media library in the image that follows:

Take note of the new image replacing the initial one in my media library above.

Also, see that the unique Cloudinary ID matches that in my database with the new title. See image below:

Yayeh! You did awesomely great! 💪

We just completed an image management application with Node.js, Cloudinary and Postgres.

Code Optimisation With Express Routing

Express Routing enables us to make our Node.js code more optimized or give it a more modular structure by separating the business logic from the controllers. We want to use that to clean up our code so far.

We'll begin by creating a new folder with the name routes in the root directory:

mk dir routes

In the routes folder, create a file with the name: routes.js.

For Windows:

echo . > routes.js

For Mac:

touch routes.js

Empty the routes.js file if anything is there and enter the following code:

const express = require('express');

const router = express.Router();



module.exports = router;

Add the following code above the last line:

const cloudinary = require("cloudinary").v2;
require("dotenv").config();
const db = require("../services/dbConnect.js");

// cloudinary configuration
cloudinary.config({
  cloud_name: process.env.CLOUD_NAME,
  api_key: process.env.API_KEY,
  api_secret: process.env.API_SECRET,
});

Back in the App.js file, delete the following code:

const cloudinary = require("cloudinary").v2;
require("dotenv").config();
const db = require("./services/dbConnect.js");

// cloudinary configuration
cloudinary.config({
  cloud_name: process.env.CLOUD_NAME,
  api_key: process.env.API_KEY,
  api_secret: process.env.API_SECRET,
});

Move all the APIs to routes.js.

Change all occurence of app to router carefully.

My routes.js file now looks like this.

Back in the app.js file, import the routes.js file like so:

// import the routes file
const routes = require("./routes/routes")

Now register the routes like so:

// register the routes 
app.use('/', routes);

This is my app.js file at the moment:

const express = require("express");
const app = express();

// import the routes file
const routes = require("./routes/routes")

// body parser configuration
const bodyParser = require("body-parser");
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

// register the routes 
app.use('/', routes);

module.exports = app;

It's time to test and see if our routes are still working like before.

Make sure yours are working like mine below:

persist-image

retrieve-image

update-image

delete-image

Wow! We have been able to separate our routes from our app.js file.

The code for this is here.

Even though our routes.js file is still lengthy, we have a good basis to separate our business logic from our controllers. The time has arrived to do just that.

How to Move Each Endpoint to a Different File

Begin by creating a new folder in the routes folder and name it controllers.

In the controllers folder, create 5 files and name them after the 5 endpoints.

Our folder and files should be structured as follows:

Back in the routes.js file, let's work on the image-upload API. Cut the following code:

(request, response) => {
  // collected image from a user
  const data = {
    image: request.body.image,
  };

  // upload image here
  cloudinary.uploader
    .upload(data.image)
    .then((result) => {
      response.status(200).send({
        message: "success",
        result,
      });
    })
    .catch((error) => {
      response.status(500).send({
        message: "failure",
        error,
      });
    });
}

In the imageUpload file, equate the code you already cut from the image-upload endpoint to exports.imageUpload like so:

exports.imageUpload = (request, response) => {
    // collected image from a user
    const data = {
      image: request.body.image,
    };

    // upload image here
    cloudinary.uploader
      .upload(data.image)
      .then((result) => {
        response.status(200).send({
          message: "success",
          result,
        });
      })
      .catch((error) => {
        response.status(500).send({
          message: "failure",
          error,
        });
      });
  }

Now let's import what is necessary for this code to work. So this is my imageUpload file right now:

const cloudinary = require("cloudinary").v2;
require("dotenv").config();

// cloudinary configuration
cloudinary.config({
  cloud_name: process.env.CLOUD_NAME,
  api_key: process.env.API_KEY,
  api_secret: process.env.API_SECRET,
});

exports.imageUpload = (request, response) => {
    // collected image from a user
    const data = {
      image: request.body.image,
    };

    // upload image here
    cloudinary.uploader
      .upload(data.image)
      .then((result) => {
        response.status(200).send({
          message: "success",
          result,
        });
      })
      .catch((error) => {
        response.status(500).send({
          message: "failure",
          error,
        });
      });
  }

Let's import and register the imageUpload API in the routes.js file like so:

const imageUpload = require("./controllers/imageUpload");

// image upload API
router.post("image-upload", imageUpload.imageUpload);

Now we have this line of code pointing to the imageUpload API in the imageUpload.js file from the routes.js file.

How awesome! Our code is more readable.

Make sure to test the API to be sure it's working properly. Mine works perfectly. See image below:

Now, it's your turn!

Apply what you have learnt to the other APIs. Let's see what you have got.

I will be waiting on the other side...

If you are here, then I believe you have done yours and they're working perfectly – or at least, you already gave it your best shot. Kudos!

Checkout mine here.

Congratulations. You are awesome :)

The code optimisation code is here.

Alright, on to the next step.

How to Deploy to GitHub And Heroku

Now that we've completed our application, let's deploy it on Heroku so that we can access it even without being on our laptop where the code was written.

I will be walking you through uploading our application to GitHub and deploying it to Heroku.

Without further ado, let's get our hands dirty.

How to Upload the Code to GitHub

Uploading or pushing to GitHub is as easy as eating your favorite meal. Check out this resource to learn how to push your project from you local machine to GitHub.

How to Deploy to Heroku

Let's begin by creating an account on Heroku.

If you have created an account, you may have been prompted to create an app (that is a folder where your app will be housed). You can do that, but I will do mine using my terminal since the terminal comes with a few added functionalities that we will need later.

Open your project in a terminal if you have not done so already. I will be using the VS Code integrated terminal.

Install Heroku CLI:

npm install heroku

Login to Heroku CLI. This will open a browser window, which you can use to log in.

heroku login

Create an app. It can have any name. I am using node-postgres-cloudinary.

heroku create node-postgres-cloudinary

Go to your Heroku dashboard and you will find the newly created app.

Waalaah!

That is how mine looks in the image above. I have some apps there already but you can see the one I just created.

Let's now add the PostgreSQL database to the app.

How to Add Heroku Postgres

Click on the app you just created. It will take you to the app's dashboard.

Click on the Resources tab/menu.

In the Add-ons Section, search and select Heroku Postgres.

Make sure you select the Hobby Dev - Free plan in the pop-up window that follows:

Click on the provision button to add it to the app like so:

Click on the Heroku Postgres to take you to the Heroku Postgres dashboard.

Click on the settings tab:

Click on View Credentials:

In the Credentials, we are interested in the Heroku CLI. We will be using it in a bit.

Back to the terminal.

Let's confirm if the Heroku Postgres was added successfully. Enter the following in the terminal:

heroku addons

Yeeeeaaaah! It was added successfully.

Before we proceed, make sure that your PostgreSQL path is set correctly if you are on Windows. Follow this link to learn how to set a path. The path should be like this: C:\Program Files\PostgreSQL\<VERSION>\bin.

The version will depend on the one installed on you machine. Mine is: C:\Program Files\PostgreSQL\12\bin since I am using the version 12.

The following image might be helpful:

You may have to navigate to the folder where PostgreSQL is installed on your machine to find out your own path.

Login into the Heroku Postgres using the Heroku CLI from our Heroku Postgres credentials. This is mine – yours will be different:

heroku pg:psql postgresql-slippery-19135 --app node-postgres-cloudinary

If you got an error, it is most likely because your path is not set properly.

How to Prepare our Database Connection to Match Heroku's

At the moment, my database looks like this:

const pg = require("pg");

const config = {
  user: "tutorial",
  database: "tutorial",
  password: "tutorial",
  port: 5432,
  max: 10, // max number of clients in the pool
  idleTimeoutMillis: 30000,
};

const pool = new pg.Pool(config);

pool.on("connect", () => {
  console.log("connected to the Database");
});

If you try connecting Heroku to this, you are going to get an error. This is because Heroku has a connection string setup already. So we have to setup our connection such that Heroku can easily connect.

I am going to refactor my database connection file (dbConnect.js) and .env file to make this happen.

• dbConnect.js

const pg = require('pg');
require('dotenv').config();

// set production variable. This will be called when deployed to a live host
const isProduction = process.env.NODE_ENV === 'production';

// configuration details
const connectionString = `postgresql://${process.env.DB_USER}:${process.env.DB_PASSWORD}@${process.env.DB_HOST}:${process.env.DB_PORT}/${process.env.DB_DATABASE}`;

// if project has been deployed, connect with the host's DATABASE_URL
// else connect with the local DATABASE_URL
const pool = new pg.Pool({
  connectionString: isProduction ? process.env.DATABASE_URL : connectionString,
  ssl: isProduction,
});

// display message on success if successful
pool.on('connect', () => {
  console.log('Teamwork Database connected successfully!');
});

• .env file

DB_USER="tutorial"
DB_PASSWORD="tutorial"
DB_HOST="localhost"
DB_PORT="5432"
DB_DATABASE="tutorial"

With the setup of the dbconnect and .env file, we are ready to export our database and tables from our local machine to heroku postgres.

How to Export Database and Tables

Go to your pgAdmin and locate the database for this tutorial. Mine is tutorial.

Right-Click on it and select Backup. This will bring up a new window.

Enter a name for the SQL file like I did. Select the plain format. Then click Backup. This will save the file to your documents folder.

Locate the file and move it into the project directory. It can be anywhere in the directory but I choose to move mine into the services directory because it holds the database related files.

Back in the terminal, navigate to the folder containing the SQL file and run the following code to add the tables we just exported to the heroku postgres database:

cat <your-SQL-file> | <heroku-CLI-from-the-heroku-posgres-credentials>

This is what mine looks like:

cat tutorial.sql | heroku pg:psql postgresql-slippery-19135 --app node-postgres-cloudinary

Did you notice that I changed directory to services (cd services)? That is where my sql file is located.

Wow! We have just successfully exported our database and tables to Heroku.

It is almost over...

How to Tell GitHub that We Made Changes

Add the files we have made changes to:

$ git add .

The period (.) adds all files.

Commit your latest changes:

$ git commit -m "refactored the dbConnect and .env file to fit in to heroku; Added the database SQL file"

Push the committed files:

$ git push origin -u master

Finally deploying our App

Go to you app's dashboard:

Select the GitHub Deployment method:

Search and select a repo, and click on connect:

Select the branch you want to deploy (in my own case, it is the master branch):

Enable automatic deployment by clicking the Enable automatic deployment button as in the image above.

Click on the Deploy button in the manual deploy

We will not have to do all this for subsequent deployments.

Now you have a button telling you to "view site" after build is completed. Click it. This will open your app in a new tab.

Oh no! A bug? Application error??

Don't worry, it just a small issue. Something you should never forget to do while making deployments. Most hosting service will require it.

How to Fix the Heroku Application Error

Get back to the root directory of your project.

Create a file and name it Procfile (it has no extension).

In the file, enter the following code:

web: node index.js

This directs Heroku to the server file (index.js) which is the entry point of the application. If your server is in a different file, please modify as required.

Save the file and push the new changes to GitHub.

Wait 2 to 5 minutes for Heroku to automatically detect changes in your GitHub repo and render the changes on the app.

You can now refresh that error page and see your hard work paying off:

You can also test the retrieve image route and see it working.

Congratulations! What a feat you have attained.

Other routes (persist-image, update-image, and delete-image) will not be working because we have not provisioned or added cloudinary add-on. It is as simple as the PostgreSQL one we just did. So you can give it a shot.

Conclusion

We started this tutorial with the aim of learning how to build a backend application using Express, Postgres, Cloudinary, Github and Heroku.

We learned how to store, retrieve, delete, and update an image record. We then organised our code with Express Routing, pushed it to GitHub, and deployed it on Heroku. That was a lot.

I hope you will agree that it was worth it because we learnt a lot. You should try adding the Cloudinary add-on yourself to sharpen your knowledge even more.

Thanks for reading!

Heroku is a well-known PaaS widely used by developers. And as a fun and useful project, you can easily make your own Heroku-like PaaS with Dokku.

What is Heroku?

Heroku is a platform as a service (PaaS) company founded in 2007. The platform runs on AWS, and its ephemeral storage system is called "Dyno".

Heroku is one of the most used PaaS by developers and for a good reason– it is easy to use, well documented, and supports several programming languages.

But what if you could deploy your own Heroku-like platform, including a CI/CD pipeline, database connections, HTTPS connections, and much more with one simple-to-use application?

Well, that is what Dokku provides and more. Let's take a look.

What is a PaaS?

Platform-as-a-Service (PaaS) is a software architecture style that provides an easy-to-use abstraction layer for deploying your application's code and managing it.

This allows you to focus on writing business logic rather than worrying about the platform itself.

PaaS providers usually provide their own database service as well as other related services, which can greatly simplify common development tasks.

The great advantage of PaaS is that the application developer doesn't need to perform any system administration work. Instead, you can just upload your code and configuration settings to a central server platform.

The service then takes care of deploying the code, scaling it as needed, backing up data, handling hosting and uptime concerns, and so on.

What is Dokku?

Dokku is a hosted Platform as a Service that enables developers to deploy their applications with ease.

From their website:

The smallest PaaS implementation you've ever seen

Dokku is based on Docker and uses Heroku's build-packs to compile and package your applications.

One of the best things about Dokku is that it is very lightweight and can be installed on a single server or VM.

It includes scalable hosting using Docker containers, continuous deployment with Git, and other popular DevOps tools.

Dokku also offers a variety of features, such as support for multiple languages, custom domains, automated deployments, and many more.

You can easily connect Postgres databases and even file storage to your applications.

You can check out more information at https://dokku.com/ or the documentation at: https://dokku.com/docs/getting-started/installation/.

You can also show some love for the open-source GitHub project here.

How to Install Dokku

In order to install Dokku you will need a Linux VPS and a domain name.

You can install and use Dokku without a domain name but it is much simpler using a domain. I recommend a cloud VPS because it makes it easier to access and configure.

When connecting a domain, either a single domain or a wildcard can be associated with the server's IP.

I will be using a VPS hosted on Hetzner with Ubuntu 20.04 installed.

We first start by making sure that our system is up to date with these commands:

# Update the linux installation
$ sudo apt update
$ sudo apt upgrade -y

Then we can download and run the installation script for Dokku:

# Install Dokku with the install script
$ wget https://raw.githubusercontent.com/dokku/dokku/v0.26.8/bootstrap.sh;
$ sudo DOKKU_TAG=v0.26.8 bash bootstrap.sh

--> Ensuring we have the proper dependencies
--> Note: Installing dokku for the first time will result in removal of
    files in the nginx 'sites-enabled' directory. Please manually
    restore any files that may be removed after the installation and
    web setup is complete.

    Installation will continue in 10 seconds.

    [...........]

    --> Running post-install dependency installation

 ! Setup a user's ssh key for deployment by passing in the public ssh key as shown:

     echo 'CONTENTS_OF_ID_RSA_PUB_FILE' | dokku ssh-keys:add admin

The installation script will install Docker and all necessary dependencies and also Dokku itself, as seen in the code above.

After the installation is complete we need to assign the SSH keys to access and also configure our domain name.

In case you have set up access to your VPS with SSH (which you should) then you already have the necessary keys – you just need to add them to Dokku:

# Assign SSH key to Dokku
$ cat ~/.ssh/authorized_keys | dokku ssh-keys:add admin

SHA256:6O1TLVOUkWV+zmTWXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

In case you don't already have an SSH key in the server, then you need to generate a key pair:

# Generate SSH key
$ ssh-keygen

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:7T6BbRCVWjGtcSUXXXXXXXXXXXXXXXXXXXXXXXXXXXXX root@freeDokku
The key's randomart image is:
+---[RSA 3072]----+
[.................]
|     . oS*.o . . |
[.................]
+----[SHA256]-----+

Then you can add it to Dokku:

# Add SSH key to Dokku
$ dokku ssh-keys:add admin /root/.ssh/id_rsa.pub

SHA256:7T6BbRCVWjGtcSUXXXXXXXXXXXXXXXXXXXXXXXX

Next, and the final step, is to assign the domain for your Dokku installation. We do that with the command:

# Set installation global domain
$ dokku domains:set-global domain.com

-----> Set domain.com

Make sure you replace 'domain.com' with your own domain, and that your domain name DNS points to the server's IP address.

And that is all you need to do to install and set up Dokku. It is really that simple.

You can now start adding your applications.

Let's see an example of that by adding a standard Django application in the next section.

How to Create Your Application in Dokku

To create and deploy our first application, there is some preparation work we need to do on Dokku.

To deploy an application on Dokku, follow these steps:

• Create the application on Dokku, which means giving it a name.
• Create the associate database (or other plugins, if needed). This will create and provision a database for use with an automatic DATABASE_URL added to the application for ease of deployment.
• Push the necessary code to Dokku's application internal GitHub endpoint. This can include also the necessary release steps (like running Django migrations, for example).

After the code is pushed, Dokku will generate any necessary Docker container and will run our application with any associated databases (or other plugins).

Now that we've covered the necessary steps, let's go through them in practice.

Let's start by creating our application. For this tutorial, I will create a very simple Django website that contains all the necessary logic for us to test Dokku.

We create an application on Dokku with this command (in the server where we installed Dokku):

# Creating our application on Dokku
$ dokku apps:create djangotutorial

-----> Creating djangotutorial...

By default, datastores (or databases) are not created when an application is created.

The datastores are handled by a series of plugins. You can check here for all available plugins.

For our application, we will create a Postgres datastore. Since by default no plugins are installed, we first need to install the Postgres plugin:

# install the postgres plugin
# plugin installation requires root, hence the user change
sudo dokku plugin:install https://github.com/dokku/dokku-postgres.git

Then we can create our Postgres datastore:

# Create a Postgres datastore
$ dokku postgres:create djangotutorial_datastore

       Waiting for container to be ready
       Creating container database
       Securing connection to database
=====> Postgres container created: djangotutorial_datastore
=====> djangotutorial_datastore postgres service information
       Config dir:          /var/lib/dokku/services/postgres/djangotutorial_datastore/data
       Config options:
       Data dir:            /var/lib/dokku/services/postgres/djangotutorial_datastore/data
       Dsn:                 postgres://postgres:ea706cc108c805d5124d134d934024c5@dokku-postgres-djangotutorial-datastore:5432/djangotutorial_datastore
       Exposed ports:       -
       Id:                  782a04fe6bbd25958752c17c304358fd5ec1f3c54d6d53175b6481b3b957d94b
       Internal ip:         172.17.0.5
       Links:               -
       Service root:        /var/lib/dokku/services/postgres/djangotutorial_datastore
       Status:              running
       Version:             postgres:14.1

We can check that our Docker container for the datastore is already up and running with:

# Check running containers
$ docker ps

CONTAINER ID   IMAGE                      COMMAND                  CREATED              STATUS              PORTS      NAMES
782a04fe6bbd   postgres:14.1              "docker-entrypoint.s…"   About a minute ago   Up About a minute   5432/tcp   dokku.postgres.djangotutorial_datastore

Now that we have the datastore up and running, we need to associate it with our application:

# Associate datastore with the application
$ dokku postgres:link djangotutorial_datastore djangotutorial

-----> Setting config vars
       DATABASE_URL:  postgres://postgres:ea706cc108c805d5124d134d934024c5@dokku-postgres-djangotutorial-datastore:5432/djangotutorial_datastore
-----> Restarting app djangotutorial
 !     App image (dokku/djangotutorial:latest) not found

You can see that a DATABASE_URL is automatically created and associated with the application.

The example above mentions that our application image is not found because we haven't pushed any code to it yet.

We can check our application's environment variables to confirm that our DATABASE_URL is present:

# Checking an application environment variables
$ dokku config:show djangotutorial

=====> djangotutorial env vars
DATABASE_URL:  postgres://postgres:ea706cc108c805d5124d134d934024c5@dokku-postgres-djangotutorial-datastore:5432/djangotutorial_datastore

We now have all the necessary configurations done on the Dokku side to support the deployment of our application.

Next, we will create the code for our application and deploy that to Dokku for an automated CI/CD pipeline.

How to Create Our Application Code on PyCharm

Before we can deploy an application, we need to have its source code to push to Dokku.

For this tutorial, we are going to create a very simple Django application that shows also the use of the Postgres database.

We will be using PyCharm as our IDE to create and manage our project.

We create a new project in PyCharm – let's call it 'DjangoTutorial':

Creating a new project on PyCharm - Screenshot by author

I personally prefer to create new projects with a virtual environment already in place, which makes life much easier.

If you created the project with a default main.py file (like I did because I keep forgetting to remove the checkmark), you can safely delete it now. We are not going to use it.

The first step is, of course, to install Django so we can build our application. We do that install using pip:

$ pip install django

Collecting django
  Downloading Django-4.0.2-py3-none-any.whl (8.0 MB)
     |████████████████████████████████| 8.0 MB 6.4 MB/s
Collecting sqlparse>=0.2.2
  Using cached sqlparse-0.4.2-py3-none-any.whl (42 kB)
Collecting tzdata
  Using cached tzdata-2021.5-py2.py3-none-any.whl (339 kB)
Collecting asgiref<4,>=3.4.1
  Downloading asgiref-3.5.0-py3-none-any.whl (22 kB)
Installing collected packages: tzdata, sqlparse, asgiref, django
Successfully installed asgiref-3.5.0 django-4.0.2 sqlparse-0.4.2 tzdata-2021.5

Then we create our Django project with:

$ django-admin startproject DjangoTutorial .

Notice the '.' at the end of the command. I like to use that so that it creates the project in the current directory instead of creating an extra sub-directory.

You should now have a project structure like this in PyCharm:

PyCharm folder structure for our Django application - screenshot by the author

We can run our project with the standard Django run:

$ python manage.py runserver   

Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 18 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
February 02, 2022 - 16:49:27
Django version 4.0.2, using settings 'DjangoTutorial.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.

We have not yet applied our migrations, so we will do that next after we discuss the database configuration for both local and Dokku access.

Navigating to the link http://127.0.0.1:8000/, we can now access our standard Django welcome page:

Django welcome page - screenshot by the author

We have our Django installation up and running so now we can start building the rest of the project.

Like most projects, we will need to store data in a database (or database using the Dokku naming).

We also want to be able to debug and run our application locally on the development machine (using a local database, in this SQLite) and run it on the cloud with Dokku using the Postgres database.

This means we need to change some configuration in our settings.py to be able to support both use cases without us needing to change any flags or configs every time.

We start by installing the package dj-database-url with:

# Install packages for the database url
$ pip install dj-database-url
$ pip install psycopg2


# We also install this package to be able to use environment variables
$ pip install python-decouple

This package enables us to have a Django database connection dictionary, populated with all the data by simply specifying a database URL.

With the package install, let's update the configuration on the settings.py:

# We need to add this import at the beginning to use environment variables
import dj_database_url
from decouple import config
from django.conf.global_settings import DATABASES

.....

# Let's also updated the allowed host so we can use it later on
ALLOWED_HOSTS = config('ALLOWED_HOSTS').split(',')

.....

# We replace the default database configuration from Django with this one
# Database
# https://docs.djangoproject.com/en/4.0/ref/settings/#databases

# DATABASE URL
DATABASES['default'] = dj_database_url.parse(config('DATABASE_URL'),conn_max_age=600)

We will also need to create '.env' file in the root dir of our project:

#HOST SETTINGS
ALLOWED_HOSTS = 127.0.0.1

#DATABASE SETTINGS
DATABASE_URL='sqlite:///db.sqlite3'

As you can see, with this change we can use the database URL from the local '.env' file on the local development machine, and then on Dokku it will automatically use the already defined DATABASE_URL that was created when we linked the datastore to the application on Dokku.

We can now create our first (and only) web page of this tutorial), a simple counter that stores and read the value from the database.

Let's create a separate application to contain our logic:

$ python manage.py startapp counter

We now should have a new folder called 'counter' in our project. Let's add a new model by opening the 'models.py' file:

from django.db import models


class Counter(models.Model):
    count = models.IntegerField(default=0)

    def __str__(self):
        return self.count

We can now add a new URL to load our counter page. We do that by adding a new file called 'urls.py' to our 'counter' folder:

from django.urls import path
from . import views

urlpatterns = [
    path('counter/', views.counter, name='counter'),
]

We now have both the model and the URL to load our test page. All we need now is the view and HTML template to render the page.

Let's create the view by editing the 'views.py' file:

from django.shortcuts import render
from .models import Counter


def counter(request):
    counter_value = Counter.objects.last()

    if counter_value is None:
        counter_value = Counter(count=0)
        counter_value.save()

    if request.method == 'POST':
        counter_value.count += 1
        counter_value.save()

    return render(request, 'counter.html', {'counter': counter_value.count})

Now we can create our HTML template to show the counter value on the page. We create a new file called 'counter.html' inside a new 'templates' folder:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Counter</title>
</head>
<body>
  <form method="post">
      {%csrf_token%}
    <h4>Counter value is: {{ counter }}</h4>
    <input type="submit" name="submit" value="Increase Counter">
  </form>
</body>
</html>

The last step is to add our newly created application to the 'settings.py' file in order for Django to recognize it:

.....

INSTALLED_APPS = [
    'counter',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

.....

And the URL to our main URLs file:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('', include('counter.urls')),
    path('admin/', admin.site.urls),
]

With all the necessary code and HTML in place, we can now create and run our migrations to create our new model in the database. We first do that on the local server by running:

# Create and run migrations
$ python manage.py makemigrations
$ python manage.py migrate

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, counter, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying counter.0001_initial... OK
  Applying sessions.0001_initial... OK

As you can see, we not only applied the migrations for our new application but we also run the initial migrations for the other Django applications since this was the first time we ran the migrations.

We can again run our server locally and we should be able to access the URL http://127.0.0.1:8000/counter/ and increment the counter:

Running our counter application - GIF by the author

As you can see, reloading the page keeps our counter value, meaning that the value is been stored in the database with our model.

How to Deploy Our Application to Dokku

We now have a very simple application running with database integration to store our counter value.

We are ready to deploy it to the cloud so we can test it there and make sure our database is also working in the cloud.

Before we do the Git push to deploy the code to Dokku, we need to do some preparation:

• Install our web server (gunicorn)
• Create our requirements file (for our packages)
• Create our Procfile (for our deployment commands)

Let's start with installing our web server to use in the cloud:

# Install our web server
$ pip install gunicorn

With our packages in place we can now create our requirements file with:

# Create requirements file
$ pip freeze > requirements.txt

Now we need to create the 'Procfile'. This file is used by Dokku to determine which commands to run on deployment and after deployment.

So let's create a new file called 'Procfile' in the root directory with the contents:

web: gunicorn DjangoTutorial.wsgi
release: python manage.py migrate

We have created two commands for Dokku to run:

• release – this command is executed on the deployment of our application in Dokku. We use it to migrate our database.
• web – this command allows Dokku to know which webserver to run to allow access to the application.

Finally, to make sure that we can collect any static files when our code is deployed to Dokku, we need to create a new directory called 'static' on the root directory. Inside we create an empty file called '.gitkeep' (this will allow us to add the directory to the Git repository later).

We also need to add this path for the static files to our 'settings.py' file:

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/4.0/howto/static-files/

STATIC_URL = 'static/'
STATIC_ROOT = BASE_DIR / "static"

Now all the files and logic are in place and we can deploy to Dokku with a standard Git push. Let's check our current file structure:

PyCharm folder structure - screenshot by the author

To be able to push our code to Dokku, we need to add our project to a Git repository.

Since we don't want to push all the files from our folder structure to the Dokku git repository, we create a '.gitignore' to exclude certain files and directories. I use the contents of this excellent Gist to populate the file:

We can now initialize and commit our code to a Git repository locally:

# Initialize repository
$ git init -b main

# Add and commit our files
$ git add . && git commit -m "initial commit"

[main (root-commit) e77a16a] initial commit
 20 files changed, 438 insertions(+)       
 create mode 100644 .gitignore
 create mode 100644 DjangoTutorial/__init__.py
 create mode 100644 counter/tests.py
 create mode 100644 counter/urls.py
 create mode 100644 counter/views.py
 create mode 100644 db.sqlite3
 create mode 100644 manage.py
 create mode 100644 requirements.txt

With our repository committed, we can now push it to a remote repository, that is the Dokku Git repository for our application:

# Adding our remote repository (replace domain.com with your domain name)
$ git remote add dokku dokku@domain.com:djangotutorial

# Time to push our code to the remote repository
$ git push dokku main

Enumerating objects: 34, done.
Counting objects: 100% (34/34), done.
Delta compression using up to 8 threads
Compressing objects: 100% (31/31), done.
Writing objects: 100% (34/34), 11.41 KiB | 402.00 KiB/s, done.
Total 34 (delta 7), reused 0 (delta 0)
-----> Set main to DOKKU_DEPLOY_BRANCH.
-----> Cleaning up...
-----> Building djangotutorial from herokuish
-----> Adding BUILD_ENV to build environment...
       BUILD_ENV added successfully
-----> Python app detected
-----> No Python version was specified. Using the buildpack default: python-3.9.9
       To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
-----> No change in requirements detected, installing from cache
-----> Installing python-3.9.9
-----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
-----> Installing SQLite3
-----> Installing requirements with pip
       Collecting asgiref==3.5.0
       Downloading asgiref-3.5.0-py3-none-any.whl (22 kB)
       Collecting dj-database-url==0.5.0
       Downloading dj_database_url-0.5.0-py2.py3-none-any.whl (5.5 kB)
       Collecting Django==4.0.2
       Downloading Django-4.0.2-py3-none-any.whl (8.0 MB)
       Collecting gunicorn==20.1.0
       Downloading gunicorn-20.1.0-py3-none-any.whl (79 kB)
       Collecting psycopg2==2.9.3
       Downloading psycopg2-2.9.3.tar.gz (380 kB)
       Preparing metadata (setup.py): started
       Preparing metadata (setup.py): finished with status 'done'
       Collecting python-decouple==3.5
       Downloading python_decouple-3.5-py3-none-any.whl (9.6 kB)
       Collecting sqlparse==0.4.2
       Downloading sqlparse-0.4.2-py3-none-any.whl (42 kB)
       Collecting tzdata==2021.5
       Downloading tzdata-2021.5-py2.py3-none-any.whl (339 kB)
       Building wheels for collected packages: psycopg2
       Building wheel for psycopg2 (setup.py): started
       Building wheel for psycopg2 (setup.py): finished with status 'done'
       Created wheel for psycopg2: filename=psycopg2-2.9.3-cp39-cp39-linux_x86_64.whl size=579484 sha256=9d6a2810a5d766738526d6f411e5e9ce514cce882b6c80a47a13c02dc7529e02
       Stored in directory: /tmp/pip-ephem-wheel-cache-8k0chg5g/wheels/b3/a1/6e/5a0e26314b15eb96a36263b80529ce0d64382540ac7b9544a9
       Successfully built psycopg2
       Installing collected packages: sqlparse, asgiref, tzdata, python-decouple, psycopg2, gunicorn, Django, dj-database-url
       Successfully installed Django-4.0.2 asgiref-3.5.0 dj-database-url-0.5.0 gunicorn-20.1.0 psycopg2-2.9.3 python-decouple-3.5 sqlparse-0.4.2 tzdata-2021.5
-----> $ python manage.py collectstatic --noinput
       128 static files copied to '/tmp/build/static'.

-----> Discovering process types
       Procfile declares types -> release, web
-----> Releasing djangotutorial...
-----> Checking for predeploy task
       No predeploy task found, skipping
-----> Checking for release task
-----> Executing release task from Procfile: python manage.py migrate
=====> Start of djangotutorial release task (a602cab30) output
       Operations to perform:
         Apply all migrations: admin, auth, contenttypes, counter, sessions
       Running migrations:
         Applying contenttypes.0001_initial... OK
         Applying auth.0001_initial... OK
         Applying admin.0001_initial... OK
         Applying admin.0002_logentry_remove_auto_add... OK
         Applying admin.0003_logentry_add_action_flag_choices... OK
         Applying contenttypes.0002_remove_content_type_name... OK
         Applying auth.0002_alter_permission_name_max_length... OK
         Applying auth.0003_alter_user_email_max_length... OK
         Applying auth.0004_alter_user_username_opts... OK
         Applying auth.0005_alter_user_last_login_null... OK
         Applying auth.0006_require_contenttypes_0002... OK
         Applying auth.0007_alter_validators_add_error_messages... OK
         Applying auth.0008_alter_user_username_max_length... OK
         Applying auth.0009_alter_user_last_name_max_length... OK
         Applying auth.0010_alter_group_name_max_length... OK
         Applying auth.0011_update_proxy_permissions... OK
         Applying auth.0012_alter_user_first_name_max_length... OK
         Applying counter.0001_initial... OK
         Applying sessions.0001_initial... OK
=====> End of djangotutorial release task (a602cab30) output
-----> App Procfile file found
=====> Processing deployment checks
       No CHECKS file found. Simple container checks will be performed.
       For more efficient zero downtime deployments, create a CHECKS file. See https://dokku.com/docs/deployment/zero-downtime-deploys/ for examples
-----> Deploying djangotutorial via the docker-local scheduler...
-----> Deploying web (count=1)
       Attempting pre-flight checks (web.1)
       Waiting for 10 seconds (web.1)
       Default container check successful (web.1)
-----> Deploying release (count=0)
-----> Running post-deploy
-----> Creating new app virtual host file...
-----> Configuring djangotutorial.domain.com...(using built-in template)
-----> Creating http nginx.conf
       Reloading nginx
-----> Renaming containers
       Renaming container djangotutorial.web.1.upcoming-7101 (f8d229ebd8bc) to djangotutorial.web.1
-----> Checking for postdeploy task
       No postdeploy task found, skipping
-----> Updated schedule file
=====> Application deployed:
       http://djangotutorial.domain.com

To domain.com:djangotutorial
 * [new branch]      main -> main

We have just deployed our application to Dokku.

What just happened? Well, Dokku did a lot of work for us:

• Installed Python
• Installed the requirements
• Collected the static files
• Performed the migrations
• And finally started a gunicorn server to deploy our application

If you had a permission error, then your private key should be registered within your local development environment. If you get a permission denied error when pushing, you can register your private key as follows: ssh-add -k ~/<your private key>.

You may also see an error regarding the ALLOWED_HOSTS when accessing the application. In that case, all you need to do is to run the following command on the Dokku server to set the environment variable to the correct value:

# Set ALLOWED_HOSTS environment variable (make sure to use your domain name)
$ dokku config:set djangotutorial ALLOWED_HOSTS=djangotutorial.domain.com

We can now access and test our application at the above URL:

Running our counter application on Dokku - GIF by the author

Congratulations, you just deployed your application on Dokku.

How to Add SSL with Let's Encrypt

One final configuration that we can do is to add SSL security to our application by installing a Let's Encrypt SSL certificate.

We can do this very easily on Dokku with the Let's Encrypt plugin:

# Install the Let's Encrypt plugin
sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git

# Configure the plugin (make sure to replace to your email)
dokku config:set --global DOKKU_LETSENCRYPT_EMAIL=email@domain.com

# set a custom domain that you own for your application
dokku domains:set djangotutorial djangotutorial.your.domain.com

# Enable Let's Encrypt
dokku letsencrypt:enable djangotutorial

# Enable Let's Encrypt auto-renewal
dokku letsencrypt:cron-job --add

Now we have a more secure application. After all, our counter is very important.

Conclusion

Using a PaaS makes a developer's life easier when building web applications.

You can use hosted PaaS like Heroku and there are many others, so the choice is there.

But there are some main drawbacks:

• Price – hosted solutions might have limits in terms of database storage or file storage, among others
• You don't control the hosting where the PaaS is deployed. Recent examples of AWS shows that not even the biggest hosting is free of problems.

You can work around these issues by self-hosting your PaaS.

This allows for more control in terms of pricing. You can use hosting provider like Digital Ocean, Hetzner, and others who have quite cheap VPSs that work perfectly with Dokku.

There are no database limits. The only limits you might have are memory and disk space, but you can always upgrade your VPS for a smaller price than getting a new database at Heroku.

Dokku is easy to install and like we saw. Creating and deploying an application is a 3 step process:

• Create an application on Dokku
• Create a datastore on Dokku (if needed, like Postgres) and link to the application
• Deploy your code to Dokku with Git

Additionally, you might need to configure some environment variables and SSL certificates, but that is all.

Dokku is really the smallest PaaS implementation.

Full source code for the Django application is available at:

Follow me on Twitter: https://twitter.com/DevAsService

Check out my website at: https://developer-service.io/

Or check out my blog at: https://blog.developer-service.io/

Live2D is a technology that allows artists to easily transform traditional 2D illustrations to create fluid expressions and motions.

The most popular software for Live2D modeling and animation is Cubism, which also provides well-documented SDK for web, native apps, and the Unity game development engine.

In this tutorial, I will walk you through how to build on top of Cubism's official Live2D Web SDK sample and deploy it to Heroku, a popular cloud app-hosting platform.

How to Set Up the Environment

To follow along with this tutorial, clone my GitHub repo and checkout the start branch. The finished project is on the develop branch.

I've also recorded a video tutorial on YouTube.

git clone https://github.com/RuolinZheng08/heroku-live2d.git
git checkout start

# update the submodule, Cubism's Live2d Web Framework
git submodule update --init

Install Node.js and npm using Homebrew:

# if you need to install homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# homebrew will install node and npm at the same time
brew install node

I'll be using Visual Studio Code as my main IDE, but you may follow along using any editor of your choice.

How to Run the Starter Code Locally

The directory structure is as follows. Our web app will be served from Samples/TypeScript/Demo.

.
├─ .vscode          # Visual Studio Code project setting
├─ Core             # Live2D Cubism Core JavaScript and TypeScript source code
├─ Framework        #  Source code for the rendering and animation features
└─ Samples
   ├─ Resources     # Live2D model files and web image assets
   └─ TypeScript    # [IMPORTANT] TypeScript sample project

Inside the heroku-live2d directory, run the following commands:

cd Samples/TypeScript/Demo/

npm install

npm run-script build

npm run-script serve

Navigate to http://localhost:5000/Samples/TypeScript/Demo/ and you should be able to see a Live2D character.

To interact with the model, hold down your mouse cursor and the character's head and eyes will follow your cursor. Tap on the body of the character to see a special animation. Tap on the gear icon in the top right corner to switch between different models.

Mark from Samples/Resources/Mark

How to Deploy to Heroku

The starter code uses npm, TypeScript, and webpack.

To deploy our project to Heroku, we need to create a package.json file that Heroku can use to build our project in our project root directory. We also need to modify Samples/TypeScript/Demo/package.json and the webpack configuration in Samples/TypeScript/Demo/webpack.config.js.

Top-level package.json

The boilerplate package.json for a Node.js Heroku app looks like this:

{
    "name": "heroku-live2d",
    "description": "Live2D Cubism Heroku Demo",
    "scripts": {
        "start": ...,
        "build": ...
    },
    "dependencies": {
        ...
    }
}

Inspect the dependencies and devDependencies attributes in Samples/TypeScript/Demo/package.json and add both sets of dependencies as dependencies to heroku-live2d/package.json.

Remember that when building and serving locally, we used npm run-script [build|serve] from inside the Samples/TypeScript/Demo directory.

Therefore, to run these npm commands from the project root, we need to prepend cd Samples/TypeScript/Demo before the npm commands. The build command, for example, will become:

cd Samples/TypeScript/Demo && npm run-script build

With these changes, the top-level package.json should look like this:

{
    "name": "heroku-live2d",
    "description": "Live2D Cubism Heroku Demo",
    "scripts": {
        "start": "cd Samples/TypeScript/Demo && npm run-script start",
        "build": "cd Samples/TypeScript/Demo && npm run-script build"
    },
    "dependencies": {
        "@typescript-eslint/eslint-plugin": "^2.18.0",
        "@typescript-eslint/parser": "^2.18.0",
        "eslint": "^6.8.0",
        "eslint-config-prettier": "^6.10.0",
        "eslint-plugin-prettier": "^3.1.2",
        "prettier": "^1.19.1",
        "rimraf": "^3.0.1",
        "serve": "^11.3.0",
        "ts-loader": "^6.2.1",
        "typescript": "^3.7.5",
        "webpack": "^4.41.5",
        "webpack-cli": "^3.3.10",
        "webpack-dev-server": "^3.10.1",
        "whatwg-fetch": "^3.0.0"
    }
}

Samples/TypeScript/Demo/package.json

On localhost, we were running on port 5000. However, Heroku will dynamically assign our web app a port stored in a variable $PORT. Therefore, we need the npm run-script start command inside Samples/TypeScript/Demo/package.json to start the webpack server on port $PORT.

Append to scripts > start > webpack-dev-server --progress so it looks like this:

"scripts": {
    "start": "webpack-dev-server --progress --port $PORT",
    ...
}

Samples/TypeScript/Demo/webpack.config.js

Add disableHostCheck to the configuration of devServer and remove port since we have configured it dynamically above.

module.exports = {
    ...,
    devServer: {
        contentBase: path.resolve(__dirname, '../../..'),
        watchContentBase: true,
        inline: true,
        hot: true,
        port: 5000, // delete this line
        host: '0.0.0.0',
        disableHostCheck: true, // add this line
        compress: true,
        useLocalIp: true,
        writeToDisk: true
    },
    ...
}

Add watchOptions so that our node_modules won't be watched. If we don't do this, we will run into an error about exceeding the maximum number of watchers when we deploy to Heroku.

module.exports = {
    ...,
    watchOptions: {
        ignored: /node_modules/
    },
    ...
}

Deploy to Heroku

To download the Heroku command line client, run

brew tap heroku/brew && brew install heroku

Log into Heroku from the command line using heroku login.

Create a Heroku app and append some numbers (for example, 123) to the app name to ensure uniqueness.

heroku create heroku-live2d-NUMBERS

Set up Node.js as the buildpack:

heroku buildpacks:set heroku/nodejs

Add and commit your project using git. Note that we don't necessarily need git push:

git add .
git commit -m "Ready to deploy to heroku"

Push the project to Heroku, assuming you are following along on the start branch. You can always check the branch you are on and push from that branch.

# check which branch we are on
git branch

# the syntax is
# git push heroku GIT_BRANCH_NAME:HEROKU_BRANCH_NAME
git push heroku start:master

You may need to wait for a few minutes for the build process to complete.

After that, navigate to YOUR-HEROKU-APP-NAME.herokuapp.com/Samples/TypeScript/Demo. In my case, the URL is https://heroku-live2d.herokuapp.com/Samples/TypeScript/Demo/. The Live2D characters will be there to greet you :)

Notice that the highlighted URL is already on Heroku

How to Redirect index.html to Samples/TypeScript/Demo

You might have noticed that YOUR-HEROKU-APP-NAME.herokuapp.com shows a list of the directory structure instead of the Live2D models. We can solve this by adding a dummy top-level index.html that redirects to Samples/TypeScript/Demo.

<!DOCTYPE html>
<html>

<head>
    <title></title>
    <!-- Just a dummy html to redirect to my subdirectory -->
    <meta http-equiv="refresh" content="0; url=Samples/TypeScript/Demo">
</head>

<body>

</body>

</html>

Rerun the deployment command git push heroku start:master. Now when you visit YOUR-HEROKU-APP-NAME.herokuapp.com, you will be automatically redirected to the Live2D model page.

Congratulations on making it to the end of this tutorial! You now have a Live2D Web App deployed to Heroku.

I hope you enjoyed this tutorial. Let's keep in touch! Connect with me on LinkedIn, GitHub, Medium, or check out my personal website.

Resources & Links

My GitHub repo for this tutorial

My Heroku App

My YouTube Video Tutorial

Cubism's Official SDK Documentation

When it comes to deploying an application, there are usually two options: a VPS or a PaaS (platform as a service). This article will show you a recipe for deploying an application to production on a PaaS like Heroku.

Step 1 - Create the project

The first step is to create a simple structure for our project with some basic files. For this article, I’ll create a demo server with NodeJS.

In a new folder I’ll open a terminal and run the command npm init -y in order to create a new project. The dummy server will be written in Express, so we need to run the npm install express command to install this module.

Once this library is installed, we can create a new file for our project, named app.js. Inside it we'll write the code for our simple server:

[RAW](https://carbon.now.sh/?bg=rgba(171%2C%20184%2C%20195%2C%201)&t=seti&wt=none&l=javascript&ds=true&dsyoff=20px&dsblur=68px&wc=true&wa=true&pv=56px&ph=56px&ln=false&fl=1&fm=Hack&fs=14px&lh=133%25&si=false&es=2x&wm=false&code=const%2520app%2520%253D%2520require(%2522express%2522)()%253B%250A%250Aconst%2520PORT%2520%253D%2520process.env.PORT%2520%257C%257C%25203000%253B%250A%250Aapp.get(%2522%2522%252C%2520(req%252C%2520res)%2520%253D%253E%2520%257B%250A%2520%2520res.send(%2522Hello%2520world%2522)%253B%250A%257D)%253B%250A%250Aapp.listen(PORT%252C%2520()%2520%253D%253E%2520%257B%250A%2520%2520console.log(%2560App%2520up%2520at%2520port%2520%2524%257BPORT%257D%2560)%253B%250A%257D)%253B)

We can start the application by running node app.js. Then we can try it out at the following URL http://localhost:3000. At this point you should see the message Hello World in the browser.

Step 2 - Version control system

The next step is to choose a version control system and to place our code in a development platform in a repository.

The most popular version control system is Git along with Github as a development platform, so that's what we'll use here.

On GitHub, go ahead and create a new repository for your application, like this:

To upload your local code into a repository, you need to run the commands that are listed on Github after you click Create repository button:

Commands to upload our code to the Github repo

! Before we do this, we must ignore some files. We want to upload to the repository only the code that we write, without the dependencies (the installed modules).

For that, we need to create a new file .gitignore and inside it write the file that we want to ignore.

Folder structure and .gitignore file

Now, we can write the commands listed in the picture above (the one from GitHub).

If you ran the commands correctly, then it'll be on your repository’s page. If you refresh it you should see your files, except the one that you explicitly ignored, namely node_modules.

Step 3 - Link the repository with Heroku

At this step, we can link the repository from Github to our Heroku application.

First, create a new application on Heroku and follow the steps listed on the platform.

Once the application has been created, a window similar to this should appear:

App dashboard

Now, if you look at the navigation at the top, you'll see Overview, Resources, Deploy, Metrics and so on. Be sure that Deploy is selected. Then on the second row, click on the GitHub icon.

Click connect

Search for the desired application, which is demo-deploy-app-09 in our case. Then click Connect.

Deploy Branch

Once the application is successfully connected with your Heroku account, you can click Deploy Branch to deploy your application.

If you want, you can also select the option Enable Automatic Deploys which will automatically pull the code from your Github repository every time you make a push to that repository.

Once the application has been deployed, you can click on View to open your application.

Step 4 - Configure Heroku to properly run the application

If you open the application at this point, you should see something like this:

That’s right, an error. That’s because Heroku doesn’t know how to start our application.

If you remember, we ran the command node app.js to start the application locally.
Heroku has no way of knowing what commands it needs to run to start the application, and that's why it threw an error.

To solve this problem, we must create a new file named Procfile with the following content: web: node ./app.js.

To update our application, all we need to do is push a new commit to GitHub. If we have enabled the Automatic Deploys option, then the code will be automatically pulled to Heroku. Otherwise we need to click on Deploy Branch again.

Once the application is rebuilt, we should see it working like so:

Step 5 - How to add an add-on

One key benefit that Heroku provides is the fact that you can easily add resources in the form of add-ons to your project. These external resources come in the form of databases, logging & monitoring tools, CI and CD tools, or testing tools.

So now let's see how to add a new resource to your project. First, we'll go to Resources, and from there we'll add a new tool for testing.

Go ahead and click on Find more add-ons and then search for Loadmill.

Loadmill is a testing tool which is really great for regression testing and load testing.

Go ahead and click on Install…. Then choose the application you want to link.

At this step, Heroku will automatically create a new account for you on the provisioned platform.

On the resources tab, you can see the newly added resource:

If you go ahead and access this add-on, you should see its dashboard with an intro tutorial and a demo test created for you.

Loadmill dashboard

Conclusion

Heroku allows developers to quickly and almost painlessly deploy an application on a web server.

It also provides a lot of plugins that you can integrate into your application.

A PaaS solution will always allow you to move faster than the solution with a VPS where you have to configure everything from the ground up.

What is the best way to master Data Science? What will get me hired?

My answer remains constant: There is no alternative to working on portfolio-worthy projects.

Even after passing the TensorFlow Developer Certificate Exam, I’d say that you can only really prove your competency with projects that showcase your research, programming skills, mathematical background, and so on.

In my post how to build an effective Data Science Portfolio, I shared many project ideas and other tips to prepare an awesome portfolio. This post is dedicated to one of those ideas: building an end-to-end data science/ML project.

Agenda

This tutorial is intended to walk you through all the major steps involved in completing an and-to-end Machine Learning project. For this project, I’ve chosen a supervised learning regression problem.

Here are the major topics covered:

• Pre-requisites and Resources

• Data Collection and Problem Statement

• Exploratory Data Analysis with Pandas and NumPy

• Data Preparation using Sklearn

• Selecting and Training a few Machine Learning Models

• Cross-Validation and Hyperparameter Tuning using Sklearn

• Deploying the Final Trained Model on Heroku via a Flask App

Let’s start building…

Pre-requisites and Resources

To go through this project and tutorial, you should be familiar with Machine Learning algorithms, Python environment setup, and common ML terminologies. Here are a few resources to get you started:

• Read the first 2–3 chapters of The hundred page ML book: http://themlbook.com/wiki/doku.php

• List of Tasks for almost every Machine Learning Project — Keep referring to this list while working on this (or any other) ML project.

• You need a Python Environment set up — a virtual environment dedicated to this project.

• You should be familiar with Jupyter Notebook.

That’s it, so make sure you have an understanding of these concepts and tools and you’re ready to go!

Data Collection and Problem Statement

The first step is to get your hands on the data. But if you have access to data (as most product-based companies do), then the first step is to define the problem that you want to solve. We don’t have the data yet, so we are going to collect the data first.

We are using the Auto MPG dataset from the UCI Machine Learning Repository. Here is the link to the dataset:

• http://archive.ics.uci.edu/ml/datasets/Auto+MPG

The data concerns city-cycle fuel consumption in miles per gallon, to be predicted in terms of 3 multivalued discrete and 5 continuous attributes.

Once you have downloaded the data, move it to your project directory, activate your virtualenv, and start the Jupyter local server.

You can download the data into your project from the notebook as well using wget :

!wget "http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data"

The next step is to load this .data file into a pandas datagram. For that, make sure you have pandas and other general use case libraries installed. Import all the general use case libraries like so:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

Then read and load the file into a dataframe using the read_csv() method:

# defining the column names
cols = ['MPG','Cylinders','Displacement','Horsepower','Weight',
                'Acceleration', 'Model Year', 'Origin']
# reading the .data file using pandas
df = pd.read_csv('./auto-mpg.data', names=cols, na_values = "?",
                comment = '\t',
                sep= " ",
                skipinitialspace=True)
#making a copy of the dataframe
data = df.copy()

Next, look at a few rows of the dataframe and read the description of each attribute from the website. This helps you define the problem statement.

Problem Statement — The data contains the MPG (Mile Per Gallon) variable which is continuous data and tells us about the efficiency of fuel consumption of a vehicle in the 70s and 80s.

_Our aim here is to predict the MPG value for a vehicl_e, given that we have other attributes of that vehicle.

Exploratory Data Analysis with Pandas and NumPy

For this rather simple dataset, the exploration is broken down into a series of steps:

Check for data type of columns

##checking the data info
data.info()

Check for null values.

##checking for all the null values
data.isnull().sum()

The horsepower column has 6 missing values. We’ll have to study the column a bit more.

Check for outliers in horsepower column

##summary statistics of quantitative variables
data.describe()

##looking at horsepower box plot
sns.boxplot(x=data['Horsepower'])

Since there are a few outliers, we can use the median of the column to impute the missing values using the pandas median() method.

##imputing the values with median
median = data['Horsepower'].median()
data['Horsepower'] = data['Horsepower'].fillna(median)
data.info()

Look for the category distribution in categorical columns

##category distribution

data["Cylinders"].value_counts() / len(data)
data['Origin'].value_counts()

The 2 categorical columns are Cylinders and Origin, which only have a few categories of values. Looking at the distribution of the values among these categories will tell us how the data is distributed:

Plot for correlation

##pairplots to get an intuition of potential correlations

sns.pairplot(data[["MPG", "Cylinders", "Displacement", "Weight", "Horsepower"]], diag_kind="kde")

The pair plot gives you a brief overview of how each variable behaves with respect to every other variable.

For example, the MPG column (our target variable) is negatively correlated with the displacement, weight, and horsepower features.

Set aside the test data set

This is one of the first things we should do, as we want to test our final model on unseen/unbiased data.

There are many ways to split the data into training and testing sets but we want our test set to represent the overall population and not just a few specific categories. Thus, instead of using simple and common train_test_split() method from sklearn, we use stratified sampling.

Stratified Sampling — We create homogeneous subgroups called strata from the overall population and sample the right number of instances to each stratum to ensure that the test set is representative of the overall population.

In task 4, we saw how the data is distributed over each category of the Cylinder column. We’re using the Cylinder column to create the strata:

from sklearn.model_selection import StratifiedShuffleSplit

split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
for train_index, test_index in split.split(data, data["Cylinders"]):
    strat_train_set = data.loc[train_index]
    strat_test_set = data.loc[test_index]

Checking for the distribution in training set:

##checking for cylinder category distribution in training set

strat_train_set['Cylinders'].value_counts() / len(strat_train_set)

Testing set:

strat_test_set["Cylinders"].value_counts() / len(strat_test_set)

You can compare these results with the output of train_test_split() to find out which one produces better splits.

Checking the Origin Column

The Origin column about the origin of the vehicle has discrete values that look like the code of a country.

To add some complication and make it more explicit, I converted these numbers to strings:

##converting integer classes to countries in Origin 

columntrain_set['Origin'] = train_set['Origin'].map({1: 'India', 2: 'USA', 3 : 'Germany'})
train_set.sample(10)

We’ll have to preprocess this categorical column by one-hot encoding these values:

##one hot encoding
train_set = pd.get_dummies(train_set, prefix='', prefix_sep='')
train_set.head()

Testing for new variables — Analyze the correlation of each variable with the target variable

## testing new variables by checking their correlation w.r.t. MPG
data['displacement_on_power'] = data['Displacement'] / data['Horsepower']
data['weight_on_cylinder'] = data['Weight'] / data['Cylinders']
data['acceleration_on_power'] = data['Acceleration'] / data['Horsepower']
data['acceleration_on_cyl'] = data['Acceleration'] / data['Cylinders']

corr_matrix = data.corr()
corr_matrix['MPG'].sort_values(ascending=False)

We found acceleration_on_power and acceleration_on_cyl as two new variables which turned out to be more positively correlated than the original variables.

This brings us to the end of the Exploratory Analysis. We are ready to proceed to our next step of preparing the data for Machine Learning.

Data Preparation using Sklearn

One of the most important aspects of Data Preparation is that we have to keep automating our steps in the form of functions and classes. This makes it easier for us to integrate the methods and pipelines into the main product.

Here are the major tasks to prepare the data and encapsulate functionalities:

Preprocessing Categorical Attribute — Converting the Oval

##onehotencoding the categorical values
from sklearn.preprocessing import OneHotEncoder

cat_encoder = OneHotEncoder()
data_cat_1hot = cat_encoder.fit_transform(data_cat)
data_cat_1hot   # returns a sparse matrix

data_cat_1hot.toarray()[:5]

Data Cleaning — Imputer

We’ll be using the SimpleImputer class from the impute module of the Sklearn library:

##handling missing values
from sklearn.impute import SimpleImputer

imputer = SimpleImputer(strategy="median")imputer.fit(num_data)

Attribute Addition — Adding custom transformation

In order to make changes to datasets and create new variables, sklearn offers the BaseEstimator class. Using it, we can develop new features by defining our own class.

We have created a class to add two new features as found in the EDA step above:

• acc_on_power — Acceleration divided by Horsepower

• acc_on_cyl — Acceleration divided by the number of Cylinders

from sklearn.base import BaseEstimator, TransformerMixin

acc_ix, hpower_ix, cyl_ix = 4, 2, 0

##custom class inheriting the BaseEstimator and TransformerMixin
class CustomAttrAdder(BaseEstimator, TransformerMixin):
    def __init__(self, acc_on_power=True):
        self.acc_on_power = acc_on_power  # new optional variable
    def fit(self, X, y=None):
        return self  # nothing else to do
    def transform(self, X):
        acc_on_cyl = X[:, acc_ix] / X[:, cyl_ix] # required new variable
        if self.acc_on_power:
            acc_on_power = X[:, acc_ix] / X[:, hpower_ix]
            return np.c_[X, acc_on_power, acc_on_cyl] # returns a 2D array

        return np.c_[X, acc_on_cyl]

attr_adder = CustomAttrAdder(acc_on_power=True)
data_tr_extra_attrs = attr_adder.transform(data_tr.values)
data_tr_extra_attrs[0]

Setting up Data Transformation Pipeline for numerical and categorical attributes

As I said, we want to automate as much as possible. Sklearn offers a great number of classes and methods to develop such automated pipelines of data transformations.

The major transformations are to be performed on numerical columns, so let’s create the numerical pipeline using the Pipeline class:

def num_pipeline_transformer(data):
    '''
    Function to process numerical transformations
    Argument:
        data: original dataframe 
    Returns:
        num_attrs: numerical dataframe
        num_pipeline: numerical pipeline object

    '''
    numerics = ['float64', 'int64']

    num_attrs = data.select_dtypes(include=numerics)

    num_pipeline = Pipeline([
        ('imputer', SimpleImputer(strategy="median")),
        ('attrs_adder', CustomAttrAdder()),
        ('std_scaler', StandardScaler()),
        ])
    return num_attrs, num_pipeline

In the above code snippet, we have cascaded a set of transformations:

• Imputing Missing Values — using the SimpleImputer class discussed above.

• Custom Attribute Addition— using the custom attribute class defined above.

• Standard Scaling of each Attribute — always a good practice to scale the values before feeding them to the ML model, using the standardScaler class.

Combined Pipeline for both Numerical and Categorical columns

We have the numerical transformation ready. The only categorical column we have is Origin for which we need to one-hot encode the values.

Here’s how we can use the ColumnTransformer class to capture both of these tasks in one go.

def pipeline_transformer(data):
    '''
    Complete transformation pipeline for both
    nuerical and categorical data.

    Argument:
        data: original dataframe 
    Returns:
        prepared_data: transformed data, ready to use
    '''
    cat_attrs = ["Origin"]
    num_attrs, num_pipeline = num_pipeline_transformer(data)
    full_pipeline = ColumnTransformer([
        ("num", num_pipeline, list(num_attrs)),
        ("cat", OneHotEncoder(), cat_attrs),
        ])
    prepared_data = full_pipeline.fit_transform(data)
    return prepared_data

To the instance, provide the numerical pipeline object created from the function defined above. Then call the OneHotEncoder() class to process the Origin column.

Final Automation

With these classes and functions defined, we now have to integrate them into a single flow which is going to be simply two function calls.

1. Preprocessing the Origin Column to convert integers to Country names:

##preprocess the Origin column in data
def preprocess_origin_cols(df):
    df["Origin"] = df["Origin"].map({1: "India", 2: "USA", 3: "Germany"})    
    return df

2. Calling the final pipeline_transformer function defined above:

##from raw data to processed data in 2 steps

preprocessed_df = preprocess_origin_cols(data)
prepared_data = pipeline_transformer(preprocessed_df)prepared_data

Voilà, your data is ready to use in just two steps!

The next step is to start training our ML models.

Selecting and Training Machine Learning Models

Since this is a regression problem, I chose to train the following models:

1. Linear Regression

2. Decision Tree Regressor

3. Random Forest Regressor

4. SVM Regressor

I’ll explain the flow for Linear Regression and then you can follow the same for all the others.

It’s a simple 4-step process:

1. Create an instance of the model class.

2. Train the model using the fit() method.

3. Make predictions by first passing the data through pipeline transformer.

4. Evaluating the model using Root Mean Squared Error (typical performance metric for regression problems)

from sklearn.linear_model import LinearRegression

lin_reg = LinearRegression()
lin_reg.fit(prepared_data, data_labels)

##testing the predictions with first 5 rows
sample_data = data.iloc[:5]
sample_labels = data_labels.iloc[:5]
sample_data_prepared = pipeline_transformer(sample_data)

print("Prediction of samples: ", lin_reg.predict(sample_data_prepared))

Evaluating model:

from sklearn.metrics import mean_squared_error

mpg_predictions = lin_reg.predict(prepared_data)
lin_mse = mean_squared_error(data_labels, mpg_predictions)
lin_rmse = np.sqrt(lin_mse)lin_rmse

RMSE for Linear regression: 2.95904

Cross-Validation and Hyperparameter Tuning using Sklearn

Now, if you perform the same for Decision Tree, you’ll see that you have achieved a 0.0 RMSE value which is not possible – there is no “perfect” Machine Learning Model (we’ve not reached that point yet).

Problem: we are testing our model on the same data we trained on, which is a problem. Now, we can’t use the test data yet until we finalize our best model that is ready to go into production.

Solution: Cross-Validation

Scikit-Learn’s K-fold cross-validation feature randomly splits the training set into K distinct subsets called folds. Then it trains and evaluates the model K times, picking a different fold for evaluation every time and training on the other K-1 folds.

The result is an array containing the K evaluation scores. Here’s how I did for 10 folds:

from sklearn.model_selection import cross_val_score

scores = cross_val_score(tree_reg, 
                         prepared_data, 
                         data_labels, 
                         scoring="neg_mean_squared_error", 
                         cv = 10)
tree_reg_rmse_scores = np.sqrt(-scores)

The scoring method gives you negative values to denote errors. So while calculating the square root, we have to add negation explicitly.

For Decision Tree, here is the list of all scores:

Take the average of these scores:

Fine-Tuning Hyperparameters

After testing all the models, you’ll find that RandomForestRegressor has performed the best but it still needs to be fine-tuned.

A model is like a radio station with a lot of knobs to handle and tune. Now, you can either tune all these knobs manually or provide a range of values/combinations that you want to test.

We use GridSearchCV to find out the best combination of hyperparameters for the RandomForest model:

from sklearn.model_selection import GridSearchCV

param_grid = [
    {'n_estimators': [3, 10, 30], 'max_features': [2, 4, 6, 8]},
    {'bootstrap': [False], 'n_estimators': [3, 10], 'max_features': [2, 3, 4]},
  ]

forest_reg = RandomForestRegressor()

grid_search = GridSearchCV(forest_reg, param_grid,
                           scoring='neg_mean_squared_error',
                           return_train_score=True,
                           cv=10,
                          )

grid_search.fit(prepared_data, data_labels)

GridSearchCV requires you to pass the parameter grid. This is a python dictionary with parameter names as keys mapped with the list of values you want to test for that param.

We can pass the model, scoring method, and cross-validation folds to it.

Train the model and it returns the best parameters and results for each combination of parameters:

Check Feature Importance

We can also check the feature importance by enlisting the features and zipping them up with the best_estimator’s feature importance attribute as follows:

# feature importances
feature_importances = grid_search.best_estimator_.feature_importances_

extra_attrs = ["acc_on_power", "acc_on_cyl"]
numerics = ['float64', 'int64']
num_attrs = list(data.select_dtypes(include=numerics))

attrs = num_attrs + extra_attrs
sorted(zip(attrs, feature_importances), reverse=True)

We see that acc_on_power, which is the derived feature, has turned out to be the most important feature.

You might want to keep iterating a few times before finalizing the best configuration.

The model is now ready with the best configuration.

Evaluate the Entire System

It’s time to evaluate this entire system:

##capturing the best configuration
final_model = grid_search.best_estimator_

##segregating the target variable from test set
X_test = strat_test_set.drop("MPG", axis=1)
y_test = strat_test_set["MPG"].copy()

##preprocessing the test data origin column
X_test_preprocessed = preprocess_origin_cols(X_test)

##preparing the data with final transformation
X_test_prepared = pipeline_transformer(X_test_preprocessed)

##making final predictions
final_predictions = final_model.predict(X_test_prepared)
final_mse = mean_squared_error(y_test, final_predictions)
final_rmse = np.sqrt(final_mse)

If you want to look at my complete project, here is the GitHub repository:

With that, you have your final model ready to go into production.

For deployment, we save our model into a file using the pickle model and develop a Flask web service to be deployed in Heroku. Let's see how that works now.

What do you need to deploy the application?

In order to deploy any trained model, you need the following:

• A trained model ready to deploy — save the model into a file to be further loaded and used by the web service.

• A web service — that gives a purpose for your model to be used in practice. For our fuel consumption model, it can be using the vehicle configuration to predict its efficiency. We’ll use Flask to develop this service.

• A cloud service provider — you need special cloud servers to deploy the application. For simplicity, we are going to use Heroku for this (I'll cover AWS and GCP in other articles).

Let’s get started by looking at each of these processes one by one.

Saving the Trained Model

Once you’re confident enough to take your trained and tested model into the production-ready environment, the first step is to save it into a .h5 or .bin file using a library like pickle .

Make sure you have pickle installed in your environment.

Next, let’s import the module and dump the model into a .bin file:

import pickle

##dump the model into a file
with open("model.bin", 'wb') as f_out:
    pickle.dump(final_model, f_out) # write final_model in .bin file
    f_out.close()  # close the file

This will save your model in your present working directory unless you specify some other path.

It’s time to test if we are able to use this file to load our model and make predictions. We are going to use the same vehicle config as we defined above:

##vehicle config
vehicle_config = {
    'Cylinders': [4, 6, 8],
    'Displacement': [155.0, 160.0, 165.5],
    'Horsepower': [93.0, 130.0, 98.0],
    'Weight': [2500.0, 3150.0, 2600.0],
    'Acceleration': [15.0, 14.0, 16.0],
    'Model Year': [81, 80, 78],
    'Origin': [3, 2, 1]
}

Let’s load the model from the file:

##loading the model from the saved file
with open('model.bin', 'rb') as f_in:
    model = pickle.load(f_in)

Make predictions on the vehicle_config:

##defined in prev_blog
predict_mpg(vehicle_config, model)


##output: array([34.83333333, 18.50666667, 20.56333333])

The output is the same as we predicted earlier using final_model.

Developing a web service

The next step is to package this model into a web service that, when given the data through a POST request, returns the MPG (Miles per Gallon) predictions as a response.

I am using the Flask web framework, a commonly used lightweight framework for developing web services in Python. In my opinion, it is probably the easiest way to implement a web service.

Flask gets you started with very little code and you don’t need to worry about the complexity of handling with HTTP requests and responses.

Here are the steps:

• Create a new directory for your flask application.

• Set up a dedicated environment with dependencies installed using pip.

• Install the following packages:

pandas
numpy
sklearn
flask
gunicorn
seaborn

The next step is to activate this environment and start developing a simple endpoint to test the application:

Create a new file, main.py and import the flask module:

from flask import Flask

Create a Flask app by instantiating the Flask class:

##creating a flask app and naming it "app"
app = Flask('app')

Create a route and a function corresponding to it that will return a simple string:

@app.route('/test', methods=['GET'])
def test():
    return 'Pinging Model Application!!'

The above code makes use of decorators — an advanced Python feature. You can read more about decorators here.

We don’t need a deep understanding of decorators, just that adding a decorator @app.route on top of the test() function assigns that web service address to that function.

Now, to run the application we need this last piece of code:

if __name__ == ‘__main__’:
    app.run(debug=True, host=’0.0.0.0', port=9696)

The run method starts our flask application service. The 3 parameters specify:

• debug=True — restarts the application automatically when it encounters any change in the code

• host=’0.0.0.0' — makes the web service public

• port=9696 — the port that we use to access the application

Now, in your terminal run the main.py:

python main.py

Opening the URL http://0.0.0.0:9696/test in your browser will print the response string on the webpage:

With the application now running, let’s run the model.

Create a new directory model_files to store all the model-related code.

In this directory, create a ml_model.py file which will contain the data preparation code and the predict function we wrote here.

Copy and paste the libraries you imported earlier in the article and the preprocessing/transformation functions. The file should look like this:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.impute import SimpleImputer

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.compose import ColumnTransformer


##functions


def preprocess_origin_cols(df):
    df["Origin"] = df["Origin"].map({1: "India", 2: "USA", 3: "Germany"})
    return df


acc_ix, hpower_ix, cyl_ix = 3, 5, 1

class CustomAttrAdder(BaseEstimator, TransformerMixin):
    def __init__(self, acc_on_power=True): # no *args or **kargs
        self.acc_on_power = acc_on_power
    def fit(self, X, y=None):
        return self  # nothing else to do
    def transform(self, X):
        acc_on_cyl = X[:, acc_ix] / X[:, cyl_ix]
        if self.acc_on_power:
            acc_on_power = X[:, acc_ix] / X[:, hpower_ix]
            return np.c_[X, acc_on_power, acc_on_cyl]

        return np.c_[X, acc_on_cyl]


def num_pipeline_transformer(data):
    numerics = ['float64', 'int64']

    num_attrs = data.select_dtypes(include=numerics)

    num_pipeline = Pipeline([
        ('imputer', SimpleImputer(strategy="median")),
        ('attrs_adder', CustomAttrAdder()),
        ('std_scaler', StandardScaler()),
        ])
    return num_attrs, num_pipeline


def pipeline_transformer(data):

    cat_attrs = ["Origin"]
    num_attrs, num_pipeline = num_pipeline_transformer(data)

    full_pipeline = ColumnTransformer([
        ("num", num_pipeline, list(num_attrs)),
        ("cat", OneHotEncoder(), cat_attrs),
        ])
    full_pipeline.fit_transform(data)
    return full_pipeline    


def predict_mpg(config, model):

    if type(config) == dict:
        df = pd.DataFrame(config)
    else:
        df = config

    preproc_df = preprocess_origin_cols(df)
    print(preproc_df)
    pipeline = pipeline_transformer(preproc_df)
    prepared_df = pipeline.transform(preproc_df)
    print(len(prepared_df[0]))
    y_pred = model.predict(prepared_df)
    return y_pred

In the same directory add your saved model.bin file as well.

Now, in the main.py we are going to import the predict_mpg function to make predictions. But to do that we are required to create an empty __init__.py file to tell Python that the directory is a package.

Your directory should have this tree:

Next up, define the predict/ route that will accept the vehicle_config from an HTTP POST request and return the predictions using the model and predict_mpg() method.

In your main.py, first import:

import pickle
from flask import Flask, request, jsonify
from model_files.ml_model import predict_mpg

Then add the predict route and the corresponding function:

@app.route('/predict', methods=['POST'])
def predict():
    vehicle = request.get_json()
    print(vehicle)
    with open('./model_files/model.bin', 'rb') as f_in:
        model = pickle.load(f_in)
        f_in.close()
    predictions = predict_mpg(vehicle, model)

    result = {
        'mpg_prediction': list(predictions)
    }
    return jsonify(result)

Here, we’ll only be accepting POST request for our function and thus we have methods=[‘POST’] in the decorator.

• First, we capture the data( vehicle_config) from our request using the get_json() method and store it in the variable vehicle.

• Then we load the trained model into the model variable from the file we have in the model_files folder.

• Now, we make the predictions by calling the predict_mpg function and passing the vehicle and model.

• We create a JSON response of this array returned in the predictions variable and return this JSON as the method response.

We can test this route using Postman or the requests package and then start the server running the main.py. Then in your notebook, add this code to send a POST request with the vehicle_config:

import requests

url = “http://localhost:9696/predict"
r = requests.post(url, json = vehicle_config)
r.text.strip()

##output: '{"mpg_predictions":[34.60333333333333,19.32333333333333,14.893333333333333]}'

Great! Now, comes the last part: this same functionality should work when deployed on a remote server.

Deploying the application on Heroku

To deploy this flask application on Heroku, you need to follow these very simple steps:

1. Create a Procfile in the main directory — this contains the command to get the run the application on the server.

2. Add the following in your Procfile:

web: gunicorn wsgi:app

We are using gunicorn (installed earlier) to deploy the application:

Gunicorn is a pure-Python HTTP server for WSGI applications. It allows you to run any Python application concurrently by running multiple Python processes within a single dyno. It provides a perfect balance of performance, flexibility, and configuration simplicity.

Now, create a wsgi.py file and add:

##importing the app from main file
from main import app

if __name__ == “__main__”: 
    app.run()

Make sure you delete the run code from the main.py .

Write all the python dependencies into requirements.txt.

You can use pip freeze > requirements.txt or simply put the above-mentioned list of packages + any other package that your application is using.

Now, using the terminal,

• initialize an empty git repository,

• add the files to the staging area,

• and commit files to the local repository:

$ git init 
$ git add .
$ git commit -m "Initial Commit"

Next, create a Heroku account if you haven’t already. Then login to the Heroku CLI:

heroku login

Approve the login from the browser as the page pops up.

Now create a flask app:

heroku create <name of your app>

I named it mpg-flask-app. It will create a flask app and will give us a URL on which the app will be deployed.

Finally, push all your code to Heroku remote:

$ git push heroku master

And Voilà! Your web service is now deployed on https://mpg-flask-app.herokuapp.com/predict.

Again, test the endpoint using the request package by sending the same vehicle config:

With that, you have all the major skills you need to start building more complex ML applications.

You can refer to my GitHub repository for this project.

And you can develop this entire project along with me:

Next Steps

This was still a simple project. For the next steps, I’d recommend you take up a more complex dataset – maybe pick up a classification problem and repeat these tasks until deployment.

Check out Data Science with Harshit — My YouTube Channel

Here is the complete tutorial (in playlist form) on my YouTube channel where you can follow me while working on this project.

With this channel, I plan to roll out a couple of series covering the entire data science space. Here is why you should be subscribing to the channel:

• These series would cover all the required/demanded quality tutorials on each of the topics and subtopics like Python fundamentals for Data Science.

• Explained Mathematics and derivations of why we do what we do in ML and Deep Learning.

• Podcasts with Data Scientists and Engineers at Google, Microsoft, Amazon, etc, and CEOs of big data-driven companies.

• Projects and instructions to implement the topics learned so far. Learn about new certifications, Bootcamp, and resources to crack those certifications like this TensorFlow Developer Certificate Exam by Google.

If this tutorial was helpful, you should check out my data science and machine learning courses on Wiplane Academy. They are comprehensive yet compact and helps you build a solid foundation of work to showcase.

Micro-frontends are the future of front end web development.

Inspired by microservices, which allow you to break up your backend into smaller pieces, micro-frontends allow you to build, test, and deploy pieces of your frontend app independently of each other.

Depending on the micro-frontend framework you choose, you can even have multiple micro-frontend apps — written in React, Angular, Vue, or anything else — coexisting peacefully together in the same larger app.

In this article, we’re going to develop an app composed of micro-frontends using single-spa and deploy it to Heroku.

We’ll set up continuous integration using Travis CI. Each CI pipeline will bundle the JavaScript for a micro-frontend app and then upload the resulting build artifacts to AWS S3.

Finally, we’ll make an update to one of the micro-frontend apps and see how it can be deployed to production independently of the other micro-frontend apps.

Overview of the Demo App

Demo app — end result

Before we discuss the step-by-step instructions, let’s get a quick overview of what makes up the demo app. This app is composed of four sub-apps:

1. A container app that serves as the main page container and coordinates the mounting and unmounting of the micro-frontend apps
2. A micro-frontend navbar app that’s always present on the page
3. A micro-frontend “page 1” app that only shows when active
4. A micro-frontend “page 2” app that also only shows when active

These four apps all live in separate repos, available on GitHub, which I’ve linked to above.

The end result is fairly simple in terms of the user interface, but, to be clear, the user interface isn’t the point here.

If you’re following along on your own machine, by the end of this article you too will have all the underlying infrastructure necessary to get started with your own micro-frontend app.

Alright, grab your scuba gear, because it’s time to dive in!

Creating the Container App

To generate the apps for this demo, we’re going to use a command-line interface (CLI) tool called create-single-spa. The version of create-single-spa at the time of writing is 1.10.0, and the version of single-spa installed via the CLI is 4.4.2.

We’ll follow these steps to create the container app (also sometimes called the root config):

mkdir single-spa-demo

cd single-spa-demo

mkdir single-spa-demo-root-config

cd single-spa-demo-root-config

npx create-single-spa

We’ll then follow the CLI prompts:

1. Select “single spa root config”
2. Select “yarn” or “npm” (I chose “yarn”)
3. Enter an organization name (I used “thawkin3,” but it can be whatever you want)

Great! Now, if you check out the single-spa-demo-root-config directory, you should see a skeleton root config app. We'll customize this in a bit, but first let's also use the CLI tool to create our other three micro-frontend apps.

Creating the Micro-Frontend Apps

To generate our first micro-frontend app, the navbar, we’ll follow these steps:

cd ..

mkdir single-spa-demo-nav

cd single-spa-demo-nav

npx create-single-spa

We’ll then follow the CLI prompts:

1. Select “single-spa application / parcel”
2. Select “react”
3. Select “yarn” or “npm” (I chose “yarn”)
4. Enter an organization name, the same one you used when creating the root config app (“thawkin3” in my case)
5. Enter a project name (I used “single-spa-demo-nav”)

Now that we’ve created the navbar app, we can follow these same steps to create our two page apps. But, we’ll replace each place we see “single-spa-demo-nav” with “single-spa-demo-page-1” the first time through and then with “single-spa-demo-page-2” the second time through.

At this point we’ve generated all four apps that we need: one container app and three micro-frontend apps. Now it’s time to hook our apps together.

Registering the Micro-Frontend Apps with the Container App

As stated before, one of the container app’s primary responsibilities is to coordinate when each app is “active” or not. In other words, it handles when each app should be shown or hidden.

To help the container app understand when each app should be shown, we provide it with what are called “activity functions.” Each app has an activity function that simply returns a boolean, true or false, for whether or not the app is currently active.

Inside the single-spa-demo-root-config directory, in the activity-functions.js file, we'll write the following activity functions for our three micro-frontend apps.

export function prefix(location, ...prefixes) {
  return prefixes.some(
    prefix => location.href.indexOf(`${location.origin}/${prefix}`) !== -1
  );
}

export function nav() {
  // The nav is always active
  return true;
}

export function page1(location) {
  return prefix(location, 'page1');
}

export function page2(location) {
  return prefix(location, 'page2');
}

Next, we need to register our three micro-frontend apps with single-spa. To do that, we use the registerApplication function. This function accepts a minimum of three arguments: the app name, a method to load the app, and an activity function to determine when the app is active.

Inside the single-spa-demo-root-config directory, in the root-config.js file, we'll add the following code to register our apps:

import { registerApplication, start } from "single-spa";
import * as isActive from "./activity-functions";

registerApplication(
  "@thawkin3/single-spa-demo-nav",
  () => System.import("@thawkin3/single-spa-demo-nav"),
  isActive.nav
);

registerApplication(
  "@thawkin3/single-spa-demo-page-1",
  () => System.import("@thawkin3/single-spa-demo-page-1"),
  isActive.page1
);

registerApplication(
  "@thawkin3/single-spa-demo-page-2",
  () => System.import("@thawkin3/single-spa-demo-page-2"),
  isActive.page2
);

start();

Now that we’ve set up the activity functions and registered our apps, the last step before we can get this running locally is to update the local import map inside the index.ejs file in the same directory.

We'll add the following code inside the head tag to specify where each app can be found when running locally:

<% if (isLocal) { %>
  <script type="systemjs-importmap">
    {
      "imports": {
        "@thawkin3/root-config": "http://localhost:9000/root-config.js",
        "@thawkin3/single-spa-demo-nav": "http://localhost:9001/thawkin3-single-spa-demo-nav.js",
        "@thawkin3/single-spa-demo-page-1": "http://localhost:9002/thawkin3-single-spa-demo-page-1.js",
        "@thawkin3/single-spa-demo-page-2": "http://localhost:9003/thawkin3-single-spa-demo-page-2.js"
      }
    }
  </script>
<% } %>

Each app contains its own startup script, which means that each app will be running locally on its own development server during local development. As you can see, our navbar app is on port 9001, our page 1 app is on port 9002, and our page 2 app is on port 9003.

With those three steps taken care of, let’s try out our app.

Test Run for Running Locally

To get our app running locally, we can follow these steps:

1. Open four terminal tabs, one for each app
2. For the root config, in the single-spa-demo-root-config directory: yarn start (runs on port 9000 by default)
3. For the nav app, in the single-spa-demo-nav directory: yarn start --port 9001
4. For the page 1 app, in the single-spa-demo-page-1 directory: yarn start --port 9002
5. For the page 2 app, in the single-spa-demo-page-2 directory: yarn start --port 9003

Now, we’ll navigate in the browser to http://localhost:9000 to view our app.

We should see… some text! Super exciting.

Demo app — main page

On our main page, the navbar is showing because the navbar app is always active.

Now, let’s navigate to http://localhost:9000/page1. As shown in our activity functions above, we’ve specified that the page 1 app should be active (shown) when the URL path begins with “page1.” So, this activates the page 1 app, and we should see the text for both the navbar and the page 1 app now.

Demo app — page 1 route

One more time, let’s now navigate to http://localhost:9000/page2. As expected, this activates the page 2 app, so we should see the text for the navbar and the page 2 app now.

Demo app — page 2 route

Making Minor Tweaks to the Apps

So far our app isn’t very exciting to look at, but we do have a working micro-frontend setup running locally. If you aren’t cheering in your seat right now, you should be!

Let’s make some minor improvements to our apps so they look and behave a little nicer.

Specifying the Mount Containers

First, if you refresh your page over and over when viewing the app, you may notice that sometimes the apps load out of order, with the page app appearing above the navbar app.

This is because we haven’t actually specified where each app should be mounted. The apps are simply loaded by SystemJS, and then whichever app finishes loading fastest gets appended to the page first.

We can fix this by specifying a mount container for each app when we register them.

In our index.ejs file that we worked in previously, let's add some HTML to serve as the main content containers for the page:

<div id="nav-container"></div>
<main>
  <div id="page-1-container"></div>
  <div id="page-2-container"></div>
</main>

Then, in our root-config.js file where we've registered our apps, let's provide a fourth argument to each function call that includes the DOM element where we'd like to mount each app:

import { registerApplication, start } from "single-spa";
import * as isActive from "./activity-functions";

registerApplication(
  "@thawkin3/single-spa-demo-nav",
  () => System.import("@thawkin3/single-spa-demo-nav"),
  isActive.nav,
  { domElement: document.getElementById('nav-container') }
);

registerApplication(
  "@thawkin3/single-spa-demo-page-1",
  () => System.import("@thawkin3/single-spa-demo-page-1"),
  isActive.page1,
  { domElement: document.getElementById('page-1-container') }
);

registerApplication(
  "@thawkin3/single-spa-demo-page-2",
  () => System.import("@thawkin3/single-spa-demo-page-2"),
  isActive.page2,
  { domElement: document.getElementById('page-2-container') }
);

start();

Now, the apps will always be mounted to a specific and predictable location. Nice!

Styling the App

Next, let’s style up our app a bit. Plain black text on a white background isn’t very interesting to look at.

In the single-spa-demo-root-config directory, in the index.ejs file again, we can add some basic styles for the whole app by pasting the following CSS at the bottom of the head tag:

<style>
  body, html { margin: 0; padding: 0; font-size: 16px; font-family: Arial, Helvetica, sans-serif; height: 100%; }
  body { display: flex; flex-direction: column; }
  * { box-sizing: border-box; }
</style>

Next, we can style our navbar app by finding the single-spa-demo-nav directory, creating a root.component.css file, and adding the following CSS:

.nav {
  display: flex;
  flex-direction: row;
  padding: 20px;
  background: #000;
  color: #fff;
}

.link {
  margin-right: 20px;
  color: #fff;
  text-decoration: none;
}

.link:hover,
.link:focus {
  color: #1098f7;
}

We can then update the root.component.js file in the same directory to import the CSS file and apply those classes and styles to our HTML. We'll also change the navbar content to actually contain two links so we can navigate around the app by clicking the links instead of entering a new URL in the browser's address bar.

import React from "react";
import "./root.component.css";

export default function Root() {
  return (
    <nav className="nav">
      <a href="/page1" className="link">
        Page 1
      </a>
      <a href="/page2" className="link">
        Page 2
      </a>
    </nav>
  );
}

We’ll follow a similar process for the page 1 and page 2 apps as well. We’ll create a root.component.css file for each app in their respective project directories and update the root.component.js files for both apps too.

For the page 1 app, the changes look like this:

.container1 {
  background: #1098f7;
  color: white;
  padding: 20px;
  display: flex;
  align-items: center;
  justify-content: center;
  flex: 1;
  font-size: 3rem;
}

import React from "react";
import "./root.component.css";

export default function Root() {
  return (
    <div className="container1">
      <p>Page 1 App</p>
    </div>
  );
}

And for the page 2 app, the changes look like this:

.container2 {
  background: #9e4770;
  color: white;
  padding: 20px;
  display: flex;
  align-items: center;
  justify-content: center;
  flex: 1;
  font-size: 3rem;
}

import React from "react";
import "./root.component.css";

export default function Root() {
  return (
    <div className="container2">
      <p>Page 2 App</p>
    </div>
  );
}

Adding React Router

The last small change we’ll make is to add React Router to our app. Right now the two links we’ve placed in the navbar are just normal anchor tags, so navigating from page to page causes a page refresh. Our app will feel much smoother if the navigation is handled client-side with React Router.

To use React Router, we’ll first need to install it. From the terminal, in the single-spa-demo-nav directory, we'll install React Router using yarn by entering yarn add react-router-dom. (Or if you're using npm, you can enter npm install react-router-dom.)

Then, in the single-spa-demo-nav directory in the root.component.js file, we'll replace our anchor tags with React Router's Link components like so:

import React from "react";
import { BrowserRouter, Link } from "react-router-dom";
import "./root.component.css";

export default function Root() {
  return (
    <BrowserRouter>
      <nav className="nav">
        <Link to="/page1" className="link">
          Page 1
        </Link>
        <Link to="/page2" className="link">
          Page 2
        </Link>
      </nav>
    </BrowserRouter>
  );
}

Cool. That looks and works much better!

Demo app — styled and using React Router

Getting Ready for Production

At this point we have everything we need to continue working on the app while running it locally. But how do we get it hosted somewhere publicly available?

There are several possible approaches we can take using our tools of choice, but the main tasks are:

1. to have somewhere we can upload our build artifacts, like a CDN, and
2. to automate this process of uploading artifacts each time we merge new code into the master branch.

For this article, we’re going to use AWS S3 to store our assets, and we’re going to use Travis CI to run a build job and an upload job as part of a continuous integration pipeline.

Let’s get the S3 bucket set up first.

Setting up the AWS S3 Bucket

It should go without saying, but you’ll need an AWS account if you’re following along here.

If we are the root user on our AWS account, we can create a new IAM user that has programmatic access only. This means we’ll be given an access key ID and a secret access key from AWS when we create the new user. We’ll want to store these in a safe place since we’ll need them later.

Finally, this user should be given permissions to work with S3 only, so that the level of access is limited if our keys were to fall into the wrong hands.

AWS has some great resources for best practices with access keys and managing access keys for IAM users that would be well worth checking out if you’re unfamiliar with how to do this.

Next we need to create an S3 bucket. S3 stands for Simple Storage Service and is essentially a place to upload and store files hosted on Amazon’s servers. A bucket is simply a directory.

I’ve named my bucket “single-spa-demo,” but you can name yours whatever you’d like. You can follow the AWS guides for how to create a new bucket for more info.

AWS S3 bucket

Once we have our bucket created, it’s also important to make sure the bucket is public and that CORS (cross-origin resource sharing) is enabled for our bucket so that we can access and use our uploaded assets in our app.

In the permissions for our bucket, we can add the following CORS configuration rules:

<CORSConfiguration>
 <CORSRule>
   <AllowedOrigin>*</AllowedOrigin>
   <AllowedMethod>GET</AllowedMethod>
 </CORSRule>
</CORSConfiguration>

In the AWS console, it ends up looking like this after we hit Save:

CORS configuration

Creating a Travis CI Job to Upload Artifacts to AWS S3

Now that we have somewhere to upload files, let’s set up an automated process that will take care of uploading new JavaScript bundles each time we merge new code into the master branch for any of our repos.

To do this, we’re going to use Travis CI. As mentioned earlier, each app lives in its own repo on GitHub, so we have four GitHub repos to work with. We can integrate Travis CI with each of our repos and set up continuous integration pipelines for each one.

To configure Travis CI for any given project, we create a .travis.yml file in the project's root directory. Let's create that file in the single-spa-demo-root-config directory and insert the following code:

language: node_js
node_js:
  - node
script:
  - yarn build
  - echo "Commit sha - $TRAVIS_COMMIT"
  - mkdir -p dist/@thawkin3/root-config/$TRAVIS_COMMIT
  - mv dist/*.* dist/@thawkin3/root-config/$TRAVIS_COMMIT/
deploy:
  provider: s3
  access_key_id: "$AWS_ACCESS_KEY_ID"
  secret_access_key: "$AWS_SECRET_ACCESS_KEY"
  bucket: "single-spa-demo"
  region: "us-west-2"
  cache-control: "max-age=31536000"
  acl: "public_read"
  local_dir: dist
  skip_cleanup: true
  on:
    branch: master

This implementation is what I came up with after reviewing the Travis CI docs for AWS S3 uploads and a single-spa Travis CI example config.

Because we don’t want our AWS secrets exposed in our GitHub repo, we can store those as environment variables. You can place environment variables and their secret values within the Travis CI web console for anything that you want to keep private, so that’s where the .travis.yml file gets those values from.

Now, when we commit and push new code to the master branch, the Travis CI job will run, which will build the JavaScript bundle for the app and then upload those assets to S3. To verify, we can check out the AWS console to see our newly uploaded files:

Uploaded files as a result of a Travis CI job

Neat! So far so good. Now we need to implement the same Travis CI configuration for our other three micro-frontend apps, but swapping out the directory names in the .travis.yml file as needed. After following the same steps and merging our code, we now have four directories created in our S3 bucket, one for each repo.

Four directories within our S3 bucket

Creating an Import Map for Production

Let’s recap what we’ve done so far. We have four apps, all living in separate GitHub repos. Each repo is set up with Travis CI to run a job when code is merged into the master branch, and that job handles uploading the build artifacts into an S3 bucket.

With all that in one place, there’s still one thing missing: How do these new build artifacts get referenced in our container app? In other words, even though we’re pushing up new JavaScript bundles for our micro-frontends with each new update, the new code isn’t actually used in our container app yet!

If we think back to how we got our app running locally, we used an import map. This import map is simply JSON that tells the container app where each JavaScript bundle can be found.

But, our import map from earlier was specifically used for running the app locally. Now we need to create an import map that will be used in the production environment.

If we look in the single-spa-demo-root-config directory, in the index.ejs file, we see this line:

<script type="systemjs-importmap" src="https://storage.googleapis.com/react.microfrontends.app/importmap.json"></script>

Opening up that URL in the browser reveals an import map that looks like this:

{
  "imports": {
    "react": "https://cdn.jsdelivr.net/npm/react@16.13.1/umd/react.production.min.js",
    "react-dom": "https://cdn.jsdelivr.net/npm/react-dom@16.13.1/umd/react-dom.production.min.js",
    "single-spa": "https://cdn.jsdelivr.net/npm/single-spa@5.5.3/lib/system/single-spa.min.js",
    "@react-mf/root-config": "https://react.microfrontends.app/root-config/e129469347bb89b7ff74bcbebb53cc0bb4f5e27f/react-mf-root-config.js",
    "@react-mf/navbar": "https://react.microfrontends.app/navbar/631442f229de2401a1e7c7835dc7a56f7db606ea/react-mf-navbar.js",
    "@react-mf/styleguide": "https://react.microfrontends.app/styleguide/f965d7d74e99f032c27ba464e55051ae519b05dd/react-mf-styleguide.js",
    "@react-mf/people": "https://react.microfrontends.app/people/dd205282fbd60b09bb3a937180291f56e300d9db/react-mf-people.js",
    "@react-mf/api": "https://react.microfrontends.app/api/2966a1ca7799753466b7f4834ed6b4f2283123c5/react-mf-api.js",
    "@react-mf/planets": "https://react.microfrontends.app/planets/5f7fc62b71baeb7a0724d4d214565faedffd8f61/react-mf-planets.js",
    "@react-mf/things": "https://react.microfrontends.app/things/7f209a1ed9ac9690835c57a3a8eb59c17114bb1d/react-mf-things.js",
    "rxjs": "https://cdn.jsdelivr.net/npm/@esm-bundle/rxjs@6.5.5/system/rxjs.min.js",
    "rxjs/operators": "https://cdn.jsdelivr.net/npm/@esm-bundle/rxjs@6.5.5/system/rxjs-operators.min.js"
  }
}

That import map was the default one provided as an example when we used the CLI to generate our container app. What we need to do now is replace this example import map with an import map that actually references the bundles we’re using.

So, using the original import map as a template, we can create a new file called importmap.json, place it outside of our repos and add JSON that looks like this:

{
  "imports": {
    "react": "https://cdn.jsdelivr.net/npm/react@16.13.0/umd/react.production.min.js",
    "react-dom": "https://cdn.jsdelivr.net/npm/react-dom@16.13.0/umd/react-dom.production.min.js",
    "single-spa": "https://cdn.jsdelivr.net/npm/single-spa@5.5.1/lib/system/single-spa.min.js",
    "@thawkin3/root-config": "https://single-spa-demo.s3-us-west-2.amazonaws.com/%40thawkin3/root-config/179ba4f2ce4d517bf461bee986d1026c34967141/root-config.js",
    "@thawkin3/single-spa-demo-nav": "https://single-spa-demo.s3-us-west-2.amazonaws.com/%40thawkin3/single-spa-demo-nav/f0e9d35392ea0da8385f6cd490d6c06577809f16/thawkin3-single-spa-demo-nav.js",
    "@thawkin3/single-spa-demo-page-1": "https://single-spa-demo.s3-us-west-2.amazonaws.com/%40thawkin3/single-spa-demo-page-1/4fd417ee3faf575fcc29d17d874e52c15e6f0780/thawkin3-single-spa-demo-page-1.js",
    "@thawkin3/single-spa-demo-page-2": "https://single-spa-demo.s3-us-west-2.amazonaws.com/%40thawkin3/single-spa-demo-page-2/8c58a825c1552aab823bcbd5bdd13faf2bd4f9dc/thawkin3-single-spa-demo-page-2.js"
  }
}

You’ll note that the first three imports are for shared dependencies: react, react-dom, and single-spa. That way we don’t have four copies of React in our app causing bloat and longer download times. Next, we have imports for each of our four apps. The URL is simply the URL for each uploaded file in S3 (called an “object” in AWS terminology).

Now that we have this file created, we can manually upload it to our bucket in S3 through the AWS console.

Note: This is a pretty important and interesting caveat when using single-spa: The import map doesn’t actually live anywhere in source control or in any of the git repos. That way, the import map can be updated on the fly without requiring checked-in changes in a repo. We’ll come back to this concept in a little bit.

Import map manually uploaded to the S3 bucket

Finally, we can now reference this new file in our index.ejs file instead of referencing the original import map.

<script type="systemjs-importmap" src="//single-spa-demo.s3-us-west-2.amazonaws.com/%40thawkin3/importmap.json"></script>

Creating a Production Server

We are getting closer to having something up and running in production! We’re going to host this demo on Heroku, so in order to do that, we’ll need to create a simple Node.js and Express server to serve our file.

First, in the single-spa-demo-root-config directory, we'll install express by running yarn add express (or npm install express). Next, we'll add a file called server.js that contains a small amount of code for starting up an express server and serving our main index.html file.

const express = require("express");
const path = require("path");
const PORT = process.env.PORT || 5000;

express()
  .use(express.static(path.join(__dirname, "dist")))
  .get("*", (req, res) => {
    res.sendFile("index.html", { root: "dist" });
  })
  .listen(PORT, () => console.log(`Listening on ${PORT}`));

Finally, we’ll update the NPM scripts in our package.json file to differentiate between running the server in development mode and running the server in production mode.

"scripts": {
  "build": "webpack --mode=production",
  "lint": "eslint src",
  "prettier": "prettier --write './**'",
  "start:dev": "webpack-dev-server --mode=development --port 9000 --env.isLocal=true",
  "start": "node server.js",
  "test": "jest"
}

Deploying to Heroku

Now that we have a production server ready, let’s get this thing deployed to Heroku! In order to do so, you’ll need to have a Heroku account created, the Heroku CLI installed, and be logged in. Deploying to Heroku is as easy as 1–2–3:

1. In the single-spa-demo-root-config directory: heroku create thawkin3-single-spa-demo (changing that last argument to a unique name to be used for your Heroku app)
2. git push heroku master
3. heroku open

And with that, we are up and running in production. Upon running the heroku open command, you should see your app open in your browser. Try navigating between pages using the nav links to see the different micro-frontend apps mount and unmount.

Demo app — up and running in production

Making Updates

At this point, you may be asking yourself, “All that work for this? Why?” And you’d be right. Sort of. This is a lot of work, and we don’t have much to show for it, at least not visually. But, we’ve laid the groundwork for whatever app improvements we’d like.

The setup cost for any microservice or micro-frontend is often a lot higher than the setup cost for a monolith; it’s not until later that you start to reap the rewards.

So let’s start thinking about future modifications. Let’s say that it’s now five or ten years later, and your app has grown. A lot. And, in that time, a hot new framework has been released, and you’re dying to re-write your entire app using that new framework.

When working with a monolith, this would likely be a years-long effort and may be nearly impossible to accomplish. But, with micro-frontends, you could swap out technologies one piece of the app at a time, allowing you to slowly and smoothly transition to a new tech stack. Magic!

Or, you may have one piece of your app that changes frequently and another piece of your app that is rarely touched. While making updates to the volatile app, wouldn’t it be nice if you could just leave the legacy code alone?

With a monolith, it’s possible that changes you make in one place of your app may affect other sections of your app. What if you modified some stylesheets that multiple sections of the monolith were using? Or what if you updated a dependency that was used in many different places?

With a micro-frontend approach, you can leave those worries behind, refactoring and updating one app where needed while leaving legacy apps alone.

But, how do you make these kinds of updates? Or updates of any sort, really?

Right now we have our production import map in our index.ejs file, but it's just pointing to the file we manually uploaded to our S3 bucket. If we wanted to release some new changes right now, we'd need to push new code for one of the micro-frontends, get a new build artifact, and then manually update the import map with a reference to the new JavaScript bundle.

Is there a way we could automate this? Yes!

Updating One of the Apps

Let’s say we want to update our page 1 app to have different text showing. In order to automate the deployment of this change, we can update our CI pipeline to not only build an artifact and upload it to our S3 bucket, but to also update the import map to reference the new URL for the latest JavaScript bundle.

Let’s start by updating our .travis.yml file like so:

language: node_js
node_js:
  - node
env:
  global:
    # include $HOME/.local/bin for `aws`
    - PATH=$HOME/.local/bin:$PATH
before_install:
  - pyenv global 3.7.1
  - pip install -U pip
  - pip install awscli
script:
  - yarn build
  - echo "Commit sha - $TRAVIS_COMMIT"
  - mkdir -p dist/@thawkin3/root-config/$TRAVIS_COMMIT
  - mv dist/*.* dist/@thawkin3/root-config/$TRAVIS_COMMIT/
deploy:
  provider: s3
  access_key_id: "$AWS_ACCESS_KEY_ID"
  secret_access_key: "$AWS_SECRET_ACCESS_KEY"
  bucket: "single-spa-demo"
  region: "us-west-2"
  cache-control: "max-age=31536000"
  acl: "public_read"
  local_dir: dist
  skip_cleanup: true
  on:
    branch: master
after_deploy:
  - chmod +x after_deploy.sh
  - "./after_deploy.sh"

The main changes here are adding a global environment variable, installing the AWS CLI, and adding an after_deploy script as part of the pipeline. This references an after_deploy.sh file that we need to create. The contents will be:

echo "Downloading import map from S3"
aws s3 cp s3://single-spa-demo/@thawkin3/importmap.json importmap.json
echo "Updating import map to point to new version of @thawkin3/root-config"
node update-importmap.mjs
echo "Uploading new import map to S3"
aws s3 cp importmap.json s3://single-spa-demo/@thawkin3/importmap.json --cache-control 'public, must-revalidate, max-age=0' --acl 'public-read'
echo "Deployment successful"

This file downloads the existing import map from S3, modifies it to reference the new build artifact, and then re-uploads the updated import map to S3. To handle the actual updating of the import map file’s contents, we use a custom script that we’ll add in a file called update-importmap.mjs.

// Note that this file requires node@13.2.0 or higher (or the --experimental-modules flag)
import fs from "fs";
import path from "path";
import https from "https";

const importMapFilePath = path.resolve(process.cwd(), "importmap.json");
const importMap = JSON.parse(fs.readFileSync(importMapFilePath));
const url = `https://single-spa-demo.s3-us-west-2.amazonaws.com/%40thawkin3/root-config/${process.env.TRAVIS_COMMIT}/root-config.js`;

https
  .get(url, res => {
    // HTTP redirects (301, 302, etc) not currently supported, but could be added
    if (res.statusCode >= 200 && res.statusCode < 300) {
      if (
        res.headers["content-type"] &&
        res.headers["content-type"].toLowerCase().trim() ===
          "application/javascript"
      ) {
        const moduleName = `@thawkin3/root-config`;
        importMap.imports[moduleName] = url;
        fs.writeFileSync(importMapFilePath, JSON.stringify(importMap, null, 2));
        console.log(
          `Updated import map for module ${moduleName}. New url is ${url}.`
        );
      } else {
        urlNotDownloadable(
          url,
          Error(`Content-Type response header must be application/javascript`)
        );
      }
    } else {
      urlNotDownloadable(
        url,
        Error(`HTTP response status was ${res.statusCode}`)
      );
    }
  })
  .on("error", err => {
    urlNotDownloadable(url, err);
  });

function urlNotDownloadable(url, err) {
  throw Error(
    `Refusing to update import map - could not download javascript file at url ${url}. Error was '${err.message}'`
  );
}

Note that we need to make these changes for these three files in all of our GitHub repos so that each one is able to update the import map after creating a new build artifact.

The file contents will be nearly identical for each repo, but we’ll need to change the app names or URL paths to the appropriate values for each one.

A Side Note on the Import Map

Earlier I mentioned that the import map file we manually uploaded to S3 doesn’t actually live anywhere in any of our GitHub repos or in any of our checked-in code. If you’re like me, this probably seems really odd! Shouldn’t everything be in source control?

The reason it’s not in source control is so that our CI pipeline can handle updating the import map with each new micro-frontend app release.

If the import map were in source control, making an update to one micro-frontend app would require changes in two repos: the micro-frontend app repo where the change is made, and the root config repo where the import map would be checked in. This sort of setup would invalidate one of micro-frontend architecture’s main benefits, which is that each app can be deployed completely independent of the other apps.

In order to achieve some level of source control on the import map, we can always use S3’s versioning feature for our bucket.

Moment of Truth

With those modifications to our CI pipelines in place, it’s time for the final moment of truth: Can we update one of our micro-frontend apps, deploy it independently, and then see those changes take effect in production without having to touch any of our other apps?

In the single-spa-demo-page-1 directory, in the root.component.js file, let's change the text from "Page 1 App" to "Page 1 App - UPDATED!" Next, let's commit that change and push and merge it to master.

This will kick off the Travis CI pipeline to build the new page 1 app artifact and then update the import map to reference that new file URL.

If we then navigate in our browser to https://thawkin3-single-spa-demo.herokuapp.com/page1, we’ll now see… drum roll please… our updated app!

Demo app — successfully updating one of the micro-frontend apps

Conclusion

I said it before, and I’ll say it again: Micro-frontends are the future of frontend web development.

The benefits are massive, including independent deployments, independent areas of ownership, faster build and test times, and the ability to mix and match various frameworks if needed.

There are some drawbacks, such as the initial set up cost and the complexity of maintaining a distributed architecture, but I strongly believe the benefits outweigh the costs.

Single-spa makes micro-frontend architecture easy. Now you, too, can go break up the monolith!

Building a full stack web app is no mean feat. Learning to deploy one to production so that you can share it with the world can add an additional layer of complexity.

In this new tutorial, we'll learn how to deploy a full stack MEVN app to Heroku!

Follow along (46 minute watch):

You can follow the original tutorial to build this app here.

Happy coding!

If you enjoyed this article, please consider checking out my games and books, subscribing to my YouTube channel, or joining the Entromancy Discord.

M. S. Farzan, Ph.D. has written and worked for high-profile video game companies and editorial websites such as Electronic Arts, Perfect World Entertainment, Modus Games, and MMORPG.com, and has served as the Community Manager for games like Dungeons & Dragons Neverwinter and Mass Effect: Andromeda. He is the Creative Director and Lead Game Designer of Entromancy: A Cyberpunk Fantasy RPG and author of The Nightpath Trilogy. Find M. S. Farzan on Twitter @sominator.

Building a full stack app is no small endeavor, and deploying it comes with its own host of things to consider.

I'm a tabletop game developer, and recently deployed a simple roleplaying game tracker that uses the M-E-V-N stack (you can follow my tutorial to create your own app here).

In deploying the app, I came across three key takeaways that may be useful as you begin considering the best way to bring your projects from development to production.

You can check out the code to my app on GitHub, and I should mention that it includes Chad Carteret's very cool CSS statblock in prettifying what's otherwise very basic HTML.

If you're thinking of following the same path to deployment as I have here, be sure to check out the official documentation on Heroku, the Vue CLI, and this tutorial by Nick Manning.

You'll also want to take a look at Will Abramson's article on a similar topic.

On to deployment!

Your front end and back end can be deployed together or separately, depending on your app's complexity.

One snag that seems to appear immediately when considering production is the structural question of how to deploy the front and back ends of your app.

Should the client (or static files) live in the same place as the server and database? Or should they be separate, with the front end making HTTP requests from elsewhere to the back end using CORS?

The answer is yes! Or no. Maybe??

For better or worse, there's no one-size-fits-all solution to this question, as it will likely depend on your app's architecture and complexity. In the roleplaying game tracker that I linked to above, I have the whole stack running on a single Heroku dyno, with the following folder structure:

All of the front and back end files live in the same place, with the Vue client built for production in a folder located at /client/dist.

In server.js, along with a bunch of database and routing code, there's a little line that says:

server.use(serveStatic(__dirname + "/client/dist"));

In Express, this tells the app to serve my static client files from a particular folder, and enables me to run the front and back ends all within the same environment. If you're deploying a similarly simple app, this type of solution might work for you as well.

Conversely, and depending on your project's complexity, you may have to separate the front and back ends and treat them as separate applications, which is no big deal. In the app above, my client is making calls to static API endpoints that are handled by the server, like this:

getQuests: function () {
    axios
        .get('https://mevn-rpg-app.herokuapp.com/quests')
        .then(response => (this.questData = response.data))                   
 }

Technically, my client could be making those requests from anywhere - even a static GitHub Pages site. This type of solution can help separate your app into two distinct entities to tackle, which is sometimes better than trying to cram the whole project into one location.

One note: if you're going to be making cross-origin HTTP requests - that is, requests from a client that lives on a separate domain from the API or server - you'll need to become familiar with CORS. You can read more about it in this article.

Your code will need to change to support a production environment.

When you're knee deep in the development process, it can be easy to lose track of how much of your code depends on local files or other data.

Consider the following in a locally-running server.js:

server.listen(3000, () => console.log("Server started!"));

On a local machine, the code just asks the server to listen on port 3000 and log to the console that we're ready for liftoff.

In a production environment, the server has no concept of where the "localhost" should be, or to whose port 3000 it should be listening. With this example, you'd have to change your code to something like:

const port = process.env.PORT || 3000;

server.listen(port, () => console.log("Server started!"));

The above instructs the server to instead listen at port 3000 of the process that's currently running, wherever that might be (check out this article for further reading on this topic).

Similarly, in my app, I have several modules that need to be imported by one another to function:

In /routes/Quests.js, for example, I have a router that tells the server what to do when receiving API requests to interact with quest-related items in the database. The router needs to import a Mongoose schema from /models/quest.js to function properly. If the application were running locally, we could just say:

const Quest = require('../models/quest');

Pretty simple! Yet, unfortunately, our server won't know where to find the root directory of our project once deployed. In Express, we'd change our code to something like:

const path = require('path');
const Quest = require(path.join(__dirname, '../models/quest'));

Your particular case might be different, depending on your language and framework(s), but you'll need to get specific about what your code looks like in a production environment rather than in your local development environment.

Additionally, you're probably already familiar with whatever bundler you're using for your front end (e.g., webpack), and will want to build your client for production to optimize it for deployment.

You have a multitude of deployment platforms from which to choose.

If you've deployed a front end website or some other type of static app, you might be familiar with just pushing your files to some remote repository and calling it a day.

Deploying a full stack app (or even just a back end) is eminently more complex. You'll need a dedicated server, or something that emulates one, to respond to the HTTP requests that it will be receiving and work with an online database.

There are a number of services that will do this very thing for you, and the spectrum ranges based on price, scalability, complexity, and other factors.

There's a bunch of articles out there that compare PaaS options for deployment, but here are some thoughts as you consider platforms for your first project:

• Heroku: If you have a small project like mine or just want to learn about deployment, a good first step could be Heroku.
• AWS, Docker, and Kubernetes: If you're seeking a career in full stack web development or DevOps, now's a good time to familiarize yourself with Amazon Web Services and/or container platforms like Docker and Kubernetes.
• Azure: If you're a C# or .NET developer, Azure appears to be a seamless way to deploy your apps without having to leave the safety of the Microsoft ecosystem.

There are, of course, several other options out there, and your particular use-case scenario might depend on pricing or the specific feature sets that are on offer.

Additionally, you'll want to consider any addons that will be necessary to replicate your app's functionality in a production environment. My roleplaying game tracker, for example, uses MongoDB, but the production version certainly can't use the little database on my local machine! Instead, I've used the mLab Heroku addon to get the live site up and running with the same functionality as in my development environment.

Your app's success, as well as your own progress as a full stack web developer, depend on your ability to consider deployment options and create a successful pipeline for production. With a little research, I'm certain that you can find the best solution that fits all of your app's needs.

Happy coding!

If you enjoyed this article, please consider checking out my games and books, subscribing to my YouTube channel, or joining the Entromancy Discord.

M. S. Farzan, Ph.D. has written and worked for high-profile video game companies and editorial websites such as Electronic Arts, Perfect World Entertainment, Modus Games, and MMORPG.com, and has served as the Community Manager for games like Dungeons & Dragons Neverwinter and Mass Effect: Andromeda. He is the Creative Director and Lead Game Designer of Entromancy: A Cyberpunk Fantasy RPG and author of The Nightpath Trilogy. Find M. S. Farzan on Twitter @sominator.

If you are new to the world of web development, you will spend a lot of time learning how to build static sites with HTML, CSS and JavaScript.

You might then start learning how to use popular frameworks such as React, VueJS or Angular.

But after trying out a few new ideas and running some sites locally, you might wonder how to actually deploy your site or app. And as it turns out, it can sometimes be difficult to know where to start.

Personally, I find running an Express server hosted on Heroku one of the simplest ways to get going. This article will show you how to do this.

Heroku is a cloud platform which supports a number of different programming languages and frameworks.

This is not a sponsored post - there are of course many other solutions available, such as:

• Digital Ocean
• Amazon Web Services
• Azure
• Google Cloud Platform
• Netlify
• ZEIT Now

Check them all out and see which suits your needs best.

Personally, I found Heroku the quickest and easiest to start using "out of the box". The free tier is somewhat limited in terms of resources. However, I can confidently recommend it for testing purposes.

This example will host a simple site using an Express server. Here are the high-level steps:

1. Setting up with Heroku, Git, npm
2. Create an Express.js server
3. Create static files
4. Deploy to Heroku

It should take about 25 minutes in total (or longer if you want to spend more time on the static files).

This article assumes you already know:

• Some HTML, CSS and JavaScript basics
• Basic command line usage
• Beginner-level Git for version control

You can find all the code in this repository.

Setting up

The first step in any project is to set up all the tools you know you'll need.

You'll need to have:

• Node and npm installed on your local machine (read how to do this here)
• Git installed (read this guide)
• The Heroku CLI installed (here's how to do it)

1. Create a new directory and initialise a Git repository

From the command line, create a new project directory and move into it.

$ mkdir lorem-ipsum-demo
$ cd lorem-ipsum-demo

Now you are in the project folder, initialise a new Git repository.

⚠️This step is important because Heroku relies on Git for deploying code from your local machine to its cloud servers ⚠️

$ git init

As a final step, you can create a README.md file to edit at a later stage.

$ echo "Edit me later" > README.md

2. Login to the Heroku CLI and create a new project

You can login to Heroku using the Heroku CLI (command line interface). You will need to have a free Heroku account to do this.

There are two options here. The default is for Heroku to let you login through the web browser. Adding the -i flag lets you login through the command line.

$ heroku login -i

Now, you can create a new Heroku project. I called mine lorem-ipsum-demo.

$ heroku create lorem-ipsum-demo

Naming your project:

• Heroku will generate a random name for your project if you don't specify one in the command.
• The name will form part of the URL you can use to access your project, so choose one you like.
• This also means that you need to choose a unique project name that no one else has used.
• It is possible to rename your project later (so don't worry too much about getting the perfect name right now).

3. Initialise a new npm project and install Express.js

Next, you can initialise a new npm project by creating a package.json file. Use the command below to do this.

⚠️This step is crucial. Heroku relies on you providing a package.json file to know this is a Node.js project when it builds your app ⚠️

$ npm init -y

Next, install Express. Express is a widely used server framework for NodeJS.

$ npm install express --save

Finally, you are ready to start coding!

Writing a simple Express server

The next step is to create a file called app.js, which runs an Express server locally.

$ touch app.js

This file will be the entry point for the app when it is ready. That means, the one command needed to launch the app will be:

$ node app.js

But first, you need to write some code in the file.

4. Edit the contents of app.js

Open app.js in your favourite editor. Write the code shown below and click save.

// create an express app
const express = require("express")
const app = express()

// use the express-static middleware
app.use(express.static("public"))

// define the first route
app.get("/", function (req, res) {
  res.send("<h1>Hello World!</h1>")
})

// start the server listening for requests
app.listen(process.env.PORT || 3000, 
    () => console.log("Server is running..."));

The comments should help indicate what is happening. But let's quickly break the code down to understand it further:

• The first two lines simply require the Express module and create an instance of an Express app.
• The next line requires the use of the express.static middleware. This lets you serve static files (such as HTML, CSS and JavaScript) from the directory you specify. In this case, the files will be served from a folder called public.
• The next line uses app.get() to define a URL route. Any URL requests to the root URL will be responded to with a simple HTML message.
• The final part starts the server. It either looks to see which port Heroku will use, or defaults to 3000 if you are running locally.

⚠️The use of process.env.PORT || 3000 in the last line is important for deploying your app successfully ⚠️

If you save app.js and start the server with:

$ node app.js

You can visit localhost:3000 in your browser and see for yourself the server is running.

Create your static files

The next step is to create your static files. These are the HTML, CSS and JavaScript files you will serve up whenever a user visits your project.

Remember in app.js you told the express.static middleware to serve static files from the public directory.

The first step is of course to create such a directory and the files it will contain.

$ mkdir public
$ cd public
$ touch index.html styles.css script.js

5. Edit the HTML file

Open index.html in your preferred text editor. This will be the basic structure of the page you will serve to your visitors.

The example below creates a simple landing page for a Lorem Ipsum generator, but you can be as creative as you like here.

<!DOCTYPE html>
<head>
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script>
    <link href="https://fonts.googleapis.com/css?family=Alegreya|Source+Sans+Pro&display=swap" rel="stylesheet">
    <link rel="stylesheet" type="text/css" href="styles.css">
</head>
<html>
<body>
<h1>Lorem Ipsum generator</h1>
  <p>How many paragraphs do you want to generate?</p>
  <input type="number" id="quantity" min="1" max="20" value="1">
  <button id="generate">Generate</button>
  <button id="copy">Copy!</button>
<div id="lorem">
</div>
<script type="text/javascript" src="script.js"></script>
</body>
</html>

6. Edit the CSS file

Next up is editing the CSS file styles.css. Make sure this is linked in your HTML file.

The CSS below is for the Lorem Ipsum example. But again, feel free to be as creative as you want.

h1 {
    font-family: 'Alegreya' ;
}

body {
    font-family: 'Source Sans Pro' ;
    width: 50%;
    margin-left: 25%;
    text-align: justify;
    line-height: 1.7;
    font-size: 18px;
}

input {
    font-size: 18px;
    text-align: center;
}

button {
    font-size: 18px;
    color: #fff;
}

#generate {
    background-color: #09f;
}

#copy {
    background-color: #0c6;
}

7. Edit the JavaScript file

Finally, you might want to edit the JavaScript file script.js. This will let you make your page more interactive.

The code below defines two basic functions for the Lorem Ipsum generator. Yes, I used JQuery - it's quick and easy to work with.

$("#generate").click(function(){
    var lorem = $("#lorem");
    lorem.html("");
    var quantity = $("#quantity")[0].valueAsNumber;
    var data = ["Lorem ipsum", "quia dolor sit", "amet", "consectetur"];
    for(var i = 0; i < quantity; i++){
        lorem.append("<p>"+data[i]+"</p>");
    }
})

$("#copy").click(function() {
    var range = document.createRange();
    range.selectNode($("#lorem")[0]);
    window.getSelection().removeAllRanges();
    window.getSelection().addRange(range);
    document.execCommand("copy");
    window.getSelection().removeAllRanges();
    }
)

Note that here, the data list is truncated to make it easier to show. In the actual app, it is a much longer list of full paragraphs. You can see the entire file in the repo, or look here for the original source.

Deploying your app

After writing your static code and checking it all works as expected, you can get ready to deploy to Heroku.

However, there are a couple more things to do.

8. Create a Procfile

Heroku will need a Procfile to know how to run your app.

A Procfile is a "process file" which tells Heroku which command to run in order to manage a given process. In this case, the command will tell Heroku how to start your server listening on the web.

Use the command below to create the file.

⚠️This is an important step, because without a Procfile, Heroku cannot put your server online. ⚠️

$ echo "web: node app.js" > Procfile

Notice that the Procfile has no file extension (e.g., ".txt", ".json").

Also, see how the command node app.js is the same one used locally to run your server.

9. Add and commit files to Git

Remember you initiated a Git repository when setting up. Perhaps you have been adding and committing files as you have gone.

Before you deploy to Heroku, make sure to add all the relevant files and commit them.

$ git add .
$ git commit -m "ready to deploy"

The final step is to push to your Heroku master branch.

$ git push heroku master

You should see the command line print out a load of information as Heroku builds and deploys your app.

The line to look for is: Verifying deploy... done.

This shows that your build was successful.

Now you can open the browser and visit your-project-name.herokuapp.com. Your app will be hosted on the web for all to visit!

Quick recap

Below are the steps to follow to deploy a simple Express app to Heroku:

1. Create a new directory and initialise a Git repository
2. Login to the Heroku CLI and create a new project
3. Initialise a new npm project and install Express.js
4. Edit the contents of app.js
5. Edit the static HTML, CSS and JavaScript files
6. Create a Procfile
7. Add and commit to Git, then push to your Heroku master branch

Things to check if your app is not working

Sometimes, despite best intentions, tutorials on the Internet don't work exactly as you expected.

The steps below should help debug some common errors you might encounter:

• Did you initialise a Git repo in your project folder? Check if you ran git init earlier. Heroku relies on Git to deploy code from your local machine.
• Did you create a package.json file? Check if you ran npm init -y earlier. Heroku requires a package.json file to recognise this is a Node.js project.
• Is the server running? Make sure your Procfile uses the correct file name to start the server. Check you have web: node app.js and not web: node index.js.
• Does Heroku know which port to listen on? Check you used app.listen(process.env.PORT || 3000) in your app.js file.
• Do your static files have any errors in them? Check them by running locally and seeing if there are any bugs.

Thanks for reading - if you made it this far, you might want to checkout the finished version of the demo project.

Developers, teams, and businesses of all sizes use Heroku to deploy, manage, and scale apps. Whether you're building a simple prototype or a business-critical product, Heroku's fully-managed platform gives you the simplest path to delivering apps quickly.

With features like Heroku Runtime, Heroku Postgres (SQL), Heroku Redis, Add-ons, Data Clips, App metrics, Smart containers, Enterprise-grade support, GitHub Integration and lots more, Heroku gives developers the freedom to focus on their core product without the distraction of maintaining servers, hardware, or infrastructure.

One of Heroku's core feature is deploying, managing, and scaling apps with your favorite languages [Node, Ruby, Python, Java, PHP, Go, and more].
In this article, I'll show you how to take an existing Node.js app and deploy it to Heroku – everything from creating your Heroku account to adding a database to your deployed application.

Prerequisites

In my previous article, I wrote about "Building a SlackBot with Node.js and SlackBots.js" and I promised to write a follow-up article to show how to host the SlackBot on either Heroku, Zeit or Netlify and publish it to the Slack Apps store. Well, this is the follow-up article but without the "Publishing to Slack Apps" part. We'll cover that in another article.

I assume you have/ know the following already:

• Read my previous article

• Built the inspireNuggets SlackBot

• Git, Node, and npm installed

• A free Heroku account

• Heroku CLI installed

Bonus

If you don't have npm, Node, and Heroku CLI installed or a Heroku account already, here's a quick bonus [ Yes, you're welcome :) ].

Installing npm and Node

• Node.js is a JavaScript runtime built on Chrome's V8 JavaScript engine.

• npm is the package manager for Node.js. An open-source project created to help JavaScript developers easily share packaged modules of code.

You can simply download Node.js here. Don't worry, npm comes with Node.js, so doing this installs both ✨

Creating a free Heroku account

Kindly head here and fill the Signup form. It's pretty simple.

Installing Heroku CLI

The Heroku Command Line Interface (CLI) makes it easy to create and manage your Heroku apps directly from the terminal. It’s an essential part of using Heroku. [ Well, you can decide to use the GitHub integration feature and Heroku Dashboard but yes you should learn how to use the CLI ]
Heroku CLI requires Git, the popular version control system. If you don’t already have Git installed, I wrote this article to help you.

Heroku CLI for Mac OS

brew tap heroku/brew && brew install heroku

or download the installer.

Heroku CLI for Ubuntu

sudo snap install --classic heroku

Heroku CLI for Windows

Download the installer for 64-Bit or 32-Bit.

Other installation methods

Please read this.

Getting started with Heroku CLI

• Verify your installation

heroku --version

heroku/7.30.1 linux-x64 node-v11.14.0

• Login to your Heroku account

There are two ways to do this:

• Web based auth

heroku login

Follow the instructions and login via your web browser then return to your terminal.

• CLI auth

This is a safer option as it saves your email address and an API token to ~/.netrc for future use.

heroku login -i

Deploying your Node.js App

I presume you've built the SlackBot already. If you haven't, please clone the finished project.

The project is a simple Slackbot that displays random inspiring techie quotes and jokes for developers/designers.

git clone https://github.com/BolajiAyodeji/inspireNuggetsSlackBot.git && cd inspireNuggetsSlackBot

Now let's deploy our app to Heroku ??. I'll show you two ways to do this:

Deploy via the Heroku Git

This is done via the Heroku CLI.

☑️ Checklist

• Specify the version of Node.js that will be used to run your application on Heroku in your package.json file.

"engines": {
    "node": "10.16.0"
  },

• Specify your start script.
  Simply create a Procfile (without any file extension) and add

web: node index.js

Heroku first looks for this Procfile. If none is found, Heroku will attempt to start a default web process via the start script in your package.json.

• Start your app locally using the heroku local command to be sure everything works fine

heroku local web

Your app should now be running on http://localhost:5000.

• Don't forget to .gitignore

/node_modules
.DS_Store
/*.env

? Let's Deploy

How this works is, you have the project working on local already and you've pushed to GitHub already.

• Run heroku create

Basically, this command creates a new Heroku app for you with some randomly generated domain and adds Heroku to your local Git repository.

• Now run git push heroku master

This is the magic command, it pushes your app to Heroku, installs it there, and launches it on your allocated domain.

In the example above, it's https://lit-cove-58897.herokuapp.com/

You can always make changes to your app settings and domains in your Heroku Dashboard

• Now visit your app in your browser

heroku open

• You can also view information about your running app using one of the logging commands. This is very useful in debugging errors.

heroku logs --tail

Deploy via GitHub integration

You can configure GitHub integration in the Deploy tab of apps in the Heroku Dashboard.

☑️ Checklist

• All previous checklists apply here – ensure you have the app deployed to GitHub already

? Let's Deploy

How this method works is that you push your entire project to GitHub and integrate it to Heroku. Every time you push, it deploys from GitHub to Heroku. Pretty cool right?

• Login to your Heroku Dashboard and create a new app

• Select your app name and region

Now your app has successfully been created

• Click the deploy tab and scroll to the Deployment method section

• Click the Connect to GitHub button

• Now you have the Connect to GitHub section, search for the repository and deploy.

• Now your app was deployed successfully

Automatic deploys

Now your app is deployed but you'll have to keep deploying manually. You need to enable automatic deploys for a GitHub branch, so Heroku builds and deploys all pushes to that branch.

• Scroll to the Automatic Deploys section

Select the branch you want to deploy. Ideally, this should be the master branch but change this according to your preference.

Now every push to master (or the branch you chose) will deploy a new version of this app.

Node.js Buildpack

In Heroku, Buildpacks are scripts that are run when your app is deployed. They are used to install dependencies for your app and configure your environment.

After deploying your app, ensure you add a Node.js buildpack to your project.

• Go to Settings and scroll to the Buildpack section

• Click the Add Buildpack button and select Node.js in the Popup modal.

• Now the new buildpack configuration will be used when this app is next deployed. Make some changes to your app and push to GitHub – it will automatically deploy.

Adding a Database to your deployed App'

The Heroku add-on marketplace has a large number of data stores, from Redis and MongoDB providers, to Postgres and MySQL.

Heroku provides three managed data services to all customers in the form of Add-ons:

• Heroku Postgres

• Heroku Redis

• Apache Kafka on Heroku

Writing about this three will make this article too long. It's pretty simple and I'll add some links to the Heroku Docs.

• Heroku Postgresql Docs

• Heroku Redis Docs

• Apache Kafka on Heroku Docs

Conclusion

Every Heroku account is allocated a pool of free dyno hours. Heroku (free) dynos are great for hosting apps and personal projects. The downside, however, is that your app will fall asleep if it doesn't receive any web traffic within 30-minutes :(.

You can use external tools to ping your server periodically so it never falls asleep.

Here are some to consider:

• Pingmydyno

• Heroku self ping

• Wakemydyno

• Kaffeine

Heroku is meticulously designed to help developers be as productive as possible. The platform removes frustrating obstacles and mundane tasks, so you can stay free of distraction in your development flow. Wherever you are on the learning path, Heroku helps you love app development even more. - Heroku

The Heroku experience provides services, tools, workflows, and polyglot support—all designed to enhance developer productivity. There is more to using Heroku and I hope you explore more and build amazing stuff with Heroku.

If you're a student, Kindly register for the GitHub Student Developer Pack to get One free Hobby Dyno for up to two years.

The pack give students free access to the best developer tools in one place so you can learn by doing.

In this tutorial we will be doing a basic React + Node app deploy to Heroku.

There are a lot of tutorials that do this only using the command line, so to change things up a bit, I will do it completely without the command line.

For things like generating React and Express apps, we have no choice but to use the command line. For everything else we'll use a GUI.

I also assume you have a Github and Heroku account. They are both free, so no worries about signing up.

sample project:
https://github.com/iqbal125/react-express-sample

React and Express Setup

First, let's start by creating two directories named Server and Client.

The Client directory will hold the contents of the create-react-app command, and Server will hold the contents of the express command. This library just creates a simple Express app for us automatically, similar to create-react-app. It needs to be installed globally, which you can do so with the command:

npm install -g express-generator

After this, simply run these commands in each of the respective directories to install the starter projects:

npx create-react-app app1 in the Client directory

express in the Server directory

Change to the app1 directory generated by create-react-app and run:

npm run build

This will generate a production build version of the project that is optimized for a production deployment, with things like the error handling code and white space removed.

Note: In a development build you would use a proxy to http://localhost:5000 to communicate from your React app to your Express server, but here the React app and the Express server are just one project. The Express server serves the React files.

Next, cut and paste the entire build directory into the Server directory. Your project structure should look like this:

We can now add some code to let our Express server know to serve our React project.:

....

app.use(express.static(path.join(__dirname, 'build')));


app.get('/*', (req, res) => {
  res.sendFile(path.join(__dirname, 'build', 'index.html'));
});

....

The first line of code serves all our static files from the build directory.

The second piece of code is to keep our client side routing functional. This code essentially serves the index.html file on any unknown routes. Otherwise we would need to rewrite our entire routing to work with this Express server setup.

To test your app, just run npm start in the Server directory and go to http://localhost 3000 in the browser. Then you should be see your running React app.

Now we are ready to upload this project to GitHub.

GitHub

Instead of using the command line to upload to GitHub, we will do this with the GUI. First, go to the GitHub homepage and create a new repository. Name it whatever you want, but make sure the Initialize this Repository with a README option checked:

Next upload all the project files without the node_modules directory.

Click commit and we are done. Your uploaded project files will appear on GitHub like so:

Now we can move on to setting up Heroku.

Heroku

Go to the Heroku dashboard, create a new app, and name it whatever you like.

Next, go to the Deploy tab and select GitHub under Deployment method:

If you haven't connected your GitHub account to your Heroku account yet, you will be prompted through the GitHub Auth flow.

After this, search for your project on GitHub and connect to it:

Finally, we can just deploy our app by clicking the Deploy Branch button:

Heroku will install all the Node modules for you automatically. You can view your project by clicking on the View button.

And that's it, we're done! Thanks for reading.

Connect with me on Twitter for more updates on future tutorials: https://twitter.com/iqbal125sf

Hi everyone :) Today I am beginning a new series of posts specifically aimed at Python beginners. The concept is rather simple: I'll do a fun project, in as few lines of code as possible, and will try out as many new tools as possible.

For example, today we will learn to use the Twilio API, the Twitch API, and we'll see how to deploy the project on Heroku. I'll show you how you can have your own "Twitch Live" SMS notifier, in 30 lines of codes, and for 12 cents a month.

Prerequisite: You only need to know how to run Python on your machine and some basic commands in git (commit & push). If you need help with these, I can recommend these 2 articles to you:

Python 3 Installation & Setup Guide

The Ultimate Git Command Tutorial for Beginners from Adrian Hajdin.

What you'll learn:

• Twitch API
• Twilio API
• Deploying on Heroku
• Setting up a scheduler on Heroku

What you will build:

The specifications are simple: we want to receive an SMS as soon as a specific Twitcher is live streaming. We want to know when this person is going live and when they leave streaming. We want this whole thing to run by itself, all day long.

We will split the project into 3 parts. First, we will see how to programmatically know if a particular Twitcher is online. Then we will see how to receive an SMS when this happens. We will finish by seeing how to make this piece of code run every X minutes, so we never miss another moment of our favorite streamer's life.

Is this Twitcher live?

To know if a Twitcher is live, we can do two things: we can go to the Twitcher URL and try to see if the badge "Live" is there.

Screenshot of a Twitcher live streaming.

This process involves scraping and is not easily doable in Python in less than 20 or so lines of code. Twitch runs a lot of JS code and a simple request.get() won't be enough.

For scraping to work, in this case, we would need to scrape this page inside Chrome to get the same content like what you see in the screenshot. This is doable, but it will take much more than 30 lines of code. If you'd like to learn more, don't hesitate to check my recent web scraping without getting blocked guide. (I recently launch ScrapingBee, a web-scraping tool hence my knowledge in the field ;))

So instead of trying to scrape Twitch, we will use their API. For those unfamiliar with the term, an API is a programmatic interface that allows websites to expose their features and data to anyone, mainly developers. In Twitch's case, their API is exposed through HTTP, witch means that we can have lots of information and do lots of things by just making a simple HTTP request.

Get your API key

To do this, you have to first create a Twitch API key. Many services enforce authentication for their APIs to ensure that no one abuses them or to restrict access to certain features by certain people.

Please follow these steps to get your API key:

• Create a Twitch account
• Now create a Twitch dev account -> "Signing up with Twitch" top right
• Go to your "dashboard" once logged in
• "Register your application"
• Name -> Whatever, Oauth redirection URL -> http://localhost, Category -> Whatever

You should now see, at the bottom of your screen, your client-id. Keep this for later.

Is that Twitcher streaming now?

With your API key in hand, we can now query the Twitch API to have the information we want, so let's begin to code. The following snippet just consumes the Twitch API with the correct parameters and prints the response.

# requests is the go to package in python to make http request
# https://2.python-requests.org/en/master/
import requests

# This is one of the route where Twich expose data, 
# They have many more: https://dev.twitch.tv/docs
endpoint = "https://api.twitch.tv/helix/streams?"

# In order to authenticate we need to pass our api key through header
headers = {"Client-ID": "<YOUR-CLIENT-ID>"}

# The previously set endpoint needs some parameter, here, the Twitcher we want to follow
# Disclaimer, I don't even know who this is, but he was the first one on Twich to have a live stream so I could have nice examples
params = {"user_login": "Solary"}

# It is now time to make the actual request
response = request.get(endpoint, params=params, headers=headers)
print(response.json())

The output should look like this:

{
   'data':[
      {
         'id':'35289543872',
         'user_id':'174955366',
         'user_name':'Solary',
         'game_id':'21779',
         'type':'live',
         'title':"Wakz duoQ w/ Tioo - GM 400LP - On récupère le chall après les -250LP d'inactivité !",
         'viewer_count':4073,
         'started_at':'2019-08-14T07:01:59Z',
         'language':'fr',
         'thumbnail_url':'https://static-cdn.jtvnw.net/previews-ttv/live_user_solary-{width}x{height}.jpg',
         'tag_ids':[
            '6f655045-9989-4ef7-8f85-1edcec42d648'
         ]
      }
   ],
   'pagination':{
      'cursor':'eyJiIjpudWxsLCJhIjp7Ik9mZnNldCI6MX19'
   }
}

This data format is called JSON and is easily readable. The data object is an array that contains all the currently active streams. The key type ensures that the stream is currently live. This key will be empty otherwise (in case of an error, for example).

So if we want to create a boolean variable in Python that stores whether the current user is streaming, all we have to append to our code is:

json_response = response.json()

# We get only streams
streams = json_response.get('data', [])

# We create a small function, (a lambda), that tests if a stream is live or not
is_active = lambda stream: stream.get('type') == 'live'
# We filter our array of streams with this function so we only keep streams that are active
streams_active = filter(is_active, streams)

# any returns True if streams_active has at least one element, else False
at_least_one_stream_active = any(streams_active)

print(at_least_one_stream_active)

At this point, at_least_one_stream_active is True when your favourite Twitcher is live.

Let's now see how to get notified by SMS.

Send me a text, NOW!

So to send a text to ourselves, we will use the Twilio API. Just go over there and create an account. When asked to confirm your phone number, please use the phone number you want to use in this project. This way you'll be able to use the $15 of free credit Twilio offers to new users. At around 1 cent a text, it should be enough for your bot to run for one year.

If you go on the console, you'll see your Account SID and your Auth Token , save them for later. Also click on the big red button "Get My Trial Number", follow the step, and save this one for later too.

Sending a text with the Twilio Python API is very easy, as they provide a package that does the annoying stuff for you. Install the package with pip install Twilio and just do:

from twilio.rest import Client
client = Client(<Your Account SID>, <Your Auth Token>)
client.messages.create(
    body='Test MSG',from_=<Your Trial Number>,to=<Your Real Number>)

And that is all you need to send yourself a text, amazing right?

Putting everything together

We will now put everything together, and shorten the code a bit so we manage to say under 30 lines of Python code.

import requests
from twilio.rest import Client
endpoint = "https://api.twitch.tv/helix/streams?"
headers = {"Client-ID": "<YOUR-CLIENT-ID>"}
params = {"user_login": "Solary"}
response = request.get(endpoint, params=params, headers=headers)
json_response = response.json()
streams = json_response.get('data', [])
is_active = lambda stream:stream.get('type') == 'live'
streams_active = filter(is_active, streams)
at_least_one_stream_active = any(streams_active)
if at_least_one_stream_active:
    client = Client(<Your Account SID>, <Your Auth Token>)
    client.messages.create(body='LIVE !!!',from_=<Your Trial Number>,to=<Your Real Number>)

Avoiding double notifications

This snippet works great, but should that snippet run every minute on a server, as soon as our favorite Twitcher goes live we will receive an SMS every minute.

We need a way to store the fact that we were already notified that our Twitcher is live and that we don't need to be notified anymore.

The good thing with the Twilio API is that it offers a way to retrieve our message history, so we just have to retrieve the last SMS we sent to see if we already sent a text notifying us that the twitcher is live.

Here what we are going do to in pseudocode:

if favorite_twitcher_live and last_sent_sms is not live_notification:
    send_live_notification()
if not favorite_twitcher_live and last_sent_sms is live_notification:
    send_live_is_over_notification()

This way we will receive a text as soon as the stream starts, as well as when it is over. This way we won't get spammed - perfect right? Let's code it:

# reusing our Twilio client
last_messages_sent = client.messages.list(limit=1)
last_message_id = last_messages_sent[0].sid
last_message_data = client.messages(last_message_id).fetch()
last_message_content = last_message_data.body

Let's now put everything together again:

import requests
from twilio.rest import Client
client = Client(<Your Account SID>, <Your Auth Token>)

endpoint = "https://api.twitch.tv/helix/streams?"
headers = {"Client-ID": "<YOUR-CLIENT-ID>"}
params = {"user_login": "Solary"}
response = request.get(endpoint, params=params, headers=headers)
json_response = response.json()
streams = json_response.get('data', [])
is_active = lambda stream:stream.get('type') == 'live'
streams_active = filter(is_active, streams)
at_least_one_stream_active = any(streams_active)

last_messages_sent = client.messages.list(limit=1)
if last_messages_sent:
    last_message_id = last_messages_sent[0].sid
    last_message_data = client.messages(last_message_id).fetch()
    last_message_content = last_message_data.body
    online_notified = "LIVE" in last_message_content
    offline_notified = not online_notified
else:
    online_notified, offline_notified = False, False

if at_least_one_stream_active and not online_notified:
    client.messages.create(body='LIVE !!!',from_=<Your Trial Number>,to=<Your Real Number>)
if not at_least_one_stream_active and not offline_notified:
    client.messages.create(body='OFFLINE !!!',from_=<Your Trial Number>,to=<Your Real Number>)

And voilà!

You now have a snippet of code, in less than 30 lines of Python, that will send you a text a soon as your favourite Twitcher goes Online / Offline and without spamming you.

We just now need a way to host and run this snippet every X minutes.

The quest for a host

To host and run this snippet we will use Heroku. Heroku is honestly one of the easiest ways to host an app on the web. The downside is that it is really expensive compared to other solutions out there. Fortunately for us, they have a generous free plan that will allow us to do what we want for almost nothing.

If you don't already, you need to create a Heroku account. You also need to download and install the Heroku client.

You now have to move your Python script to its own folder, don't forget to add a requirements.txt file in it. The content of the latter begins:

requests
twilio

cd into this folder and just do a heroku create --app <app name>.

If you go on your app dashboard you'll see your new app.

We now need to initialize a git repo and push the code on Heroku:

git init
heroku git:remote -a <app name>
git add .
git commit -am 'Deploy breakthrough script'
git push heroku master

Your app is now on Heroku, but it is not doing anything. Since this little script can't accept HTTP requests, going to <app name>.herokuapp.com won't do anything. But that should not be a problem.

To have this script running 24/7 we need to use a simple Heroku add-on call "Heroku Scheduler". To install this add-on, click on the "Configure Add-ons" button on your app dashboard.

Then, on the search bar, look for Heroku Scheduler:

Click on the result, and click on "Provision"

If you go back to your App dashboard, you'll see the add-on:

Click on the "Heroku Scheduler" link to configure a job. Then click on "Create Job". Here select "10 minutes", and for run command select python <name_of_your_script>.py. Click on "Save job".

While everything we used so far on Heroku is free, the Heroku Scheduler will run the job on the $25/month instance, but prorated to the second. Since this script approximately takes 3 seconds to run, for this script to run every 10 minutes you should just have to spend 12 cents a month.

Ideas for improvements

I hope you liked this project and that you had fun putting it into place. In less than 30 lines of code, we did a lot, but this whole thing is far from perfect. Here are a few ideas to improve it:

• Send yourself more information about the current streaming (game played, number of viewers ...)
• Send yourself the duration of the last stream once the twitcher goes offline
• Don't send you a text, but rather an email
• Monitor multiple twitchers at the same time

Do not hesitate to tell me in the comments if you have more ideas.

Conclusion

I hope that you liked this post and that you learned things reading it. I truly believe that this kind of project is one of the best ways to learn new tools and concepts, I recently launched a web scraping API where I learned a lot while making it.

Please tell me in the comments if you liked this format and if you want to do more.

I have many other ideas, and I hope you will like them. Do not hesitate to share what other things you build with this snippet, possibilities are endless.

Happy Coding.

Pierre

Don't want to miss my next post:

You can subscribe here to my newsletter.

Don’t you just love automated emails? I know I do. I mean, who doesn’t enjoy waking up to 236 new messages from Nike, Ticketmaster, and Adobe Creative Cloud every morning? What a fantastic way to start my day! ??

Anyway, today I’ll be showing you how to drown your inbox in more clutter, for God-knows-what reason. We’re going to be using Python to create a custom Reddit email-notification system. That means we’ll be writing a script that looks for Reddit posts matching some keywords and then emails us when such posts appear.

Want quality email content like this? Read on!

There are a few reasons that you might be doing this. Maybe you’re really excited about some topic on Reddit. Maybe you’re trying to discover a new karma-farming technique because Internet points are important to you. Maybe you want to send annoying emails to your friends. Or maybe you just want more emails in your inbox to deal with your crippling loneliness. Oops, sorry — went too far. Let’s get started.

Looking through Reddit

Reddit has a nice API that you can do a lot with. To make things even easier, we will be using PRAW, the Python Reddit API Wrapper.

You’ll need a Reddit account first. Once you have one, go here to create an app. Name it anything, and make sure “script” is selected. As per the docs, you can just put [http://localhost:8080](http://localhost:8080) for your redirect URI.

Now, you’re ready to start that nifty script! In the code below, I look through a subreddit, picking out posts that match my needs.

I consider a post a match if it is relevant enough and if it is popular enough. More specifically, the post is relevant enough when it has a keyword_count that’s not -1 (I’ll explain this below) and popular enough when it has a weighted_score greater than a predefined MIN_RELEVANT_WEIGHTED_SCORE. The weighted score simply factors in the score of the post and the number of comments on the post. Anyway, this is what best fit my needs, so feel free to better define what a match means to you.

Now, I promised you I would talk about the keyword_count party going on. Spoiler: it’s not really a party. I just devised this simple way of assessing relevancy: there are required terms and secondary terms. A post is relevant if and only if all the required terms are in the title, and at least X number of secondary terms are in the title (where X is some predefined number). Again, this part can be re-imagined in infinitely different ways, but this is just what I did.

Now we have everything to comb through our subreddit and tease out the good stuff about conspiracies or whatever. Cool. So, like my homie Ariana says, “thank u, next.”

Emailing notifications

Time to start spamming. In the code below, I’m using smtplib (the Simple Mail Transfer Protocol client) to help me send my emails. I then craft the beautiful email with HTML, using the info from Reddit that we got above to populate it. And the best (or worst?) part is, if you want to notify everyone you know about the latest and greatest Reddit posts, you can simply add more email addresses to the email_list.

Important side note: make sure the email you use to send the emails have less secure app access enabled if it’s a Gmail address, or this will not work.

Make it run forever

If you don’t have time to continually browse Reddit, you don’t have time to continually run this script. I used Heroku Scheduler to run this script every 10 minutes, as suggested by this Stack Overflow answer. It’s pretty easy to follow: add in a few additional files and a dummy web server, push to Heroku, add the Heroku Scheduler add-on, and BAM! You’re set until you run out of free dyno-hours. ??

Is this the best solution? No. But is it sufficient for my purposes? Yep. If you know of a similarly trivial way to do this, please let me know!

In conclusion

That’s pretty much all to this project. This GitHub repo contains all my code. Because of all the work that literally everyone else has already done, it’s quite a simple task to build this custom Reddit notification system. Gotta love the ✨magic✨ of software development.

Me after setting up my custom Reddit notifications

If you made it all the way down to here, please comment “North Dakota is the top producer of barley in the USA” in the box below.

Thanks for reading!