Table of Contents

• What is "React2Shell"?

• Why is this Happening Now?

• Should You Worry About this Change?

• Is this Mandatory?

• How Bad Can It Get? The Extent of Exploitation

• What Would be the Code Change for This?

• Advanced: Verify with the Original Exploit (PoC)

• Emergency Response: What If You Were Already Compromised?

• Conclusion

What is "React2Shell"?

Think of your server receiving data like a mailroom receiving packages.

Usually, a mailroom checks if a package is safe before opening it. But in vulnerable versions of React and NextJS, the "Flight" protocol (used to communicate between the server and client) acts like a mailroom that blindly opens every package and follows any instructions inside immediately.

This vulnerability (CVE-2025-55182) allows an attacker to send a specifically crafted "package" (HTTP request) that forces your server to execute malicious code – like stealing passwords or installing viruses –without even logging in.

Why is this Happening Now?

It's all about how modern frameworks handle data serialization. There are a few reasons why this was just discovered.

First, React has complex serialization. To make Server Components seamless, React sends complex data structures back and forth.

Second, it has the "Flight" protocol. The vulnerability was found in how this specific protocol de-serializes (unpacks) data. It was too trusting of the input it received from the client side.

Should You Worry About this Change?

You need to pay attention if your app qualifies for any of the below:

• You are using NextJS App Router: This is the default in newer NextJS versions (v13+).

• You are using React 19: Specifically versions with Server Components enabled.

• You use Server Actions: If your app takes user input and processes it on the server using React's server actions.

Is this Mandatory?

Yes. This is a critical security update. If your app qualifies in any of the above scenarios, you need to act immediately. Because, this vulnerability is being exploited right now.

How Bad Can It Get? The Extent of Exploitation

You might be thinking, "My site is just a simple content wrapper, surely I'm not a target?" Unfortunately, with Remote Code Execution (RCE), the attacker doesn't just "break" your site – they own the server it runs on.

Here is exactly what a hacker can do once they exploit this vulnerability:

Total Environment Theft

The most immediate risk is your .env file. Attackers can execute code to read your environment variables, instantly gaining access to your AWS Secret Keys, Database passwords, Stripe API keys, and OpenAI tokens.

The "Shell" Access

As the name "React2Shell" implies, attackers can open a reverse shell. This gives them a command-line interface on your server, allowing them to browse your file system as if they were sitting in front of your computer.

Lateral Movement

Once inside your NodeJS server, they are behind your firewall. They can now attack your internal services (like Redis, internal databases, or private micro-services) that are usually blocked from the outside world.

Supply Chain Poisoning

If your build server is vulnerable, an attacker could potentially inject malicious code into your deployment pipeline, affecting every user who visits your site in the future.

Botnet Recruitment

Hackers often automate these attacks to install crypto-miners, using your server's CPU (which you pay for!) to mine digital currency for them, often crashing your application in the process.

What Would be the Code Change for This?

You don’t need to rewrite your application code, but you must update your dependencies in your release line.

The vulnerability is fully resolved in the following patched NextJS releases:

• 15.0.5

• 15.1.9

• 15.2.6

• 15.3.6

• 15.4.8

• 15.5.7

• 16.0.7

Patched canary releases for NextJS 15 and 16:

• 15.6.0-canary.58 (for 15.x canary releases)

• 16.1.0-canary.12 (for 16.x canary releases)

These versions include the hardened React Server Components implementation.

Here are the patched versions for React JS:

• 19.0.1

• 19.1.2

• 19.2.1

Frameworks and bundlers using the aforementioned packages should install the latest versions provided by their respective maintainers.

Alternatively, you can run npx fix-react2shell-next in your NextJS project to launch an interactive tool which can check versions and perform deterministic version bumps per the recommended versions above. See the GitHub repository for full details.

There is no workaround other than upgrading to a patched version.

It’s highly recommended to rotate all your application secrets, once you have patched your version and re-deployed your application.

Advanced: Verify with the Original Exploit (PoC)

If you want to be 100% sure your patch is working, or if you want to understand how the attack actually works, you can use the original Proof of Concept (PoC) created by the security researcher (Lachlan Davidson) who found the bug.

Repository: React2Shell-CVE-2025-55182-original-poc

Lachlan provided three variations of the exploit script. The most important one for testing is 01-submitted-poc.js, which is the exact, simplified version submitted to Meta for the bug bounty.

How the Exploit Works

According to the repository, the attack works by tricking the parser:

1. The attacker sends a payload using $@x to access a specific data Chunk.

2. They "plant" a .then function on a fake object.

3. The JavaScript runtime thinks it is handling a Promise and tries to "unravel" it.

4. This allows the attacker to re-enter the parser with a malicious fake chunk, giving them access to internal server gadgets (like _response) to execute code (RCE).

Steps to Recreate the Issue

⚠️ WARNING: Only run this against a local development server (localhost) that you own. Never run this against production servers or public websites.

Note: I forked Lachlan’s repo and made minor changes to make it easy for you to run the script.

Step 1: Clone the Repository

Run the following commands to clone the repository, navigate into the project, and install dependencies:

git clone https://github.com/arunachalam-b/React2Shell-CVE-2025-55182-original-poc.git
cd React2Shell-CVE-2025-55182-original-poc
npm i

Step 2: Run a Vulnerable Local Server

Start your NextJS application locally (ensure it’s running a vulnerable version, for example NextJS 15.0.0, for the test to succeed).

npm run dev
# usually runs on http://localhost:3000

Step 3: Execute the Test

You will need to modify the script or use a tool like curl to send the payload structure found in 01-submitted-poc.js to your server's endpoint (usually a Server Action endpoint). Or simply run the following command if your app is accessible at http://localhost:3000:

node 01-submitted-poc.js

If the exploit succeeds (on the vulnerable version), the console will log the execution of the code (RCE). If the exploit fails (after you patch), the server will either reject the request or error out safely.

You can also confirm if your infected web server prints 50 in the console. Because we inject the code to do a calculation (look at _prefix field in the below JSON) that results in 50.

After you apply the fix, you should see an error while running the script. In this case, as I’m using NextJS v15.1, the fix is upgrading the next package to version 15.1.9. Here are the screenshots after upgrading the package and running the script.

Step 4: Verification

Once you have confirmed the exploit works on the old version, update your packages (as shown in the section above) and run the script again. It should no longer trigger the code execution.

Emergency Response: What If You Were Already Compromised?

If you suspect your server was exposed to the internet with a vulnerable version, assume the worst. A hacker may have already stolen your keys or left a "backdoor" to return later. Patching the code alone is NOT enough in this case.

Follow this "Nuke and Pave" protocol immediately:

Step 1: Isolate and Shutdown

Take the compromised server offline immediately. Do not try to "fix" it while it is running.

Step 2: Rotate ALL Secrets (Crucial Step)

Assume every secret in your .env file is in the hands of a hacker. You must generate new ones:

• Change the password for your database users.

• Rotate AWS Access Keys, Google Cloud Service Account keys, and so on.

• Roll your Stripe/PayPal/Razorpay API keys.

• Rotate your NEXTAUTH_SECRET or any JWT signing keys.

Step 3: Do Not "Clean" — Rebuild

Do not attempt to find and delete malware files on the server. Hackers are good at hiding.

• Destroy the existing container, droplet, or EC2 instance entirely.

• Build a fresh instance from your source code (after applying the patch).

Step 4: Audit Your Logs

Look at your database and cloud provider logs. Did anyone download your entire user database? Did anyone spin up expensive GPU instances on your AWS account? Check for unusual activity that occurred before you patched.

Conclusion

In this article, you learned about the "React2Shell" vulnerability, how to verify it using the original developer's tools, and how to upgrade your app to secure your Server Components. I hope you have a clear idea about why this update is urgent. By being proactive now, you can avoid a catastrophic data breach.

You can follow my Twitter/X account to receive the top AI news everyday. If you wish to learn more about cybersecurity, subscribe to my email newsletter and follow me on social media.

Table of Contents

• What is a Page Size?

• Why is this Change Being Implemented Now?

• What are the Pros and Cons of this Change?

• Should You worry About this Change?

• Is this Mandatory?

• What if You Don’t Upgrade Your App?

• How Does this Affect Hybrid Apps?

• What Would be the Code Change for This?

• How to Verify if Your App is Upgraded to a 16 KB Page Size

• Conclusion

What is a Page Size?

Think of your device's memory like a book. An operating system doesn't read memory one tiny word at a time; it reads in chunks. These chunks are called "pages." For a long time, on most ARM64 Android devices, these pages were 4 KB in size. Now, for some newer Android devices (specifically those launching with Android 13 and later), that page size has quadrupled to 16 KB.

Why is this Change Being Implemented Now?

It's all about making Android run better on modern hardware. Here are some of the reasons why is it being implemented:

Better Performance: Modern processors can handle larger memory chunks more efficiently. A 16 KB page size means the CPU spends less time managing tiny bits of memory and more time doing actual work, which can lead to faster app performance.

Smoother Operations: With fewer, larger pages to keep track of, the system itself has a little less overhead, making things a bit more streamlined.

Keeping Up with Tech: This change helps Android align with how newer ARM64 processors are designed to work best.

What are the Pros and Cons of this Change?

Every big change has its own pros and cons.

Pros

• Apps that move a lot of data around or are memory-intensive might just feel a bit snappier

• The system could run a bit more efficiently, benefiting all apps indirectly

Cons

• If your native code is constantly asking for very small bits of memory (less than 16 KB), each of those might now take up a full 16 KB page, potentially using a little more memory than before.

• If your native code makes assumptions that "memory pages are always 4 KB," it could run into issues on 16 KB page devices.

Should You worry About this Change?

You need to pay attention if:

• Your app includes native libraries (like .so files) written in C/C++. This is where the impact is most direct. If your native code does anything with memory mapping (mmap, shmem) or file I/O where it calculates offsets or sizes based on a fixed page size.

• You're developing games or other highly performance-sensitive apps with native components.

• You're targeting Android 15+ with your app updates.

You need not worry if:

• Your app is built purely in Java or Kotlin with no native components. The Android Runtime (ART) handles memory for you, so these underlying page size changes are largely invisible. You'll still get the performance benefits!

• You're using React Native or Flutter, unless you've added custom native modules that directly deal with memory mapping or page-size-dependent operations

Is this Mandatory?

Yes. Google Play is making this a requirement for app updates. You would have received an email from Google Play if your app does not support 16 KB page size yet.

As the screenshot clearly shows, "From Nov 1, 2025, if your app updates do not support 16 KB memory page sizes, you won't be able to release these updates" for apps targeting Android 15+. This gives us a solid timeframe to get things ready.

What if You Don’t Upgrade Your App?

You could notice some serious issues if your app has native libraries that aren't ready for the 16 KB page size by the deadline. Here are a few:

• Crashes: This is the most serious. Your app might crash unexpectedly (often with a "segmentation fault") if it tries to access memory incorrectly due to old page size assumptions.

• Wasted Memory: If your code allocates memory in smaller chunks than 16 KB, it could end up using more memory than necessary, potentially slowing things down or hitting memory limits.

• Performance Hit: Instead of gaining speed, your app might actually run slower if its memory operations aren't aligned with the larger page size.

Essentially, your app might work fine today, but become unstable or inefficient on newer Android devices if its native components aren't updated.

How Does this Affect Hybrid Apps?

Generally, if you're building a standard hybrid app (React Native or Flutter app) without custom native modules, you're in a pretty good spot. The frameworks themselves, and the underlying runtimes (JavaScript engine for React Native, Dart VM for Flutter), usually handle memory management, abstracting away the page size.

However, if you've implemented custom native modules in C++ for performance-critical tasks or specific hardware interactions, then you do need to check those modules.

For the vast majority of standard React Native and Flutter apps, you likely won't need direct code changes related to page size, but always ensure you're using the latest SDK versions for your framework to benefit from any underlying platform updates.

What Would be the Code Change for This?

The biggest thing to avoid in your native code is making assumptions about memory page sizes. Instead of hardcoding 4096 (for 4 KB), always ask the operating system what its current page size is.

Steps to Take:

1. Audit Your Native Code: Search your .cpp, .c, and .h files for any direct use of 4096 or 4 KB in memory allocation, buffer sizing, or alignment calculations

2. Replace with sysconf(_SC_PAGESIZE) or getpagesize(): Update any fixed values to dynamically retrieve the actual page size.

3. Recompile with Latest NDK: Make sure you're building your native libraries with a recent Android NDK (r25 or newer is a good target). This ensures your toolchain is aware of the 16 KB page size and provides correct system definitions.

How to Verify if Your App is Upgraded to a 16 KB Page Size

You can verify if your app is upgraded by running extensive testing. However, here are the few more steps.

1. Check Your Test Device's Page Size:

   • Connect your Android 13+ test device (preferably a newer one like a Pixel) via ADB

   • Run adb shell getconf PAGE_SIZE

   • If it returns 16384, you're testing on a 16 KB page device! If it returns 4096, you'll need to find a different device to properly test for this change

   • Here’s an example screenshot from my device

     

2. Run Your App Extensively: Once you have a 16 KB page device, put your app through its paces. Try all features, especially those involving native code, heavy data loading, or complex operations.

3. Monitor for Crashes: Keep a close eye on your crash reporting tools (like Crashlytics). Specifically look for native crashes (SIGSEGV, SIGBUS) coming from Android 13+ devices, as these could be related to page size issues.

4. Memory Profiling: While less direct, if you suspect memory inefficiency in your native code, use Android Studio's Memory Profiler to see if allocations are unexpectedly large or if there's excessive memory usage.

Conclusion

In this blog, we learnt about page size in Android, and why and how to upgrade your app to support 16 KB page size. I hope you have a clear idea about 16 KB page size in Android. By being proactive now, you can avoid last-minute scrambling and ensure your apps continue to perform beautifully on the latest Android devices, well past the November 2025 deadline!

You can follow my Twitter/X account to receive the top AI news everyday. If you wish to learn more about mobile app development, subscribe to my email newsletter (https://5minslearn.gogosoon.com/) and follow me on social media.

And I’m not an exception to this. I wanted to create more content, and had an idea based on something I’ve observed. I subscribe to a few technology newsletters, and I read lots of updates every day about the tech ecosystem. But I’ve noticed that many of my peers often don’t seem to be aware of this news. So, I decided to post my top three news stories (especially about AI) on my Twitter/X account every day.

I did this for a couple of weeks, but after that I couldn’t find the time to keep it going. So, I did some research into how I could automate the process, and I found a solution. In this guide, I’ll explain the process so you can use it, too.

By the end of this tutorial, you’ll have created your own AI bot that:

• Fetches data from an API or crawls a webpage

• Processes the data using AI

• Posts the results on Twitter/X

And the great thing: this entire process is automated.

Table of Contents

• Prerequisites

• How to Build the Bot

  • Step 1: Generate the Twitter API Key

  • Step 2: Generate Access Token and Secret

  • Step 3: Generate an API Key in Google Gemini

• Node.js Project Setup

  • Step 1: Fetch Data from the API

  • Step 2: Upload the Data as a File to Gemini API

  • Step 3: Prompt Gemini to Get the Latest AI News

  • Step 4: Post Using the Twitter/X API

  • Step 5: Delete the File Uploaded in the Gemini API

  • The Result

• Conclusion

Prerequisites

Before we begin creating a bot, you’ll need to have the following setup and tools ready to go:

• NodeJS - A simple NodeJS app to code the bot

You’ll also need some API keys, secrets, and tokens. So, you’ll need to have the following accounts created:

• Twitter Developer – To generate the Twitter/X API keys, secrets, and tokens

• Google AI Studio – To generate the Gemini API key

How to Build the Bot

There are a number of steps I’ll walk you through to build your bot.

We’ll start by generating an API Key and Secret so we can use the Twitter/X API. Then we’ll generate an access token and access token secret with “Read and Write” permissions that’ll be able to post in your account. After that we’ll generate an API Key in Google Gemini (we’ll be using the Gemini API to process the data).

With all that taken care of, we’ll start working on the Node.js app. The app will be able to fetch data from an API, process the data using AI, and then post that data in the form of tweets on Twitter/X.

Finally, we’ll automate the entire process and schedule it to run daily.

Step 1: Generate the Twitter API Key

1. Navigate to Twitter Developer Website.

2. Click on the “Developer Portal” in the top right:

   

3. Signup using your account.

4. You’ll be asked to fill out a form asking how will you use the Twitter API, and a few basic details. It may take up to 24 hours to get approved. But, it’s approved instantly for me.

   

5. After login, Navigate to "Projects and Apps" and under “Overview” click on "Create App":

   

6. Enter a name for your app and click “Next” to proceed with creating your app. At the end, you’ll be shown your API Key and Secret. Don’t copy that now.

7. Click on the project you created from the left side drawer and click on the "Edit" option in “User authentication settings” section.

   

8. Select “Read and Write” in App Permissions section, “Web App, Automated App or Bot” in Type of App section, and enter your website URL (it can be any URL including http://localhost) in the “Callback URI” and “Website URL”. Then hit “Save”.

9. Go to “Keys and tokens” tab.

10. Click on “Regenerate” button in “API Key and Secret” section.

11. Copy and save the API Key and Secret somewhere securely.

Step 2: Generate Access Token and Secret

1. Go to “Keys and tokens” tab.

2. Click on “Generate” or “Regenerate” button in “Access Token and Secret” section.

3. Copy and save the Access Token and Secret somewhere securely.

   

Step 3: Generate an API Key in Google Gemini

1. Navigate to Google AI Studio.

2. Login to your account.

3. Click on “Get API Key” button at the top right.

4. Click on “Create API Key” button.

   

5. Copy and save the API Key somewhere securely.

Alright, we are done with creating the necessary API Keys and Secrets for our project. Let’s put on our coding shoes.

Node.js Project Setup

There are 5 major steps for this part of the project. They are:

1. Fetch data from the API

2. Upload the data as a file to Gemini API

3. Prompt Gemini with the uploaded file to get the latest AI news

4. Post news to Twitter/X using their API

5. Delete the file uploaded in Gemini API

These are just the snippets of code that can be assembled together to run this project.

Step 1: Fetch Data from the API

In my case, I’ll be using techmeme.com to get the latest news. But this site does not offer an API. So, I’ll be downloading the HTML of this site.

In the User-Agent header, we pass the value that mimics a browser user agent to avoid potential blocks.

Step 2: Upload the Data as a File to Gemini API

Now we need to store this HTML in a separate file. We cannot directly pass the HTML code in the prompt to the Gemini API, as it’ll result in an error. This is because Gemini accepts only a limited number of tokens in this API. The HTML code of any website will always result in huge number of tokens. So, we’ll create a separate file.

Upload the file to the Gemini API. Refer to the file id in the prompt to Gemini.

Step 3: Prompt Gemini to Get the Latest AI News

Let’s write a prompt to Gemini asking it to generate top news by referring to the HTML file provided. We’ll ask it to provide a headline, short description, URL, and three relevant hashtags for each tweet. We’ll also give some example data of how it should look. We’ll ask it to generate a structured response by providing the format of the JSON that we want the output to be.

You can use whatever model you want to, but I’ll be using the gemini-2.5-pro-exp-03-25 model for this use case. I’m using this model because we need a thinking model that thinks and picks the correct top news – not just one that predicts the next token/word. The Gemini 2.5 Pro model best qualifies for this.

Step 4: Post Using the Twitter/X API

Here’s the core of our app. We need to post all the tweets we received from Gemini. We’ll be posting the tweet as a thread. This means that the first tweet will be the root tweet and subsequent tweets will be in the comments of the prior tweet. This makes it a thread.

To do this, we’ll take the id of each tweet after it’s posted and pass it on to the next tweet as a reference. One additional thing to note here is, after each successful tweet, we’ll give a pause of 5 seconds before posting the next tweet. There are few reasons for doing it this way.

1. When any script runs, it usually runs at a much higher speed (usually in milliseconds). So, the second tweet may get posted before the first tweet was posted (maybe due to some poor internet connection). Also, I believe Twitter implements some queue system which may quickly process the second tweet before your first. So it’s always better to leave a small gap – if not 5 seconds then at least 1 second

2. Twitter may have implemented some rate limiting mechanism. So if there are multiple request received from a same IP within a short time frame, they may block the IP and consider your account as spam.

3. Since we’re using a Free tier API, we are limited to 1500 tweets per month. If you’ve paid for this API, you won’t have to worry about this (since you’ll have a higher limit and the rate limiting mechanism –refer to point #2 – might not be applicable). All of this depends on their pricing, so just refer to that and make your call accordingly.

I’m using the free tier, and since it’s a hobby project, having a 5 seconds wait time makes sense. I have not faced any issues so far with this.

Step 5: Delete the File Uploaded in the Gemini API

After posting all the tweets, it’s time to clean up the system. The only thing we need to do as a clean up is delete the uploaded file. It’s always a best practice to remove an unused file that’s no longer needed. And since we’ve already posted the tweets, we no longer need that file. So, we’ll be deleting it in this step.

That’s it. We’re all done. You just need to copy these blocks of code into an index.js file and install some dependencies into the project and you should be good to go.

To make this even more simple, I have created a repo and made it public. Here’s the Github repo URL. You just need to clone the repo, install the dependencies, and run the post command

git clone https://github.com/arunachalam-b/existential-crisis-alert-bot.git
cd existential-crisis-alert-bot
npm i

Create a .env file and update your API keys and secrets in that file:

GEMINI_API_KEY=
TWITTER_API_KEY=
TWITTER_API_SECRET=
TWITTER_ACCESS_TOKEN=
TWITTER_ACCESS_TOKEN_SECRET=

Run the following command to post the latest AI news to your account:

npm run post

The Result

Here’s a sample output of that command:

You can modify the code/prompt to fetch data from a different API and post the top results in your Twitter account.

Conclusion

I hope you now understand how you can automate a slightly complex process using AI and some APIs. Just note that this example is not completely automated. You still have to manually run the command everyday to post the tweets.

But you can automate that process as well. Just drop me a message if you wish to know about that. That topic itself deserves to be a separate tutorial. Also, I would request that you give a star for my project if you enjoyed this tutorial.

Meanwhile, you can follow my Twitter/X account to receive the top AI news everyday. If you wish to learn more about automation, subscribe to my email newsletter (https://5minslearn.gogosoon.com/) and follow me on social media.

In this tutorial, you’ll learn about what scripting is, its many use cases, and some advantages and disadvantages of using scripts. We’ll also go through a few example scripts so you can see them in action.

What is a Script?

A script is a set of instructions written in any scripting language (like Bash, Python, JavaScript, and others) that helps you automate tasks or control processes. Unlike compiled programs, scripts are typically interpreted, meaning they are executed directly by a runtime environment without prior compilation.

Scripts are powerful tools for automating repetitive tasks, managing workflows, and solving small (and sometimes large) problems efficiently. Whether you’re a beginner or an experienced developer, understanding how to write scripts can enhance your productivity and broaden your technical capabilities.

Why Write Scripts?

I’ve already touched on what you can do with scripts. So let’s look at some of their advantages (and challenges, too) so you understand why they’re so powerful – and when to use them.

Advantages of scripts

1. Automation: Scripts can help you simplify repetitive tasks such as data processing or file management.

2. Efficiency: they can also save you time by automating tasks that you would otherwise have to do manually.

3. Error reduction: Scripts can help reduce human errors through consistent execution of instructions.

4. Flexibility: Scripts can adapt to a wide variety of tasks with minimal modification.

5. Integration: They can also seamlessly integrate with other systems, tools, or workflows.

Challenges with scripts

1. Performance: Scripts can be slower than compiled programs due to interpretation overhead.

2. Scalability: They’re not always suitable for large-scale or highly complex tasks.

3. Debugging: Debugging scripts can sometimes be challenging due to their dynamic nature.

4. Security risks: Poorly written scripts can expose vulnerabilities, especially if they execute system-level commands.

When to Use vs Not Use Scripts

Scripts are ideal for:

1. Tasks are simple, well-defined, or one-off

2. Prototyping or quickly automating a process

3. The scope is small enough to avoid complexity

Scripts are not ideal for:

1. Performance-critical tasks requiring high efficiency. Instead of a script, try using a dedicated ETL (Extract, Transform, Load) tool or a message broker, or similar alternative tools that fit your use-case.

2. Applications with extensive user interfaces. Instead, you can build a small application or a modular system with proper logging, testing, and documentation.

3. Scenarios needing long-term maintenance, where compiled programs might be more stable. Instead, use task schedulers or workflow managers like CRON, Airflow, AWS Lambda/GCP Functions.

How to Write Effective Scripts

This is the process I use for writing helpful scripts. After we go through this, we’ll see some examples of scripts in different languages so you can get some hands-on practice.

1. Define the problem: Before writing a script, identify the problem it will solve. Be clear about the tasks to automate and the expected outcomes.

2. Choose the right language:

   • Bash: Ideal for system-level tasks like file operations or server management.

   • Python: Great for data processing, web scraping, and more complex automation.

   • JavaScript: Suited for web development and browser-based automation.

3. Write the script: Use a text editor or an Integrated Development Environment (IDE), and make sure you follow best practices like using comments, meaningful variable names, and modular code. We’ll cover these below.

4. Test the script: Test the script in a controlled environment to ensure it performs as expected without causing errors.

5. Execute and deploy: Run the script in its intended environment. If necessary, schedule its execution using tools like Cron (for Bash) or task schedulers.

Example Scripts

Now that you know the basics, let's have some practice. Assume you have around 100 files with the names “book-part-1.pdf”, “book-part-2.pdf”, …, “book-part-100.pdf”. You want to replace all the hyphens (-) in the file names with underscores (_), because the website where you're trying to upload these documents doesn’t allow you to upload files with names containing hyphens.

Here are scripts written in three different languages that all perform the same operation. The process looks like this:

1. find all the files in a directory,

2. check if they contain hyphens (-) in their name, and

3. replace any hyphens with underscores (_).

Here are the file names to start (containing hyphens):

Bash Script

We’ll start with a bash script. Here it is:

#!/bin/bash
# Replace "-" with "_" in file names
DIRECTORY="/path/to/your/folder"
for FILE in "$DIRECTORY"/*; do
    if [[ "$FILE" == *-* ]]; then
        NEW_NAME=$(echo "$FILE" | sed 's/-/_/g')
        mv "$FILE" "$NEW_NAME"
        echo "Renamed: $FILE -> $NEW_NAME"
    fi
done

We define the directory (folder) at the top where our files reside. For each file in the directory, we check if the name contains a -. In such case, we create a new file name and store it in the variable NEW_NAME by copying the old file name using the echo command and replace the - with _ using the sed command. Finally we use the move command mv with the old and new file names as arguments.

Python Script

Next, let’s see what it would look like in Python:

import os
# Replace "-" with "_" in file names
directory = "/path/to/your/folder"
for filename in os.listdir(directory):
    if "-" in filename:
        old_path = os.path.join(directory, filename)
        new_filename = filename.replace("-", "_")
        new_path = os.path.join(directory, new_filename)
        os.rename(old_path, new_path)
        print(f"Renamed: {filename} -> {new_filename}")

The steps are pretty similar in Python. First, we define the directory and then iterate through each file in the directory. To find all files in the directory, we have to use the listdir method from the os package.

Then we check if the file name contains a - in the next line. In such a case, we find the current path (old_path) of the file by merging the directory and its file name. We can create the new file name by replacing the - with _ using the replace method.

We then generate the new file path (new_path) in the similar way we generate the old_path. Finally, we call the rename method in os package with old and new file paths as arguments.

JavaScript Script

And now let’s see how it would look in JavaScript:

const fs = require('fs');
const path = require('path');
const directory = '/path/to/your/folder';

fs.readdir(directory, (err, files) => {
    if (err) {
        console.error('Error reading directory:', err);
        return;
    }
    files.forEach(file => {
        if (file.includes('-')) {
            const oldPath = path.join(directory, file);
            const newFilename = file.replace(/-/g, '_');
            const newPath = path.join(directory, newFilename);
            fs.rename(oldPath, newPath, err => {
                if (err) {
                    console.error(`Error renaming ${file}:`, err);
                } else {
                    console.log(`Renamed: ${file} -> ${newFilename}`);
                }
            });
        }
    });
});

The JavaScript implementation is somewhat similar to the Python implementation – but you’ll need to write more code. Generally, devs don’t prefer JavaScript for these kind of scripts. Most of them rely on Bash/Python. JavaScript is better-suited for browser-based automation scripts.

Still, let’s see what we have here. In this JavaScript code, you have to use two different packages, fs and path. We define the directory at the top, read the files in the directory using the readdir method from the fs package, and pass the directory as the argument. Along with directory, we also pass a callback function that will be executed once the files are read.

Inside the callback function, we loop through each file and check if the file name includes a hyphen (-). If it does, we find the old path using the path package with the directory and file names as arguments. We then construct the new file name by replacing all hyphens with underscores using the replace method.

Similar to the old path, we find the new path using the new file name as an argument. Then we use the rename method from the fs package to rename the file by passing both the old and new file names. If there are errors during renaming or reading the files in a directory, we log the error message. Otherwise, we log the success message.

How to run these scripts

Ok, here’s how you can actually use these scripts:

1. Replace /path/to/your/folder with the actual directory containing the files.

2. Run the script in the corresponding environment:

   • Bash: Save as a .sh file, then execute with bash script.sh

   • Python: Save as a .py file, then execute with python script.py

   • JavaScript: Save as a .js file, then execute with node script.js

The screenshot below shows running the bash script to change the names of the files.

Recurring Scripts

Recurring scripts are designed to execute at regular intervals, like checking a system’s status weekly, cleaning up logs, or fetching data updates. These scripts typically use some form of task scheduler.

Common Approaches

1. CRON jobs: Most operating systems support CRON, which can trigger scripts based on a defined schedule.

2. Task queues: Tools like Celery (Python), Bull (Node.js), or Sidekiq (Ruby) can handle scheduled jobs with more flexibility.

3. Cloud schedulers: Services like AWS Lambda with EventBridge, Google Cloud Scheduler, or Azure Logic Apps allow you to set up recurring scripts in a serverless architecture.

One good example use case for recurring scripts would be sending a daily/weekly report of your system’s usage/performance. You could write a script that finds the number of users who have joined and subscribed to your product and send that report as an email every day/week.

Best Practices for Writing Scripts

Here are some things to keep in mind when you’re writing scripts:

1. Use comments: Explain complex parts of the script with comments.

In the below example, without the comment, someone might have to spend extra time figuring out why the tax rate is a decimal and not a percentage.

# Calculate the total price with tax
def calculate_price_with_tax(price, tax_rate):
    # Tax rate is expressed as a decimal (e.g., 0.07 for 7%)
    return price + (price * tax_rate)

2. Error handling: Account for possible errors and handle them gracefully.

In the below example, if the file is missing, the script won’t crash – instead, it will show a helpful error message.

try:
    with open('data.csv', 'r') as file:
        data = file.readlines()
except FileNotFoundError:
    print("Error: 'data.csv' file not found. Make sure the file exists before running the script.")
except Exception as e:
    print(f"An unexpected error occurred: {e}")

3. Modular design: Break down the script into reusable functions or modules.

In the below example, by separating functionality into smaller, reusable functions, you can debug or reuse parts of the script independently.

def fetch_data_from_api(api_url):
    # Fetch data from the given API
    pass

def process_data(data):
    # Process the data into the desired format
    pass

def save_to_file(data, filename):
    # Save processed data to a file
    pass

# Main script
if __name__ == "__main__":
    data = fetch_data_from_api("https://example.com/api")
    processed_data = process_data(data)
    save_to_file(processed_data, "output.json")

4. Input validation: Validate user inputs to prevent unexpected errors or security risks.

Without validation, someone could input invalid or malicious data (for example, SQL injection strings in certain scenarios).

import re

# Validate that the input is a valid email address
def validate_email(email):
    pattern = r'^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$'
    if not re.match(pattern, email):
        raise ValueError("Invalid email address format")
    return email

# Example usage
try:
    user_email = validate_email(input("Enter your email: "))
    print(f"Valid email: {user_email}")
except ValueError as e:
    print(e)

5. Version control: Use Git or other version control tools to track changes.

If a change breaks the script, you can easily revert to a previous commit using git checkout. Plus, you can collaborate seamlessly with team members.

git init
git add script.py
git commit -m "Initial commit"

Conclusion

Writing scripts is a skill that can significantly boost your productivity and problem-solving abilities. By understanding the basics of scripting languages like Bash, Python, and JavaScript, you can automate tasks, streamline workflows, and save valuable time. Start small, build incrementally, and practice writing scripts for different use cases to master this invaluable skill.

I have an exercise for you. To run and verify this example script, you may think you have to manually create 100 files. That consumes a lot of time.

I wrote a script to generate those 100 files. I would also recommend that you try writing a script to generate 100 files with hyphens in their file names. Then try to run the example script to convert the hyphens to underscores.

This may sound difficult at the beginning, but believe me you just need to write 5 lines of bash code to generate 100 files. Not just 100 – you can even generate a million/billion/trillion files with just 5 lines of code.

If you wish to learn more about Scripts, subscribe to my email newsletter (https://5minslearn.gogosoon.com/) and follow me on social media.

Happy scripting!

During this process, I came across the Sliding Window technique. I found this algorithm to be very interesting, so I wanted to share my learnings with the community.

This tutorial will be useful for you if you're preparing for competitive programming interviews. So, let's get started.

What is the Sliding Window Technique?

The sliding window technique is an algorithmic approach used in computer science and signal processing. It involves selecting a fixed-size subset, or "window," from a larger dataset and moving this window through the dataset in a step-wise fashion.

The window slides over the data, typically one element at a time, and performs some operation on the elements within the window at each step.

Are you confused? Let me elaborate on this technique with an example.

Sliding Window example

Assume you're practicing for competitive programming and you encounter the following problem:

"Find the maximum sum of sub-array of size k with the time complexity of O(N).

Array = [1, 2, 6, 2, 4, 1], k = 3"

If you're not familiar with the concept of time complexity, here's a quick definition:

In theoretical computer science, the time complexity is the computational complexity that describes the amount of computer time it takes to run an algorithm.

And here's a course if you'd like to learn more.

Back to our problem. Basically, we have to find the sub-array of size 3, whose sum is the maximum (largest number). Here's a pictorial view of how we can go about solving this:

Find sum of sub-array of size k

Manual solution

Let's solve it manually. From the above image, let's find the sum of each of the sub-arrays of size 3.

• Sum of 1st sub-array = 1 + 2 + 6 = 9
• Sum of 2nd sub-array = 2 + 6 + 2 = 10
• Sum of 3rd sub-array = 6 + 2 + 4 = 12
• Sum of 4th sub-array = 2 + 4 + 1 = 7

The maximum (biggest) number among 9, 10, 12, and 7 is 12 – or the 3rd sub-array. That is our solution.

Code solution

Alright, let's wear our programming shoes and try to solve this using code.

Here's my solution to the problem:

Here's a quick walk through of the above code.

I'm defining the input array and window size at the bottom, and calling the method findMaxSumOfSequence with these parameters.

Initially I'm doing a validation if the input array size is more than the window size, else the calculation is not possible, so return null.

I'm assuming the maximum sum to be negative infinity.

I iterate over the array, and for each item in the array, I iterate for next k items, find their sum, and assign the current window sum to maximum sum variable if the current window sum is more than existing maximum sum. Finally, return the maximum sum.

Let's try to run the code.

Find maximum sum of a sub-array of size k

Here we go. We got the right answer.

But, the problem does not stop here. If you look at the problem carefully, we have to find the solution in O(N) time complexity.

So, you might wonder what's the time complexity of the above solution. Well, the time complexity of the above solution is O(N*k). This means, we're iterating k times for each item in the array (nested for-loop).

O(N) time complexity basically describes that you have to find the maximum value by iterating over the given array only once.

Using the Sliding Window technique

How do we solve this with one iteration? Here's where sliding window technique comes into play. Let's take a look at the pictorial representation of our solution once again:

Sliding window technique

Here, you may notice that the window slides over the array. It's initially covering the indices 0, 1, and 2 in the 1st sub-array. For the next sub-array, it slides one-position to the right, eliminating 0th index to the left and adding the 3rd index to the right. So, now it covers the 1, 2, and 3 in the 2nd sub-array...and so on.

The calculation goes this way for the sub-arrays:

• 1st sub-array = 1 + 2 + 6 = 9
• 2nd sub-array = 9 (sum of 1st sub-array) - 1 + 2 = 10

Let's take a look at this carefully. We find the sum of 1st sub-array to be 9. To calculate the sum of 2nd sub-array, we subtract the number that's going out (1 at 0th index) and add the number that's coming in (2 at 3rd index).

• 3rd sub-array = 10 (sum of 2nd sub-array) - 2 + 4 = 12
• 4th sub-array = 12 (sum of 3rd sub-array) - 6 + 1 = 7

This is the sliding window technique. Following this technique, we'll able to find the sum of maximum sub-array in a single iteration.

How to implement Sliding Window in code

Alright, let's put on our coding shoes again and try to implement this.

Let's try to understand the above code. I made some changes to the findMaxSumOfSequence method. I introduced start and end variables which describe the window block.

In this implementation, we have 2 loops but they're not nested. This is because, in the first loop, we have to find the sum of the 1st window. The second loop will subtract and add items from the result of the first loop.

From the above example, the first loop will iterate over the first k items (3) which are 1, 2, and 6. I calculate the sum and store them in the maxSum and windowSum variables.

In the next loop, I'm iterating over each item in the array. For each item, I'm subtracting the previous number and adding the next number, and updating the result in the windowSum variable. I compare the windowSum and maxSum variables and update the maxSum variable if the windowSum is larger. Then I move the window to the next sub-array by incrementing the start and end variables. Finally, I return the maximum sum.

Here's the output of the above code:

Find maximum sum of a sub-array of size k using sliding window technique

With this implementation, we have satisfied the problem's requirement by iterating over the array only once and finding the maximum sum of a sub-array (with O(N) time complexity).

Applications of the Sliding Window Technique

The Sliding Window Technique is versatile and finds applications in various domains.

Array and String Manipulation: In array or string processing, a sliding window can be used to efficiently perform operations such as finding subarrays or substrings that satisfy certain conditions.

Data Compression: Sliding window compression algorithms, like LZ77 and its variants, use a window to find repeated patterns in the input data and replace them with references to previous occurrences.

Image Processing: In image processing, a sliding window can be employed for tasks such as feature extraction, object detection, or image segmentation.

Signal Processing: Time-series data can be analyzed using a sliding window to capture local patterns, trends, or anomalies.

Network Protocols: Sliding window protocols are used in computer networks for reliable and efficient data transmission. The sender and receiver maintain a window of allowable sequence numbers to manage the flow of data.

Conclusion

Hope you now have a clear idea of how the Sliding Window Technique works after seeing these examples. I would recommend that you try to solve some other problems with this technique to get familiar with it. Trying to find the minimum sum of the sub-array of size k on your own using this technique would be a helpful exercise.

As I mentioned earlier, I'm actively looking to switch my job. If you have a good position available in your organization, please refer me.

Hope you enjoyed reading this article. If you wish to learn more about techniques to crack competitive programming interviews, subscribe to my newsletter by visiting my site which also has a consolidated list of all my blogs.

Binary classification is used in a wide range of applications, such as spam email detection, medical diagnosis, sentiment analysis, fraud detection, and many more.

In this article, we'll explore binary classification using TensorFlow, one of the most popular deep learning libraries.

Before getting into the Binary Classification, let's discuss a little about classification problem in Machine Learning.

What is Classification problem?

A Classification problem is a type of machine learning or statistical problem in which the goal is to assign a category or label to a set of input data based on their characteristics or features. The objective is to learn a mapping between input data and predefined classes or categories, and then use this mapping to predict the class labels of new, unseen data points.

Sample Multi Classification

The above diagram represents a multi-classification problem in which the data will be classified into more than two (three here) types of classes.

Sample Binary Classification

This diagram defines Binary Classification, where data is classified into two type of classes.

This simple concept is enough to understand classification problems. Let's explore this with a real-life example.

Heart Attack Analytics Prediction Using Binary Classification

In this article, we will embark on the journey of constructing a predictive model for heart attack analysis utilizing straightforward deep learning libraries.

The model that we'll be building, while being a relatively simple neural network, is capable of achieving an accuracy level of approximately 80%.

Solving real-world problems through the lens of machine learning entails a series of essential steps:

1. Data Collection and Analytics
2. Data preprocessing
3. Building ML Model
4. Train the Model
5. Prediction and Evaluation

Data Collection and Analytics

It's worth noting that for this project, I obtained the dataset from Kaggle, a popular platform for data science competitions and datasets.

I encourage you to take a closer look at its contents. Understanding the dataset is crucial as it allows you to grasp the nuances and intricacies of the data, which can help you make informed decisions throughout the machine learning pipeline.

This dataset is well-structured, and there's no immediate need for further analysis. However, if you are collecting the dataset on your own, you will need to perform data analytics and visualization independently to achieve better accuracy.

Let's put on our coding shoes.

Here I am using Google Colab. You can use your own machine (in which case you will need to create a .ipynb file) or Google Colab on your account to run the notebook. You can find my source code here.

As the first step, let's import the required libraries.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
import sklearn
import pandas as pd
import keras
from keras.models import Sequential
from keras.layers import Dense
import tensorflow as tf
from sklearn.metrics import confusion_matrix,ConfusionMatrixDisplay
from sklearn.preprocessing import MinMaxScaler

I have the dataset in my drive and I'm reading it from my drive. You can download the same dataset here.

Remember the replace the path of your file in the read_csv method:

df = pd.read_csv("/content/drive/MyDrive/Datasets/heart.csv")
df.head()

Sample 5 record in the dataset

The dataset contains thirteen input columns (age, sex, cp, and so on) and one output column (output), which will contain the data as either 0 or 1.

Considering the input readings, 0 in the output represents the person will not get heart attack, while the 1 represents the person will be affected by heart attack.

Let's split our input and output from the above dataset to train our model:

target_column = "output"
numerical_column = df.columns.drop(target_column)
output_rows = df[target_column]
df.drop(target_column,axis=1,inplace=True)

Since our objective is to predict the likelihood of a heart attack (0 or 1), represented by the target column, we split that into a separate dataset.

Data preprocessing

Data preprocessing is a crucial step in the machine learning pipeline, and binary classification is no exception. It involves the cleaning, transformation, and organization of raw data into a format that is suitable for training machine learning models.

A dataset will contain multiple type of data such as Numerical Data, Categorical Data, Timestamp Data, and so on.

But most of the Machine Learning algorithms are designed to work with numerical data. They require input data to be in a numeric format for mathematical operations, optimization, and model training.

In this dataset, all the columns contain numerical data, so we don't need to encode the data. We can proceed with simple normalization.

Remember if you have any non-numerical columns in your dataset, you may have to convert it into numerical by performing one-hot encoding or using other encoding algorithms.

There are lot of normalization strategies. Here I am using Min-Max Normalization:

Min-Max Scaling Formula

Don't worry – we don't need to apply this formula manually. We have some machine learning libraries to do this. Here I am using MinMaxScaler from sklearn:

scaler = MinMaxScaler()
scaler.fit(df)
t_df = scaler.transform(df)

scaler.fit(df) computes the mean and standard deviation (or other scaling parameters) necessary to perform the scaling operation. The fit method essentially learns these parameters from the data.

t_df = scaler.transform(df): After fitting the scaler, we need to transform the dataset. The transformation typically scales the features to have a mean of 0 and a standard deviation of 1 (standardization) or scales them to a specific range (for example, [0, 1] with Min-Max scaling) depending on the scaler used.

We have completed the preprocessing. The next crucial step is to split the dataset into training and testing sets.

To accomplish this, I will utilize the train_test_split function from scikit-learn.

X_train and X_test are the variables that hold the independent variables.

y_train and y_test are the variables that hold the dependent variable, which represents the output we are aiming to predict.

X_train, X_test, y_train, y_test = train_test_split(t_df, output_rows, test_size=0.25, random_state=0)

print('X_train:',np.shape(X_train))
print('y_train:',np.shape(y_train))
print('X_test:',np.shape(X_test))
print('y_test:',np.shape(y_test))

Sample training and testing dataset size

We split the dataset by 75% and 25%, where 75% goes for training our model and 25% goes for testing our model.

Building ML Model

A machine learning model is a computational representation of a problem or a system that is designed to learn patterns, relationships, and associations from data. It serves as a mathematical and algorithmic framework capable of making predictions, classifications, or decisions based on input data.

In essence, a model encapsulates the knowledge extracted from data, allowing it to generalize and make informed responses to new, previously unseen data.

Here, I am building a simple sequential model with one input layer and one output layer. Being a simple model, I am not using any hidden layer as it might increase the complexity of the concept.

Initialize Sequential Model

basic_model = Sequential()

Sequential is a type of model in Keras that allows you to create neural networks layer by layer in a sequential manner. Each layer is added on top of the previous one.

Input Layer

basic_model.add(Dense(units=16, activation='relu', input_shape=(13,)))

Dense is a type of layer in Keras, representing a fully connected layer. It has 16 units, which means it has 16 neurons.

activation='relu' specifies the Rectified Linear Unit (ReLU) activation function, which is commonly used in input or hidden layers of neural networks.

input_shape=(13,) indicates the shape of the input data for this layer. In this case, we are using 13 input features (columns).

Output Layer

basic_model.add(Dense(1, activation='sigmoid'))

This line adds the output layer to the model.

It's a single neuron (1 unit) because this appears to be a binary classification problem, where you're predicting one of two classes (0 or 1).

The activation function used here is 'sigmoid', which is commonly used for binary classification tasks. It squashes the output to a range between 0 and 1, representing the probability of belonging to one of the classes.

Optimizer

adam = keras.optimizers.Adam(learning_rate=0.001)

This line initializes the Adam optimizer with a learning rate of 0.001. The optimizer is responsible for updating the model's weights during training to minimize the defined loss function.

Compile Model

basic_model.compile(loss='binary_crossentropy', optimizer=adam, metrics=["accuracy"])

Here, we'll compile the model.

loss='binary_crossentropy' is the loss function used for binary classification. It measures the difference between the predicted and actual values and is minimized during training.

metrics=["accuracy"]: During training, we want to monitor the accuracy metric, which tells you how well the model is performing in terms of correct predictions.

Train model with dataset

Hurray, we built the model. Now it's time to train the model with our training dataset.

basic_model.fit(X_train, y_train, epochs=100)

X_train represents the training data, which consists of the independent variables (features). The model will learn from these features to make predictions or classifications.

y_train are the corresponding target labels or dependent variables for the training data. The model will use this information to learn the patterns and relationships between the features and the target variable.

epochs=100: The epochs parameter specifies the number of times the model will iterate over the entire training dataset. Each pass through in the dataset is called an epoch. In this case, we have 100 epochs, meaning the model will see the entire training dataset 100 times during training.

loss_and_metrics = basic_model.evaluate(X_test, y_test)
print(loss_and_metrics)
print('Loss = ',loss_and_metrics[0])
print('Accuracy = ',loss_and_metrics[1])

The evaluate method is used to assess how well the trained model performs on the test dataset. It computes the loss (often the same loss function used during training) and any specified metrics (for example, accuracy) for the model's predictions on the test data.

Sample output to find the Loss and Accuracy

Here we got around 82% accuracy.

Prediction and Evaluation

predicted = basic_model.predict(X_test)

The predict method is used to generate predictions from the model based on the input data (X_test in this case). The output (predicted) will contain the model's predictions for each data point in the training dataset.

Since I have only minimum dataset I am using the test dataset for prediction. However, it is a recommend practice to split a part of dataset (say 10%) to use as a validation dataset.

Evaluation

Evaluating predictions in machine learning is a crucial step to assess the performance of a model.

One commonly tool used for evaluating classification models is the confusion matrix. Let's explore what a confusion matrix is and how it's used for model evaluation:

In a binary classification problem (two classes, for example, "positive" and "negative"), a confusion matrix typically looks like this:

Here's the code to plot the confusion matrix from the predicted data of our model:

predicted = tf.squeeze(predicted)
predicted = np.array([1 if x >= 0.5 else 0 for x in predicted])
actual = np.array(y_test)
conf_mat = confusion_matrix(actual, predicted)
displ = ConfusionMatrixDisplay(confusion_matrix=conf_mat)
displ.plot()

Confusion matrix for the predicted output

Bravo! We've made significant progress toward obtaining the required output, with approximately 84% of the data appearing to be correct.

It's worth noting that we can further optimize this model by leveraging a larger dataset and fine-tuning the hyper-parameters. However, for a foundational understanding, what we've accomplished so far is quite impressive.

Given that this dataset and the corresponding machine learning models are at a very basic level, it's important to acknowledge that real-world scenarios often involve much more complex datasets and machine learning tasks.

While this model may perform adequately for simple problems, it may not be suitable for tackling more intricate challenges.

In real-world applications, datasets can be vast and diverse, containing a multitude of features, intricate relationships, and hidden patterns. Consequently, addressing such complexities often demands a more sophisticated approach.

Here are some key factors to consider when working with complex datasets.

1. Complex Data Preprocessing
2. Advanced Data Encoding
3. Understanding Data Correlation
4. Multiple Neural Network Layers
5. Feature Engineering
6. Regularization

If you're already familiar with building a basic neural network, I highly recommend delving into these concepts to excel in the world of Machine Learning.

Conclusion

In this article, we embarked on a journey into the fascinating world of machine learning, starting with the basics.

We explored the fundamentals of binary classification—a fundamental machine learning task. From understanding the problem to building a simple model, we've gained insights into the foundational concepts that underpin this powerful field.

So, whether you're just starting or already well along the path, keep exploring, experimenting, and pushing the boundaries of what's possible with machine learning. I'll see you in another exciting article!

If you wish to learn more about artificial intelligence / machine learning / deep learning, subscribe to my article by visiting my site, which has a consolidated list of all my articles.

Not only that – everyone wants to build and release an AI product on their own to mark their position in the global competition.

And if you already own a SaaS product, you may want to integrate AI and ML features into it to retain your customers and be competitive in the global market.

In this tutorial, you'll learn how to build a Linear Regression Model. This is one of the first things you'll learn how to do when studying Machine Learning, so it'll help you take your first step into this competitive market.

Table of Contents

• Prerequisites
• What is Linear Regression?
• Evaluation Metrics
• Linear Regression Example – Car Price Prediction Model
• How to Build the Model
• Conclusion

Prerequisites

1. Medium level experience using an IDE (preferably VS Code)
2. Basic understanding of Python Notebook (.pynb) files
3. Good understanding of the Python programming language
4. Basic knowledge of Pandas (to handle dataframes), Numpy, Scikit Learn, and Matplot libraries
5. Some knowledge of statistics is helpful for analyzing the data

What is Linear Regression?

Linear Regression is a Supervised Learning method, where the predicted output will be continuous in nature. For example, things like price prediction, marks prediction, and so on.

Linear Regression is a fundamental statistical and machine learning technique used for modeling the relationship between a dependent variable (also known as the target or response variable) and one or more independent variables (predictors or features).

It aims to establish a linear equation that best represents the association between these variables, allowing us to make predictions and draw insights from the data.

The primary goal of linear regression is to find the "best-fit" line (or hyperplane in higher dimensions) that minimizes the difference between the predicted values and the actual observed values.

This best-fit line is defined by a linear equation of the form:

Y = b0 ​+ b1​X1 ​+ b2​X2 ​+...+ bn​Xn​

In this equation:

1. Y represents the dependent variable we want to predict.
2. X1,X2,...,Xn are the independent variables or features.
3. b0 is the intercept (the value of Y when all X values are zero).
4. b1,b2,...,bn are the coefficients that determine the relationship between each independent variable and the dependent variable.

Linear regression assumes that there is a linear relationship between the predictors and the target variable.

The goal of the model is to estimate the coefficients (b0,b1,...,bn) that minimize the sum of the squared differences between the predicted values and the actual values in the training data. This process is often referred to as "fitting the model."

Evaluation Metrics

The evaluation metrics for a Linear Regression model are:

1. Coefficient of Determination or R-Squared (R2)
2. Root Mean Squared Error (RSME)

Let's see what each of these are.

R-Squared

R-Squared describes the amount of variation that is captured by the developed model. It always ranges between 0 and 1. The higher the value of R-squared, the better the model fits with the data.

Root Mean Squared Error

RMSE measures the average magnitude of the errors or residuals between the predicted values generated by a model and the actual observed values in a dataset. It always ranges between 0 and positive infinity. Lower RMSE values indicate better predictive performance.

Linear Regression Example – Car Price Prediction Model

In this example, we'll try to predict the car price by building a Linear Regression model. I found this problem and the dataset in Kaggle. I noticed that there's a submission for this problem, which was perfect. In fact I built my solution by taking a part of that solution.

Let's dive into the problem.

We're given a dataset of used cars, which contains the name of the car, year, selling price, present price, number of kilometers it has driven, type of fuel, type of the seller, transmission, and if the seller is the owner. Our goal is to predict the selling price of the car.

Let's explore the solution.

Import the required packages:

You'll need various packages to work through this problem. Here's how you can import them all:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn import metrics
from sklearn.model_selection import KFold
from sklearn.pipeline import make_pipeline
from statsmodels.stats.diagnostic import normal_ad
from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.stats.stattools import durbin_watson
from scipy import stats

Import the dataset

You can download the dataset (car data.csv) from Kaggle or you can download it from my Github repo.

df = pd.read_csv('./car data.csv')

Pre-process the dataset

The below code shows the columns and their datatype and the number of rows. Our dataset has 9 columns and 301 rows.

df.info()

Datatype of columns in the dataset

The "Car_Name" column describes the car name. This field should be ignored from our dataset. This is because, only the features of the car matters and not its name.

The below code returns the number of unique car names in our dataset.

df['Car_Name'].nunique()

We have 98 unique car names in our dataset.

Number of unique car names

Clearly, it does not add any meaning to our dataset, since there are so many categories. Let's drop that column.

df.drop('Car_Name', axis=1, inplace=True)

The dataset has the column named "Year". Ideally, we need the age of car over the year it was bought / sold. So, let's convert that to "Age" and remove the "Year" column.

df.insert(0, "Age", df["Year"].max()+1-df["Year"] )
df.drop('Year', axis=1, inplace=True)

"Age" is calculated by finding the difference between the maximum year available in our dataset and the year of that particular car. This is because, our calculations will be specific to that particular time period and to this dataset.

Visualize the data in the dataset

How to find outliers

An outlier is a data point that differs significantly from other observations. They can cause the performance of the model to drop.

How to determine Outliers?

Categorical columns will have the datatype as "object". Let's group the numerical columns and categorical columns in a NumPy array. The first 5 elements in the array will be numerical columns and the rest 3 will be categorical columns.

We can plot the data in the columns using the seaborn library. Categorical columns will contain multiple bars, whereas numerical columns will contain single bars.

Let's try to find the outliers in our dataset using the following code:

sns.set_style('darkgrid')
colors = ['#0055ff', '#ff7000', '#23bf00']
CustomPalette = sns.set_palette(sns.color_palette(colors))

OrderedCols = 
np.concatenate([df.select_dtypes(exclude='object').columns.values,  df.select_dtypes(include='object').columns.values])

fig, ax = plt.subplots(2, 4, figsize=(15,7),dpi=100)

for i,col in enumerate(OrderedCols):
    x = i//4
    y = i%4
    if i<5:
        sns.boxplot(data=df, y=col, ax=ax[x,y])
        ax[x,y].yaxis.label.set_size(15)
    else:
        sns.boxplot(data=df, x=col, y='Selling_Price', ax=ax[x,y])
        ax[x,y].xaxis.label.set_size(15)
        ax[x,y].yaxis.label.set_size(15)

plt.tight_layout()    
plt.show()

Outliers determination in the dataset

Let's try to find the outliers using the InterQuartile Range rule.

This is based on the concept of quartiles, which divide a dataset into four equal parts. The IQR (InterQuartile Range rule) rule specifically focuses on the range of values within the middle 50% of the data and uses this range to identify potential outliers.

We have to find the minimum and maximum quantile values for each unique value in the categorical columns and filter the outlier samples which do not fit into the 25th and 75th percentile of our target column (Selling Price).

On the other hand, the outliers in numerical columns can be filtered by the 25th and 75th percentiles of the same column. We don't need to filter out with respect to the target column.

outliers_indexes = []
target = 'Selling_Price'

for col in df.select_dtypes(include='object').columns:
    for cat in df[col].unique():
        df1 = df[df[col] == cat]
        q1 = df1[target].quantile(0.25)
        q3 = df1[target].quantile(0.75)
        iqr = q3-q1
        maximum = q3 + (1.5 * iqr)
        minimum = q1 - (1.5 * iqr)
        outlier_samples = df1[(df1[target] < minimum) | (df1[target] > maximum)]
        outliers_indexes.extend(outlier_samples.index.tolist())


for col in df.select_dtypes(exclude='object').columns:
    q1 = df[col].quantile(0.25)
    q3 = df[col].quantile(0.75)
    iqr = q3-q1
    maximum = q3 + (1.5 * iqr)
    minimum = q1 - (1.5 * iqr)
    outlier_samples = df[(df[col] < minimum) | (df[col] > maximum)]
    outliers_indexes.extend(outlier_samples.index.tolist())

outliers_indexes = list(set(outliers_indexes))
print('{} outliers were identified, whose indices are:\n\n{}'.format(len(outliers_indexes), outliers_indexes))

By running the above code, we found that there are 38 outliers in our dataset.

List of initial outliers indices

But keep in mind that it's not always the right decision to remove the outliers. They can be legitimate observations and it’s important to investigate the nature of the outlier before deciding whether to drop it or not.

We can delete outliers in two cases:

1. Outlier is due to incorrectly entered or measured data
2. Outlier creates a significant association

Let's dig even more and find the perfect outliers.

To do that, let's assume that if the selling price is more than 33 Lakhs or if the car has been driven more than 400,000 Kilometers, those are outliers. We'll mark them in green. Save all the indices in the removing_indices variable. Plot them in the scatterplot format using seaborn library, comparing each column against our target column.

# Outliers Labeling
df1 = df.copy()
df1['label'] = 'Normal'
df1.loc[outliers_indexes,'label'] = 'Outlier'

# Removing Outliers
removing_indexes = []
removing_indexes.extend(df1[df1[target]>33].index)
removing_indexes.extend(df1[df1['Kms_Driven']>400000].index)
df1.loc[removing_indexes,'label'] = 'Removing'

# Plot
target = 'Selling_Price'
features = df.columns.drop(target)
colors = ['#0055ff','#ff7000','#23bf00']
CustomPalette = sns.set_palette(sns.color_palette(colors))
fig, ax = plt.subplots(nrows=3 ,ncols=3, figsize=(15,12), dpi=200)

for i in range(len(features)):
    x=i//3
    y=i%3
    sns.scatterplot(data=df1, x=features[i], y=target, hue='label', ax=ax[x,y])
    ax[x,y].set_title('{} vs. {}'.format(target, features[i]), size = 15)
    ax[x,y].set_xlabel(features[i], size = 12)
    ax[x,y].set_ylabel(target, size = 12)
    ax[x,y].grid()

ax[2, 1].axis('off')
ax[2, 2].axis('off')
plt.tight_layout()
plt.show()

Outliers marked in Green color

Let's see the perfect outliers:

removing_indexes = list(set(removing_indexes))
removing_indexes

List of original outliers indices

We got 2. We have to remove them. But before that, we have to check if there's any null data in our dataset.

df.isnull().sum()

There are no null values in our dataset.

No null values in the dataset

Let's remove the identified outliers and reset the index of the dataframe.

df1 = df.copy()
df1.drop(removing_indexes, inplace=True)
df1.reset_index(drop=True, inplace=True)

Analyze the Dataset

Let's analyze the data to see how much each field/category is correlated with the selling price of the car. We need to do some analysis on our dataset to be able to come to some conclusions about it.

To do that, we have to identify the numerical and categorical fields in our dataset, as the way to plot this differs for each type.

NumCols = ['Age', 'Selling_Price', 'Present_Price', 'Kms_Driven', 'Owner']
CatCols = ['Fuel_Type', 'Seller_Type', 'Transmission']

Bivariate Analysis

If you're not familiar what bivariate analysis is, here's a basic definition:

Bivariate analysis is one of the simplest forms of quantitative analysis. It involves the analysis of two variables, for the purpose of determining the empirical relationship between them. Bivariate analysis can be helpful in testing simple hypotheses of association.

Let's compare the Selling Price with the other columns using Bivariate Analysis and try to derive some conclusions out of that data.

Selling Price vs Numerical Features Bivariate Analysis

Let's compare the numerical features with the Selling Price using Bivariate analysis. The Numerical columns will be plotted in a scatter graph.

fig, ax = plt.subplots(nrows=2 ,ncols=2, figsize=(10,10), dpi=90)
num_features = ['Present_Price', 'Kms_Driven', 'Age', 'Owner']
target = 'Selling_Price'
c = '#0055ff'

for i in range(len(num_features)):
    row = i//2
    col = i%2
    ax[row,col].scatter(df1[num_features[i]], df1[target], color=c, edgecolors='w', linewidths=0.25)
    ax[row,col].set_title('{} vs. {}'.format(target, num_features[i]), size = 12)
    ax[row,col].set_xlabel(num_features[i], size = 12)
    ax[row,col].set_ylabel(target, size = 12)
    ax[row,col].grid()

plt.suptitle('Selling Price vs. Numerical Features', size = 20)
plt.tight_layout()
plt.show()

Selling Price vs Numerical Features Bivariate Analysis

Selling Price vs Categorical Features Bivariate Analysis

Let's compare the categorical features with the Selling Price using Bivariate analysis. The Categorical columns will be plotted in a stripplot graph. This gives the comparison among multiple values in a category.

fig, axes = plt.subplots(nrows=1 ,ncols=3, figsize=(12,5), dpi=100)
cat_features = ['Fuel_Type', 'Seller_Type', 'Transmission']
target = 'Selling_Price'
c = '#0055ff'

for i in range(len(cat_features)):
    sns.stripplot(ax=axes[i], x=cat_features[i], y=target, data=df1, size=6, color=c)
    axes[i].set_title('{} vs. {}'.format(target, cat_features[i]), size = 13)
    axes[i].set_xlabel(cat_features[i], size = 12)
    axes[i].set_ylabel(target, size = 12)
    axes[i].grid()

plt.suptitle('Selling Price vs. Categorical Features', size = 20)
plt.tight_layout()
plt.show()

Selling Price vs Categorical Features Bivariate Analysis

Here are the conclusions we can make from our data analysis:

1. As Present Price increases, Selling Price increases as well. They're directly proportional.
2. Selling Price is inversely proportional to Kilometers Driven.
3. Selling Price is inversely proportional to the car's age.
4. As the number of previous car owners increases, its Selling Price decreases. So Selling Price is inversely proportional to Owner.
5. Diesel Cars > CNG Cars > Petrol Cars in terms of Selling Price.
6. The Selling Price of cars sold by individuals is lower than the price of cars sold by dealers.
7. Automatic cars are more expensive than manual cars.

Categorical Variables Encoding

We can't use the Categorical fields as they are. They have to be converted to numbers because machines can only understand numbers.

For an example, let's take the Fuel column. As per our dataset, we have cars running on two types of fuel. They are Petrol and Diesel. The categorical variable encoding will split the fuel column into 2 columns (Fuel_Type_Petrol and Fuel_Type_Diesel).

Let's assume a car runs on Petrol. For this car, the data will be converted as Fuel_Type_Petrol column set to 1 (True), and Fuel_Type_Diesel column set to 0 (False). Computers can understand 1 and 0 rather than "Petrol" and "Diesel".

To do that, we'll perform one-hot encoding for the categorical columns. Pandas offers the get_dummies method to encode the columns.

CatCols = ['Fuel_Type', 'Seller_Type', 'Transmission']

df1 = pd.get_dummies(df1, columns=CatCols, drop_first=True)
df1.head(5)

Categorical Variables Encoding

Assume True and False are 0 and 1, respectively.

Correlation Analysis

A correlation matrix is a matrix that summarizes the strength and direction of the linear relationships between pairs of variables in a dataset. It is a crucial tool in statistics and data analysis, used to examine the patterns of association between variables and understand how they may be related.

The correlation is directly proportional if the values are positive, and inversely proportional if the values are negative.

Here's the code to find the correlation matrix with relation to Selling Price.

target = 'Selling_Price'
cmap = sns.diverging_palette(125, 28, s=100, l=65, sep=50, as_cmap=True)
fig, ax = plt.subplots(figsize=(9, 8), dpi=80)
ax = sns.heatmap(pd.concat([df1.drop(target,axis=1), df1[target]],axis=1).corr(), annot=True, cmap=cmap)
plt.show()

Correlation Matrix

From the above matrix, we can infer that the target variable "Selling Price" is highly correlated with Present Price, Seller Type, and Fuel Type.

How to Build the Model

We have come to the final stage. Let's train and test our model.

Let's remove the "Selling_Price" from input and set it to output. This means that it has to be predicted.

X = df1.drop('Selling_Price', axis=1)
y = df1['Selling_Price']

Let's split our dataset by taking 70% of data for training and 30% of data for testing.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

Let's make a backup of our test data. We need this for the final comparison.

y_test_actual = y_test

Normalize the dataset

The StandardScaler is a preprocessing technique commonly used in machine learning and data analysis to standardize or normalize the features (variables) of a dataset. Its primary purpose is to transform the data such that each feature has a mean (average) of 0 and a standard deviation of 1.

Let's normalize our dataset using StandardScaler.

scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

It is very important that StandardScaler transformation should only be gotten from the training set, otherwise it will lead to data leakage.

Train the model

linear_reg = LinearRegression()
linear_reg.fit(X_train_scaled, y_train)

Let's find the intercept and co-efficient for each column in our training dataset.

pd.DataFrame(data = np.append(linear_reg.intercept_ , linear_reg.coef_), index = ['Intercept']+[col+" Coef." for col in X.columns], columns=['Value']).sort_values('Value', ascending=False)

Slope and Intercept values

How to evaluate the model

Scikit Learn provides a metrics feature which helps us to measure the metrics of our model. We can use that to determine metrics include Mean Squared Error, Mean Absolute Error, Root Mean Squared Error, and R2-Score.

Now it's time to evaluate the model:

def model_evaluation(model, X_test, y_test, model_name):
    y_pred = model.predict(X_test)

    MAE = metrics.mean_absolute_error(y_test, y_pred)
    MSE = metrics.mean_squared_error(y_test, y_pred)
    RMSE = np.sqrt(MSE)
    R2_Score = metrics.r2_score(y_test, y_pred)

    return pd.DataFrame([MAE, MSE, RMSE, R2_Score], index=['MAE', 'MSE', 'RMSE' ,'R2-Score'], columns=[model_name])

model_evaluation(linear_reg, X_test_scaled, y_test, 'Linear Reg.')

Evaluation of the model

Evaluate the model using K-fold Cross-Validation

In k-fold cross-validation, the dataset is divided into k roughly equal-sized subsets or "folds." The model is trained and evaluated k times, each time using a different fold as the validation set and the remaining folds as the training set.

The results (for example accuracy, error) of these k runs are then averaged to obtain a more robust estimate of the model's performance.

The advantage is that each data point is used for both training and validation, reducing the risk of bias in the evaluation.

linear_reg_cv = LinearRegression()
scaler = StandardScaler()
pipeline = make_pipeline(StandardScaler(),  LinearRegression())

kf = KFold(n_splits=6, shuffle=True, random_state=0) 
scoring = ['neg_mean_absolute_error', 'neg_mean_squared_error', 'neg_root_mean_squared_error', 'r2']
result = cross_validate(pipeline, X, y, cv=kf, return_train_score=True, scoring=scoring)

MAE_mean = (-result['test_neg_mean_absolute_error']).mean()
MAE_std = (-result['test_neg_mean_absolute_error']).std()
MSE_mean = (-result['test_neg_mean_squared_error']).mean()
MSE_std = (-result['test_neg_mean_squared_error']).std()
RMSE_mean = (-result['test_neg_root_mean_squared_error']).mean()
RMSE_std = (-result['test_neg_root_mean_squared_error']).std()
R2_Score_mean = result['test_r2'].mean()
R2_Score_std = result['test_r2'].std()

pd.DataFrame({'Mean': [MAE_mean,MSE_mean,RMSE_mean,R2_Score_mean], 'Std': [MAE_std,MSE_std,RMSE_std,R2_Score_std]},
             index=['MAE', 'MSE', 'RMSE' ,'R2-Score'])

Model Evaluation using Cross-Validation

Results Visualization

Let's create a dataframe with the actual and predicted values.

y_test_pred = linear_reg.predict(X_test_scaled)
df_comp = pd.DataFrame({'Actual':y_test_actual, 'Predicted':y_test_pred})

Let's compare the actual and predicted target values for the test data with the help of a bar plot.

def compare_plot(df_comp):
    df_comp.reset_index(inplace=True)
    df_comp.plot(y=['Actual','Predicted'], kind='bar', figsize=(20,7), width=0.8)
    plt.title('Predicted vs. Actual Target Values for Test Data', fontsize=20)
    plt.ylabel('Selling_Price', fontsize=15)
    plt.show()

compare_plot(df_comp)

Comparison between actual and predicted values

In the above graph, the blue lines indicates the actual price and orange lines indicate the predicted price of the cars. You can see that some predicted values are negative. But in most of our cases, our model has predicted it pretty well.

This is not the perfect model. But if you want to fine tune it to make better predictions, let me know via my email. I'll write a tutorial about that if I receive more requests.

Conclusion

In this tutorial, you learned about Linear Regression with a practical example. I hope it helps you to make progress in your ML journey. You can access the above dataset and the code for it in from this Github repo.

Hope you enjoyed reading this article. If you wish to learn more about artificial intelligence / machine learning / deep learning, subscribe to my article by visiting my site which has a consolidated list of all my blogs.

Flutter's framework allows developers to build beautiful user interfaces, deliver native performance, and streamline development.

In this tutorial, I will walk you through building and releasing a Flutter app on the Android platform, from setting up your development environment to distributing your app on the Google Play Store.

Prerequisites:

1. Install Flutter and Dart: Make sure you have Flutter and Dart installed on your machine. Follow the official Flutter installation guide for your platform.
2. Android Studio: Download and install Android Studio, which comes with useful tools for Android app development.
3. Google Play Developer Account: Create an account on the Google Play Console to publish your app on the Play Store.

How to Develop your app

In this guide, I'll be building and releasing a simple todo app I built in one of my earlier blogs.

However, you can follow along with this tutorial and release your own app based on what you build.

As a learner, and if you want to experience the app releasing process, this will be good practice for you.

Run the following command to clone my repo:

git clone https://github.com/5minslearn/Flutter-Todo-App

Before proceeding with the app-building process, it's crucial to ensure that our app runs smoothly on a simulator or a physical device, without encountering any errors.

Open the repo in VS Code and press F5 to run the app on your phone / emulator.

How to Customize the Default Launcher Icon in Flutter

When a new Flutter app is created, it has a default launcher icon.

To customize this icon we need to follow the steps below:

1. You can create your own icon by following this guideline.
2. In the [project]/android/app/src/main/res/ directory, place your icon files in folders. The default mipmap- folders demonstrate the correct naming convention.
3. In AndroidManifest.xml, update the application tags android:icon attribute to reference icons from the previous step (for example, <application android:icon="@mipmap/custom_icon" ...).

I created an icon named custom_icon.xml with multiple resolutions (mdpi , hdpi, xhdpi, xxhdpi, xxxhdpi):

48 × 48 (mdpi)
72 × 72 (hdpi)
96 × 96 (xhdpi)
144 × 144 (xxhdpi)
192 × 192 (xxxhdpi)

I also placed each icon in the respective mipmap- folder. And finally, I mentioned them in the AndroidManifest.xml:

_Updating customicon in AndroidManifest.xml

<manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.example.todo" >
    <application
        android:label="todo"
        android:name="${applicationName}"
        android:icon="@mipmap/custom_icon">
        .....
        .....
    </application>
</manifest>

I am utilizing ColorControlNormal in my custom icon file (custom_icon.xml):

<vector xmlns:android="http://schemas.android.com/apk/res/android"
    android:width="48dp"
    android:height="48dp"
    android:viewportWidth="960"
    android:viewportHeight="960"
    android:tint="?attr/colorControlNormal">
  <path
      android:fillColor="@android:color/white"
      android:pathData="M380,....,453.5Q592,463 613,483Z"/>
</vector>

colorControlNormal is a value that refers to a color attribute that is resolved at runtime, allowing the drawable to match the color theme of the app.

As I'm using an older API version, this function isn't accessible within the older API.

As a solution, I've included the appcompat library in my app/build.gradle file:

...
dependencies {
    ....
    implementation 'androidx.appcompat:appcompat:1.3.1'
    ....
}
....

Implement appcombat library in build.gradle

That's it. Our configurations are done. Let's try to run the app and look at the icon in the device.

App Icon Changed for Todo App

The one with the rabbit's face is the one I created. It has been successfully applied to our app.

Without any delay, let's get into building our Android application.

What is a Keystore and Why DO You Need it in Flutter?

A keystore in Flutter, specifically for Android, is a secure container used to store cryptographic keys and certificates.

It's important to maintain the security of your app, especially when dealing with sensitive information such as API keys, authentication tokens, and encryption keys.

The keystore ensures that these sensitive assets are stored in a way that makes them difficult to extract from the device.

Security

Storing sensitive information in a Keystore ensures that it is protected from unauthorized access, even if the device is compromised.

Compliance

Many regulations require apps to protect sensitive data. A keystore helps you meet compliance standards.

Encryption

If your app uses encryption, the keystore provides a secure location for storing encryption keys.

Credentials

If your app communicates with APIs or services that require credentials, using a keystore can prevent those credentials from being easily extracted.

How to Create a Keystore in Flutter

To create a Keystore, you typically use the keytool utility that comes with the Java Development Kit (JDK):

keytool -genkey -v -keystore my-release-key.jks -keyalg RSA -keysize 2048 -validity 10000 -alias my-key

Sample output for generate a keystore file

How to Generate APK or AAB in Flutter

APK (Android Package) and AAB (Android App Bundle) are distribution formats used to package and distribute Android applications.

It is recommended to build an app as AAB over APK due to the following benefits:

• Smaller downloads and faster installations due to optimized APKs
• Improved efficiency in resource usage and reduced storage consumption
• Dynamic feature delivery support for delivering features on-demand
• Enhanced optimization for specific device configurations

flutter build appbundle
or
flutter build apk

Sample output to build abb

We cannot directly install an Android App Bundle (AAB) file on a mobile device.

An AAB is not an executable package like an APK (Android Package) file that can be directly installed on a device.

Instead, the AAB is a publishing format designed for optimized delivery on the Google Play Store.

When you publish your app on the Google Play Store using an AAB, the Play Store uses it to generate APKs that are tailored to each user's device configuration.

This dynamic generation allows the Play Store to deliver only the necessary resources and code for a specific device, reducing the app's size and improving performance during installation.

How to Release a Flutter App on Google Play Store

To release our app on the Google Play Store, we'll require a Google Play developer account. We can create an account by visiting this link.

Please be aware that there is a registration fee of at least $25 associated with the process.

Google play dashboard to create a new App

Click on the Create app button in the All apps section. Enter the app details in the "Create app" form.

Creating an app in Google play for release

After creating the app, we have to go through multiple tasks to release the application.

Steps to testing the app in Goole Play Console

Steps to setting up the app in Goole Play Console

Steps to release the app in Goole Play Console

However, all these steps will be self-explanatory.

At each step, you'll be prompted for information about your application, advertisements, privacy policy, and more.

It's important to note that you can proceed to publish only after completing these steps.

Make sure to mention everything clearly. Especially the questions about data collection. Because, at any time if Google found that your prescribed information is wrong, your app will not be published or will be taken out, if it's published.

After completing these steps, we have to send the app for review. Our app will be published once the review is completed from Google Play team.

It's important to note that there's a higher chance of the app getting rejected if it has security or advertising-related issues.

Therefore, make sure to thoroughly review your app to ensure that it doesn't have any potential problems in data collection. This will contribute to a smoother and more successful app release experience.

Conclusion

Throughout this tutorial, we've covered the process of changing the app icon and building a Flutter app for Android.

While this example provides a foundational understanding of Flutter app development, it's important to note that real-world app complexities may require additional custom configurations in Android.

For further insights, I encourage you to explore the official documentation, which offers comprehensive guidance on more advanced aspects.

I hope this tutorial has provided you with valuable insights in your Flutter journey.

Thank you for investing your time, and best of luck in your app development endeavors!

If you wish to learn more about Flutter, subscribe to my article by visiting my site which has a consolidated list of all my blogs.

Cheers!

But let's take a step back and explore something more elementary yet equally intriguing: addition! Yes, you read that right – addition.

In this tutorial, we'll build a neural network that can learn the art of adding two numbers together.

A quick note before we get into this: I don't recommend using machine learning to find the sum of two numbers in practice. I tried this out of curiosity when I started learning machine learning. And I just wished to share this with you all to make learning fun.

You can use this tutorial as a sort of starter guide in your machine learning journey. Sometimes its hard to find good, clean datasets as a beginner machine learning engineer. This makes it harder to work on and learn about machine learning problems if you don't have a solid dataset.

But don't worry – in this tutorial, we'll be creating our own dataset (pairs of numbers to add) and cleaning the data. So it'll give you a good dataset you can use in your own problems and with your own models.

Alright, before diving in, let's brush up on some machine learning and deep learning basics.

Deep Learning Basics

There are a few machine and deep learning terms which I'll be using in the exercise. So, it's better to understand them on a high level in a couple of sentences before diving in.

What is a Neural Network?

A neural network is a computational model inspired by the structure and functions of the human brain. It consists of interconnected nodes (neurons) organized in layers. Neural networks are trained on data to learn patterns and make predictions.

What is an Activation Function?

An activation function is applied to the output of a neuron to introduce non-linearity. It allows neural networks to learn complex relationships in data. Common activation functions include ReLU (Rectified Linear Unit) and Sigmoid.

What is a Loss Function?

A loss function is a measure of how well a model's predictions match the true target values. During training, the goal is to minimize the loss function, guiding the model to make better predictions.

What is Gradient Descent?

Gradient descent is an optimization algorithm used to minimize the loss function. It adjusts the model's parameters iteratively in the direction of steepest descent, guided by the gradients of the loss function with respect to the parameters.

What is Backpropagation?

Backpropagation is a fundamental algorithm used in training neural networks. It calculates the gradients of the loss function with respect to each model parameter and propagates them backward through the network to update the weights during gradient descent.

What is Batch Size?

Batch size represents the number of training samples used in one forward/backward pass during training. Larger batch sizes can speed up training but require more memory.

What is an Epoch?

An epoch represents one complete iteration through the entire training dataset during training.

These are just a few of the many terms you'll encounter in the vast field of machine and deep learning. But they're enough to help you understand the following exercise.

Prerequisites

Here's a checklist to help you get started with machine learning basics. Before going through this tutorial, you should have them installed and ready (but it's not mandatory).

1. Install Anaconda (it's packaged with many default machine learning libraries).
2. Create an environment in Anaconda: This is highly recommended, because only the created environment will be affected if something goes wrong. Your entire Anaconda installation will not be affected.
3. Make sure you have a good code editor/IDE like Visual Studio Code.
4. Install Keras (this requirement is specific to this exercise).

Do you have all these items ready? Hope you're excited. Let's jump into our exercise.

How to Sum Two Numbers using Machine Learning

Create a folder and a file

Create a new folder with any name. Navigate into the folder and create a file named addition.ipynb. Open the folder in Visual Studio Code or whatever IDE you're using.

Next, create code blocks for each of the following sections by pressing the "+ Code" button at the top left in VS Code.

Import libraries

Import the numpy and keras libraries with these commands:

import numpy as np
from keras.models import Sequential
from keras.layers import Dense

Prepare the data

The accuracy of your machine learning models depends on the data with which you train your model.

To create the addition data we'll need here, let's create pairs of 1000 random numbers which will be considered our input. The output will be the sum of the numbers in each pair.

num_samples = 1000
X_train = np.random.rand(num_samples, 2)
y_train = X_train[:, 0] + X_train[:, 1]

Define the neural network

Let's build a neural network with two input layers – one hidden layer with 8 neurons, and an output layer with a single neuron. We'll use the "relu" activation function.

model = Sequential()
model.add(Dense(8, input_shape=(2,), activation='relu'))
model.add(Dense(1))

Compile the model

Compile the model using the MSE (Mean Squared Error) as the loss function and Adam optimizer.

model.compile(loss='mse', optimizer='adam')

Train the model

Train the model for 100 epochs, with a batch size of 32.

batch_size = 32
epochs = 100
model.fit(X_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1)

This may take few seconds depending on your CPU configuration. It consumed around 10 to 15 seconds in my laptop to complete.

Test the model

Now that we have trained our model, let's test it with a few custom inputs. I have taken two inputs but you can test your model with any number of inputs.

test_input = np.array([[1, 2], [0.3, 0.4]])
predicted_sum = model.predict(test_input)

Print the values

The prediction is complete. Let's see if they're right by printing the predicted values:

print("Predicted sums:")
print(predicted_sum)

Sample output of predicted values

Pretty close, right?

Conclusion

In this tutorial, you learned how to build a neural network to perform addition.

If you're curious, though, you can try building a neural network to perform subtraction just for fun. Good luck :)

Hope you enjoyed reading this article. If you wish to learn more about artificial intelligence / machine learning / deep learning, subscribe to my article by visiting my site which has a consolidated list of all my blogs.

Flutter, Google's UI toolkit for building natively compiled applications for mobile, web, and desktop, offers a powerful animation system that allows developers to create stunning custom animations.

In this step-by-step guide, we will explore how to build beautiful custom animations in Flutter to take your app's UI to the next level.

Prerequisites

Before we start, make sure you have Flutter installed on your system. It's also useful to have a basic understanding of the framework's fundamental concepts, such as widgets, state management, and gesture handling.

Finally, but most importantly, muster up a small spark of interest in learning animation! :) Because once you see the widgets come to life with animation magic, that spark's gonna grow into a blazing inferno of excitement.

In this guide, we are going to see how to implement animation in two kinds of tasks:

1. Animated List
2. Animated Loader

We are going to create a simple Todo app with Animated List and Animated Loader. So, saddle up, and let's ride into the world of animating lists and loaders in Flutter. 🤠🐴.

How to Build an Animated List in Flutter

First, we will create a simple Flutter list with animation. Animated List is a Flutter widget that allows developers to create dynamic and animated lists with smooth and visually appealing transitions. It is part of the Flutter animation framework and is an extension of the ListView widget.

The Animated List automatically animates changes to the list's content, such as inserting or removing items, providing an engaging and interactive user experience.

Key Features

Insertion and Removal Animations

When you add or remove items from the list, the Animated List animates these changes with predefined or custom animations, making the list's modifications visually seamless.

Built-in Animation Controllers

Animated List comes with built-in animation controllers that handle the timing and ease curves of the animations, simplifying the process of creating smooth and fluid transitions.

Customizable Animations

While Animated List provides default animations, developers can also customize the animations to fit the app's unique visual style and requirements.

Now, while theory is essential, practical examples bring concepts to life. So, let's dive into a practical example of using an Animated List in Flutter.

Project Setup and Dependencies

To create our Flutter app, we'll use Visual Studio Code as our development environment.

If you're not familiar with setting up a new Flutter project, don't worry – you can refer to my previous blogs for step-by-step instructions. If you're already comfortable creating Flutter projects, skip this part and proceed with the app development.

No need to install any external plugin to create an Animated List.

How to Create a Task Model

In this blog, we will focus more on animation so we can keep the functionalities simple. Define a Task class that represents a single task with a title and status.

I created a file called todo_list.dart in the lib folder that will contain the Animated List. First I created a simple class that represents a Task with title and status describing whether it's completed or not.

todo_list.dart

class Task {
  String title;
  bool isCompleted;
  Task(this.title, this.isCompleted);
}

How to Create a Simple List with Animation

Technically our goal is to create a scrolling container that animates items when inserted or removed. This widget's AnimatedListState can be used to dynamically insert or remove items.

To refer to the AnimatedListState either provide a GlobalKey or use the static of method from an item's input callback.

To do that, let's create a StatefulWidget to create an Animated List.

todo_list.dart

// todo_list.dart
import 'package:flutter/material.dart';
void main() {
  runApp(TodoListApp());
}
class Task {
  String title;
  bool isCompleted;
  Task(this.title, this.isCompleted);
}

class TodoListApp extends StatefulWidget {
  @override
  _TodoListAppState createState() => _TodoListAppState();
}

class _TodoListAppState extends State<TodoListApp> {
  List<Task> tasks = [];
  bool isLoading = false;

  final GlobalKey<AnimatedListState> _animatedListKey = GlobalKey();

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        appBar: AppBar(title: const Text('ToDo List')),
        body: AnimatedList(
          key: _animatedListKey,
          initialItemCount: tasks.length,
          itemBuilder: (context, index, animation) {
            return _buildTaskItem(tasks[index], animation, index);
          },
        ),
        floatingActionButton: FloatingActionButton(
          onPressed: _addTask,
          child: const Icon(Icons.add),
        ),
        backgroundColor: Colors.white60,
      ),
    );
  }

  Widget _buildTaskItem(Task task, Animation<double> animation, int index) {
    return SizeTransition(
        sizeFactor: animation,
        child: Card(
          color: Colors.white,
          child: ListTile(
            title: Text(task.title),
            onLongPress: () => _removeTask(index),
          ),
        ));
  }

  void _addTask() async {
    Task newTask = Task('New Task ${tasks.length + 1}', false);
    tasks.add(newTask);
    _animatedListKey.currentState!.insertItem(tasks.length - 1);
  }

  void _removeTask(int index) async {
    _animatedListKey.currentState!.removeItem(index,
        (context, animation) => _buildTaskItem(tasks[index], animation, index));
    tasks.removeAt(index);
  }
}

Here, we used AnimatedList (a default Flutter package ). The AnimatedList class in Flutter is a powerful widget that allows us to create dynamic and animated lists with smooth transitions. It is an extension of the ListView widget, providing built-in animation support for adding, removing, and updating items in the list.

The primary goal of AnimatedList is to enhance the user experience by animating changes to the list's content, making the app feel more interactive and visually engaging.

AnimatedList has multiple properties or parameters that control various aspects of the widget's behavior, appearance, and animations. Understanding and using these props correctly is crucial to achieving the desired behavior and visual effects in an AnimatedList.

To know about the properties and behavior of AnimatedList please refer to the official documentation.

Simple Animated List in Flutter

In this tutorial, we will be focusing on Animated List and Animated Loader. If you are not familiar with Flutter basics (like widgets, states, and so on) I would recommend that you read my previous tutorial.

How to Build the Animated Loader

A loader is commonly used to provide visual feedback to users while waiting for data to load, processing content, or completing network requests. Loaders help improve the user experience by giving a sense of activity and preventing the app from appearing unresponsive during waiting periods.

There are various ways to implement loaders in Flutter, including using built-in widgets, third-party packages, or creating custom loaders. Additionally, "animated loaders" add an extra touch of dynamism to the loading process by incorporating smooth animations.

Let's add an animated loader in our TodoApp while creating and deleting a task.

How to Add the Flutter Loader Package

You'll need to install the package with the following command:

flutter pub add loading_animation_widget

Then you should see the following screen:

Installing loading animation package in Flutter

Now your package should be ready for use.

How to Implement AnimatedLoader

The loading_animation_widget package offers various loader animations that we can use to display loading indicators in our app. By importing the package, we gain access to these loader animations and can utilize them to enhance the user experience during loading operations or any other asynchronous tasks.

All loading animation APIs follow the same straightforward implementation. There is a static method for each animation inside the LoadingAnimationWidget class, which returns the Object of that animation. Both size and color are required some animations need more than one color.

loading_animation_widget offers multiple animated loaders with customized animation. Let's explore a few of those and integrate into our Todo App.

Now I created a file called animated_loader.dart, which has AnimatedLoader Widget.

// animated_loader.dart
import 'package:flutter/material.dart';
import 'package:loading_animation_widget/loading_animation_widget.dart';

void main() {
  runApp(AnimatedLoader());
}
class AnimatedLoader extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: MyLoaderScreen(),
    );
  }
}
class MyLoaderScreen extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: Center(
        child: LoadingAnimationWidget.staggeredDotsWave(
            size: 75, color: Colors.deepPurple),
      ),
      backgroundColor: Colors.transparent,
    );
  }
}

Let's show this animated loader in our animated list while adding or removing a task.

todo_list.dart

import 'package:flutter/material.dart';
import 'package:flutter_animation/animated_loader.dart';
import 'package:loader_overlay/loader_overlay.dart';

void main() {
  runApp(TodoListApp());
}

class Task {
  String title;
  bool isCompleted;
  Task(this.title, this.isCompleted);
}

class TodoListApp extends StatefulWidget {
  @override
  _TodoListAppState createState() => _TodoListAppState();
}

class _TodoListAppState extends State<TodoListApp> {
  List<Task> tasks = [];
  bool isLoading = false;

  final GlobalKey<AnimatedListState> _animatedListKey = GlobalKey();

  Future<void> loadData() async {
    setState(() {
      isLoading = true;
    });
    await Future.delayed(const Duration(seconds: 2));
    setState(() {
      isLoading = false;
    });
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        appBar: AppBar(title: const Text('ToDo List')),
        body: Stack(
          children: [
            AnimatedList(
              key: _animatedListKey,
              initialItemCount: tasks.length,
              itemBuilder: (context, index, animation) {
                return _buildTaskItem(tasks[index], animation, index);
              },
            ),
            if (isLoading)
              const Opacity(
                opacity: 0,
                child: ModalBarrier(dismissible: false, color: Colors.black),
              ),
            if (isLoading)
              Center(
                child: Center(child: AnimatedLoader()),
              ),
          ],
        ),
        floatingActionButton: FloatingActionButton(
          onPressed: _addTask,
          child: const Icon(Icons.add),
        ),
        backgroundColor: Colors.white60,
      ),
    );
  }

  Widget _buildTaskItem(Task task, Animation<double> animation, int index) {
    return SizeTransition(
        sizeFactor: animation,
        child: Card(
          color: Colors.white,
          child: ListTile(
            title: Text(task.title),
            onLongPress: () => _removeTask(index),
          ),
        ));
  }

  void _addTask() async {
    Task newTask = Task('New Task ${tasks.length + 1}', false);
    await loadData();
    tasks.add(newTask);
    _animatedListKey.currentState!.insertItem(tasks.length - 1);
  }

  void _removeTask(int index) async {
    await loadData();
    _animatedListKey.currentState!.removeItem(index,
        (context, animation) => _buildTaskItem(tasks[index], animation, index));
    tasks.removeAt(index);
  }
}

To explore more about the Loader you can refer to the docs. It has multiple loaders with customization options.

Animated List with Animated Loader in Flutter

Hurray! We can see that the Animation Loader and Animation List are rendering very smoothly, and they look even better.

If you are implementing this from scratch, that's great and it'll really help you learn. But if you prefer, you can also clone the repo from GitHub. Either way, I'm excited to see the Animation Loader and Animation List in action from you.

Note: In the context of creating a task in real time, a loader might not be necessary since the creation of a task typically happens quickly and doesn't involve any lengthy processes like fetching data from an API or performing complex computations. But adding a loader during task creation can still be a useful visual cue to indicate that the task is being processed and provide immediate feedback to the user.

Conclusion

We explored the world of creating beautiful custom animations in Flutter, focusing on the implementation of Animated List and Animated Loader. By understanding the Animated List, we learned how to create dynamic and interactive lists with smooth item insertions and removals.

Through these examples, you saw Flutter's ability to make animation implementation enjoyable and straightforward. By incorporating custom animations into your apps, you can create engaging and visually appealing interfaces that captivate users and set your apps apart.

If you wish to learn more about Flutter, subscribe to my email newsletter (https://5minslearn.gogosoon.com/) and follow me on social media.

Happy Animating and Fluttering! 🚀

In this article, we'll dive into the Serverless Stack Toolkit (SST), a framework for building serverless applications. We'll deploy a Next.js application and set up a custom domain, all without visiting the AWS console.

Let's begin this journey!

What Does Serverless Mean?

The term "serverless" refers to a cloud computing model where developers can build and deploy applications without the need to manage servers. In a serverless architecture, the cloud provider handles server provisioning, scaling, and maintenance. This allows developers to focus solely on writing code for their applications.

With serverless, developers are billed based on actual usage rather than fixed server costs, making it a cost-effective and scalable solution. It offers increased flexibility and agility, as resources are automatically allocated and released based on demand. This eliminates the need for developers to worry about infrastructure management.

Now that we have a good idea of what serverless means, let's see what the Serverless Stack Toolkit (SST) is.

Understanding Serverless Stack Toolkit (SST)

The Serverless Stack Toolkit, or SST in short, is a flexible, open-source framework designed to enable faster development and reliable deployment of serverless applications on AWS.

It aims to make it easier for developers to define their application's infrastructure using AWS CDK (Cloud Development Kit).

You can use it to test applications in real-time with Live Lambda Development, debug code in Visual Studio Code, manage applications through a web-based dashboard, and deploy to multiple environments and regions seamlessly.

Benefits of Using SST

Here are some benefits of using the SST stack:

Infrastructure as Code

With SST, developers can define their application's infrastructure programmatically using AWS CDK. This improves version control and collaboration among team members.

Efficient Testing and Debugging

SST enables live Lambda development, making it easier to test and debug serverless applications locally before deployment to AWS. This reduces potential issues and ensures smoother deployment.

Simplified Deployment

SST simplifies the deployment process, allowing developers to deploy applications to multiple environments and regions effortlessly.

Language Flexibility

SST supports multiple programming languages, including JavaScript, TypeScript, Go, Python, C#, and F#, providing developers with the flexibility to use their preferred language for building serverless applications.

Now that we have understood what SST is and some of its benefits, let's see the power of SST in action.

How to Configure AWS

Before we add SST we have to configure some AWS credentials. To do that, type the below command in your terminal:

aws configure

AWS Configure

You'll be required to enter your AWS Access Key ID, Secret Access Key, Region name and output format. If you don't have these keys, please create an IAM user and enter the credentials.

How to Add SST to Your Next.js App

We can use SST in an existing Next.js app in drop-in mode or inside a monorepo app in standalone mode.

In this article, we'll create new Next.js project and add SST which follows drop-in mode installation using the commands below:

yarn create next-app
cd my-app
yarn create sst
yarn install

Note: You should ensure that you have the index.tsx file inside the /pages folder. Without the file, you'll get errors while deploying your app using SST. You don't need to make any changes to this file.

Folder structure

Once you run the above commands, SST will create two new files —sst.config.ts and sst-env.d.ts

We have to define all our infrastructure and stacks in the sst.config.ts file.

You can use these commands to run the app locally:

# Start SST locally
yarn sst dev

# Start Next.js locally
yarn dev

On executing the yarn sst dev command, you'll be asked to enter the stage name. Please enter your environment name. I'll use dev for this project's stage name.

Start SST locally

Just sit back and watch. It will automatically create the necessary IAM roles, permissions and CloudFormation stacks.

SST - Creating the neccessary IAM roles, permissions and stack

Notice in the image above that you can see the Console URL, https://console.sst.dev/sst-demo/dev. With the Console URL, you can view real-time logs, invoke functions, replay invocations, make queries, run migrations, view uploaded files, query your GraphQL APIs, and more!

Just awesome right? I would recommend you to visit the official documentation to learn more about the services they offer.

Next, start the Next.js site by running yarn dev. You should see the default page after that.

Next.js default page

Our Next.js app is now ready to be deployed to AWS! Just run the following command and see the magic.

yarn sst deploy --stage prod

OpenNext building the Next.js App

It will automatically start building the app using OpenNext , deploy it to AWS using CDK, and output the CloudFront URL. Click on the link and you should be able to see your app up and running.

SST - Deployed changes and outputs CloudFront url

The Next.js app up and running

How to Create Infrastructure using SST

To create an infrastructure, we simply need to edit sst.config.ts and import any AWS services like S3 bucket, RDS, API Gateway, and so on from sst/constructs

Let's add a simple S3 file upload feature. Open sst.config.ts file and add the code below:

import { SSTConfig } from "sst";
import {Bucket, NextjsSite } from "sst/constructs";

export default {
  config(_input) {
    return {
      name: "sst-tutorial",
      region: "us-east-1",
    };
  },
  stacks(app) {
    app.stack(function Site({ stack }) {
      const bucket = new Bucket(stack, "public");
      const site = new NextjsSite(stack, "site",{
        bind:[bucket],
      });
      stack.addOutputs({
        SiteUrl: site.url,
      });
    });
  },
} satisfies SSTConfig;

Here, we're creating a new public S3 bucket and binding it with our NextjsSite.

Let's edit our index page to add file upload feature.

How to Upload Files to S3 using SST

To upload a file to S3, we need to generate a pre-signed URL. To do that, we need to add this package @aws-sdk/s3-request-presigner in our repo.

yarn add @aws-sdk/s3-request-presigner

Open index.tsx file and create a function called getServerSideProps above the Home function, as shown in the below code snippet.

...
import { Bucket } from "sst/node/bucket";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
...
export async function getServerSideProps() {
  const command = new PutObjectCommand({
    ACL: "public-read",
    Key: crypto.randomUUID(),
    Bucket: Bucket.public.bucketName,
  });
  const url = await getSignedUrl(new S3Client({}), command);
  const bucketName = Bucket.public.bucketName
  console.log(bucketName)
  return { props: { url } };
}

Update the Home() function with the following code.

import styles from "@/styles/Home.module.css";
export default function Home({ url }: { url: string }) {
  return (
    <main className={styles.main}>
     <div className={styles.center}>
        <a
          href="https://5minslearn.gogosoon.com/?ref=github_sst_app"
          className={styles.card}
          target="_blank"
          rel="noopener noreferrer"
        >
          <h2 className={inter.className}>
            5minslearn <span>-&gt;</span>
          </h2>
          <p className={inter.className}>Learn tech in 5mins</p>
        </a>
      </div>
      <form
        className={styles.form}
        onSubmit={async (e) => {
          e.preventDefault();

          const file = (e.target as HTMLFormElement).file.files?.[0]!;

          const image = await fetch(url, {
            body: file,
            method: "PUT",
            headers: {
              "Content-Type": file.type,
              "Content-Disposition": `attachment; filename="${file.name}"`,
            },
          });

          window.location.href = image.url.split("?")[0];
        }}
      >
        <input name="file" type="file" accept="image/png, image/jpeg" />
        <button type="submit" className={inter.className}>
          Upload
        </button>
      </form>
    </main>
  );
}

I added an input with a file type and a button for submitting the form. The selected image will be uploaded to S3 when the form is submitted. It's time to deploy the changes.

To deploy the changes, run the yarn sst deploy command.

Once deployed you'll see a page like this:

Next.js up and running with updated changes

Now you can upload any image and check your S3. The selected file will be uploaded to your S3 bucket.

Great, we have successfully deployed the changes. But we still have the random URL generated by CloudFront which may be difficult to memorize for humans. Let's configure a custom domain.

How to Configure Custom Domains

To configure custom domains, we need a valid domain or sub-domain. You can create one using Route 53 or your preferred domain provider like GoDaddy, Namecheap, and so on.

If you have a domain on an external DNS provider, you'll need to create an SSL certificate on AWS Certificate Manager (ACM).

I have my domain on Cloudflare. If you have yours with other providers like Namecheap or GoDaddy, then the steps below should still work for you.

How to Point CNAME to CloudFront

1. Login into your DNS provider.
2. Add a CNAME. In my case, I used aws as the name because my domain is aws.gogosoon.com, and target as the CloudFront URL without https.

We've successfully pointed our CNAME to CloudFront. Now let's create an SSL certificate for our domain.

How to Create ACM Certificate

ACM certificates are managed SSL/TLS certificates that can be used with a variety of AWS services, including CloudFront.

However, there is a specific requirement for using ACM certificates with CloudFront: the certificate must be created in the US East (N. Virginia) region (us-east-1). The reason for this is that CloudFront has all of its provisioning/administrative infrastructure based in us-east-1.

Quoting from their documentation:

To use a certificate in AWS Certificate Manager (ACM) to require HTTPS between viewers and CloudFront, make sure you request (or import) the certificate in the US East (N. Virginia) Region (us-east-1).

Here are the steps to follow to create an ACM:

1. Login into AWS console.
2. Search for certificate manager, switch to us-east-1 and click on "Request Certificate" in the sidebar.

AWS ACM - Request Certificate

3. Enter the domain name you pointed to in your DNS provider configuration. Under "Validation method", select "Email validation" and click next.

AWS ACM - Request Certificate

4. A certificate with the status of "Pending Validation" will be created. You'll receive an email from AWS with a link to validate the request.

ACM certificate with pending status

5. Once you click on the link in the email, the status of the certificate will be changed to "Issued". Copy the ARN – we'll need it in the next steps.

AWS ACM certificate issued

Now that we've created the certificate successfully, let's create the alternate domain for CloudFront.

How to Create an Alternate Domain for CloudFront Distribution

1. Log into the AWS Console and search for CloudFront.
2. Click on the distribution created by SST.
3. In the "General" tab, click the "Edit" button.

Edit CloudFront distribution

4. Enter the alternate domain name and select the certificate that we created. Leave all other options as default and click on the "Save changes" button.

Add alternate domain for CloudFront distribution

All set! Let's edit our app to deploy the changes to our custom domain.

How to Configure External Custom Domain using SST

Update the sst.config.ts file with the following code. Paste the ARN you copied while creating the certificate as a value for the variable certArn. Replace the domainName with your domain:

import { SSTConfig } from "sst";
import {Bucket, NextjsSite } from "sst/constructs";
import { Certificate } from "aws-cdk-lib/aws-certificatemanager";


export default {
  config(_input) {
    return {
      name: "sst-tutorial",
      region: "us-east-1",
    };
  },
  stacks(app) {
    app.stack(function Site({ stack }) {
      const bucket = new Bucket(stack, "public");
      const certArn = 'Paste the certificate arn'
      const site = new NextjsSite(stack, "site",{
        bind:[bucket],
        customDomain: {
          isExternalDomain: true,
          domainName: "aws.gogosoon.com",
          cdk: {
            certificate: Certificate.fromCertificateArn(stack, "MyCert", certArn),
          },
        },
      });
      stack.addOutputs({
        SiteUrl: site.customDomainUrl || site.url,
      });
    });
  },
} satisfies SSTConfig;

sst.config.ts - File changes

Run yarn sst deploy to deploy the changes to a custom domain. Once deployed, you should have the app running on the custom URL.

Next.js deployed with custom domain using SST

Next.js app up and running with custom domain

Conclusion

Voila! Our Next.js app is now deployed to AWS, and we've connected it with our custom domain. Please check out the source code here.

The SST framework provides an excellent toolset for deploying serverless applications, contributing significantly to development speed, scalability, and error handling.

Feel free to explore more about SST and its potential in transforming your cloud development experience. Happy coding!

If you wish to learn more about AWS Services, subscribe to my email newsletter (https://5minslearn.gogosoon.com/) and follow me on social media.

spaCy excels at large-scale information extraction tasks. It's written from the ground up in carefully memory-managed Cython.

spaCy is designed to help us build real products, or gather real insights. It's built with 73+ languages, and supports custom models built with Pytorch and Tensorflow. It's robust and has rigorously evaluated accuracy.

You may not know much about Cython. So let's have a quick look at it.

What is Cython?

Cython is a Python compiler that makes writing C extensions for Python as easy as Python itself. Cython is based on Pyrex, but supports more cutting edge functionality and optimizations.

To put in simple words, it's a Python to C compiler.

Quoting from Wikipedia,

Cython is a programming language, a superset of the Python programming language, designed to give C-like performance with code that is written mostly in Python with optional additional C-inspired syntax.

So you might be wondering if you need to learn Cython to help you fine-tune your spaCy models.

Well, don't worry. You don't need to learn Cython to to work with spaCy. I just wanted to make sure you're aware of what it is to help you get the most out of this tutorial.

Prerequisites

Basic knowledge of spaCy

The official documentation site of spaCy provides a lot of information about the tool. Alternatively, you can read my other tutorial which gives some basic information about spaCy.

Basic knowledge of how to gather data

In order to fine-tune any model, you need to have the data ready. And it should be good quality data.

In this tutorial, let's assume we've built a event management software. We want to add voice assistance to our software. We have a module that converts voice input into text. Our next step is to process this text and extract data from the given sentence using spaCy.

We have to gather some basic sentences that we hear from people trying to schedule a event. Here are a few examples:

1. Schedule event for visit to Trivandrum on July 18
2. Create event happening tomorrow on AI
3. Schedule Pongal celebration event in Oaks HOA at June 20, 2023

Similarly, we have to collect prompts related to event scheduling. The more data you collect and input, more accurate your model will be.

I created 7 sentences, which is much too small for a event management software company to train its model. But from a demo standpoint, it should be enough.

How to Pre-Process the Data

Collecting data covers just one part of the equation. We need to pre-process the data and transform it in a way that spaCy can easily understand. We should also define what kind of data (tags) should be identified from the given sentences.

Let's take the following sentence as an example:

"Schedule event for visit to Trivandrum on July 18".

Let's try to split out some tags from above sentence:

• Schedule – this belongs to the "action" tag
• event – this belongs to the "domain" tag
• visit to Trivandrum – this belongs to the "name" tag
• July 18 – this belongs to the "date" tag

Every tag defined above may contain alternatives in other sentences. For an example, we may input the following sentences:

1. Cancel client meeting scheduled tomorrow
2. Change time of mall visit to 6 PM

From the above sentences, the action tags are "Cancel" and "Change". Similarly data for each tag may vary for each sentence.

Our next step is to teach spaCy about the words for each tag. We need to prepare a JSON file that contains examples with the tags and their indices.

For example, in the above sentence ("Schedule event for visit to Trivandrum on July 18"), the index for the "action" tag starts from 0 (indices always start from 0 in Python) and ends at 7.

Similarly for all 7 sentences I've chosen, I've prepared the index for each tag and created the JSON file.

How to Fine-Tune the spaCy Model

Let's try to fine-tune spaCy with the data that we have.

Create a folder and download the above JSON file and place it into the folder. Create a new file named custom_model.ipynb.

All the following sections below need a code block to be created. Create a code block wherever you see a heading. Here's a sample screenshot.

Creating and running blocks of code

Import spaCy

import spacy

Load the pre-trained model

nlp = spacy.load("en_core_web_lg")

Import the JSON file

Import the above downloaded JSON file.

import json

with open('./event_schedule_data.json', 'r') as f:
    data = json.load(f)

Convert the data

Convert the data read from JSON file into tuple of dictionaries containing original text and entities.

training_data = []
for example in data['examples']:
    temp_dict = {}
    temp_dict['text'] = example['content']
    temp_dict['entities'] = []
    for annotation in example['annotations']:
        start = annotation['start']
        end = annotation['end'] + 1
        label = annotation['tag_name'].upper()
        temp_dict['entities'].append((start, end, label))
    training_data.append(temp_dict)
print(training_data[0])

The above code will convert the data to the required format and print the first dictionary in the tuple, which will look something like below:

{'text': 'Schedule a calendar event in Teak oaks HOA about competitions happening tomorrow', 'entities': [(0, 8, 'ACTION'), (11, 25, 'DOMAIN'), (29, 42, 'HOA'), (49, 71, 'EVENT'), (72, 80, 'DATE')]}

Import training libraries

from spacy.tokens import DocBin
from tqdm import tqdm
from spacy.util import filter_spans

nlp = spacy.blank('en')

Train the model

The below code will create a custom model with the data that we give. A binary file named train.spacy will be generated at the end.

doc_bin = DocBin()
for training_example in tqdm(training_data):
    text = training_example['text']
    labels = training_example['entities']
    doc = nlp.make_doc(text)
    ents = []
    for start, end, label in labels: 
        span = doc.char_span(start, end, label=label, alignment_mode="contract")
        if span is None:
            print("Skipping entity")
        else:
            ents.append(span)
    filtered_ents = filter_spans(ents)
    doc.ents = filtered_ents
    doc_bin.add(doc)

doc_bin.to_disk("train.spacy")

Create config files

Create a new file named base_config.cfg and copy the below code into it.

Create another file named config.cfg and copy the below code into it.

Don't worry. These are default configurations that I've taken from their official documentation and I've not made any changes to it.

Initialize spaCy with the config files

Run the following command in the notebook code block to initialize spaCy with the config file. This config file will be used to train the spaCy model with our generated custom model.

!python -m spacy init fill-config base_config.cfg config.cfg

Train spaCy model

Run the following command to train the spaCy model:

!python -m spacy train config.cfg --output ./ --paths.train ./train.spacy --paths.dev ./train.spacy

This may take some time depending on your system configuration. Ideally not too long (around 5 to 10 minutes). At the end, it'll generate 2 folders named model-best and model-last.

Load the best model

nlp_ner = spacy.load("model-best")

Test our model

Let's test our model with the following input.

"Could you please reserve a team brainstorming session on coming Wednesday at 11 AM?"

doc = nlp_ner("Could you please reserve a team brainstorming session on coming Wednesday at 11 AM?")

spacy.displacy.render(doc, style="ent")

You should be surprised to see the output.

Entity representation of our input sentence

That's great right?

Eventually, you may have a question: as a programmer, how can I get this data in my backend code?

Well, that's something everyone asks.

spaCy has an answer for it. You can expose the above data as JSON.

Convert extracted data to JSON

json_obj = doc.to_json()
json_obj

This will show a similar output like the one below.

JSON of our test sentence input

Write a REST API and expose this data as JSON. That's it. But remember, spaCy will give you only the indices, you have parse your sentence and extract words in between those indices.

Conclusion

In this article, we learnt about how to customize and fine-tune a pre-trained spaCy model with the data that corresponds to our domain knowledge.

Similarly you can also train with your domain specific data. The model that you fine-tune will be private to you unless you expose it to the public. So it's best suited for training with the domain data that is not publicly available.

If you wish to learn more about NLP/Machine Learning, subscribe to my email newsletter (https://5minslearn.gogosoon.com/) and follow me on social media.

Firebase Cloud Messaging (FCM) is a powerful push notification service provided by Firebase. It offers a seamless way to connect with your app's users and keep them engaged.

In this tutorial, we will delve into the integration of FCM in Flutter. We'll explore its benefits and showcase real-world examples of how it can enhance user engagement and improve app performance.

What is Firebase Cloud Messaging?

Firebase Cloud Messaging (FCM) provides a reliable and battery-efficient connection between your server and devices. It allows you to deliver and receive messages and notifications on iOS, Android, and the web at no cost.

In this tutorial, we will explore the process of setting up and using Firebase Cloud Messaging (FCM) in Flutter using Firebase as the backend service. While the main focus will be on Android implementation, it's worth noting that the process is similar for iOS and Android (with a few configuration differences).

Here is what we'll cover:

1. How to create an app in Firebase
2. How to set up Firebase in Flutter
3. How to implement push notifications using FCM tokens

In this tutorial, you'll learn how to send a simple notification using Firebase to the app running in Flutter. Let's get started.

How to Create an App in Firebase

I'll create a new project in the Firebase console to get started. I'll walk through the necessary steps, including project setup, how to configure Firebase Cloud Messaging, and how to get the required credentials and configuration files for our Flutter app.

Before creating the app you need to signup for the Firebase console if you don't have an account. After sign up, try to create a project.

Create a Project in Firebase

It will take a little time to create a project.

Creating project in Firebase

After creating the project, it will redirect you to the project dashboard.

Project Overview in Firebase Console

Once you've created the project in Firebase console, it's time to get started with our Flutter app.

How to Set Up Firebase in Flutter

I have created a simple Flutter project using Visual Studio Code. If you are unfamiliar with building a Flutter project, you can refer to my previous tutorial. (If you are already familiar, you can skip this step.)

Simple Flutter Application running on Android Device

Let's integrate Firebase into our Flutter project. To do this, we need a Firebase CLI command line tool. I have already installed the Firebase CLI. If you haven't done this, you can refer to the official documentation.

Then we need to log in to Firebase using Firebase CLI.

firebase login

Login to Firebase using FirebaseCLI

This will navigate you to the browser to log in to Firebase. You'll be navigated back once the authentication is successfully completed.

After successful login, we need to install FlutterFire CLI. We can use the FlutterFire CLI to configure our Flutter apps to connect to Firebase. Run the following command to activate the FlutterFire CLI:

dart pub global activate flutterfire_cli

The FlutterFire CLI is a command-line interface tool that simplifies the integration of Firebase services into Flutter applications. It provides a convenient way to add, configure, and manage Firebase plugins in our Flutter project.

Installing FlutterFireCLI

The next step is to add firebase_core library to our Flutter project.

The following command will automatically add the firebase_core package as a dependency in your project's pubspec.yaml file and fetch the latest version of the package from pub.dev. After running this command, you can import the firebase_core package into the Dart files and use Firebase services in our Flutter app.

flutter pub add firebase_core

Installing Firebase Core package

The flutterfire configure command is used to configure Firebase services in our Flutter project using the FlutterFire CLI. This command helps us set up Firebase authentication, Firestore, Cloud Messaging, and other Firebase services easily and efficiently.

flutterfire configure

The first step is to choose the project,

Connect Flutter App with Firebase app

The next is to choose the platform. I am using it for Android here, so I choose Android.

Choosing platform

After the successful configuration, the Firebase App Id will be displayed.

Finally, we need to add some code changes to our main.dart file.

Import the following packages:

import 'package:firebase_core/firebase_core.dart';
import 'firebase_options.dart';

Add the following configuration to initialize the Firebase config inside the main function of the main.dart file.

await Firebase.initializeApp(
  options: DefaultFirebaseOptions.currentPlatform,
);

Alright, we have successfully completed the Firebase configuration in our Flutter app! Let's take a moment to celebrate this milestone. Configuring Firebase services is a crucial step in building powerful and feature-rich applications.

How to Implement Push Notification using FCM Tokens

We'll implement the process of registering devices for push notifications and retrieving the unique FCM tokens assigned to each device. This step is crucial for sending targeted notifications to specific devices.

We'll dive into the implementation of sending push notifications to devices using Firebase Cloud Messaging. We'll explore how to structure and send notification messages from the Firebase console and demonstrate how to handle these messages within our Flutter app.

flutter pub add firebase_messaging

Installing firebse messaging Package

Next, we need to trigger the setAutoInitEnabled function to enable automatic initialization of Firebase Cloud Messaging (FCM) in our Flutter app. This means that FCM will automatically initialize and retrieve a device token when the app starts.

Add the following function call in the main method:

import 'package:firebase_messaging/firebase_messaging.dart';
...
...
await FirebaseMessaging.instance.setAutoInitEnabled(true);

Let's run our Flutter app and verify if we receive the notification.

Navigate to the Firebase messaging console. As it is our first message, we need to select "Create your first campaign". Select "Firebase Notification messages" and click "Create".

Sample test messaging template

Now we need to enter the notification title, text, and name for the message.

Then we can get the FCM token manually for testing purposes using the code below. To retrieve the current registration token for an app instance, call getToken() in the main() method. This method will ask the user for notification permissions if notification permission has not been granted. Otherwise, it returns a token or rejects if there's any error.

final fcmToken = await FirebaseMessaging.instance.getToken();
log("FCMToken $fcmToken");

Copy the FCM token printed on the console and paste it into the "Add an FCM registration token" input box.

Sent test message using FCM Token

Click on the Test button. The targeted client device (with the app in the background) should receive the notification.

Received push notification in android device

Hurray! We got the notification on our Android device. If we click on the notification it will open the app by default.

When we tap a notification, the default behavior on both Android and iOS is to open the application. If the application is terminated, it will be started. If it is in the background, it will be brought to the foreground.

Here, we can see the basic configuration to initialize Firebase messaging.

main.dart

import 'package:flutter/material.dart';
import 'package:firebase_core/firebase_core.dart';
import 'package:firebase_messaging/firebase_messaging.dart';
import 'firebase_options.dart';

void main() async {
  runApp(const MyApp());
  await Firebase.initializeApp(
    options: DefaultFirebaseOptions.currentPlatform,
  );
  final fcmToken = await FirebaseMessaging.instance.getToken();
  await FirebaseMessaging.instance.setAutoInitEnabled(true);
  log("FCMToken $fcmToken");
}

Conclusion

In this tutorial we have covered the essential steps for implementing push notifications in Flutter using Firebase Cloud Messaging (FCM).

By following the outlined steps, you can set up Firebase, integrate it into your Flutter project, and implement push notification functionality.

With the ability to send and receive notifications seamlessly, you can enhance the user experience and engage with your app's users effectively. Stay tuned for more advanced topics and features in future tutorials.

If you wish to learn more about Flutter, subscribe to my email newsletter (https://5minslearn.gogosoon.com/) and follow me on social media.

NLP is a sub-field of artificial intelligence. It focuses on enabling machines to understand, interpret, and generate human language.

In this tutorial, we'll explore the fundamental concepts of NLP and we'll look at a particular implementation with spaCy. This will showcase its immense potential to revolutionize various industries.

Let's have a quick look at Natural Language Processing before we begin with spaCy.

What is Natural Language Processing?

NLP involves the intersection of linguistics, computer science, and machine learning. Its primary objective is to bridge the gap between human language and machine understanding.

NLP encompasses a wide range of tasks, including Text Classification, Named Entity Recognition (NER), Sentiment Analysis, and more.

Text Classification

This involves categorizing text into predefined classes or categories based on its content.

This has applications in sentiment analysis, spam detection, topic classification, and more.

Named Entity Recognition (NER)

This involves identifying and extracting named entities such as names, organizations, locations, and dates from text.

NER is crucial for information extraction, question answering systems, and recommendation engines.

Sentiment Analysis

This involves determining the sentiment or emotion expressed in a piece of text, whether it's positive, negative, or neutral.

Sentiment analysis is extensively used for brand monitoring, customer feedback analysis, and social media monitoring.

Challenges in Natural Language Processing

While NLP has made significant advancements, several challenges persist:

1. Human language is inherently ambiguous, making it challenging sometimes for machines to accurately understand and interpret meaning.
2. Different languages, dialects, slang, and cultural nuances add complexity to NLP tasks, requiring models to be language-specific and adaptable.
3. Capturing contextual information and understanding the underlying semantics of text remains a significant challenge for NLP algorithms.
4. NLP models heavily rely on training data, and biased or low-quality data can result in biased or inaccurate predictions, leading to potential ethical concerns.

What is spaCy?

In the world of Natural Language Processing (NLP), spaCy has emerged as a powerful and efficient library, revolutionizing the way developers and researchers work with text data.

spaCy is an open-source Python library designed specifically for NLP tasks such as part-of-speech tagging, named entity recognition, dependency parsing, and more.

It was developed with the goal of providing industrial-strength performance, while still being easy to use and integrate into existing workflows.

spaCy is built on the latest research and implements state-of-the-art techniques, making it an ideal choice for both beginners and experienced NLP practitioners.

Key Features of spaCy

Linguistic Annotations

spaCy provides a wide range of pre-trained models that can quickly analyze text and extract various linguistic features. These features include part-of-speech tags, named entities, syntactic dependencies, sentence boundaries, and more.

The pre-trained models are trained on large corpora and have high accuracy, allowing developers to focus on their specific NLP tasks without worrying about training models from scratch.

Tokenization and Sentence Segmentation

Tokenization is a crucial step in NLP that breaks down text into individual words or subwords. spaCy's tokenization algorithms are highly efficient and language-specific, allowing for accurate and customizable tokenization.

spaCy can also automatically segment text into sentences, making it easy to work with text data at a granular level.

Entity Recognition

Named Entity Recognition (NER) is the task of identifying and classifying named entities such as persons, organizations, locations, dates, and more.

spaCy's NER capabilities are exceptional, providing out-of-the-box support for multiple languages. It allows developers to train custom NER models using their own labeled data, enabling domain-specific entity recognition.

Dependency Parsing

Dependency parsing involves analyzing the grammatical structure of a sentence by determining the relationships between words.

spaCy's dependency parsing is based on efficient algorithms and achieves high accuracy. It provides a rich set of syntactic annotations, including the head of each word, the dependency label, and the subtree structure.

This information is invaluable for tasks like information extraction, question answering, and sentiment analysis.

Customization and Extensibility

One of spaCy's major strengths is its flexibility and extensibility. Developers can easily customize and fine-tune spaCy's models to adapt to specific domains or improve performance on specific tasks.

The library also provides a straightforward API for adding custom components, such as new tokenizers, entity recognizers, or syntactic parsers, making it a versatile tool for research and development.

Performance and Scalability

spaCy is known for its exceptional performance and scalability. The library is implemented in Cython, a programming language that compiles Python-like code into highly efficient C/C++ modules. This allows spaCy to process text data blazingly fast, making it suitable for large-scale NLP applications and real-time systems.

Named Entity Recognition Example in spaCy

Let's try to implement NER using spaCy.

I'll be using Google Colab. Google Colab is a hosted Jupyter Notebook service that requires no setup to use and provides free access to computing resources, including GPUs and TPUs.

You can use Kaggle instead or run it on your own computer if you'd like. Since spaCy is a pre-trained model, it does not required much computing power to get started.

But I'd advise that you setup Anaconda on your machine if you're working on Machine learning problems.

Navigate to https://colab.research.google.com and click on the "New Notebook" button.

Google Colab Console

On the header, enter a name of your file. Ensure your file name ends with .pynb extension.

Change file name and create a code block

Click on the "+ Code" button to create a code block.

By default, Google Colab is packed with some machine tools and Python libraries pre-installed. So, we don't have to worry about installations and getting our development environment ready.

But it doesn't come with the spacy library.

Run the following command inside the code block to install the spacy library.

!pip install -U spacy

Install spacy library

Choose whichever option you want and proceed. The major difference between each one of them is the amount of data it has been trained with.

• Small – en_core_web_sm
• Medium – en_core_web_md
• Large – en_core_web_lg
• Transformer – en_core_web_trf

Our next step is to download one of these models. Add a code block and choose any of the ones from the above list and run the following command. I'll be downloading the large model.

!python -m spacy download en_core_web_lg

Download pre-trained model

Add a code block and run the following command to load the model.

import spacy
nlp = spacy.load("en_core_web_lg")

Alright. We're all set.

Let's try to split entities from a sentence. Add a code block and run the following block of code:

doc = nlp("Apple is looking at buying U.K. startup for $1 billion")

for ent in doc.ents:
    print(ent.text, ent.start_char, ent.end_char, ent.label_)

In the above code, we're asking the spaCy model to find the entities from the sentence "Apple is looking at buying U.K. startup for $1 billion".

We're then iterating through each entity, and displaying the entity, start and end characters index in the sentence, and the entity label.

You should be seeing the following output:

Named Entity Recognition example 1 from spaCy

The above output describes that "Apple" is an entity and it is present from index 0 to index 5 in the given sentence and it is an Organisation (ORG).

If you're confused about the index, remember that it starts from 0. The first 5 characters in our given input text is "Apple". So, it is from 0 to 5.

Similarly it figures out "U.K." as an entity and describes it as a Geopolitical entity (GPE). It labels "$1 billion" as a Money (MONEY) entity.

Let's try a different sentence this time.

"Prime Minister of India Narendra Modi met US President Joe Biden at Washington DC".

Let's see what are the entities it finds out. Add a code block and run the following code:

doc = nlp("Prime Minister of India Narendra Modi met US President Joe Biden at Washington DC")

for ent in doc.ents:
    print(ent.text, ent.start_char, ent.end_char, ent.label_)

Named Entity Recognition example 2 from spaCy

That's awesome, isn't it?

It has identified "India", "US", and "Washington DC" as Geopolitical entities (GPE). It has also identified "Narendra Modi" and "Joe Biden" as Person entities (PERSON).

Try to input different sentences and play around with it. I'm sure you'll be amazed at its capabilities in identifying entities.

Conclusion

In this tutorial, we learnt about NLP with a simple implementation using the spaCy library.

Natural Language Processing holds immense potential to transform the way we interact with machines and analyze vast amounts of textual data. spaCy has become a go-to library for many NLP practitioners due to its powerful features, ease of use, and exceptional performance.

If you wish to learn more about NLP/Machine Learning, subscribe to my email newsletter (https://5minslearn.gogosoon.com/) and follow me on social media.

In this tutorial, I'll introduce you to API Gateway and explain the benefits of using this helpful tool. Then I'll show you how to create and deploy a Rest API, and create usage plans to offer API keys. Alright, let's get started.

What is API Gateway?

AWS API Gateway is a fully managed service provided by Amazon Web Services (AWS) that simplifies the creation, deployment, and management of APIs at any scale.

It acts as a front door for applications, and allows you to create APIs that act as bridges between clients and back-end services. This enables secure and efficient communication.

Why Do You Need API Gateway?

AWS API Gateway offers many benefits for businesses and developers. Here are a few benefits of using API Gateway.

Scalability and High Availability

With AWS API Gateway, scaling your APIs becomes much easier. It seamlessly handles traffic spikes by automatically scaling the underlying infrastructure. This results in high availability and helps prevent service disruptions.

Security and Authentication

API Gateway offers robust security features, including built-in authentication and authorization mechanisms.

It supports User Authentication through IAM Roles for internal applications, Cognito for external applications (for example Mobile users), and it also supports custom authorizers.

Integration with other AWS Services

As part of the AWS ecosystem, API Gateway seamlessly integrates with a range of other AWS services. This enables you to leverage additional functionalities like AWS Lambda functions, AWS Cognito for user management, and AWS CloudWatch for monitoring and logging.

API Lifecycle Management

With API Gateway, you can easily version, deploy, and manage different stages of your APIs. This simplifies the process of rolling out updates, testing new features, and managing different environments such as development, staging, and production.

I hope by now you understood what API Gateway is and why it's valuable. Let's dive into creating our very own API Gateway.

How to Create an AWS API Gateway

In this section, we will:

• Create a Rest API with the GET method
• Integrate it with a simple hello world lambda function and deploy it

Let's begin with creating a lambda function

How to Create an AWS Lambda Function

Log in to the AWS Management Console and search for "Lambda" in the AWS Management Console search bar. Click on Create Function.

Navigate to AWS Lambda Console

Select the "Author from scratch" option, enter a name for your lambda function, select the "Python" runtime, and click the Create Function button at the bottom right.

Create a AWS Lambda Function

Once the function is created, update the following code and deploy the changes:

import json

def lambda_handler(event, context):
    body = "Hello from 5minslearn!"
    statusCode = 200
    return {
        "statusCode": statusCode,
        "body": json.dumps(body),
        "headers": {
            "Content-Type": "application/json"
        }
    }

Deploy a Lambda Function

Congratulations! You have successfully created an AWS Lambda function. Now let's create the Rest API.

How to Create a Rest API and Integrate it with AWS Lambda

Search for API Gateway in the search bar. In the REST API section, click on the Build button.

Create a Rest API

Choose the Protocol as Rest and select New API in the Create new API section. In the settings section enter the API name of your choice and leave Endpoint Type as the default. Then click the Create API button.

Configure creating a Rest API

Click the Actions Button on the top left. Next, Click Method and select the method as GET and click the Tick icon.

Create a Method

Choose "GET" method

Select Lambda Function as the Integration type and enter the name of the Lambda function you created previously. Then save the function.

Select Method configuration

Once you click save, "Add Permission to Lambda Function" will prompt for confirmation. This basically means that you're allowing the API Gateway to invoke a Lambda function. In this case, it is "DemoFunction" Lambda function. Accept the confirmation and proceed to the next step.

Allow Permission to invoke Lambda Function from API Gateway

Click on Test. It will take you to a new page. Click on the "Test" button. You'll be able to see the response from the Lambda function on the right side panel.

Our API Architecture

Test our API Gateway

As you have successfully tested your API, you're ready to deploy the API. To deploy the API, click on the Actions button once again and click on Deploy API.

Deploy the API Gateway

The Deploy API dialogue will popup. Select New Stage for Deployment stage and name it whatever you want. Click the Deploy button.

Configure API Gateway deployment

Click on Invoke URL shown at the top. You can see the response from the Lambda function.

API Gateway Created

Test our API

Great! We successfully created the Rest API, integrated it with the Lambda function, and deployed it.

But you can do this with multiple services available on the market. Why would you choose AWS API Gateway?

Well. That's a interesting question. First of all, you can configure the usage plan for your API. The best part is you don't have to write any code for it.

Now let's create a Usage Plan, generate an API key, and make our Rest API accessible only by passing the API key in the Header.

How to Create an API Gateway Usage Plan

In the left side bar click on Usage Plans and click the Create button. Enter the Name of your plan – I chose "Basic". Enter the Throttling and Quota sections as per your requirements and click Next.

Create AWS API Gateway usage plan

Click on the Add API Stage button. Select the API and its stage. Click on the tick icon at right corner and select Next.

Create a Stage for our API

Create a Stage for our API

Click on Create API Key and add to Usage Plan. A modal will pop up. Enter the Name for API Key. For the API key, I selected Auto Generate but if you want to give a custom key you can enter a custom key. Hit the Save button.

Create a API Key to access the service

Configure the API Key

Select Resources from the Sidebar, click on the GET API you just created, and click the Method Request.

Select the method

In the Settings section, update the API Key Required field to true and click the Tick icon. Once updated, don't forget to deploy the changes by hitting the Action dropdown. Your changes will not be updated otherwise.

Enable API Key Required field

Deploy the API

Hit the same URL now and see the magic.

Forbidden!

Because our API layer is protected now. You have to pass the API key in the header to access the data.

Forbidden access if no API Key is provided

Now Click on the Usage Plans from the Sidebar. Select your plan and navigate to the API Keys tab.

Access your API Key

Click on the API key you created in Step 3. Click Show. Copy the API key.

List of API Keys

Reveal your API Key

You have to pass the API Key in the 'x-api-key' header. Let's switch to the terminal to test this out.

Verify your Rest API without passing the API key at first. Open the terminal, and enter the following curl command. You will once again see the forbidden message.

curl --location --request GET '[enter your invoke url]'
--header 'Content-Type: application/json

Forbidden access without API Key in Terminal

Now pass the API key this time. Run the following curl command:

curl --location --request GET '[your invoke url]' \
--header 'x-api-key: [your api key]' \
--header 'Content-Type: application/json' \
--data-raw ''

Data received on passing API Key in x-api-key Header

You can see the output of the Lambda function because you passed 'x-api-key' in the header.

Awesome! You have successfully created the Usage plan, generated the API key, and attached it to the Rest API method and verified the integration.

Conclusion

In this tutorial, you learned what AWS API gateway is and how to create Usage Plans for the Rest API.

If you wish to learn more about AWS Services, subscribe to my email newsletter (https://5minslearn.gogosoon.com/) and follow me on social media.