I ran into this firsthand while trying to build a live options dashboard. HTTP requests weren't going to cut it, and everything I was reading seemed overly complex until I went back to the basics. This article is the result of that process.

We'll cover Python's websockets library from scratch, then move into FastAPI, where many Python backends live. It's worth noting that WebSockets aren't the only solution for real-time communication. WebRTC may be a better fit depending on your use case, but understanding WebSockets is the right starting point before exploring further.

Table of Contents

1. WebSocket Connections and Methods

2. How to Build Your First WebSocket in Python

3. File Transfer Over WebSockets

4. How to Connect to an External WebSocket

5. WebSockets in FastAPI

6. How to Handle WebSocket Disconnections in FastAPI

7. Conclusion

WebSocket Connections and Methods

A WebSocket connection enables bi-directional communication between a client and a server. Once a connection is established, both sides can communicate freely without either having to ask first. This is different from a regular HTTP request, where the client always has to ask before the server can respond.

It looks something like this:

        CLIENT  <===== open connection =====>  SERVER

Note that a WebSocket URL is not a regular web page, so you can't "visit it" like a website. You need a client to talk to it.

Different frameworks provide different methods for handling WebSocket connections. With Python’s websockets library, for instance, a connection is automatically accepted the moment a client connects. With frameworks like FastAPI, you have to explicitly call await websocket.accept(), otherwise the connection gets rejected.

Let’s look at the core methods provided by Python’s websockets library:

1. websockets.serve(...):  starts a WebSocket server.

2. websockets.connect(...): connects to a WebSocket server.

3. websockets.send(...): sends a message from either side.

4. websockets.recv(): receives a message from client or server.

recv() takes no arguments because it's purely a waiting operation. It waits for the next message and returns it:

message = await websocket.recv()

How to Build Your First WebSocket in Python

Before we dive into frameworks, let’s explore Python’s websockets library. You’ll set up a simple server and client, and exchange messages over a WebSocket connection, giving you a solid foundation for understanding WebSockets under the hood.

Environment Setup

Run the following in your virtual environment to install or verify the WebSockets package:

pip install websockets
# or, to check if it's already installed:
pip show websockets

Create the WebSocket Server

Create server.py in your project folder, and paste this:

import asyncio
import websockets

async def handler(connection):
    print("Client connected")

    message = await connection.recv()
    print("Received from client:", message)
    await connection.send("Hello client!")


async def main():
    async with websockets.serve(handler, "localhost", 8000):
        print("Server running at ws://localhost:8000")
        #await asyncio.Future()  # runs forever
        await asyncio.sleep(30)

asyncio.run(main())

When this line executes:

async with websockets.serve(handler, "localhost", 8000):

The library opens a TCP socket on the specified host and port and waits for incoming clients. When one connects, it creates a connection object and passes it into your handler function.

The handler is required because it defines what the server does with each connection. The host and port arguments are also important. Both default to None – passing neither raises an error because the OS cannot bind a network server without a port.

You could pass port=0 to let the OS assign a free port automatically, but then you'd need an extra step to figure out which port was chosen, so the client can connect:

server.sockets[0].getsockname()

It’s simpler to specify both host and port explicitly, so the client knows exactly where the server is running.

Set Up the Client

Create client.py in the same folder and add this:

import asyncio
import websockets

async def client():
    async with websockets.connect("ws://localhost:8000") as websocket:
        await websocket.send("Hello server!")
        response = await websocket.recv()
        print("Server replied:", response)

asyncio.run(client())

Test the Connection

First, open a terminal and run server.py. You should see:

Server running at ws://localhost:8000

In a second terminal, run client.py. Messages should appear in both terminals confirming that the connection is active and both sides are communicating.

Note that the server must be running before you start the client – otherwise the client has nothing to connect to, and the connection will fail.

Keeping the server alive: a note on asyncio.Future()

In server.py, there’s a line currently commented out:

await asyncio.Future()

This keeps the server running indefinitely. For local development and testing however, await asyncio.sleep(30) is a simpler alternative. It keeps the server alive for a fixed period without running forever.

File Transfer Over WebSockets

WebSockets aren't limited to text. They support raw bytes too, which means you can send files directly over the connection. Here’s how a client can send a file to a server over a WebSocket connection:

Update server.py

async def file_handler(ws):
    print("Client connected, waiting for file...")
    file_bytes = await ws.recv()  # receive bytes
    with open("received_file.png", "wb") as f:
        f.write(file_bytes)
    print("File received and saved!")
    await ws.send("File received successfully!")

async def main():
    async with websockets.serve(file_handler, "localhost", 8000):
        print("Server running on ws://localhost:8000")
        await asyncio.sleep(50)  # keep server alive

asyncio.run(main())

The handler waits for incoming bytes with await ws.recv(); the websockets library automatically detects whether the incoming message is text or bytes, so no extra configuration is needed. Once received, the file is written to disk in binary mode ("wb") and the server sends a confirmation message back to the client.

Update client.py

import asyncio
import websockets

async def send_file():
    uri = "ws://localhost:8000"
    async with websockets.connect(uri) as ws:
        with open("portfolio-image.png", "rb") as f:  #open file in binary mode
            file_bytes = f.read()
        await ws.send(file_bytes)  # send bytes
        response = await ws.recv()
        print("Server response:", response)

asyncio.run(send_file())

The client opens the image in binary mode ("rb"), reads the entire file into memory as bytes, and sends it in a single ws.send() call. It then waits for the server's confirmation before closing the connection.

Test it

Add an image to your project folder and make sure the filename in client.py matches. Run server.py first, then client.py in a second terminal.

Once the transfer completes, the server saves the file as received_file.png in the same directory. You should see it appear in your workspace immediately.

This approach loads the entire file into memory before sending. For large files, it’s better to read and send them in chunks. But this is the easiest way to understand WebSocket byte transfer.

How to Connect to an External WebSocket

So far you've been connecting to servers you built yourself. But WebSocket clients can also connect to public servers. For example, a client can connect to Postman’s echo server:

import asyncio
import websockets

async def connect_external():
    uri = "wss://ws.postman-echo.com/raw"  # public WebSocket server
    async with websockets.connect(uri) as ws:
        print("Connected to external server!")

        # Send a message
        await ws.send("Hello external server!")
        print("Message sent")

        # Receive response
        response = await ws.recv()
        print("Received from server:", response)
asyncio.run(connect_external())

Notice the client connects to Postman’s echo server using the wss:// URI scheme instead of ws://. This indicates the connection is encrypted using TLS, similar to how https:// secures regular web requests.

An echo server returns exactly what you send it. So "Hello external server!" comes straight back as the response. It's a useful sandbox for testing your client-side WebSocket code without needing your own server.

WebSockets in FastAPI

FastAPI provides a WebSocket object (via Starlette under the hood) to manage real-time connections. You can define WebSocket endpoints just like HTTP routes, while Uvicorn handles the event loop – no manual asyncio server management needed. This makes FastAPI a natural fit for real-time projects, from chat apps to live dashboards and data feeds.

Before jumping into code, here's a quick reference of the core methods you'll be working with.

Accepting:

• await websocket.accept(): the accept() method must be called first, before anything else. Skip it and the connection gets rejected.

Sending:

• await websocket.send_text(data): sends a string.

• await websocket.send_bytes(data): sends binary data.

• await websocket.send_json(data): serializes and sends JSON.

Receiving:

• await websocket.receive_text(): waits for a text message.

• await websocket.receive_bytes(): waits for binary data.

• await websocket.receive_json(): receives and deserializes JSON.

• async for msg in websocket.iter_text(): iterates over incoming messages, exits cleanly on disconnect.

Closing:

• await websocket.close(code=1000): standard code for a normal closure. It accepts an optional “reason” argument.

Here's what the WebSocket lifecycle looks like in FastAPI:

Building a Simple Echo Server with FastAPI

As you saw with the Postman example, an echo server sends back the message a client provides. Let's build one with FastAPI.

1. Install FastAPI:

pip install "fastapi[standard]"

2. Update server.py:

from fastapi import FastAPI, WebSocket

app = FastAPI()

@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()
    data = await websocket.receive_text()
   
    await websocket.send_text(f"You said: {data}")

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="127.0.0.1", port=8000)

A few things to note here compared to the plain websockets library:

• WebSocket endpoints are defined with @app.websocket("/ws") just like an HTTP route.

• await websocket.accept() is required before anything else. FastAPI won't accept connections without it.

• Uvicorn handles the event loop and server startup for you via the if name == "__main__" block. No asyncio.run() or asyncio.Future() needed.

3. Update client.py:

async def test_client():
    uri = "ws://127.0.0.1:8000/ws"
    async with websockets.connect(uri) as ws:
        await ws.send("Hello FastAPI server!")
        response = await ws.recv()
        print("Server replied:", response)

asyncio.run(test_client())

Since the FastAPI server isn't secured with TLS, the client URI uses ws:// instead of wss://. Make sure to match the host and port from your server code.

4. Interact with the echo server:

Start server.py, then run client.py in another terminal. The server terminal should show the echoed message.

How to Handle WebSocket Disconnections in FastAPI

Clients will inevitably disconnect in real-time applications, sometimes intentionally, sometimes unexpectedly. If not handled properly, this can crash your server or leave it in a broken state.

The WebSocketDisconnect exception in FastAPI is raised whenever a client unexpectedly closes the connection, allowing the server to handle disconnects gracefully, log the event, and clean up resources without crashing.

Here’s an example:

@app.websocket("/ws")
async def websocket_endpoint(ws: WebSocket):
    await ws.accept()
    try:
        while True:
            data = await ws.receive_text()
   
            if "bye" in data or "quit" in data:
                await ws.send_text("Closing connection")
                await ws.close(code=1000, reason="Server requested close")  
                break
            await ws.send_text(f"I got your request: {data}")
    except WebSocketDisconnect:
        print("Client disconnected")  # connection already closed

The server runs a continuous loop waiting for messages. If the client message contains "bye" or "quit", the server responds, calls await ws.close(code=1000), and breaks out of the loop cleanly.

But if the client disconnects unexpectedly, WebSocketDisconnect is caught by the except block and the server moves on without crashing. At this point the connection is already closed on the client side, so calling ws.close() inside the except block is unnecessary.

Conclusion

WebSockets make real-time communication possible by keeping a persistent connection open between client and server. Starting with Python’s websockets library helps clarify how the protocol works under the hood, while frameworks like FastAPI provide the structure needed for production applications.

The parts that trip most people up early on are asyncio and FastAPI's explicit websocket.accept(). With asyncio, the question is usually why it's needed and why the server dies instantly without something keeping it alive. And it's easy to ignore websocket.accept() if you're coming from the plain websockets library where that happens automatically. Once those click, everything else follows naturally.

We just posted a course on the freeCodeCamp.org YouTube channel that will teach you how to construct a basic backend for a website using popular technologies like Node.js, Express.js, and the MongoDB NoSQL database.

The tutorial covers the core components of backend development, walking you through the initial server setup, database connection, and code structure using a model, route, and controller pattern.

You will learn to build practical APIs for user authentication, including password hashing, as well as full CRUD (Create, Read, Update, Delete) operations to manage data. Finally, the video demonstrates how to utilize Postman to test your server's requests and ensure your APIs are functioning correctly.

This course was developed by Shivani. She is part of Hack Club. Hack Club is a global non-profit organization that creates a community for high school students interested in coding and making things with technology.

Here are the sections covered in this course:

• Introduction & Overview

• What is a Backend?

• Core Components: Languages, Databases, Runtimes, Frameworks

• Backend Architecture Flowchart

• How Frontend Connects to Backend (APIs)

• Prerequisites & Installing Node.js

• Project Folder Structure

• Project Initialization (Git & npm)

• Setting up MongoDB Atlas Database

• Environment Variables (.env)

• Constants & ES Modules Setup

• Creating the Express App (app.js)

• Connecting Database to Server (database.js)

• Server Entry Point (index.js)

• Setting up Nodemon & Running the Server

• Understanding Models & ER Diagrams

• Creating the User Model

• Understanding Routes

• Setting up User Routes

• Understanding Controllers

• Coding the Register Controller

• The Journey of a Request

• HTTP Methods & Status Codes Explained

• Introduction to Postman

• Testing the Register API

• Viewing Data in MongoDB Atlas

• Coding the Login Controller

• Hashing Passwords with Bcrypt

• Comparing Passwords for Login

• Testing the Login API

• Coding the Logout Controller

• Testing the Logout API

• Intro to CRUD APIs

• Creating the Post Model

• Create Post API (Controller & Route)

• Testing Create Post

• Read All Posts API

• Testing Get Posts

• Update Post API

• Testing Update Post

• Delete Post API

• Testing Delete Post

• Final Commit & Conclusion

Watch the full course on the freeCodeCamp.org YouTube channel (2-hour watch),.

But once you move beyond basic apps, CRUD starts to fall apart.

Real-world systems don’t just “update” or “delete” things. In a loan application system, for example, borrowers “submit” applications, loan officers “approve” or “reject” them, and applications are eventually “archived”. These aren’t generic CRUD operations—they’re domain-specific actions that carry meaning.

And that’s the problem: CRUD hides the meaning of our systems behind vague verbs. REST APIs inherit the same issue, mapping HTTP verbs onto CRUD but still failing to express real workflows clearly.

In this article, we’ll explore:

• Why CRUD works fine for simple apps but becomes an anti-pattern at scale

• How concepts like upsert, archive, and bulk operations reveal its cracks

• Why REST doesn’t solve these issues

• How to design APIs around domain actions and workflows instead.

By the end, you’ll see CRUD for what it really is—a teaching tool, not a design philosophy.

Table of Contents

• What is CRUD?

• Stretching CRUD: Upsert, Archive, Bulk

• Breaking CRUD: Domain Actions

• Breaking CRUD: Domain Authorization

• CRUD Alternative: Align to Workflows

• Conclusion

What is CRUD?

Create, Read, Update, Delete—these are the four basic operations we perform on data in a database.

• Create – adds a new record.

• Read – fetches an existing record (or list of records).

• Update – changes one or more fields in a record.

• Delete – removes a record.

For example, in a typical Node.js + Express app managing users:

// Create a user
app.post('/users', createUser);

// Read a user
app.get('/users/:id', getUser);

// Update a user
app.put('/users/:id', updateUser);

// Delete a user
app.delete('/users/:id', deleteUser);

This maps directly to the underlying SQL:

INSERT INTO users (...);
SELECT * FROM users WHERE id = ...;
UPDATE users SET ... WHERE id = ...;
DELETE FROM users WHERE id = ...;

And that’s CRUD in its purest form—four operations that can describe almost any database interaction.

Stretching CRUD: Upsert, Archive, Bulk

Developers quickly realize CRUD isn’t enough, so they invent extensions:

• Upsert: a mix of “update” and “insert.” If the record exists, update it; if not, create it.

• Archive: instead of deleting a record, we “soft delete” or mark it as inactive so the history stays intact.

• Bulk operations: run create, update, or delete on many records at once for efficiency.

These solve real problems, but they stretch CRUD’s simple model. We now need to distinguish between single and bulk resource actions. And we also need to factor in the technical concerns of upsertions and soft deletions.

Breaking CRUD: Domain Actions

The technical domain itself stretches CRUD substantially, but business domain concerns break it entirely. Take a loan application system:

• A borrower doesn’t “create” and “update” an application—they start, submit, or withdraw it.

• A loan officer doesn’t “update” an application—they review, approve, or reject it.

• Applications don’t get “deleted”—they’re usually archived so there’s a record for compliance.

If we try to model these as plain CRUD, the meaning gets lost:

PATCH /applications/123 { "status": "approved" }

Technically, it works. But what does “update” really mean here? Was the application submitted, rejected, or archived? You can’t tell from the API call.

The core problem: CRUD hides intent behind generic, technical language. Real business processes are expressed as domain-specific actions, not generic updates or deletes.

Breaking CRUD: Domain Authorization

CRUD not only obscures intent—it also creates authorization gaps. Using the same loan application example:

• Only loan officers should approve applications.

• Borrowers should only edit their own information or withdraw their applications.

If “approve” is just modeled as a generic update, the system can’t distinguish between roles without additional checks. A naive authorization rule like “can this user update?” suddenly lets borrowers perform actions reserved for officers.

This mismatch between technical verbs and business rules can lead to:

• Security issues—unauthorized actions performed by the wrong user.

• Audit problems—it’s unclear who did what, and when.

• Workflow confusion—state transitions get lost in generic updates.

The solution: treat each domain action as its own API call with explicit authorization rules:

POST /applications/123/approve   # Only accessible to loan officers
POST /applications/123/withdraw  # Only accessible to the borrower

By modeling actions instead of CRUD operations, intent and permissions are clear, reducing both bugs and security risks.

CRUD Alternative: Align to Workflows

Real-world applications follow workflows—sequences of states that a resource moves through. Take our loan application example:

Here’s what the corresponding API endpoints might look like:

# Borrower actions
POST /applications/123/submit       # Draft → Submitted
POST /applications/123/withdraw     # Draft/Submitted → Closed

# Loan officer actions
POST /applications/123/approve      # Submitted → Approved
POST /applications/123/reject       # Submitted → Rejected

# System/Admin actions
POST /applications/123/close        # Approved/Rejected → Closed

# Side effect: spawning a Loan (after Approved)
POST /loans
{
  "applicationId": "123",
  "amount": 50000,
  "borrowerId": "456",
  "terms": { ... }
}

At this point, our API calls are almost entirely outside the CRUD pattern—the only one that resembles a CRUD action is the spawning of a loan, which looks like a “create”. Behind the scenes, we’ll still use INSERT, SELECT, and UPDATE statements in SQL, but at the API level we’re aligning to the actual business workflow. Because of it, we’re able to easily support the following:

1. Actions reflect business intent — each API call maps to a real-world task like submit, approve, or withdraw.

2. Built-in authorization — endpoints clearly separate borrower, loan officer, and admin responsibilities.

3. Auditability and workflow enforcement — state transitions are explicit and invalid transitions are prevented.

4. Controlled side effects — spawning loans, notifications, and downstream processes are handled deliberately.

Conclusion

By moving away from CRUD and modeling domain actions instead, our API aligns with real business workflows, clearly communicates intent, and enforces rules and authorization naturally. State transitions, side effects, and auditing become explicit, reducing errors and security risks. While CRUD still powers the underlying database operations, thinking in terms of actions and workflows ensures that the system behaves the way the business expects.

Spring Boot’s multi-module structure allows you to manage different parts of the application independently, which lets your team develop, test, and deploy components separately. This structure keeps code organized and modular, making it useful for both microservices and large monolithic systems.

In this tutorial, you’ll build a multi-module Spring Boot project, with each module dedicated to a specific responsibility. You’ll learn how to set up modules, configure inter-module communication, handle errors, implement JWT-based security, and deploy using Docker.

Prerequisites:

• Basic knowledge of Spring Boot and Maven.

• Familiarity with Docker and CI/CD concepts (optional but helpful).

Table of Contents

1. Why Multi-Module Projects?

2. Project Structure and Architecture

3. How to Set Up the Parent Project

4. How to Create the Modules

5. Inter-Module Communication

6. Common Pitfalls and Solutions

7. Testing Strategy

8. Error Handling and Logging

9. Security and JWT Integration

10. Deployment with Docker and CI/CD

11. Best Practices and Advanced Use Cases

12. Conclusion and Key Takeaways

1. Why Multi-Module Projects?

In single-module projects, components are often tightly coupled, making it difficult to scale and manage complex codebases. A multi-module structure offers several advantages:

• Modularity: Each module is dedicated to a specific task, such as User Management or Inventory, simplifying management and troubleshooting.

• Team Scalability: Teams can work independently on different modules, minimizing conflicts and enhancing productivity.

• Flexible Deployment: Modules can be deployed or updated independently, which is particularly beneficial for microservices or large applications with numerous features.

Real-World Example

Consider a large e-commerce application. Its architecture can be divided into distinct modules:

• Customer Management: Responsible for handling customer profiles, preferences, and authentication.

• Product Management: Focuses on managing product details, stock, and pricing.

• Order Processing: Manages orders, payments, and order tracking.

• Inventory Management: Oversees stock levels and supplier orders.

Case Study: Netflix

To illustrate these benefits, let's examine how Netflix employs a multi-module architecture.

Netflix is a leading example of a company that effectively uses this approach through its microservices architecture. Each microservice at Netflix is dedicated to a specific function, such as user authentication, content recommendations, or streaming services.

This modular structure enables Netflix to scale its operations efficiently, deploy updates independently, and maintain high availability and performance. By decoupling services, Netflix can manage millions of users and deliver content seamlessly worldwide, ensuring a robust and flexible system that supports its vast and dynamic platform.

This architecture not only enhances scalability but also improves fault isolation, allowing Netflix to innovate rapidly and respond effectively to user demands.

2. Project Structure and Architecture

Now let’s get back to our example project. Your multi-module Spring Boot project will use five key modules. Here’s the layout:

codespring-boot-multi-module/
 ├── common/               # Shared utilities and constants
 ├── domain/               # Domain entities
 ├── repository/           # Data access layer (DAL)
 ├── service/              # Business logic
 └── web/                  # Main Spring Boot application and controllers

Each module has a specific role:

• common: Stores shared utilities, constants, and configuration files used across other modules.

• domain: Contains data models for your application.

• repository: Manages database operations.

• service: Encapsulates business logic.

• web: Defines REST API endpoints and serves as the application’s entry point.

This structure aligns with separation of concerns principles, where each layer is independent and handles its own logic.

The diagram below illustrates the various modules:

3. How to Set Up the Parent Project

Step 1: Create the Root Project

Let’s run these commands to create the Maven parent project:

mvn archetype:generate -DgroupId=com.example -DartifactId=spring-boot-multi-module -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false
cd spring-boot-multi-module

Step 2: Configure the Parent pom.xml

In the pom.xml, let’s define our dependencies and modules:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://www.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.example</groupId>
    <artifactId>spring-boot-multi-module</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>pom</packaging>
    <modules>
        <module>common</module>
        <module>domain</module>
        <module>repository</module>
        <module>service</module>
        <module>web</module>
    </modules>
    <properties>
        <java.version>11</java.version>
        <spring.boot.version>2.5.4</spring.boot.version>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>${spring.boot.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

This pom.xml file centralizes dependencies and configurations, making it easier to manage shared settings across modules.

4. How to Create the Modules

Common Module

Let’s create a common module to define shared utilities like date formatters. Create this module and add a sample utility class:

mvn archetype:generate -DgroupId=com.example.common -DartifactId=common -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

Date Formatter Utility:

package com.example.common;

import java.time.LocalDate;
import java.time.format.DateTimeFormatter;

public class DateUtils {
    public static String formatDate(LocalDate date) {
        return date.format(DateTimeFormatter.ofPattern("yyyy-MM-dd"));
    }
}

Domain Module

In the domain module, you will define your data models.

package com.example.domain;

import javax.persistence.Entity;
import javax.persistence.Id;

@Entity
public class User {
    @Id
    private Long id;
    private String name;

    // Getters and Setters
}

Repository Module

Let’s create the repository module to manage data access. Here’s a basic repository interface:

package com.example.repository;

import com.example.domain.User;
import org.springframework.data.jpa.repository.JpaRepository;

public interface UserRepository extends JpaRepository<User, Long> {}

Service Module

Let’s create the service module to hold your business logic. Here’s an example service class:

package com.example.service;

import com.example.domain.User;
import com.example.repository.UserRepository;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class UserService {

    @Autowired
    private UserRepository userRepository;

    public User getUserById(Long id) {
        return userRepository.findById(id).orElse(null);
    }
}

Web Module

The web module serves as the REST API layer.

@RestController
public class UserController {

    @Autowired
    private UserService userService;

    @GetMapping("/users/{id}")
    public User getUserById(@PathVariable Long id) {
        return userService.getUserById(id);
    }
}

5. Inter-Module Communication

To avoid direct dependencies, you can use REST APIs or message brokers (like Kafka) for inter-module communication. This ensures loose coupling and allows each module to communicate independently.

The diagram below demonstrates how modules communicate with each other:

The diagram illustrates how different system components communicate to handle requests efficiently.

The Web Module processes incoming API requests and forwards them to the Service Module, which contains the business logic. The Service Module then interacts with the Repository Module to fetch or update data in the Database. This layered approach ensures that each module operates independently, promoting flexibility and easier maintenance.

Example Using Feign Client:

In the context of inter-module communication, using tools like Feign Clients is a powerful way to achieve loose coupling between services.

The Feign client allows one module to seamlessly communicate with another through REST API calls, without requiring direct dependencies. This approach fits perfectly within the layered architecture described earlier, where the Service Module can fetch data from other services or microservices using Feign clients, rather than directly accessing databases or hard-coding HTTP requests.

This not only simplifies the code but also improves scalability and maintainability by isolating service dependencies.

@FeignClient(name = "userServiceClient", url = "http://localhost:8081")
public interface UserServiceClient {
    @GetMapping("/users/{id}")
    User getUserById(@PathVariable("id") Long id);
}

6. Common Pitfalls and Solutions

When implementing a multi-module architecture, you may encounter several challenges. Here are some common pitfalls and their solutions:

• Circular Dependencies: Modules may inadvertently depend on each other, creating a circular dependency that complicates builds and deployments.

  • Solution: Carefully design module interfaces and use dependency management tools to detect and resolve circular dependencies early in the development process.

• Over-Engineering: There's a risk of creating too many modules, leading to unnecessary complexity.

  • Solution: Start with a minimal set of modules and only split further when there's a clear need, ensuring each module has a distinct responsibility.

• Inconsistent Configurations: Managing configurations across multiple modules can lead to inconsistencies.

  • Solution: Use centralized configuration management tools, such as Spring Cloud Config, to maintain consistency across modules.

• Communication Overhead: Inter-module communication can introduce latency and complexity.

  • Solution: Optimize communication by using efficient protocols and consider asynchronous messaging where appropriate to reduce latency.

• Testing Complexity: Testing a multi-module project can be more complex due to the interactions between modules.

  • Solution: Implement a robust testing strategy that includes unit tests for individual modules and integration tests for inter-module interactions.

By being aware of these pitfalls and applying these solutions, you can effectively manage the complexities of a multi-module architecture and ensure a smooth development process.

7. Testing Strategy and Configuration

Testing each module independently and as a unit is critical in multi-module setups.

Unit Tests

Here, we’ll use JUnit and Mockito for performing unit tests:

@RunWith(MockitoJUnitRunner.class)
public class UserServiceTest {

    @Mock
    private UserRepository userRepository;

    @InjectMocks
    private UserService userService;

    @Test
    public void testGetUserById() {
        User user = new User();
        user.setId(1L);
        user.setName("John");

        Mockito.when(userRepository.findById(1L)).thenReturn(Optional.of(user));

        User result = userService.getUserById(1L);
        assertEquals("John", result.getName());
    }
}

Integration Tests

And we’ll use Testcontainers with an in-memory database for integration tests:

@Testcontainers
@ExtendWith(SpringExtension.class)
@SpringBootTest
public class UserServiceIntegrationTest {

    @Container
    private static PostgreSQLContainer<?> postgresqlContainer = new PostgreSQLContainer<>("postgres:latest");

    @Autowired
    private UserService userService;

    @Test
    public void testFindById() {
        User user = userService.getUserById(1L);
        assertNotNull(user);
    }
}

8. Error Handling and Logging

Error handling and logging ensure a robust and debuggable application.

Error Handling

In this section, we'll explore how to handle errors gracefully in your Spring Boot application using a global exception handler. By using @ControllerAdvice, we'll set up a centralized way to catch and respond to errors, keeping our code clean and our responses consistent.

@ControllerAdvice
public class GlobalExceptionHandler {

    @ExceptionHandler(UserNotFoundException.class)
    public ResponseEntity<String> handleUserNotFoundException(UserNotFoundException ex) {
        return new ResponseEntity<>("User not found", HttpStatus.NOT_FOUND);
    }
}

In the code example above, we define a GlobalExceptionHandler that catches any UserNotFoundException and returns a friendly message like "User not found" with a status of 404. This way, you don’t have to handle this exception in every controller—you’ve got it covered in one place!

Now, let’s take a look at the diagram. Here’s how it all flows: when a client sends a request to our Web Module, if everything goes smoothly, you'll get a successful response. But if something goes wrong, like a user not being found, the error will be caught by our Global Error Handler. This handler logs the issue and returns a clean, structured response to the client.

This approach ensures that users get clear error messages while keeping your app’s internals hidden and secure.

Logging

Structured logging in each module improves traceability and debugging. You can use a centralized logging system like Logback and include correlation IDs to trace requests.

9. Security and JWT Integration

In this section, we’re going to set up JSON Web Tokens (JWT) to secure our endpoints and control access based on user roles. We'll configure this in the SecurityConfig class, which will help us enforce who can access what parts of our application.

@EnableWebSecurity
public class SecurityConfig extends WebSecurityConfigurerAdapter {

    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http.authorizeRequests()
            .antMatchers("/admin/**").hasRole("ADMIN")
            .antMatchers("/user/**").hasAnyRole("USER", "ADMIN")
            .anyRequest().authenticated()
            .and()
            .oauth2ResourceServer().jwt();
    }
}

In the code example above, you can see how we’ve defined access rules:

• The /admin/** endpoints are restricted to users with the ADMIN role.

• The /user/** endpoints can be accessed by users with either the USER or ADMIN roles.

• Any other requests will require the user to be authenticated.

Next, we set up our application to validate incoming tokens using .oauth2ResourceServer().jwt();. This ensures that only requests with a valid token can access our secured endpoints.

Now, let’s walk through the diagram. When a client sends a request to access a resource, the Security Filter first checks if the provided JWT token is valid. If the token is valid, the request proceeds to the Service Module to fetch or process the data. If not, access is denied right away, and the client receives an error response.

This flow ensures that only authenticated users can access sensitive resources, keeping our application secure.

10. Deployment with Docker and CI/CD

In this section, we'll containerize each module using Docker to make our application easier to deploy and run consistently across different environments. We’ll also set up a CI/CD pipeline using GitHub Actions (but you can use Jenkins too if you prefer). Automating this process ensures that any changes you push are automatically built, tested, and deployed.

Step 1: Containerizing with Docker

We start by creating a Dockerfile for the Web Module:

FROM openjdk:11-jre-slim
COPY target/web-1.0-SNAPSHOT.jar app.jar
ENTRYPOINT ["java", "-jar", "/app.jar"]

Here, we’re using a lightweight version of Java 11 to keep our image size small. We copy the compiled .jar file into the container and set it up to run when the container starts.

Step 2: Using Docker Compose for Multi-Module Deployment

Now, we'll use a Docker Compose file to orchestrate multiple modules together:

version: '3'
services:
  web:
    build: ./web
    ports:
      - "8080:8080"
  service:
    build: ./service
    ports:
      - "8081:8081"

With this setup, we can run both the Web Module and the Service Module at the same time, making it easy to spin up the entire application with a single command. Each service is built separately from its own directory, and we expose the necessary ports to access them.

CI/CD Example with GitHub Actions

name: CI Pipeline

on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up JDK 11
      uses: actions/setup-java@v2
      with:
        java-version: '11'
    - name: Build with Maven
      run: mvn clean install

This pipeline automatically kicks in whenever you push new code or create a pull request. It checks out your code, sets up Java, and runs a Maven build to ensure everything is working correctly.

11. Best Practices and Advanced Use Cases

The following best practices ensure maintainability and scalability.

Best Practices

• Avoid Circular Dependencies: Ensure modules don’t have circular references to avoid build issues.

• Separate Concerns Clearly: Each module should focus on one responsibility.

• Centralized Configurations: Manage configurations centrally for consistent setups.

Advanced Use Cases

1. Asynchronous Messaging with Kafka: Use Kafka for decoupled communication between services. Modules can publish and subscribe to events asynchronously.

2. REST Client with Feign: Use Feign to call services within modules. Define a Feign client interface for communication.

3. Caching for Performance: Use Spring Cache in the service module for optimizing data retrieval.

Conclusion and Key Takeaways

A multi-module Spring Boot project provides modularity, scalability, and ease of maintenance.

In this tutorial, you learned to set up modules, manage inter-module communication, handle errors, add security, and deploy with Docker.

Following best practices and using advanced techniques like messaging and caching will further optimize your multi-module architecture for production use.

These bad elements access the back-end servers through vulnerable points in the gateways to wreak havoc, stealing relevant info and negatively affecting application performance and efficiency via various forms of API attacks such as SQL and non-SQL-based injections, Distributed Denial-of-Service (DDoS) attacks, code malware, and other methods to exploit vulnerabilities.

In this article, I will be focusing on rate limiting, an important hack that helps protect the back-end API from being exploited by hackers via the use of DDoS, Brute-force attacks and other related malicious activities. But first of all, what does rate limiting mean?

Table of Contents

1. What is Rate Limiting?

2. Importance of Rate Limiting

3. Adoption and Usage of Rate-Limiting by Popular Sites

4. Other Real life Use-cases of API Rate Limiting

5. How Does Rate Limiting Work?

6. Examples of Rate Limiting Algorithms

7. Rate Limiting Best Practices

8. Conclusion

What is Rate Limiting?

Rate limiting is a mechanism put in place to regulate the frequency of requests made by a client to the back-end server. It prevents the repetition of a client request within a defined time frame.

Why do we have to implement rate limiting in the API development? We'll discuss that in the next section.

Importance of Rate Limiting

Here are some of the reasons why rate limiting is used in back-end application development.

DDoS Attacks

First of all, it serves as a preventive measure to mitigate against DDoS attacks. DDoS attacks are malicious attacks on servers which involves flooding the server endpoints with multiple requests, often millions, resulting in reduced server efficiency and disruption of server functions. In most cases, they occur with the use of automated bots.

These attacks can be volumetric, protocol-based or application layer-based. A key example of this form of attack occurred on the GitHub website in the year 2018.

Web Scraping

Rate limiting also plays a role in protecting web applications and web servers from unauthorized web scrapers and web crawlers. These tools, also automated, usually emit requests continually to collect relevant website data which can be exposed to unauthorized. Having a good rate limiter in place helps to prevent all these.

Brute Force Attack

This involves trying to gain access to a server's resources by trying all possible configurations to get access to the resource. This could be done manually but is mostly automated with the use of bots as it is resource-consuming. Rate limiting also proves effective in preventing these form of attacks by disabling the requests if they exceed the required number of requests within a specific time frame.

Resource Optimization

Server requests usually cost the API owners some expenses in terms of running and maintenance cost. Having a rate limiter in place helps to regulate the number of requests the server can handle, help conserve cost and maximize efficiency. Subsequently, we will highlight some algorithms which rate limiters are built on.

Adoption and Usage of Rate-Limiting by Popular Sites

Rate limiting as a security measure has been adopted by a lot of tech products, ranging from large-scale to small-scale applications. For example, Twitter (X) has a rale limit feature implemented in the application programming interfaces they provide to developers.

These interfaces allow access to Twitter sign-up extension and other features made available by Twitter. To guarantee the efficient running of these interfaces, Twitter imposed a rate limit of 50 tweet post requests per user every 24 hours. More details about this can be found here.

Other Real life Use-cases of API Rate Limiting

The use of an application programming interface isn't limited to just what popular sites like Twitter use it for. Here are some other real-life applications of rate-limiting in today's world.

Reducing the Incidence of Spamming

Research reveals that over 160 billion spam emails are sent daily. Hence this has prompted the implementation of rate-limiting to curb the spread of unsolicited messages and spam content via messaging and emailing platforms over a specific time range. By so doing, it encourages responsible use of these platforms.

Tackling Fraudulent Activities

Rate limiting is currently implemented across web applications to help detect unusual web application activities by some users which may possess fraudulent intents. This measure serves to prevent and mitigate the ongoing fraudulent transactions being performed over the application server.

Disabling Malicious User Authentication

Individuals with malicious intent may want to compromise the web servers by taking several measures such as brute force, DDoS and other techniques to take over other users' accounts.

However, several sites have efficient rate limit systems in places which limit the amount of login attempts an individual has to a site within a specific time range. This also contributes to web security measures.

How Does Rate Limiting Work?

Rate-limiting tools used in applications are implemented based on different algorithm structures. These algorithms guide the functionality of the rate-limiting tool whose end goal is to limit the number of requests a server receives per time to enhance its efficiency.

Examples of Rate Limiting Algorithms

Here are some of the most popular algorithms currently in use.

Fixed Window Algorithm

This algorithm is based on fixing a static definite time interval by the server for all clients, regulating the number of requests that can be made to the server, irrespective of the number of clients accessing the API.

For example, setting a request limit of five minutes prevents any client from accessing the endpoint until the expiration of the five minutes window. This model isn’t cost-effective.

Sliding Window Algorithm

This algorithm is similar in configuration to the fixed window algorithm but it provides a solution to the fixed window algorithm by individualizing client access to a given number of requests within a specific time interval by creating independent time intervals for each client.

For example, if Client A accesses the request by 10:00, the client is allowed to make 10 requests until the expiration of the time by 10:03, while Client B who accesses the request by 10:02 is allowed to make 10 requests until the expiration by 10:05.

Leaking Bucket Algorithm

This algorithm is based on the literal meaning of its name: the leaking bucket. It ensures that only a specific number of requests can be processed by the server at any given time. Any requests exceeding this number will be discarded and issued an "error 429". This is to ensure that the server is not overloaded and to guarantee maintenance of server efficiency and speed.

Token Bucket Algorithm

This model is similar to the leaking bucket as there is a hypothetical bucket which serves as the rate limiter. This bucket serves to manage tokens and new tokens are added periodically into the bucket.

When a request is made, a token gets discarded, and this continues until all the tokens in the bucket get depleted. At that point, any request made will be discarded with an "error 429". This also helps to prevent server congestion and ensure maximal efficiency.

Rate Limiting Best Practices

Efficient web API development is mainly achievable by following the best API development practices. To maximize the use of a rate limiter as an API security measure, the following needs to be implemented.

• Firstly, choose a compatible rate-limiting algorithm. Having a strong rate-limiting algorithm in place will be essential to achieve the desired result. Choosing the best rate-limiting algorithm in sync with your API endpoint will also be needed.

• Ensure that the limit sets are within the reasonable limit ranges. Having arbitrary rate limit parameters set could negatively affect user experience and that could defeat its purpose. Setting a reasonable time limit to maximize user experience and militate against attacks has proven to be much more effective.

• Ensure efficient error handling and provide necessary feedback to the client. The default rate limiting error code is error code 429. Appropriate handling of the errors that occur during the API usage especially due to the abuse of the API will be necessary to provide the necessary feedback to the user.

• Implement flexible rate-limiting mechanisms across several parameters. Setting a fixed time interval across all endpoints seems to be a bad practice as some API endpoints are much more data-sensitive than others. Hence, having a flexible rate limiter that sets parameters in order of relevance helps to maximize server efficiency and ensure security.

• Ensure the provision of appropriate application logging, monitoring and observability tools. Having in place API metrics, logging, monitoring and observability tools also helps to serve as an additional security hack for Web APIs as they help monitor server activity, and through the use of monitoring alerts, notify the server developer when suspicious requests are detected on the server.

• Ensure synchronicity of rate limiting and other API security measures. Proper synchronicity of rate limiters with other API security hacks should be harnessed to potentiate the API security measures. Adequate knowledge of the security measures and expertise is needed in order not to counteract the existing security measures.

• Ensure proper API documentation. Adequate API documentation is also needed to ensure users, other developers and clients alike are aware of the rate-limiting practice in place to ensure compliance with the rate-limiting rules.

Conclusion

In conclusion, we have highlighted rate limiting as an important API security hack and some of it's real life use cases.

Feel free to check out my other articles here. Till next time, keep on coding!

Tomi Tokko developed this course. He's a popular teacher, and despite his young age has taught many courses over the past 4 years.

This course not only enhances your backend development skills but also gives you a taste of combining frontend technologies to create full-fledged applications.

Tomi offers an engaging and detailed course that guides you through building three projects.

Here are the projects you will build:

1. AI Blog Article Generator: Dive into the world of artificial intelligence as you create a tool that generates blog articles. You will learn about the integration of AI in web development.
2. Netflix Clone: Build a clone of Netflix, where you'll implement user authentication, video streaming, and a dynamic, responsive user interface.
3. Spotify Clone: Create a music streaming platform, learning how to manage audio files, user playlists, and real-time data streaming.

And here are the technologies you will use:

• Python: Known for its readability and efficiency, Python is a favorite among backend developers. You'll use Python to build robust backend logic and handle server-side operations.
• Django: This high-level Python web framework encourages rapid development and clean, pragmatic design. With Django, you'll structure your backend, manage databases, and ensure your applications are secure and scalable.
• JavaScript: While primarily known for frontend development, JavaScript's role in this course is to add interactivity and enhance the user experience of your applications.
• PostgreSQL: This powerful, open-source object-relational database system offers reliability, feature robustness, and performance. You'll use PostgreSQL to manage and query your application data efficiently.
• Tailwind CSS: A utility-first CSS framework, Tailwind CSS enables you to style your applications without leaving your HTML. It's a powerful tool for designing responsive and visually appealing user interfaces.

Whether you're new to backend development or looking to expand your skill set, this course offers a structured and detailed path to mastering essential technologies and concepts. Watch the full course on the freeCodeCamp.org YouTube channel (10-hour watch).

Imagine being able to build full-featured web applications, handle large-scale data analysis, and create robust architecture designs using Python. With its vast ecosystem of libraries and built-in support for machine learning, Python has become the go-to programming language for back-end development.

In this guide, we’ll explore everything you need to know about Python back-end development. We’ll discuss the role and responsibilities of a Python back-end developer, popular web frameworks, and the latest trends in the industry. We’ll also explore real-world applications and case studies, giving you practical insights into building efficient and scalable back-end systems.

Whether you’re interested in data-driven applications or API development, this tutorial will provide you with valuable knowledge and resources to excel in the world of Python back-end development. So, let’s get started.

Table Of Contents

1. How to Launch Your Career as a Python Back-End Developer
2. What is Python Back-end Development?
3. Python Back-end Developer Skills
4. Python Back-end Career Paths
5. Recommended Python Libraries for Back-end Development
6. How to Become a Python Back-end Developer
7. Database Management in Python
8. Security and Authentication in Back-end Development
9. How to Prepare for a Back-end Python Developer Job
10. Conclusion

How to Launch Your Career as a Python Back-End Developer

Are you ready to embark on an exciting journey as a Python back-end developer? Python's incredible versatility and power have made it a standout choice in the tech industry, especially for back-end development. This guide is your first step toward a flourishing career in this field.

Here are some of the topics we'll cover in this handbook:

Python's Critical Role in Back-End Development

Python, often thought of as a front-end language, is actually a powerhouse in back-end development. Its clear syntax, robust support, and expansive library ecosystem are pivotal in crafting efficient, scalable web applications.

So try to dispel any myths about Python's performance limitations – advancements in interpreter technology and optimized libraries mean Python adeptly manages large code bases and numerous concurrent connections.

Career Pathways in Python Back-End Development

As a Python back-end developer, numerous career avenues await you. From junior developers to system architects, Python opens doors to diverse roles. Your expertise in Python can lead you to high-demand sectors, where your skills in data analysis, machine learning, and API development are highly valued.

Industry Demand and Career Growth

The demand for skilled Python back-end developers is soaring. Businesses across various industries seek professionals who can navigate Python's capabilities to solve complex challenges. By learning Python, you position yourself at the forefront of an ever-growing job market.

Community Involvement and Networking

Engage with the Python community. Participate in forums, contribute to open-source projects, and network with fellow developers. This involvement is not just about learning – it's about building relationships that can propel your career forward.

Structured Learning for Success

Begin your Python journey with a structured learning plan. Start with the basics – Python's syntax and features – and utilize the wealth of online tutorials and resources available.

Then you can progress to exploring web frameworks like Django, Flask, and FastAPI. Each framework has its unique strengths and will equip you with the skills to build complex web applications.

Build a Personal Portfolio

Create a portfolio that showcases your Python projects, from simple to complex. These projects are proof of your skills and will serve as a cornerstone for your professional growth. They are not just demonstrations of your ability, but valuable learning experiences.

Seek Mentorship and Continuous Learning

Remember, becoming proficient in Python back-end development is a journey of continuous learning and practice. Seek mentorship from experienced developers and remain curious and motivated. Your path to becoming a Python back-end developer is filled with endless possibilities and opportunities for growth.

Embrace this journey with enthusiasm and determination, and watch as doors open in the dynamic world of Python back-end development.

What is Python Back-end Development?

Python allows you to create innovative backend solutions. This versatile language is swiftly becoming a go-to for developers seeking to create powerful server-side components for web applications.

But its simplicity and versatility are just the beginning. Python enables you to design solutions that make real-world impacts, bringing efficiency and innovation to various industries.

The Expansive Role of Python in Crafting Web Applications

When you write server-side code using Python, you're directly handling requests from the front-end and engaging with databases and other resources.

The beauty of Python lies in its support for frameworks like Django, Flask, and FastAPI, simplifying the development of complex web applications. This is where you transform your code into functional, user-centric web applications.

How to Harness Python's Advantages for Career Advancement

Python is renowned for its readability and clean syntax, making it an ideal language for both beginners and experienced developers. Its extensive library ecosystem equips you with tools to expedite development processes.

But there's more – Python's prowess in machine learning and data analysis opens up avenues in sophisticated back-end operations. This versatility allows for seamless integration with various databases, both SQL and NoSQL, catering to a wide array of applications.

As a career coach, I recommend leveraging these advantages by exploring real-world applications and community projects. This practical exposure not only enhances your skill set but also showcases your capabilities to potential employers.

How to Tackle Python's Misconceptions with Confidence

Python, like any language, has its debates. Concerns about its performance, particularly in comparison to statically typed languages, do arise.

But Python's dynamic nature and interpreted execution, when harnessed with efficient libraries and optimization techniques, are more than capable of powering large-scale web applications. Its multi-threading and asynchronous features allow handling numerous concurrent connections, a vital aspect of modern web development.

You can enhance your expertise in these areas through specialized training and community workshops, focusing on Python’s advanced features and performance optimization.

Stay Focused and Curious

As you embark on this journey, remember that Python back-end development is a continuous learning curve. Stay updated with the latest trends and advancements in Python. Engage in lifelong learning, explore new challenges, and actively seek opportunities for growth.

Building a strong foundation in Python, coupled with an eagerness to adapt and innovate, will set you apart in this dynamic field.

So as you can see, Python back-end development is not just about coding – it's about creating, innovating, and constantly evolving. By embracing the versatility and power of Python and staying committed to continuous learning and skill enhancement, you are opening doors to a thriving career in technology.

Python Back-End Developer Skills

Aspiring Python back-end developers need to acquire a specific set of skills and competencies to excel in their roles. These skills encompass a combination of programming knowledge, familiarity with web frameworks, database management, and an understanding of security and authentication protocols.

Proficiency in Python Programming

Being proficient in Python is an essential skill for any Python back-end developer. A deep understanding of the language’s syntax, features, and best practices allows developers to write efficient and reliable code.

Python’s simplicity, readability, and vast ecosystem of libraries make it a popular choice for back-end development.

If you want to get started learning Python, freeCodeCamp has a couple certifications just for you. Check out Scientific Computing with Python here, and Data Analysis with Python here.

There are also a number of great beginner-friendly Python courses on the freeCodeCamp YouTube channel – like this one from Harvard.

And if you want some great Python coding examples, check out these handbooks here and here.

Learn Python Web Frameworks

Python offers several powerful web frameworks that simplify the development process and provide built-in support for building full-featured web applications.

Popular frameworks like Django, Flask, and FastAPI offer different approaches to web development, allowing developers to choose the one that best suits their project’s requirements.

You can learn all about Django from the famous Dr. Chuck in this course on freeCodeCamp's YouTube channel.

And you can learn about Flask by building an e-commerce app in this course.

And this handbook will teach you FastAPI basics while this course will solidify your knowledge.

Learn Database Management

Python back-end developers must have a good grasp of database management, including both SQL (Structured Query Language) and NoSQL (Non-relational) databases.

Understanding how to design database schemas, write efficient queries, and ensure data integrity is crucial for building robust back-end systems.

Here's a course that'll teach you SQL basics for data storage and retrieval.

And here's a quick tutorial that teaches you how to use Python to read data from and write data to a table in SQL databases.

For NoSQL databases, here's a full course that teaches you all about how they work.

Learn Security and Authentication

As back-end developers handle sensitive user data and interact with external systems, they must possess knowledge of security and authentication protocols. This includes understanding concepts like encryption, secure communication, user authentication, and authorization. Familiarity with frameworks and libraries that provide security features is also essential.

Here's a helpful tutorial on web security in Django, and here's a tutorial about setting up authentication in a Flask app. Hopefully these will give you some basic ideas and inspire you to learn more if you need to do so.

Developers who possess these skills will be well-equipped to tackle the challenges of back-end development using Python.

By continuously honing their skills and staying updated with the latest trends and best practices, Python back-end developers can deliver high-quality and secure web applications.

Python Back-End Career Paths

As a Python back-end developer, you have a wide range of career paths to choose from. The demand for skilled Python developers is on the rise, and there are numerous opportunities in various fields.

Here are some of the career paths you can explore:

Web Development

Web development is one of the most common career paths for Python back-end developers. You can work on building and maintaining websites, web applications, and content management systems.

With Python’s versatility and the availability of web frameworks like Django and Flask, you can create robust and scalable web solutions.

Data Analysis

Python is widely used for data analysis and manipulation. As a Python back-end developer, you can work with large datasets, perform statistical analysis, and create data visualizations. Tools like NumPy, Pandas, and Matplotlib are commonly used in this field.

Machine Learning

Python is the language of choice for machine learning and artificial intelligence. As a Python back-end developer, you can develop machine learning models, implement deep learning algorithms, and work on natural language processing projects. Libraries like TensorFlow and scikit-learn are popular for this purpose.

DevOps and Deployment

Python is also valuable in the DevOps field. You can work on automating deployment processes, managing infrastructure, and building scalable systems. With tools like Docker and Kubernetes, you can streamline the deployment and maintenance of applications.

Backend Architecture and Design

If you have a strong grasp of backend architecture and design principles, you can pursue a career as a backend architect or technical lead. You will be responsible for designing scalable, efficient, and secure backend systems and ensuring smooth integration with frontend components.

The salaries for Python back-end developers vary based on experience, location, and industry. But the average salary for Python professionals is quite competitive, with ample room for growth and advancement.

Remember, this is just a glimpse of the career paths available to Python back-end developers. With the continuous growth and evolution of technology, new opportunities are constantly emerging. Stay updated with the latest trends and technologies to make the most out of your career in Python back-end development.

Recommended Python Libraries for Backend Development

When it comes to Python backend development, leveraging the right libraries can significantly streamline the development process and enhance the functionality of your web applications.

Here are five Python tools that can simplify and optimize your backend development workflow:

Flask

Flask is a micro web framework that offers simplicity and ease of use without sacrificing functionality. It provides a minimalistic syntax and focuses on the core features of web development, making it an excellent choice for small to medium-sized projects.

Flask’s modular design allows developers to add the necessary extensions to create a full-featured web application. It is highly extensible and beginner-friendly, making it a great library for developers getting started with Python backend development.

FastAPI

FastAPI is relatively new, but it's rapidly gaining popularity as a high-performance web framework. It combines modern, asynchronous programming techniques with the simplicity and productivity of the Python language.

FastAPI leverages the power of type hints and automatic documentation generation, making it easier for developers to build robust and well-documented APIs. Its speed and efficiency make it an excellent choice for building scalable backend systems.

Django

Django is a widely used, batteries-included web framework that simplifies the process of building complex web applications. It provides built-in support for various functionalities, including authentication, database management, and URL routing.

Django’s robustness and scalability have made it a preferred choice for a wide range of applications, from basic websites to large-scale enterprise systems.

Tornado

Tornado is a popular open-source web framework that excels in handling high-performance, real-time web applications. Its non-blocking architecture makes it ideal for building scalable applications that require handling a large number of concurrent connections.

Tornado’s lightweight and efficient design, along with its support for long polling and websockets, makes it a valuable tool for developers working on real-time applications.

Pyramid

Pyramid is a powerful and flexible web framework that prioritizes simplicity and speed. It provides a solid foundation for building both small and large-scale web applications.

Pyramid follows the principle of “pay only for what you need,” allowing developers to choose and integrate various components according to their requirements.

With its extensive documentation and active community, Pyramid is a popular choice among Python developers.

By utilizing these libraries in your Python backend development projects, you can benefit from their extensive features and community support. Each library has its own strengths, so it’s essential to evaluate your specific project requirements and choose the one that best fits your needs.

How to Become a Python Back-End Developer

Becoming a Python back-end developer requires a combination of technical skills, practical experience, and continuous learning. Here are some actionable steps and guidance to help you kickstart your journey:

Fortify Your Python Foundations

Embarking on your Python back-end development journey begins with a deep dive into Python’s fundamentals. Set specific, measurable goals to master Python’s syntax and core concepts.

For instance, aim to understand basic syntax within the first couple weeks, or complete a mini-project using object-oriented principles within a month.

Supplement your learning with practical exercises at the end of each session. These exercises reinforce theoretical knowledge and aid in retention.

It's also a good idea to engage in peer review and collaboration by joining coding forums or local Python groups. This exposure can offer new perspectives and enhance your problem-solving skills, forming a solid foundation for your journey.

Expand Your Web Development Horizons

As you progress, broaden your understanding of web development essentials. While your focus may be on back-end development, a well-rounded knowledge of front-end technologies like HTML, CSS, and JavaScript is crucial.

Explore resources or short courses that offer a comprehensive overview of these front-end technologies. You can also analyze real-world applications to see how these technologies are applied in practice.

This approach provides a holistic view of web development, preparing you to create more integrated and efficient back-end systems.

Learn Python Web Frameworks

Venture further into Python’s powerful web frameworks, such as Django, Flask, and FastAPI that we discussed above. Each framework has its unique strengths, so consider undertaking a comparative project where you utilize different frameworks for the same application.

This practical experience will deepen your understanding of each framework’s capabilities.

Engage actively in community forums, workshops, and contribute to open-source projects related to these frameworks. Such community engagement is not only enriching but also keeps you abreast of the latest developments and best practices in Python web frameworks.

Learn Database Management

Database management is a critical aspect of back-end development. Familiarize yourself with both SQL and NoSQL databases – we'll discuss these in more detail in subsequent chapters.

Hands-on projects involving the setup and management of databases will provide invaluable experience. Incorporate case studies of real-world database management scenarios to understand the challenges and best practices in this field. This knowledge is crucial for storing, retrieving, and manipulating data efficiently in your applications.

Create a Personal Portfolio

Building a personal portfolio is essential in showcasing your skills. Select projects that not only demonstrate your technical prowess but also align with your career aspirations.

For instance, if data science intrigues you, focus on projects that involve data analysis and manipulation. Regular portfolio reviews by mentors or peers can provide constructive feedback, helping you refine your portfolio to better reflect your capabilities and aspirations.

Go Through Certifications and Courses

Enhance your skills and professional standing by enrolling in relevant courses and obtaining certifications. Choose educational paths that align with your career goals, ensuring that your learning journey strategically builds towards your desired role. Customize your learning path to fit your personal learning style and pace, making your educational experience more effective and enjoyable.

Keep Pace with Industry Trends

Stay updated with the latest trends in Python back-end development. Seek mentorship from industry professionals who can provide insights into current industry trends and offer valuable career advice. Regularly subscribe to industry newsletters, listen to podcasts, and follow influencers in the Python development space to stay informed.

Your path to becoming a skilled Python back-end developer involves a combination of solid foundational learning, practical application, continuous education, and active community engagement. Each step should be approached with dedication and enthusiasm, keeping you inspired and informed as you explore the vast possibilities of Python and back-end development.

Now let's get into some of the more nitty-gritty details of what it takes to work with Python on the back end.

Database Management in Python

Database management is a crucial aspect of building robust and efficient applications in Python. It involves effectively organizing and manipulating data to support various operations.

In this chapter, we will explore the definition and significance of database management and provide an overview of SQL and NoSQL databases in Python.

Database management refers to the process of organizing, storing, and retrieving data in a structured manner. It plays a vital role in ensuring data integrity, enabling efficient data access, and facilitating data analysis.

By effectively managing databases, developers can create scalable and reliable systems that support various business needs.

SQL vs. NoSQL: Making the Informed Choice

In the realm of databases, understanding the distinction between SQL and NoSQL is crucial.

SQL databases shine with their structured query language, offering unparalleled precision for applications that rely on complex queries and transactional integrity. They are the bedrock for scenarios demanding strict data consistency, detailed relationships, and clear schema definitions.

On the other hand, NoSQL databases bring flexibility to the table, catering to applications that deal with large volumes of unstructured data or need to scale rapidly. Their schema-less nature allows for quick adaptations and supports a variety of data models, including key-value, document, wide-column, and graph formats.

The decision between SQL and NoSQL hinges on the specific needs of your project. If your application requires meticulous data organization and complex transactions, SQL is the dependable choice. For projects that must scale dynamically or handle a diverse set of data structures, NoSQL offers the agility and scalability needed.

Choose wisely, focusing on the demands of your application and the data it will handle. Both SQL and NoSQL databases have their place in the technology landscape, each bringing strengths that, when matched with the right project requirements, can significantly enhance your application's performance and scalability.

SQL Databases in Python

SQL databases play a crucial role in storing and managing data in Python applications. They provide a structured approach to data storage, utilizing tables with predefined schemas.

With SQL databases, you can efficiently organize and retrieve data, making them ideal for applications that require complex querying and relationships between data entities.

Let's look at an example.

When working with SQL databases in Python, you can use libraries like sqlite3 and SQLAlchemy to connect to the database and perform operations.

For instance, you can establish a connection to an SQLite database using sqlite3.connect() and then execute SQL queries to retrieve and manipulate data.

Here, we're creating a table called users to store user information:

import sqlite3

# Connect to the database
conn = sqlite3.connect('example.db')

# Create a cursor object
cursor = conn.cursor()

# Execute SQL queries
cursor.execute("SELECT * FROM users")
users = cursor.fetchall()

# Print the results
for user in users:
    print(user)

# Close the connection
conn.close()

Connecting to SQL databases using Python libraries like sqlite3 and SQLAlchemy

To connect to SQL databases in Python, you can leverage libraries such as sqlite3 and SQLAlchemy. These libraries provide convenient APIs to establish connections, execute queries, and handle database operations.

Designing database schemas for SQL databases

Designing an effective database schema is crucial for SQL databases. A well-designed schema ensures efficient data organization, optimal querying performance, and data integrity. When designing a schema, consider the relationships between different entities and define appropriate tables, columns, and constraints.

Writing efficient SQL queries and ensure data integrity

To maximize performance and ensure data integrity, it is important to write efficient SQL queries. Consider using indexes, optimizing joins, and using appropriate SQL clauses to retrieve the required data. You can also enforce data integrity by defining constraints, such as primary key and foreign key constraints.

By understanding SQL databases, connecting to them using libraries like sqlite3 and SQLAlchemy, designing effective schemas, and writing efficient queries, you can effectively manage data storage in your Python applications.

NoSQL Databases in Python

NoSQL databases provide a flexible and scalable approach to data storage, making them suitable for various use cases in Python.

One popular NoSQL database is MongoDB, which allows for storing data in a document format. This makes it ideal for scenarios where data structures may evolve or have varying attributes.

For example, let's say you are working on a Python project that involves analyzing social media data. By using MongoDB and its document-oriented model, you can easily store and retrieve data such as user profiles, posts, and comments, without the need for a predefined schema. This flexibility allows for efficient handling of dynamic data.

Let's look at an example. We can use the pymongo library to connect and interact with the database. Here's an example of inserting a document into a collection called products:

from pymongo import MongoClient

# Connect to the MongoDB database
client = MongoClient('mongodb://localhost:27017/')

# Access the database
db = client['social_media']

# Access the collection for user profiles
profiles_collection = db['profiles']

# Insert a new user profile document
profile = {
    'username': 'john_doe',
    'name': 'John Doe',
    'followers': 1000
}
profiles_collection.insert_one(profile)

# Retrieve all user profiles
profiles = profiles_collection.find()

# Print the user profiles
for profile in profiles:
    print(profile)

# Close the connection
client.close()

How Do NoSQL Databases Work?

NoSQL databases represent a broad class of database management systems that differ significantly from traditional relational database systems (RDBMS). These databases are designed to excel in areas where relational databases might struggle, particularly in handling large volumes of unstructured or semi-structured data, accommodating rapid scaling, and facilitating agile development practices that require schema flexibility.

Unlike SQL databases that store data in tables with predefined schemas, NoSQL databases use a variety of data models, including key-value pairs, documents, wide-column stores, and graphs. This diversity allows them to store and manage data in a format that is closer to the data's native structure, enhancing the efficiency of data retrieval and manipulation.

As a result, NoSQL databases have become a popular choice for developing modern applications in web, mobile, and IoT domains, where the ability to handle vast amounts of varied data types and structures quickly is paramount.

One of the key strengths of NoSQL databases is their scalability. They are designed to scale out by distributing data across multiple servers, often in a distributed system environment. This contrasts with the scale-up approach of traditional RDBMS, where a single server's resources are increased.

NoSQL's distributed architecture enables horizontal scalability, improving performance and availability while managing the cost implications of dealing with big data.

Furthermore, NoSQL databases often provide flexible schemas, allowing developers to alter the data structure as the application evolves without needing to predefine the schema. This flexibility facilitates rapid development and iteration, making NoSQL databases particularly suited to projects with evolving data models or projects that must go to market quickly.

Connecting to NoSQL Databases with Python Libraries

Diving into NoSQL databases with Python opens a world of possibilities for handling diverse data types and structures. Python offers a variety of libraries tailored to different NoSQL databases, such as pymongo for MongoDB, redis-py for Redis, and cassandra-driver for Cassandra. Each library simplifies the connection process, enabling developers to interact with the database using Pythonic conventions.

Establishing a connection typically involves specifying the database URI and credentials, after which you can perform CRUD operations seamlessly. This step is crucial for leveraging the full potential of NoSQL databases in your Python applications, providing the flexibility to work with data as your project requires.

Designing Data Models for NoSQL Databases

When it comes to NoSQL databases, designing effective data models is a game-changer. Unlike SQL databases that follow a strict schema, NoSQL databases allow for a more flexible data structure, which can be optimized based on the specific use case.

Whether you’re working with document, key-value, wide-column, or graph databases, understanding the strengths and use cases of each type enables you to design data models that enhance performance and scalability.

This involves considering how data is accessed and manipulated, ensuring that the model supports efficient querying and updates. Effective data modeling in NoSQL databases paves the way for building dynamic applications that can adapt to changing requirements.

Writing Effective Queries in NoSQL Databases

To tap into the power of NoSQL databases, writing effective queries is key. Each type of NoSQL database offers unique querying capabilities, from MongoDB’s BSON-based syntax to the graph traversal languages of graph databases. Mastering these query languages allows developers to retrieve, update, and manipulate data efficiently.

The goal is to optimize query performance without compromising the flexibility and scalability that NoSQL databases provide. This requires a deep understanding of the database’s indexing capabilities and query execution plan, enabling developers to write queries that are both powerful and efficient.

Ensuring Data Integrity in NoSQL Databases

Maintaining data integrity in NoSQL databases presents unique challenges, given their schema-less nature and eventual consistency model. But strategies such as implementing application-level validation, using database features for atomic operations, and understanding the database’s consistency guarantees can mitigate these challenges.

It’s also important to consider transaction support, where needed, to ensure that data remains consistent across operations. By carefully planning for data integrity, you can build NoSQL-backed applications that are not only scalable and flexible but also reliable, meeting the demands of modern web and mobile applications.

How to design database schemas for NoSQL databases and their types (Document, Key-Value, and so on)

When designing a database schema for a NoSQL database, it is crucial to consider the specific type of NoSQL database you are working with. For example, if you are using a document-oriented database like MongoDB, you can structure your data as JSON-like documents.

Let's say you are developing a Python application for an e-commerce platform. With a document-oriented NoSQL database, you can design your schema to represent products as documents, including attributes like name, price, and category. This schema flexibility allows for easy expansion and modification of data structures as your application evolves.

How to perform CRUD operations with NoSQL databases in Python

CRUD operations (Create, Read, Update, Delete) are fundamental tasks when working with databases. In Python, you can perform these operations with NoSQL databases using libraries like pymongo.

For example, imagine you are building a sentiment analysis application in Python, and you are using a NoSQL database to store user reviews. With pymongo, you can easily create a new document to store a user's review, retrieve reviews based on specific criteria, update review ratings, or delete unwanted reviews. This flexibility in performing CRUD operations allows you to effectively manage and manipulate data in your NoSQL database.

Database Schema Design

A well-designed database schema is of utmost importance in Python frontend and backend development, NLP, data science, and machine learning. It provides the foundation for efficient data organization and retrieval.

Let's explore techniques for creating effective database schemas and tools for visualizing and documenting them.

When designing a database schema, consider the specific requirements of your application.

Start by identifying the entities and relationships that need to be represented. For example, in an e-commerce application, you may have entities such as customers, products, and orders. Each entity can be represented as a table with columns representing specific attributes.

Next, define the relationships between the entities using foreign keys. For instance, an order can be associated with a customer and contain multiple products. By defining these relationships, you ensure data consistency and enable efficient querying.

To create a well-designed database schema, it is essential to follow best practices such as normalizing the data. This involves breaking down data into smaller, logical units to minimize redundancy and improve data integrity.

For example, instead of storing customer addresses directly in the orders table, you can create a separate table for addresses and link them using foreign keys.

You should also consider the performance implications of your schema design. Indexing key columns can significantly improve query performance by allowing the database to quickly locate specific data. Be mindful of the trade-off between the number of indexes and the overhead they introduce during data modification operations.

When it comes to visualizing and documenting the database schema, there are various tools available. One popular tool is MySQL Workbench, which provides a graphical interface for designing and visualizing database schemas. Another useful tool is Lucidchart, which allows you to create professional-looking entity-relationship diagrams.

Here's an example of using the MySQL Workbench tool to design a database schema:

Screenshot of MySQL Workbench

Let's illustrate these concepts with an example. Suppose we are building a blog application, and we need to design the database schema.

We can start by identifying the entities: authors, posts, and comments. The authors table can have columns like id, name, and email. The posts table can have columns like id, title, content, and author_id (a foreign key referencing the authors table). The comments table can have columns like id, content, post_id (a foreign key referencing the posts table), and author_id (a foreign key referencing the authors table).

import sqlite3

# Connect to the database
conn = sqlite3.connect('example.db')

# Create a cursor object
cursor = conn.cursor()

# Create the tables
cursor.execute('''CREATE TABLE authors
                  (id INTEGER PRIMARY KEY, name TEXT, email TEXT)''')

cursor.execute('''CREATE TABLE posts
                  (id INTEGER PRIMARY KEY, title TEXT, content TEXT, author_id INTEGER,
                  FOREIGN KEY(author_id) REFERENCES authors(id))''')

cursor.execute('''CREATE TABLE comments
                  (id INTEGER PRIMARY KEY, content TEXT, post_id INTEGER, author_id INTEGER,
                  FOREIGN KEY(post_id) REFERENCES posts(id),
                  FOREIGN KEY(author_id) REFERENCES authors(id))''')

# Close the connection
conn.close()

By following these techniques and utilizing appropriate tools, you can create a well-designed database schema that supports efficient data management and retrieval in Python applications. Remember to constantly evaluate and refine your schema as your application evolves and new requirements arise.

Query Optimization and Performance

Optimizing SQL queries for better performance is essential in Python frontend and backend development, NLP, data science, and machine learning. Here are some tips to improve query performance:

Efficiently use indexes

Indexes help speed up query execution by allowing the database to quickly locate specific data. Identify frequently accessed columns and create indexes on them. For example, if a column is frequently used in WHERE clauses or JOIN conditions, consider creating an index on that column.

Example: Suppose we have a table called "products" with a column "name" frequently used in search queries. We can create an index on the "name" column to improve search performance.

import sqlite3

# Create an index on the "name" column
conn = sqlite3.connect('example.db')
cursor = conn.cursor()
cursor.execute('CREATE INDEX idx_products_name ON products(name)')
conn.close()

Optimize JOIN operations

JOIN operations can be resource-intensive. To enhance performance, minimize the number of JOIN operations and optimize the join conditions. Avoid unnecessary JOINs and use appropriate join types (e.g., INNER JOIN, LEFT JOIN) based on the relationship between tables.

Example: If we have two tables, "customers" and "orders," with a one-to-many relationship, use INNER JOIN to retrieve customers who have placed orders.

import sqlite3

# Retrieve customers who have placed orders using INNER JOIN
conn = sqlite3.connect('example.db')
cursor = conn.cursor()
cursor.execute('SELECT customers.name, orders.order_number FROM customers INNER JOIN orders ON customers.id = orders.customer_id')
result = cursor.fetchall()
conn.close()

Use appropriate SQL clauses

Utilize SQL clauses like WHERE, GROUP BY, and HAVING to filter and aggregate data efficiently. Restrict the amount of data returned by applying filters early in the query.

Example: When retrieving customer data, use WHERE clauses to filter by specific criteria, such as age or location.

import sqlite3

# Retrieve customers older than 18 years
conn = sqlite3.connect('example.db')
cursor = conn.cursor()
cursor.execute('SELECT * FROM customers WHERE age > 18')
result = cursor.fetchall()
conn.close()

Minimize data retrieval

Only retrieve the necessary columns and rows to reduce the amount of data transferred between the database and application. Avoid using SELECT * and limit the result set to the required data.

Example: Instead of retrieving all columns from the "users" table, specify only the required columns, such as name and email.

import sqlite3

# Retrieve only the required columns from the "users" table
conn = sqlite3.connect('example.db')
cursor = conn.cursor()
cursor.execute('SELECT name, email FROM users')
result = cursor.fetchall()
conn.close()

Use Execution Plans

Analyzing and tuning database queries using execution plans is another valuable technique. Execution plans provide insights into how the database executes queries and can help identify performance bottlenecks. Examine the execution plan to ensure optimal query execution and identify areas for optimization.

Example: Use the EXPLAIN keyword in SQL to obtain the execution plan of a query and analyze it to identify potential performance improvements.

import sqlite3

# Retrieve the execution plan of a query
conn = sqlite3.connect('example.db')
cursor = conn.cursor()
cursor.execute('EXPLAIN SELECT * FROM products WHERE price > 100')
result = cursor.fetchall()
conn.close()

Best practices for database indexing and transaction management are crucial to ensure data integrity and optimize performance:

Identify appropriate columns for indexing

Select columns that are frequently used in WHERE clauses, JOIN conditions, and ORDER BY clauses. Avoid over-indexing, as it can introduce overhead during data modification operations.

Example: If we often search for products by category, create an index on the "category" column to speed up these queries.

import sqlite3

# Create an index on the "category" column
conn = sqlite3.connect('example.db')
cursor = conn.cursor()
cursor.execute('CREATE INDEX idx_products_category ON products(category)')
conn.close()

Manage transactions effectively

Use transactions to group multiple database operations into a single logical unit. This ensures data consistency and allows for rollback in case of errors.

Example: When updating customer information, encapsulate the update operation within a transaction to guarantee that all changes are either committed or rolled back.

import sqlite3

# Update customer information within a transaction
conn = sqlite3.connect('example.db')
cursor = conn.cursor()
cursor.execute('BEGIN')
cursor.execute('UPDATE customers SET email = ? WHERE id = ?', ('new_email@example.com', 1))
cursor.execute('COMMIT')
conn.close()

By following these practices, you can optimize SQL queries, analyze and tune execution plans, and implement effective database indexing and transaction management in Python frontend and backend development, NLP, data science, and machine learning projects.

Advanced Topics in Database Management

Exploring database sharding, replication, and clustering is essential for Python backend development. These strategies address scalability challenges by distributing data across multiple servers.

Sharding involves partitioning a database into smaller pieces, allowing for horizontal scaling. Replication creates multiple copies of a database to improve read performance and ensure data availability. Clustering connects servers to work as a single system, providing high availability and load balancing.

These strategies enable Python backend developers to build scalable, high-performance systems that can handle growing data requirements.

import pymongo

# Connect to the MongoDB database
client = pymongo.MongoClient('mongodb://localhost:27017/')

# Enable sharding for the database
admin_db = client.admin
admin_db.command('enableSharding', 'my_database')

# Shard the collection based on a specific criterion
admin_db.command('shardCollection', 'my_database.my_collection', key={'user_id': 1})

# Close the connection
client.close()

Databases are crucial for Python backend development, especially in big data and real-time applications. Python's simplicity and extensive library ecosystem make it an ideal choice for handling large amounts of data and enabling real-time data processing.

With databases, developers can efficiently manage and query big datasets, extract insights, and handle real-time data streams. Python's integration with databases like Hadoop, Apache Spark, and PostgreSQL, as well as real-time data processing frameworks like Apache Kafka and Redis, empowers developers to build scalable and responsive applications.

By understanding the role of databases in big data and real-time applications, Python backend developers can effectively handle data challenges and build efficient and reliable applications.

from pyspark.sql import SparkSession

# Create a SparkSession
spark = SparkSession.builder \\\\
    .appName('Real-Time NLP Analysis') \\\\
    .getOrCreate()

# Read data from a big data source, such as Apache Kafka
df = spark \\\\
    .readStream \\\\
    .format('kafka') \\\\
    .option('kafka.bootstrap.servers', 'localhost:9092') \\\\
    .option('subscribe', 'my_topic') \\\\
    .load()

# Perform NLP analysis on the real-time data
# ...

# Write the results to a database for further analysis
df.write \\\\
    .format('database') \\\\
    .option('url', 'jdbc:mysql://localhost:3306/my_database') \\\\
    .option('dbtable', 'nlp_results') \\\\
    .mode('append') \\\\
    .save()

# Stop the SparkSession
spark.stop()

Integrating Python applications with cloud-based database services is crucial for Python backend development. Cloud-based databases offer scalability, high availability, and easy management, making them ideal for handling the increasing demands of modern applications. By leveraging cloud-based databases, Python developers can focus on building robust and efficient backend systems without the need to manage infrastructure.

There are several benefits to using cloud-based database services in Python backend development.

First, they provide automatic backups and disaster recovery, ensuring data is protected and can be easily restored in case of any issues.

Second, these services offer built-in security features, such as encryption and access control, to protect sensitive data. Third, they enable seamless scalability, allowing applications to handle growing user bases and increasing data volumes without significant performance issues.

Cloud-based database services also provide built-in monitoring and analytics capabilities, giving you insights into the performance and usage of your applications. They also offer integration with other cloud services, such as serverless computing platforms and container orchestration tools, allowing you to build highly scalable and efficient application architectures.

Using cloud-based databases in Python backend development simplifies the deployment process. You can easily provision and configure database instances, manage database schemas, and perform backups and restores through simple APIs or web-based interfaces. This allows for faster development and deployment cycles, reducing time-to-market for new features and updates.

import boto3

# Connect to the Amazon RDS database
client = boto3.client('rds')

# Create a new database instance
client.create_db_instance(
    DBInstanceIdentifier='my-instance',
    Engine='mysql',
    DBInstanceClass='db.t2.micro',
    AllocatedStorage=10,
    MasterUsername='admin',
    MasterUserPassword='password'
)

# Connect to the database
conn = pymysql.connect(
    host='my-instance.abc123.us-west-2.rds.amazonaws.com',
    user='admin',
    password='password',
    database='my_database'
)

# Perform database operations
# ...

# Close the connection
conn.close()

By examining this example code, I hope you can better understand how to implement database sharding and how to leverage databases in big data and real-time applications. You should also understand how to integrate Python applications with cloud-based database services in Python frontend and backend development, NLP, data science, and machine learning projects.

Security and Authentication in Back-End Development

In back-end development, security is essential for protecting data integrity and confidentiality. Neglecting security can lead to serious issues like data breaches and loss of user trust. To prevent this, a security-first approach in the development process is vital.

Let's briefly go over some key security practices with code examples.

User Authentication:

Implementing strong authentication methods, such as passwords, tokens, and multi-factor authentication, is crucial to prevent unauthorized access. For instance, multi-factor authentication adds an additional security layer beyond just passwords.

Here's an example of implementing User Authentication in Python:

# Password-based authentication
def authenticate_user(username, password):
    # Check if username and password match in the database
    # Return True if authentication is successful, False otherwise

# Token-based authentication
def generate_token(user_id):
    # Generate a unique token for the user
    # Save the token in the database along with the user ID
    # Return the generated token

def verify_token(token):
    # Check if the token exists in the database
    # Return the associated user ID if the token is valid, None otherwise

# Multi-factor authentication
def send_verification_code(user_id):
    # Generate and send a verification code to the user
    # Store the verification code in the database

def verify_verification_code(user_id, code):
    # Check if the provided code matches the one stored in the database
    # Return True if the code is valid, False otherwise

User Authorization

Managing user permissions ensures that individuals access only what they're allowed to, maintaining data security. For example, administrative users may access more sensitive data than regular users.

Here's a Python code sample of implementing User Authorization:

# Role-based access control
def check_user_permission(user_id, resource_id):
    # Query the database to check if the user has permission to access the resource
    # Return True if the user has permission, False otherwise

# Attribute-based access control
def check_user_access(user_id, resource_id):
    # Query the database to retrieve the user's attributes and resource attributes
    # Apply access control policies based on the attributes
    # Return True if the user has access, False otherwise

Data Encryption

Protecting data in transit and at rest through encryption safeguards it from unauthorized access. Using robust algorithms like AES is a common practice.

Here's an example of Data Encryption in Python:

import hashlib
from Crypto.Cipher import AES

# Encrypt data
def encrypt_data(data, key):
    cipher = AES.new(key, AES.MODE_ECB)
    encrypted_data = cipher.encrypt(data)
    return encrypted_data

# Decrypt data
def decrypt_data(encrypted_data, key):
    cipher = AES.new(key, AES.MODE_ECB)
    decrypted_data = cipher.decrypt(encrypted_data)
    return decrypted_data

Secure Communication Protocols

Utilizing HTTPS and TLS ensures encrypted and secure data transmission, especially when handling sensitive user information.

Here's an example of a Secure Communication Protocol implemented in Python:

import requests

# Make a secure HTTPS request
response = requests.get('<https://example.com>')

# Use TLS/SSL certificates for secure communication
app.run(ssl_context='adhoc')

Secure Storage and Data Handling

Encrypting stored data and employing secure storage technologies are necessary for maintaining data confidentiality.

Here's an example of Secure Storage and Data Handling in Python:

import bcrypt

# Hash passwords for secure storage
def hash_password(password):
    hashed_password = bcrypt.hashpw(password.encode('utf-8'), bcrypt.gensalt())
    return hashed_password

# Verify hashed passwords
def verify_password(password, hashed_password):
    return bcrypt.checkpw(password.encode('utf-8'), hashed_password)

Key Management and Access Controls

Proper management of encryption keys and access credentials prevents unauthorized resource access. Role-based access control is an effective strategy here.

Here's an example of Key Management and Access Controls in Python:

# Key management
def generate_encryption_key():
    # Generate a secure encryption key
    # Store the key securely, such as in a key management system or hardware security module

# Access controls
def check_access(user_id, resource_id):
    # Query the database to check if the user has access to the resource based on their role or attributes
    # Return True if the user has access, False otherwise

Regular Security Testing

Conducting audits and penetration testing helps identify and rectify vulnerabilities. Adhering to industry standards and regulations is also crucial for system security.

Here's an example of how to perform regular Security Testing in Python:

import unittest
from unittest.mock import patch
from app import app

# Example unit test with mock patching
class APITestCase(unittest.TestCase):
    def test_secure_endpoint(self):
        with patch('app.authenticate_user') as mock_authenticate:
            mock_authenticate.return_value = True
            response = app.test_client().get('/secure-endpoint')
            self.assertEqual(response.status_code, 200)
            self.assertEqual(response.data, b'Success')

Remember to customize and adapt all these code snippets to your specific application's needs. These examples provide a starting point for implementing the security practices in your back-end development projects.

Stay Informed

Keeping up with the latest security trends and threats enables continuous improvement in security practices.

Security in back-end development involves robust authentication, careful user authorization, data encryption, secure protocols, vigilant storage practices, stringent access controls, ongoing testing, and staying current with security advancements. These measures collectively ensure the creation of secure and trustworthy applications.

Fundamentals of Web Security

In back-end development, a deep understanding of security threats is key to implementing effective measures. Embracing a security-first approach ensures data integrity and confidentiality.

Key security challenges include:

Cross-Site Scripting (XSS)

Cross-Site Scripting attacks occur when an attacker manages to inject malicious scripts into content that is then served to a user. These scripts execute within the victim's browser under the guise of a trusted site, potentially stealing cookies, session tokens, or other sensitive information that can be used to impersonate the victim.

Mitigation Strategies:

Sanitizing User Inputs: Ensure that all user-provided data is sanitized, which means filtering out any potentially malicious content before it's used or displayed on your site.

This often involves stripping out HTML tags or JavaScript from inputs that are not expected to contain such content.

Using Output Encoding: When displaying user-generated content, employ output encoding techniques.

This involves converting special characters into their HTML or URL encoded equivalents so that they are rendered harmless. For example, characters like <, >, and " are converted to <, >, and ", respectively.

SQL Injection

SQL Injection attacks involve inserting or "injecting" a SQL query via the input data from the client to the application. A successful SQL injection exploit can read sensitive data from the database, modify database data, execute administrative operations on the database, and in some cases, issue commands to the operating system.

Mitigation Strategies:

Parameterized Queries: Use parameterized queries (also known as prepared statements) whenever interacting with the database. This approach allows the database to distinguish between code and data, regardless of the input content.

Avoid Direct SQL Query Concatenation: Never build SQL queries by directly concatenating user input with SQL code. This practice is prone to SQL injection because it treats user input as part of the SQL syntax. Always use safe methods provided by your database access libraries to parameterize inputs.

Cross-Site Request Forgery (CSRF)

Cross-Site Request Forgery is an attack that tricks the victim into executing unwanted actions on a web application in which they're currently authenticated.

If the victim is a regular user, a successful CSRF attack can force them to perform state-changing requests like transferring funds, changing their email address, and so on. If the victim is an administrative account, CSRF can compromise the entire web application.

Mitigation Strategies:

Use CSRF Tokens: A CSRF token is a unique, secret, unpredictable value that is generated by the server-side application and transmitted to the client in such a way that it is included in a subsequent HTTP request made by the client. When the next request is made, the server checks the submitted token against the one it issued and rejects the request if the values do not match.

SameSite Cookie Attribute: Utilize the SameSite attribute in cookies, which tells the browser to only send the cookie in requests originating from the same domain as the target domain.

This can effectively prevent CSRF attacks by ensuring that authenticated session cookies are not sent along with requests initiated by third-party websites.

Implementing these strategies can significantly enhance the security of web applications by mitigating some of the most common and impactful security vulnerabilities.

Regular security audits and staying updated with the latest trends are crucial for ongoing protection. In Python development, utilizing frameworks like Flask-Security or Django’s authentication features helps in secure user authentication. They offer password hashing, token authentication, and multi-factor authentication support.

Let's get into each of these security risks in a bit more detail with some code examples.

Cross-Site Scripting (XSS):

# Sanitizing user inputs to prevent XSS attacks
def sanitize_input(input_string):
    # Remove HTML tags and special characters from the input string
    sanitized_string = input_string.replace('<', '&lt;').replace('>', '&gt;')
    return sanitized_string

# Using output encoding to prevent XSS attacks
def output_encode(input_string):
    # Encode special characters to their HTML entity representation
    encoded_string = html.escape(input_string)
    return encoded_string

SQL Injection:

import sqlite3

# Using parameterized queries to prevent SQL injection
def get_user(username):
    conn = sqlite3.connect('database.db')
    cursor = conn.cursor()

    query = "SELECT * FROM Users WHERE username = ?"
    cursor.execute(query, (username,))
    user = cursor.fetchone()

    conn.close()

    return user

Cross-Site Request Forgery (CSRF):

from flask import Flask, render_template, request, session
import secrets

app = Flask(__name__)
app.secret_key = secrets.token_hex(16)

@app.route('/login', methods=['POST'])
def login():
    # Generate and store a CSRF token in the session
    session['csrf_token'] = secrets.token_hex(16)

    # Validate the CSRF token sent in the request
    if session['csrf_token'] != request.form['csrf_token']:
        return "Invalid CSRF token"

    # Continue with the login process

    return "Login successful"

@app.route('/form', methods=['GET'])
def form():
    # Generate a CSRF token and render the form template
    csrf_token = secrets.token_hex(16)
    return render_template('form.html', csrf_token=csrf_token)

Remember to customize and adapt the code snippets to your specific application's needs. These examples provide a starting point for implementing security measures against XSS, SQL injection, and CSRF attacks in your back-end development projects.

User Authentication

User authentication is a critical process in ensuring the security of your application. It involves verifying the identity of users before granting them access to protected resources. There are several authentication mechanisms you can implement, such as passwords, tokens, and multi-factor authentication.

For passwords, it is important to enforce strong password policies that require a combination of uppercase and lowercase letters, numbers, and special characters. Also, consider implementing features like password complexity checks and password expiration to enhance security. An example of a strong password would be "P@ssw0rd123".

Tokens are another commonly used authentication mechanism. They are unique strings that are generated and issued to users upon successful authentication. Tokens can be stored securely and used for subsequent requests to authenticate the user.

An example of a token-based authentication flow is using JSON Web Tokens (JWT), where a user logs in and receives a token that is included in subsequent requests for authentication.

Multi-factor authentication (MFA) adds an extra layer of security by requiring users to provide additional verification, such as a one-time password sent to their mobile device or a fingerprint scan. Implementing MFA can significantly reduce the risk of unauthorized access to user accounts.

For example, when a user logs in, they might be prompted to enter a verification code received via SMS in addition to their password.

To ensure secure authentication, consider the following best practices:

• Implement secure storage for passwords and tokens, such as hashing passwords using strong hashing algorithms like bcrypt or Argon2.
• Utilize encryption for transmitting sensitive authentication data, such as using HTTPS for secure communication.
• Regularly review and update authentication mechanisms to address any known vulnerabilities.

Remember, user authentication is a crucial aspect of your application's security. By implementing robust authentication mechanisms and following best practices, you can protect user accounts and maintain the trust of your users.

User Authentication:

import bcrypt
import jwt

# Password-based authentication
def authenticate_user(username, password):
    # Retrieve the user's hashed password from the database
    hashed_password = get_hashed_password(username)

    # Verify the provided password against the hashed password
    if bcrypt.checkpw(password.encode('utf-8'), hashed_password):
        return True
    else:
        return False

# Token-based authentication
def generate_token(user_id):
    # Generate a JSON Web Token (JWT) with the user ID as the payload
    token = jwt.encode({'user_id': user_id}, 'secret_key', algorithm='HS256')
    return token

def verify_token(token):
    try:
        # Verify and decode the JWT
        decoded_token = jwt.decode(token, 'secret_key', algorithms=['HS256'])
        user_id = decoded_token['user_id']
        return user_id
    except jwt.InvalidTokenError:
        return None

# Multi-factor authentication
def send_verification_code(user_id):
    # Generate and send a verification code to the user
    verification_code = generate_verification_code()
    store_verification_code(user_id, verification_code)
    send_sms(user_id, verification_code)

def verify_verification_code(user_id, code):
    # Retrieve the stored verification code for the user
    stored_code = get_verification_code(user_id)

    # Check if the provided code matches the stored code
    if code == stored_code:
        return True
    else:
        return False

Remember to customize and adapt the code snippet to your specific application's needs. This example provides a starting point for implementing user authentication in your back-end development projects.

User Authorization

Defining user permissions and access levels is a crucial aspect of user authorization. In Python frontend and backend development, you can achieve this by implementing role-based access control (RBAC) or attribute-based access control (ABAC) mechanisms.

With RBAC, you assign roles to users based on their responsibilities or job titles. For example, you may have roles such as "admin," "manager," and "user." Each role has a predefined set of permissions that determine what actions a user with that role can perform. By assigning appropriate roles to users, you can control their access to different parts of the application.

On the other hand, ABAC takes a more fine-grained approach by considering various attributes of the user, the resource being accessed, and the context in which the access request is made. Attributes like user location, time of day, or user department can influence access decisions. For instance, you can define policies that allow only users from a specific department to access certain sensitive data.

To effectively manage user authorization, consider using a well-established authentication and authorization framework like Django's built-in authentication system or Flask-Security. These frameworks provide convenient methods and decorators to handle user permissions and access control.

Remember, authentication and authorization are distinct but interconnected concepts. While authentication verifies the identity of users, authorization determines what actions they are allowed to perform. Understanding the difference is crucial for building secure and reliable applications.

Example:

In a Python web application, let's say you have defined three roles: "admin," "manager," and "user."

• The "admin" role has permissions to perform administrative tasks such as creating and managing users, modifying system settings, and accessing sensitive data.
• The "manager" role can perform tasks related to managing projects, assigning tasks to team members, and generating reports.
• The "user" role has limited permissions and can perform basic actions like creating and updating their own profile, viewing project details, and submitting feedback.

By assigning the appropriate role to each user, you ensure that they have access to the necessary features and data based on their responsibilities and level of authority within the application.

Role-based access control (RBAC) example:

# Define roles and their corresponding permissions
ROLES = {
    'admin': ['create_user', 'delete_user', 'modify_settings', 'access_sensitive_data'],
    'manager': ['manage_projects', 'assign_tasks', 'generate_reports'],
    'user': ['update_profile', 'view_project_details', 'submit_feedback']
}

# Check user's role and perform actions based on permissions
def perform_action(user_role, action):
    if user_role in ROLES:
        if action in ROLES[user_role]:
            # Perform the action
            print(f"Performing {action} as {user_role}")
        else:
            print(f"Access denied. {user_role} does not have permission to perform {action}")
    else:
        print(f"Invalid role: {user_role}")

# Example usage
perform_action('admin', 'create_user')  # Performing create_user as admin
perform_action('manager', 'delete_user')  # Access denied. manager does not have permission to perform delete_user
perform_action('user', 'update_profile')  # Performing update_profile as user

Attribute-based access control (ABAC) example:

# Define user attributes and their values
USER_ATTRIBUTES = {
    'user1': {'department': 'HR', 'location': 'office1'},
    'user2': {'department': 'IT', 'location': 'office2'},
    'user3': {'department': 'Finance', 'location': 'office1'}
}

# Define resource attributes and their values
RESOURCE_ATTRIBUTES = {
    'resource1': {'department': 'HR'},
    'resource2': {'department': 'IT'},
    'resource3': {'department': 'Finance'}
}

# Check user's attributes and resource attributes to determine access
def check_access(user_id, resource_id):
    if user_id in USER_ATTRIBUTES and resource_id in RESOURCE_ATTRIBUTES:
        user_attributes = USER_ATTRIBUTES[user_id]
        resource_attributes = RESOURCE_ATTRIBUTES[resource_id]

        if user_attributes['department'] == resource_attributes['department']:
            # Grant access
            print(f"Access granted for {user_id} to {resource_id}")
        else:
            print(f"Access denied. {user_id} does not have access to {resource_id}")
    else:
        print(f"Invalid user ID or resource ID")

# Example usage
check_access('user1', 'resource1')  # Access granted for user1 to resource1
check_access('user2', 'resource3')  # Access denied. user2 does not have access to resource3
check_access('user3', 'resource2')  # Access denied. user3 does not have access to resource2

Remember to customize and adapt the code snippets to your specific application's needs. These examples provide a starting point for implementing role-based access control (RBAC) and attribute-based access control (ABAC) mechanisms in your Python backend development projects.

Encryption Techniques

In Python frontend and backend development, encryption plays a crucial role in safeguarding data. There are different types of encryption to consider: in transit, at rest, and end-to-end.

In Transit Encryption

When data is being transmitted between different systems or over a network, it is important to ensure that it is encrypted. This prevents unauthorized access and eavesdropping on sensitive information.

Implementing secure communication protocols like HTTPS and TLS is essential for encrypting data in transit. For example, you can use the requests library in Python to make API calls using HTTPS, ensuring secure transmission of data.

At Rest Encryption

When data is stored in databases or on physical storage devices, it should be encrypted to protect it from unauthorized access. Choosing the right encryption algorithms and implementing them effectively is crucial for ensuring the security of the stored data.

For instance, in Python, you can use libraries like cryptography to encrypt sensitive data before storing it in a database. By encrypting data at rest, even if the storage medium is compromised, the data remains protected.

End-to-End Encryption

End-to-end encryption provides an additional layer of security by ensuring that data remains encrypted throughout its entire journey, from the sender to the recipient. This means that only the intended recipients can decrypt and access the data.

For example, in a messaging application, end-to-end encryption ensures that only the sender and the intended recipient can read the messages, even if they pass through intermediate servers.

Implementing end-to-end encryption requires careful consideration of cryptographic protocols and key management practices.

Choosing the right encryption algorithms and implementing them effectively is crucial for maintaining the security of your application. It is recommended to use widely accepted and tested encryption algorithms like AES (Advanced Encryption Standard). Also, regularly updating encryption libraries and frameworks to the latest versions helps address any known vulnerabilities.

Remember, encryption is an important aspect of data security in both frontend and backend development. By implementing encryption techniques, you can protect sensitive data and ensure the confidentiality and integrity of your application's information.

Secure Communication Protocols

In Python frontend and backend development, ensuring secure data transmission is vital to protect sensitive information. Two key practices for achieving this are utilizing HTTPS and TLS.

Use HTTPS

When transmitting data over networks, it is crucial to use HTTPS (Hypertext Transfer Protocol Secure). HTTPS encrypts the data being transmitted, preventing unauthorized access and ensuring data integrity.

By implementing HTTPS, you can secure the communication between clients and servers, safeguarding sensitive user information.

For example, in a Python web application, you can configure your server to use HTTPS by obtaining and installing an SSL/TLS certificate.

Protect API Endpoints and Backend Services

It is essential to secure API endpoints and backend services to prevent unauthorized access and data breaches. Implementing authentication and authorization mechanisms is crucial here.

For instance, in Python backend development, you can use frameworks like Flask or Django to implement token-based authentication, where users are issued tokens upon successful authentication. These tokens are then included in subsequent requests to authenticate the user and grant access to protected endpoints.

For example, in a Python backend application, let's consider an API endpoint that allows users to retrieve sensitive user data.

To protect this endpoint, you can implement authentication using JWT (JSON Web Tokens). When a user successfully logs in, they receive a JWT that contains their identity and authorization claims. For subsequent requests to the sensitive endpoint, the user includes this JWT in the request headers. The backend server then validates and verifies the token before granting access to the protected data.

By utilizing HTTPS and implementing robust authentication mechanisms, such as token-based authentication, you can ensure secure data transmission and protect API endpoints and backend services in your Python applications.

Remember, adopting these security practices in Python frontend and backend development is crucial for safeguarding sensitive information and maintaining the trust of your users.

Secure Storage and Data Handling

In Python frontend and backend development, ensuring the secure storage of sensitive data and preventing data leaks are critical considerations. By following best practices, you can enhance data integrity and safeguard confidential information.

Best Practices for Storing Sensitive Data

When it comes to storing sensitive data, several best practices can help maintain its security:

1. Data Encryption: Encrypting sensitive data before storing it adds an extra layer of protection. Utilize cryptographic algorithms like AES (Advanced Encryption Standard) to ensure secure storage. For example, in Python, you can use libraries like cryptography to implement encryption in your application.
2. Secure Key Management: Properly managing encryption keys is crucial for data security. Store keys securely using key management systems or hardware security modules (HSMs) to prevent unauthorized access. Maintain a well-defined key rotation policy to minimize the risk of key compromise.
3. Access Controls: Implement robust access controls to restrict data access to authorized personnel only. Use role-based access control (RBAC) or attribute-based access control (ABAC) mechanisms to enforce granular permissions. Regularly review and update access control policies to align with changing requirements.
4. Secure Backup and Disaster Recovery: Regularly back up sensitive data and store backup copies securely. Test the restoration process to ensure data availability in case of emergencies. Implement disaster recovery strategies to minimize the impact of data breaches or system failures.

Example:

Let's consider an example in Python backend development. Suppose you are developing a web application that handles sensitive user data, such as personal information. To ensure secure storage, you can employ the following practices:

• Encrypt the sensitive user data using AES encryption before storing it in the database. This ensures that even if the database is compromised, the data remains protected.
• Implement secure key management by using a key management system to securely store and manage encryption keys. This prevents unauthorized access to the keys, enhancing overall data security.
• Apply access controls to restrict data access to authorized users only. For example, implement RBAC to assign different roles to users and grant them permissions based on their responsibilities and level of authority.
• Regularly back up the encrypted data and store the backup copies securely. Test the restoration process to ensure data availability in case of data loss or system failures.

By following these best practices, you can ensure the secure storage of sensitive data in your Python application, minimizing the risk of data leaks and maintaining the confidentiality of user information.

How to prevent data leaks and ensure data integrity

Preventing data leaks and maintaining data integrity are vital aspects of data security. Consider the following practices:

1. Input Validation: Validate all user inputs to prevent common vulnerabilities like SQL injection and cross-site scripting (XSS). Implement server-side input validation and sanitize user inputs to mitigate the risk of code injection attacks.
2. Secure File Uploads: Implement strict file upload validation to prevent unauthorized execution of malicious files. Validate file types, limit file sizes, and store uploaded files in a secure location to prevent unauthorized access or execution.
3. Secure Logging: Log events and errors securely to prevent sensitive information from being exposed. Avoid logging sensitive data like passwords or personally identifiable information (PII). Utilize secure logging frameworks and ensure log files are protected from unauthorized access.
4. Secure Configuration: Maintain secure configurations for your application, server, and related components. Disable unnecessary services, apply security patches promptly, and follow security best practices for server and application configurations.

Example:

Let's illustrate these practices with a Python frontend development example. Suppose you are developing a web application that accepts user inputs and performs database operations.

To prevent data leaks and ensure data integrity, you can adopt the following measures:

• Implement server-side input validation to validate and sanitize user inputs. This prevents common vulnerabilities like SQL injection and helps maintain data integrity.
• Apply strict file upload validation to ensure that only allowed file types and sizes are accepted. Store uploaded files outside the web root directory to prevent unauthorized access and execution.
• Log events and errors securely using a logging framework that supports secure logging practices. Avoid logging sensitive information, and ensure log files are protected from unauthorized access.
• Maintain secure configurations for your application and server. Regularly update and patch software components, disable unnecessary services, and follow security best practices for configurations.

By implementing these practices, you can significantly reduce the risk of data leaks and maintain the integrity of your application's data in Python frontend development.

Remember, ensuring secure data storage and preventing data leaks are critical for maintaining the trust of your users and protecting sensitive information. By following these best practices, you can enhance data security in your Python frontend and backend development projects.

Here are some code examples that show various ways to protect your Python applications:

User Authorization:

# Role-based access control
def check_user_permission(user_id, resource_id):
    # Query the database to check if the user has permission to access the resource
    user_role = get_user_role(user_id)
    resource_permissions = get_resource_permissions(resource_id)

    if user_role in resource_permissions:
        return True
    else:
        return False

# Attribute-based access control
def check_user_access(user_id, resource_id):
    # Query the database to retrieve the user's attributes and resource attributes
    user_attributes = get_user_attributes(user_id)
    resource_attributes = get_resource_attributes(resource_id)

    # Apply access control policies based on the attributes
    if user_attributes['department'] == resource_attributes['department']:
        return True
    else:
        return False

Data Encryption:

import hashlib
from Crypto.Cipher import AES

# Encrypt data
def encrypt_data(data, key):
    cipher = AES.new(key, AES.MODE_ECB)
    encrypted_data = cipher.encrypt(data)
    return encrypted_data

# Decrypt data
def decrypt_data(encrypted_data, key):
    cipher = AES.new(key, AES.MODE_ECB)
    decrypted_data = cipher.decrypt(encrypted_data)
    return decrypted_data

Secure Communication Protocols:

import requests

# Make a secure HTTPS request
response = requests.get('<https://example.com>')

# Use TLS/SSL certificates for secure communication
app.run(ssl_context='adhoc')

Secure Storage and Data Handling:

import bcrypt

# Hash passwords for secure storage
def hash_password(password):
    hashed_password = bcrypt.hashpw(password.encode('utf-8'), bcrypt.gensalt())
    return hashed_password

# Verify hashed passwords
def verify_password(password, hashed_password):
    return bcrypt.checkpw(password.encode('utf-8'), hashed_password)

Remember to customize and adapt the code snippets to your specific application's needs. These examples provide a starting point for implementing user authentication, user authorization, data encryption, secure communication protocols, and secure storage and data handling in your back-end development projects.

Key Management and Access Controls

In Python front-end and back-end development, secure key management and robust access control are crucial for data security.

Key management involves storing encryption keys in secure locations like key management systems or hardware security modules (HSMs). Regular key rotation and limiting key access to authorized personnel are also essential. Implement strong authentication, such as multi-factor authentication (MFA), to access these keys.

For access control, define granular policies based on user roles and responsibilities, using systems like role-based access control (RBAC) or attribute-based access control (ABAC). Regularly update these policies and combine them with strong authentication methods, including passwords, tokens, or MFA, to safeguard sensitive resources.

For instance, in a Python web application handling sensitive user data or patient records, these practices ensure that only authorized users can access and decrypt critical information, maintaining data integrity and user trust.

In summary, managing encryption keys securely and implementing effective access control are fundamental in Python development to protect data and maintain application integrity.

Secure Key Management:

import os
import cryptography
from cryptography.fernet import Fernet

# Generate a new encryption key
def generate_key():
    key = Fernet.generate_key()
    return key

# Store the encryption key securely
def store_key(key):
    # Store the key in a key management system or hardware security module (HSM)
    # Follow best practices for securing the key storage

# Retrieve the encryption key
def retrieve_key():
    # Retrieve the key from the secure storage
    key = get_key_from_storage()
    return key

# Example usage
key = generate_key()
store_key(key)
key = retrieve_key()

And you can always go back to the examples of Role-Based Access Control (RBAC) and Attribute-Based Access Control (ABAC) above if you need to review those security processes.

Remember to customize and adapt the code snippets to your specific application's needs. These examples provide a starting point for implementing secure key management and access control mechanisms in your Python back-end development projects.

How to Prepare for a Back-end Python Developer Job

Becoming a back-end Python developer goes beyond learning Python and its frameworks; it involves showcasing your skills and solving complex issues through a well-crafted portfolio. This subchapter guides you on preparing for a back-end Python developer role, emphasizing the importance of a portfolio that reflects your technical skills and problem-solving capabilities.

Your portfolio is crucial—it demonstrates your expertise, creativity, and commitment to Python back-end development. Choose projects that mirror your career goals and exhibit essential skills like database management, API development, and system integration.

Select projects that challenge you yet are achievable within a realistic timeline. A mix of complex and straightforward projects shows versatility. Projects like a Data Dashboard or IoT Data Processing Service highlight your ability to work with data-centric applications.

Each project is a chance to display your technical skills and innovative approach. Tackling challenges such as developing secure authentication systems or engaging user interfaces can significantly showcase your development capabilities.

How to Create a Python Back-end Portfolio

In Python back-end development, crafting a personal portfolio is crucial. It's more than just a display of your skills – it’s a showcase of your practical expertise and your ability to solve real-world problems. The projects you choose to include in your portfolio provide tangible proof of your capabilities to future employers or clients. It’s important to remember that you don’t have to excel in every possible project.

When deciding on the right projects to undertake, align them with your career goals and personal interests. Your portfolio should accurately reflect your professional aspirations in Python back-end development.

Start by identifying the skills you want to highlight. Whether it’s database management, API development, or system integration, select projects that emphasize these abilities. For example, if your aim is to work in data-centric roles, projects like a Data Dashboard or an IoT Data Processing Service would be particularly relevant.

Next, consider the complexity and scope of each project. It’s essential to find a balance – challenging yourself is good, but you also want to complete projects in a reasonable time frame. A well-rounded portfolio that includes a mix of both intricate and simpler, well-executed projects can be more impressive than one filled with unfinished, overly ambitious projects.

Additionally, choose projects that genuinely interest you. Your enthusiasm for a project can lead to more dedication and a higher quality final product. If the idea of creating a Real-time Chat Application excites you, pursue it. Your passion can be a driving force in your learning and can result in more creative and engaging projects.

Keep in mind, your portfolio is an evolving element of your professional growth. Begin with one or two projects that align with your professional objectives and personal interests, and expand your portfolio over time. Each project you undertake is a step forward in your Python back-end development career, enhancing your expertise and confidence.

So, take the initiative, select a project, and begin coding. Every line of code you write contributes not just to an application, but to your future in the dynamic field of Python back-end development.

1. Blog Platform

• Skills Demonstrated: Implementing user authentication and designing relational databases not only strengthens your foundation in security and data management but also demonstrates your capability to handle core web functionalities.
• Technical Highlights: Develop user registration and login systems using Django or Flask, highlighting your expertise in handling user data securely. Designing a database schema to manage posts and comments will showcase your proficiency with Django’s ORM or Flask with SQL extensions.
• Additional Features: Enhance your platform with post categorization, robust search functionality, and rich text editing to demonstrate your attention to detail and user experience.

2. E-commerce Website

• Skills Demonstrated: Developing an e-commerce site tests your ability to design complex databases and create APIs, crucial for handling dynamic user interactions and integrating external services.
• Technical Highlights: Build a structured product catalog and order management system, integrating RESTful services for seamless product interactions. Adding payment processing using APIs like Stripe or PayPal illustrates your adeptness in third-party integrations.
• Additional Features: Implement inventory management and user reviews, incorporating recommendation algorithms to display your skills in enhancing user engagement.

3. Social Media API

• Skills Demonstrated: Building a RESTful API for social media sharpens your skills in data modeling and secure authentication, essential for modern web applications.
• Technical Highlights: Develop robust APIs for managing user profiles and interactions. Implement OAuth for secure user authentication, and optimize data queries for efficient performance.
• Additional Features: Incorporate real-time notifications, an intelligent news feed algorithm, and media upload support to elevate your application's user experience.

4. Task Management Application

• Skills Demonstrated: Creating a task management tool focuses on your ability to implement CRUD operations and manage user sessions, integrating these with a front-end framework for a comprehensive application development experience.
• Technical Highlights: Develop functionalities for task creation, updates, and organization. Securely manage user sessions, showcasing your ability to handle sensitive data. Integrate with front-end frameworks like React or Angular to demonstrate your full-stack development capabilities.
• Additional Features: Add task prioritization, set deadlines, and reminders to demonstrate your proficiency in enhancing user productivity.

5. Real-time Chat Application

• Skills Demonstrated: This project tests your skills in asynchronous programming and handling real-time data, crucial for applications requiring immediate data reflection and interaction.
• Technical Highlights: Implement WebSocket for real-time, bidirectional communication. Efficiently manage concurrent user connections and ensure seamless message synchronization across clients.
• Additional Features: Introduce chat rooms, file sharing, and user status indicators to demonstrate your ability to create feature-rich, interactive applications.

6. Data Dashboard

• Skills Demonstrated: Building a data dashboard allows you to showcase your expertise in data fetching, processing, and visualization — key skills for handling and presenting data insights.
• Technical Highlights: Utilize Python libraries like Pandas for data manipulation and Matplotlib or Plotly for creating interactive visualizations. Efficiently fetch and process data from various sources.
• Additional Features: Implement customizable widgets and user-specific views, demonstrating your ability to create tailored user experiences.

7. Machine Learning Model Deployment

• Skills Demonstrated: Deploying a machine learning model as a web service shows your readiness to delve into the world of AI, highlighting your skills in integrating complex algorithms into practical applications.
• Technical Highlights: Deploy a pre-trained model and develop APIs for user interaction. Focus on scalability and performance, ensuring your application can handle varying loads.
• Additional Features: Add features for model retraining and incorporate user feedback loops, showcasing your commitment to continuous improvement and user-centric design.

8. IoT Data Processing Service

• Skills Demonstrated: This service demonstrates your ability to handle streaming data and process information from IoT devices, a rapidly growing field in technology.
• Technical Highlights: Develop a system to collect, process, and store IoT device data, utilizing cloud platforms like AWS or Azure for scalability and reliability.
• Additional Features: Include real-time data visualization, device management features, and anomaly detection algorithms, showcasing your innovative approach to data handling.

9. Content Management System (CMS)

• Skills Demonstrated: Creating a CMS from scratch tests your abilities in database schema design, user authentication, and implementing role-based access controls — key elements in creating secure and scalable web applications.
• Technical Highlights: Develop a flexible content system with diverse user roles and permissions. Focus on streamlined content creation, editing, and publishing workflows.
• Additional Features: Incorporate plugin or theme support, add SEO tools, and implement content versioning to demonstrate your foresight in creating a versatile and user-friendly system.

10. Online Reservation System

• Skills Demonstrated: Developing an online reservation system showcases your ability to integrate external APIs and handle complex queries, essential for creating applications that interact with various external services.
• Technical Highlights: Build a comprehensive booking and reservation management system. Seamlessly integrate with additional services like payment processing and mapping.
• Additional Features: Implement user recommendation algorithms, reservation reminders, and promotional offers, highlighting your dedication to creating an engaging and user-friendly experience.

The significance of a well-developed portfolio in Python back-end development is immense. It not only showcases your technical skills but also demonstrates your problem-solving capabilities to potential employers or clients. However, remember that quality trumps quantity. It's more effective to focus on a few projects that align with your career goals and highlight your core competencies.

Projects like a Blog Platform or an E-commerce Website are ideal for demonstrating a range of abilities in Python back-end development. Each project should reflect professional standards, including well-written code and comprehensive documentation. As you grow in your career, keep your portfolio updated with your latest work, ensuring it evolves with industry trends and best practices.

In essence, carefully select projects that showcase your strengths, and let your portfolio narratively represent your professional growth in Python back-end development. It's your key to unlocking new opportunities in this dynamic field.

Back-end Python Interview Questions

Preparing for a back-end Python developer interview requires a solid grasp of both the technical and soft skills essential for the role. This chapter is designed to arm you with a comprehensive set of questions that cover a wide range of topics, from general programming knowledge and specific Python skills to problem-solving abilities and understanding of back-end technologies.

Whether you're a seasoned developer or new to back-end development, these questions will help you assess your preparedness and identify areas for improvement.

General and Behavioral Questions

1. Can you describe your experience with back-end development?
2. How do you stay updated with the latest back-end technologies and trends?
3. Describe a challenging back-end project you worked on.
4. How do you approach debugging a complex problem?
5. Can you discuss a time you improved the performance of a web application?
6. How do you ensure the security of your web applications?
7. Describe your experience with team collaboration in development projects.
8. What's your process for testing and quality assurance?
9. How do you handle tight deadlines and pressure?
10. What motivates you in back-end development?

Technical Skills and Coding Questions

1. What languages are you most proficient in for back-end development?
2. Can you explain RESTful API design?
3. What are the differences between SQL and NoSQL databases? When would you use each?
4. How do you manage database transactions and prevent data inconsistencies?
5. What are some common HTTP methods, and how are they used?
6. Explain the concept of MVC architecture.
7. How do you optimize database queries?
8. What is containerization, and how does it benefit development?
9. Describe the process of API authentication and authorization.
10. How do you manage cross-origin resource sharing (CORS) in web applications?

Frameworks and Technologies

1. What experience do you have with frameworks like Django, Flask, or Express.js?
2. How do you handle session management in web applications?
3. Can you explain the concept of middleware in web development?
4. What is your experience with cloud services like AWS, Azure, or Google Cloud?
5. How have you used caching mechanisms to improve application performance?
6. What are some challenges you have faced using ORM tools?
7. Describe your experience with microservices architecture.
8. How do you ensure the scalability of your web applications?
9. What tools do you use for version control?
10. How do you approach mobile backend development differently from web backend?

Problem-Solving and Algorithm Questions

1. How would you design a system for handling high traffic loads?
2. Describe how you would implement a rate-limiting feature.
3. How would you handle data synchronization across different services?
4. Describe your approach to implementing a full-text search in a database.
5. Can you explain the concept of load balancing and how it works?
6. How would you design a notification system for a high-traffic application?
7. Discuss how you would secure sensitive data in transit and at rest.
8. How do you approach error handling and logging in back-end applications?
9. Describe a time when you had to refactor code for better efficiency.
10. How do you handle database schema migrations?

Specific Technologies/Concepts Questions

1. Explain the concept of stateless architecture.
2. What is the difference between SOAP and REST APIs?
3. How do you implement a WebSocket, and what are its advantages?
4. What are some best practices for API versioning?
5. How do you handle file uploads in your web applications?
6. Describe the process of integrating third-party services via APIs.
7. What is OAuth, and how do you implement it?
8. How do you handle concurrency issues in a database?
9. Explain the role of a reverse proxy in web applications.
10. What is the difference between horizontal and vertical scaling?

Scenario-Based Questions

1. How would you optimize an application with slow response times?
2. Describe how you would scale a database for a growing application.
3. How would you troubleshoot a memory leak in an application?
4. Imagine a scenario where users report frequent timeouts. How would you investigate?
5. Describe how you would handle a security breach in a web application.
6. How would you manage data consistency across distributed systems?
7. If a service is failing intermittently, how would you go about diagnosing it?
8. Explain how you would design a back-end for an e-commerce site.
9. Describe the steps you would take to migrate a monolithic application to microservices.
10. How would you design a back-end system for real-time data processing?

Advanced Conceptual Questions

1. What are idempotent operations, and why are they important?
2. Can you explain the concept of eventual consistency?
3. Describe the trade-offs of a monolithic vs. microservices architecture.
4. What is the CAP theorem, and how does it apply to database systems?
5. How do you approach dependency injection in back-end applications?
6. Explain the concept of Domain-Driven Design (DDD) in back-end development.
7. What are design patterns, and can you give examples of how you've used them?
8. How does event-driven architecture work in back-end systems?
9. Explain the role of service discovery in microservices.
10. What are some strategies for dealing with database replication lag?

Performance and Optimization Questions

1. How do you monitor the performance of your applications?
2. Can you discuss strategies for reducing server load?
3. What are some ways to handle high-volume data processing efficiently?
4. How do you approach optimizing application boot time?
5. What tools do you use for performance profiling?
6. Discuss how you would reduce the response time of a database query.
7. What strategies can be used to reduce the size of a web application's payload?
8. How do you optimize resource utilization in cloud services?
9. What are some common performance bottlenecks in web applications, and how do you address them?
10. How do you ensure efficient memory management in your applications?

Collaboration and Workflow Questions

1. How do you approach code reviews?
2. What's your experience working in Agile development environments?
3. How do you handle merge conflicts in a version control system?
4. Describe your experience with Continuous Integration/Continuous Deployment (CI/CD).
5. How do you prioritize tasks in a fast-paced development environment?
6. Describe a time you had to collaborate with front-end developers to solve a problem.
7. How do you document your code and API for other developers?
8. What strategies do you use to ensure clear communication in a remote team?
9. How do you stay aligned with product management and other stakeholders?
10. Describe your approach to mentoring or onboarding new team members.

Personal Growth and Learning Questions

1. How do you keep your programming skills sharp?
2. What's the last programming book you read, or tech talk you watched?
3. How do you approach learning a new programming language or framework?
4. Can you share an example of a recent technical challenge and how you solved it?
5. What areas of back-end development are you looking to improve in?
6. Describe a technical topic you're passionate about.
7. How do you balance work with personal development and learning?
8. What's your experience with open-source projects?
9. How do you handle feedback and criticism on your code?
10. What goals do you have for your future in back-end development?

Conclusion

As we wrap up this comprehensive exploration of Python back-end development, it's clear that the journey you're embarking on is both challenging and rewarding.

Python, with its simplicity, versatility, and powerful libraries, stands as a cornerstone for building robust back-end systems. This guide aimed to equip you with the foundational knowledge, best practices, and practical insights necessary to thrive as a Python back-end developer.

Key Takeaways:

• Python's Role: You've learned that Python is not just versatile but pivotal in back-end development. Its frameworks, like Django, Flask, and FastAPI, enable you to build efficient, scalable applications.
• Database Mastery: Understanding the nuances of SQL and NoSQL databases enhances your ability to manage data effectively, ensuring performance and scalability.
• Security First: Emphasizing security practices, from authentication to encryption, is critical. This ensures the integrity and confidentiality of data, building trust and reliability.
• Practical Portfolio: Developing a portfolio with diverse projects demonstrates your skills and problem-solving capabilities, making you stand out in the job market.
• Interview Preparedness: The detailed set of interview questions prepares you to articulate your knowledge and experience confidently, showcasing your readiness for back-end roles.

Moving Forward:

• Continuous Learning: The field of back-end development is ever-evolving. Stay curious, keep learning, and embrace the changes in technologies and methodologies.
• Community Engagement: Join forums, contribute to open-source projects, and network with professionals. The insights and connections you gain are invaluable for your growth and career advancement.
• Practical Application: Apply what you've learned in real-world scenarios. Whether through personal projects, freelancing, or in your job, hands-on experience solidifies your skills and opens up new opportunities.
• Mentorship and Sharing: As you grow, seek mentorship from experienced developers. Similarly, share your knowledge with those who are starting their journey. Teaching is a powerful way to reinforce your own learning and contribute to the community.

Embarking on your career as a Python back-end developer is a journey of continuous growth, innovation, and discovery. With the skills and knowledge you've acquired, you're well-equipped to tackle the challenges and opportunities that lie ahead. Embrace the journey with confidence, passion, and a willingness to explore new horizons in the dynamic world of Python back-end development.

Resources

Kickstart your journey in technology with our specialized program that dives into Artificial Intelligence (AI) and machine learning. This initiative is crafted to build your programming expertise, supplemented with dedicated mentorship and career guidance to pave your way in the tech industry.

To enrich your learning experience, here's a helpful selection of targeted resources:

• How to Enter Gen AI in 2024: This guide breaks down the essentials of emerging AI technologies and prepares you for future trends.
• Land Your Software Engineering Internship: This resource provides step-by-step instructions for finding and landing a valuable internship in software engineering, giving you a competitive edge.
• Machine Learning Fundamentals eBook: Begin your exploration of machine learning with this eBook, which provides a concise overview of its core principles and techniques.

For access to these resources and detailed information about our program, visit LunarTech's website. Embark on your tech career path with the right tools and support from LunarTech.

Connect with Me:

• Follow me on LinkedIn for a ton of Free Resources in CS, ML and AI
• Visit my Personal Website
• Subscribe to my The Data Science and AI Newsletter

About the Author

I'm Vahe Aslanyan, specializing in the world of computer science, data science, and artificial intelligence. Explore my work at vaheaslanyan.com. My expertise encompasses robust full-stack development and the strategic enhancement of AI products, with a focus on inventive problem-solving.

My experience includes spearheading the launch of a prestigious data science bootcamp, an endeavor that put me at the forefront of industry innovation. I've consistently aimed to revolutionize technical education, striving to set a new, universal standard.

As we close this handbook, I extend my sincere thanks for your focused engagement. Imparting my professional insights through this book has been a journey of professional reflection. Your participation has been invaluable. I anticipate these shared experiences will significantly contribute to your growth in the dynamic field of technology.

When you’re building the backend of an application, you’ve got to figure out how all the different components are going to talk to each other.

It’s like setting up a communication network for your app’s brain, and the way you do it can seriously impact how well your app performs.

But here’s the fun part: there’s no one-size-fits-all answer. The communication pattern you choose depends on what your application needs to do.

So, in this tutorial, we’re going to take a look at five different ways backend systems like to chat it up. We’ll explore what they’re good at and when you should invite them to the conversation. Let’s dive right in!

What is a Design Pattern?

Before we get into all the cool patterns, let’s talk about what a design pattern is, shall we?

Design patterns are nifty, tried-and-true reusable solutions to common problems encountered during software design and development. You can call them the cheat codes of software design and development.

They can help provide a structured approach to solving recurring challenges, offering a set of guidelines and best practices that can be adapted for various scenarios.

Now that we’ve got that down, let’s explore five of these emerging design patterns used for backend communication.

Request-Response Pattern

The request-response pattern is a fundamental building block for how the front-end and back-end of web applications chat with each other. This pattern is like a conversation between the client (say your browser) and the server, where they take turns speaking. Imagine it as a “ping-pong” of data.

Here's a diagram illustrating what that looks like:

request/response communication model

How does the Request-Response pattern work?

This pattern is all about synchronization. The client sends a request to the server, kind of like raising your hand to ask a question in class. Then it patiently waits for the server to respond before it can move on.

It’s like a polite conversation — one speaks, the other listens, and then they swap roles.

You’ve probably heard of RESTful APIs, right? Well, they’re a prime example of the request-response model in action.

When your app needs some data or wants to do something on the server, it crafts an HTTP request — say GET, POST, PUT, or DELETE (like asking nicely for a page), and sends it to specific endpoints (URLs) on the server. The server then processes your request and replies with the data you need or performs the requested action.

It’s like ordering your favorite dish from a menu — you ask, and the kitchen (server) cooks it up for you.

Interestingly, there’s more than one way to have this conversation. Besides REST, there’s GraphQL, an alternative that lets you ask for exactly the data you want. It’s like customizing your order at a restaurant — you get to pick and choose your ingredients.

It’s important to note that this pattern isn’t just limited to web applications. You’ll spot it in Remote Procedure Calls (RPCs), database queries (with the server being the client and the database, the server), and network protocols (HTTP, SMTP, FTP) to name a few. It’s like the language of communication for the web.

Benefits of the Request-Response pattern

Ease of Implementation and Simplicity: The way communication flows in this model is pretty straightforward, making it a go-to choice for many developers, especially when they’re building apps with basic interaction needs.

Flexibility and Adaptability (One Size Fits Many): The request-response pattern seamlessly fits into a wide range of contexts. You can use it for making API calls, rendering web pages on the server, fetching data from databases, and more.

Scalability: Each request from the client is handled individually, so the server can easily manage multiple requests at once. This is highly beneficial for high-traffic websites, APIs that get tons of calls, or cloud-based services.

Reliability: Since the server always sends back a response, the client can be sure its request is received and processed. This helps maintain data consistency and ensures that actions have been executed as intended even during high-traffic scenarios.

Ease of Debugging: If something goes wrong, the server kindly sends an error message with a status code stating what happened. This makes error handling easy.

Limitations of the Request-Response Pattern

Latency Problem: Because it’s a back-and-forth conversation, there’s often a waiting period. This amounts to idle periods and amplifies latency, especially when the request requires the server to perform time-consuming computing tasks.

Data Inconsistency in Case of Failures: If a failure occurs after the server has processed the request but before the response is delivered to the client, data inconsistency may result.

Complexity in Real-Time Communication: For applications that need lightning-fast real-time communication (like live streaming, gaming, or chat apps), this pattern can introduce delays and is therefore unsuitable for these use-cases.

Inefficiency in Broadcasting: In scenarios where you need to send the same data to multiple clients at once (broadcast), this pattern can be a bit inefficient. It’s like mailing individual letters instead of sending one group message.

Here’s a code example that shows the request-response pattern using Nodejs.

First, we have the server.js file. Here we’ve set up the server to listen for incoming requests from the client.

const http = require("http");

const server = http.createServer((req, res) => {
 res.statusCode = 200;
 res.setHeader("Content-Type", "text/plain");

 //check request method and receive data from client
 if (req.method === "POST") {
  let incomingMessage = "";
  req.on("data", (chunk) => {
   incomingMessage += chunk;
  });

  //write back message received from the client on the console
  req.on("end", () => {
   console.log(`Message from client: ${incomingMessage}`);
   res.end(`Hello client, message received!`);
  });
 } else {
  res.end("Hey there,  Client!\n");
 }
});

const PORT = 3030;
server.listen(PORT, () => {
 console.log(
  `Server is listening for incoming request from client on port:${PORT}`
 );
});

And here's the client.js file:

const http = require("http");

const options = {
 method: "POST",
 hostname: "localhost",
 port: 3030,
 path: "/",
};

//message to server
let messageToServer = "Hey there, server!";

//send a http request to the server
const req = http.request(options, (res) => {
 let incomingData = "";

 res.on("data", (chunk) => {
  incomingData += chunk;
 });

 res.on("end", () => {
  console.log(`Response from the server: ${incomingData}`);
 });
});

req.on("error", (error) => {
 console.log(`Error message: ${error.message}`);
});

//send message to the server
req.write(messageToServer);

//end your request
req.end();

Here's the output:

code output on the terminal

The Publish/Subscribe Pattern

Publish/Subscribe is another communication design pattern you may see on the backend. It is used in distributed systems for asynchronous communication between several (usually decoupled) components. It’s perfect for when you’ve got a bunch of components that need to work together but want to keep their distance.

Here's a diagram showing how it works:

Pub/Sub communication model

How does the Publish/Subscribe pattern work?

This pattern involves the use of message queues (often called message brokers) which serve as intermediaries between the publishers and subscribers. These message brokers group messages into something called channels (or topics).

Publishers are the components that create and send messages while subscribers are the components that receive and consume the messages.

The publishers simply toss messages (or events) into specific channels/topics within the message broker. These channels act as the distribution point for the messages. The subscribers then indicate interest by subscribing to those channels within the message broker and whenever a message or event is published to that channel, they receive a copy.

Message queuing tools like Apache Kafka and MQTT use the publish/subscribe communication pattern under the hood.

Benefits of the Publish/Subscribe pattern

Asynchronous Communication: Unlike the request-response model, pub/sub is designed to be asynchronous by nature. This makes it ideal for building real-time applications with reduced latency bottlenecks.

Loose Coupling of Components: The components in a publish/subscribe model are loosely coupled. This means that they are not tied together and can freely interact by triggering and responding to events.

Highly Scalable: There is no limit to the number of subscribers a publisher can publish events to. Also, there’s no limit to the number of publishers subscribers can subscribe to.

Independent of Language and Protocol: The Pub/Sub model can be easily integrated into any tech stack because it is language-agnostic. Also, it often supports a wide range of environments and platforms making it cross-platform compatible.

Load Balancing: In cases where multiple subscribers subscribe to a particular event, the pub/sub model can distribute the events evenly among the subscribers, providing load-balancing capabilities out of the box.

Limitations of the Publish/Subscribe pattern

Complexity in Implementation: Setting up a Pub/Sub system can be more complex than simpler communication models like the request-response pattern. You need to configure and manage message brokers, channels, and subscriptions, which can add overhead to your system.

Message Duplication: Depending on the configuration and network issues, messages can be duplicated. Subscribers might receive the same message more than once, which can lead to redundancy and extra processing.

Scalability Challenges: While Pub/Sub is highly scalable, managing extremely high volumes of messages and subscribers can become complex. You might need to consider how to distribute messages efficiently and handle massive numbers of subscribers.

Complex Error Handling: Dealing with errors in a Pub/Sub system can be challenging. Handling situations like message delivery failures or subscriber errors requires careful consideration and design.

When should you use it?

• In building features that require real-time and low latency responsiveness for the end-users, for example live chat or gaming applications for multiple players.
• In event notification systems
• In building distributed systems that rely on logging and caching

Here’s a code snippet showing a simple pub/sub implementation using the npm package pubsub-js.

Here are the contents of the pubsub.js file:

const PubSub = require("pubsub-js");

/*
create  a topic of choice.
Any subscriber that subscribes to this topic
receives the published messages
*/
const TOPIC = "chat";

//a function to publish a messgae to subscribers
function publishMessageToSubscribers(message) {
 PubSub.publish(TOPIC, message);
 console.log(`Message Published: ${message}`);
}

let subscriber1 = function (msg, data) {
 console.log(msg, data);
 console.log("subscriber1 received: ", data);
};

let subscriber2 = function (msg, data) {
 console.log(msg, data);
 console.log("subscriber2 received: ", data);
};

// Subscriber1 subscribes to the topic
PubSub.subscribe(TOPIC, subscriber1);

//subscriber2 subscribes to topic
PubSub.subscribe(TOPIC, subscriber2);

publishMessageToSubscribers("Hello subscriber!");

console.log("Subscriber 1 is listening....");

console.log("Subscriber 2 is listening....");

Here's the output:

The Short Polling Pattern

Short polling is another communication pattern that facilitates data exchange between client and server. It uses the pull-based communication mechanism (which is basically the client pulling data from the backend) to continuously poll the server for new updates.

How does Short Polling work?

Imagine you’re waiting for a message from a friend, and you don’t want to miss it. What do you do? You keep asking, “Got a message for me yet?”

So, the client makes a request to the server at regular (fixed) intervals (say every x unit of time) to check for new data or updates. The server sends back a response and if there’s new available data or an update, the server includes the data in the response.

Here’s what the client-server communication would look like:

• Client: “Hey, server, any new messages?”
• Server: “Nah, nothing yet.”
• Client: “Alright, I’ll check back later.”
• [Some time passes…]
• Client: “Hey, server, any new messages?”
• Server: “Bingo! Here they are!”

Although polling has some similarities with the request-response model we discussed earlier, a key difference between them is timing.

While polling occurs at regular, predefined intervals regardless of whether updates are available, the request-response model allows clients to request data or actions on-demand when needed, reducing unnecessary communication.

Benefits of Polling

Simplicity: Polling is easy to understand and implement and it’s completely stateless between the client and server. This makes it perfect for scenarios where the complexities need to be minimized.

Compatibility: It can be used with a wide range of technologies and protocols which makes it highly compatible with a number of platforms and environments.

Some example use cases of polling pattern includes simple dashboards, weather apps that require periodic updates, resource monitoring, or in cases where you’re considering cross-platform compatibility.

Limitations of Polling

Latency: Polling introduces latency, as clients must wait for predefined intervals before receiving updates. This can lead to delays in accessing real-time data or receiving notifications.

Inefficiency: Constantly polling the server for updates can be inefficient and can result in unnecessary network and server overhead.

Scaling: Handling a large number of simultaneous clients using polling can be resource-intensive for the server. It may require significant server resources to manage numerous concurrent polling requests.

Here's a code snippet to illustrate short polling. Here, the client sends a periodic request(polls ) to the server to check for upload progress.

The app.js file:

const express = require("express");

const app = express();

//create a dictionary to store the upload progress
const uploadStatus = {};

//simulate upload of file
app.post("/upload", (req, res) => {
 //create a unique request id for incoming request
 const requestId = Math.floor(Math.random() * 1000000);
 uploadStatus[requestId] = 0;

 simulateUploadProgress(requestId);

 res.json({ requestId });
});

//endpoint to check upload progress
app.get("/status/:requestId", (req, res) => {
 const requestId = parseInt(req.params.requestId);

 if (!isNaN(requestId) && uploadStatus[requestId] !== undefined) {
  if (uploadStatus[requestId] === 100) {
   //upload completed
   res.json({ progress: 100, message: "UPLOAD COMPLETED!" });
  } else {
   // upload still in progress
   res.json({ progress: uploadStatus[requestId] });
  }
 } else {
  res.status(404).json({ error: "Request ID not found" });
 }
});

//update upload progress by 10% every 5 secs
function simulateUploadProgress(requestId) {
 if (uploadStatus[requestId] < 100) {
  setTimeout(() => {
   uploadStatus[requestId] += 10;
   simulateUploadProgress(requestId);
  }, 5000);
 }
}
const PORT = 4000;

app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

Here's the output in the terminal

The Long Polling Pattern

Long polling is like polling but uses a push-based communication mechanism. In long polling, instead of the client asking the server, “Any updates?” all the time, it says, “Let me know when something’s up.”

Here’s how Long Polling works:

The client pings the server, just like in regular polling, but this time, the server doesn’t immediately answer. It holds the connection open, like keeping a phone line on hold. When there’s something to share, it responds. It’s like the server’s saying, “Hey, I’ll call you when I have news.”

It is used in web applications to achieve real-time or near-real-time updates between a client and a server.

Benefits of the Long Polling Pattern

Low Latency: Long polling provides low latency when compared to traditional polling as data is immediately sent back to clients as soon as they are available.

Real-Time Updates: With the long polling technique, applications can achieve real-time updates without the need for constantly polling the server.

Limitations of the Long Polling Pattern

Resource Intensive: Long polling requires keeping many connections open. As a result, it can be resource-consuming on both the server and client side.

Increased Latency: Although long polling helps cut down on frequent polling, it can still introduce latency when compared to other real-time communication protocols like Web Socket. Clients may experience delays between updates since they have to wait for the server to respond.

Difficult to Scale: When dealing with a large number of concurrent clients, long polling can strain server resources. As more clients establish long-polling connections, the server may struggle to manage and respond to all these connections efficiently.

Typical use cases of long polling include real-time updates, event-driven applications, and event notification systems.

Here's a code snippet to illustrate the concept of long polling where the client waits for updates from the server.

const express = require("express");
const app = express();

const uploadStatus = {};

app.post("/upload", (req, res) => {
 const requestId = Math.floor(Math.random() * 1000000);
 uploadStatus[requestId] = 0;

 updateUploadProgress(requestId, uploadStatus[requestId]);

 res.json({ requestId });
});

app.get("/status/:requestId", async (req, res) => {
 const requestId = parseInt(req.params.requestId);

 //simulate long polling (the server will not respond until done)
 while ((await checkUploadComplete(requestId)) === false);
 res.end("\n\n Upload status: completed " + uploadStatus[requestId]);
});

//update progress by 20% after every 5 seconds
function updateUploadProgress(requestId, progress) {
 uploadStatus[requestId] = progress;
 console.log(`Updated progress to ${progress}`);
 if (progress === 100) return;
 this.setTimeout(() => updateUploadProgress(requestId, progress + 20), 5000);
}

//check upload status
function checkUploadComplete(requestId) {
 return new Promise((resolve, reject) => {
  if (uploadStatus[requestId] < 100) {
   this.setTimeout(() => resolve(false), 1000);
  } else {
   resolve(true);
  }
 });
}
const PORT = 4000;

app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

And here's the output in the terminal.

code output

The Push Pattern

Push is a communication model that is used to deliver real-time updates to connected clients.

In this model, the client opens a connection to the server and awaits messages or updates from the server. Whenever there is a new update or message, the server immediately pushes that update to the client without the client explicitly requesting it — as long it is connected.

This model allows for bidirectional communication between the client and server. Web sockets, a popular protocol, uses the push model as its underlying data exchange method.

The Push model provides the most real-time or near real-time end-user experience when compared to other closely related paradigms such as polling and long polling.

To explain how the push pattern works, imagine a chat room with multiple connected users as an example. The client and server conversation will look like this:

• user1 (client): “Hello…”
• Server: “Oh, user1 has a message!” Instantly sends it to everyone else in the room.
• User2 (client): Gets the message from user1 without doing anything.
• User3 (client): Also receives the message from user1 — no need to refresh or ask for it.

In the example above, the push pattern enables real-time communication such that whenever there’s a message from a client in the chat room, it pushes the message to all other connected clients in that room without them having to continuously poll or request updates.

Popular technologies that uses the push pattern are RabbitMQ and WebSocket.

Benefits of Push Pattern

Real-Time Updates: The push model allows clients to receive updates from the server as soon as they are available. This is key, especially in applications where real-time updates are crucial.

Reduced Latency: Since the server pushes updates to the client as soon as they are available, it potentially reduces the latency.

Efficiency: Because there is no need for continuous polling or frequent client requests, there is efficient use of network resources and reduced server load.

While the push model is widely adopted as the best fit for providing real-time updates, it does have its own disadvantages.

Limitations of Push Pattern

Scalability: It can become difficult to scale as the number of connected clients increases. At this point, it becomes resource-intensive, especially on the server side since the server needs to maintain open connections with multiple clients.

Client support: Some clients might not be able to handle pushed messages as not all client platforms support push technologies. This may lead to compatibility issues and may need some sort of fallback mechanism for unsupported clients.

Some example use cases of the push pattern include chat and messaging apps, notification systems, IoT data streaming, and online gaming, to name a few.

Here’s a simple Node.js code example to illustrate the push pattern using Web Sockets. To run the code, you have to install the ws library using npm.

//create a server - this server will push updates to the clients at intervals
const WebSocket = require("ws");
const server = new WebSocket.Server({ port: 8080 });

server.on("connection", (client) => {
 console.log("Client connected to server");

 //Simulate client to receive real-time updates from the server every 2 seconds
 const interval = setInterval(() => {
  const message = `Message received at: ${new Date().toLocaleTimeString()}`;
  client.send(message);
 }, 2000);

 // Handle client disconnection
 client.on("close", () => {
  clearInterval(interval);
  console.log("Client disconnected");
 });
});

// Client (listens for updates from server)
const clientSocket = new WebSocket("ws://localhost:8080");

clientSocket.onmessage = (event) => {
 console.log(`Message from server: "${event.data}"`);
};

Wrapping Up

In this tutorial, we explored five key communication design patterns: Request-Response, Publish/Subscribe, Short Polling, Long Polling, and Push.

Each pattern has its unique strengths and limitations which makes them suitable for various use cases.

Depending on your application’s goal, understanding these patterns will help you design efficient backend systems.

Oracle, the company that owns Java, claims that Java is used on 3 billion devices worldwide, making it one of the most popular programming languages.

In this article, we will discuss the history of Java, reasons to use Java for backend development, the top tech companies that already use Java for this purpose, and the top Java frameworks for backend development.

We will also explore the benefits and drawbacks of using Java for backend development.

History of the Java Programming Language

Java was launched in 1991 when James Gosling and his team at Sun Microsystems began development on the language.

Soon after, the team shifted its focus to developing the language for the newest specialized market, the World Wide Web.

Java was offered to the public in 1995 for usage in various applications ranging from the internet to computer programming. According to learncodewith.me, Java is one of the top three programming languages in the world for backend development. It's also ranked first among the top ten programming languages for 2022.

Why You Should Use Java for Backend Infrastructure

The following are some benefits of using the Java programming language for backend development.

Scalability and Robustness

Java has a robust type-checking mechanism that makes it durable. The Java virtual machine (JVM) enables dynamic linking and a secure environment, allowing Java to run anywhere.

Java's automatic memory management and disposal collection make it highly scalable and these features help speed up web application development.

Open Source Library

The vast majority of Java libraries are free and open-source, with expert support. The use of such libraries speeds up the back-end programming of web projects dramatically.

There are many Java libraries for various uses, including logging, JSON parsing, unit testing, XML and HTML parsing, messaging, PDF and Excel reading, cryptography, and many others.

Diversity

Java has long been the dominant programming language for developing Web applications, Android applications, and software tools such as Eclipse, IntelliJ IDEA, NetBeans IDE, and others.

Java's use cases have now grown to include data science applications, machine learning applications, and even IoT applications.

Simplicity

Java syntax is relatively simple, easy to develop, learn, maintain, and understand, and the code is easily debuggable.

Java is also less complex than languages such as C and C++ because many of these languages' sophisticated features, such as explicit pointers, storage classes, operator overloading, and many others, have been eliminated from Java.

Security

Java reduces security concerns and dangers by reducing the use of explicit pointers. A pointer is a value that keeps the memory address of another value and can be exploited to gain unwanted memory access. This issue is solved by removing the concept of pointers.

Platform Independency

image from techavidan

Java is platform-independent, which means it's a "Write Once, Run Anywhere" (WORA) language.

The compiled code (the byte code of Java) is platform-independent and can run on any machine, irrespective of the operating system. We can run this code on any machine that supports the Java Virtual Machine (JVM), as shown in the figure above.

Multithreading Support

Java is a multithreaded language, which means that multiple threads can run at the same time.

A thread is a process's smallest unit. Multithreading allows us to maximize CPU use. Multiple threads share the same memory space, increasing the application's efficiency and performance.

Drawbacks of the Java Programming Language

Lack of Backup Facility

Java is primarily concerned with storage and does not prioritize data backup. This is a huge drawback, and it is losing user interest and ratings as a result.

Code Complexity

Java code is extensive, which means it contains many words and long, complex sentences that are hard to read and comprehend. This reduces the code's readability.

Java strives to be more manageable, but it must settle for unnecessarily complex programs and detailed explanations for each thing.

Speed and Performance Level

Java consumes a large amount of memory and is substantially slower than native languages such as C or C++. This is partly because each bit of code must be interpreted into machine-level code. This slow performance is caused by the JVM's additional level of compilation and abstraction.

Memory Capacity

When compared to other languages such as C and C++, Java uses a significant amount of memory. Memory efficiency and system performance may suffer during cleanup execution.

Top Companies That Use Java

To begin with the list, according to statistics based on Codegym, 10,130 companies are said to employ Java in their IT stacks. Not unexpectedly, the United States leads among enterprises that use Java, accounting for more than 60% of Java clients (about 64,000 businesses).

Linkedin: This application is primarily written in Java, with some C++ parts thrown in for good measure. Java is excellent for LinkedIn's search and analytics. More specifically, it addresses scale concerns, allowing the server to function faster while using fewer resources.

Uber: Uber is the next big Java-based firm. The organization works with a large amount of real-time data, tracking drivers and incoming ride requests. As a result, Uber should be able to sift the data and match users swiftly. That's where Java comes in, handling requests and transmitting data as quickly as possible.

Microsoft: Even though Java does not control Windows or anything similar, Microsoft uses it for various purposes. For example, it requires Java to construct its proprietary Edge web browser.

NASA World Wind: NASA built the Word Wind app, which features a realistic 3D virtual globe and can display precise geographical data, largely thanks to Java (the program uses actual images from satellites to develop 3D representations of the planets). It's an open-source program that runs on practically any operating system because it's developed in Java.

Netflix: Netflix, the same as PayPal, primarily uses Java for practically everything. And because Netflix is one of the best-known entertainment platforms in the world, there is a significant demand for Java expertise in this organization.

In addition to the tech titans mentioned earlier, Java is used by Airbnb, Google, eBay, Spotify, TripAdvisor, Intel, Pinterest, Groupon, Slack, Flipkart, and many more. Without a doubt, you can find Java practically anywhere.

Differences Between a Java Backend and a Node.js Backend

Java and Node.js are both popular choices for building server-side applications. But they have some key differences that make each one better suited for different use cases.

In terms of strengths, Java has the advantage of being a statically-typed language, which provides better performance and stability. Java also provides better security features, such as the Java Virtual Machine (JVM), which has memory protection and isolation. Java also has a large number of libraries and tools available, which makes it easier to develop and maintain large-scale applications.

Node.js, on the other hand, has the advantage of being a dynamically-typed language, which makes it easier to learn and more flexible. Node.js is also well-suited for real-time applications, such as chat applications, thanks to its event-driven architecture. Node.js also has a large and active community, which provides a wealth of libraries and tools for various tasks.

In terms of weaknesses, Java can be slower to develop and more complex compared to Node.js, due to its strong typing and verbose syntax. Java also requires more memory compared to Node.js, which can make it less suitable for low-memory devices.

Node.js, on the other hand, has weaker type checking compared to Java, which can lead to unexpected errors. Node.js also has a single-threaded architecture, which means that it is not as well-suited for multi-threaded applications compared to Java.

In conclusion, both Java and Node.js have their own strengths and weaknesses for backend development. Java is well-suited for large-scale and complex applications, while Node.js is well-suited for real-time and scalable applications. The choice between the two ultimately depends on the specific requirements of the project.

Top Java Frameworks for Backend Development

There are various Java backend frameworks, each with its own capabilities and features. Below are a few popular examples:

Spring

The Spring framework is a lightweight Java application framework and an inversion of a control container.

The framework's core functionality can be used by any Java application, although there are enhancements for constructing web applications on top of the Java EE framework.

Hibernate

Hibernate ORM is an object-relational mapping tool written in Java that provides a framework for translating a relational database into an object-oriented domain model.

Play

The Play Framework combines productivity and performance, making it simple to create scalable Java and Scala online applications. Play is developer-friendly, with a "simply hit refresh" workflow and testing support built in.

Because of Play's stateless and non-blocking architecture, applications scale reliably. Play is ideal for current online and mobile apps because it is RESTful by default and includes asset compilers, JSON, and WebSocket support.

Struts

Apache Struts is a free and open-source web application framework used to create Java EE web applications. It uses and improves the Java Servlet API to encourage developers to use a model-view-controller paradigm. Craig McClanahan designed it and presented it to the Apache Nation.

Conclusion

In this article, you learned about the Java programming language's history. You have also learned the key distinctions between Java and Node.js, clearing up any uncertainty.

Finally, you understand what Java is used for, the industries in which it can be found, and some of its distinguishing traits. You can now use it in your own web apps for seamless backend development.

• [The many layers of development](#The many layers of development)
• But we’re not all full stack
• So what is the difference between Front End Development and Back End Development?
• What is Front End Development?
• What is Back End Development?
• Where things get fuzzy
• Resources to learn

The many layers of development

Whether you’re working on a website or a native iOS app, all development environments share a common theme — there’s a front end to an application and a back end.

This line can get blurry, especially given the rise of javascript and the serverless world. With the tooling somewhat merging together, we might sometimes wonder if we’re a full stack developer.

But we’re not all full stack

As much as we might all want to be, we’re not all full stack developers. Personally, I find myself able to be productive in the back end of an application, but it’s not my strength and I much prefer to be heads down building UIs.

And some people are the opposite, where they are strongest dealing with building APIs in the back end of an application and while they can build out a UI, it might be more of a prototype-like experience than a fleshed out application.

So what is the difference between Front End Development and Back End Development?

Even if you are a full stack developer, that doesn’t mean there’s not a division of responsibilities.

Front End Engineer vs Back End Engineer

So what do those look like?

What is Front End Development?

The front end of an application typically refers to the layer that represents the UI (user interface). This can include anything from a static site with HTML and CSS to a full React app that powers the UI.

What did Front End Development traditionally look like?

Javascript currently rules the front end web, but that wasn’t always the case. While it could have been used to add little bits of interaction to a site, typically front ends were rendered using server-side templating languages like framework-driven PHP and Template Toolkit (Perl).

This grew to be super popular in practice with home grown frameworks or tools like Wordpress that used PHP to drive a massive community of developers who built their websites with those tools.

The way it worked was the templating language was able to get its data straight from the server as it was rendered. When a browser requested the page directly from the origin (the server itself), whatever data the template would need, the application logic would provide at that time.

Some of the more traditional front end tools include:

• Libraries like jQuery or MooTools
• Website frameworks like Wordpress
• Plain CSS
• Abundant use of Table elements

But as time went on, javascript kept getting more mature as a language and browsers kept getting more powerful, which led to the idea that we could move more of that work to the browser to build faster and more interactive experiences.

What does Front End Development look like now?

Now it’s common to see javascript-heavy websites and apps built using UI frameworks like React, Vue, and Angular. These tools provide abstractions that allow developers to build complex UIs with reusable patterns like components.

When the browser loads the page, the page receives an initial HTML document that also includes the script tag to the javascript (same as always). But once that javascript loads, it reaches out to APIs using browser requests that when completed, update the page to fill in any kind of dynamic data that you’d typically get along with that first HTML document.

It's like building a website... with more steps

While it sounds like more steps, it commonly provides a faster initial page load and render, not to mention it has a great developer experience. By delivering less on that first request and prioritizing what loads after that, it usually ends up as a better user experience.

Some of the front end tools that are more common and growing in popularity include:

• UI frameworks like React or Vue
• Web frameworks like Gatsby
• Compilers like Babel
• Bundlers like Webpack
• CSS tools like Sass

But those APIs, whether ones we pay for or create ourselves, need to be built somewhere. That’s where the back end comes in.

What is Back End Development?

The back end layer is usually where the business logic occurs. This can be super complex like the rules that determine revenue for an e-commerce company or something more common like a user profile.

What did Back End Development traditionally look like?

The back ends of applications were historically built using server-side languages like PHP or Ruby. The idea is that you have a server that you need to perform complex operations on, so the way to do that is with a language that server would understand.

On each request to the server, the backend would perform the full stack of the operations, including rendering out the front end. By using frameworks or DIY architectures, the back end would accept the request, figure out what it should do with that request, run any business logic needed with the request, and provide the front end any data that it would need to display a response to that request.

Back end giving the front end a 500 Internal Server Error

Some of the more traditional back end tools include:

• On-premise or remotely managed servers like Rackspace
• HTTP servers using Apache
• Databases like MySQL
• Server side languages like PHP or Perl
• Application frameworks like Ruby on Rails

What does Back End Development look like now?

Back end stacks look somewhat similar to the way they did before, aside from newer code patterns, except more often you’ll see the back ends provide data through APIs via HTTP requests instead of directly to the templates the front end team are working on.

While the foundation isn’t super different, it actually be comes increasingly complex as you have to deal with different security implications that could compromise your system if not properly configured such as leaving an API open to the public that returns sensitive user data.

But also how the server operates can be completely different. While previously, we might run our python on our own managed server (we still can), we can now make use of serverless functions with tools like AWS Lambda that simplify how we manage code.

While “serverless” doesn’t necessarily mean there are literally no servers, it means that as a service, the developer doesn’t have to worry about maintaining that server and can instead just focus on the code they need to run.

Some of the back end tools that are more common and growing in popularity include:

• Cloud servers like AWS EC2
• Serverless services like AWS Lambda
• NoSQL databases like MongoDB
• Languages like Python or javascript via NodeJS
• Web application frameworks like Serverless Framework

Where things get fuzzy

Part of the twist with back ends is now you can write your back end with javascript. With the inception of Node.js, developers were given the ability to use their favorite browser language to do most of the same things they were used to and familiar with but now on a server.

Never stopped to think if we should write JS on a server

While not everyone is fond of running javascript as a server side language, it became a little easier to use the same language to write the full stack of an application. This changed the game a bit as far as front ends and back ends were concerned.

But it’s also started to come full circle where you now see systems that build APIs right next to the front end similar to what you might see in a traditional stack.

Front End vs Back End

Regardless of the stack, there will always be the separation of concerns. The UI and all of the interaction, whether rendered on the server or in the browser, is what makes the front end the front end and the data and business logic, whether coming from the server in your company’s closet or a managed function, is what makes the back end the back end.

Whether you prefer to work on the user facing features or build the logic that lets them do things, there are plenty of resources to get started.

Resources to learn

Front End

• freecodecamp.org Responsive Web Design Certification (freecodecamp.org)
• Beginner Javascript (beginnerjavascript.com - Wes Bos)
• React Tutorial for Beginners (youtube.com - Programming with Mosh)
• Front End Masters (frontendmasters.com)

Back End

• freecodecamp.org APIs and Microservices Certification (freecodecamp.org)
• Super simple start to serverless (kentcdodds.com)
• AWS Certified Cloud Practitioner Training 2019 - A Free 4-hour Video Course (freecodecamp.org)
• CS50's Introduction to Computer Science (edx.org)

All the above

• How to Become a Full Stack Web Developer in 2020 (colbyfayock.com)
• Egghead.io (egghead.io)
• 100 Days of Code (100daysofcode.com)
• The Web Developer Bootcamp (udemy.com - Colt Steele)

My previous article described how you can get into frontend development. It also discussed how the front end can be a place filled with landmines – step in the wrong place and you'll be overwhelmed by the many frameworks of the JavaScript ecosystem.

In this blog article, let's see how you can get into back end development. Along the way, I'll answer some of the most common questions people ask me about it.

What is Backend Development?

Front end development involves what a user sees on the screen when they open a specific URL owned by you. Even in a completely static environment (with only HTML/CSS), when someone opens a website, some server on the planet needs to respond to you with those HTML and CSS files.

That server is just a computer, just like the one you use yourself to browse the internet. But it has been tuned for performance, and doesn't have unnecessary components like a mouse or keyboard attached. And it sits with tons of other computers probably in a data warehouse.

Programming those computers in some special way is called back end development.

You may think that backend development is called what it is because it runs behind the user's back. A visitor to your website never really "accesses" the back end completely. They just communicate with your server, either directly through ports for very limited access (like transferring HTML/CSS files) or not even that – buried deep under CDNs or firewalls (like Cloudflare).

Now that we have a raw understanding of what back end development means, let's get into some real questions.

Is front end programming knowledge required for the back end?

TLDR; No.

Back end development, as mentioned above, involves the programming of a computer sitting probably on the other side of the planet responsible for responding to what your users say from their own computers.

If you're a full-time backend developer, you do not really need to care about what goes on inside those HTML, CSS and JavaScript files you send to the user's browser. Instead, you've to focus more on the performance of the server, the server code, and throughput.

What goes into back end development?

Well, going by the books, you may say that a person who codes an application that can respond to HTTP requests is a back end developer.

But in reality, sometimes back end developers are able to do much more than just writing server scripts. They have the knowledge to set up reverse proxy servers (NGiNX/HAProxy), enable compression and other ways to speed up the site, and set up a production docker environment.

To qualify as a back end developer, I'd say the bare minimum skills you need are:

1. Good knowledge about a programming language in which you can write HTTP servers. Examples: C#, Java, Node, PHP, Python, etc. (there are many!)
2. Manage to host using cPanel (traditional) or using bash terminal (cloud hosting/traditional)
3. Working with Version Control Systems (VCS) like git for managing and deploying builds

Just like every game comes with minimum and recommended specifications, for back end developers, my recommend specifications would be (inclusive of the minimum skills):

1. NGiNX for static file assets and server management
2. Database Management skills (SQL/NoSQL)
3. Security of backend (Writing safe and robust code, running applications in docker containers with limited privileges, protection against DoS attacks)
4. Autoscaling/Load balancing

Alright, too much talking about what goes into back end development. But how do you become one?

Start with minimum requirements

Like I said, for the back end, just like games, we have a set of minimum requirements and recommended requirements. The minimum requirements consists of 3 things:

Learn a backend programming language

When people learn by themselves, they usually do not have a team or anyone who can do front end development. They're all on their own. So you'll often have to create webpages and servers all by yourself, at least in the beginning.

Although there are a lot of choices for back end programming languages, and I cannot think of any popular system language which doesn't support HTTP servers out of the box. The advantage of choosing Node is that your front end JavaScript skills are transferrable to the back end.

Nonetheless, you can choose from a variety of languages like Java, C++, C#, Python, PHP, etc.

How do you pick one, you might ask. The answer is the same as it was in the front end development article: you have gotta try everything initially and see which one clicks the best with you.

Node is easy as you might have already done JS programming for the front end. But if you're a Python or Java developer, you might find those easy to pick up. It depends on your profession and taste completely.

Learn about managing hosting

Gone are the days when you'll have to manually purchase servers and set them up in your home, connect to your ISP, do all that stuff yourself. This is the era of cloud computing. Now, when hosting your website, you have mainly 2 options:

1. Going for managed hosting servers like HostGator or GoDaddy.
2. Going for cloud hosting providers like GCP, AWS, or DigitalOcean.

What is the difference between the two? In both cases, the servers are owned and operated by the respective companies. But the major difference is that managed hosting is more GUI friendly, has a rich set of tools for seeing the filesystem, monitoring usage, managing your official domain emails, uploading/downloading files from your server, and so on. It's basically a setup for people with less technical skills.

For that reason, I do not recommend managed sites like HostGator or GoDaddy for seasoned developers. Still, it might be a good platform to make mistakes and learn on, primarily because you usually have prepaid plans for them. You'll also have a nice UI for managing things, which doesn't allow you to accidentally shoot up your bills.

But when you start picking up speed, I recommend that you switch to a cloud provider. This takes away all the nice tools from cPanel that you used to manage files and folders on servers. But at the same time, it will challenge you to level up your skills a lot.

Today, a lot of cloud providers offer a decent free trial, too, so that you can actually try out their platform before going full in. I host my website for developers - codedamn - on DigitalOcean and find it to be at a sweet balance of site complexity and features.

You can use this link to signup on DigitalOcean and get free $100 credits. DigitalOcean instances are as cheap as $5 a month, so you have a runway of about 20 months on that instance, great deal, huh?

Anyway, you can choose any cloud provider. Then it's important to learn to manage the server using just the command line by ssh'ing into it.

Learn about Version Control Systems

There are other solutions apart from Git for VCS. But Git is the most used and simplest to understand.

As an individual, you might not appreciate it right away. But you'll understand why it is so important the moment you start working either in a team on multiple features simultaneously in your project.

Git allows you to manage your workflow using commits and branches. Commits are like checkpoints in your codebase - the ones you can always revert to if you screw up.

Branches are like alternate realities of your project, where something completely different could happen. These alternate realities can be created from any point in time and can be merged back again at any time.

If those realities can be merged together with compatibility, then it's fine. But if there's a conflict (like if you're alive in one reality and dead in other), then you have to manually make a choice. Other changes can be merged automatically.

Git is super interesting, and once you get hang of it, you'll want to use it in every project. You get to keep a history of your work in an efficient manner (it compresses and stores only the difference between commits).

It also allows you to create online git repositories on sites like GitHub, which acts as a central source of truth for your website. Sites like GitHub can be configured with special webhooks that can actually update your website whenever you add a new checkpoint (a new commit) without you ever needing to manually go to the server and update it yourself.

Go for recommended skills

I'm a big believer in learning by doing. And the best way to do something comes out of necessity or interest. Once you consider yourself good enough with the minimum requirements, it's time to acquire the recommended skills. This includes all the tools like Docker and NGiNX mentioned above.

DevOps is also something which fits in super nicely with back end developers. You could try and explore TravisCI or CircleCI for automated build deployments. Continuous Integration and Deployment (CI/CD) is a topic that could take another whole blog post, so I'll not get into that. In fact, once it is set up correctly, it'll save you a ridiculous amount of developer time!

Then comes databases, which I placed in recommended skills. But you're gonna need databases for pretty much any application which involves some sort of data persistence generated by the user.

Databases are usually easy to begin working with, but harder to maintain and tweak properly. The best way to start working on a back end tech stack is to have everything together on a single server - the code of your application, the reverse proxy servers, the database, etc. Then as you become more proficient in each thing, you can decouple it from the existing business logic.

By doing this, you're enabling an architecture that can be highly scaled. A database-operation intensive application could have an optimized solution for databases. And a heavy traffic bound site should have a good CDN mechanism to offload static assets, and so on.

Conclusion

There's so much to learn, but it's all achievable if you don't give up. Let me know what you think about this post through my twitter and Instagram handles. It'll mean a lot to me if we connect over there!

Also, if you're interested, checkout codedamn - a developer-focused platform for learning technologies like backend development! I even posted a YT video on spinning up your own simple website server in 2 minutes! Check that out and let me know what you think!

Peace!

I've been shopping around for a back end framework to support a tabletop game app, and decided to do some research to determine the best fit for my needs.

The objective was straightforward: to build a simple RESTful API that would allow a front end app to perform basic CRUD operations, providing me with an introduction to what the development process would look like.

There are a lot of back end framework options out there, and I'm most familiar with JavaScript, C#, and Python (in that order), which limited my options somewhat. The natural starting point was to build a simple front end that would send requests to an API, which would in turn read from and write to a local database.

I began my development process with Express, which, for reasons I'll soon explain, led me to also check out Flask and ASP.NET. I thought my findings might prove useful to others out there who are researching back end frameworks for small projects. In this article, I'll also provide code examples and the resources that I used to build everything.

You can access the full code on GitHub, as well.

I should caveat that I won't be promoting one framework over another, and haven't yet compared things like deployment, authentication, or scalability. Your mileage may vary if those particulars are important to you!

I will, however, provide a TL;DR at the bottom if you just want to get the summary and key learnings.

Here we go!

Defining the API

If you're new to web development, you might be asking, "what's an API?"

I've had to ask the question a hundred times to find an answer that made sense. And it really wasn't until I built my own that I could say I understood what an API does.

Put simply, an API, or "application programming interface", allows two different computing systems to talk to one another. In this article, I'll show a simple front end app that displays a "quest" tracker that players can view for their tabletop roleplaying game. Each quest has a "name" and a "description," both of which are displayed in the web browser.

If I already had all of the quests listed on the website and just wanted players to view them, I would have no need for an API or back end. For this project, however, I want the ability to allow users to add quests, search for them, delete them, and so on. For those operations, I need to store the quests somewhere, but my front end app isn't able to transfer information directly to a database.

For that, I need an API that can receive HTTP requests from the website, figure out what to do with those requests, interact with my database, and send more information back up the chain so that the user can see what happened.

The whole thing - the front end "client", the back end "API" or server, and the database - is called a "stack," or more precisely, the "full stack." For this project, I built a simple front end website as the top of the stack, and switched out everything beneath it as I tried out different frameworks and databases.

Project Structure

The structure for this project was fairly simple, with the front end client separated from three different servers that I would spin up as necessary to serve the API.

I used Visual Studio Community as my code editor and IDE, with the requisite language packages installed for JavaScript, Python, and C#.

I'll provide an overview of my experience with each framework in turn, with links to the tutorials and packages that I used to get them to work with the client. But first, let's take a look at the front end!

The Client: Vue.js

The goal for the client was to have a simple website that would receive information from the database through the API and display it to the user. To streamline the process, my requirements were that the client would only need to "read" all of the items in the database, and provide the user with the ability to "create" a new quest.

These "read" and "create" operations - the "R" and "C" in "CRUD" - are analogous to the HTTP methods of "GET" and "POST," which we'll see in the code below.

In front end development, I'm most comfortable using Vue, and used the Vue CLI to scaffold a basic client, with the following file structure:

I replaced the boilerplate markup provided by the Vue CLI with the following:

<template>
    <div id="app">
        <h1>RPG Quests</h1>
        <p v-for="(quest, index) in quests" v-bind:key="index">{{quest.name}}: {{quest.description}}</p>
        <input type="text" v-model="newQuestName" placeholder="Quest Name" /> <br />
        <input type="text" v-model="newQuestDescription" placeholder="Quest Description" /><br /><br />
        <button v-on:click="postQuest">Add Quest</button>
    </div>
</template>

And the corresponding Vue code:

import axios from 'axios';

    export default {
        name: 'App',
        data: function () {
            return {
                quests: null,
                newQuestName: null,
                newQuestDescription: null
            }
        },
        methods: {
            getQuests: function () {
                axios
                    .get('http://localhost:3000/quests')
                    .then(response => (this.quests = response.data));
            },
            addQuest: function () {
                axios
                    .post('http://localhost:3000/quests', {
                        name: this.newQuestName,
                        description: this.newQuestDescription
                    });
            },
            postQuest: function () {
                axios.all([this.addQuest(), this.getQuests()]);
                this.$forceUpdate();
            }
        },
        mounted: function () {
            this.getQuests();
        }
    }

If you're not familiar with Vue, the specifics of the front end aren't that important! Of significance here is that I'm using a JavaScript package called Axios to make my GET and POST requests to a potential server.

When the client loads, it'll make a GET request to the URL http://localhost:3000/quests to load all quests from the database. It also provides a couple of input fields and a button that will POST a new quest.

Using the Vue CLI to serve the client on http://localhost:8080, the front end of the app looks like this in action:

Once quests are added to the database, they'll start appearing in between the "RPG Quests" header and the input fields.

Client Resources

To build the client, I used:

• NodeJS/NPM for package management
• Vue CLI for scaffolding, serving, and building projects
• Axios for making HTTP requests to the API
• Vue Axios Documentation for making sense of how to use Axios in concert with the API
• Postman for testing API requests through the browser before implementing them in the client.

JavaScript API: Express

Express is a lightweight web framework for NodeJS that allows you to write server-side applications with JavaScript.

It's un-opinionated, which means that you can build your applications how you like without it defining the architecture for you. You can add packages to improve functionality as you fancy, which I found to be a double-edged sword as a newbie to the framework. More on that later.

Being most comfortable in JavaScript, I was excited by the prospect of having the entire stack run on just one language instead of several. I had heard of the "MEVN Stack," which denotes a full stack application that is comprised of MongoDB, Express, Vue, and NodeJS, and decided to try that out for this iteration of the project.

I followed a web API tutorial to first build a template app, then used another MEVN tutorial to fill in the details of how to get the API to communicate with the Vue client that I had built. The Express API that I created for this project follows a similar structure to the former, using MongoDB as the database:

If you're coming from a JavaScript background, Express is fairly easy to read, even if you're not familiar with some of the back end terminology. The following is a snippet from /routes/quests.js, for example, which handles the HTTP endpoint requests:

router.get('/', async (req, res) => {
    try {
        const quests = await Quest.find();
        res.json(quests);
    } catch (err) {
        res.status(500).json({ message: err.message });
    }
});

router.post('/', async (req, res) => {
    const quest = new Quest({
        name: req.body.name,
        description: req.body.description
    });
    try {
        const newQuest = await quest.save();
        res.status(201).json(newQuest);
    } catch (err) {
        res.status(400).json({ message: err.message });
    }
});

The general theme of the code is to receive a request, attempt to contact the database to do work, and then send a response back to whoever's asking. The specifics can be quite complex, particularly if you're writing your own middleware that does things in between the request and response, but the code is at least readable.

I found MongoDB to be painless to work with as a NoSQL database. If you're working with Express, you'll most likely use Mongoose as an ODM - basically like a "middle person" that translates a model of what your data looks like to the database.

The model in this app (called a "schema" in Mongoose terms) is really simple, located in /models/quests.js:

const questSchema = new mongoose.Schema({
    name: {
        type: String,
        required: true
    },
    description: {
        type: String,
        required: true
    }
});

The above indicates that the database should store our two fields: a quest name and a quest description. Both of these fields are strings, and required. All GET and POST requests will have to conform to this model to interact with the database.

After wiring all of this up and POSTing a few new quests, the front end site started populating with data:

The process of setting up the Express API was not without its hair pulling, however. Being a primarily front end and 2D game developer, I've become intimately familiar with how dispersed the JavaScript ecosystem can feel. This frustration was magnified in attempting to build a back end app. There are a lot of packages required to get everything up and running, each of which having its own required configuration and implementation.

If you're looking for a framework that just does everything out of the box, Express is most certainly not the choice for you. It's lightweight, flexible, and easy to read, in a very "choose-your-own-adventure" fashion. I quite like how clean the code is and the ability to structure my projects as I see fit, but troubleshooting and error handling do leave a lot to be desired.

JavaScript/Express Resources

To build the JavaScript API, I used:

• NodeJS/NPM for package management
• Express as the main web framework
• Dotenv to create environment-specific variables
• Nodemon to watch files for changes and restart the server so I didn't have to
• CORS to allow for cross-origin requests (basically a pain if you're trying to make requests from a client to a server that are both running locally on your machine)
• MongoDB for the NoSQL database
• Mongoose for writing models that map onto MongoDB
• This API tutorial to provide a basic understanding of how to create an Express-MongoDB stack
• This MEVN tutorial to fill in the gaps of running a MongoDB-Express-Vue-Node full stack

Python API: Flask

In the process of building the Express API, I had a conversation with a data science friend who works in Python. This gave me the idea of trying out non-JavaScript frameworks to see if they were better suited for my app.

I took a cursory look at Django, since I'd been hearing about it as a powerhouse back end framework that provides everything out of the box. I was a little intimidated by how opinionated it seemed, and opted to try out Flask instead, which kind of felt like the Python equivalent of Express.

I followed the first few bits of the excellent Flask Mega-Tutorial to get my app structure set up, using the companion RESTful API tutorial to fill in the pieces of HTTP requests. The file structure turned out to be only a shade more complex than that of the Express API:

The tutorial I followed uses SQLite for its database, with Flask-SQLAlchemy as an ORM. The HTTP request code that's most analogous to the Express API is located in /app/routes.py:

@app.route('/quests', methods=['GET'])
def get_quests():
    questQuery = Quest.query.all()
    quests = {}
    for quest in questQuery:
        quests[quest.name] = quest.description
    return jsonify(quests)

@app.route('/quests', methods=['POST'])
def post_quest():
    newQuest = Quest(name=request.json['name'], description=request.json['description'])
    db.session.add(newQuest)
    db.session.commit()
    return "Quest Added!"

Similarly, the database model (akin to the Mongoose "schema") is in /app/models.py:

class Quest(db.Model):
    name = db.Column(db.String(256), primary_key=True, index=True, unique=True)
    description = db.Column(db.String(256), index=True, unique=True)

As I mentioned, I'm more familiar with JavaScript and C# than with Python, and working with the latter to build the Flask API felt like cheating. Certain things like pathing, package handling, and writing workable code were just easy, although I did get hung up on getting the API to correctly parse JSON for the client. I suspect that was more of an issue of my unfamiliarity with the language than anything else, but it did take time to troubleshoot.

To be quite honest, coming from a non-Flask background, I did kind of expect to complete a couple of tutorials and spin up an API without having to do all that much work for it.

I can't say that it turned out that way, as Python does have its own particulars that require some time to get used to. Still, the Python ecosystem appears to be extremely well organized, and I enjoyed my time building the Flask API.

I've also heard that Django is a better and more scalable option for larger projects. But it seems like it would involve a separate, and steeper, learning curve to become proficient.

Flask was easy enough for me as a non-Python developer to pick up and build something over a weekend. I suspect that learning Django would take quite a bit longer, but with potentially greater dividends over the long run.

Python/Flask Resources

To build the Flask API, I used:

• Python 3/pip for package management
• Flask as the main web framework
• python-dotenv to configure environment variables
• SQLite as the database
• Flask-SQLAlchemy as the ORM to work with SQLite
• Flask-Migrate as an additional tool to migrate data to SQLite
• Flask-CORS to handle the same CORS issue as with the Express API
• The Flask Mega-Tutorial to learn the basics
• The Flask REST API tutorial to understand how to receive HTTP requests

C# API: ASP.NET

I can't tell you how many times I've Googled ".NET" to understand what it is, how it's different from ASP.NET, and why I'd want to use any of it. My C# knowledge comes mainly from working with Unity, which exists somewhat adjacent to .NET and doesn't provide for a lot of exposure to Microsoft's larger ecosystem.

I've spent some time researching Razor Pages and MVC, and finally came to understand ASP.NET's breadth of features as Microsoft's open source web framework. I decided to toss ASP.NET into the hat for a potential back end for my app, and set about working through the official web API tutorial with ASP.NET Core and MongoDB.

The file structure for this version of the API was more complex than the others, given that .NET projects tend to have a much larger footprint:

I should also mention that I already had Visual Studio and all of the required workloads installed, which made the setup process easier. Plus, having spent time with MongoDB for the Express API, I found the database portion of the project to be similar, although by default, ASP.NET seems to prefer using Microsoft's SQL Server and the Entity Framework ORM.

The ASP.NET code for HTTP requests is a bit more complex than what we've seen with the two other APIs, but it's no match for all of the code that sits around it.

First, consider this snippet in /Controllers/QuestController.cs that handles requests:

namespace QuestAPI.Controllers
{
    [Route("quests/")]
    [ApiController]
    public class QuestsController : ControllerBase
    {
        private readonly QuestService _questService;

        public QuestsController(QuestService questService)
        {
            _questService = questService;
        }

        [HttpGet]
        public ActionResult<List<Quest>> Get() =>
            _questService.Get();

        [HttpPost]
        public ActionResult<Quest> Create(Quest quest)
        {
            _questService.Create(quest);
            return CreatedAtRoute("GetQuest", new { id = quest.Id.ToString() }, quest);
        }
    }
}

Not too terrible, almost kind of readable, in a C# sort of way. The data model in /Models/Quest.cs is even easier:

namespace QuestAPI.Models{
    public class Quest
    {
        [BsonId]
        [BsonRepresentation(BsonType.ObjectId)]
        public string Id { get; set; }

        [BsonElement("Name")]
        public string Name { get; set; }

        public string Description { get; set; }
    }
}

These two snippets essentially do the same things as the previous examples that we've seen: take requests from the front end, process them to get or modify data in the database, and send a response back to the client.

Yet, as you can probably tell from the complex file structure, there's so much code that surrounds these snippets, along with Interfaces, Dependency Injection, and other abstractions, that it can be challenging to understand how it all works together.

Consider the following configuration code in /Startup.cs:

        public void ConfigureServices(IServiceCollection services)
        {
            services.Configure<QuestDatabaseSettings>(Configuration.GetSection(nameof(QuestDatabaseSettings)));

            services.AddSingleton<IQuestDatabaseSettings>(sp => sp.GetRequiredService<IOptions<QuestDatabaseSettings>>().Value);

            services.AddSingleton<QuestService>();

            services.AddCors(options =>
            {
                options.AddPolicy(MyAllowSpecificOrigins, builder =>
                {
                    builder.WithOrigins("http://localhost:3000/quests", "http://localhost:8080").AllowAnyHeader().AllowAnyMethod();
                });
            });

            services.AddControllers();
        }

Or this particularly nested bit from a separate SQL Server web API tutorial:

    [HttpGet]
    public async Task<ActionResult<IEnumerable<TodoItemDTO>>> GetTodoItems()
    {
        return await _context.TodoItems
            .Select(x => ItemToDTO(x))
            .ToListAsync();
    }

Lol. What?? As a new user, even familiar as I am with C#, I can go line-by-line to understand each abstraction, or I can just trust that the framework is handling everything for me and forget about it.

I tend to want to know exactly how my code works so that I can fix or alter it if necessary. But I certainly feel like my time spent learning the ins-and-outs of ASP.NET could be better utilized towards mastering another framework.

To be fair, ASP.NET appears to be similar to Django in being more opinionated and providing you with a ton of stuff out of the box, including an authentication solution, database management, and the lot. If these things are important to you, it's certainly worth considering.

It also has the full support of Microsoft and an open source community. So if you're looking at developing enterprise-level applications that need to scale, you might want to take a longer look at ASP.NET as a potential solution.

C#/ASP.Net Resources

To build the ASP.Net API, I used the following resources:

• Visual Studio Community as my code editor and IDE, with the ASP.NET and web development workload installed (I already had MongoDB running from the Express API)
• Microsoft's official tutorial for building web APIs with ASP.NET and MongoDB

TL;DR

In all, with some slight variations and hiccups among them, I've gotten each of the web APIs to work with the Vue client, with the ability to view quests from and add quests to the database. Hopefully, my explanation of the process has been helpful in your own search for a back end framework, but here are some additional recommendations just in case:

• If you're a JavaScript developer and/or want to manage everything that your application does, including its architecture, consider using Express.
• If you're a Python developer and/or want a pleasant experience in developing small projects, try Flask, but consider using Django if you need more out-of-the-box support and don't mind conforming to an opinionated framework.
• If you're a C# developer and willing to spend the time to learn the most arcane details of C# coding best practices, consider using ASP.NET. Alternatively, if you need enterprise-level support right out of the box, you'd be hard-pressed to find better.
• If you don't know what to use and just want to learn back end development, take a look at Flask. It's easy to work with and will teach you the basics that you'll need to know for building web apps in any coding language.
• If you don't know what to use and want an adventure, choose Express. There's a rabbit hole of package management and Stack Overflow questions waiting that may make you tear your hair out, but you'll learn a lot about the JavaScript ecosystem and web development in general.

Additionally, two things bear mentioning that threw me for a spin in this process: CORS and environment variables. The former I've mentioned in this article a couple of times already, but it's worth discussing again to understand the scope of building a full stack app on your machine.

Unless you have an integrated development environment that's handling the whole stack for you, you'll likely have a client, a server, and a database that are all running independently of one another.

In the Express API section above, for example, I was running

1. the Vue CLI server, which rendered my front end app on port 8080;
2. an NPM script to spin up the Express API server on port 3000; and
3. a separate instance of the Mongo database to get everything working together. That's three command prompts open and a general mess!

If you dig into the Vue code above (or on GitHub), you'll see that the requests made on behalf of the client, running on http://localhost:8080, are to the server on http://localhost:3000, which is where the Express API is listening. This is called "cross-origin resource sharing," or CORS, and it's blocked by the browser for security concerns. Most frameworks require you to install an additional package to get the whole thing running in your local environment.

Second, you'll want to become comfortable with environment variables, which can really help smooth some rough pathing edges at runtime. I used dotenv and Flask-Env for the Express and Flask projects, respectively.

Both packages allow you to configure things like where your database lives, or what default port your application should be using, in one document. Your application then uses that document at runtime to figure out where to find everything, without any further configuration needed from you.

One final note that may be helpful if you're just working on a back end project and don't want to go through the trouble of building a front end client: consider using a third-party app like Postman. I used it to make HTTP requests to each of the APIs to make sure they were working properly before layering on the Vue client and trying to get the whole stack running altogether.

I hope this article has been helpful for you in your own process of looking for a back end framework. Let me know what you find!

If you enjoyed this article, please consider checking out my games and books, subscribing to my YouTube channel, or joining the Entromancy Discord.

M. S. Farzan, Ph.D. has written and worked for high-profile video game companies and editorial websites such as Electronic Arts, Perfect World Entertainment, Modus Games, and MMORPG.com, and has served as the Community Manager for games like Dungeons & Dragons Neverwinter and Mass Effect: Andromeda. He is the Creative Director and Lead Game Designer of Entromancy: A Cyberpunk Fantasy RPG and author of The Nightpath Trilogy. Find M. S. Farzan on Twitter @sominator.

This is for those of you out there who are about to start your job search and who may be worried that you can’t land a top-tier tech job without a Stanford CS degree. Someone told you that you’re not good enough to get a job at Microsoft or Facebook.

But I’m here to tell you that you can get that job. Here’s how I landed my dream job at Twitter.

Read more about my courses here to learn how I prepared.

You can read about my experiences after a year at Twitter here.

What this article covers:

• My background
• How I landed interviews with top tech companies in the world: Facebook Google, Amazon, LinkedIn, Microsoft, Twitter, Pinterest, Snapchat, and others.
• How I landed multiple offers as a full-time software engineer
• Lessons from my interview experience
• Subscribe here for more article updates from me

If you prefer to watch my story instead, I made a video here:

Background

I did not graduate from an Ivy league school. I went to a community college in Idaho for two years, and then finished my CS degree at a small Catholic university.

I started learning computer science in my junior year of college, because it sounded fun to me at the time. The only thing resembling a computer I had growing up was a Chinese copycat of the Nintendo SNES. Even then, it would break every time I put a cartridge in it.

To support myself through college, I took multiple part-time jobs like cleaning floors and working stand-up concessions.

When I graduated, I didn’t have a job lined up. I applied to as many big tech companies as I could, and had the good fortune of landing a few phone interviews.

At this point, I didn’t have a single notion of what a technical screen would be like, much less how to prepare for it. I headed into these interviews thinking that the interviewer would ask me what a linked list or binary tree was.

I didn’t pass any of those interviews.

Moving forward

I didn’t delve too much into whether I was good. I knew that I could learn things fast. I just needed an opportunity.

As the saying goes, cast your net far and wide. So that’s what I did.

What I did next is something I’m particularly proud of. I wrote a simple Python script that scraped job listings on Craigslist with titles containing keywords from a list, and collected the emails in a spreadsheet. For the actual war story, you can read the article here.

It wasn’t the smartest solution, but people who post on Craigslist are surprisingly accurate with their titles.

Craigslist, however, didn’t like people scraping their website. To work around this, I ran my script through a VPN, and had a timer that would pause the script every few minutes or so. It wasn’t perfect, but it worked well enough.

In the end I collected about 500 emails from around San Francisco, Portland, Spokane, and Seattle. I filtered the results by how specific and recent they were, and kept improving it by adding more and more features.

As it turned out, there were a few bots in the market already that crawled Craigslist and sent out automated emails. These were mostly offshore companies that were looking to pitch their company to the US market.

One of my workarounds was that I crafted emails that used keywords from their listings in the title of my emails. I then added more details using the body of the postings to make it seem more personable. I did a quick A/B test, and the replies I received had increased quite a bit from around 2–3% to 10%.

Out of the 500 or so emails, I received about 50 replies, and landed phone screens with a small percentage of those. I stopped at 500 because I was short on time and needed to finalize a job as soon as possible. I was optimizing for results rather than reach at that point.

As luck would have it, I finally landed a job at a startup in Seattle as a junior software engineer. The startup was located in Kirkland at the time, so I had to take a 45-min bus ride to make it in time for the interview.

I then stayed there for the next 3.5 years, where I learned a great deal of stuff like Amazon AWS, EC2, DynamoDB, SQS, and Docker. I grew a lot during this period. I learned how to write modular, maintainable code. I learned how to reason about software design. And I learned how to handle people problems.

I was working next to a group of smart people who held jobs at Microsoft, Amazon, and LinkedIn, and I tried to be the “sponge” in the group. I absorbed anything and everything they threw at me. I believe this made a huge impact in my career.

Startup Days

During my stint at the startup, I worked almost exclusively on backend development, with some dev-ops in between. I started out writing some functions to add/modify a feature that were mostly small in scope. But it was a great opportunity to understand the codebase and get some code reviews.

A year into it, I started owning parts of the codebase, and then I was tasked with turning a set of features into a service. That was the start of the SOA phase for the startup. We started turning various components of the site into services, and that’s how I started learning more about RESTful services, authentication, AWS services, pub-sub, distributed systems and so forth.

The interesting part here is that I didn’t learn about any of these through books or formal education. Rather, I needed to get that set of features done and there were the bottlenecks.

So I thought, let’s go solve it!

There were many times where I was stuck in analysis paralysis — a state where I over-analyzed scenarios and ended up not able to make progress.

Those trying times were the greatest learning opportunities. I started to learn feature scoping, negotiations, monitoring, alerting, and documentation. Each step of the process revealed more things I needed to learn. I grew the most during these 2–3 years, both as an individual and software engineer.

How I prepared for my interviews

After suffering through my first job search, I told myself that I must be prepared in future interviews.

I started preparing for interviews by charting out an overview of what I was good at, bad at, and where I could improve. I broke it down into three categories: data structures, algorithms, and system design.

Having worked in PHP for most of my professional career, and C++ in college, I wanted to try something a little simpler and less verbose for interviewing.

For this reason, I picked Python. It is a great language to learn, easy to pick up, supports many data structures out of the box, and can be written quickly on the whiteboard. I learned Python by going through YouTube tutorials like these, and also reading their documentation. I prefer Python 2.x, but you can go for either 2.x or 3.

Also, another reason why I picked Python is that it’s highly readable and easy to write on a whiteboard. Here’s a trivial comparison between C++ and Python.

A C++ program to sort in descending order:

#include <bits/stdc++.h>
using namespace std; 
int main() {   
    int arr[] = {1,10,0,4,5};
    int n = size(arr)/sizeof(arr[0]);   
    sort(arr, arr + n, greater<int>());   
    for (int i = 0; i < n; i++) {       
        cout << arr[i] << " ";   
    }    
    return 0;
}

Compare that with Python’s version:

a = [1,2,4,5,1000]
a.sort(reverse=True)
print a

I’ve received feedback from interviewers to err on the side of brevity in an interview. In a 45-minute interview, you want to use most of your time solving the actual problem.

Pro tip: pick a language that’s less verbose so that you can write the code more quickly on the whiteboard.

Preparation mode

I spent about a week going through simple challenges on LeetCode, HackerRank, and Project Euler to familiarize myself with their interfaces, and to get used to writing code in Python.

The first week gave me insights into my competence level at certain programming languages. I spent another week going through some design challenges like “design X” and went as wide and deep as I could.

This was a lot of fun for me, because I often looked at iOS apps and tried to figure out how they did it. For example, how would you build Instagram from scratch? (I was asked this at Facebook.)

My background is in API designs and service-oriented architecture, so I took this opportunity to show how I would design my own version of Instagram. And because I have some iOS programming experience from my side-projects, I could talk a little bit about callbacks and push/long-polls here.

I started the conversation with some features I’d like to have on my own version of Instagram: likes, upload a photo, and a simple timeline. Feature scoping enabled me to build a very solid API because I know these scenarios well.

I then drew some pictures of a high-level design, of how the client would interact with the backend, and of how the backend would store the data.

I started small, and then added more components where needed and proactively sought where the bottlenecks were. I made educated guesses (read educated, not blind guesses) on what the requirements would be, and how each technology would fit in well. And also equally important, what technologies would not fit well.

For example, why would you use Cassandra over MySQL to store certain information (hint: scale, speed of development, schema reviews), why use OAuth over simple authentication, Redis vs Memcached for caching data, streaming vs batch processing, and so on.

There are many areas you can explore here, so typically a one-hour session is not enough. To do well on these questions, you have to read and learn about trade-offs. Pros and cons of technologies in the industry. For this, I recommend a site like HighScalability.

Take it like a typical brainstorming session with a coworker, so explore as widely and as deeply as you can.

It’s crucial to know that these design interviews are meant to explore how much you know and how well you know it, and it’s an opportunity for you to shine. I watched this YouTube video from an ex-Facebook engineer about how to solve design problems, and it gave me insights that helped me tremendously with my design interviews. My two main lessons from it: drive the design conversation, and show what you know.

I listed out my competency level for: data structures (linked list, hash map, binary tree, binary search tree, heap, array), algorithms (binary search, hashing, dynamic programming, sorting), and language-specific syntax and libraries (like sort, lambda for Python, appending, indexing).

I picked the area I was worst at, and started working on it: algorithms.

Algorithms have never been my forte. It’s been a while since my college days, and I didn’t spend much time doing binary search in my day-to-day career. I had an inkling of how each algorithm would perform, and in what scenarios to use them. But I wasn’t 100% comfortable with writing a binary search in under 10 mins. On a whiteboard. In front of an interviewer.

I also picked up a bunch of fine-point markers from Amazon, which work amazingly well. Perhaps it’s just me, but the fine-point markers in interviewing rooms usually don’t work at all. I’d usually scramble for 2–3 mins looking for a working pen, and that’s 2–3 mins you can’t afford to waste. Plus, fine-point markers allow you to write 5–8 more lines of code on a typical whiteboard vs. thicker ones. :)

Pro tip: Get your own set of fine-point markers.

I got a whiteboard from Costco for $50, some books from Amazon (listed in the tools I recommend section below), and started practicing. I made sure I ramped up on binary search, recursion, dynamic programming, BFS and DFS. A lot of interviewing questions revolved around recursion and binary search or some variations of it.

The best interviewing questions I’ve seen had many different solutions to them, and there’s an additional layer added on top as you progress through.

One Google question I had was related to file-system directories, and how to traverse them (hint: recursion). I solved that relatively quickly, and the interviewer asked how to identify a missing file in that directory. That was a little more difficult, but I got through it. And we then moved into how to rebuild the directory, how to serialize/deserialize it, and we spent a good chunk of time debating how file directories work underneath the hood. It was a very enjoyable session for me.

Interviewing at top-tier companies

It was a nerve-wracking experience, to say the least, and a real roller-coaster.

I allocated my time in the following manner: 20% resume, 20% research and 60% interview preparation.

I spent 20% of my time fixing up my resume, which hadn’t been updated in at least three years. I took a hard look at the stuff I’ve done in the past, and picked projects I handled end-to-end, regardless of complexity.

The reason for doing this is two-fold. Taking a project from start to completion demands discipline and leadership — two of the traits I’d like to be identified with.

Secondly, ownership of a project end-to-end means I can talk about each aspect of the project at length and in depth. This proved critical in helping me navigate my design round at Twitter, where they grilled me hard on not only the designs of my projects, but also the decisions behind them.

20% of my time was used for research. Research in this case meant doing due diligence on companies I was interested in and reaching out for referrals. Having referrals helps with return calls.

From my experience, I sent out 20 or so cold messages to startups and mid-stage companies, and only heard back from a handful. But, almost all the companies I was referred to by an existing employee sent me a message within a week. This is anecdotal, but there’s value to be had there.

I am not that sociable, and I didn’t know many people who’d be able to refer me to a company I was interested in. To solve that problem, I went on LinkedIn. They have a search functionality that I used to search for 1st and 2nd-level connections. 2nd-level connections are people who’re one hop away from your immediate circle. In other words, we have mutual friends who can vouch for my credibility.

LinkedIn search

This is incredibly important, because cold-calling someone for a job is very, very hard, especially in today’s market. People tend to err on the side of caution when it comes to cold-callers. Using LinkedIn was super helpful for my research phase.

Looking back at all the companies I interviewed at, here are my thoughts on each of them:

• Facebook/Google — very mechanical. The standard interviewing process, and I didn’t feel any personal connection to them.
• Pinterest — not the best interviewing experience, but a cool product and company.
• Microsoft — loved the team and especially the manager and her manager. Standard interviewing questions, but very personable. Close-second choice. Your mileage may vary, though — each team at Microsoft interviews differently.
• Amazon — standard interviewing process. About 50% of the people love it, the others don’t.
• Twitter — incredibly fun and personal. Loved the interviewing process, gave a lot of emphasis on the individual and what I’d done in the past.
• Snapchat — cool office in LA, great bunch of people who decided to jump on the startup bandwagon. Felt like things were shrouded under a cloud of secrecy.
• Lyft — near to where I live, nice office, standard interviewing process. No strong feelings about it.

Let’s talk about my favorite

In many ways, I’d say Twitter’s interviewing style was hard. But at the same time, it was more interesting and personable than other companies I’ve interviewed at.

Their interviewing process starts with an introductory phone call with an engineering manager. That’s followed up by one or two technical phone screens, depending on how you perform. If you do well, they’ll fly you out to the office you’re interviewing for, which was Seattle in my case. There are three 1-hour-and-15-minute rounds, each with two interviewers.

The first two technical phone screens are the standard, run-of-the-mill technical screens where you solve coding problems on a shared coding doc.

The onsite rounds, however, are much more conversational and feel much less intimidating. The interviewers will ask you in-depth questions about your past projects, and they’ll grill you on what you’ve done in the past. If you claim ownership of a project, you should expect some questions about it. You’re encouraged to use them for references and to bounce ideas off of.

I never felt any pressure to magically come up with a fully working solution, and it felt highly collaborative.

The others

In comparison, interviewing at Facebook and Google felt much more mechanical. They have one or two technical phone screens, and five to six onsite coding rounds. Each round involves some coding on a whiteboard, and you’re expected to come up with a near-perfect solution in a reasonable amount of time.

Facebook has two coding rounds, one design round, and one behavioral round.

I went through an additional shadow round at the end of the day, which didn’t count towards my overall score.

Google had five coding rounds, none of which focused on designs, and not a single interviewer asked about my previous projects. I don’t necessarily think this is bad. But I think it felt very mechanical and didn’t give much opportunity for the engineer to show what they’re capable of. Some people do well in these scenarios, much like some students do well in exams.

I did not enjoy my interview with Pinterest. I think the product itself is interesting, and their engineering team seems to be working on very cool technical problems. But I definitely had a negative experience during my interview there.

Pinterest has three coding rounds and one design round. Of those four rounds, the design round was most disappointing to me. Here’s why:

The interviewer came in late, and he spent a few minutes glancing over my resume before proceeding to draw some APIs on the board. He gave a short description of what he expected the API to do, and asked how I would solve it. We clarified the features of the API, and I started describing my solution using the whiteboard. About 5 minutes into it, I turned around and saw him taking a nap!

Not cool.

I gave the recruiter my feedback in a survey, and I didn’t hear back from them after that.

I won’t delve into specifics of the questions I was asked during all the interviews. Instead, I’ll share some of the insights and useful tips I learned from my preparation process.

What I learned:

• Be honest on your resume. Most companies will ask you questions about your resume, and they can tell if you made it up. It’s better to be able to know 100% about one project than to know 10% about 10 different projects.
• One-page resumes are recommended. This is especially true for tech companies, and it seems that the wisdom within the tech sphere is that you should reserve two pages and longer for post-doctoral work, or if you’ve done a lot of projects that you know and care deeply about. A friend of mine runs a company called Jobscan that scans resumes and makes specific, actionable improvements on them. They’re pretty awesome, so try them out :)
• Socialize and establish a network. There’s a lot of competition for software engineering jobs, and these top tech companies are filtering through thousands of resumes a day. Having a referral will help you get some eyes on your resume.
• Nail your pitch. Every company that’s interested in you wants to know why you’re interested in them. A bad answer: I just need a job right now to pay bills. A less-bad answer: I was browsing online and found you guys. Sounds like you’re working on interesting things. A good answer: I know you’re doing some interesting work in X to achieve Y. I’ve done some work in the past and here’s what I learned about A, B, C that might be related to X. I am passionate about Y because blah. (Don’t use this as a template. Instead, you should see the pattern here — do your research, use your background, and show the company why both of you would fit well together.)

Some more advice

Technical interviews are incredibly difficult, and sometimes it’s a hit-or-miss. The best opportunities, however, are reserved for those who are prepared.

• Prepare early, prepare well. Everyone knows that they should prepare for an interview, but most don’t know how to do it well. As with anything worth doing, it takes deliberate practice to do well at something. And deliberate practice means you need to have a system.
• Build a system to practice technical skills. I started by rating myself from 1–10 on how good I was, and worked on the ones I was worst at. I spent days on different types of questions until I fully mastered each concept. And I wrote notes daily on Evernote. I had a note that serves as a brain dump for all things programming. It is full of programming tips & tricks, common errors and misconceptions, frameworks for solving specific types of questions, and much more.

My notebook

• Keep a notebook of the things you’ve learned. I use both Evernote and OneNote to keep track of things. OneNote for technical stuff/code, because I like that I can easily format the note any way I like. I use Evernote for essays/thoughts. The image above shows a note I keep on architecture and system designs.

Evernote for thoughts/tips

• Jot everything down, even if you don’t think you’ll use it. I tend to forget very easily, so anything that I learn I write it down, including shell commands. I read technical blogs from time-to-time, and if I find anything interesting I jot it down on Evernote right away. I’ll revise it every week or month and reorganize accordingly. This has helped me tremendously over my career.
• Get mock interviews. This was definitely very valuable and I highly advise it. I had mock interviews with friends and tried to practice as much as I could. If you can’t find friends to practice with, then I recommend Refdash, which is an Interview-As-A-Service. They have a group of interviewers who work at big tech companies like Google, Facebook, and Microsoft. These interviewers will assess you on your coding and design skills. The best part of it is they’ll give you a score at the end of it with specific actionable items on how to improve.
• It’s OK to fail. I failed multiple interviews during this whole process. Sometimes you just have a bad day. It’s not the end of the world if you fail. Companies are biased towards saying no because it’s a lower risk for them. A false positive costs more than a false negative in the long run. The first few rejections definitely stung the most. I failed multiple phone screens when I first started interviewing, and my confidence level sunk. I had doubts in my mind about my abilities and started fearing that my skills weren’t relevant in today’s job market. However, I gave myself a tip: If you fail 10 times, then try 10 times more. All you need is one success. That reassurance gave me a lot of confidence to keep pushing through and when my first offer came through, the other offers came much more easily.

It took me about 2 months of deliberate practice and preparation for my interviews. I spent about 20 hours/week, or 80 hours/month, learning and writing notes on top of a full time job.

To build up my resume, it took 3.5 years of focused, deliberate work. I intentionally picked things that were tough and icky so that I could learn more than anyone else. Even though I don’t have a brand name university or top-tier tech company on my resume, I made up for it with a clear, thorough understanding of the projects I worked on. And this was possible because I researched and wrote down notes of everything I learned, and have a system to review them.

Remember: the strong survives, the tough thrives.

TL;DR: Don’t give up, set yourself up for opportunities, practice a lot, and stay hopeful. Focus on the process, and take a disciplined, dedicated approach to the process.

Tools I Recommend

• Designing Data-Intensive Applications: Awesome book for learning about scaling distributed systems! Highly recommended.
• Elements of Programming Interviews: Great for solving coding problems.
• Cracking The Coding Interview: Great for covering foundational CS coding problems.
• Daily Coding Problem.com: This is a free-to-try website that offers free daily coding problems. You can sign up for interesting daily coding challenges, and you can pay for solutions if you want.
• Dropbox: I keep all my files, pictures, resume here. Easy access, installed once and available everywhere. Love it ❤️ (If you sign up thru this link, both of us will get free 500MB!
• CodeRunner: I love this Mac app! I used this multiple times to run ad-hoc Python scripts/functions and it just works amazingly well. ?
• Kafka the Guide: I used this book as a reference guide, and enjoyed it for the high-level description.

(I share more resources I personally have used and recommend on zhiachong.com, if you’re interested in learning more.)

Thanks for reading my story! You can find me on Twitter and LinkedIn. I would love to connect and talk more about tech, startups, travel :D

Credits:

Brandon O’brien, my mentor and good friend, for proof-reading and providing valuable feedback on how to improve this article.

YK Sugishita, an up-and-coming Youtube star who left his job at Google to pursue his dreams, for proof-reading and giving critical feedback.

This article will teach you how to build and deploy everything you need to be able to build a back-end for your application. We'll be using AWS to host all of this and deploying it all using the Serverless Framework.

By the end of this article you'll know how to:

• Set up your AWS account to work with the Serverless Framework
• Set up a Serverless Project and deploy a Lambda
• Create private cloud storage with S3 bucket and upload files from your computer
• Deploy an API using API Gateway and AWS Lambda
• Create a serverless database table with AWS DynamoDB
• Create an API to get data from your DynamoDB table
• Create an API to add data to your DynamoDB table
• Create APIs to store files and get files from your S3 bucket
• Secure all of your API endpoints with API keys

Being able to do all these things gives you the ability to create all the functionality needed from most application back ends.

Serverless Setup with AWS

The Serverless Framework is a tool that we can use as developers to configure and deploy services from our computers. There's a bit of setup to allow all of this to work together and this section will show you how to do that.

Embedded content

To allow Serverless to do work on your account, you need to set up a user for it. To do this, navigate into AWS and search for "IAM" (Identity and Access Management).

Once on the IAM Page, click on Users in the list on the left hand side. This will open the list of users on your account. From here we'll be clicking Add user.

We need to create a user which has Programmatic access and I've called my user ServerlessAccount, but the name doesn't matter too much.

Next, we need to give the user some permissions. When on the permissions screen, select Attach existing policies directly and then select AdministratorAccess. This will give the Serverless Framework permission to create all the resources it needs to.

We don't need to add any tags, so we can move straight onto Review.

On the review window, you'll see the user has been given an Access key ID and a Secret access key. We'll be needing those in the next part so keep this page open.

Serverless Install and Configuration

Now that we've created our user, we need to install the Serverless Framework on our machine.

Open up a terminal and run this command to install Serverless globally on your computer. If you haven't got NodeJS installed check out this page.

npm install -g serverless

Now that we've got Serverless installed, we need to set up the credentials for Serverless to use. Run this command, putting your access key ID and Secret access key in the correct places:

serverless config credentials --provider aws --key ${Your access key ID} --secret ${Your secret access key} --profile serverlessUser

Once this has been run, you're all set up with Serverless.

Deploying Your First AWS Lambda

With out serverlessUser set up, we want to deploy something using the Serverless Framework. We can use Serverless templates to setup a basic project that we can deploy. This will be the base for the whole of this Serverless project.

Embedded content

In your terminal we can create a Serverless project from a template. This command will create a NodeJS Serverless project in the myServerlessProject folder:

serverless create --template aws-nodejs --path myServerlessProject

If you now open the folder up in your code editor we can look at what we've created.

We've got two file worth talking about: handler.js and serverless.yml

handler.js

This file is a function that will be uploaded as a Lambda function to your AWS account. Lambda functions are great and we'll use a lot more of them later on in the series.

serverless.yml

This is a very important file for us. This is where all the configuration for our deployment goes. It tells Serverless what runtime to use, which account to deploy to, and what to deploy.

We need to make a change to this file so that our deployment works properly. In the provider object we need to add a new line of profile: serverlessUser. This tells Serverless to use the AWS credentials we created in the last section.

We can scroll down to functions and see that we have one function which is called hello and points to the function within the handler.js file. This means we will be deploying this Lambda function as part of this project.

We'll learn a lot more about this serverless.yml file later on in this article.

Deploying Our Project

Now that we've looked at the files it's time to do our first deployment. Open up a terminal and navigate to our project folder. Deploying is as simple as typing:

serverless deploy

This takes a while, but when it's done we can check that everything has deployed successfully.

Open up your browser and navigate to your AWS account. Search for Lambda and you'll see a list of all your Lambda functions. (If you don't see any then check that your region is set to N. Virginia). You should see the myserverlessproject-dev-hello Lambda which contains the exact code that is in the handler.js file in your project folder.

Deploying an S3 Bucket and Uploading Files

In this section we're going to learn how we can deploy an Amazon S3 bucket and then sync up files from our computer. This is how we can start using S3 as cloud storage for our files.

Embedded content

Open up the serverless.yml file and remove all the commented out lines. Scroll to the bottom of the file and add the following code to include our S3 resources:

resources:
    Resources:
        DemoBucketUpload:
            Type: AWS::S3::Bucket
            Properties:
                BucketName: EnterAUniqueBucketNameHere

Change the name of the bucket and we're ready to deploy again. Open up your terminal again and run serverless deploy. You may get an error saying that the bucket name is not unique, in which case you'll need to change the bucket name, save the file and rerun the command.

If it is successful we can then go and see our new S3 bucket in our AWS Console through our browser. Search for S3 and then you should see your newly created bucket.

Syncing up your files

Having a bucket is great, but now we need to put files in the bucket. We're going to be using a Serverless plugin called S3 Sync to do this for us. To add this plugin to our project we need to define the plugins. After your provider object, add this code:

plugins:
    - serverless-s3-sync

This plugin also needs some custom configuration, so we add another field to our serverless.yml file, changing out the bucket name for yours:

custom:
    s3Sync:
        - bucketName: YourUniqueBucketName
          localDir: UploadData

This section of code is telling the S3 Sync plugin to upload the contents of the UploadData folder to our bucket. We don't currently have that folder so we need to create it and add some files. You can add a text file, an image, or whatever you want to be uploaded, just make sure there is at least 1 file in the folder.

The last thing we need to do is to install the plugin. Luckily, all Serverless plugins are also npm packages, so we can install it by running npm install --save-dev serverless-s3-sync in our terminal.

As we've done before, we can now run serverless deploy and wait for the deployment to complete. Once it is complete we can go back into our browser and into our bucket and we should see all the files that we put in the UploadData folder in our project.

Creating an API with Lambda and API Gateway

In this section we'll learn to do one of the most useful things with Serverless: create an API. Creating an API allows you to do so many things, from getting data from databases, S3 storage, hitting other APIs, and much more!

Embedded content

To create the API we first need to create a new Lambda function to handle the request. We're going to make a few Lambdas, so we're going to create a lambdas folder in our project with two subfolders, common and endpoints.

Inside the endpoints folder we can add a new file called getUser.js. This API is going to allow someone to make a request and get back data based on the ID of a user. This is the code for the API:

const Responses = require('../common/API_Responses');

exports.handler = async event => {
    console.log('event', event);

    if (!event.pathParameters || !event.pathParameters.ID) {
        // failed without an ID
        return Responses._400({ message: 'missing the ID from the path' });
    }

    let ID = event.pathParameters.ID;

    if (data[ID]) {
        // return the data
        return Responses._200(data[ID]);
    }

    //failed as ID not in the data
    return Responses._400({ message: 'no ID in data' });
};

const data = {
    1234: { name: 'Anna Jones', age: 25, job: 'journalist' },
    7893: { name: 'Chris Smith', age: 52, job: 'teacher' },
    5132: { name: 'Tom Hague', age: 23, job: 'plasterer' },
};

If the request doesn't contain an ID then we return a failed response. If there is data for that ID then we return that data. If there isn't data for that user ID then we also return a failure response.

As you may have noticed we are requiring in the Responses object from API_Responses. These responses are going to be common to every API that we make, so making this code importable is a smart move. Create a new file called API_Responses.js in the common folder and add this code:

const Responses = {
    _200(data = {}) {
        return {
            headers: {
                'Content-Type': 'application/json',
                'Access-Control-Allow-Methods': '*',
                'Access-Control-Allow-Origin': '*',
            },
            statusCode: 200,
            body: JSON.stringify(data),
        };
    },

    _400(data = {}) {
        return {
            headers: {
                'Content-Type': 'application/json',
                'Access-Control-Allow-Methods': '*',
                'Access-Control-Allow-Origin': '*',
            },
            statusCode: 400,
            body: JSON.stringify(data),
        };
    },
};

module.exports = Responses;

This set of functions are used to simplify the creation of the correct response needed when using a Lambda with API Gateway (which we'll do in a sec). The methods add headers, a status code, and stringify any data that needs to be returned.

Now that we have the code for our API, we need to set it up in our serverless.yml file. Scroll to the functions section of the serverless.yml file. In the last part of this guide we deployed the hello function, but we no longer need that. Delete the functions object and replace it with this:

functions:
    getUser:
        handler: lambdas/endpoints/getUser.handler
        events:
            - http:
                  path: get-user/{ID}
                  method: GET
                  cors: true

This code is creating a new Lambda function called getUser which is in the file of lambdas/getUser on the method of handler. We then define the events that can trigger this lambda function to run.

To make a Lambda into an API we can add a http event. This tells Serverless to add an API Gateway to this account and then we can define the API endpoint using path. In this case get-user/{ID} means the URL will be https://${something-provided-by-API-Gateway}/get-user/{ID}, where the ID is passed into the Lambda as a path parameter. We also set the method to GET and enable CORS so that we could access this endpoint from a front end application if we wanted.

We can now deploy again, and this time we can use the shorthand command sls deploy. This only saves a few characters, but helps avoid a lot of typos. When this is completed we'll get an output that also includes a list of endpoints. We can copy our endpoint and head over to a browser to test it out.

If we paste our API URL into our browser and then add an ID to the end of 5132 we should get back a response of { name: 'Tom Hague', age: 23, job: 'plasterer' }. If we enter a different ID such as 1234 we'll get different data, but entering an ID of 7890 or not entering an ID will return an error.

If we want to add more data to our API, we can simply add a new row to the data object in the getUser.js file. We can then run a special command which only deploys one function, sls deploy -f ${functionName}. So for us that is:

sls deploy -f getUser

If you now make a request using the ID of the new data, the API will return that new data instead of an error.

Creating a Database on AWS

DynamoDB is a fully hosted, non-relational database on AWS. This is the perfect solution for storing data that you need to access and update regularly. In this section we're going to learn how we can create a DynamoDB table with Serverless.

Embedded content

In our serverless.yml file we're going to add some configuration to the Resources section:

resources:
    Resources:
        DemoBucketUpload:
            Type: AWS::S3::Bucket
            Properties:
                BucketName: ${self:custom.bucketName}
        # New Code
        MyDynamoDbTable:
            Type: AWS::DynamoDB::Table
            Properties:
                TableName: ${self:custom.tableName}
                AttributeDefinitions:
                    - AttributeName: ID
                      AttributeType: S
                KeySchema:
                    - AttributeName: ID
                      KeyType: HASH
                BillingMode: PAY_PER_REQUEST

In this code we can see that we are creating a new DynamoDB table with a TableName of ${self:custom.tableName}, defining an attribute of ID and setting the billing mode to pay per request.

This is our first look at the use of variables in our serverless.yml file. We can use variables for a few reasons and they can make our jobs much easier. In this case, we're referencing the variable custom.tableName. We can then reference this variable from multiple locations without having to copy and paste the table name. To get this to work we also need to add tableName to the custom section. In our case we're going to add the line tableName: player-points to create a table to store the points a player has. This table name only needs to be unique to your account.

When defining a table you need to define at least one of the fields which will be your unique identifying field. Because DynamoDB is a non-relational database, you don't need to define the full schema. In our case we've defined the ID, stating that it has an attribute type of string and a key type of HASH.

The last part of the definition is the billing mode. There are two ways to pay for DynamoDB:

• pay per request
• provisioned resources.

Provisioned resources lets you define how much data you're going to be reading and writing to the table. The issues with this are that if you start using more your requests get throttled, and that you pay for the resource even if no one is using it.

Pay Per Request it's much simpler as you just pay per request. This means if you have no one using it then you pay nothing and if your have hundreds of people using it at once, all the requests work. For this added flexibility you pay slightly more for Pay Per Request, but in the long run it usually works out to be cheaper.

Once we've run sls deploy again we can open up our AWS console and search for DynamoDB. We should be able to see our new table and we can see that there is nothing in there.

To add data to the table, click Create item, give it a unique ID, click the plus button and append to a new field and select the type of string. We need to give it a field of name and a value of Jess. Add a number field of score set to 12. Click save and you now have data in your dynamo table.

Getting Data from your DynamoDB Table

Now that we have our Dynamo table created, we want to be able to get and add data to the table. We're going to start with getting data from the table with a get endpoint.

Embedded content

We're going to create a new file in our endpoints folder called getPlayerScore.js. This Lambda endpoint is going to handle the requests for a user and get that data from the Dynamo table.

const Responses = require('../common/API_Responses');
const Dynamo = require('../common/Dynamo');

const tableName = process.env.tableName;

exports.handler = async event => {
    console.log('event', event);

    if (!event.pathParameters || !event.pathParameters.ID) {
        // failed without an ID
        return Responses._400({ message: 'missing the ID from the path' });
    }

    let ID = event.pathParameters.ID;

    const user = await Dynamo.get(ID, tableName).catch(err => {
        console.log('error in Dynamo Get', err);
        return null;
    });

    if (!user) {
        return Responses._400({ message: 'Failed to get user by ID' });
    }

    return Responses._200({ user });
};

The code used here is very similar to the code inside the getUser.js file. We are checking that a path parameter of ID exists, getting the user data, and then returning the user. The main difference is how we are getting the user.

We have imported the Dynamo function object and are calling Dynamo.get. We're passing in the ID and the table name and then catching any errors. We now need to create that Dynamo function object in a new file called Dynamo.js in the common folder.

const AWS = require('aws-sdk');

const documentClient = new AWS.DynamoDB.DocumentClient();

const Dynamo = {
    async get(ID, TableName) {
        const params = {
            TableName,
            Key: {
                ID,
            },
        };

        const data = await documentClient.get(params).promise();

        if (!data || !data.Item) {
            throw Error(`There was an error fetching the data for ID of ${ID} from ${TableName}`);
        }
        console.log(data);

        return data.Item;
    },
};
module.exports = Dynamo;

Reading and writing to Dynamo requires a reasonable amount of code. We could write that code every time we want to use Dynamo but it is much cleaner to have functions to simplify the process for us.

The file first imports AWS and then creates an instance of the DynamoDB Document Client. The document client is the easiest way for us to work with Dynamo from our Lambdas. We create a Dynamo object with an async get function. The only things we need to make a request are an ID and a table name. We format those into the correct parameter format for the DocumentClient, await a documentClient.get request, and make sure that we add .promise() to the end. This turns the request from a callback to a promise which is much easier to work with. We check that we managed to get an item from Dynamo and then we return that item.

We now have the all the code that we need, we have to update our serverless.yml file too. The first thing to do is to add our new API endpoint by adding it to our list of functions.

    getPlayerScore:
        handler: lambdas/endpoints/getPlayerScore.handler
        events:
            - http:
                  path: get-player-score/{ID}
                  method: GET
                  cors: true

There are two more changes that we need to make to get our endpoint working:

• environment variables
• permissions

You may have noticed in the getPlayerScore.js file we had a line of code like this:

const tableName = process.env.tableName;

This is where we are getting the table name from the environment variables of the Lambda. To create our Lambda with the correct environment variables, we need to set a new object in the provider called environment with a field of tableName and a value of ${self:custom.tableName}. This will ensure that we are making the request to correct table.

We also need to give our Lambdas permissions to access Dynamo. We have to add another field to the provider called iamRoleStatements. This has an array of policies which can allow or disallow access to certain services or resources:

provider:
    name: aws
    runtime: nodejs10.x
    profile: serverlessUser
    region: eu-west-1
    environment:
        tableName: ${self:custom.tableName}
    iamRoleStatements:
        - Effect: Allow
          Action:
              - dynamodb:*
          Resource: '*'

As all this has been added to the provider object, it will be applied to all Lambdas.

We can now run sls deploy again to deploy our new endpoint. When that is done we should get an output with a new endpoint of https://${something-provided-by-API-Gateway}/get-player-score/{ID}. If we copy that URL into a browser tab and add the ID of the player that we created in the last section, we should get a response.

Adding New Data to DynamoDB

Being able to get data from Dynamo is cool, but it's quite useless if we can't also add new data to the table as well. We're going to be creating a POST endpoint to create new data in our Dynamo table.

Embedded content

Start by creating a new file in our endpoints folder called createPlayerScore.js and adding this code:

const Responses = require('../common/API_Responses');
const Dynamo = require('../common/Dynamo');

const tableName = process.env.tableName;

exports.handler = async event => {
    console.log('event', event);

    if (!event.pathParameters || !event.pathParameters.ID) {
        // failed without an ID
        return Responses._400({ message: 'missing the ID from the path' });
    }

    let ID = event.pathParameters.ID;
    const user = JSON.parse(event.body);
    user.ID = ID;

    const newUser = await Dynamo.write(user, tableName).catch(err => {
        console.log('error in dynamo write', err);
        return null;
    });

    if (!newUser) {
        return Responses._400({ message: 'Failed to write user by ID' });
    }

    return Responses._200({ newUser });
};

This code is very similar to the getPlayerScore code with a few changes. We are getting the user from the body of the request, adding the ID to the user, and then passing that to a Dynamo.write function. We need to parse the event body as API Gateway stringifies it before passing it to the Lambda.

We now need to modify the common Dynamo.js file to add the .write method. This performs very similar steps to the .get function and returns the newly created data:

    async write(data, TableName) {
        if (!data.ID) {
            throw Error('no ID on the data');
        }

        const params = {
            TableName,
            Item: data,
        };

        const res = await documentClient.put(params).promise();

        if (!res) {
            throw Error(`There was an error inserting ID of ${data.ID} in table ${TableName}`);
        }

        return data;
    }

We've created the endpoint and common code, so the last thing we need to do is modify the serverless.yml file. As we added the environment variable and permissions in the last section, we just need to add the function and API configuration. This endpoint is different from the previous two because the method is POST instead of GET:

    createPlayerScore:
        handler: lambdas/endpoints/createPlayerScore.handler
        events:
            - http:
                  path: create-player-score/{ID}
                  method: POST
                  cors: true

Deploying this with sls deploy will now create three endpoints, including our create-player-score endpoint. Testing a POST endpoint is more complex than a GET request, but luckily there are tools to help us out. I use Postman to test all my endpoints as it makes it quick and easy.

Create a new request and paste in your create-player-score URL. You need to change the request type to POST and set the ID at the end of the URL. Because we're doing a POST request we can send up data within the body of the request. Click body then raw and select JSON as the body type. You can then add the data that you want to put into your table. When you click Send, you should get a successful response:

To validate that your data has been added to the table, you can make a get-player-score request with the ID of the new data you just created. You can also go into the Dynamo console and look at all the items in the table.

Creating S3 GET and POST Endpoints

Dynamo is a brilliant database storage solution, but sometimes it isn't the best storage solution. If you've got data that isn't going to change and you want to save some money, or if you want to store files other than JSON, then you might want to consider Amazon S3.

Embedded content

Creating endpoints to get and create files in S3 is very similar to DynamoDB. We need to create two endpoint files, a common S3 file, and modify the serverless.yml file.

We're going to start with adding a file to S3. Create a createFile.js file in the endpoints folder and add this code:

const Responses = require('../common/API_Responses');
const S3 = require('../common/S3');

const bucket = process.env.bucketName;

exports.handler = async event => {
    console.log('event', event);

    if (!event.pathParameters || !event.pathParameters.fileName) {
        // failed without an fileName
        return Responses._400({ message: 'missing the fileName from the path' });
    }

    let fileName = event.pathParameters.fileName;
    const data = JSON.parse(event.body);

    const newData = await S3.write(data, fileName, bucket).catch(err => {
        console.log('error in S3 write', err);
        return null;
    });

    if (!newData) {
        return Responses._400({ message: 'Failed to write data by filename' });
    }

    return Responses._200({ newData });
};

This code is almost identical to the createPlayerScore.js code, but uses a filename instead of an ID and S3.write instead of Dynamo.write.

Now we need to create our S3 common code to simplify requests made to S3:

const AWS = require('aws-sdk');
const s3Client = new AWS.S3();

const S3 = {
    async write(data, fileName, bucket) {
        const params = {
            Bucket: bucket,
            Body: JSON.stringify(data),
            Key: fileName,
        };
        const newData = await s3Client.putObject(params).promise();
        if (!newData) {
            throw Error('there was an error writing the file');
        }
        return newData;
    },
};
module.exports = S3;

Again, the code in this file is very similar to the code in Dynamo.js, with a few differences around the parameters for the request.

The last thing we need to do for writing to S3 is change the severless.yml file. We need to do four things: add environment variables, add permissions, add the function, and add an S3 bucket.

In the provider we can add a new environment variable of bucketName: ${self:custom.s3UploadBucket}.

To add permission to read and write to S3 we can add a new permission to the existing policy. Straight after - dynamodb:* we can add the line - s3:*.

Adding the function is the same as we've been doing with all our other functions. Make sure that the path has a parameter of fileName as that is what you are checking for in your endpoint code:

    createFile:
        handler: lambdas/endpoints/createFile.handler
        events:
            - http:
                  path: create-file/{fileName}
                  method: POST
                  cors: true

Lastly we need to create a new bucket to upload these files into. In the custom section we need to add a new field s3UploadBucket and set it to a unique bucket name. We also need to configure the resource. After the Dynamo table config, we can add this to create a new bucket for our file uploads:

        s3UploadBucket:
            Type: AWS::S3::Bucket
            Properties:
                BucketName: ${self:custom.s3UploadBucket}

With this set up it is time to deploy again. Running sls deploy again will deploy the new upload bucket as well as the S3 write endpoint. To test the write endpoint, we'll need to head back over to Postman.

Copy in the create-file URL that you get when Serverless has completed the deployment and paste it into Postman and change the request type to POST. Next, what we need to do is to add the filename that we are uploading. In our case we're going to be uploading car.json. The last thing we need to do is add the data to the request. Select Body then raw with a type of JSON. You can add whatever JSON data you would like but here's some example data:

{
    "model": "Ford Focus",
    "year": 2018,
    "colour": "red"
}

When you post this data up, you should get a 200 response with an ETag reference to the file. Going into the console and your new S3 bucket you should be able to see car.json.

Getting Data from S3

Now that we can upload data to S3, we want to be able to get it back too. We start by creating a getFile.js file inside the endpoints folder:

const Responses = require('../common/API_Responses');
const S3 = require('../common/S3');

const bucket = process.env.bucketName;

exports.handler = async event => {
    console.log('event', event);

    if (!event.pathParameters || !event.pathParameters.fileName) {
        // failed without an fileName
        return Responses._400({ message: 'missing the fileName from the path' });
    }

    const fileName = event.pathParameters.fileName;

    const file = await S3.get(fileName, bucket).catch(err => {
        console.log('error in S3 get', err);
        return null;
    });

    if (!file) {
        return Responses._400({ message: 'Failed to read data by filename' });
    }

    return Responses._200({ file });
};

This should look pretty similar to previous GET endpoints we've created before. Differences are the use of the fileName path parameter, S3.get, and returning the file.

Inside the common s3.js file we need to add the get function. The main difference between this and getting from Dynamo is that when we get from S3, the result is not a JSON response, but a Buffer. This means that if we upload a JSON file, it won't come back down in JSON format, so we check if we're getting a JSON file and then transform it back to JSON:

    async get(fileName, bucket) {
        const params = {
            Bucket: bucket,
            Key: fileName,
        };
        let data = await s3Client.getObject(params).promise();
        if (!data) {
            throw Error(`Failed to get file ${fileName}, from ${bucket}`);
        }
        if (fileName.slice(fileName.length - 4, fileName.length) == 'json') {
            data = data.Body.toString();
        }
        return data;
    }

Back in our serverless.yml file, we can add a new function and endpoint for getting files. We've already configured the permissions and environment variables:

    getFile:
        handler: lambdas/endpoints/getFile.handler
        events:
            - http:
                  path: get-file/{fileName}
                  method: GET
                  cors: true

As we're creating a new endpoint we need to do a full deployment again with sls deploy. We can then take the new get-file endpoint and paste it into a browser or Postman. If we add car.json to the end of the request we'll receive the JSON data that we uploaded earlier in this section.

Securing Your Endpoints with API Keys

Being able to create API endpoints quickly and easily with Serverless is great for starting a project and creating a proof of concept. When it comes to creating a production version of your application, you need to start being more careful about who can access your endpoints. You don't want anybody being able to hit your APIs.

Embedded content

To secure your APIs there are loads of methods, and in this section we're going to be implementing API keys. If you don't pass the API key with the request then it fails with an unauthorised message. You can then control who you give the API keys to, and therefore who has access to your APIs.

You can also add usage policies to your API keys so that you can control how much each person uses your API. This allows you to created tiered usage plans for your service.

To start we're going to be creating a simple API Key. To do this we need to go into our serverless.yml file and add some configuration to the provider.

    apiKeys:
        myFirstAPIKey

This will create a new API key. Now we need to tell Serverless which API endpoints to protect with the API key. This has been done so that we can have some of the APIs protected, whilst some of them stay public. We specify that an endpoint needs to be protected by adding the option private: true:

    getUser:
        handler: lambdas/endpoints/getUser.handler
        events:
            - http:
                  path: get-user/{ID}
                  method: GET
                  cors: true
                  private: true

You can then add this field to as many of your APIs as you would like. To deploy this we can run sls deploy again. When this completes, you will get back an API key in the return values. This is very important and we'll use it very soon. If you try and make a request to your get-user API you should get a 401 Unauthorised error.

To get the request to succeed, you now need to pass up an API key in the headers of the request. To do this we need to use Postman or another API request tool and add header to our get request. We do this by selecting Authorisation using the API type. The key needs to be X-API-KEY and the value is the key that you got as an output from your Serverless deploy:

Now when we make the request we get a successful response. This means that the only people who can access your API are people who you have given your API key to.

This is great, but we can do more. We can add a usage policy to this API key. This is where we can limit the number of requests a month as well as the rate at which requests can be made. This is great for running a SAAS product as you can provide an API key that gives users a set amount of API calls.

To create a usage plan we need to add a new object in the provider. The quota section defines how many requests can be made using that API key. You can change the period to either DAY or WEEK if that would suit your application better.

The throttle section allows you to control how frequently your API endpoints can be hit. Adding a throttle rate limit sets a maximum number of requests per second. This is very useful as it stops people from setting up a denial of service attack. The burstLimit allows the API to be hit more often than your rateLimit but only for a short period of time, normally a few seconds:

    usagePlan:
        quota:
            limit: 10
            period: MONTH
        throttle:
            burstLimit: 2
            rateLimit: 1

If we were to deploy this again, the deployment would fail as we would be trying to deploy the same API key. API keys need to be unique so we have to change the name of the API key. When we deploy this and copy our new API key into Postman, we'll be able to make requests as we normally would. If we try and make too many requests per second or reach the maximum number of requests then we'll get a 429 error of

{
    "message": "Limit Exceeded"
}

This means that you can't use this API key again until next month.

Whilst creating a usage plan is great, you often want to give different people different levels of access to your services. You might give free users 100 requests per month and paying users get 1000. You might want different payment plans which give different number of requests. You would also probably want a master API key for yourself which has unlimited requests!

To do this we can set up multiple groups of API keys that each have their own usage policy. We need to change the apiKeys and usagePlan sections:

    apiKeys:
        - free:
              - MyAPIKey3
        - paid:
              - MyPaidKey3
    usagePlan:
        - free:
              quota:
                  limit: 10
                  period: MONTH
              throttle:
                  burstLimit: 2
                  rateLimit: 1
        - paid:
              quota:
                  period: MONTH
                  limit: 1000
              throttle:
                  burstLimit: 20
                  rateLimit: 10

Once you've saved and deployed this you'll get two new API keys, each with a different level of access to your API endpoints.

Thanks for reading this guide! If you've found it useful, please subscribe to my Youtube channel where I release weekly videos on Serverless and software development.

What is a Thread?

A thread is a lightweight process. Any process can have multiple threads running in it.

For example in a web browser, we can have one thread which will load the user interface and another thread which will actually retrieve all the data that needs to be displayed in that interface.

What is MultiThreading?

Multithreading enables us to run multiple threads concurrently.

For example in a web browser, we can have one thread which handles the user interface, and in parallel we can have another thread which fetches the data to be displayed.

So multithreading improves the responsiveness of a system.

What is Concurrency?

Concurrency in the context of threads enables us to run multiple threads at the same time.

But do the threads really run at the same time?

Single Core Systems

The Thread Scheduler provided by the JVM decides which thread runs at any given time. The scheduler gives a small time slice to each thread.

So at any given time we have only one thread which is actually running in the processor. But because of the time slicing we get the feeling that multiple threads are running at the same time.

Multi Core Systems

Even in multiple core systems the thread scheduler is involved. But since we have multiple cores, we can actually have multiple threads running at the exact same time.

For example if we have a dual core system, then we can have 2 threads running at the exact same time. The first thread will run in the first core, and the second thread will run in the second core.

Why is Multithreading needed?

Multithreading enables us to improve the responsiveness of a system.

For example in a web browser, if everything ran in a single thread, then system would be completely unresponsive whenever data was being fetched to display. For example, if it takes 10 seconds to fetch the data, then in that 10 seconds we wont be able to do anything else in the web browser like opening new tabs, or even closing the web browser.

So running different parts of a program in different threads concurrently helps improve the responsiveness of a system.

How to write Multithreaded Programs in Java

We can create threads in Java using the following

• Extending the thread class
• Implementing the runnable interface
• Implementing the callable interface
• By using the executor framework along with runnable and callable tasks

We will look at callables and the executor framework in a separate blog. In this article I will be mainly focussing on extending the thread class and implementing the runnable interface.

Extending the Thread Class

In order to create a piece of code which can be run in a thread, we create a class and then extend the thread class. The task being done by this piece of code needs to be put in the run() function.

In the below code you can see that worker is a class which extends the thread class, and the task of printing numbers 0 to 5 is being done inside the run() function.

class Worker extends Thread {

    @Override
    public void run() {
        for (int i = 0; i <= 5; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
        }
    }

}

In the above code Thread.currentThread().getName() is used to get the name of the current thread which is running the code.

In order to create a thread, we just need to create an instance of the worker class. And then we can start the thread using the start() function.

public class ThreadClassDemo {
    public static void main(String[] args) {
        Thread t1 = new Worker();
        Thread t2 = new Worker();
        Thread t3 = new Worker();
        t1.start();
        t2.start();
        t3.start();

    }
}

In the above code, we are creating 3 threads (t1,t2 and t3) from the worker class. Then we are starting the threads using the start() function.

Here is the final code for creating a thread by extending a thread class:

class Worker extends Thread {

    @Override
    public void run() {
        for (int i = 0; i <= 5; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
        }
    }

}

public class ThreadClassDemo {
    public static void main(String[] args) {
        Thread t1 = new Worker();
        Thread t2 = new Worker();
        Thread t3 = new Worker();
        t1.start();
        t2.start();
        t3.start();

    }
}

Here is the output we get by running the above code:

You can see that all the 3 threads have printed the numbers from 0 to 5.

You can also clearly see from the output that the 3 threads do not run in any particular sequence

Implementing the Runnable Interface

In order to create a piece of code which can be run in a thread, we create a class and then implement the runnable interface. The task being done by this piece of code needs to be put in the run() function.

In the below code you can see that RunnableWorker is a class which implements runnable interface, and the task of printing numbers 0 to 4 is being done inside the run() function.

class RunnableWorker implements Runnable{

    @Override
    public void run() {
        for (int i = 0; i <= 4; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
        }
    }

}

In order to create a thread, first we need to create an Instance of RunnableWorker which implements the runnable interface.

Then we can create a new thread by creating an instance of the thread class and passing the instance of RunnableWorker as the argument. This is shown in the code below:

public class RunnableInterfaceDemo {

    public static void main(String[] args) {
        Runnable r = new RunnableWorker();
        Thread t1 = new Thread(r);
        Thread t2 = new Thread(r);
        Thread t3 = new Thread(r);

        t1.start();
        t2.start();
        t3.start();

    }

}

The above code creates a runnable instance r. Then it create 3 threads (t1, t2 and t3) and passes r as the argument to the 3 threads. Then the start() function is used to start all 3 threads.

Here is the complete code for creating a thread by implementing the runnable interface:

class RunnableWorker implements Runnable{

    @Override
    public void run() {
        for (int i = 0; i <= 4; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
        }
    }

}

public class RunnableInterfaceDemo {

    public static void main(String[] args) {
        Runnable r = new RunnableWorker();
        Thread t1 = new Thread(r);
        Thread t2 = new Thread(r);
        Thread t3 = new Thread(r);

        t1.start();
        t2.start();
        t3.start();

    }

}

On running the above code, we will get the following output. The sequence of the output will change every time the code is run.

Implementing the runnable interface is a better option than extending the thread class since we can extend only one class, but we can implement multiple interfaces in Java.

Runnable Interface in Java 8

In Java 8, the runnable interface becomes a FunctionalInterface since it has only one function, run().

The below code shows how we can create a runnable instance in Java 8.

public class RunnableFunctionalInterfaceDemo {

    public static void main(String[] args) {

        Runnable r = () -> {
            for (int i = 0; i <= 4; i++) {
                System.out.println(Thread.currentThread().getName() + ": " + i);
            }
        };

        Thread t1 = new Thread(r);
        Thread t2 = new Thread(r);
        Thread t3 = new Thread(r);

        t1.start();
        t2.start();
        t3.start();
    }

}

Here, instead of creating a class and then implementing the runnable interface, we can directly use a lambda expression to create a runnable instance as shown below:

Runnable r = () -> {
        for (int i = 0; i <= 4; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
        }
    };

Code

The code in this article is available in the following GitHub repo: https://github.com/aditya-sridhar/basic-threads-demo

Congrats ?

You now know how to create threads by extending the thread class and by implementing the runnable interface.

I will discuss the thread life cycle and challenges while using threads in my next blog post.

My Website: https://adityasridhar.com/

Feel free to connect with me on LinkedIn or follow me on Twitter