The support channel has the same complaint arriving in six different tones. Is it down? Mine is just spinning. It worked this morning. Somebody has posted a screenshot of a loading indicator with no caption, which you suspect they're enjoying.

So you open the dashboard.

Nodes ready. Pods running. CPU at 20%. No restarts, nothing in CrashLoopBackOff, no alert fired. By every signal Kubernetes gives you, the system is healthy.

It is not healthy. Somebody is eleven seconds into waiting for the first word of an answer, and they're about to go back to doing it the old way, and they're not coming back.

That is the failure this piece is about. Not the outage. The quiet one, where everything is technically up and the product is still unusable.

Table of Contents

• Your First Instinct is Wrong

• You Are Not the Only One

• The Thing Nobody Told You When You Deployed It

• Where the Eleven Seconds Went

  • Stage One, Routing: Your Load Balancer is Guessing

  • Stage Two, The Queue: Nothing is Coming to Help

  • Stage Three, Prefill: The GPUs Are There, and Useless

  • Stage Four, Decode: Your Ingress is Cutting People Off

  • And Underneath All Four: Bursty Traffic and Nobody's Name on the Problem

• Meanwhile, the Platform Did Not Stand Still

• How This Got Here in the First Place

• What To Do Once the Fire is Out

  • Capacity

  • Performance

  • Placement

  • Resilience

  • Ownership

• Four Things Your Monitoring Still Isn't Telling You

• The Thing That is Still Unsettled

• Monday Morning

• Back to Thursday

Your First Instinct is Wrong

The instinct at that moment is to suspect the model. Wrong version, bad quantization, context too long, or that someone changed the system prompt.

It is almost never the model.

Walk one request end to end and you find four stages. It gets routed to a replica. It waits in a queue. The model reads the prompt. The model streams an answer. Only the last two involve the model doing anything you are paying for. The first two are queueing and routing, which is to say infrastructure.

When people say their model is slow, the model is usually fine. The wait is somewhere else.

You Are Not the Only One

It's worth knowing before you go hunting that this is the most common complaint in the field, not an unlucky configuration on your part.

A practitioner survey of 200 people running AI in production found that nearly half, 49.5%, named latency at peak load as their single hardest scaling problem. Not accuracy. Not hallucination. Not cost per token. Latency, at the exact moment people are trying to use the thing.

Two more figures from the same research are worth holding alongside it. 59.5% said running inference closer to users or decision points is critical or very important. 45.5% still serve from a single cloud region. People know proximity matters and have not managed to act on it, which tells you something about what multi-region GPU capacity costs to stand up.

The same research mentions organizations mandating sub-250ms response times alongside 99.9% availability. Read that carefully, because taken literally it is impossible. No meaningful LLM response completes in a quarter of a second. It has to mean time to first token (how long before the first word of the answer appears), and that is the right thing to hold yourself to anyway. Once tokens are flowing at a readable pace, people stop counting. The silence before the first one is what loses them.

The Thing Nobody Told You When You Deployed It

Here's the part that explains everything else.

A normal web request is short, small, and costs roughly what the last one cost. Kubernetes scheduling assumes that. Service load balancing assumes it. The Horizontal Pod Autoscaler assumes it. Your ingress controller's default timeout assumes it. Nobody wrote the assumption down, because for fifteen years it was simply true.

An LLM request breaks it in three places.

It runs in two phases with completely different profiles. Prefill: the model reads the entire prompt in one pass. Compute-bound, bursty, and this is what decides how long your user stares at nothing. Then decode: one token at a time, each needing the accumulated state of every token before it. That state is the KV cache. It lives in GPU memory next to the weights, and it grows as the answer grows.

So requests are not short, they can run for a minute. They do not cost the same, since a prompt with a document pasted into it can cost fifty times what a one-liner costs. And the genuinely scarce resource is GPU memory, which does not appear on a single dashboard you inherited.

That is why your monitoring said everything was fine. It was measuring the wrong machine.

Where the Eleven Seconds Went

Follow the request through the four stages and the failure modes fall out in order. These six are the ones practitioners name most often, and they map neatly onto the path.

Stage One, Routing: Your Load Balancer is Guessing

Round-robin stops being fair the moment request costs diverge. Three heavy prompts land on one pod while its neighbour handles one-liners. The loaded pod's queue grows, its tail latency climbs, and the fleet average still looks completely reasonable, which is why nobody noticed before today.

Long-lived connections make it worse. With HTTP/2 or keepalive, a client opens one connection and sends everything down it, and a Kubernetes Service balances per connection rather than per request. All that traffic pins to a single backend and stays there.

Then there is the cost most teams have never considered. If a replica already holds the prefix of this prompt in its KV cache, routing there skips real prefill work. Since most production prompts share a long system preamble, that is not an edge case, it is most of your traffic. Round-robin throws the benefit away at random, every time.

Stage Two, The Queue: Nothing is Coming to Help

Your autoscaler could add capacity. It will be late, and the reason is unglamorous: a new replica has to pull a multi-gigabyte serving image, fetch 20GB to 70GB of weights over the network, and load them into GPU memory. Minutes, not seconds, and that assumes a node with a free card already exists. If it does not, add provisioning time, and add whether your provider has stock in that zone today.

There is a second problem stacked on the first. The signal is usually wrong. CPU tells you nothing here, because the CPU is idle while the GPU works. GPU utilization is barely better: it reports that a kernel was executing during the sample window, and a server handling one request and a server handling sixty both read 100%. It cannot tell you whether anyone is waiting.

Queue depth can. vLLM already exposes running and waiting request counts as Prometheus metrics. Scale on those:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-server
spec:
  scaleTargetRef:
    name: llm-server
  minReplicaCount: 2
  maxReplicaCount: 8
  cooldownPeriod: 600
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: sum(vllm:num_requests_waiting{app="llm-server"})
        threshold: "5"

Check the metric names against your vLLM version. Several have been renamed across releases, and the failure is silent.

Two values there are deliberate. The minimum of 2, because a cold start means you cannot sit at zero and expect to serve. The ten minute cooldown, because scaling down eagerly means paying that cold start again shortly afterwards.

Which leads to the uncomfortable conclusion about autoscaling inference. Reactive scaling is permanently late by however long your cold start is, and no trigger fixes that. The fix is to keep more spare capacity than feels comfortable. Run extra replicas at all times, scale up earlier than you think you need to, and scale down more slowly. This also means scale to zero isn't a good fit for user-facing inference. Save it for batch jobs, where nobody is sitting there waiting for a response.

Stage Three, Prefill: The GPUs Are There, and Useless

Kubernetes hands out CPU in millicores and GPUs whole. A pod asks for one card and gets the entire card, whether it needs 10% of it or all of it.

The waste is the obvious problem. Fragmentation is the one that ruins a Thursday. A 70B model in 16-bit needs roughly 140GB of weights, so a replica needs several GPUs on one node. You can have six free GPUs spread two-two-one-one across four nodes, and the replica sits in Pending indefinitely. Capacity on paper, none of it usable, and no alert, because nothing is broken.

Sharing a card gives three options, each with a catch you should know before choosing. Time-slicing is easy to enable and gives no memory isolation, so one greedy pod can take down its neighbour. Fine for development, not for anything customer-facing. MIG gives real hardware isolation but fixed slice geometry, which means predicting your workload mix before you have one. Dynamic Resource Allocation is the structurally correct answer and brings accelerator requests properly into the core Kubernetes API.

None of these are on by default. Each needs a deliberate decision from somebody who understands the workload, who is usually not in the room when the cluster gets built.

Stage Four, Decode: Your Ingress is Cutting People Off

Many ingress controllers ship with a 60 second read timeout and response buffering enabled.

The timeout terminates long generations mid-sentence. Users report that as a crash, and you cannot reproduce it, because your test prompt is short. The buffering holds tokens and releases them in clumps, which destroys the feel of streaming while the model is behaving perfectly.

Two config lines, written for a workload that no longer exists.

And Underneath All four: Bursty Traffic and Nobody's Name on the Problem

Internal AI tools don't get steady traffic. They get sudden spikes: when everyone starts work at 9am, in the hour after a company all-hands meeting, or when someone shares the link in a busy Slack channel and says it's actually quite good. A fixed replica count handles exactly one of those. Most teams set it once during a quiet week and never revisit it, so they get both waste and collapse on the same day.

Then the sixth failure mode, which is not technical at all, and which is the reason the other five survive for quarters.

Platform engineering owns the cluster. The cluster is green. From where they sit, the job is done and done well. Application developers can see latency is bad and cannot see why, because every cause sits in scheduling, scaling and routing that they neither control nor observe. MLOps owns the model artifact and almost nothing that determines how it performs at quarter past two on a Thursday.

Three teams, all telling the truth, and the problem living in the gap between their on-call rotas, where no alert is configured and no dashboard points.

Meanwhile, the Platform Did Not Stand Still

The encouraging part of this story is that the gaps above are known, and the fixes have been shipping.

Dynamic Resource Allocation brings accelerator hardware requests into the core Kubernetes API, so the scheduler can reason about the device rather than counting opaque units. Kueue adds job queueing and multi-tenant GPU quota alongside CPU and memory, which is how you stop a research job starving the endpoint your users depend on. Gateway API Inference Extension and llm-d add model-aware routing: request criticality, dynamic balancing from live model metrics, and awareness of cache state. That last one is the direct answer to the guessing load balancer in stage one.

Published benchmarks give a sense of scale: Kueue reducing multi-stage workload makespan by up to 15%, dynamic accelerator slicing cutting mean job completion time by 36%, and Gateway API plus llm-d improving tail time to first token by up to 90% under heavy load.

Treat those as headroom rather than forecast. Up to 90% is measured on a configuration chosen to demonstrate the improvement, and your gain depends entirely on how poor your baseline is. If you already run least-outstanding-requests balancing with a warm cache, expect far less. If you are on stock round-robin behind a default ingress, possibly something dramatic, because that baseline is genuinely bad.

The direction is what matters. The biggest published gain is in tail time to first token, which is precisely what half the surveyed practitioners named as their worst problem. Somebody built the fix for the thing people were complaining about.

How This Got Here in the First Place

Worth a short detour, because it explains why the tooling arrived late.

When generative AI landed in enterprise planning documents, the confident position was that Kubernetes could not hold it. Container orchestration was built for small fungible units of CPU and memory, cheap restarts, interchangeable replicas. AI needed scheduled specialized silicon carrying enormous state that takes minutes to place. A purpose-built platform was coming.

It never arrived. AI went into the microservices stack and Kubernetes absorbed it.

The survey data explains why. More than half of enterprises do not train models at all, and only 7% deploy a model on any given day. Meanwhile 82% of container users run Kubernetes in production, and 66% of organizations hosting generative AI already serve it there. The industry spent years designing for the workload almost nobody has, while everybody else quietly downloaded weights and put them behind an endpoint.

What actually converged was not a place to run AI. It was a standard way to run it, with the location left negotiable. Which is exactly why your problems today are placement, routing and capacity rather than anything model-shaped.

What To Do Once the Fire is Out

The path from working to reliable comes down to five things: capacity, performance, placement, resilience and ownership.

Capacity

Requests per second is meaningless when one request is 50 tokens and the next is 8,000. Plan in tokens per second and track prompt and completion separately, because they stress different parts of the system. Prompt tokens are a compute burst during prefill. Completion tokens are a sustained drip during decode that holds GPU memory for the life of the response.

Load test with your worst realistic prompt at your worst realistic concurrency, and plan from that number. A benchmark built on short prompts gives a reassuring figure that evaporates in week one.

On supply, treat GPUs as something you reserve ahead of time rather than request when needed. Keep a committed baseline and burst on top. Find out your provider's quota and regional stock position before the evening you need it, and use Kueue so one team cannot quietly consume the pool.

Performance

Two numbers matter most. Time to first token (TTFT) is how long a user waits between sending a prompt and seeing the first word of the answer. It decides whether people trust the product. Inter-token latency is the gap between each word after that, and it decides whether the answer feels smooth to read.

Track both at p95 and p99, not the average. These are percentiles. p95 is the time that 95% of requests come in under, so it shows what your slowest 1 in 20 users experience. p99 does the same for the slowest 1 in 100. An average can look healthy while those users wait far too long, and they're the ones who give up.

Next, tune the serving layer. This is the software that loads the model and handles requests, such as vLLM. It's where you'll get the biggest improvements for the least effort and cost. Continuous batching keeps the GPU busy without making early requests wait for a batch to fill. Quantization roughly halves memory footprint where the quality tradeoff is acceptable, and freed memory becomes concurrent requests. Set maximum context length to what your product actually needs rather than what the model supports. If the model handles 128K and your longest genuine prompt is 6K, you are reserving KV cache for a scenario that never occurs. One config line, and it can substantially raise effective concurrency.

Placement

Taint GPU nodes so general workloads cannot land on them and only inference pods with matching tolerations schedule there. A logging sidecar should never be the reason a model replica cannot place.

Keep weights close. Node-local NVMe or a zone-local cache turns a 40GB network pull into a fast local read, which feeds directly into cold start, autoscaling lag, and the peak-hour latency you spent Thursday afternoon on. This is the highest-leverage unglamorous fix on the list.

Respect topology. Cards connected by NVLink behave very differently from cards that merely share a PCIe bus, and for a sharded model that interconnect sits on the critical path of every token generated.

Then be honest about geography. If your users are in London and your GPUs are in Virginia, every token crosses an ocean, and a well-tuned model behind a long network path is still a slow product.

Resilience

Readiness should pass only when the model is loaded and genuinely able to serve. Use a startup probe with a generous failure threshold, otherwise liveness kills the pod partway through loading weights and you get a restart loop that presents as a mystery and costs an hour.

Set a PodDisruptionBudget so a routine node upgrade cannot remove half your replicas at once.

Raise the termination grace period past the 30 second default and add a preStop hook. A generation can easily run longer than that, so otherwise every rollout severs in-flight streams and your users experience deployments as random failure.

Shed load deliberately. Past a queue depth threshold, return a fast busy-try-again rather than accepting a request that will hang for two minutes. This feels wrong to engineers and is right for users. People forgive a quick honest retry. Nobody forgives a frozen screen.

Have somewhere to fall back to: a second region, a smaller model covering the common cases, or an external API at the edge of disaster. Given that 45.5% of practitioners run from a single region, this is the most commonly skipped item on the list, which makes it the most likely candidate for your next incident.

Ownership

Agree on the numbers and write them down: time to first token, inter-token latency, error rate, queue time. Put them on one dashboard showing cluster and model metrics side by side, and make sure both teams open the same link rather than maintaining separate versions of reality.

Split responsibility explicitly. Platform owns the ability to hit the target: capacity, scheduling, routing, scaling, placement. Application owns how the model is configured and called: context length, batching parameters, prompt size, retry behaviour. When the number slips, both show up, and the conversation starts from the same graph instead of two that disagree.

Four Things Your Monitoring Still Isn't Telling You

Latency, traffic, errors and saturation are still necessary and no longer sufficient. Four more belong on the board.

Token consumption rate, split prompt and completion, because that is your real unit of capacity. Cost per request by tenant and query type, because we run eight GPUs is not an answer to what a feature costs per user. Output quality, tracking hallucination rate and guardrail drift, because latency work that degrades quality is not a win. Model attribution, recording which checkpoint and which prompt version produced a response, because when quality moves you need to know what changed.

That last one connects to a shift worth noticing. Model registries have become peers to container registries, and prompts and agent configurations are increasingly versioned as GitOps artifacts in the same delivery pipelines as everything else.

The prompt is code now. It changes production behaviour, it can break things, and it should go through review, versioning and rollback like anything else that does. Plenty of teams are still editing prompts in a web console and wondering why quality shifted on a Tuesday.

The Thing That is Still Unsettled

The substrate has converged, and the speed of it is the evidence. A Kubernetes AI conformance programme launched in late 2025 with 18 platforms. Within roughly four months it had grown to 31 and extended to agentic workloads, covering the major hyperscalers, the enterprise and private cloud distributions, GPU neoclouds and edge networks. Companies that agree on almost nothing agreed on this.

The layer above has not converged at all. AI gateways, evaluation platforms, agent control planes, agent observability. All advancing faster in commercial products than in any standard body, and all sitting exactly where the interesting business logic is heading.

The risk is worth naming plainly. You can be perfectly portable at the Kubernetes layer and completely locked in at the layer where you actually build. Your pods will migrate between clouds beautifully while your agent definitions, eval suites and gateway policies stay exactly where they are, because there is nowhere standard to move them to.

That is not a reason to avoid those tools. They solve real problems today. It is a reason to know which of your components has an exit and which does not, and to make that a decision you wrote down rather than one you discover during a renewal negotiation.

Monday Morning

Start with one measurement. Time a cold start end to end, from pod created to first token served, and write the number down. That figure is your autoscaling floor, and your minimum replica count falls directly out of it.

Put time to first token and queue depth on your main dashboard at p99. Replace CPU-based autoscaling with a queue-depth trigger and minimums that respect the cold start. Check your ingress read timeout and buffering against a real long streaming response rather than a test prompt. Fix the termination grace period so deployments stop cutting people off.

Then, over the next few weeks: taint your GPU nodes, move weights to node-local or zone-local storage, set maximum context length to what you actually use, and find out whether you are still on round-robin. You probably are.

And before the quarter ends, decide who owns the end-to-end latency number, and say it out loud in a room with the other team present.

Back to Thursday

The dashboard was never lying. It was answering a different question.

It was telling you whether the containers were alive, which they were. It could not tell you that requests were queueing behind a badly routed batch, that no new replica was coming for four minutes, that six GPUs were free and unusable, or that the ingress was buffering tokens it should have been streaming. Nothing in the standard toolkit is pointed at any of that.

Getting a model to answer on Kubernetes takes an afternoon. Getting it to answer well when everyone logs in at once is a different discipline, and it looks far more like ordinary infrastructure engineering than the current conversation suggests. Scheduling, routing, capacity, ownership. We have been solving those since long before any of this carried the AI label. The skills are already in the building. They need pointing at the right metric.

A green cluster is where the work starts. What counts is what the person typing the prompt sees.

I hope you’ve enjoyed this and learned something new. I’m always open to suggestions and discussions on LinkedIn. Hit me up with direct messages.

If you’ve enjoyed my writing and want to keep me motivated, consider leaving stars on GitHub and endorsing me for relevant skills on LinkedIn.

Till the next one, happy exploring!

Automating alert provisioning allows you to enforce consistency, secure sensitive credentials, and integrate monitoring into your deployment pipelines.

This guide dives deep into how you can use the SigNoz Terraform Provider to define and manage alert configurations as code, making your observability setup resilient and adaptable.

Here’s what we’ll cover:

1. Why Automate Alert Provisioning?

2. What are SigNoz and Terraform?

3. Overview of the Setup

4. Prerequisites

5. Steps to Set Up the Project

6. Best Practices and Security Considerations

7. Integrating with CI/CD Pipelines

8. Advanced Customizations and Troubleshooting

9. Conclusion

Why Automate Alert Provisioning?

It’s a good idea to automate your alert provisioning for various reasons.

First of all, configuring things manually often leads to discrepancies between environments (development, staging, production). Automating alerts ensures that all environments adhere to the same monitoring standards, reducing the likelihood of configuration drift and improving consistency and uniformity.

Also, when alerts are defined as code, every change is tracked in your version control system. This audit trail makes it easier to trace and review changes, collaborate with team members, and roll back configurations if issues arise.

Something else to consider is that as your infrastructure grows, manually managing alerts becomes unsustainable. Automation allows you to quickly and efficiently update your alerting rules across multiple services without the need for repetitive manual intervention.

Automation also helps improve security. Storing sensitive information like API tokens as environment variables or in secret management systems helps maintain security. Automating the process also minimizes human exposure to critical credentials.

And finally, defining alerts as code enables you to integrate monitoring configurations into your CI/CD pipelines. This leads to continuous testing, validation, and deployment of alert rules alongside application updates.

So as you can see, there are many compelling reasons to go the automation route. Now let’s see how you can do this in practice.

What Are SigNoz and Terraform?

SigNoz is an open-source observability platform designed to collect, analyze, and visualize metrics, logs, and traces from your applications. Its most helpful features include:

• It has comprehensive monitoring abilities: Provides detailed insights into system performance, error rates, and user behaviors.

• It comes equipped with real-time analytics: Enables proactive issue detection and performance optimization.

• It’s community-driven: As an open-source solution, it benefits from community contributions, transparency, and customization.

• It’s cost-effective: Offers powerful observability capabilities without the hefty licensing fees of proprietary solutions.

Terraform is an Infrastructure as Code (IaC) tool developed by HashiCorp. It allows you to define and provision infrastructure using declarative configuration files. Terraform’s core advantages include:

• Its declarative syntax: You specify the desired state of your infrastructure, and Terraform handles the implementation.

• Its version Control: Configuration files can be managed in Git repositories, enabling traceability and rollback of changes.

• Powerful automation: Facilitates automated provisioning and updates, reducing manual effort and errors.

• Multi-cloud support: Manages resources across different cloud providers with a consistent workflow.

So you might be wondering: why should you use Terraform with SigNoz?

First of all, Terraform ensures that your infrastructure is managed consistently across different environments, reducing the risk of configuration drift. It also simplifies managing multiple alerts and resources, making it easier to scale your observability setup.

Beyond this, automating the provisioning process reduces manual setup efforts and minimizes the potential for human error.

And finally, Terraform configurations can be version-controlled, allowing teams to track changes over time and collaborate more effectively.

Overview of the Setup

This setup utilizes the SigNoz Terraform Provider to manage alerts and notification channels within SigNoz Cloud. The configuration includes:

• Provider configuration: Establishes the connection to SigNoz using the API endpoint and a securely provided API token.

• Notification channels: Defines where alerts are sent (for example, via email) to ensure the right teams are notified.

• Alert rules: Specifies the conditions under which alerts are triggered, including thresholds and evaluation windows.

• External variables: Enhances flexibility by allowing critical values (like CPU thresholds and email addresses) to be managed externally.

Prerequisites

Before diving into the setup, make sure you have the following:

1. SigNoz Cloud account: If you don't have one, sign up for SigNoz Cloud to host your observability data and configure alerts.

2. Terraform installed: Install Terraform on your machine. Terraform is the tool you'll use to manage your infrastructure as code.

3. SigNoz API token:

   • Log in to your SigNoz Cloud dashboard.

   • Navigate to Settings > API Tokens.

   • Click Generate API Token.

   • Copy the token, as you'll need it to authenticate Terraform with SigNoz.

4. Basic knowledge of Terraform: Familiarity with Terraform's syntax and concepts, including writing configuration files and running Terraform commands, is essential.

5. Text editor: Use any code editor like Visual Studio Code or Sublime Text to write your Terraform configuration files.

Steps to Set Up the Project

1. Understand the signoz_alert Resource

The signoz_alert resource allows you to create and manage alert rules in SigNoz via Terraform. It supports various alert types, conditions, and configurations. Understanding this resource is crucial as it forms the basis of your alert configuration.

2. Set Up Your Terraform Configuration

Create a new directory for your Terraform configuration:

mkdir signoz-terraform
cd signoz-terraform

Create a main.tf file with the following content:

terraform {
  required_providers {
    signoz = {
      source  = "SigNoz/signoz"
      version = "0.1.3" # Use the latest version from the Terraform Registry
    }
  }
}

provider "signoz" {
  endpoint  = "https://api.us.signoz.cloud" # Replace with your SigNoz Cloud API endpoint
  api_token = var.signoz_api_token
}

variable "signoz_api_token" {}

The provider block configures the SigNoz provider, where endpoint specifies the API endpoint and api_token is passed through a variable for security.

3. Define a Notification Channel (Optional)

If you plan to send alerts to specific channels, define them using signoz_notification_channel. For example, create a channels.tf file:

resource "signoz_notification_channel" "email_channel" {
  name = "Email Channel"
  type = "email"

  receivers {
    email_config {
      to = ["alerts@example.com"]
    }
  }
}

Defining a notification channel ensures that alerts are sent to the correct recipients, enhancing the utility of your alerting system.

4. Create an Alert Using the signoz_alert Resource

Create an alerts.tf file to define your alert:

resource "signoz_alert" "cpu_high_usage" {
  alert            = "High CPU Usage Alert"
  alert_type       = "METRIC_BASED_ALERT"
  severity         = "critical"
  description      = "Alert when CPU usage exceeds 80% over 5 minutes"
  rule_type        = "threshold_rule"
  broadcast_to_all = false
  disabled         = false
  eval_window      = "5m0s"
  frequency        = "1m0s"
  version          = "v4"

  condition = jsonencode({
    compositeQuery = {
      builderQueries = {
        A = {
          aggregateOperator = "avg"
          dataSource        = "metrics"
          metricName        = "cpu_usage_user"
          reduceTo          = "avg"
          filters           = {
            items = []
            op    = "AND"
          }
          groupBy = []
        }
      }
      queryType = "builder"
      panelType = "graph"
      unit      = "%"
    }
    op                = ">"
    target            = 80
    matchType         = "EQUALS"
    selectedQueryName = "A"
    targetUnit        = "%"
  })

  preferred_channels = [signoz_notification_channel.email_channel.name]

  labels = {
    severity = "critical"
    team     = "DevOps"
  }
}

This configuration creates a high CPU usage alert with specific conditions and notifications. The condition parameter is crucial as it defines the alert triggering logic.

5. Provide the API Token

Set the signoz_api_token as an environment variable:

export TF_VAR_signoz_api_token="YOUR_SIGNOZ_API_TOKEN"

This ensures that your API token is securely used by Terraform without hardcoding it in your configuration files.

6. Initialize Terraform

Run:

terraform init

This command initializes your Terraform working directory, downloading necessary plugins, and preparing the environment.

7. Review the Execution Plan

Generate the execution plan:

terraform plan

This step previews the changes Terraform will make, allowing you to verify the configuration before applying it.

8. Apply the Configuration

Apply the changes:

terraform apply

Type yes when prompted. This command applies the configuration, creating or updating resources as specified.

9. Verify the Alert in SigNoz Cloud

To do this, follow these steps:

• Log in to your SigNoz Cloud dashboard.

• Navigate to Alerts.

• Confirm that the "High CPU Usage Alert" is listed.

• Click on the alert to view its details and ensure it matches your configuration.

10. Modify the Alert (Optional)

To change the CPU usage threshold to 75%, follow these steps:

• Update the target in alerts.tf:

    target = 75
  

• Apply the changes:

    terraform apply
  

11. Destroy the Resources (Optional)

To remove the alert and notification channel:

terraform destroy

Type yes to confirm. This command will delete the resources created by Terraform.

Best Practices and Security Considerations

In modern infrastructure automation, robust best practices and security measures are paramount.

Use version pinning

To ensure your alert provisioning remains reliable and maintainable, start with strict version control. Avoid using the latest tag and instead specify an exact version number. This ensures your infrastructure configuration remains consistent and predictable.

By pinning your provider version (for example, use version = "0.1.3" instead of version = ">= 0.1.3".), you eliminate unexpected behavior that can arise from upstream changes. This practice is critical for long-term stability, especially when your infrastructure scales across multiple environments.

Externalize Credentials

Security is non-negotiable. Instead of embedding sensitive details like API tokens in your codebase, leverage environment variables or dedicated secret management tools such as HashiCorp Vault or AWS Secrets Manager.

For instance, storing your SigNoz API token as an environment variable (TF_VAR_signoz_api_token) not only mitigates the risk of credential exposure but also simplifies the process of credential rotation. Also, enforce access control policies around your configuration repositories and CI/CD pipelines to further secure these secrets.

Use Version Control

A mature setup also demands rigorous infrastructure version control. Hosting your Terraform configuration in a Git repository with branch protection and code review policies allows you to track changes meticulously, roll back problematic updates, and maintain an audit trail. This traceability is essential when troubleshooting issues or validating compliance during audits.

You should also document your configuration decisions extensively—explain why a particular CPU threshold was chosen or why specific labels (like severity and team) are used. Such documentation becomes invaluable for onboarding new team members or when revisiting configurations months later.

Integrating with CI/CD Pipelines

Integrating Terraform with your CI/CD pipeline is a cornerstone of a modern, automated deployment strategy. A well-architected pipeline not only validates your infrastructure changes but also ensures that your alerting rules remain in sync with your evolving application environment.

Continuous Integration (CI) involves automatically merging code changes into a shared repository and running automated tests on each commit. In practice, embedding Terraform plan into your pull request workflow provides early feedback, catching misconfigurations before they reach production. For instance, a GitHub Actions workflow can automatically check your changes:

name: Terraform CI/CD

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  terraform:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v3
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
      - name: Terraform Init
        run: terraform init
      - name: Terraform Plan
        run: terraform plan -no-color
        env:
          TF_VAR_signoz_api_token: ${{ secrets.SIGNOZ_API_TOKEN }}

This workflow uses GitHub secrets to securely manage your API tokens while validating the configuration changes. Continuous Delivery (CD) takes this further by automating deployments. Once your plan is approved, an automated Terraform apply step (often scheduled during off-peak hours or coordinated with application deployments) ensures smooth, coordinated rollouts.

Advanced pipelines can also include automated rollback mechanisms. For example, if a deployment triggers an anomaly, scripts can automatically revert to a previous version using your version control history—minimizing downtime and reinforcing the feedback loop between application performance and infrastructure configuration.

Advanced Customizations and Troubleshooting

As your observability requirements evolve, you may need to implement advanced customizations. One powerful approach is using multi-metric composite alerts. Instead of triggering an alert on a single threshold, you can design rules that combine multiple conditions—for example, firing only when both CPU usage and memory consumption exceed critical levels. This nuanced alerting minimizes false positives and ensures alerts are issued only during genuine performance issues.

Terraform’s modular design is especially useful here. By creating reusable modules that encapsulate your alert configurations, you can parameterize key variables—such as thresholds, evaluation windows, and notification channels—across a microservices architecture. This modularity enforces consistency while simplifying management and scaling.

Troubleshooting advanced configurations starts with reviewing your terraform plan output to ensure every change aligns with expectations. If an alert isn’t triggering as expected, inspect the JSON structure generated by the jsonencode function. Even minor syntax errors can cause significant issues.

When integrating with incident management tools like PagerDuty or Opsgenie, run comprehensive end-to-end tests in a staging environment. For example, deploy a test alert to a dedicated channel to verify that the complete alerting pipeline—from condition detection to incident escalation—is functioning correctly.

In one real-world scenario, a misconfigured composite query in an alert’s JSON payload led to intermittent failures. By enabling detailed provider logs and iteratively validating the JSON output, the issue was rapidly isolated and resolved. Such experiences underscore the importance of rigorous logging, validation, and testing in production-grade setups.

Conclusion

Automating alert provisioning is a transformative approach to managing observability in modern infrastructures.

By defining alerts and notification channels as code, you make your systems more consistent, scalable, secure, and easily integratabtle with CI/CD. You can set up uniform alert rules across all environments, quickly update and deploy monitoring configs, easily handle secure credentials, and automate CI/CD workflows that stay in sync with application changes. They also become easier to integrate with CI/CD workflows.

I hope you’ve enjoyed this tutorial and have learned something new. I’m always open to suggestions and discussions on LinkedIn. Hit me up with direct messages.

If you’ve enjoyed my writing and want to keep me motivated, consider leaving starts on GitHub and endorsing me for relevant skills on LinkedIn.

Till the next one, happy coding!

This article compares three leading CDEs: Harness CDE, Gitpod, and Coder. My goal here is to offer an objective analysis to help you make informed decisions based on your specific needs.

What is a Cloud Development Environment (CDE)?

A Cloud Development Environment (CDE) is a cloud-hosted workspace where developers can write, test, and deploy code without relying on local machines. Unlike traditional setups, CDEs provide pre-configured environments accessible via a browser or IDE, eliminating the "it works on my machine" problem.

How CDEs differ from traditional development environments

• CDEs are more consistent and help standardize tools, dependencies, and configurations across teams.

• They’re also accessible from anywhere, enabling remote collaboration.

• They’re more scalable, as resources (CPU, memory) scale dynamically based on workload.

• And they’re secure, with centralized security controls and compliance adherence (for example, SOC 2, GDPR).

Common CDE Features

Most CDEs come with a variety of helpful features. They typically have pre-built environment templates (for example, Python, Node.js) and integrate easily with Git repositories and CI/CD pipelines. They also have various real-time collaboration tools, as well as the ability to automate backups and recovery.

You’ll learn more about specific features when we discuss each of our CDE options below.

The Case for Cloud-Based Development

CDEs can help you and your team solve some critical pain points:

CDEs make setup easier

When using a CDE, you don’t have the hassle of setting up local machines. Instead, you have a pre-configured development environment that’s ready to go in minutes. With a traditional setup, you have to install dependencies, configure environments, and resolve compatibility issues – and this can take hours or even days. CDEs make this process much easier.

Let’s say a new developer joins your project that requires a complex stack – it needs a specific Python version, multiple frameworks, and environment variables. Instead of spending hours configuring their local machine, they can just launch a cloud-based workspace (like one of the tools we’ll discuss here), which comes pre-loaded with everything they need.

CDEs help reduce costs

CDEs can reduce your costs by making sure that development resources are allocated only when they’re needed. Unlike local machines, which require upfront investment in high-performance hardware, cloud environments help you scale resources dynamically and pay only for the compute power you and your team use.

Perhaps your team is developing a resource-intensive AI application. If you’re using a CDE, you’ll no longer need to provide every developer with an expensive workstation. Instead, you can just provision high-performance cloud instances when needed and shut them down when idle. This cuts down on unnecessary spending.

CDEs enhance security

With cloud-based environments, code and sensitive data remain on secure, centralized servers rather than being stored on individual developer machines. This helps reduce the risk of data loss or theft. CDEs also provide audit logs, identity management, and automated backups, all of which help make things more security.

Let’s say a financial services company requires strict security controls over customer data. By using a CDE, the developers on the team can access code via secure connections without storing sensitive files locally. This helps ensure compliance with industry regulations like SOC 2 or GDPR.

CDEs enable global collaboration

CDEs make collaboration among distributed teams much easier by allowing multiple developers to work in the same environment with shared configurations. Remote developers can contribute from anywhere without worrying about compatibility issues or inconsistent setups.

For example, perhaps your global development team is working on a SaaS product. They can use a CDE to collaborate in real time. A member of your dev team in India can start debugging an issue, and then a teammate in the US can pick up where they left off hours later without needing to set up the same environment locally.

Methodology

This analysis is based on official documentation, user reviews, and independent testing. All information is current as of the last update date. The article is focused on key aspects such as features, deployment options, security, pricing, and use cases.

Overview of Each Tool

Harness CDE

Harness CDE is part of the broader Harness platform, designed to streamline software delivery with integrated CI/CD pipelines, feature flags, and cloud cost management. It provides enterprise-grade security, a user-friendly interface, and robust integration capabilities, making it ideal for large-scale applications.

With its comprehensive suite of tools and advanced cost management, Harness CDE helps enterprises efficiently manage their entire development lifecycle. Harness CDE's intuitive interface and detailed documentation further enhance its suitability for large-scale applications.

Drawbacks

Despite its many strengths, Harness CDE is relatively new to the market, meaning its features and capabilities are still evolving. Deep integration with the Harness platform could make switching challenging.

Gitpod

Gitpod is a SaaS solution that provides automated, ready-to-code development environments. It integrates seamlessly with popular version control systems like GitHub, GitLab, and Bitbucket, offering fast and consistent setups.

Gitpod is known for its user-friendly web interface and quick onboarding process, which significantly reduces setup times and lets devs focus on coding rather than environment configuration. This makes it ideal for agile development teams and startups.

Drawbacks

Gitpod's SaaS model offers limited control over infrastructure, which can be a disadvantage for teams needing more customization and control. Also, dedicated instances can be more costly, potentially offsetting some of the benefits of its free plan.

Coder

Coder is an open-source platform offering both free and self-managed (paid) options. It provides highly customizable, secure, and scalable development environments that you can host on your infrastructure. This makes it suitable for organizations needing tailored solutions.

Coder excels in environments where stringent security and compliance requirements are paramount, offering extensive control and customization.

Drawbacks

Coder requires more setup time and maintenance compared to Gitpod and Harness CDE. Its reliance on self-managed infrastructure can also increase complexity and cost, particularly for smaller teams or startups without dedicated DevOps resources.

Detailed Feature Comparison

Now let’s compare some basic features to see how these three tools stack up against each other.

Deployment and Scalability

• Harness CDE: Integrated with the Harness & Gitness platforms, offering high scalability within the Harness ecosystem.

• Gitpod: SaaS model with easy scalability and options for managed dedicated instances.

• Coder: Self-managed deployment with full control over infrastructure, providing high scalability for tailored environments.

Integration and User Experience

• Harness CDE: Comprehensive suite of development tools, intuitive interface, and detailed documentation.

• Gitpod: Seamless integration with GitHub, GitLab, and Bitbucket, featuring automated setups and excellent documentation.

• Coder: Integrates with existing infrastructure and various tech stacks, providing detailed documentation and customizable configurations.

Security and Compliance

• Harness CDE: Enterprise-grade security features, including SOC 2 compliance, role-based access control, and advanced secrets management. Offers comprehensive audit logging and governance with policy-as-code support.

• Gitpod: Secure environments with data encryption, SOC 2 compliant.

• Coder: Focuses on security and compliance, supporting various standards like HIPAA and GDPR.

Cost/Pricing

• Harness CDE: Competitive pricing with integrated platform features. A lot of features are free to use. Pricing varies based on scale and needs – contact Harness for details.

• Gitpod: Varies with free and paid plans, limited customization based on SaaS offerings. The free plan includes 50 hours/month, while paid plans offer unlimited hours.

• Coder: Costs depend on self-managed infrastructure, offering high customization and control over environment setup. The tool is free for open-source use, and paid for self-managed deployments.

Setup Time and User Interface

• Harness CDE: Fast setup with integrated CI/CD pipeline and modern, intuitive interface.

• Gitpod: Quick setup in minutes with a user-friendly, web-based interface.

• Coder: Setup time varies based on custom configurations, offering a flexible and customizable interface.

Availability

• Harness CDE: Part of the Harness platform, typically targeted at enterprise users.

• Gitpod: SaaS model with both free and paid plans.

• Coder: Open-source with both free and self-managed (paid) options.

Specs

• Harness CDE: Integrated CI/CD, feature flags, cloud cost management, enterprise-grade security, and so on.

• Gitpod: Automated setups, seamless integration with VCS, user-friendly interface.

• Coder: Highly customizable, secure, scalable, extensive control.

Additional Features

Here’s a detailed table that includes info on a bunch of other features that might help you make your decision as to which tool is best for you.

How to Choose the Right Tool

As you can see, each of these cloud development environments has its strengths. It’s up to you to analyze them and decide which tool is right for you. Here’s a quick summary:

• Harness CDE offers the fastest startup times and a straightforward, performance-focused approach.

• Gitpod provides the widest language support and a large community, with competitive pricing.

• Coder excels in security, compliance, and customization.

When choosing a CDE, consider the following factors:

• Team size and structure

• Existing technology stack

• Security and compliance requirements

• Budget constraints

• Customization needs

• Scalability requirements

• Integration with existing tools and workflows

Conclusion

In this guide, you learned about three CDE tools and their main features. Which of these tools you choose will largely depend on your specific needs.

Ultimately, I recommend that you take advantage of any free trials or demos offered by these platforms to get hands-on experience before making a decision. Consider your team's specific workflows, the technologies you use, and your scalability needs when choosing a cloud development environment.

References

• Harness Docs

• Gitpod Docs

• Coder Docs

Note: This article is for informational purposes only. Everyone should conduct their own thorough evaluation based on their specific requirements before making a decision.

I hope you’ve enjoyed it and have learned something new. I’m always open to suggestions and discussions on LinkedIn. Hit me up with direct messages.

If you’ve enjoyed my writing and want to keep me motivated, consider leaving starts on GitHub and endorsing me for relevant skills on LinkedIn.

Till the next one, happy coding!

They are designed to provide the same high availability and reliability as an AWS Region, but with the added benefit of low-latency connections for applications that require it.

Using Local Zones can be particularly helpful if you have end users located in specific geographic areas and want to provide them with low-latency access to their applications. This can be especially important for applications that require real-time data processing or have strict performance requirements.

Local Zones can also minimize network expenses, which is an additional advantage. By executing applications in a Local Zone closer to end users, you can limit the quantity of data that must be transferred over long distances, which reduces network costs.

In this tutorial, we will see how to build a hybrid edge EKS cluster that spans across the AWS Regions and AWS Local Zones using both AWS Management Console and AWS Cloud Development Kit (CDK).

Before getting started, it's important to note that AWS Local Zones are physically separate locations that are connected to the main AWS region through high-speed links. They allow you to run certain services closer to your customers and reduce latency.

To build your EKS cluster across Local Zones, you will need to:

1. Create an EKS cluster in the main region
2. Create a VPC and associated resources in the Local Zones
3. Connect the Local Zone VPCs to the main region VPC
4. Launch worker nodes in the Local Zone VPCs

We'll walk through these steps in this guide.

Architecture Diagram

Prerequisites

Before we begin, you need to have the following:

1. An AWS account with permissions to create resources in AWS Wavelength and AWS Local Zones.
2. AWS CDK installed on your local machine. If you don't have it installed, you can follow the instructions in the AWS CDK documentation to install it.
3. AWS CLI installed on your local machine. If you don't have it installed, you can follow the instructions in the AWS CLI documentation to install it.

Finally, let’s get started.

Step 1: Create an AWS Local Zone

The first step is to opt-in to AWS Local Zones in the region of your choice. You can follow the instructions in the AWS Local Zones documentation to opt-in to these zones.

Step 2: Create a CDK Project

To start, we need to create a new CDK project using the following command:

cdk init --language=javascript

This will create a new directory with the required files and directories for a CDK project.

Next, install the required dependencies using the following command:

npm install

Now, let's create a new file called local-zone-eks.js in the lib directory and add the following code:

const cdk = require('aws-cdk-lib');
const ec2 = require('aws-cdk-lib/aws-ec2');
import * as autoscaling from '@aws-cdk/aws-autoscaling';
import * as ecs from '@aws-cdk/aws-eks';

require('dotenv').config();
const config = {
  env: {
    account: process.env.AWS_ACCOUNT_NUMBER,
    region: process.env.AWS_REGION,
  },
};


class VPCStack extends cdk.Stack {
  constructor(scope, id, props) {
    super(scope, id, props);

    // Create a VPC 
    const vpc = new ec2.Vpc(this, 'VPC', {
      cidr: '10.0.0.0/16',
      maxAzs: 2,
      subnetConfiguration: [
        {
          cidrMask: 24,
          name: 'Public',
          subnetType: ec2.SubnetType.PRIVATE,
        },
        {
          cidrMask: 24,
          name: 'Private',
          subnetType: ec2.SubnetType.PRIVATE,
        },
      ],
    });
  }
}

module.exports = { VPCStack };

This code creates a new VPC with a CIDR block of 10.0.0.0/16, which spans across two private subnets.

Next, let’s export the environment variable for our AWS Local Zone, which we will use to create the only public subnet in our VPC. In this example, we have selected the Las Vegas Local Zone and have configured the subnet accordingly.

const localZone: string = 'us-west-2-las-1a’

 // Create Local Zone Public Subnet
    const LocalZoneSubnet = new ec2.PublicSubnet(this, 'localzone-public-subnet', {
      availabilityZone: localZone,
      cidrBlock: '10.0.3.0/26',
      vpcId: vpc.vpcId,
      mapPublicIpOnLaunch: true,
    });

    // Add Local Zone Subnet to VPC
    vpc.publicSubnets.push(LocalZoneSubnet);

Step 3: Create an Amazon EKS Cluster

Now that we have a VPC, we can create an Amazon Elastic Container Service for Kubernetes (Amazon EKS) cluster.

Add the following code to the local-zone-eks.js file:

const eks = require('aws-cdk-lib/aws-eks');

class EKSStack extends cdk.Stack {
  constructor(scope, id, props) {
    super(scope, id, props);
    // Create the EKS cluster
    const cluster = new eks.Cluster(this, 'EKSCluster', {
      vpc: vpc,
      defaultCapacity: 0,
      version: '1.21',
      clusterName: 'local-zone-eks-demo-cluster',
    });
  }
}
module.exports = { EKSStack };

This code creates a new EKS cluster using the VPC we created earlier. It also specifies the Kubernetes version to use and the name of the cluster.

Step 4: Create Worker Nodes

Next, we need to create worker nodes to run our applications on the EKS cluster.

Add the following code to the local-zone-eks.js file:

// Define EKS-optimized image for Launch Template
    const image = new ecs.EksOptimizedAmi();

    // Create Launch Template for Auto Scaling group to reference
    const lzLaunchTemplate = new ec2.CfnLaunchTemplate(
      this,
      'eks-launch-template',
      {
        launchTemplateName: 'lz-launch-template',
        launchTemplateData: {
          networkInterfaces: [
            {
              deviceIndex: 0,
              associatePublicIpAddress: true,
              deleteOnTermination: true,
              subnetId: LocalZoneSubnet.subnetId!,
            },
          ],
          imageId: image.getImage(this).imageId,
          instanceType: 't3.medium',
          userData: cdk.Fn.base64(
            `#!/bin/bash -xe
          set -o xtrace
          /etc/eks/bootstrap.sh ‘local-zone-eks-demo-cluster’}
            )
        },
      }
    );

 // Create Auto Scaling Group
    const lz_asg = new autoscaling.AutoScalingGroup(this, 'LocalZoneWorkerNodes', {
      instanceType: ec2.InstanceType.of(ec2.InstanceClass.T3, ec2.InstanceSize.MEDIUM),
      machineImage: new ecs.EksOptimizedAmi(),
      updateType: autoscaling.UpdateType.REPLACING_UPDATE,
      desiredCapacity: 1,
      vpc: vpc,
    launchTemplate: lzLaunchTemplate
    });

This code creates a new auto-scaling group to manage the worker nodes, and adds the worker nodes to the EKS cluster using the instance bootstrap script.

Step 5: Deploy the CDK App

Now that we have all the required code, we can deploy it using the following command:

cdk deploy

With your EKS cluster up and running, you can start deploying your applications. You can use Kubernetes manifests or Helm charts to deploy your applications to the cluster.

Conclusion

AWS Local Zones provide a robust mechanism for providing high-performance applications to end users, independent of their location. They also give end users a better experience and deliver great performance.

I’m always open to suggestions and discussions on LinkedIn. Hit me up with direct messages.

If you’ve enjoyed my writing and want to keep me motivated, consider leaving stars on GitHub and endorse me for relevant skills on LinkedIn.

Till the next one, stay safe and keep learning.

Random Access Memory (RAM) is a crucial component of any computer system, and it is responsible for temporarily storing data that is required by the system to carry out its functions. But the contents of RAM can be quite volatile, and they are usually lost when the system is shut down.

One way to preserve the contents of RAM is by performing a RAM dump, which is a process of copying the contents of RAM onto a storage device, such as a hard drive. You can analyze the RAM dump, and the data contained within it can provide valuable insights into the system's state at the time the dump was taken.

In this article, I will walk you through the process of reading the contents of RAM, as well as explain what a RAM dump is and how it can be useful in analyzing a computer system. I will also provide you with step-by-step instructions on how to perform a RAM dump and how to analyze the resulting data.

Why Read RAM Data?

Reading data from the disc is something you might be familiar with doing. But can we also read data straight from the RAM, where the most essential information is stored?

As developers, we can investigate the space complexity and delve further into the RAM to understand what is going on.

Accessing and reading the contents of RAM can be useful in a variety of scenarios. One common use case is for troubleshooting and diagnosing issues with a computer system. By examining the contents of RAM, you can gain insights into the state of the system at a particular point in time.

For example, if your computer suddenly crashes, examining the contents of RAM can help you identify the cause of the crash.

Another use case is for forensic analysis. When investigating a computer system for evidence of wrongdoing, examining the contents of RAM can provide valuable insights into the activities that were being carried out on the system.

For example, a security professional may use RAM analysis to determine whether a particular process was running on the system or to identify files that were recently accessed.

In addition, accessing and reading the contents of RAM can also be useful for software developers and researchers. By analyzing the data stored in RAM, developers can gain insights into the performance of their software and identify potential issues or bottlenecks. Researchers can also use RAM analysis to study the behavior of malware or to develop new security tools and techniques.

Overall, accessing and reading the contents of RAM can be useful for troubleshooting, forensic analysis, software development, and research. It provides a valuable way to gain insights into the inner workings of a computer system and can help identify issues and potential security threats.

Before we move further into the specifics, let's take a short look at the nomenclature. This may be common knowledge, but you'll need to understand the terminology as you go through this guide, so it's worth a review.

What is RAM?

There's a physical hardware device called RAM (which stands for Random Access Memory): the physical memory, a CPU, a hard disk, and other physical components.

On top of this, we have the operating system. The operating system is always in conversation with the piece of hardware known as the "kernel" – and this is one of the most critical aspects of this software.

If we consider things from the user's point of view, we initially log in to the operating system so that we can carry out our tasks, the majority of which include executing applications.

What is the Kernel?

A kernel is a fundamental portion of an operating system that accepts the instructions from the user with the aid of the program we run behind the scenes. This may happen regardless of the program that we run behind the scenes.

Everything that has to be computed is handled by the central processing unit (CPU), but whatever we do and however we manage the services provided by the CPU, the data, instructions, code, and program must all go via the random access memory (RAM).

This implies that the results of anything we do with the data at any given moment in time will be available on top of the memory. If you are a programmer and you're constructing some kind of data structure, this indicates that all of the data will live on top of the RAM.

We can directly go to the RAM and see how the data structures have been created and how they align, and we can see how the space complexity works there.

For instance, if we are discussing any web browser, such as Chrome, there may have been security flaws generated in an application. So the most effective method is to navigate to the RAM and investigate how the data has been operating.

Let's say you open any secure website in Chrome (or any browser), like gmail.com. Any information you enter into Gmail, including your username and password, is sent over the internet to a server run by Google. That data has been fully encrypted, and it will be challenging, if not impossible, to crack the password.

But in order to enter your password, you likely use a computer with a keyboard attached to it. After that, some programs will encrypt the data and send it to the internet. This means that initially, your data was present in the RAM.

First, the password is standard data, and the password goes and lands on the top of the RAM. Then, some programs will encrypt the data and send it to the internet. If you can access the RAM, you can examine it to determine whether or not your password has been encrypted. And this process is pretty straightforward.

Let's look at an example to see how this works. We'll pretend that "a" is a variable and "9" is the data. When we execute the program, this data loads onto the RAM. When the program ends, the data from the RAM will be gone, although nobody has proven it yet. But this is the case, and we can prove it with a little investigation.

So how can we check whether or not this data is still present in the RAM after the application has finished running if it was loaded at the very top of the memory?

Well, the RAM will be cleared of these contents shortly. A "RAM dump" is the process of capturing all of the RAM. This demonstrates the actual form in which the entire data set is made accessible and loaded onto the RAM for the first time. And you'll learn how to capture or extract the data from memory next.

In order to accomplish this, you'll need some form of software. As a result, the goal of this program is to travel further into the memory. It will go into memory and retrieve all of the stored data from memory before continuing.

What is LiME?

The Linux memory extractor, sometimes referred to as LiME, is a powerful piece of software. It's what you use to extract the memory from Linux. This piece of software is also known as the driver, also referred to as the module. This is because RAM is a device, which complicates things further.

We need some form of driver to access the device so we can try to read the contents of it. LiME is an example of a driver, and if you're familiar with Linux you may know that in order to make any driver function, you need to load that driver with the assistance of the kernel.

Within the context of Linux, a driver is also referred to as a module. So LiME is a Linux kernel module. We have access to what is known as a kernel loadable module, which allows us to install the module on the operating system.

We have covered enough background. Now, let's look at how we can extract the contents of the RAM. We'll go through the process step-by-step in a hands-on way.

Setup and Installations

So, the only thing we need is the LiME driver. Here's the link to download this particular module: https://github.com/gursimarsm/LiME.

Now, boot up your Linux system (I use RedHat Enterprise Linux). You can use the free -h command to check the amount of RAM memory that's being used, that's available, and other details. Mine looks like this:

Output from running the free -m command

To access RAM, we need some software where the kernel can load some extra modules. In our case, the module name is LiME. So, the software we install are called “kernel-devel”, and “kernel-headers”. These two pieces of software are what we need to install in order to perform our subsequent actions, that is to use the LiME module.

If you want to install the software, you should have "yum" configured. For context, yum is a command you can use to install the software in the RedHat Linux operating system. I'll demo how to configure it in the appendix for reference.

Installation

So, now that you've installed that software, you need one more piece of software because you have to download some drivers from GitHub. So, now you need to install Git if you don't already have it. You can do this using the command yum install git. You also need to configure your account so that you can work with it.

# git config --global user.name "Your Name" 

# git config --global user.email "you@example.com"

I shared the repository above. It's a loadable kernel module (LKM). It lets you acquire the entire memory from your Linux operating system or any kind of language operating system (including Android devices because Android is based on Linux as well).

You can download the module using the **# g**it clone**** [**https://github.com/gursimar**sm**/LiME**](https://github.com/gursimarsm/LiME) command.

After downloading that, you need to move into the directory of the software. You'll find multiple folders there. But, to run the main code, you need to move to the "src" directory.

In this directory, you'll find multiple programs based on the C language. So, in order to make use of the files, you'll need to compile them. To do that, you can use the make command. Install that like this:

# yum install make 

# yum install gcc

In the directory /LiME/src/, run the make command to compile the entire code.

If you encounter an error, it might be because we are using the latest version of LiME, and it comes with a new feature called orc metadata generate. To implement this feature, you have to install one more thing that's part of LiME called elfutils-libelf-devel. You can do that using yum like you can see below:

How to install elfutils-libelf-devel

After that's done, if we now run the make command it will ask the GCC compiler to compile the entire code. After the compilation, it will create one final object file called the kernel object file, and that is the final module in LiME.

Output

You can find this file in the same directory by using the ls command.

How to Use the Module

With this module, the kernel now has the capability to capture or read the entire RAM. By default, we can't read the entire RAM in one go, but now because of the LiME module, we can.

To learn more about the LiME module, you can use the modinfo command. Type modinfo along with lime. This will show you some more details like where the file is available, and it also displays all the modules or drivers that come with some kind of extra parameters. Every parameter has some benefits.

modinfo

Here we are going to use two parameters which are very important: path and format.

path means when we read the entire RAM, we have to store the data of the RAM in some file. So, to specify the destination file we would like to create, we have to give that particular information over here.

The next parameter, format, specifies the format in which we want to read the RAM data. So, in this case, we want to read the format of the RAM as it is. The data stored in the RAM is mostly in binary, and we want to read the entire data in that binary format only and capture it in its raw form.

So, the format will be raw and stored in the file wherever we give the path.

Finally, it's time to read the data from the RAM. So, let's come to the main command that will help us start reading the entire RAM.

Demonstration

Type in your password for your Gmail account in Chrome for this demonstration. This will help you learn how to capture the data from the RAM and also if your password is encrypted.

To verify these two things, move to the command prompt and check if the data is still on the RAM. You'll have to load a particular module using the command insmod. This will help you insert the module.

Copy the complete path of the module and paste it along with the insmod command.

insmod command

This module will get loaded with the help of the kernel. The module will start capturing the entire data from the RAM and it'll store it in a file, for example, myram.data

It will also load the entire memory dump or RAM dump into this file and which format we want to capture. So, the format will be the raw format.

We'll use these two parameters (don't worry about this for now). We need only two parameters to perform, and now as soon as we hit this command, whatever data we have will be captured and stored in this particular file. This command typically takes a few seconds, depending on the CPU speed and the amount of data we have in the RAM.

Command

How to read the data

Now, we have this file myram.data and the entire data of the RAM is stored in this file. Because we captured this data in the raw format, the data is going to be in binary. If we try to read this data from this file directly, as human beings, we can't read it even if we try it with the initial lines using the head command to read some of the top 10 lines.

Output

So, we can use the “cat” command, which will read the entire data. But, again, the same thing is going to happen – it will read the entire data, but the data will be displayed in the binary format. Then we need to use the pipe function with this command and combine it with another new command called strings:

Command

String is a command that will convert the data into regular text in a human-readable format.

Output

The list will go on and on. You can interrupt it using Ctl+C.

Right now, it won't mean much seeing and reading the entire data. We know some data that are there on the RAM is the password called mywebpasswordgmail. So, just to confirm that this data is available on the RAM we can use one more pipe along with the grep command. The grep command helps us sort the data.

cat /myram.data | strings | grep mywebpasswordgmail

Now, we are searching for this string in the entire data. It will convert the data into regular text, and wherever that particular string shows up, grep will grab that line and let us start searching, then show us this data.

So, as you can see from this simple example, whatever you type on your keyboard can also go into the RAM – even if it's your password or any kind of secure site you are surfing, your data is there on the RAM. It doesn't matter what program you run. If you type using the keyboard everything will load on the RAM and can be extracted. This is called the memory dump.

LiME provides us with many other powerful capabilities. Right now, we are capturing the data directly from the system where we perform the actions. But we can also run LiME on the system and it can capture the data in real-time and send the data over the network to another computer.

What does this mean? Think of it this way: for example, somebody opens a website and they're typing something in real-time. This entire message is being transmitted in real-time to another computer.

We're not talking about key loggers, we are just talking about the RAM. Whatever is happening when any program is running, the database is storing some data. Programs are reading data from other parts of the hard disk. And whatever is happening on the RAM can be captured in real-time by the system and sent over the network to other computers.

Conclusion

We’ve finally come to the end of this article. I hope you've enjoyed it and have learned something new.

I’m always open to suggestions and discussions on LinkedIn. Hit me up with direct messages.

If you’ve enjoyed my writing and want to keep me motivated, consider leaving starts on GitHub and endorse me for relevant skills on LinkedIn.

Till the next one, stay safe and keep learning.

By the end, you'll be able to create your own custom container. We'll also examine why Kubernetes deprecated Docker and embraced CRI-O, as well as how to configure a multi-node Kubernetes cluster using CRI-O.

In the end, we will look at the list of available container runtimes.

Table of Contents

1. What are containers?
2. Why do we need containers?
3. What's the difference between containers and virtualization?
4. Is there a standard format for containers?
5. Types of container platforms
6. How to launch a container
7. Why did Kubernetes deprecate Docker?
8. Challenges of using Docker
9. What is a CRI (Container Runtime Interface)?
10. How to build a multi-node cluster using CRI-O

Alright, let's dive in.

What Are Containers?

Containers allow you to reliably move software from one computing environment to another.

The technology behind containers is nearly as old as that behind virtual machines. But the information technology industry didn't begin using containers until around 2013–14, when Docker, Kubernetes, and other innovations that disrupted the sector became popular.

Containers have emerged as a critical tool in the process of developing software. They can serve either as a replacement for Virtual Machines or as a supplement to them.

Containerisation helps developers construct apps more rapidly and securely while also deploying them more easily.

Why Do We Need Containers?

As we learned above, containers provide a solution to the problem of transporting software from one computer environment to another in a reliable manner. This can be just from a developer's workstation to a test environment, from a staging environment to production, or even from a real system in a data center to a private or public cloud virtual machine.

Containerization makes possible all of these transformations. And these are just some of the viewpoint alterations that can happen.

This quote gives a little perspective on why containers are helpful:

"You’re going to test using Python 2.7, and then it’s going to run on Python 3 in production and something weird will happen.

Or you’ll rely on the behavior of a certain version of an SSL library and another one will be installed.

You’ll run your tests on Debian and production is on Red Hat and all sorts of weird things happen.

The network topology might be different, or the security policies and storage might be different but the software has to run on it." – Solomon Hykes

How Do Containers Solve This Issue?

A more straightforward interpretation is that a container is an all-encompassing runtime environment.

This means that a piece of software, together with all of its dependencies, libraries, other binaries, and configuration files, is packed and distributed to customers as a single package.

The application platform and its dependencies can be protected from the consequences of changes in operating system distribution and underlying infrastructure if they are bundled within containers.

What’s the Difference Between Containers and Virtualization?

A virtual machine is a package that may be shared when employing virtualization technology. This package includes both the program and the operating system being used.

On top of a physical server running three virtual machines, you'd have an installation of a hypervisor, as well as three distinct operating systems.

On the other hand, a Docker server that hosts three containerized programs only needs to run a single operating system because all of the containers share the same kernel. The standard components of the operating system can only be read, but each container has its own mount or access mechanism, which allows it to store and retrieve data.

This hints that containers are far lighter than virtual machines and make considerably less use of their resources.

Is There a Standard Format for Containers?

When CoreOS published its own App Container Image (ACI) specification in 2015, some people worried that the rapidly growing container movement might splinter into different Linux container formats. This was because ACI stood for "App Container Image."

In contrast, the Open Container Project, which would subsequently become the Open Container Initiative (OCI), was not made public until the latter half of the same year.

The Open Container Initiative, which the Linux Foundation leads, aims to establish an industry standard for container formats as well as container runtime software that is compatible with all operating systems (OCI).

Docker technology was used to develop the Open Container Project (OCP) guidelines, and Docker gave around 5 percent of their software to help get the effort off the ground.

What is Open Container Initiative?

The Open Container Initiative (OCI) is an organization whose mission is to ensure that the fundamental aspects of container technology, such as the container's format, are standardized so that anyone can use them.

As a result, companies can concentrate their efforts on developing the supplementary software they need in order to use standardized containers in an enterprise or cloud environment (instead of developing competing technologies for containers).

Software components called container orchestration and management solutions, as well as container security systems, are essential components.

Types of Container Platforms

As a result of the development and expansion of container technology, a number of different choices are currently available.

Docker is, without a doubt, the most common and widely used container platform available.

On the other hand, the landscape of container technologies includes other technologies, each of which has its own use cases and benefits.

We will look at the two most famous ones, i.e., Docker and Podman.

Docker

Docker is the container platform that's currently the most popular and is used the most widely. It allows you to develop and use Linux containers.

Docker is a piece of software that, by using containers, simplifies the processes of creating, deploying, and operating software applications. It does this by minimising the number of steps in each process.

Docker has gained support not just from the most major Linux distributions, such as Red Hat and Canonical, but also from large organisations, such as Microsoft, Amazon, and Oracle.

Virtually all businesses concerned with information technology and cloud computing use Docker.

Docker Architecture and Components

Docker is built on a client-server architecture. The Docker daemon enables the creation, operation, and distribution of Docker containers.

The Docker client and Docker daemon can interact through a REST application programming interface (API), UNIX sockets, or network interface.

Source: docs.docker.com

Docker's architecture is comprised of the following five primary components:

1. Docker Daemon manages Docker objects like images, containers, networks, and volumes. It also responds to Docker API inquiries.
2. Docker Clients enables users to interact with Docker by allowing the user to connect with Docker. Docker client supplies a CLI interface that allows users to send application commands to a Docker daemon and start and halt such operations.
3. Docker Host provides a complete software program execution and operation environment. This system comprises the Docker daemon, Images, Containers, Networks, and Storage components.
4. Docker Registry maintains Docker Images. Docker Hub is a public registry, and by default, Docker is configured to use images saved on Docker Hub. You can use it to administer your own register.
5. Docker Images are templates that can only be read and produced by following a Dockerfile's set of instructions. Images specify the appearance we want for our packaged program, its dependencies, and the processes that should run when the application is launched.

Resource Isolation components in Docker

Namespaces:

• PID namespace for process isolation.
• NET namespace for managing network interfaces.
• IPC namespace for managing access to IPC resources.
• MNT namespace for managing filesystem mount points.
• UTS namespace for isolating kernel and version identifiers.

Control groups

• Memory cgroup that oversees accounting as well as restrictions and alerts
• HugeTBL is a cgroup that keeps track of each process group's use of huge pages.
• CPU group is responsible for regulating the time users and the system spend using CPU.
• CPUSet cgroup lets you associate a group with a certain CPU. Utilized for real-time workloads and NUMA systems with localised memory for each CPU.
• BlkIO cgroup for measuring and limiting the amount of blkIO each group produces.
• the net cls and net prio cgroups are utilised for traffic control tagging.
• Devices cgroup for accessing devices that can both read and write data.
• Cgroup for the freezing of a group referred to as the freezer. It is useful for scheduling cluster batches, relocating processes, and troubleshooting.

Union Filesystem

• Docker images are composed of layered filesystems, allowing them to be extremely lightweight and speedy. Union file systems are layering-based file systems.
• Docker Engine has the ability to use several different versions of UnionFS, such as AUFS, btrfs, vfs, and devicemapper.
• For executing five 250MB image containers, we would require 1.25GB of disc space if we didn't have UnionFS.

The Docker interface may appear to be a mysterious black box holding a variety of unknown technologies when seen from the outside. Despite their relative obscurity, these technologies are both highly intriguing and useful.

Despite the fact that we do not need to grasp these technologies in order to utilise Docker effectively, it is still beneficial to learn about and have an awareness of these technologies.

If we have a deeper understanding of the instrument, it will be much easier for us to make the proper decisions, such as those regarding performance optimization or security implications.

In addition, this facilitates the discovery of innovative new technologies, which may have many more uses for the organisation than initially thought.

Just a note:

Docker does not require cgroupfs as the control group driver. The cgroup can be changed to systemd.

Docker's own control group manager is cgroupfs. Nevertheless, for most Linux distributions, systemd is the default init system, and systemd has tight interaction with Linux control groups.

For Kubernetes, it is recommended to use systemd, as utilising cgroupfs in conjunction with systemd appears to be suboptimal.

So systemd is preferable for cgroup management. kubelet is set to utilise systemd by default. Therefore, Docker should be modified to utilise the systemd driver.

Docker Engine Sparked the Containerization Movement

The Docker Engine is the de-facto container runtime for Linux and Windows Server platforms.

Docker develops simple tools and a uniform packaging strategy that encapsulates all application requirements into a container, which is then executed by Docker Engine.

The Docker Engine enables containerized apps to operate anywhere reliably on any infrastructure, resolving "dependency hell" for developers and operations teams and removing the "it works on my laptop!" issue.

Podman

Podman is RedHat's product. Docker and Podman are fairly comparable to one another in many ways.

Podman provides a Docker-compatible command-line interface that you can alias to the Docker command-line interface with the $ alias docker = podman. Also, Podman provides a socket-activated REST API service that makes it possible for remote apps to launch containers whenever they want.

Users of docker-py and docker-compose are able to connect with Podman as a service because this REST API also supports the Docker API.

By using the libpod library, Podman is able to handle the entirety of the container ecosystem, which includes pods, containers, container images, and container volumes.

Podman is distinct from Docker in that it does not require a server and has a lightweight and Daemon-less design. This means that it makes direct contact with runC to start containers, which eliminates the need for an overhead server.

Containers managed by Podman can either be operated by the root user or by a user with fewer privileges than root.

While working with Docker, the Docker Command Line Interface is how we interface with the daemon that Docker runs in the background. The daemon, which operates on containers and produces pictures, is where the majority of the program's functionality can be found.

This daemon runs with the permissions of the root user. This also suggests that a Docker container may have access to the file system of the host computer if the configuration is not done appropriately.

In contrast, the architecture of Podman makes it possible for us to collaborate with the user who is responsible for running the container, and this user does not need to have root access in order to execute the application.

When compared to containers that run with root capabilities, non-privileged containers provide a substantial advantage. This is because if an intruder breaks into a non-privileged container and flees, the intruder will still be an unprivileged host user. Using this approach provides an additional safeguard for our data.

Just replace Docker with Podman in the instructions to use it. It has the same commands as Docker.

$ alias docker=podman

What other advantages does Podman have?

• Integrated support for systemd makes it possible to run container processes in the background without the need for a separate daemon process.
• Provides us with the ability to build and manage Podami, a collection consisting of one or more functional containers. Because of this, future workload migration to Kubernetes and the orchestration of Podman containers is now possible.
• It is possible to implement UID separation using namespaces, providing an additional layer of security isolation while containers are being executed.
• Can create a YAML file for Kubernetes from a container that is currently operating (with the command $ podman generate kube).

How to Launch a Container

To launch any container we need two things: Image and RunTime.

Container engines such as Docker and Podman are only an additional software layer that sits on top of the runtimes. They are not responsible for actually launching the containers themselves.

They also initiate interaction with the runtimes in the background to start the container processes.

A Container Runtime is a software that runs and manages the components required to run containers.

Runtime is actually a program/software which launches, deletes, and removes containers.

runC is a very famous Runtime, but we have many other Runtimes like gvisor and kata.

Docker connects to this runtime behind the scenes.

The runtime spec file is a configuration file where we give all the important things for the container to be launch like CMD, folder, network, and so on. It is the file that the container runtime uses to launch the container.

We can verify that Docker is using runC as its default runtime engine like this:

$ docker info

How to use runC – the universal container runtime

Because "containers" are a collection of complex and occasionally obscure system elements, they are combined into a single low-level component. This is runC.
As a standalone tool, infrastructure plumbers all around the world utilise runC as plumbing.

runC is a lightweight, portable runtime for containers. It is a command-line tool for creating and running containers according to the Open Container Initiative (OCI) specification. It has libcontainer, which is the original lower-layer library interface that the Docker engine used to set up what we call an operating system container.

runC is designed with the principles of security, usability at large scale, and no dependency on Docker in mind.

Whenever we launch a container, it starts within a second. It looks like a new OS has launched because it has all the things an OS would have (like all the commands, network card, and more). It looks like an independent OS.

How can a container be launched in one second?

As you may know, when you run any program, it becomes a process. So even here, every running container is a process in the host system. So whenever we launch a container, it means we have started some process.

It looks like that container is a different OS with its own file system, network, and so on, as I mentioned above. But the kernel runs this entire setup inside a process by using the concept of namespaces. We'll discuss namespaces further in a minute.

So, as the container is a process, it launches quickly.

A container is just a process running in the RAM. This process looks like it is running a full-flash OS inside an OS. Typically, the process (container) runs the bash command, which has an infinite lifetime till someone gives an exit command.

If we inspect the container, we can find the PID of the /bin/bash command running in the base OS. Now, if we kill the process of the /bin/bash, the container will also be terminated.

What is a namespace?

Again, running a container is also a process. But with Docker, a container is given with its own principal user, network configuration, mount folders, filesystem, and so on which they get as a common component of the baseOS. As a result, the container now has its own isolated environment, known as a namespace.

$ lsns

# To list down all the namespace in the baseOS

To launch a container, Docker Creates a Runtime Spec file for runC, and then runC uses that to launch the container. runC creates Namespaces for the container process.

The Image works as a Hard Drive – that is, it contains the entire file system for the container. A Container Image bundles all the data which is mounted on a storage namespace (that is, mount namespace).

The Image creates the entire bundle in the "/" directory of the containers. We have to unbundle it by untaring it.

Note that runC will not download, unzip, or untar the images for us. It can launch the container and mount the files for us to the container. For the images, we need some image management tools.

Also, runC only provides us with a network namespace, but Docker has to manage the network (that is, specify the IP range, provide IPs, and so on).

Docker can download, unzip, or unbundle the images. It can also do the required networking setup. It can also connect to the storage for us, and many such features are provided by Docker. Typically, Docker can do almost all the stuff provided by Kubernetes via its commands.

In Docker, we have a client-server architecture in which we have Docker CLI working as the client program and the container as the server. The server will now connect to the runtime and launch the container for us.

If we want to launch the container directly, we can do so with the help of runC.

First, we need to install runC using yum.

$ runc list 

# This will list the available containers.

$ runc spec

# To create a runtime spec file in current directory.

runC is written in the Go language. So, generally it supports Go programs (images are also written in the Go language).

$ go build -o name is the command to compile a Go program.

runC Commands:

$ runc create <cont_name>

# To launch a container (This will take config file from current directory).

$ runc start <cont_name>

# This command will run and give the output of the program that we’ve specified in container.

In the workspace we use the following:

$ runc spec

# This will create a config.json file for the runC configuration.

In the congif.json file, we must change the parameters-values according to our requirements.

For example, if we don’t want the terminal, we need to make it false. For that we can give the command to run in the arg, we can set the hostname, and so on.

We can connect to this container namespace by doing the following:

$ nsenter -u -t -n <pid_of_container>

# To enter user and network namespace of specific process ID.

To create a container just by using runC:

1. Install runC.
2. Create config.json by running the runC spec command.
3. Mention all the important things in the above file. We also have to give process by writing its code in the Go language.
4. Create a rootfs folder in the current workspace and move the compiled Go code into this folder.
5. Now, to launch the container:

$ start runc create <container_name>

To start the container and print on the console whatever is there in the go file, run this command:

$ runc start <container_name>

How to generate an example specification with runC:

> runc spec
> cat config.json
{
  "ociVersion": "1.0.0",
  "process": {
    "terminal": true,
    "user": { "uid": 0, "gid": 0 },
    "args": ["sh"],
    "env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "TERM=xterm"
    ],
    "cwd": "/",
    "capabilities": {
      "bounding": ["CAP_AUDIT_WRITE", "CAP_KILL", "CAP_NET_BIND_SERVICE"],
      [...]
    },
    "rlimits": [ { "type": "RLIMIT_NOFILE", "hard": 1024, "soft": 1024 } ],
    "noNewPrivileges": true
  },
  "root": { "path": "rootfs", "readonly": true },
  "hostname": "runc",
  "mounts": [
    {
      "destination": "/proc",
      "type": "proc",
      "source": "proc"
    },
    [...]
  ],
  "linux": {
    "resources": { "devices": [ { "allow": false, "access": "rwm" } ] },
    "namespaces": [
      { "type": "pid" },
      { "type": "network" },
      { "type": "ipc" },
      { "type": "uts" },
      { "type": "mount" }
    ],
    "maskedPaths": [
      "/proc/kcore",
      [...]
    ],
    "readonlyPaths": [
      "/proc/asound",
      [...]
    ]
  }
}

Why Did Kubernetes Depreciate Docker?

Docker is often the first choice when it comes to managing and creating containers and images. It is extremely fast – so you might be wondering why Kubernetes dropped Docker and went on to use the CRI-O container engine? Let’s explore.

Source: https://www.sumologic.com/blog/kubernetes-vs-docker/

We can check the Docker container engine like this:

$ systemctl status docker

Here it shows Docker Application Container Engine but conntainerD is the actual engine running.

In Docker, when kubelet needs to connect to the ContainerD it has to go through an API Docker shim to contact runC. This acts as an interface between Docker and Kubernetes. This makes the whole process fairly complex.

Source: Tutorial Works

What is ConatinerD?

ContainerD is an industry standard container runtime that emphasises simplicity, durability, and portability.

You can find a daemon-based implementation of containerD on both Linux and Windows. It is responsible for managing the whole container lifecycle of the system that it is hosted on, which includes image transfer and storage, container execution and monitoring, low-level storage, and network attachments.

Features of ContainerD:

• OCI Image Spec support
• Image push and pull support
• Network primitives for creation, modification, and deletion of interfaces
• Multi-tenant supported with CAS storage for global images
• OCI Runtime Spec support (aka runC)
• Container runtime and lifecycle support
• Management of network namespaces containers to join existing namespaces

What is dockerd?

Docker's daemon may be kicked off with the help of dockerd (so you can command the daemon to manage images, containers, and so on). Dockerd is a server that runs in the background as a daemon.

To run the Docker daemon we can specify dockerd. After the dockerd keyword, you should supply the daemon parameters you want to use.

dockerd (Docker Daemon) can listen for Docker Engine API requests via three different types of Sockets: unix, tcp, and fd.

Challenges of Using Docker

Overhead

Docker is a fairly developed platform in the container market. Along with container management, it offers many other capabilities like storage, security, and network infrastructure, among other things.

When compared to Cri-O and Podman, Docker's performance suffers as a direct result of the overhead caused by these additional functionalities.

But Kubernetes and OpenShift come equipped with all of these functions. Therefore, they want only one thing from the container engine: the ability to launch and manage the containers. In other words, they do not need any other functions.

Dockershim

In Kubernetes, the process of launching containers begins when kubelet communicates with containerD, which then contacts runC.

Because separate businesses are responsible for the production of containerD and kubelet, kubelet must have an additional layer in order to contact containerD (an API like layer). And inside the Docker ecosystem, this layer is referred to as Dockershim.

Kubernetes had depreciated dockershim because of the complexities and burden created by Docker updates.

What's the issue with dockershim?

Kubernetes suggested a temporary solution to include support for Docker so that it could serve as its container runtime. Dockershim's deprecation just signifies that Dockershim's code will no longer be maintained in the Kubernetes source repository.

Dockershim has become a significant problem for Kubernetes developers. Following this change, the Kubernetes community will only be permitted to maintain the Kubernetes Container Runtime Interface (CRI).

Kubernetes supports all CRI-compliant runtimes, such as containerD and CRI-O.

Kubernetes has come up with CRI-O as its interface to contact with runC. Kubelet will now be contacting CRI-O. Further, it will contact the runC, and the container will be launched.

However, like CRI-O and Docker, there are many container engines present. So, the Kubernetes community decided to create an abstraction layer on top of all container engines. So, a client can use any container engine according to their requirements.

This abstraction layer is called CRI (container runtime interface).

Kubernetes using Docker vs Kubernetes using CRI-O

What is a CRI (Container Runtime Interface)?

The Kubelet program abstracts the underlying container engines. The Container Runtime Interface (CRI) is a plugin interface that lets kubelet use several container runtimes without recompilation.

In addition to protocol buffers, the gRPC API, and libraries, other specifications and tools are in active development for CRI.

Kubelet establishes a connection with CRI over the gRPC Protocol.

What is CRI-O?

CRI-OCI is an abbreviation that stands for the Container Runtime Interface and OCI, which stands for the Open Container Initiative.

The term CRI-O was chosen after taking into account references made by members of the CRI and CIO communities.

CRI-O is another container engine, but it is more lightweight than Docker since it does not include the additional capabilities that Docker has, such as networking, storage, and so on.

CRI-O provides a foundation that is more secure, stable, and performant for the execution of runtimes that are compatible with the Open Container Initiative (OCI). Runtimes that are OCI-compliant can be used in conjunction with the CRI-O container engine to launch containers and pods.

Examples of such runtimes include runC, which is the default OCI runtime, and Kata Containers. But you can use any OCI-conformant runtime in its place.

The goal of the CRI-O project is to replace Docker as the container engine that implements the Kubernetes Container Runtime Interface (CRI) for OpenShift Container Platform and Kubernetes.

The stability of CRI-O may be attributed to the fact that it is developed, tested, and distributed in tandem with major and minor versions of Kubernetes and complies with OCI standards.

The O's in CRI-scope are reliant on the Container Runtime Interface (CRI). The actual container engine specifications of a Kubernetes service (kubelet) were compiled and specified by CRI.

In light of the fact that several container engines were being developed, the CRI team decided to take this measure in order to settle Kubernetes' requirements for container engines.

According to the OpenShift Docs, the tools that help replace and extend what the Docker command and service provided are:

• crictl: For working directly with CRI-O container engines & troubleshooting
• runc: For running container images
• podman: For managing pods and container images (run, stop, start, ps, attach, exec, and so on) outside of the container engine
• buildah: For building, pushing and signing container images
• skopeo: For copying, inspecting, deleting, and signing images

CRI-O Architecture

Source: cri-o.io

How to Build a Multi-Node Cluster using CRI-O

Following are the commands you'd use to create a multi-node Kubernetes cluster using CRI-O in Ubuntu 20.04.

Here's the repository that contains the set of commands:

OS=xUbuntu_20.04
VERSION=1.20

cat >>/etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list<<EOF
deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /
EOF

cat >>/etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list<<EOF
deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /
EOF

curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -

curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers-cri-o.gpg add -

apt update

apt install -qq -y cri-o cri-o-runc cri-tools

systemctl daemon-reload

systemctl enable --now crio

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -


apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"

apt install -qq -y kubeadm=1.20.5-00 kubelet=1.20.5-00 kubectl=1.20.5-00

cat >>/etc/modules-load.d/crio.conf<<EOF
overlay
br_netfilter
EOF

modprobe overlay

modprobe br_netfilter

cat >>/etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
EOF

sysctl --system

cat >>/etc/crio/crio.conf.d/02-cgroup-manager.conf<<EOF
[crio.runtime]
conmon_cgroup = "pod"
cgroup_manager = "cgroupfs"
EOF

systemctl daemon-reload

systemctl enable --now crio

systemctl restart crio

sed -i '/swap/d' /etc/fstab

swapoff -a

systemctl disable --now ufw

kubeadm init --apiserver-advertise-address=172.16.16.100 --pod-network-cidr=192.168.0.0/16

kubectl --kubeconfig=/etc/kubernetes/admin.conf create -f https://docs.projectcalico.org/v3.18/manifests/calico.yaml

kubeadm token create --print-join-command

We can use the join command to connect the nodes to the cluster and we're ready with a multi-node cluster of Kubernetes.

Here's an illustration of how Docker, Kubernetes, CRI-O, containerD and runC work together

Container Runtimes

We have seen a lot of details on how containers work, we have defined container runtimes and how we can build our custom container using runC. Now, are there any other tools in hand for us like runC?

Here we will look at landscape of all the container runtimes that are available.

Generally, they fall into two main categories:

1. Open Container Initiative (OCI) runtimes

The OCI runtimes are further classified in two broader categories: Native Runtimes and Sandboxed & Virtualized Runtimes

2. Container Runtime Interface (CRI)

The CRI consists of containerD and CRI-O.

1. Open Container Initiative (OCI) runtimes

2. Container Runtime Interface (CRI)

Conclusion

Here we saw how we can build our custom images and the various tool available at our disposal. It was a long one but I hope you've enjoyed it and have learned something new.

I'm always open to suggestions and discussions on LinkedIn. Hit me up with direct messages.

If you've enjoyed my writing and want to keep me motivated, consider leaving starts on GitHub and endorse me for relevant skills on LinkedIn.

Till the next one, stay safe and keep learning.