The question is never whether systems need to communicate. The question is always how.

For years, REST over HTTP with JSON was the default answer. It works, it's simple, and the tooling is everywhere. But as systems grow in scale (in the number of services talking to each other, the volume of data being exchanged, and the need for real-time communication), REST starts to show its limits.

This is where Remote Procedure Calls, Protocol Buffers, and gRPC enter the picture.

In this handbook, you'll learn what RPC is and the problem it was designed to solve. You'll also understand Protocol Buffers: what they are, why they exist, and how they work.

You'll then see how Google combined these ideas into gRPC, one of the most powerful communication frameworks in modern distributed systems. You'll learn all four gRPC communication patterns, see code generated across multiple languages from a single contract file, and walk through a complete end-to-end Flutter implementation with production-grade concerns including authentication, error handling, and timeouts.

By the end, you won't just know what gRPC is. You'll understand when to use it, when not to, and how to think about service communication as a systems engineer.

Table of Contents

• What is a Remote Procedure Call?

• The Problem RPC Solves

• Why gRPC Over REST: The Real Case

• Protocol Buffers: A New Language for Data

• The Proto File

• JSON vs Protocol Buffers

• The Protoc Compiler and Code Generation

• What is gRPC?

• Why HTTP/2 Matters for gRPC

• The Four gRPC Communication Patterns

• The Protobuf Repository: Organizational Best Practice

• Building a Complete gRPC System with Dart and Flutter

• Production Concerns

• gRPC vs REST vs WebSockets: When to Use What

• The Hybrid Architecture

• Conclusion

What is a Remote Procedure Call?

To understand RPCs, you first need to understand what a procedure call is.

A procedure call means invoking a procedure or function so that its code executes. For example, in Dart:

double calculateTax(double amount) {
  return amount * 0.075;
}

final tax = calculateTax(50000); // local procedure call

You call calculateTax, pass an argument, and get a result back. The function lives on the same machine, in the same process, in the same memory space. This is a local procedure call.

A Remote Procedure Call takes this same idea and stretches it across a network. The function you're calling lives on a different machine, in a different process, and potentially in a different country. But from the caller's perspective, it feels exactly like calling a local function.

// This looks like a local function call
final tax = await taxService.calculateTax(amount: 50000);

// But under the hood, this:
// 1. Serializes the argument into a binary format
// 2. Sends it over a network connection to a remote server
// 3. The server executes calculateTax with your argument
// 4. Serializes the result
// 5. Sends it back over the network
// 6. Deserializes it into a Dart object
// 7. Returns it to you as if it were local

The network complexity is completely hidden. You call a function. You get a result. Everything in between is handled by the RPC framework.

This is the fundamental idea behind RPC: make calling a remote function feel as natural as calling a local one.

The Problem RPC Solves

To appreciate why RPC matters, you need to understand what the alternative looks like.

Without RPC, calling a remote service looks like this:

Future<double> calculateTax(double amount) async {
  final response = await http.post(
    Uri.parse('https://tax-service.internal/api/v1/calculate'),
    headers: {
      'Content-Type': 'application/json',
      'Authorization': 'Bearer $token',
    },
    body: jsonEncode({'amount': amount}),
  );

  if (response.statusCode != 200) {
    throw Exception('Tax calculation failed: ${response.statusCode}');
  }

  final data = jsonDecode(response.body);
  return (data['tax'] as num).toDouble();
}

Every service call requires you to:

• Know and hardcode the endpoint URL

• Know the correct HTTP method

• Manually serialize your request to JSON

• Handle HTTP status codes yourself

• Manually deserialize the response from JSON

• Cast dynamic types to the types you actually expect

• Hope the field names in the response match what you think they are

Now multiply this by every single service call in your application. An authentication service, a user service, a payment service, a notification service, a transaction service. Every one of them requires the same manual boilerplate. Every one of them introduces the possibility of a typo in a field name, a wrong status code assumption, or a JSON deserialization failure that only surfaces at runtime.

With RPC:

// Feels like a local function call
final tax = await taxService.calculateTax(
  TaxRequest(amount: 50000),
);
// tax is already a strongly typed TaxResponse object
// No URLs. No HTTP methods. No JSON parsing. No casting.

The framework handles everything. The function signature is defined in a contract file that both the client and server use. The types are enforced at compile time. If the server changes the response shape, the client fails to compile before anything reaches production.

This is what RPC solves: it removes the accidental complexity of network communication and lets you focus on what you're actually trying to do.

Why gRPC Over REST: The Real Case

Before going into the technical details of Protocol Buffers and gRPC, it's important to make a solid case for why you'd choose gRPC over REST in specific scenarios. This isn't a claim that gRPC is always better. It's a clear look at where it genuinely wins.

Large Payloads Called by Many Internal Systems

Consider an internal enterprise API in a large telecommunications company. A single request and response payload for a plan registration or activation flow can contain over a thousand fields. This endpoint is called by dozens of internal applications: billing systems, CRM platforms, customer-facing mobile apps, internal dashboards, and partner portals.

With REST and JSON, every one of those applications sends and receives that thousand-field payload as text. Field names like subscription_activation_status, rate_plan_identifier, and network_provisioning_reference travel over the wire as strings on every single request. A significant portion of every payload isn't data. It's labels for data.

With Protocol Buffers, field names never appear in the payload at all. Only field numbers and values travel over the wire. That thousand-field payload shrinks dramatically. For an endpoint called millions of times per day by dozens of systems, the bandwidth saving is enormous and translates directly to infrastructure cost reduction.

Beyond size, the generated client guarantee is equally important. With REST, each of those dozens of applications reads the API documentation and builds its own understanding of the contract. When the backend changes a field name or type, not every application finds out immediately. Some find out in production when they break.

With a shared .proto file, every application generates its own strongly typed client from the same source. A contract change means every application regenerates. The compiler immediately reports where the breaking change affects each codebase. Nothing reaches production in a broken state.

Low Bandwidth and Remote Network Conditions

This is one of the most under-appreciated advantages of gRPC in markets where network quality varies significantly.

In many regions, a substantial portion of mobile users are on 2G or 3G connections. On a 2G connection, bandwidth can be as low as 50 to 100 kilobits per second. A REST JSON response that is 80 kilobytes takes over 6 seconds to download on a 2G connection. The equivalent protobuf binary, which can be 3 to 10 times smaller, takes under 2 seconds.

That difference isn't a technical footnote. It's the line between an application that feels usable and one that feels broken to a significant portion of your users. For any product that operates in markets with variable network conditions, protobuf's binary efficiency is a direct competitive advantage.

Beyond size, gRPC runs over HTTP/2 which maintains a single persistent connection rather than opening a new connection for every request. On slow networks where connection establishment (the TCP handshake and TLS negotiation) can itself take hundreds of milliseconds, reusing a single connection across many calls saves significant time over a session.

Microservice to Microservice Communication

When two internal services need to communicate, you have options. A message bus like Kafka or RabbitMQ is excellent when you don't need an immediate response, when the operation can happen asynchronously, and when you're broadcasting something that happened to multiple consumers.

But many service-to-service calls are synchronous by nature. An authentication service needs to validate a token right now before the request proceeds. A fraud detection service needs to assess a transaction right now before the payment is authorized. A pricing service needs to calculate a rate right now before the quote is generated. These operations can't publish an event and wait.

For synchronous service-to-service calls at high frequency, gRPC over HTTP/2 with protobuf encoding is significantly more efficient than REST. The persistent multiplexed connection means no connection setup overhead per call. The binary encoding means no JSON serialization and deserialization overhead on every hop. The generated clients mean both services compile against the same contract.

At scale, when two services are calling each other thousands of times per second, these efficiency differences compound into real performance and cost differences.

Managing API Contracts Across Multiple Teams

In a large engineering organization, multiple teams build services that others depend on. REST API contracts live in documentation. Documentation goes stale. The backend team changes a field name. The mobile team finds out when users report crashes. The data team finds out when their pipeline throws an error at 2am.

gRPC's protobuf repository approach transforms contract management from a documentation problem into a code problem. Contract changes go through pull requests. Every dependent team reviews the change. Breaking changes are caught at compile time. Nobody is surprised in production.

This governance benefit scales with team size. The larger the organization, the more valuable it becomes.

Real-Time Communication

REST is request-response. The client asks and the server answers. The conversation ends. For real-time features, you either poll (wasteful) or bolt on a separate WebSocket server alongside your REST API (two different systems to maintain).

gRPC's streaming patterns handle real-time communication natively within the same framework you use for regular calls. A live balance update, a real-time transaction notification, or a bidirectional chat session all use the same generated client, the same connection, and the same protobuf encoding as your regular unary calls.

One framework with all communication patterns. No separate infrastructure.

Protocol Buffers: A New Language for Data

RPC is a concept. To implement it, you need two things: a way to define the contract between client and server, and a way to serialize data efficiently for transmission over the network.

This is where Protocol Buffers comes in.

Protocol Buffers, commonly called protobuf, is a language-neutral, platform-neutral, extensible mechanism for serializing structured data. It was developed at Google in 2001, used internally for years, and open-sourced in 2008.

The JSON Problem at Scale

JSON is the dominant data format for web APIs. It's human-readable, flexible, and universally supported. For many use cases, it's the right choice.

But JSON has structural inefficiencies that become painful at scale.

Consider a user profile response:

{
  "id": "usr_001",
  "first_name": "John",
  "last_name": "Smith",
  "email": "john@example.com",
  "phone_number": "+2348012345678",
  "account_type": "savings",
  "balance": 500000.00,
  "currency": "NGN",
  "is_verified": true,
  "is_active": true,
  "kyc_level": 3,
  "created_at": "2024-01-15T10:30:00Z",
  "last_login": "2026-07-20T09:15:00Z"
}

Every field name travels over the network as a string on every single response. "first_name", "account_type", "phone_number" aren't data. They're labels for data. But they consume bytes on every request.

Now consider an internal enterprise API with over a thousand fields in its request and response payload, being called by dozens of internal applications thousands of times per day. A significant portion of every payload is field name strings, not actual data. The overhead accumulates into real bandwidth and processing costs.

Beyond size, JSON has another problem: it has no schema at the network level. Nothing prevents a backend engineer from renaming "first_name" to "firstName" in a new deployment. The client breaks at runtime in production with real users.

What Protocol Buffers Do Differently

Protocol Buffers solve both problems with a fundamentally different approach to data encoding.

Instead of encoding data as human-readable text with field names, protobuf encodes data as compact binary using only field numbers and values. Field names never travel over the network.

Here's the same user profile defined in protobuf:

message UserProfile {
  string id = 1;
  string first_name = 2;
  string last_name = 3;
  string email = 4;
  string phone_number = 5;
  string account_type = 6;
  double balance = 7;
  string currency = 8;
  bool is_verified = 9;
  bool is_active = 10;
  int32 kyc_level = 11;
  string created_at = 12;
  string last_login = 13;
}

When protobuf encodes this data, the output is binary that no human can read. But to a machine, it's extremely compact and fast to parse. The field numbers (1, 2, 3...) identify each field. The names never appear in the encoded output at all.

The result: the same user profile that's approximately 280 bytes in JSON is approximately 95 bytes in protobuf. That's three times smaller. For a thousand-field enterprise payload, this difference is enormous.

And because the schema is defined in a .proto file that both client and server compile against, field name changes are caught at compile time, not at runtime.

The Proto File

The .proto file is the heart of everything in the protobuf and gRPC ecosystem. It's where you define your data models and your service contracts.

It's written in Protocol Buffer Language (proto3) – not Go, not Dart, not Python, not Java. You write it in any text editor. VS Code with the vscode-proto3 extension gives you syntax highlighting, autocomplete, and inline validation.

Here's a complete .proto file for a fintech platform:

syntax = "proto3";

package banking;

option go_package = "./banking";
option java_package = "com.fintech.banking";



service BankingService {
  // Unary: one request, one response
  rpc Login (LoginRequest) returns (LoginResponse);

  // Unary: fetch user profile
  rpc GetProfile (ProfileRequest) returns (UserProfile);

  // Server streaming: real-time balance updates
  rpc WatchBalance (BalanceRequest) returns (stream BalanceResponse);

  // Server streaming: live transaction feed
  rpc StreamTransactions (TransactionRequest) returns (stream Transaction);

  // Client streaming: upload KYC documents in chunks
  rpc UploadDocument (stream DocumentChunk) returns (UploadResponse);

  // Bidirectional streaming: live chat support
  rpc Chat (stream ChatMessage) returns (stream ChatMessage);
}



message LoginRequest {
  string email = 1;
  string password = 2;
}

message LoginResponse {
  string token = 1;
  string user_id = 2;
  int64 expires_at = 3;
}

message ProfileRequest {
  string user_id = 1;
}

message UserProfile {
  string id = 1;
  string first_name = 2;
  string last_name = 3;
  string email = 4;
  string phone_number = 5;
  string account_type = 6;
  double balance = 7;
  string currency = 8;
  bool is_verified = 9;
  int32 kyc_level = 10;
}

message BalanceRequest {
  string user_id = 1;
}

message BalanceResponse {
  double balance = 1;
  string currency = 2;
  int64 timestamp = 3;
}

message TransactionRequest {
  string user_id = 1;
  int32 limit = 2;
}

message Transaction {
  string id = 1;
  double amount = 2;
  string description = 3;
  string type = 4;
  int64 timestamp = 5;
}

message DocumentChunk {
  bytes data = 1;
  string document_type = 2;
  int32 chunk_index = 3;
  bool is_last = 4;
}

message UploadResponse {
  bool success = 1;
  string document_id = 2;
  string message = 3;
}

message ChatMessage {
  string sender_id = 1;
  string content = 2;
  int64 timestamp = 3;
}

Let's walk through every part of this file carefully.

The Syntax Declaration

syntax = "proto3";

This tells the protobuf compiler which version of the Protocol Buffer language you're using. proto3 is the current standard. It must be the first non-comment line in every .proto file.

The Package Declaration

package banking;

The package name prevents naming conflicts when you have multiple .proto files across different services. It functions like a namespace. If two services both define a UserProfile message, the package name distinguishes them: banking.UserProfile versus auth.UserProfile.

Language-Specific Options

option go_package = "./banking";
option java_package = "com.fintech.banking";

These options tell the compiler how to organize the generated code for specific languages. They don't affect the proto file itself, only the generated output.

The Service Definition

service BankingService {
  rpc Login (LoginRequest) returns (LoginResponse);
  rpc WatchBalance (BalanceRequest) returns (stream BalanceResponse);
}

The service block defines the RPC contract. Think of it exactly like an abstract class in any object-oriented language. It declares what functions exist, what they accept, and what they return.

Each rpc line defines one remote procedure. The stream keyword before a type indicates that multiple messages will flow rather than just one.

Message Definitions

message LoginRequest {
  string email = 1;
  string password = 2;
}

A message is a data structure. Think of it as a class with only fields: no methods, no logic. Each field has three parts.

The type can be string, int32, int64, double, bool, bytes, or another message type.

The name is the field name as it appears in generated code. This is for human readability only. It never appears in the binary encoding.

The field number (= 1, = 2, = 3) is the unique identifier that protobuf uses in the binary output instead of the field name. This is critical: once you assign a field number, you must never change it or reuse it. The binary encoding uses these numbers, not names. If you change a field number, old encoded data becomes unreadable.

You can safely add new fields with new numbers, remove fields (the number stays reserved, never reuse it), and rename fields (names don't appear in binary). You must never change a field number, reuse a removed field's number, or change a field's type.

JSON vs Protocol Buffers

Now that you understand both formats, let's make a direct comparison.

Size Comparison

Let's look at the same login request in both formats:

JSON (text):

{
  "email": "john@example.com",
  "password": "securepassword123"
}

Approximately 55 bytes.

Protobuf binary:

Field 1 (email): tag + length + value bytes. Field 2 (password): tag + length + value bytes.

Approximately 38 bytes.

For a simple two-field message, the difference is modest. Now consider a thousand-field enterprise payload. Field names alone in JSON can account for 40-60% of the total payload size. In protobuf, field names contribute zero bytes to the payload.

On a 2G connection where bandwidth can be as low as 50 kilobits per second, the difference between an 80 kilobyte JSON response and a 15 kilobyte protobuf response is the difference between a 13-second load and a 2-second load. For users in areas with limited network infrastructure, this isn't a performance metric. It's a usability threshold.

Speed Comparison

Protobuf serialization and deserialization is significantly faster than JSON parsing because binary parsing requires no string tokenizing, quote handling, whitespace skipping, or type inference. The parser reads a field number, reads the value type, reads the value, and moves to the next field. It's a direct binary read.

JSON parsing must tokenize a string character by character, identify keys and values by their surrounding quotes and delimiters, infer types from the value format, and construct objects from dynamic maps.

On a mobile device handling hundreds of responses per session, this parsing difference translates to measurable CPU and battery savings.

Schema and Type Safety

JSON has no schema enforcement at the network level. A backend can change "balance" to "current_balance" and the client only discovers this when the app crashes in production.

Protobuf schemas are enforced at compile time. If the .proto file changes in a way that breaks the client, the client fails to compile. The problem is caught before it reaches any user.

The Honest Comparison

The Protoc Compiler and Code Generation

The protoc compiler reads your .proto file and generates code in any language you specify. This is where the universal contract becomes reality.

Generating Go code (for the backend server):

protoc \
  --go_out=. \
  --go-grpc_out=. \
  proto/banking.proto

Generates:

banking.pb.go        <- the message structs
banking_grpc.pb.go   <- the server interface

Generating Dart code (for the Flutter client):

protoc \
  --dart_out=grpc:lib/generated \
  proto/banking.proto

Generates:

lib/generated/
  banking.pb.dart        <- the message classes
  banking.pbgrpc.dart    <- the client stub

Generating Python code (for a data service):

protoc \
  --python_out=. \
  --grpc_python_out=. \
  proto/banking.proto

Generates:

banking_pb2.py         <- the message classes
banking_pb2_grpc.py    <- the client and server classes

Generating TypeScript code (for a web frontend):

protoc \
  --ts_out=. \
  proto/banking.proto

Generates:

banking.ts             <- typed message classes and client

All of this from the same single banking.proto file.

The Go backend engineer never writes serialization code. The Flutter engineer never writes deserialization code. The Python data engineer never parses binary manually. The TypeScript web engineer never constructs HTTP requests. All of that is generated automatically from the contract that every team agreed on.

Here's what the generated code looks like in each language to make this concrete:

Generated Go server interface (the backend implements this):

// Generated — do not edit
type BankingServiceServer interface {
    Login(context.Context, *LoginRequest) (*LoginResponse, error)
    WatchBalance(*BalanceRequest, BankingService_WatchBalanceServer) error
    mustEmbedUnimplementedBankingServiceServer()
}

// The Go backend engineer writes this implementation
type bankingServer struct {
    pb.UnimplementedBankingServiceServer
}

func (s *bankingServer) Login(
    ctx context.Context,
    req *pb.LoginRequest,
) (*pb.LoginResponse, error) {
    token, err := authService.Login(req.Email, req.Password)
    if err != nil {
        return nil, status.Errorf(codes.Unauthenticated, "invalid credentials")
    }
    return &pb.LoginResponse{
        Token:  token,
        UserId: user.Id,
    }, nil
}

Generated Python client (the data team uses this):

import grpc
import banking_pb2
import banking_pb2_grpc

channel = grpc.secure_channel(
    'api.fintech-platform.com:50051',
    grpc.ssl_channel_credentials()
)
stub = banking_pb2_grpc.BankingServiceStub(channel)

response = stub.Login(banking_pb2.LoginRequest(
    email='john@example.com',
    password='password123'
))

print(f"Token: {response.token}")
print(f"User ID: {response.user_id}")

Generated Dart client (you use this in Flutter):

import 'package:grpc/grpc.dart';
import 'generated/banking.pbgrpc.dart';
import 'generated/banking.pb.dart';

final channel = ClientChannel('api.fintech-platform.com', port: 50051);
final client = BankingServiceClient(channel);

final response = await client.login(
  LoginRequest(email: 'john@example.com', password: 'password123'),
);

print('Token: ${response.token}');
print('User ID: ${response.userId}');

Three different languages. Three different teams. One .proto file. All of them are generated, strongly typed, and guaranteed to be in sync with the server.

What is gRPC?

gRPC is Google's open-source Remote Procedure Call framework. It was open-sourced in 2016 and is now a Cloud Native Computing Foundation (CNCF) graduated project. This means it's been production-proven at the highest level of the cloud-native ecosystem.

gRPC combines three things:

1. Remote Procedure Calls as the programming model: calling remote functions like local ones.

2. Protocol Buffers as the interface definition language and data serialization format: strongly typed contracts and compact binary encoding.

3. HTTP/2 as the transport protocol: multiplexed, persistent connections with binary framing.

The combination of these three produces a framework that's faster than REST, more structured than WebSockets, and more powerful than any of its predecessors.

gRPC is used internally at Google for virtually all service-to-service communication. Netflix, Uber, Square, Dropbox, Lyft, and hundreds of other organizations use it for their internal microservice communication. Official support exists for Go, Java, Python, C++, C#, Ruby, Node.js, PHP, Dart, Kotlin, and more.

Why HTTP/2 Matters for gRPC

gRPC is built exclusively on HTTP/2. Understanding what HTTP/2 provides is essential to understanding why gRPC performs the way it does.

HTTP/1.1, which powers most REST APIs, has fundamental performance constraints. Each request must complete before the next one begins on the same connection. Headers are sent as verbose text on every request. The server can't send data unless the client asks first.

HTTP/2 was designed to fix these constraints at the protocol level.

Multiplexing

HTTP/2 introduces streams within a single connection. Multiple independent requests can travel over the same TCP connection simultaneously.

Single TCP connection to api.fintech-platform.com

Stream 1: Login request ---------> Login response
Stream 2: Profile request -------> Profile response
Stream 3: Balance request -------> Balance stream (ongoing)
Stream 4: Transactions request --> Transaction stream (ongoing)

All four streams active simultaneously over ONE connection

In HTTP/1.1, you would need four separate connections or wait for each to complete before starting the next. HTTP/2 handles all four over a single persistent connection with no waiting.

This is the foundation of gRPC's streaming capabilities. A persistent multiplexed connection is what allows the server to keep pushing balance updates and transaction notifications while the client continues making other calls.

Binary Framing

HTTP/1.1 sends everything as text. HTTP/2 sends everything as binary frames. Binary is more compact and significantly faster for machines to parse.

Every gRPC message is broken into binary frames and sent over the HTTP/2 connection. Combined with protobuf's binary encoding, gRPC data travels in the most compact form possible at every layer.

Header Compression (HPACK)

HTTP/1.1 sends full headers on every request. An Authorization header carrying a JWT token can be 500 bytes or more, repeated on every request.

HTTP/2 uses HPACK compression. Headers sent on previous requests are cached. Subsequent requests only send headers that changed. The Authorization header, once sent, is referenced by a short index rather than retransmitted in full.

On a mobile application making dozens of authenticated requests per session, this compression is a meaningful bandwidth saving, especially on slow networks where every byte matters.

Server Push

HTTP/2 allows the server to proactively send data to the client without waiting for a request. The client opens a stream and the server keeps pushing messages through it as events occur.

This is the mechanism behind gRPC server streaming. The client sends one WatchBalance request and the server pushes a new BalanceResponse every time the balance changes. No polling or repeated requests. The connection stays open and the server speaks whenever it has something new to say.

The Four gRPC Communication Patterns

This is the most important section of this article. gRPC doesn't have one communication model. It has four. Each one is defined precisely in the .proto file and serves different use cases.

Pattern 1: Unary RPC

One request from the client and one response from the server. This is identical to a REST API call in terms of the request-response flow.

rpc Login (LoginRequest) returns (LoginResponse);

Client ----LoginRequest----> Server
Client <---LoginResponse---- Server
Done.

When to use Unary RPC: Login, profile fetch, payment initiation, data creation, configuration retrieval: any operation that follows a simple ask-and-answer pattern.

Dart implementation:

Future<LoginResponse> login(String email, String password) async {
  try {
    return await _client.login(
      LoginRequest(email: email, password: password),
    );
  } on GrpcError catch (e) {
    throw _mapGrpcError(e);
  }
}

Go server implementation:

func (s *bankingServer) Login(
    ctx context.Context,
    req *pb.LoginRequest,
) (*pb.LoginResponse, error) {
    user, err := s.authService.Login(req.Email, req.Password)
    if err != nil {
        return nil, status.Errorf(codes.Unauthenticated, "invalid credentials: %v", err)
    }
    token, _ := s.tokenService.Generate(user.Id)
    return &pb.LoginResponse{
        Token:  token,
        UserId: user.Id,
    }, nil
}

Pattern 2: Server Streaming RPC

One request from the client and a continuous stream of responses from the server. The connection stays open and the server pushes messages as they become available.

rpc WatchBalance (BalanceRequest) returns (stream BalanceResponse);
rpc StreamTransactions (TransactionRequest) returns (stream Transaction);

Client ----BalanceRequest----> Server
Client <---BalanceResponse---- Server (balance: 500000)
Client <---BalanceResponse---- Server (balance: 495000, after a debit)
Client <---BalanceResponse---- Server (balance: 995000, after a credit)
[stream stays open, server pushes on every change]

When to use Server Streaming: Live account balance, real-time transaction notifications, live stock prices, sports scores, news feeds, system monitoring dashboards: anything where the server has an ongoing series of updates to deliver.

Dart implementation:

Stream<BalanceResponse> watchBalance(String userId) {
  return _client.watchBalance(
    BalanceRequest(userId: userId),
  );
}

In Flutter, consume this with a StreamBuilder:

StreamBuilder<BalanceResponse>(
  stream: _dataSource.watchBalance(currentUserId),
  builder: (context, snapshot) {
    if (snapshot.connectionState == ConnectionState.waiting) {
      return const CircularProgressIndicator();
    }

    if (snapshot.hasError) {
      return Text('Error: ${snapshot.error}');
    }

    if (!snapshot.hasData) {
      return const Text('Waiting for balance...');
    }

    final balance = snapshot.data!;
    return Column(
      children: [
        Text(
          '${balance.currency} ${balance.balance.toStringAsFixed(2)}',
          style: const TextStyle(
            fontSize: 36,
            fontWeight: FontWeight.bold,
          ),
        ),
        Text(
          'Updated: ${DateTime.fromMillisecondsSinceEpoch(balance.timestamp.toInt())}',
        ),
      ],
    );
  },
)

Every time the server pushes a new balance, the StreamBuilder calls builder again and the widget shows the updated value. There's zero polling logic or manual refresh. The server speaks and the widget listens.

Go server implementation:

func (s *bankingServer) WatchBalance(
    req *pb.BalanceRequest,
    stream pb.BankingService_WatchBalanceServer,
) error {
    ticker := time.NewTicker(time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-stream.Context().Done():
            return nil
        case <-ticker.C:
            balance, err := s.accountService.GetBalance(req.UserId)
            if err != nil {
                return status.Errorf(codes.Internal, "failed to fetch balance: %v", err)
            }

            if err := stream.Send(&pb.BalanceResponse{
                Balance:   balance.Amount,
                Currency:  balance.Currency,
                Timestamp: time.Now().UnixMilli(),
            }); err != nil {
                return err
            }
        }
    }
}

Pattern 3: Client Streaming RPC

The client sends a stream of messages to the server. The server processes them all and responds once at the end.

rpc UploadDocument (stream DocumentChunk) returns (UploadResponse);

Client ----Chunk 1 (bytes 0-1024)-----> Server
Client ----Chunk 2 (bytes 1024-2048)--> Server
Client ----Chunk 3 (bytes 2048-3072)--> Server
Client ----Chunk 4 (last chunk)-------> Server
Client <---UploadResponse-------------- Server (document_id: "doc_001")

When to use Client Streaming: Uploading large files (KYC documents, profile photos) in chunks, sending a batch of sensor readings, submitting bulk records to a server.

Dart implementation:

Future<UploadResponse> uploadDocument(
  List<Uint8List> chunks,
  String documentType,
) async {
  try {
    Stream<DocumentChunk> chunkStream() async* {
      for (int i = 0; i < chunks.length; i++) {
        yield DocumentChunk(
          data: chunks[i],
          documentType: documentType,
          chunkIndex: i,
          isLast: i == chunks.length - 1,
        );
      }
    }

    return await _client.uploadDocument(chunkStream());
  } on GrpcError catch (e) {
    throw _mapGrpcError(e);
  }
}

chunkStream() is an async generator function. The async* keyword means it yields values over time rather than returning a single value. Each yield produces one DocumentChunk message that gRPC sends to the server. The server receives these one by one and processes them all before sending a single UploadResponse at the end.

Go server implementation:

func (s *bankingServer) UploadDocument(
    stream pb.BankingService_UploadDocumentServer,
) error {
    var allData []byte
    var documentType string

    for {
        chunk, err := stream.Recv()
        if err == io.EOF {
            break
        }
        if err != nil {
            return status.Errorf(codes.Internal, "failed to receive chunk: %v", err)
        }

        allData = append(allData, chunk.Data...)
        documentType = chunk.DocumentType
    }

    docId, err := s.documentService.Store(allData, documentType)
    if err != nil {
        return status.Errorf(codes.Internal, "failed to store document: %v", err)
    }

    return stream.SendAndClose(&pb.UploadResponse{
        Success:    true,
        DocumentId: docId,
        Message:    "Document uploaded successfully",
    })
}

Pattern 4: Bidirectional Streaming RPC

Both the client and server stream messages simultaneously. Both sides can send at any time. Neither waits for the other.

rpc Chat (stream ChatMessage) returns (stream ChatMessage);

Client ----"Hello"---------------------------> Server
Server <---"Hi, how can I help?"-------------- Client
Client ----"What is my account balance?"-----> Server
Server <---"Your balance is NGN 500,000"------ Client
Server <---"New transaction alert: -5,000"---- Client (server-initiated)
Client ----"Thanks"--------------------------> Server
[both sides communicate freely and simultaneously]

When to use Bidirectional Streaming: Real-time chat, live collaborative document editing, multiplayer game state synchronization, interactive trading terminals, real-time customer support sessions.

Dart implementation:

void startChat(String userId) {
  final outgoing = StreamController<ChatMessage>();

  final incoming = _client.chat(outgoing.stream);

  incoming.listen(
    (message) {
      print('${message.senderId}: ${message.content}');
    },
    onError: (error) {
      print('Chat error: $error');
    },
    onDone: () {
      print('Chat session ended');
    },
  );

  outgoing.add(ChatMessage(
    senderId: userId,
    content: 'Hello, I need help with my account',
    timestamp: DateTime.now().millisecondsSinceEpoch,
  ));
}

StreamController manages the outgoing message stream. You add messages to outgoing whenever the user sends something. The incoming stream delivers messages from the server. Both run simultaneously over the same HTTP/2 connection.

Go server implementation:

func (s *bankingServer) Chat(stream pb.BankingService_ChatServer) error {
    for {
        msg, err := stream.Recv()
        if err == io.EOF {
            return nil
        }
        if err != nil {
            return err
        }

        response := s.chatService.Process(msg)
        if err := stream.Send(&pb.ChatMessage{
            SenderId:  "support_agent",
            Content:   response,
            Timestamp: time.Now().UnixMilli(),
        }); err != nil {
            return err
        }
    }
}

The Protobuf Repository: Organizational Best Practice

In a small project, the .proto file can live inside the backend repository. The mobile engineer clones the backend repo to get it. This works at small scale.

In any organization of meaningful size, this approach breaks down. The backend repo becomes the source of truth, giving the backend team unilateral control over the contract. Other teams find out about changes when their builds break.

The industry best practice is a dedicated protobuf repository: a standalone repository that belongs to everyone and is owned exclusively by no one.

fintech-api-contracts/
  proto/
    auth/
      auth.proto
    banking/
      banking.proto
    payments/
      payments.proto
    notifications/
      notifications.proto
    kyc/
      kyc.proto
  scripts/
    generate_dart.sh
    generate_go.sh
    generate_python.sh
  README.md

How Contract Changes Work

Every API change follows the same process:

Engineer proposes a change to banking.proto
          |
          raises a Pull Request in fintech-api-contracts
          |
Flutter team lead reviews:
  "Does this break our client? Do we need to update?"

Go backend lead reviews:
  "Is this implementable? Does it follow our conventions?"

React web lead reviews:
  "Does the web client need changes?"

Python data lead reviews:
  "Does this affect our data pipelines?"
          |
All teams approve
          |
PR merges — the change is now the law
          |
Every team runs their code generation script
          |
Builds fail where breaking changes exist
Changes are caught at compile time
Before any code reaches production

This process gives you something REST with documentation can never provide: guaranteed contract synchronization across every team, enforced by the compiler, before anything reaches users.

Building a Complete gRPC System with Dart and Flutter

Now let's put everything together in a complete, production-structured example.

Project Setup

Add the gRPC dependency to your Flutter project:

# pubspec.yaml
dependencies:
  flutter:
    sdk: flutter
  grpc: ^3.2.4
  protobuf: ^3.1.0

dev_dependencies:
  protoc_plugin: ^21.1.2

Install the protoc compiler and the Dart plugin:

# macOS
brew install protobuf

# Install the Dart protoc plugin
dart pub global activate protoc_plugin

Generate the Dart code from the proto file:

protoc \
  --dart_out=grpc:lib/generated \
  -I proto \
  proto/banking/banking.proto

The Data Source Layer

// lib/features/banking/data/datasources/banking_remote_datasource.dart

import 'package:grpc/grpc.dart';
import '../../../../generated/banking.pb.dart';
import '../../../../generated/banking.pbgrpc.dart';
import '../../../../core/error/app_exception.dart';

class BankingRemoteDataSource {
  late final BankingServiceClient _client;
  late final ClientChannel _channel;

  BankingRemoteDataSource({
    required String host,
    required int port,
    required String authToken,
  }) {
   
    _channel = ClientChannel(
      host,
      port: port,
      options: const ChannelOptions(
        credentials: ChannelCredentials.secure(),
        connectionTimeout: Duration(seconds: 10),
      ),
    );

   
    _client = BankingServiceClient(
      _channel,
      options: CallOptions(
        metadata: {'authorization': 'Bearer $authToken'},
        timeout: const Duration(seconds: 30),
      ),
    );
  }

  Future<LoginResponse> login(String email, String password) async {
    try {
      return await _client.login(
        LoginRequest(email: email, password: password),
      );
    } on GrpcError catch (e) {
      throw _mapGrpcError(e);
    }
  }

  Future<UserProfile> getProfile(String userId) async {
    try {
      return await _client.getProfile(
        ProfileRequest(userId: userId),
      );
    } on GrpcError catch (e) {
      throw _mapGrpcError(e);
    }
  }

  Stream<BalanceResponse> watchBalance(String userId) {
    return _client
        .watchBalance(BalanceRequest(userId: userId))
        .handleError((error) {
      if (error is GrpcError) throw _mapGrpcError(error);
      throw error;
    });
  }

  Stream<Transaction> streamTransactions(String userId, {int limit = 20}) {
    return _client
        .streamTransactions(
          TransactionRequest(userId: userId, limit: limit),
        )
        .handleError((error) {
      if (error is GrpcError) throw _mapGrpcError(error);
      throw error;
    });
  }

  Future<UploadResponse> uploadDocument(
    List<Uint8List> chunks,
    String documentType,
  ) async {
    try {
      Stream<DocumentChunk> chunkStream() async* {
        for (int i = 0; i < chunks.length; i++) {
          yield DocumentChunk(
            data: chunks[i],
            documentType: documentType,
            chunkIndex: i,
            isLast: i == chunks.length - 1,
          );
        }
      }
      return await _client.uploadDocument(chunkStream());
    } on GrpcError catch (e) {
      throw _mapGrpcError(e);
    }
  }

  ResponseStream<ChatMessage> startChat(Stream<ChatMessage> outgoing) {
    return _client.chat(outgoing);
  }

  Future<void> dispose() async {
    await _channel.shutdown();
  }

  AppException _mapGrpcError(GrpcError error) {
    switch (error.code) {
      case StatusCode.unauthenticated:
        return AppException.unauthorized(
          message: error.message ?? 'Unauthorized',
        );
      case StatusCode.notFound:
        return AppException.notFound(
          message: error.message ?? 'Not found',
        );
      case StatusCode.deadlineExceeded:
        return AppException.timeout(message: 'Request timed out');
      case StatusCode.unavailable:
        return AppException.serverUnavailable(
          message: 'Service unavailable',
        );
      default:
        return AppException.unknown(
          message: error.message ?? 'Unknown error',
        );
    }
  }
}

Let's walk through the important decisions in this data source.

The Channel:

_channel = ClientChannel(
  host,
  port: port,
  options: const ChannelOptions(
    credentials: ChannelCredentials.secure(),
    connectionTimeout: Duration(seconds: 10),
  ),
);

The channel is the physical HTTP/2 connection to the server. You create it once and reuse it for every call. ChannelCredentials.secure() enables TLS encryption. connectionTimeout prevents the app from waiting indefinitely if the server is unreachable.

The channel is the core of gRPC's performance advantage. A single persistent channel multiplexes all requests through one HTTP/2 connection. Creating a new channel per request would eliminate this advantage entirely and perform worse than REST.

Authentication via Metadata:

_client = BankingServiceClient(
  _channel,
  options: CallOptions(
    metadata: {'authorization': 'Bearer $authToken'},
    timeout: const Duration(seconds: 30),
  ),
);

gRPC uses metadata (key-value pairs) for what HTTP uses headers. Passing the auth token as metadata on the CallOptions means every single call made through this client automatically includes the Authorization metadata. You write it once. It applies everywhere.

Error Mapping:

AppException _mapGrpcError(GrpcError error) {
  switch (error.code) {
    case StatusCode.unauthenticated:
      return AppException.unauthorized(...);
    case StatusCode.deadlineExceeded:
      return AppException.timeout(...);
    ...
  }
}

gRPC has its own set of status codes similar to HTTP status codes but not identical. Mapping them to your application's exception types at the data source layer means the rest of your code (use cases, notifiers, widgets) never deals with gRPC-specific errors directly. Your domain layer stays clean and framework-independent.

The Repository Layer

abstract class BankingRepository {
  Future<Result<UserProfile, AppException>> getProfile(String userId);
  Stream<BalanceResponse> watchBalance(String userId);
  Stream<Transaction> streamTransactions(String userId);
  Future<Result<UploadResponse, AppException>> uploadDocument(
    List<Uint8List> chunks,
    String documentType,
  );
}


class BankingRepositoryImpl implements BankingRepository {
  final BankingRemoteDataSource _dataSource;

  BankingRepositoryImpl(this._dataSource);

  @override
  Future<Result<UserProfile, AppException>> getProfile(String userId) async {
    try {
      final profile = await _dataSource.getProfile(userId);
      return Result.success(profile);
    } on AppException catch (e) {
      return Result.failure(e);
    }
  }

  @override
  Stream<BalanceResponse> watchBalance(String userId) {
    return _dataSource.watchBalance(userId);
  }

  @override
  Stream<Transaction> streamTransactions(String userId) {
    return _dataSource.streamTransactions(userId);
  }

  @override
  Future<Result<UploadResponse, AppException>> uploadDocument(
    List<Uint8List> chunks,
    String documentType,
  ) async {
    try {
      final response = await _dataSource.uploadDocument(chunks, documentType);
      return Result.success(response);
    } on AppException catch (e) {
      return Result.failure(e);
    }
  }
}

The Riverpod Providers

part 'banking_providers.g.dart';

@riverpod
Stream<BalanceResponse> balanceStream(BalanceStreamRef ref, String userId) {
  final repository = ref.watch(bankingRepositoryProvider);
  return repository.watchBalance(userId);
}

@riverpod
Stream<Transaction> transactionStream(
  TransactionStreamRef ref,
  String userId,
) {
  final repository = ref.watch(bankingRepositoryProvider);
  return repository.streamTransactions(userId);
}

With Riverpod, a provider that returns a Stream automatically becomes an AsyncValue that widgets can watch. Every new value pushed from the gRPC server stream triggers a widget rebuild automatically.

The UI

// lib/features/banking/presentation/pages/dashboard_page.dart
class DashboardPage extends ConsumerWidget {
  final String userId;

  const DashboardPage({required this.userId, super.key});

  @override
  Widget build(BuildContext context, WidgetRef ref) {
    final balanceAsync = ref.watch(balanceStreamProvider(userId));
    final transactionsAsync = ref.watch(transactionStreamProvider(userId));

    return Scaffold(
      appBar: AppBar(title: const Text('Dashboard')),
      body: Column(
        children: [
          balanceAsync.when(
            data: (balance) => BalanceCard(
              amount: balance.balance,
              currency: balance.currency,
            ),
            loading: () => const BalanceShimmer(),
            error: (e, _) => ErrorCard(message: e.toString()),
          ),

          const SizedBox(height: 24),

          Expanded(
            child: transactionsAsync.when(
              data: (transaction) => TransactionTile(
                transaction: transaction,
              ),
              loading: () => const TransactionShimmer(),
              error: (e, _) => ErrorCard(message: e.toString()),
            ),
          ),
        ],
      ),
    );
  }
}

Both the balance and transactions arrive through gRPC server streams. Both update in real-time as the server pushes new data. Both are handled identically through Riverpod's AsyncValue pattern. One framework with all patterns covered.

Production Concerns

Authentication with Interceptors

For more granular authentication control, such as refreshing an expired token and retrying automatically, you implement a client interceptor:

class AuthInterceptor extends ClientInterceptor {
  final TokenService _tokenService;

  AuthInterceptor(this._tokenService);

  @override
  ResponseFuture<R> interceptUnary<Q, R>(
    ClientMethod<Q, R> method,
    Q request,
    CallOptions options,
    ClientUnaryInvoker<Q, R> invoker,
  ) {
    final token = _tokenService.currentToken;
    final authenticatedOptions = options.mergedWith(
      CallOptions(metadata: {'authorization': 'Bearer $token'}),
    );
    return invoker(method, request, authenticatedOptions);
  }

  @override
  ResponseStream<R> interceptServerStreaming<Q, R>(
    ClientMethod<Q, R> method,
    Q request,
    CallOptions options,
    ClientServerStreamingInvoker<Q, R> invoker,
  ) {
    final token = _tokenService.currentToken;
    final authenticatedOptions = options.mergedWith(
      CallOptions(metadata: {'authorization': 'Bearer $token'}),
    );
    return invoker(method, request, authenticatedOptions);
  }
}

Pass the interceptor when creating the client:

_client = BankingServiceClient(
  _channel,
  interceptors: [AuthInterceptor(tokenService)],
);

The interceptor fires on every call automatically. The data source code never touches auth logic directly.

Error Handling: gRPC Status Codes

gRPC defines a standard set of status codes that every implementation follows:

AppException _mapGrpcError(GrpcError error) {
  switch (error.code) {
    case StatusCode.unauthenticated:
      return AppException.unauthorized(message: 'Session expired');
    case StatusCode.permissionDenied:
      return AppException.forbidden(message: 'Access denied');
    case StatusCode.notFound:
      return AppException.notFound(message: error.message ?? 'Not found');
    case StatusCode.deadlineExceeded:
      return AppException.timeout(message: 'Request timed out');
    case StatusCode.unavailable:
      return AppException.serverUnavailable(message: 'Service unavailable');
    case StatusCode.resourceExhausted:
      return AppException.rateLimited(message: 'Too many requests');
    case StatusCode.invalidArgument:
      return AppException.validation(
        message: error.message ?? 'Invalid input',
      );
    default:
      return AppException.unknown(
        message: error.message ?? 'An error occurred',
      );
  }
}

Deadlines and Timeouts

Every gRPC call should have a deadline. Without deadlines, a slow server can make your app hang indefinitely.

Per-call deadline:

Future<UserProfile> getProfile(String userId) async {
  return await _client.getProfile(
    ProfileRequest(userId: userId),
    options: CallOptions(timeout: const Duration(seconds: 10)),
  );
}

Default deadline for all calls:

_client = BankingServiceClient(
  _channel,
  options: CallOptions(
    timeout: const Duration(seconds: 30),
    metadata: {'authorization': 'Bearer $authToken'},
  ),
);

When the deadline is exceeded, the call throws a GrpcError with StatusCode.deadlineExceeded, which your error mapper handles appropriately.

Logging Interceptor

class LoggingInterceptor extends ClientInterceptor {
  @override
  ResponseFuture<R> interceptUnary<Q, R>(
    ClientMethod<Q, R> method,
    Q request,
    CallOptions options,
    ClientUnaryInvoker<Q, R> invoker,
  ) {
    final stopwatch = Stopwatch()..start();
    debugPrint('[gRPC] --> ${method.path}');

    final response = invoker(method, request, options);

    response.then((_) {
      stopwatch.stop();
      debugPrint(
        '[gRPC] <-- ${method.path} (${stopwatch.elapsedMilliseconds}ms)',
      );
    }).catchError((error) {
      stopwatch.stop();
      debugPrint(
        '[gRPC] ERROR ${method.path}: $error (${stopwatch.elapsedMilliseconds}ms)',
      );
    });

    return response;
  }
}

gRPC vs REST vs WebSockets: When to Use What

When to Use REST

The API is consumed by third-party developers or external partners. JSON over HTTP is the universal language that every developer in every language can access immediately without learning new tooling.

The operation is simple request-response with no streaming requirements and only one client platform. REST is simpler to implement, simpler to debug, and simpler to test for straightforward CRUD operations.

Public documentation and human readability matter. REST with OpenAPI/Swagger gives you browsable, testable documentation that developers can explore in a browser.

Caching is important. REST GET responses can be cached at every layer: CDN, reverse proxy, browser cache. gRPC requests can't leverage standard HTTP caching.

Use WebSockets When

You need true bidirectional real-time communication and gRPC isn't already in your stack. Chat applications, multiplayer games, and collaborative tools where both sides need to speak freely are natural WebSocket use cases.

Browser support without a proxy layer is required. WebSockets work natively in every modern browser. gRPC in the browser requires gRPC-Web and a proxy layer.

When to Use gRPC

Multiple platform teams share the same service contract. When Flutter, React, Go, and Python services all call the same backend, a .proto file enforced by the compiler prevents contract drift across every team.

Large payloads are called by many internal applications. The more fields in the payload and the more applications consuming it, the stronger the case for protobuf's binary encoding and generated clients.

Network conditions are variable and payload size matters. Users on 2G or 3G connections benefit directly from protobuf's compact binary format. The same data in protobuf can be 3 to 10 times smaller than JSON, translating to faster load times and lower data consumption for users on limited data plans.

Real-time streaming is required and you want one unified framework for all communication patterns. gRPC's four patterns cover every scenario without requiring a separate WebSocket server alongside your API.

Service-to-service communication at high frequency is involved. Two internal services calling each other thousands of times per second over a persistent multiplexed HTTP/2 connection with binary protobuf encoding will significantly outperform REST with JSON over HTTP/1.1.

The Hybrid Architecture

The mature engineering decision is not choosing gRPC over REST or REST over gRPC. It's knowing where each belongs and using both deliberately.

Most organizations of meaningful scale end up with a hybrid:

The public REST API serves external consumers who need simplicity and JSON. The internal gRPC network handles high-frequency, high-performance service-to-service calls. The mobile gRPC endpoints give Flutter clients real-time capabilities over efficient binary connections.

Each layer uses the right tool for its specific requirements. No ideological commitment to one protocol. Pure engineering pragmatism.

Conclusion

Remote Procedure Calls began with a simple observation: network communication shouldn't require developers to think about network communication. Calling a function on another machine should feel like calling a function on your own.

Protocol Buffers took this further by solving the data problem. JSON is readable but verbose. Binary encoding with a compiler-enforced schema produces payloads that are smaller, faster to parse, and guaranteed to match the contract every team agreed on. For users on slow networks and internal systems processing millions of requests daily, this efficiency is a business advantage.

gRPC combined RPC semantics, Protocol Buffer encoding, and HTTP/2 transport into a framework that supports four distinct communication patterns: unary request-response, server streaming, client streaming, and bidirectional streaming. All from the same generated client, using the same persistent connection, and enforced by the same .proto contract.

The organizational practice of a shared protobuf repository transforms gRPC from a technical tool into an engineering discipline. Contract changes go through review. Breaking changes are caught by the compiler. Every team generates their own strongly typed client from the same source of truth and stays in sync automatically, regardless of programming language.

In Flutter specifically, gRPC server streams integrate naturally with Dart's Stream type and Riverpod's stream providers. Real-time balance updates and live transaction feeds that would require polling with REST or a separate WebSocket implementation become simple stream subscriptions. The server pushes, the widget listens, and nothing else is required.

The decision of when to use gRPC versus REST versus WebSockets isn't about preference. It's about matching the tool to the requirement. Public APIs belong behind REST. High-frequency internal service communication belongs on gRPC. Large payloads consumed by many internal systems belong in protobuf. Real-time bidirectional features belong on gRPC streaming. Users on variable networks deserve the smallest payloads you can give them.

Understanding all of these tools, understanding why they exist, and knowing when to reach for each one is what separates engineers who use tools from engineers who think in systems.

Happy Coding!

We'll implement core features like file replication, metadata management, and versioning, all while demonstrating how to achieve eventual consistency in a distributed environment. By the end, you'll have a fully functional distributed file storage system that can handle high traffic, optimize storage, and ensure data integrity.

What You Will Learn

• How to set up MinIO for distributed object storage.

• How to use gRPC for efficient client-server communication.

• How to implement file replication and metadata management.

• How to understand data consistency in a distributed system.

• How to use Docker to deploy a scalable, distributed architecture.

Prerequisites

Before starting, ensure you have the following installed:

• Node.js (v14 or higher)

• MinIO

• gRPC and gRPC-tools

• Docker

You’ll also need to have a basic understanding of Node.js, object storage, and distributed systems.

Table of Contents

• Project Overview

• Step 1: Setting Up the Project

• Step 2: Setting Up MinIO Distributed Storage Nodes

• Step 3: Defining the gRPC Protocol

• Step 4: Implementing the gRPC Server

• Step 5: Creating the Client

• Step 6: Running the System

• Conclusion: What You’ve Learned

Project Overview

We'll build a distributed file storage system where:

1. Users can upload and download files.

2. Files are replicated across multiple storage nodes to ensure high availability.

3. Metadata (like file names, upload times, and versions) is managed centrally.

4. The system handles eventual consistency by syncing file updates across nodes.

System Architecture

Our system will consist of:

1. gRPC Server: Manages file uploads, downloads, and metadata.

2. MinIO Distributed Storage Nodes: Handles object storage and replication.

3. Client Interface: Allows users to interact with the system via HTTP.

Step 1: Setting Up the Project

Create a new directory for the project and initialize a Node.js application:

mkdir distributed-file-storage
cd distributed-file-storage
npm init -y

Now, install the necessary dependencies:

npm install grpc @grpc/grpc-js @grpc/proto-loader express multer dotenv minio

• grpc: For building gRPC server and client.

• @grpc/proto-loader: Loads gRPC protocol files.

• express: For the client-side HTTP server.

• multer: For handling file uploads.

• dotenv: For managing environment variables.

• minio: MinIO client for interacting with storage nodes.

Create a .env file with the following content:

MINIO_ENDPOINT_1=localhost:9001
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin
PORT=5000

Step 2: Setting Up MinIO Distributed Storage Nodes

We'll use Docker to run multiple MinIO instances, simulating a distributed environment. Run the following commands to set up three MinIO containers:

docker run -p 9001:9000 --name minio1 -e "MINIO_ACCESS_KEY=minioadmin" -e "MINIO_SECRET_KEY=minioadmin" -d minio/minio server /data
docker run -p 9002:9000 --name minio2 -e "MINIO_ACCESS_KEY=minioadmin" -e "MINIO_SECRET_KEY=minioadmin" -d minio/minio server /data
docker run -p 9003:9000 --name minio3 -e "MINIO_ACCESS_KEY=minioadmin" -e "MINIO_SECRET_KEY=minioadmin" -d minio/minio server /data

These commands will start three MinIO nodes, each listening on a different port.

Step 3: Defining the gRPC Protocol

Create a new folder named protos and inside it, create a file called storage.proto:

syntax = "proto3";

service FileStorage {
  rpc UploadFile(stream FileRequest) returns (UploadResponse);
  rpc DownloadFile(FileDownloadRequest) returns (stream FileResponse);
  rpc GetMetadata(FileMetadataRequest) returns (MetadataResponse);
}

message FileRequest {
  bytes fileData = 1;
  string fileName = 2;
}

message UploadResponse {
  string message = 1;
}

message FileDownloadRequest {
  string fileName = 1;
}

message FileResponse {
  bytes fileData = 1;
}

message FileMetadataRequest {
  string fileName = 1;
}

message MetadataResponse {
  string fileName = 1;
  string uploadTime = 2;
  string version = 3;
}

• UploadFile: Streams file data from the client to the server.

• DownloadFile: Streams file data from the server to the client.

• GetMetadata: Retrieves metadata like file name, upload time, and version.

Step 4: Implementing the gRPC Server

Create a file called server.js:

require('dotenv').config();
const grpc = require('@grpc/grpc-js');
const protoLoader = require('@grpc/proto-loader');
const Minio = require('minio');
const fs = require('fs');
const path = require('path');

const packageDefinition = protoLoader.loadSync('protos/storage.proto');
const storageProto = grpc.loadPackageDefinition(packageDefinition).FileStorage;

// Set up MinIO clients for each node
const minioClients = [
  new Minio.Client({
    endPoint: process.env.MINIO_ENDPOINT_1.split(':')[0],
    port: parseInt(process.env.MINIO_ENDPOINT_1.split(':')[1]),
    accessKey: process.env.MINIO_ACCESS_KEY,
    secretKey: process.env.MINIO_SECRET_KEY,
    useSSL: false,
  })
];

// Upload file to MinIO
async function uploadFile(call, callback) {
  const chunks = [];
  call.on('data', (chunk) => chunks.push(chunk.fileData));
  call.on('end', async () => {
    const buffer = Buffer.concat(chunks);
    const fileName = call.metadata.get('fileName')[0];

    // Store file in MinIO
    const client = minioClients[0];
    await client.putObject('files', fileName, buffer);
    callback(null, { message: `File ${fileName} uploaded successfully` });
  });
}

// Download file from MinIO
function downloadFile(call) {
  const { fileName } = call.request;
  const client = minioClients[0];

  client.getObject('files', fileName, (err, stream) => {
    if (err) return call.emit('error', err);
    stream.on('data', (chunk) => call.write({ fileData: chunk }));
    stream.on('end', () => call.end());
  });
}

function main() {
  const server = new grpc.Server();
  server.addService(storageProto.FileStorage.service, { uploadFile, downloadFile });
  server.bindAsync('0.0.0.0:5000', grpc.ServerCredentials.createInsecure(), () => {
    console.log('gRPC server running on port 5000');
    server.start();
  });
}

main();

Here’s what’s going on in this code:

1. uploadFile: Handles file uploads by streaming data to the server and storing it in MinIO.

2. downloadFile: Streams the requested file back to the client from MinIO.

3. MinIO Clients: We set up multiple MinIO clients to handle distributed storage.

Step 5: Creating the Client

Create a file named client.js:

const grpc = require('@grpc/grpc-js');
const protoLoader = require('@grpc/proto-loader');
const fs = require('fs');

const packageDefinition = protoLoader.loadSync('protos/storage.proto');
const storageProto = grpc.loadPackageDefinition(packageDefinition).FileStorage;
const client = new storageProto('localhost:5000', grpc.credentials.createInsecure());

function uploadFile(filePath) {
  const call = client.uploadFile();
  const fileName = filePath.split('/').pop();
  const stream = fs.createReadStream(filePath);

  stream.on('data', (chunk) => call.write({ fileData: chunk }));
  stream.on('end', () => call.end());
  call.on('data', (response) => console.log(response.message));
}

function downloadFile(fileName) {
  const call = client.downloadFile({ fileName });
  const writeStream = fs.createWriteStream(`downloaded_${fileName}`);

  call.on('data', (chunk) => writeStream.write(chunk.fileData));
  call.on('end', () => console.log(`Downloaded ${fileName}`));
}

uploadFile('test.txt');  // Example usage

Step 6: Running the System

1. Start the gRPC Server:

    node server.js
   

2. Run the Client:

    node client.js
   

Conclusion: What You’ve Learned

Congratulations! You've built a distributed file storage system using MinIO and gRPC. In this tutorial, you learned how to:

1. Set up a distributed object storage system using MinIO.

2. Use gRPC to handle file uploads, downloads, and metadata management.

3. Implement file replication and eventual consistency across multiple nodes.

4. Utilize Docker to simulate a scalable distributed environment.

Next Steps:

1. Add File Versioning: Store multiple versions of files for rollback.

2. Implement Authentication: Secure your gRPC endpoints with JWT.

3. Deploy with Kubernetes: Scale your system across multiple nodes for high availability.

Happy coding!