If you use or have used SharedPreferences in your applications, you might be thinking of making the switch. But as with everything, the main question here is: what is going to be the cost in development?

There are benefits for using DataStore, but only the Proto DataStore allows you to save objects while providing type safety.

If you look at the documentation for Proto DataStore, you will find that it is a bit outdated and missing some crucial steps when working with it. So that is why, in this article, we are going to go over how to integrate Proto DataStore into your application and show that it’s not that big of a hassle to use it.

What is DataStore?

Jetpack DataStore has two variants:

• Preferences DataStore
• Proto DataStore

We won’t be discussing the first, due to it’s similarity to SharedPreferences and also the fact that is has been covered widely. So now let’s understand what the Proto in Proto DataStore means.

Proto is the name Google chose to represent Protocol Buffers. These are (Google’s) mechanism that help you serialize structured data. They are not coding language-specific and in general, you define the type of data that you wish to work with and then code is generated that helps you read and write your data.

✋ We will be using the Proto 3 version in this article.

How does that definition look like?

message MyItem {
    string itemName = 1;
    int32 itemId = 2;
}

First, you define an object with the message keyword. Inside it, you list the fields associated with that object. The numbers at the end of each field are used to identify the field itself and cannot be changed once being set and the object is in use.

But what if we wanted to have multiple objects in our .proto file? Assuming the objects are related to one another, you can do this simply by adding more message objects:

message MyItem {
    string itemName = 1;
    int32 itemId = 2;
}

message MyListOfItems {
   repeated MyItem items = 1;
}

Notice that above we have added another message object that relies on the MyItem object defined above. If you want to define a list of objects, you need to use the repeated keyword.

How to Set Up Proto DataStore

To get started, you'll need to add the following dependencies to your application level build.gradle:

 implementation "androidx.datastore:datastore-preferences:1.0.0"
 implementation  "com.google.protobuf:protobuf-javalite:3.18.0"

Then, you will need to create a proto directory inside your project. This directory needs to be a sibling of the Java folder in your project structure.

Inside of the proto directory, you will be creating a .proto file. This file is responsible for generating the data types you wish to store in Proto DataStore.

Inside the proto directory, create a file with the .proto extension. Our .proto file will hold objects representing a Todo list (what else?). So we will call our file todo.proto and it will look like this:

syntax = "proto3";

option java_package = "com.yourPackageName.todo";
option java_multiple_files = true;

message TodoItem {
  string itemId = 1;
  string itemDescription = 2;
}

message TodoItems {
  repeated TodoItem items = 1;
}

Notice how we defined two message objects:

1. TodoItem – that defines a todo item
2. TodoItems – that defines a list of TodoItem objects

Next, build the project so that classes will be generated for TodoItem and TodoItems.

After our data objects have been defined, we need to create a class to serialize them. This class will tell the DataStore how to read/write our objects.

// 1
object TodoItemSerializer: Serializer<TodoItems> {
   // 2
    override val defaultValue: TodoItems = TodoItems.getDefaultInstance()
    // 3
    override suspend fun readFrom(input: InputStream): TodoItems {
        try {
            return TodoItems.parseFrom(input)
        } catch (exception: InvalidProtocolBufferException) {
            throw CorruptionException("Cannot read proto.", exception)
        }
    }
    // 3
    override suspend fun writeTo(
        t: TodoItems,
        output: OutputStream
    ) = t.writeTo(output)
}

Let’s review what we have in this class:

1. When we declare the class, we need to implement the Serializer interface with our object as the type (T)
2. We define a default value for the serializer in case the file is not created
3. We override the readFrom/writeTo methods and we make sure to have our object as the data type there

We have our .proto file with our data types and our serializer, so the next step is to instantiate the DataStore. We do this by using the property delegate created by dataStore, which requires giving a filename where our data will be saved and our serializer class (which we defined above).

private const val DATA_STORE_FILE_NAME = "todo.pb"

private val Context.todoItemDatastore: DataStore<TodoItems> by dataStore(
    fileName = DATA_STORE_FILE_NAME,
    serializer = TodoItemSerializer,
)

This piece of code needs to reside at the top of a class of your choosing above the definition of the class itself. That is:

private const val DATA_STORE_FILE_NAME = "todo.pb"

private val Context.todoItemDatastore: DataStore<TodoItems> by dataStore(
    fileName = DATA_STORE_FILE_NAME,
    serializer = TodoItemSerializer,
)

class YourClassName {

}

To access this object in the rest of our application, we will need to use a context. An example is to use the application context in your viewmodel class:

class MyViewModel(application: Application): AndroidViewModel(application) {

   val todoDataStore = application.todoItemDataStore
   //...
}

How to Use Kotlin Flow

Now that we have gone through setting up everything we need for our DataStore, we'll discuss how we are actually going to interact with it. We'll want to read and write data to/from it. But the way we can do so is different from what you may be familiar with from SharedPreferences.

The DataStore we defined above has a data field that exposes a Flow for the properties we defined in our DataStore.

🚰 If you are not familiar with flows, this is a good place to start.

val todoItemFlow: Flow<TodoItems> = todoItemDataStore.data
        .catch { exception ->
            if (exception is IOException) {
                emit(TodoItems.getDefaultInstance())
            } else {
                throw exception
            }
        }

The code above shows how you can define a Flow that collects data from the Proto DataStore. A catch block was added in case an exception occurs. You can place this logic in the class where you defined your DataStore and use it like so in your viewmodel:

val todoItemsFlow: LiveData<TodoItems> = todoItemsRepository.todoItemFlow.asLiveData()

Notice how we converted our Flow to LiveData. We did this for two reasons:

1. Flows can stay active regardless of the activity/fragment that uses them
2. LiveData is something familiar to many developers, and I wanted to make this example as approachable as possible

To be able to do this, you need to add the following dependency to your build.gradle file:

implementation "androidx.lifecycle:lifecycle-livedata-ktx:2.6.2"

In your activity/fragment class, you can observe this live data like so:

myViewModel.todoItemFlow.observe(LocalLifecycleOwner.current) { todoItems ->
                // Logic to access data from DataStore
            }

Why and When to Use DataStore

After everything we reviewed, it’s time to talk about the elephant in the room. Should you go ahead and use DataStore (either Preferences or Proto) in your existing or next project?

In my opinion, the answer should be Yes. Besides the fact that Google is moving away from SharedPreferences, DataStore offers plenty of benefits to help you focus on your application and not the persistence of your data.

It’s safe to interact with the DataStore from the UI thread (as it moves work to I/O automatically), and it forces you to use Flow (if you haven’t still) and enjoy all the benefits within. There is also an option to migrate easily from SharedPreferences to Preferences DataStore.

If you are contemplating using Room instead of Proto DataStore, well that depends on your use case. If the amount of data you are going to save (or persist) is rather small and won’t require partial updating, the Proto DataStore is the way to go. If you have a larger data set or one that may be complex, you should opt for using Room instead.

If you want to see how all this code looks like in an application, you can see it here:

If you want to read other articles I have written, you can see them here:

Thanks for reading!

References:

• Protocol Buffers Documentation (proto 3)
• Working With Proto DataStore Codelab
• DataStore Documentation

gRPC is a powerful framework for working with Remote Procedure Calls. RPCs allow you to write code as though it will be run on a local computer, even though it may be executed on another computer.

These past few days I have been diving deep into gRPC. I'm going to share some of my big discoveries here in this article.

Note that I will focus more on concepts than implementation details. You will learn the core architecture of gRPC itself. You'll also learn:

• Why gRPC is so widely used by developers
• How it performs so well
• And how it all works under the hood.

Let’s go back a bit

Before we rush into gRPC, we should take a look at what Remote Procedure Calls are.

A RPC is a form of Client-Server Communication that uses a function call rather than a usual HTTP call.

It uses IDL (Interface Definition Language) as a form of contract on functions to be called and on the data type.

RPC Architecture

If you all haven’t realized it yet, the RPC in gRPC stands for Remote Procedure Call. And yes, gRPC does replicate this architectural style of client server communication, via function calls.

So gRPC is technically not a new concept. Rather it was adopted from this old technique and improved upon, making it very popular in just the span of 5 years.

Overview of gRPC

In 2015, Google open sourced their project which eventually would be the one called gRPC. But what does the "g" in gRPC actually stand for?

Lots of people might assume its for Google because Google made it, but it does not.

Google changes the meaning of the "g" for each version to the point where they even made a README to list all the meanings.

Since gRPC has been introduced it has gained quite a bit of popularity and many companies use it.

What makes gRPC so popular?

There are plenty of reasons why gRPC is so popular:

• Abstraction is easy (it’s a function call)
• It is supported in a lot of languages
• It is very performant
• HTTP calls are often confusing, so this makes it easier

And aside from all of the reasons above, gRPC is popular because microservices are very popular.

Microservices will often be running several services in different programming languages. They'll also often have a lot of service to service interactions.

This is where gRPC helps out the most by providing the support and capability to solve the typical issues that arise from those situations.

Microservices

gRPC is very popular in service to service calls, as often HTTP calls are harder to understand at first glance.

gRPC functions are much easier to reason about, so developers don't have to worry about writing a lot of documentation because the code itself should explain everything.

Some of the services might also be written in different languages and gRPC comes with multiple libraries to support that.

Performance is the cherry on top – and it’s a big cherry.

gRPC Architecture

The rough architecture of gRPC. It's more or less the same as regular RPC.

I have mentioned several times that gRPC's performance is very good, but you might wonder what makes it so good? What makes gRPC so much better than RPC when their designs are pretty similar?

Here are a few key differences that make gRPC so performant.

HTTP/2

HTTP has been with us for a long time. Now, almost all backend services use this protocol.

History of HTTP

As the picture above shows, HTTP/1.1 stayed relevant for a long time.

Then in 2015, HTTP/2 came out and essentially replaced HTTP/1.1 as the most popular transport protocol on the internet.

If you remember that 2015 was also the year that gRPC came out, it was not a coincidence at all. HTTP/2 was also created by Google to be used by gRPC in its architecture.

HTTP/2 is one of the big reasons why gRPC can perform so well. And in this next section, you'll see why.

Request/Response Multiplexing

In a traditional HTTP protocol, is not possible to send multiple requests or get multiple responses together in a single connection. A new connection will need to be created for each of them.

This kind of request/response multiplexing is made possible in HTTP/2 with the introduction of a new HTTP/2 layer called binary framing.

This binary layer encapsulates and encodes the data. In this layer, the HTTP request/response gets broken down into frames.

The headers frame contains typical HTTP headers information, and the data frame contains the payload. Using this mechanism, it's possible to have data from multiple requests in a single connection.

This allows payloads from multiple requests with the same header, thus identifying it as a single request.

Header Compression

You might have encountered many cases where HTTP headers are even bigger than the payload. And HTTP/2 has a very interesting strategy called HPack to handle that.

For one, everything in HTTP/2 is encoded before it's sent, including the headers. This does help with performance, but that’s not the most important thing about header compression.

HTTP/2 maps the header on both the client and the server side. From that, HTTP/2 is able to know if the header contains the same value and only sends the header value if it is different from the previous header.

As seen in the picture above, Request #2 will only send the path since the other values are exactly the same. And yes, this does cut down a lot on the payload size, and in turn, improves HTTP/2's performance even more.

Protocol Buffer, a.k.a. Protobuf

Protocol Buffer

Protobuf is the most commonly used IDL (Interface Definition Language) for gRPC. It's where you basically store your data and function contracts in the form of a proto file.

message Person {
    required string name = 1;
    required int32 id = 2;
    optional string email = 3;
}

As this is in the form of a contract, both the client and server need to have the same proto file. The proto file acts as the intermediary contract for client to call any available functions from the server.

Protobuf also has it owns mechanisms, unlike a usual REST API that just sends over strings of JSON as bytes. These mechanisms allow the payload to be much smaller and enable faster performance.

The encoding method Protobuf uses is pretty complicated. If you want to take a deep dive into how it works, check out this comprehensive documentation.

What else does gRPC offer?

_Photo by [Unsplash](https://unsplash.com/@kyledevaras?utm_source=ghost&utm_medium=referral&utm_campaign=api-credit">Kyle Gregory Devaras / <a href="https://unsplash.com/?utm_source=ghost&utm_medium=referral&utmcampaign=api-credit)

Now you should have a basic understanding of the architecture of gRPC, how it works, and what it's capable of.

But here are a few other interesting things gRPC offers us.

Metadata

Instead of using a usual HTTP request header, gRPC has something called metadata. Metadata is a type of key-value data that can be set from either the client or server side.

Header can be assigned from the client side, while servers can assign Header and Trailers so long as they're both in the form of metadata.

Streaming

Streaming is one of the core concepts of gRPC where several things can happen in a single request. This is made possible by the multiplexing capability of HTTP/2 mentioned earlier.

There are several types of streaming:

• Server Streaming RPC: Where the client sends a single request and the server can send back multiple responses. For example, when a client sends a request for a homepage that has a list of multiple items, the server can send back responses separately, enabling the client to use lazy loading.
• Client Streaming RPC: Where the client sends multiple requests and the server only sends back a single response. For example, a zip/chunk uploaded by the client.
• Bidirectional Streaming RPC: Where both the client and server send messages to each other at the same time without waiting for a response.

Interceptors

gRPC supports the usage of interceptors for its request/response. Interceptors, well, intercept messages and allow you to modify them.

Does this sound familiar? If you have played around with HTTP processes on a REST API, interceptors are very similar to middleware.

gRPC libraries usually support interceptors, and allow for easy implementation. Interceptors are usually used to:

• Modify the request/response before being passed on. It can be used to provide mandatory information before being sent to the client/server.
• Allow you to manipulate each function call such as adding additional logging to track response time.

Load Balancing

If you aren't already familiar with load balancing, it's a mechanism that allows client requests to be spread out across multiple servers.

But load balancing is usually done at the proxy level (for example, NGINX). So why am I talking about it here?

Turns out that gRPC supports a method of load balancing by the client. It's already implemented in the Golang library, and can be used with ease.

While it might seem like some sort of crazy magic, it's not. There's some sort of DNS resolver to get an IP list, and a load balancing algorithm under the hood.

Call Cancellation

gRPC clients are able to cancel a gRPC call when it doesn't need a response anymore. Rollback on the server side is not possible, though.

This feature is especially useful for server side streaming where multiple server requests might be coming. The gRPC library comes equipped with an observer method pattern to know if a request is cancelled and allow it to cancel multiple corresponding requests at once.

Wrapping Up

_Photo by [Unsplash](https://unsplash.com/@rcrazy?utm_source=ghost&utm_medium=referral&utm_campaign=api-credit">Ricardo Rocha / <a href="https://unsplash.com/?utm_source=ghost&utm_medium=referral&utmcampaign=api-credit)

Everything I shared today just scratches the surface of what gRPC is, what it's capable of, and roughly how it works.

I truly hope this article helped you understand more about gRPC. But there's still a lot more learn, so don't stop here! Keep digging.

Thanks for reading!

When people who speak different languages get together and talk, they try to use a language that everyone in the group understands.

To achieve this, everyone has to translate their thoughts, which are usually in their native language, into the language of the group. This “encoding and decoding” of language, however, leads to a loss of efficiency, speed, and precision.

The same concept is present in computer systems and their components. Why should we send data in XML, JSON, or any other human-readable format if there is no need for us to understand what they are talking about directly? As long as we can still translate it into a human-readable format if explicitly needed.

Protocol Buffers are a way to encode data before transportation, which efficiently shrinks data blocks and therefore increases speed when sending it. It abstracts data into a language- and platform-neutral format.

Table of Contents

• Why do we need Protocol Buffers?
• What are Protocol Buffers and how do they work?
• Protocol Buffers in Python
• Final notes

Why Protocol Buffers?

The initial purpose of Protocol Buffers was to simplify the work with request/response protocols. Before ProtoBuf, Google used a different format which required additional handling of marshaling for the messages sent.

In addition to that, new versions of the previous format required the developers to make sure that new versions are understood before replacing old ones, making it a hassle to work with.

This overhead motivated Google to design an interface that solves precisely those problems.

ProtoBuf allows changes to the protocol to be introduced without breaking compatibility. Also, servers can pass around the data and execute read operations on the data without modifying its content.

Since the format is somewhat self-describing, ProtoBuf is used as a base for automatic code generation for Serializers and Deserializers.

Another interesting use case is how Google uses it for short-lived Remote Procedure Calls (RPC) and to persistently store data in Bigtable. Due to their specific use case, they integrated RPC interfaces into ProtoBuf. This allows for quick and straightforward code stub generation that can be used as starting points for the actual implementation. (More on ProtoBuf RPC.)

Other examples of where ProtoBuf can be useful are for IoT devices that are connected through mobile networks in which the amount of sent data has to be kept small or for applications in countries where high bandwidths are still rare. Sending payloads in optimized, binary formats can lead to noticeable differences in operation cost and speed.

Using gzip compression in your HTTPS communication can further improve those metrics.

What are Protocol buffers and how do they work?

Generally speaking, Protocol Buffers are a defined interface for the serialization of structured data. It defines a normalized way to communicate, utterly independent of languages and platforms.

Google advertises its ProtoBuf like this:

Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once …

The ProtoBuf interface describes the structure of the data to be sent. Payload structures are defined as “messages” in what is called Proto-Files. Those files always end with a .proto extension.

For example, the basic structure of a todolist.proto file looks like this. We will also look at a complete example in the next section.

syntax = "proto3";

// Not necessary for Python, should still be declared to avoid name collisions 
// in the Protocol Buffers namespace and non-Python languages
package protoblog;

message TodoList {
   // Elements of the todo list will be defined here
   ...
}

Those files are then used to generate integration classes or stubs for the language of your choice using code generators within the protoc compiler. The current version, Proto3, already supports all the major programming languages. The community supports many more in third-party open-source implementations.

Generated classes are the core elements of Protocol Buffers. They allow the creation of elements by instantiating new messages, based on the .proto files, which are then used for serialization. We’ll look at how this is done with Python in detail in the next section.

Independent of the language for serialization, the messages are serialized into a non-self-describing, binary format that is pretty useless without the initial structure definition.

The binary data can then be stored, sent over the network, and used any other way human-readable data like JSON or XML is. After transmission or storage, the byte-stream can be deserialized and restored using any language-specific, compiled protobuf class we generate from the .proto file.

Using Python as an example, the process could look something like this:

First, we create a new todo list and fill it with some tasks. This todo list is then serialized and sent over the network, saved in a file, or persistently stored in a database.

The sent byte stream is deserialized using the parse method of our language-specific, compiled class.

Most current architectures and infrastructures, especially microservices, are based on REST, WebSockets, or GraphQL communication. However, when speed and efficiency are essential, low-level RPCs can make a huge difference.

Instead of high overhead protocols, we can use a fast and compact way to move data between the different entities into our service without wasting many resources.

But why isn’t it used everywhere yet?

Protocol Buffers are a bit more complicated than other, human-readable formats. This makes them comparably harder to debug and integrate into your applications.

Iteration times in engineering also tend to increase since updates in the data require updating the proto files before usage.

Careful considerations have to be made since ProtoBuf might be an over-engineered solution in many cases.

What alternatives do I have?

Several projects take a similar approach to Google’s Protocol Buffers.

Google’s Flatbuffers and a third party implementation, called Cap’n Proto, are more focused on removing the parsing and unpacking step, which is necessary to access the actual data when using ProtoBufs. They have been designed explicitly for performance-critical applications, making them even faster and more memory efficient than ProtoBuf.

When focusing on the RPC capabilities of ProtoBuf (used with gRPC), there are projects from other large companies like Facebook (Apache Thrift) or Microsoft (Bond protocols) that can offer alternatives.

Python and Protocol Buffers

Python already provides some ways of data persistence using pickling. Pickling is useful in Python-only applications. It's not well suited for more complex scenarios where data sharing with other languages or changing schemas is involved.

Protocol Buffers, in contrast, are developed for exactly those scenarios.
The .proto files, we’ve quickly covered before, allow the user to generate code for many supported languages.

To compile the .proto file to the language class of our choice, we use protoc, the proto compiler.

If you don’t have the protoc compiler installed, there are excellent guides on how to do that:

• MacOS / Linux
• Windows

Once we’ve installed protoc on our system, we can use an extended example of our todo list structure from before and generate the Python integration class from it.

syntax = "proto3";

// Not necessary for Python but should still be declared to avoid name collisions 
// in the Protocol Buffers namespace and non-Python languages
package protoblog;

// Style guide prefers prefixing enum values instead of surrounding
// with an enclosing message
enum TaskState {
    TASK_OPEN = 0;
    TASK_IN_PROGRESS = 1;
    TASK_POST_PONED = 2;
    TASK_CLOSED = 3;
    TASK_DONE = 4;
}

message TodoList {
    int32 owner_id = 1;
    string owner_name = 2;

    message ListItems {
        TaskState state = 1;
        string task = 2;
        string due_date = 3;
    }

    repeated ListItems todos = 3;
}

Let’s take a more detailed look at the structure of the .proto file to understand it.
In the first line of the proto file, we define whether we’re using Proto2 or 3. In this case, we’re using Proto3.

The most uncommon elements of proto files are the numbers assigned to each entity of a message. Those dedicated numbers make each attribute unique and are used to identify the assigned fields in the binary encoded output.

One important concept to grasp is that only values 1-15 are encoded with one less byte (Hex), which is useful to understand so we can assign higher numbers to the less frequently used entities. The numbers define neither the order of encoding nor the position of the given attribute in the encoded message.

The package definition helps prevent name clashes. In Python, packages are defined by their directory. Therefore providing a package attribute doesn’t have any effect on the generated Python code.

Please note that this should still be declared to avoid protocol buffer related name collisions and for other languages like Java.

Enumerations are simple listings of possible values for a given variable.
In this case, we define an Enum for the possible states of each task on the todo list.
We’ll see how to use them in a bit when we look at the usage in Python.

As we can see in the example, we can also nest messages inside messages.
If we, for example, want to have a list of todos associated with a given todo list, we can use the repeated keyword, which is comparable to dynamically sized arrays.

To generate usable integration code, we use the proto compiler which compiles a given .proto file into language-specific integration classes. In our case we use the --python-out argument to generate Python-specific code.

protoc -I=. --python_out=. ./todolist.proto

In the terminal, we invoke the protocol compiler with three parameters:

1. -I: defines the directory where we search for any dependencies (we use . which is the current directory)
2. --python_out: defines the location we want to generate a Python integration class in (again we use . which is the current directory)
3. The last unnamed parameter defines the .proto file that will be compiled (we use the todolist.proto file in the current directory)

This creates a new Python file called _pb2.py. In our case, it is todolist_pb2.py. When taking a closer look at this file, we won’t be able to understand much about its structure immediately.

This is because the generator doesn’t produce direct data access elements, but further abstracts away the complexity using metaclasses and descriptors for each attribute. They describe how a class behaves instead of each instance of that class.

The more exciting part is how to use this generated code to create, build, and serialize data. A straightforward integration done with our recently generated class is seen in the following:

import todolist_pb2 as TodoList

my_list = TodoList.TodoList()
my_list.owner_id = 1234
my_list.owner_name = "Tim"

first_item = my_list.todos.add()
first_item.state = TodoList.TaskState.Value("TASK_DONE")
first_item.task = "Test ProtoBuf for Python"
first_item.due_date = "31.10.2019"

print(my_list)

It merely creates a new todo list and adds one item to it. We then print the todo list element itself and can see the non-binary, non-serialized version of the data we just defined in our script.

owner_id: 1234
owner_name: "Tim"
todos {
  state: TASK_DONE
  task: "Test ProtoBuf for Python"
  due_date: "31.10.2019"
}

Each Protocol Buffer class has methods for reading and writing messages using a Protocol Buffer-specific encoding, that encodes messages into binary format.
Those two methods are SerializeToString() and ParseFromString().

import todolist_pb2 as TodoList

my_list = TodoList.TodoList()
my_list.owner_id = 1234

# ...

with open("./serializedFile", "wb") as fd:
    fd.write(my_list.SerializeToString())


my_list = TodoList.TodoList()
with open("./serializedFile", "rb") as fd:
    my_list.ParseFromString(fd.read())

print(my_list)

In the code example above, we write the Serialized string of bytes into a file using the wb flags.

Since we have already written the file, we can read back the content and Parse it using ParseFromString. ParseFromString calls on a new instance of our Serialized class using the rb flags and parses it.

If we serialize this message and print it in the console, we get the byte representation which looks like this.

b'\x08\xd2\t\x12\x03Tim\x1a(\x08\x04\x12\x18Test ProtoBuf for Python\x1a\n31.10.2019'

Note the b in front of the quotes. This indicates that the following string is composed of byte octets in Python.

If we directly compare this to, e.g., XML, we can see the impact ProtoBuf serialization has on the size.

<todolist>
    <owner_id>1234</owner_id>
    <owner_name>Tim</owner_name>
    <todos>
        <todo>
            <state>TASK_DONE</state>
            <task>Test ProtoBuf for Python</task>
            <due_date>31.10.2019</due_date>
        </todo>
    </todos>
</todolist>

The JSON representation, non-uglified, would look like this.

{
    "todoList": {
        "ownerId": "1234",
        "ownerName": "Tim",
        "todos": [
            {
                "state": "TASK_DONE",
                "task": "Test ProtoBuf for Python",
                "dueDate": "31.10.2019"
            }
        ] 
    }
}

Judging the different formats only by the total number of bytes used, ignoring the memory needed for the overhead of formatting it, we can of course see the difference.

But in addition to the memory used for the data, we also have 12 extra bytes in ProtoBuf for formatting serialized data. Comparing that to XML, we have 171 extra bytes in XML for formatting serialized data.

Without Schema, we need 136 extra bytes in JSON for formatting serialized data.

If we’re talking about several thousands of messages sent over the network or stored on disk, ProtoBuf can make a difference.

However, there is a catch. The platform Auth0.com created an extensive comparison between ProtoBuf and JSON. It shows that, when compressed, the size difference between the two can be marginal (only around 9%).

If you’re interested in the exact numbers, please refer to the full article, which gives a detailed analysis of several factors like size and speed.

An interesting side note is that each data type has a default value. If attributes are not assigned or changed, they will maintain the default values. In our case, if we don’t change the TaskState of a ListItem, it has the state of “TASK_OPEN” by default. The significant advantage of this is that non-set values are not serialized, saving additional space.

If we, for example, change the state of our task from TASK_DONE to TASK_OPEN, it will not be serialized.

owner_id: 1234
owner_name: "Tim"
todos {
  task: "Test ProtoBuf for Python"
  due_date: "31.10.2019"
}

b'\x08\xd2\t\x12\x03Tim\x1a&\x12\x18Test ProtoBuf for Python\x1a\n31.10.2019'

Final Notes

As we have seen, Protocol Buffers are quite handy when it comes to speed and efficiency when working with data. Due to its powerful nature, it can take some time to get used to the ProtoBuf system, even though the syntax for defining new messages is straightforward.

As a last note, I want to point out that there were/are discussions going on about whether Protocol Buffers are “useful” for regular applications. They were developed explicitly for problems Google had in mind.

If you have any questions or feedback, feel free to reach out to me on any social media like twitter or email :)

This post is originally from www.jaredwolff.com

In Part 1 we’ve learned the anatomy of a Protocol Buffer. In Part 2 we’ve learned how a Bluetooth Low Energy Service gets pieced together on the Nordic NRF52 SDK. This final post brings together all the elements in an end-to-end working example.

Let’s get going!

P.S. this post is lengthy. If you want something to download, click here for a a beautifully formatted PDF. (Added bonus, the PDF has all three parts of this series!)

Setting Everything Up

Go look at the Readme.md in each of the example repositories. (You will have to clone the repositories, sign up here to get the code).

I’ve made the setup for the firmware dead simple. For the javascript side, it’s a bit more hairy but’ doable! I’ll also include the instructions here as well:

Firmware Setup (For Mac)

1. Initialize the full repository (there are submodules!): git submodule update --init
2. Install protoc using Homebrew: brew install protobuf
3. Run make sdk. This will download your SDK files.
4. Run make tools_osx. This will download your ARMGCC toolchain (for Mac). For other environments see below.
5. Run make gen_key once (and only once)! This will set up your key for DFU.
6. Run make and this will build your bootloader and main app.

Note: You only have to do steps 1-5 once.

Javascript App Setup (For Mac)

Prerequisite: you will need Xcode command line tools. You can get those here.

1. Clone this repo to a place on your computer
2. Make sure you have nvm installed
3. Run nvm install v8.0.0
4. Run nvm install v10.15.3
5. Run nvm use v8.0.0
6. Run yarn (if you don’t have yarn npm install yarn -g)
7. Once installed, run nvm use v10.15.3
8. Then run node index.js to start the example

Using NVM helps mitigate a compile issue with the Bluetooth library. You mileage may vary on this one.

How the heck does this work?

In this project, the Protocol Buffer has two functions. The first as a “command” to the bluetooth device. Secondly, a “response” to that command from the device.

Our example javascript app command uses “This is” as the payload. With the power of some string operations, let's make a full sentence out of it!

The series of events looks something like this:

1. The test app connects and sends that data using our Bluetooth Low Energy service.
2. On the firmware decodes the message.
3. The firmware modifies the original data by adding “ cool.” Resulting in "This is cool"
4. The firmware encodes the payload and makes the data available for reading.
5. The app finally reads the characteristic, decodes and displays the result!

Seems complicated but there's some benefits to this:

1. In Protocol Buffers, data structures are well defined. This means that it has some great error checking abilities. If you're using any type of data structure you may have to code your own data validation.
2. Encoding and decoding is simple and straight forward. You can encode same data on different platforms and always get the same decoded result.

Want to get the example up and running? Seeing is believing after all. Let's do this.

Is This Thing On?

Once you're done with the setup (above), let's get this firmware programmed! Lucky for you it's only two quick steps!

1. Plug in your NRF52 DK board
2. Run make flash_all. (This compiles and flashes all the code for the project.)

Once flashed, the easiest way to know if things are working is when Led 1 is blinking. That means it’s advertising and ready for a connection.

Let’s make sure it’s advertising correctly though. You can grab a tool like NRF Connect for iOS or Android. Or, you can grab LightBlue. (Also available for iOS and Android)

Open up one of the app and scan for advertising devices. You’ll be looking for Nordic_Buttonless.

Now let’s connect and double check our connectable services & characteristics:

As you can see there are two services. One is Nordic’s DFU service. The other is our Protocol Buffer service!

You can compare PROTOBUF_UUID_BASE in ble_protobuf.h with the UUID of the “Unknown Service”. Nordic’s chips are Little Endian while the data here is Big Endian format. (i.e. you’ll have to reverse the bytes to see that they’re the same!)

You can even click in further to see the characteristics in NRF Connect. In the case of LightBlue, the characteristic UUIDs are already shown.

Using the Javascript App

So, once we’re up and advertising, it’s time to run the app.

Simply change to the ble-protobuf-js directory and run node index.js

If you've installed everything correctly, you should start seeing output like:

example started scanning peripheral with ID 06f9b62ec5334454875b9f53d2f3fa74 found with name: Nordic_Buttonles

It should then immediately connect and send data to the device. It should receive the response immediately and print it to the screen.

connected to Nordic_Buttonles Sent: This is Received: This is cool.

Bingo!

You’ve made it!

? Congrats if you made it this far through the tutorial. You should now be feeling confident enough to start playing with the software code. Maybe you’ll cook up something cool?

Here’s links to some resources that you may find handy.

• Protocol Buffer documentation
• NanoPB - (the implementation of Protocol Buffers we used for this project)
• Nordic SDK Documentation
• Again, here are the links to Part 1 and Part 2
• If you haven’t already, get all the code for this project

Conclusion

Protocol Buffers are quite versatile and convenient. Using them with Bluetooth Low Energy service is one way that you can take advantage. As the connected world of IoT continues to expand, I have no doubt that we'll be seeing more use cases in the future!

I hope this has inspired you to incorporate Protocol Buffers into your own project(s). Make sure you download the sample code and get started right now.

Has this tutorial been useful for you? Let me know what you think.

This post is originally from www.jaredwolff.com.

This is Part 2 of configuring your own Bluetooth Low Energy Service using a Nordic NRF52 series processor. If you haven’t seen Part 1 go back and check it out. I’ll be waiting right here..

If you’re with me thus far, high five. ?

Let’s jump in!

So far we’ve created an efficient cross platform data structure using Protocol Buffers. This Protocol Buffer in particular can be used to send these defined data structures a Bluetooth Low Energy Service. In this part, I’ll show you the inner workings of creating the service from scratch.

P.S. this post is lengthy. If you want something to download, click here for a a beautifully formatted PDF. (Added bonus, the PDF has all three parts of this series!)

Creating the Service

Dealing with Bluetooth Low Energy in general can seem overwhelming. As I discussed here, there’s a few moving parts that you need to keep in mind.

The best way to create a new service is to copy an already existing one! I’ve done this by:

1. Go to the sdk -> components -> ble -> ble_services -> ble_bas
2. Copy ble_bas.h to include/ble
3. Copy ble_bas.c to src/ble

I’ve then renamed them from ble_bas to ble_protobuf to be consistent. I’ve also done the same inside the files. (BAS is the battery level service used to report battery voltage or relative charge using a percentage)

I’ve also gone ahead and removed all the battery measurement functions as they will be replaced. This part of the process is fairly tedious and prone to error. If you’re new to the Nordic SDK, I highly recommend you download the example code for this post.

Adding a UUID

Normally, for a vendor defined service you’ll have to use your own UUID. There are certain ranges of UUIDs that are reserved for the Bluetooth SIG. Supposedly you can also reserve your own UUID if you’re a member. Here’s a handy post on Stack Overflow on the subject.

In our case, I’ve defined a UUID that I’ve used for other projects. If you go to ble_protobuf.h you can see the UUIDs for both the service and the characteristic:

// UUID for the Service & Char
#define PROTOBUF_UUID_BASE   {0x72, 0x09, 0x1a, 0xb3, 0x5f, 0xff, 0x4d, 0xf6, \
                               0x80, 0x62, 0x45, 0x8d, 0x00, 0x00, 0x00, 0x00}
#define PROTOBUF_UUID_SERVICE     0xf510
#define PROTOBUF_UUID_CONFIG_CHAR (PROTOBUF_UUID_SERVICE + 1)

Both PROTOBUF_UUID_BASE PROTOBUF_UUID_SERVICE are used in ble_protobuf_init The last one is used in command_char_add (I’ll describe that a bit more below).

You can go without defining a BASE ID but I highly recommend you go the extra mile. That way your application will be impervious to future Bluetooth protocol updates.

Creating the Characteristic

Nordic has a fairly straight forward way of initializing separate characteristics in each service. For each characteristic, there is a char_add function which then configures and add the characteristic to the service.

Configure your Characteristic withble_add_char_params_t

Nordic has put the configurations parameters for a BLE Characteristic into a logical (and helpful) struct. If you’re developing on a different platform you may find the same settings though all not in one place! (Or handled logically… ?)

Here’s the breakdown of the struct:

typedef struct
{
    uint16_t                    uuid;
    uint8_t                     uuid_type;
    uint16_t                    max_len;
    uint16_t                    init_len;
    uint8_t *                   p_init_value;
    bool                        is_var_len;
    ble_gatt_char_props_t       char_props;
    ble_gatt_char_ext_props_t   char_ext_props;
    bool                        is_defered_read;
    bool                        is_defered_write;
    security_req_t              read_access;
    security_req_t              write_access;
    security_req_t              cccd_write_access;
    bool                        is_value_user;
    ble_add_char_user_desc_t    *p_user_descr;
    ble_gatts_char_pf_t         *p_presentation_format;
} ble_add_char_params_t;

It’s here that you tell the BLE Stack about what type of characteristic this is. In this example I use a handful of parameters to define the Protobuf Command Characteristic.

uuid defines the address of how the characteristic will be accessed. If you’re defining your own service. max_len is the maximum length of data that you may send though the characteristic. That’s why it’s important to set max_size in your Protocol Buffer .options file for all variable length parameters. Once you compile your Protocol Buffers you’ll get a *_size variable much like the one defined in command.pb.h Here's a snippet of what it looks like below:

/* Maximum encoded size of messages (where known) */
#define event_size                               67

Thus when defining ble_add_char_params_t I set max_len to event_size:

add_char_params.uuid                      = PROTOBUF_UUID_CONFIG_CHAR;
add_char_params.max_len                   = event_size;
add_char_params.is_var_len                = true;

Along the same lines, because we’re using a string as one of the parameters in the Protocol Buffer, this data can be of variable size. is_var_len is handy to make sure that the right amount of bytes is sent and received. The decode function of the Protocol Buffers will fail if more data is fed in than necessary. (I learned this the hard way!)

char_props define the permissions for the characteristic. If you’re familiar with file system permissions on a computer, this will be second nature to you. In this example, read and write is what we’re looking for.

Finally, parameters ending in _access determine the security type used. In most cases SEC_OPEN or SEC_JUST_WORKSis more than sufficient. If you’re handling critical data (passwords, etc) you may have to implement a second layer of encryption or enable a higher level security mode.

If you’re looking on more info about Bluetooth Low Energy security, here’s a useful post on the subject.

Add. dat. char.

Once you’ve defined your params, it’s as easy as calling the characteristic_add function. This will associate this new characteristic with the related service. The first argument is the service handle, the second the configuration parameters and the third is a pointer to the handles for the characteristic. (See below)

uint32_t characteristic_add(uint16_t                   service_handle,
                            ble_add_char_params_t *    p_char_props,
                            ble_gatts_char_handles_t * p_char_handle)

Getting it Running

Setting everything up inside ble_protobuf.c is 90% of the battle. The final mile requires some bits being added to services_init in main.c

    ble_protobuf_init_t       protobuf_init = {0};

    protobuf_init.evt_handler          = ble_protobuf_evt_hanlder;
    protobuf_init.bl_rd_sec            = SEC_JUST_WORKS;
    protobuf_init.bl_cccd_wr_sec       = SEC_JUST_WORKS;
    protobuf_init.bl_wr_sec            = SEC_JUST_WORKS;

    err_code = ble_protobuf_init(&m_protobuf,&protobuf_init);
    APP_ERROR_CHECK(err_code);

The above allows events to be funneled back to the main context. That way your app becomes much more interactive with the core logic of your firmware code. In Nordic’s examples they’ve also brought the security parameters out so they can be defined in the main context as well.

Side note: m_protobuf is defined using a macro from ble_protobuf.h It not only creates a static instance of the service but it also defines the callback that is used for handling events.

/**@brief Macro for defining a ble_protobuf instance.
 *
 * @param   _name  Name of the instance.
 * @hideinitializer
 */
#define BLE_PROTOBUF_DEF(_name)                          \
    static ble_protobuf_t _name;                         \
    NRF_SDH_BLE_OBSERVER(_name ## _obs,             \
                         BLE_PROTOBUF_BLE_OBSERVER_PRIO, \
                         ble_protobuf_on_ble_evt,        \
                         &_name)

If you're making your own service, you'll have to update the function name for the event handler. If you need to tweak priorities you can define/update that as well.

What happens when this characteristic is written to?

ble_protobuf_on_ble_evt is the main way that events are handled within Bluetooth Low Energy services. We’re most concerned with the BLE_GATTS_EVT_WRITE event but you can trigger on any GATT event that tickles your fancy.

on_write is where the action happens. It takes the data that is written to the characteristic and decodes it according to event_fields It’s all put conveniently into a struct for additional processing, etc. If an error happens in decoding, pb_decode returns an error. Once modified, the data is encoded and made available for reading. Since reading Part 1, the calls to pb_decode and pb_encode should look very familiar!

Of course, you can have your firmware do whatever you want to. The Bluetooth Energy World is your oyster.

Final Notes

When adding new services to a Bluetooth Low Energy example, you may have to make some changes to the underlying code.

For example, sdk_config.h may need some changes. Particularly NRF_SDH_BLE_VS_UUID_COUNT needs to be increased depending how many service UUIDs are made available. For this project, I am also using the DFU service (as it should be a default for all connected projects!!)

Another important aspect is memory and flash management. The default .ld file that comes with the BLE DFU service may not be sufficient for another BLE Service. The only way you’ll know there’s not enough is when you compile and flash it to a NRF52 device. If the device boots up stating there’s not enough memory, you’ll have to make the suggested changes. The error will show up on the debug console where this message normally shows up:

<info> app: Setting vector table to bootloader: 0x00078000
<info> app: Setting vector table to main app: 0x00026000

Learn more about how to get the Debug Console set up in the example code here.

Conclusion

In this part I’ve shown you the inner workings of a custom Bluetooth Low Energy service using Protocol Buffers. In the last part, I’ll show you how to load the firmware, run the example javascript app and test our freshly developed Protocol Buffer!

This post is originally from www.jaredwolff.com

One of the most confusing things about Bluetooth Low Energy is how data is moved around. Depending on your application, your device state may be fairly complex. That means having an individual endpoint for every piece of data is suicide by Bluetooth.

So, what’s he solution?

Protocol Buffers.

A protocol buffer is a programatic way to encode/decode optimized structured data. They can be shared and manipulated across almost any platform. Nordic actually uses a variant of it for their DFU service.

There was a lot of buzz words in the first few sentences. Hopefully, by the end of this post you’ll understand exactly what I’m talking about.

In this tutorial, i'll include fully flushed out example code that you can clone and start using immediately. All you need is one of these:

So, how do you use this magical software?

Read on!

P.S. this post is lengthy. If you want something to download, click here for a a beautifully formatted PDF. (Added bonus, the PDF has all three parts of this series!)

Install

The first part of the process is to make sure you’ve installed all the correct utilities. Depending on what programming language will determine what you install and use. In this case I’ll outline the utilities that I have used for several projects in the past using C, Go and Javascript.

protoc is the most important utility you’ll have to install here. It's the Protobuf "compiler" which takes your .proto and .options files and turns them into static code.

1. For Mac, download the appropriate release here.
2. Unzip the folder
3. Run ./autogen.sh && ./configure && make in the folder
4. If you get an error autoreconf: failed to run aclocal: No such file or directory install autoconf using Homebrew:

brew install autoconf && brew install automake

Then, re-run step 3.

1. Then run:

make check
sudo make install
which protoc

Consider protoc the compiler for Protocol Buffers. It can either output raw files or libraries directly. That’s because it’s got Go support built in.

That raw data can also be used to generate static libraries for other languages. That usually requires an extra utility (or utilities). I describe the two that the Dondi Lib project used below.

2. nanopb is a python script used to create C libraries that encode/decode your structured data. It can be installed by navigating to the nanopb git repo and downloading the appropriate files. The most important pieces to include:

3. pb_encode.c, pb_decode.c and pb_common.c

4. /generator/nanopb_generator.py

5. And the /generator/nanopb/directory co-located with nanopb_generator.py

   nanopb is meant for deployment on embedded platforms. It's different from protoc-c (the regular C variant) because it is optimized for resource constrained systems like embedded processors. Buffers have finite sizes. There's no memory allocation! Depending on if there's bi-directional communication, you can only import and use the encoding functionality or decoding functionality.

6. pbjs uses the output from protoc to generate a static javascript library. This is powerful because you can then use it in any javascript application. The best way to install pbjs is by running:

      npm install -g protobufjs
   

I've simplified this step a bit in the example code. Get started by cloning the repos here.

Setting up the protocol buffer

Create a file called command.proto. You can make the contents of that file what's below:

syntax = "proto3";

message event {
  enum event_type {
    command = 0;
    response = 1;
  }
  event_type type = 1;
  string message = 2;
}

It may look foreign at first but once you take a deeper look, it’s not that much different than a standard C struct or hash table.

I'm using two types of data in this example: a string and enum as a type. There are actually a few more which you can read up at the documentation. When compiled, the equivalent c struct looks like:

/* Struct definitions */
typedef struct _event {
    event_event_type type;
    char message[64];
/* @@protoc_insertion_point(struct:event) */
} event;

Where event_event_type is

/* Enum definitions */
typedef enum _event_event_type {
    event_event_type_command = 0,
    event_event_type_response = 1
} event_event_type;

You can nest as many messages inside each other as your hearts content. Typically though, a message is as small as possible so data transmission is as efficient as possible. This is particularly important for resource constrained systems or LTE deployments where you're charged for every megabyte used. Note: when elements are not used or defined they are typically not included in the encoded Protocol Buffer payload.

Normally, when you create a generic message like this, there is no limit to the size of the string message. That option can be set in the .options file:

event.message    max_size:64

This way, the memory can be statically allocated in my microprocessor code at compile time. If the message size is greater than 64 bytes then it will get chopped off in the code (or you'll simply get an error during decode). It's up to you, the software engineer, to figure out the absolute maximum amount of bytes (or characters) that you may need for this type of data.

You can look at more of the nanopb related features at their documentation.

Generating the appropriate static libraries

In order to make this as easy as possible, I put all the following code into a Makefile. When you make a change to the Protocol Buffer, that every library for every language used gets generated.

If we want to generate a static Go file the command looks like:

protoc -I<directory with .proto> --go_out=<output directory> command.proto

If you've installed the nanopb plugin, you can do something similar to generate C code:

protoc -I<directory with .proto> -ocommand.pb command.proto
<path>/<to>/protogen/nanopb_generator.py -I<directory with .proto> command

The first file creates a generic "object" file. The second actually creates the static C library.

For javascript:

pbjs -t static-module -p<directory with .proto> command.proto > command.pb.js

You can test each of these commands with the .proto and .options file examples above. I also built this manual process into one command in the example repository. Get access here.

Encoding and Decoding

In the examples below, I show you how to use your freshly compiled static code! This is where the fun begins.

Encoding using Javascript

Here’s a typical flow that you can follow when using a statically generated javascript library. First, initialize the library.

// Import the config message
var protobuf  = require('./command.pb.js');

Then create an instance of event:

// setup command
var event = protobuf.event.create();
event.type = protobuf.event.event_type.command;
event.message = "This is";

Then, compile the payload. i.e. turn human readable JSON into nicely packed binary. See below.

// make sure it's valid
var err = protobuf.event.verify(event);
if( err != null ) {
   console.log("verify failed: " + err);
   return;
}

You'll get errors during this step if your object is malformed or if if you are missing required elements. I don't recommend using the required prefix when defining your .proto file. Any checks for required elements can be easily done in your application code.

Finally, the last step is to encode and turn it into raw bytes:

// encode into raw bytes
var payload = protobuf.event.encode(event).finish();

You can then use payload and send it over BLE, HTTP or whatever. If there's a communication protocol, you can send this buffer over it!

Decoding in C

Once the data is received it’s decoded on the embedded end. nanopb is confusing. But luckily I have some code here that will work for you:

// Setitng up protocol buffer data
event evt;

// Read in buffer
pb_istream_t istream = pb_istream_from_buffer((pb_byte_t *)data, data_len);

if (!pb_decode(&istream, event_fields, &evt)) {
   NRF_LOG_ERROR("Unable to decode: %s", PB_GET_ERROR(&istream));
   return;
}

// Validate code & type
if( evt.type != event_event_type_command ) {
   return;
}

First, you create an input stream based on the raw data and the size of the data.

Then, you use the pb_decode function. You point the first argument to the input stream. The second to the definition of our Protocol Buffer we’ve been working with. It's located in the command.pb.h file.

/* Defines for backwards compatibility with code written before nanopb-0.4.0 */
#define event_fields &event_msg

The last argument is a pointer to the struct to put the decoded data into. (In this case it's evt defined right before pb_istream_from_buffer above).

Encoding in C

Let's now say we're going to reply to the message that was just decoded above. So now we have to create data, encode it and send it back. Here's the process:

// Encode value
pb_byte_t output[event_size];

// Output buffer
pb_ostream_t ostream = pb_ostream_from_buffer(output,sizeof(output));

if (!pb_encode(&ostream, event_fields, &evt)) {
   NRF_LOG_ERROR("Unable to encode: %s", PB_GET_ERROR(&ostream));
   return;
}

First create a buffer that holds the maximum amount of bytes that your Protocol buffer takes up. This is also defined in your command.pb.h. In this case event_size is set to 67. Then, similarly to the decode command, you create a stream and connect it to your buffer. Then finally encode the data by pointing your evt struct along with the stream and event_fields.

As long as pb_encode returns without error, the encoded data has been written to output! The structure can be variable length so the best way to handle when sending it is to get the bytes_written from ostream:

NRF_LOG_INFO("bytes written %d",ostream.bytes_written);

Conclusion

Nice you made it! I hope you're starting to grasp the power of Protocol Buffers. Don't worry, it took me a little while to figure it all out. You too can be a Protocol Buffer master! ?

If you're not too thrilled with Protocol Buffers, there are other alternatives. I've used MessagePack with some success on previous products. It's straightforward and has tons of support for a majority of programming languages.

If you are interested how to roll this into a Bluetooth Low Energy project, stay tuned for Part Two. In Part two, I’ll show you how to set up a very simple Bluetooth Service and Characteristic that will be used to transfer our freshly encoded data to-and-fro.

Also, if you want to see all the code in action, you can download everything here.