Protobuf is Google's answer to the need for an effective, efficient, and flexible serialization format as an alternative to JSON or XML. Its benefits pave the way for groundbreaking solutions to the problems faced by modern enterprises in creating a digital landscape in the cloud.

Cloud-native means rethinking

Constructing a half-timbered house using traditional methods and tools certainly has its charm and is an achievement not to be sneezed at. However, modern zero-emission houses with a smart home connection cannot be built this way. And let's be honest, the number of customers desperately looking for a half-timbered house is very small.

Cloud-based applications present new and unfamiliar challenges to today's software architects. They need modern processes that are specifically designed to meet the new challenges of the cloud, so that applications can run reliably while conserving resources. Whether it is CPU, memory or disk space, anything that is used less saves money in running the cloud and is also a step towardss green computing.

Numerous challenges

I cannot cover all the challenges in the necessary depth in this article. The text would be so long and complicated that no one would finish reading it. Instead, I will focus on serialization. Serialization is the conversion of structured data into a format suitable for transmission or storage.

In modern distributed systems, the choice of a fast and reliable serialisation format is critical. Serialization itself is not complex, but it is widely used. Choosing the optimal format therefore has a major impact on the overall design of the cloud application.

A well-chosen format ensures that data is transferred over the network quickly and with minimal overhead. In addition, a reliable format is essential to avoid errors during data reconstruction and thus ensuring data integrity.

Automation as a success factor

JSON is currently the serialization format of choice for many. It has overtaken XML over the years. However, the advantage of JSON, as a format that is easy for humans to create and read, does not really come into play when it is used for communication between microservices. Although data and processes are triggered by human interaction, they only run downstream between systems and react to events rather than users.

The argument that developers need to read the serialization format is also no longer valid. Cloud applications run in containers, which are themselves monitored and controlled by systems such as Kubernetes. Manual intervention is not the method of choice for analyzing and resolving problems.

In addition, the application is updated at short intervals via a continuous delivery pipeline to meet market demands for rapid and constant adaptation. There is no alternative to automation in this context, as it ensures secure and, in times of skills shortages, cost-effective operation.

Gamechanger Protobuf

In this article, I will briefly discuss the terms and functionality of Protobuf. I will then explain its novel concepts and features and how they address the shortcomings of JSON. Finally, I will explain why Protobuf is a vast improvement over JSON and the next step in the evolution of serialization formats.

Safe and secure thanks to precise definition and strong typing

In order to serialize data, it must first be described in a structured way. With Protobuf, the data structures are defined in files with a .proto extension. In this proto definition, an object is described by a message. A message may contain any number of fields. A field has an identifier, a type and a number that is unique for this message. The type of a field can in turn be a message, creating a tree structure similar to JSON and XML.

syntax = "proto3";

package example;

message Book {
 string title = 1;
 repeated Page pages = 2;
}

message Page {
 string content = 1;
}

The code example above describes a simplified data structure for a book. The book has a title of type string, which can contain any character string. The book also has an arbitrary number of ("repeated") pages, each of which is stored as a message of type "Page". The pages have the text they contain stored in the "content" field. The explicit specification of types in the proto definition helps to avoid errors and ensures consistency across system boundaries. Protobuf supports common programming language data types. Google also provides a collection of extended types, called WellKnownTyps. You can also create your own data types using self-defined messages.

High productivity with code generation

With other serialization formats, a lot of valuable development time is spent creating the data structures again as objects in the programming language and continually updating them as changes are made. The work is time consuming, but not very challenging, which means that errors creep in quickly. Generic mappers promise to reduce the effort, but they are difficult to configure in all cases where you deviate from the normal use case.

Protobuf relies on automatic and continuous code generation and supports multiple programming languages. The data structure code is generated directly from the proto definition, eliminating much of the effort of other approaches. It is also guarantees that the sender and receiver always use the same implementation as long as they have the same version of the proto schema. The generated source code can also be flexibly extended with additional methods to make the use of custom data types more convenient.

Strictly defined but extendable contract

The proto definition is an interface contract between the sender and the receiver. Protobuf uses only simple structural elements. For example, the keyword "repeated" refers to a list of a messages without specifying its exact number. On the one hand, this means that the sender cannot read possible restrictions on the receiver side from the contract. On the other hand, this can also be seen as an opportunity to make the contract more flexible and therefore easier to extend. The ability to easily change the contract while maintaining the integrity of the application is invaluable in a constantly evolving system landscape.

Simple schema evolution

Protobuf was designed with compatibility in mind. A change is backward compatible if the new version of a receiver can process messages from from an old version of a sender. Conversely, a change is forward compatible if the new version of a sender still works with the old version of the receiver.

By following a few simple rules, Protobuf provides both downward and upward compatibility. For example, the type and number of a field must never be changed. A message and a field cannot be deleted. The proto3 syntax also does not define any mandatory fields, as extensions to a contract must always be optional in order to maintain compatibility.

Compatibility for zero downtime deployment

Backward and forward compatibility is particularly important if it cannot be guaranteed that the sender and receiver and all messages can be updated to a new version of an interface description at the same time. This is an important feature for cloud applications.

Full compatibility makes it possible to roll out a sender and a receiver independently of each other, eliminating the need for time-consuming coordination of the sequence and making continuous delivery possible in the first place. Using a modern software solution for managing containerised applications, such as Kubernetes, also enables automatic rolling updates. This means that a new version of the application can be rolled out without any downtime. Users are unaware of this, which allows for uninterrupted service and an optimal customer experience.

Outstanding efficiency as binary and minimalist

Protobuf serializes messages in binary format. Only the number and value of the field in a message are stored. The proto definition is therefore needed to be able to read the message later. Serialization deliberately avoids unnecessary ballast that is not necessary for the actual benefit. Because of the small amount of data, the creation and reading of messages is very fast.

However, the minimal message storage and high compatibility can lead to unexpected behaviour in the event of errors during development. Under certain circumstances, messages can be deserialized by a foreign proto definition without an error occurring. It is therefore essential to ensure the use of protobuf with automated tests. These can also be used to ensure compatibility with older versions.

Manual error analysis of messages on a case-by-case basis is easily possible by converting the binary information to an object. Messages can also be easily converted to a JSON format. The code required to do this is fully generated according to the proto definition. This means that legacy systems that can only work with JSON can be connected to modern applications. The benefits of Protobuf can be at least partially exploited.

Conclusion

No serialization format, neither Protobuf nor JSON, can be used as a "golden hammer". Protobuf is certainly not an alternative to JSON for all use cases, nor was it designed to be. The strengths of Protobuf make it a flexible and at the same time particularly efficient serialization format. The focus of the application area is primarily on data exchange between software systems. It can therefore be easily used wherever automated processes are created as part of digitization.

The initial adoption of Protobuf may require some training and learning, but this is quickly recouped through the consistent use of the serialisation format across as many use cases as possible. The high degree of automation and code generation allows the development team to focus on what they do best - solving problems.

Now there are no limits to your creativity in designing solutions for your domain based on Protobuf. I look forward to your ideas and suggestions. After all, aren't we all striving to continuously improve our software through first-class design and to make it a success?

In future articles, I will give some concrete examples of how I have used Protobuf in the past. It's best to follow me so that you don't miss any of my future articles on Protobuf. ❤