Protobuf messages are gaining popularity across many domains. There are libraries like nanopb in C to support the protobuf encoding/decoding. However they are slow and also pose significant challenge in implementation. GoLang protobuf encoding/decoding is fast and provides support of unmarshalling data from JSON to proto format.

We will be creating a C shared library containing functions that read from json file, protobuf encode and decode.

Full Code can be found here:

GoLang Code

Lets start with Student.proto. I am using proto3 for this example:

syntax="proto3";

package main;

message Parents {
      string dad=1;
      string mom=2;
}

enum Gender {
    Female = 0;
    Male = 1;
}

message Student {
      string name = 1;
      int32 age = 2;
      Parents parents=3;
      Gender gender=4;
}

message Students {
      repeated bytes StudentEntry=1;
}

As you can see above proto includes enum, nested proto and repeated bytes. Student represents a student's data, each student's data will be encoded and added to Students StudentEntry and all the encoded entries will be encoded again.

Below is input JSON file student_list.json which is used to populate proto fields. We have to use the same names and nested levels for the unmarshal of Json to proto.

{
"0" : {
    "Name": "Elliot",
    "Age" : 15,
    "Parents" : {
            "Dad" : "George",
            "Mom" : "Elena"
    },
    "Gender" : 0
  },
"1" : {
    "Name": "David",
    "Age" : 14,
    "Parents" : {
            "Dad" : "Keith",
            "Mom" : "Sarah"
    },
    "Gender" : 1
   }
}

Encoding Protobuf from JSON

To integrate with C, library C is used to convert GoData to C datatypes. Comment export is used to export functions to C lib.
We will create a new GoLang program file student_en_dc.go and add encode and decode functions to it.
Json file is input parameter of this function get_student_enbuf passed from C. This function returns encoded buffer and its length to C.

//export get_student_enbuf
func get_student_enbuf(student_json_file *C.char) (unsafe.Pointer,C.int) {
       .
       .
       return C.CBytes(students_en),C.int(int32(len(students_en)))

First step in encoding is to Unmarshall JSON to proto. As there are more than one student entries, I have added entries using number keys. This cannot be directly unmarshalled, we would have to extract json data of each student and then unmarshal. I am first unmarshalling json file data to an interface. Below is the code:

jsonbytes, _ := ioutil.ReadAll(jsonFile)
var parsed_proto map[string]interface{}
json.Unmarshal(jsonbytes, &parsed_proto)

Then parse each interface to obtain student json data and unmarshall it into our proto. Aftermath proto is encoded as shown.

    for i=0;i<len(parsed_proto);i++ {
            selectedjb, err := json.Marshal(parsed_proto[strconv.Itoa(i)])
            if err != nil {
                log.Println("Error in reading json index",err)
                return nil,0
            }
            var parsed_jf map[string]interface{}
            //Each student data to proto
            json.Unmarshal(selectedjb, &parsed_jf)
            student := &Student{}
            //Proto Encoding of student
            json.Unmarshal(selectedjb, student)

Encoded data is added to StudentEntry of Students. Then Students data is encoded. We have the encoded buffer.

            students.StudentEntry=append(students.StudentEntry,student_en)
    }
    students_en, err := proto.Marshal(students)

Decoding protobuf

Students is decoded first following all the student entries.

Function decode_student_enbuf takes encoded data and its length as input passed from C and returns decoded buffer to C.

//export decode_student_enbuf
func decode_student_enbuf(data *C.char,leng C.int) *C.char {
           .
           .
           return C.CString(decoded_student)
}

Before decode, *char is converted to GoBytes and then unmarshalled to proto.

new_students := &Students{}
student_en:=C.GoBytes(unsafe.Pointer(data),leng)
err := proto.Unmarshal(student_en, new_students)

Decoded students data is used to obtain each StudentEntry and is decoded. Decoded student is appended to a string and is returned to C.

for i=0;i<len(new_students.StudentEntry);i++ {
        new_student := &Student{}
        err = proto.Unmarshal(new_students.StudentEntry[i], new_student)
        decoded_student=decoded_student+"\n"+proto.MarshalTextString(new_student)

Building C-Shared file

Lets compile proto to generate Student.pb.go.Then build c-shared file.

protoc --go_out=. Student.proto
go build  -o student.so -buildmode=c-shared Student.pb.go student_en_dc.go

student.so and student.h files are generated.
Following lines would be present in the generated header file. Golang function get_student_enbuf returns tuple(more than one return parameters), this is converted to struct when header files are generated as seen below.

/* Return type for get_student_enbuf */
struct get_student_enbuf_return {
        void* r0;
        int r1;
};

extern struct get_student_enbuf_return get_student_enbuf(char* p0);

extern char* decode_student_enbuf(char* p0, int p1);

C Code

From C side, generated header file is included. Students list JSON file is passed to function get_student_enbuf, returned encoded buffer and its length is populated in struct stu. The same are passed to decode function decode_student_enbuf, the returned decoded string is printed.
Finally both encoded and decoded buffers are freed.

#include <stdio.h>
#include "student.h"
#include <string.h>
#include <stdlib.h>
#include <stdbool.h>

struct get_student_enbuf_return stu;

int main() {
    stu = get_student_enbuf("student_list.json");
    int stu_list_len = stu.r1;
    if(stu.r0!=NULL) {
        char *decoded_stu = decode_student_enbuf((char*)stu.r0,stu.r1);
        if(decoded_stu != NULL){
                printf("%s",decoded_stu);
                free((char*)decoded_stu);
        }
    free((char*)stu.r0);
    }
    return 0;
}

Compiling and executing C

Link with the created student.so, create object student and run the object.

gcc -o student student_cgo.c ./student.so

./student
name: "Elliot"
age: 15
parents: <
  dad: "George"
  mom: "Elena"
>

name: "David"
age: 14
parents: <
  dad: "Keith"
  mom: "Sarah"
>
gender: Male

Full Code can be found here: