Name: Practical unit-testing web client in Go
Rating: 4.7 (1970 reviews)
Author: chen

I've started a Go project, a lightweight Hashicorp Vault¹ client with no dependencies, and a simple API (for the user). Part of the reason I use no 3rd party modules is, I want to better understand Go internals, structure, and improve my skills. In this post, I'll conduct a practical example of how I ended up testing it.

The the project's name is libvault, and it's my first open-source project.

Vault is a secret manager service with web API and CLI. Applications can communicate with it through HTTP. I find the CLI easy to use, however, the official Vault library is a bit more complicated; it felt like a swiss-army knife when all I needed was just a simple kitchen knife. I decided to implement a light version, that covers basic functionality while maintaining a simple API.

I'm not going to cover the basics in this post. The intention of it is to be practical and provide real-life, working examples. The opinions here are mine. It worked for me, still, it doesn't mean it would work for you.

I removed ALL the error handling from the code for brevity. Please make sure you handle your errors.

The Go standard library provides a really good testing package. You can manage without external frameworks or 3rd party packages. In my scenario, I need to test a web client that I wrote. The technique I found useful is mock testing.

Mock testing is an approach to unit testing that lets you make assertions about how the code under test is interacting with other system modules. In mock testing, the dependencies are replaced with objects that simulate the behaviour of the real ones. ... Such a service can be replaced with a mock object. ~ Wikipedia

I had to choose which component to mock:

Client
Webserver

At a high level, mocking the client means creating a new struct that implements the interface you are testing (mocking the interface). Then, provide your mocking client to the code under test. A good library with examples is testify.

I didn't find it useful for my use case, as I need to mock the server-side. I prefer not to modify any code on the client if I can. This would result in more reliable tests for my package. So, I've chosen to go with the second option. Read on for how.

The httptest package

Go is very friendly to web services; it has a utility package (httptest) for testing an http server. You can simply start a webserver in your testing code.

First I had to fetch an exact response from a real webserver (Vault server), then I could easily mock it using this package. For example, when querying the /v1/auth/approle/login endpoint, the response looks like this:

{
  "request_id": "de7c8097-1a38-50a6-b971-fe1836840e45",
  "lease_id": "",
  "renewable": false,
  ...
  ...
}

Now that I have the content, I find it easier to save it to a file (rather than put it inside the code). Go testing package has another cool feature:

The go tool will ignore a directory named "testdata", making it available
to hold ancillary data needed by the tests.²

I created a directory named testdata inside my project and placed the JSON content in a file - approleExample.json.

Start the mocking server

I'll start with code, followed by an explanation:

package main

import (
    "fmt"
    "io"
    "net/http"
    "net/http/httptest"
)

func TestClientLogin(t *testing.T*) {
    mux := http.NewServeMux()
    ts := httptest.NewServer(mux)
    defer ts.Close()

    mux.HandleFunc("/v1/auth/approle/login", func(w http.ResponseWriter, r *http.Request) {
        // request validation logic
        ...
        // read json response
        jsonPayload, _ := ioutil.ReadFile("testdata/approleExample.json")
        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(200)
        w.Write(jsonPayload)
    })

    // initalize new client pointing to the testing server
    client, err := NewClient(ts.URL)
    if err != nil {
        // error handling
    }

    // test logic
}

If you're familiar with the Go http package, this code is pretty self-explanatory. This is another advantage of using the standard library - you have one less thing to learn when you want to contribute.

mux is of ServeMux type. Which is an HTTP request multiplexer. It matches the URL of each incoming request (/v1/auth/approle/login) against a list of registered patterns and calls the handler for the pattern that most closely matches the URL (our mux.HandleFunc function body).
ts is of Server type. A Server is an HTTP server listening on a system-chosen port on the local loopback interface, for use in end-to-end HTTP tests. Note that I use httptest.NewServer to initialize it.
mux.HandleFunc(..) defines a path and an handler (a function) to call. The content inside describes our server's response.
client, err := NewClient(ts.URL) creates a new Client (my Vault client), providing it the test server URL to work with.

The elegance of this pattern is, every test case has its test webserver with all the relevant configurations. The mocked content, the test logic, etc. are all implemented inside the test itself.
This really makes life easier when debugging, reviewing a test case logic or coverage.

We can improve it further, make the code more clear and concise. This code includes some boilerplate: creating the mux, the server, and reading the json content. We just need to refactor these elements out (to a setup() function, and readJson(path string), for example). Then call this setup() function for every test case. I'll leave that to the reader to decide.

Summary

There are numerous articles about the importance of software testing. Go makes it easier. You should always write tests for your packages; it has so many advantages to just skip it. However, many people do that and I can assume the reason, which is it doesn't provide any additional functionality to your code. Don't be one of these people.

Go standard library provides great tools, and you should use them. Personally, I prefer it over other dependencies.

In this post, I gave a practical example of how you can unit-test a web client by mocking it.

The takeaways:

Always write tests. They are too valuable to give up and very easy to do with Go.
Mock the part your code connects with, don't mock your code. If you write a server, mock the client, and vice-versa. It makes your tests much more reliable.
1. If you mock a server, get a real server response and save it to a file
2. Do that for every API you would like to mock
Use testdata directory to hold your ancillary data needed by your tests

In the next article, I will cover how to test with a TLS (HTTPS) web server and self-signed certificates, adding more examples. You can also find more examples in the source code.

Feedback and comments are welcome.