Platform Engineering: Building Your Developer Portal with Backstage - Part 1

In my previous article, we looked at platform engineering: what it is, how it became a thing, platform engineering V.S. DevOps, and how it could help improve security.

If you haven't read it yet, here is the link to it for you:

Platform Engineering and Security: A Very Short Introduction

Today, let's get hands-on and build a developer portal from scratch.

1 What Are We Talking about When We Talk about Developer Portal

Since platform engineering is all about self-service, the internal developer portal is crucial in the new paradigm, in which all the self-service capabilities and integrations are implemented.

Given the significance of the developer portal, before building one, we need to figure out what exactly we are expecting from it. Typically, to get the most out of platform engineering, a developer portal would contain some key features, like:

a service catalog: giving developers a bird's-eye view of all the projects, services, deployment status, documentation, ownership, and even on-call schedules and incidents, etc.;
some kind of repository scaffolding/project generation tool: for example, something like cookiecutter, which enables developers to spin up new services from within the portal itself;
customization: you might want to integrate whatever toolchain you use into your developer portal to make it a genuinely unified one-stop experience. Possible integrations include Kubernetes, CI/CD status, on-call schedules, incident management systems, secrets managers, etc.

At this point, the portal can handle everything, like getting information about services, on-call schedules, triggering incidents, deploying things, and dealing with incident management. With these features, developers can have a truly unified experience where creating services and subsequently keeping track of and operating/maintaining those services in one place, maximizing the potential of a developer portal.

2 Choosing the Tool: Backstage

Now that we know what we really want from a developer portal, let's build one.

Since it will have multiple features, the designing/coding part could be a massive project in itself, and not all teams could afford that. That said, we need a tool for building the portal for a quick start, and luckily, we've got one - Backstage.

Backstage is an open platform for building developer portals. Backstage itself isn't a developer portal but a tool to build your developer portal. Think of "create-react-app" V.S., the actual react app you created with it. Out of the box, Backstage includes:

software catalog for managing all your software, such as microservices, libraries, data pipelines, websites, and ML models;
software templates for quickly spinning up new projects and standardizing your tooling with your organization's best practices;
docs for making it easy to create, maintain, find, and use technical documentation, using a "docs like code" approach;
a growing ecosystem of opensource plugins that further expand Backstage's customizability and functionality.

It can be so flexible in its architecture: it has a frontend written in React/TypeScript and a backend in Node.js, and it extends its power by adding plugins (which we will probably cover in the second part of this tutorial.

What's more, Backstage (created by Spotify) is now hosted by the Cloud Native Computing Foundation (CNCF) as an Incubation level project, meaning you can get all the community support you want. It's also got office hours, where you can join interactively to learn precisely how the opensource platform can drive better developer effectiveness and experience every Thursday.

Enough said; today, we will start from scratch and build a developer portal ourselves. After this tutorial, you can bootstrap a new service using templates with baked-in security CI workflows, check the CI status, and view the documentation for that service in the same place.

Without further adieu, let's get started.

3 Building The Portal

3.1 Prerequisites

Not much; probably all DevOps engineers already got them:

A Unix-based operating system. For example, you can run this on macOS, Linux, or Windows Subsystem for Linux (WSL).
curl, git, Docker
Node.js, yarn

3.2 Creating the Portal

Run:

npx @backstage/create-app@0.5.2-next.3

Author's note:

The reason we use backstage/create-app@0.5.2-next.3 instead of backstage/create-app@latest is because we need this feature to create secrets in GitHub to be used in GitHub Actions workflows. This feature has been developed, but as of now (June 18, 2023), it's not yet integrated into the latest version of backstage/create-app.

If you read this article later when the latest version points to a version released in June 2023 (or later), you can safely run npx @backstage/create-app@latest instead without specifying the weird version.

This command gives you an interactive mode where you need to enter the name, for example, "my-portal", press enter, then wait till the app is finished.

Enter the directory, and run:

yarn dev

And you should have the portal up and running already! It's that simple! Now, you can poke around with it for a bit and have a feeling of the software catalog, templates, documents, and stuff there:

But, at this moment, it's not of much use yet. So, let's continue configuring it.

4 GitHub Authentication and Integration

Since the developer portal will be in charge of bootstrapping new repositories, it requires permissions to operate GitHub, and that's why we need to do a GitHub authentication and integration.

4.1 Personal Access Token for Integration

While using GitHub Apps might be the best way to set up integrations because of its fine-grained permission settings, for this tutorial, we'll use a Personal Access Token for a quicker start:

Create your Personal Access Token by opening the GitHub token creation page.
Use a name to identify this token and put it in the notes field. Choose the number of days for expiration.
If you have a hard time picking a number, we suggest going for 7 days; it's a lucky number :) (And we will only be testing it locally, not running in production.)
For this tutorial, let's set the scope to the maximum so that your experience won't be blocked by struggling with GitHub permissions. Although, please note that you should NEVER do this in production!

Then, export the token as an environment variable:

export GITHUB_TOKEN=xxx

In the app-config.yaml file, change the integrations section to the following:

integrations:
  github:
    - host: github.com
      token: ${GITHUB_TOKEN} # leave it like this, read values from env var

4.2 Creating a GitHub OAuth App

Go to https://github.com/settings/applications/new to create your OAuth App. The Homepage URL should point to Backstage's frontend; in our tutorial, it would be http://localhost:3000. The Authorization callback URL will point to the auth backend, which will most likely be http://localhost:7007/api/auth/github/handler/frame.

Then, let's open our app-config.yaml file again and configure the authentication by updating the auth section to the following:

auth:
  environment: development
  providers:
    github:
      development:
        clientId: YOUR CLIENT ID # put your values here
        clientSecret: YOUR CLIENT SECRET # put your values here

Please note that storing the client secret in the config file as a hardcoded secret is against security best practices and should only be used for local development. For production usage, we can read them from environment variables, following the 12-factor app rules

After these config changes, we must stop our yarn dev servers and re-run yarn dev.

5 Creating a Template

Next, let's prepare a software template, which could be used to bootstrap a new service in no time.

The template should contain the following:

some basic source code/directory structure common to your services
some documentation
some CI workflows to test/build/deploy your service

For this tutorial, we will use this template that I created as an example.

The directory structure is relatively simple; there are only two parts:

a skeleton folder,
a template.yaml file.

5.1 The `skeleton` Folder

The skeleton folder contains all the templates that will be rendered when using this template to create a new service, and the variables are in the format of ${{ values.varName }}.

If you are familiar with Helm, YAML, or Go templates (or just about any template tool), you will find this easy to read and understand at no cost whatsoever.

The only item worth mentioning is a file named catalog-info.yaml, which is used by Backstage, so this file must exist; otherwise, you can't register the created service as a component in the portal.

As you can see, in the template, we have already got some baked-in GitHub Actions workflows, one would test on pull requests and push to the main branch, and the other will use ggshield to scan repositories for hardcoded secrets.

In this way, we can put all the CI/CD best practices in the template so that when others launch a new service, they already have everything they need with security features enabled by default.

5.2 The `template.yaml` File

The template.yaml file defines how the template looks in the portal UI and what it actually does.

It is long and seems overwhelming at first glance, but once you have a closer look at it, you will notice it's pretty straightforward:

the syntax is like a Kubernetes custom resource;
it has two major parts, one is parameters, and the other is steps;
the parameters define required input and their types when using this template;
the steps define what actually happens when you execute this template, and it looks a whole lot just like a GitHub Actions workflow.

Parameters example:

parameters:
  - title: Provide some simple information
    required:
      - service_name
      - owner
    properties:
      service_name:
        title: Name
        type: string
        description: Unique name of the service.
        ui:field: EntityNamePicker
      description:
        title: Description
        type: string
        description: Help others understand what this service is for; optional.
      owner:
        title: Owner
        type: string
        description: Owner of the component
        ui:field: OwnerPicker
        ui:options:
          allowedKinds:
            - Group
  ...

Steps example:

steps:

- id: template
  name: Fetch Skeleton + Template
  action: fetch:template
  input:
  url: ./skeleton
  copyWithoutTemplating: - .github/workflows/\*
  values:
  serviceName: ${{ parameters.service_name }}
  description: ${{ parameters.description }}
  destination: ${{ parameters.repoUrl | parseRepoUrl }}
  owner: ${{ parameters.owner }}
  ...

From the above file, we can infer what exactly it defines:

first, you need to enter two input parameters: service_name and owner;
then you need to choose a repository location with one extra parameter (GITGUARDIAN_API_KEY, to be used in CI pipelines);
it then fetches the template, renders it, publishes it to GitHub, and registers it in our portal.

5.3 Register the Template

Finally, let's add our template to our portal's catalog so that others can use this template.

The most straightforward configuration for the catalog is to declaratively add locations pointing to YAML files with static location configuration. Locations are added to the catalog under the catalog.locations key in the app-config.yaml file:

catalog:
  locations:
    - type: url
      target: https://github.com/IronCore864/backstage-test-template/blob/main/template.yaml
      rules:
        - allow: [Template]

The rule above allows us to add a template from the specified URL.

Remember to restart yarn dev servers.

6 Putting Everything Together

Now that we've got everything ready, it's time to see the magic in action.

Visit http://localhost:3000, and click the "Create" button, choose our template:

Input necessary information. For the GitGuardian API key, create one here: https://dashboard.gitguardian.com/api/personal-access-tokens.

When everything is set, click "next step", and the magic will happen:

Everything should be created now.

We can view it in our catalog:

And we've also got documentation created and rendered:

Last but not least, let's check out the CI status:

It seems the pipelines are already finished successfully, and we can click on them for more details with detailed steps and logs.

If you prefer to view them in the CI software (in this case, GitHub Actions), you can click the corresponding links to jump to them.

For your reference, this repo is the one I created using the template above.

This means no matter what toolchain the team uses, the team members don't have to remember 10 different URLs for 10 different tools, and they don't have to keep those 10 tabs open all the time. When they need some information, any information, they go to the developer portal, and they've got everything, and that's precisely the power of an internal developer portal.

7 Summary

In the first part of the tutorial, we reviewed the features of developer portals, learned how to use the opensource Backstage tool to create a portal ourselves, configured our portal with GitHub, created a software template, and bootstrapped a service from it.

A developer portal could be much more than this if more and more integrations are added. Imagine if you deploy services in Kubernetes, use Argo CD for GitOps deployment, use HashiCorp Vault for secrets management, and all the integrations are in your portal: when you need to check the deployment status, when you want to see the actual resources in K8s, you don't have to visit Vault, Argo CD, K8s Dashboard. Heck, you don't even have to remember the URLs for them or even know the existence of those things because you have the one tool to rule them all, and that's the developer portal.

This tutorial only works for a local quick start; there are many things to think about for production usage. For example, we use static config for now; nothing is persistent; if you restart the dev server, the catalog info is lost. For this, we need to configure Postgres for our portal. For another example, we use yarn dev to start both the frontend and backend; for production usage, you might want to separate the frontend from the backend, deploy them in K8s as containers, and maybe create Ingress and stuff for them.

In the next part of this tutorial, we will look at the Backstage plugin's mechanism to see how we can expand its power to the next level.

If you enjoy this article, please like, comment, and subscribe. See you in the next part!

Platform Engineering: Building Your Developer Portal with Backstage (Pt 1)