How Custom GitHub Actions Enabled Us to Streamline Thousands of CI/CD Pipelines

Gabriel L. Manor - Sep 18 - - Dev Community

Introduction

Since the early days of Permit.io, we have understood that building a secure and reliable Authorization service depends on how it integrates with the software development lifecycle. For this reason, besides providing our users with comprehensive authorization as a service product, we invest heavily in solving the challenge of automating CI/CD pipelines while maintaining consistent permission management across various environments.

Our main goal with this holistic Authorization approach is to ensure these processes are not only seamless but also accessible and efficient for any development team.

By integrating GitHub Actions with our APIs, we've created a model that supports development organizations at scale. This blog will walk you through how we leveraged these tools to automate environment management in our workspace, making it accessible to CI/CD pipelines.

Understanding Your Architecture for the CI/CD

The first and (maybe the most important) step in creating a GitHub action is to examine your system’s architecture and understand the structure you aim to streamline in your CI/CD.

When building Permit.io, we’ve built a system that allows developers to externalize their authorization to our hybrid cloud service. While the core of the product is, of course, letting the application code decide what users are allowed or not allowed to do, it was also built to support the common hierarchies of software development organizations.

This is evident in Permit’s workspace hierarchy:

  • Workspace - comparable the the whole development organization. You can think of it as a GitHub organization.
  • Project - a single application that your organization manages. It can be thought of as a code repository, but it is also flexible enough to support either multi- or mono-repo architectures. A workspace can include multiple projects.
  • Environment - the silo of policies and configurations inside each project. The environment allows every organization to manage their CI/CD process in the same way they are managing it in their SDLC. It is comparable to a Git branch. Each project has a default of Production and Development branches, but a user can create as many as they want.

api-model-f212cef178cf6984bd6009c26d0b62e8.svg

The Advantages of a CI/CD Structure

Understanding the structure of your CI/CD process is halfway to solving the streamlining process. The environment-project model is built to provide flexibility and control across multiple stages of development. At its core, it allows our users to:

  1. Create Isolated Environments: Each environment is independent yet can replicate production settings, making it ideal for testing and development.
  2. Manage Permissions Seamlessly: Permission management is consistent across environments, ensuring that your CI/CD process is both secure and reliable.
  3. Automate Workflows: Permit integrates smoothly with CI/CD tools like GitHub Actions, enabling automated environment creation, testing, merging, and cleanup.

As mentioned, this model is especially useful in scenarios where environments need to be dynamically created, tested, and merged—making it perfect for PR-based workflows.

With this model in mind, let’s identify the automation and streamlining challenges.

Workflow Friction: Identifying the Challenges

To fully support the seamless integration of an authorization service to existing users' environments and architecture, we need to identify the events and connection points of our permissions system with the software development system.

At the basic level, a new environment needs to be created in Permit for every pull request (PR). This allows isolating configuration from the production and staging environments, running tests to ensure branch and environment protection, and merging changes into production, all with minimal manual intervention.

This process involves the following steps:

  1. Environment Creation: We want to let our users automate the process of creating a new Permit environment for each PR. We also understand that not every PR should have an authorization configuration change, so we want to ensure that we are opening new environments only for relevant PRs.
  2. Environment Testing: For each PR, we want to run integration and product tests against the relevant environment to ensure the accuracy of permissions and settings.
  3. Merging and Cleanup: After a PR is approved, we want to merge the Permit environment with the production environment. This will allow us to ensure streamlined deployment and delivery.

Manually managing these tasks can be error-prone and time-consuming, compelling us to find a reliable way to automate them. With CI/CD in mind, we want to streamline the continuous integration of the first two steps and the continuous deployment and delivery of the last.

Now that we understand the requirements let’s create our GitHub action:

Creating a Custom GitHub Action

As described, our GitHub Action incorporates three workflows that fully automate the environment management process:

  1. new-env.yaml : Creates a new environment when a PR is opened or updated.
  2. run-pdp.yaml : Runs tests in the newly created environment.
  3. merging.yaml : Merges the environment changes into production and then deletes the PR-specific environment.

These workflows utilize Permit’s API to dynamically manage environments, providing a streamlined and error-free process. You can see this workflow in action in the following code repository: https://github.com/permitio/pink-mobile-demo-app/tree/main/.github/workflows

Let’s take a look at the code and understand its steps -

Workflow Breakdown

1. Creating a New Environment (new-env.yaml)

When a new PR is opened or updated, we trigger a workflow responsible for creating a new environment within Permit. We will create this environment using Permit’s copy environment API to replicate the production settings, ensuring consistency across all environments.

Trigger the Workflow:

The first step of running our workflow is to trigger it as a result of events in our code. We can do this by using the following command:

name: demo-new-env
on:
  pull_request:
    types: [opened, reopened]
Enter fullscreen mode Exit fullscreen mode

This will ensure our workflow is triggered. In the next step, we’ll set it up with some variables.

Set up the environment:

Define the project ID as an environment variable. This will be used in subsequent API calls.

This project is the one the environment will be created in. In production, you can get it from a GitHub variable.

env:
  PROJECT_ID: 1238e459351a8470
Enter fullscreen mode Exit fullscreen mode

Check Creation Relevancy

To ensure we are not opening a redundant environment in Permit for PRs that are not relevant for permissions changes, you’ll need to label PRs with a Permissions tag. This will serve as a condition for creating an environment:

jobs:
  demo-new-env:
    if: contains( github.event.pull_request.labels.*.name, 'permissions')
Enter fullscreen mode Exit fullscreen mode

Extract the Branch Name:

Next, extract the branch name. We will need it to name our new environment:

- name: Extract branch name
  run: echo "branch=${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}}" >> $GITHUB_OUTPUT
  id: extract_branch
Enter fullscreen mode Exit fullscreen mode

Create the New Environment:

Create a new environment in Permit. This is done by posting the environment details, including a unique key derived from the branch name:

- name: Create new environment
  run: |
    response=$(curl -X POST \\
      <https://api.permit.io/v2/projects/$>{{ env.PROJECT_ID }}/envs \\
      -H 'Authorization: Bearer ${{ secrets.PROJECT_API_KEY }}' \\
      -H 'Content-Type: application/json' \\
      -d '{
        "key": "pr-${{ steps.extract_branch.outputs.branch }}",
        "name": "pr-${{ steps.extract_branch.outputs.branch }}"
      }')

    echo "ENV_ID=$(echo "$response" | jq -r '.id')" >> $GITHUB_ENV
    echo "New env ID: $(echo "$response" | jq -r '.id')"
Enter fullscreen mode Exit fullscreen mode

Fetch the Environment API Key:

The workflow will retrieve the API key for the newly created environment, which will be used in subsequent workflows:

- name: Fetch API Key of the new environment
  run: |
    response=$(curl -X GET \\
      <https://api.permit.io/v2/api-key/$>{{ env.PROJECT_ID }}/${{ env.ENV_ID }} \\
      -H 'Authorization: Bearer ${{ secrets.PROJECT_API_KEY }}')

    ENV_API_KEY=$(echo "$response" | jq -r '.secret')
    echo "ENV_API_KEY=$ENV_API_KEY" >> $GITHUB_ENV
    echo "New env API key: $ENV_API_KEY"
Enter fullscreen mode Exit fullscreen mode

2. Running Tests in the New Environment (run-pdp.yaml)

For our next step in continuous integration, we would like to let our pipeline test the code against the newly created environment. One key aspect of Permit’s architecture is using a local policy decision point (PDP) that runs together with the application.

In this workflow, we want to run the PDP in the GitHub action. This way, the tests can call it. We’ll also use this step to set up some mock data in the Permit system:

Triggering the Workflow:

This workflow will run whenever a PR is synchronized, ensuring that any updates are tested:

name: demo-run-pdp-and-tests

on:
  pull_request:
    types: [synchronize]
Enter fullscreen mode Exit fullscreen mode

Fetching the Environment ID and API Key:

Next, we’ll fetch the existing environment ID created in the previous workflow:

- name: Fetch environment ID and API key
  run: |
    response=$(curl -s -X GET <https://api.permit.io/v2/projects/$>{{ env.PROJECT_ID }}/envs \\
      -H 'Authorization: Bearer ${{ secrets.PROJECT_API_KEY }}' \\
      -H 'Content-Type: application/json')
    EXISTING_ID=$(echo "$response" | jq -r '.[] | select(.key == "pr-${{ steps.extract_branch.outputs.branch }}") | .id')
    echo "EXISTING_ID=$EXISTING_ID" >> $GITHUB_ENV
Enter fullscreen mode Exit fullscreen mode

Running the Local PDP Instance:

Then, a local instance of the PDP needs to be spun up using Docker, allowing tests to be run against the newly created environment:

- name: local PDP running
  run: docker run -d -p 7766:7000 --env PDP_API_KEY=${{ env.ENV_API_KEY }} --env PDP_DEBUG=true permitio/pdp-v2:latest
Enter fullscreen mode Exit fullscreen mode

Executing the Tests:

Finally, we can execute tests against the local PDP instance using the environment’s API key:

- name: Run tests
  run: PERMIT_TOKEN=${{ env.ENV_API_KEY }} PDP_URL=localhost:7766 npm run test
Enter fullscreen mode Exit fullscreen mode

3. Merging Changes and Cleanup (merging.yaml)

The last step of our custom GitHub Action is to ensure an efficient and streamlined deployment and delivery process. This is where we will trigger the merge workflow, which will ensure we are safely integrating the created environment into our production environment in Permit.

Triggering the Workflow:

This workflow is triggered when a PR is closed, signaling that it’s time to merge and clean up:

name: demo-merging
on:
  pull_request:
    types: [closed]
Enter fullscreen mode Exit fullscreen mode

Fetching Production and PR Environment IDs:

Next, fetch the IDs of the production environment and the PR-specific environment:

- name: Fetch production and PR environment IDs
  run: |
    response=$(curl -s -X GET <https://api.permit.io/v2/projects/$>{{ env.PROJECT_ID }}/envs \\
      -H 'Authorization: Bearer ${{ secrets.PROJECT_API_KEY }}' \\
      -H 'Content-Type: application/json')
    PROD_ID=$(echo "$response" | jq -r '.[] | select(.key == "production") | .id')
    echo "PROD_ID=$PROD_ID" >> $GITHUB_ENV
Enter fullscreen mode Exit fullscreen mode

Merging and Deleting the PR Environment:

Finally, merge the PR-specific environment into production and then delete the PR environment, ensuring no residual environments remain:

- name: Merge and delete PR environment
  run: |
    curl -X POST <https://api.permit.io/v2/projects/$>{{ env.PROJECT_ID }}/envs/${{ env.EXISTING_ID }}/copy \\
      -H 'Authorization: Bearer ${{ secrets.PROJECT_API_KEY }}' \\
      -H 'Content-Type: application/json' \\
      -d '{
        "target_env": {
            "existing": "${{ env.PROD_ID }}"
        }
    }'

    curl -X DELETE \\
      <https://api.permit.io/v2/projects/$>{{ env.PROJECT_ID }}/envs/${{ env.EXISTING_ID }} \\
      -H 'Authorization: Bearer ${{ secrets.PROJECT_API_KEY }}'
Enter fullscreen mode Exit fullscreen mode

Our Custom GitHub Action

Looking at the three steps we just created, we can easily see how this action is streamlined the whole process of our authorization service into our users’ architecture. Here's how we tackled them using our custom GitHub Actions:

  1. Environment Creation: The process of creating new environments for every pull request (PR) was automated through the new-env.yaml workflow. This action triggers when a relevant PR is opened or updated, automating the creation of a new environment in Permit. By using Permit’s API, we replicate production settings in the new environment, ensuring consistency across the different stages. A condition was also added to avoid unnecessary environment creation for PRs unrelated to permission changes, using a label system to filter relevant PRs.

  2. Environment Testing: The run-pdp.yaml workflow addresses the need to run tests on the newly created environment. Once a new environment is set up, the workflow spins up a local Policy Decision Point (PDP) instance via Docker. This allows integration and product tests to run against the environment, ensuring that permission settings are accurate before merging changes into production. This automated testing prevents errors and guarantees that only well-tested changes move forward.

  3. Merging and Cleanup: After a PR is approved and closed, the merging.yaml workflow merges the PR-specific environment into the production environment, streamlining the deployment process. Following the merge, the workflow deletes the PR-specific environment, ensuring that no unnecessary environments linger, thereby reducing clutter and maintaining a clean development setup.

By automating these steps—environment creation, testing, merging, and cleanup—we significantly reduced manual intervention, minimized errors, and sped up the entire CI/CD process.

Conclusion

By integrating GitHub Actions with Permit.io APIs, we were able to streamline our CI/CD process and make our environment model accessible and efficient for development teams. This approach not only reduced manual effort but also minimized the risk of errors, providing a smoother and more reliable deployment process.

For any team looking to optimize their CI/CD pipeline, Permit.io’s flexible environment model combined with GitHub Actions offers a powerful solution to automate and secure your development workflows.

You can read more in our docs about how we handle Projects & Environments, or try out Permit.io for yourself!

If you have any questions, join our Slack community, where there are thousands of devs building and implementing authorization.

Image
Chatbot uses and its Effect
Image
GitHub Actions for Beginners: Essential Concepts and First Steps

github, githubactions, softwaredevelopment, cicd
Image
Boosting Incident Response Capabilities with Azure: A Practical Guide
Image
The Secrets of the 'delete' Operator in JavaScript

javascript, webdev, programming, frontend
Image
Back to development after 3 months
Image
Loading data to Google Big Query using Dataproc workflow templates and cloud Schedule
Image
Dive into the World of Decentralized Apps with Master Solidity! 🚀
Image
New to programming
Image
A Compelling Case for the Comma Operator
Image
The Start to my Open Source Development Journey
Image
One-To-One Relation in Two Different Entities using Asp .Net 8 Web Api (Introduction)
Image
Student Dev First Blog Post
Image
Debug Mode: Django Docker Pycharm
Image
React State
Image
Configuring AWS VPC, Load Balancers, and DNS for WordPress and Moodle Integration
Image
Why Real-Time Data in Bitcoin API Matters
Image
Unit, Integration, and Functional Testing: 4 main points of difference

testing, mobile, webdev, programming
Image
Solar System
Image
The Best Online Stores for Disposable Vapes: A Comprehensive Guide
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Terabox Video Player