Hello there, and welcome to another Adventure of Blink! This season we're building the game "Hangman" with a full stack of devops tools and technologies... it's going to be a wild ride!

Today we're talking about the design of our game. We've drawn this out at a high level in the previous episode, but I made a small modification in that design. Here's what it looks like now:

What's different? I moved the API application into the Docker environment. Why would I do this? Because the database and the API are tightly coupled with each other. The API code would never be deployed without the database in place, and we wouldn't ever make a change to the structure of the database without changing the API. With that in mind, it makes sense to run the API inside the same container environment (although in a separate container) as the database.

Youtube Ride-along

How to play along at home

If you're interested in following this season's build, you can find the code for each episode on my GitHub. A little note about how this repo works: Each episode will have its own branch so that you can see the code built in that specific episode. Additionally, I'll merge each episode's branch into main when it releases so that main will always reflect the latest point in our journey.

Now it's time to build an API

I've seen many people feel daunted when we start to talk about an API. It's my hope that we can dispel this mythical complexity today.

API is an acronym for "Application Programming Interface". Sounds fancy, doesn't it? It can feel like a Star Trek episode where there are inverse tachyons causing a feedback cascade in the main deflector dish or something... but let's speak a little more plainly.

An Application Programming Interface is simply a set of rules for how a program can interact with another. That sounds a little less intimidating, doesn't it?

In the case of a database, an API is just a program that allows a program to easily communicate with the database.

Why would we build a program for a program to use? Isn't this overkill?

We could just build our game with database calls directly embedded in it, and skip the need for an API. And in the case of hangman, it's a perfectly rational decision to make.

But we're not here just to build the game. We're learning about architectural principles! So... why would we want the API separate from the application code?

Separation of Concerns

We build an API layer for the database because we don't want all of our logic to get mixed together inside the application. If database calls become part of the frontend logic, we can create problems for ourselves later; "separating concerns" is a design principle that we use to make sure it's easy to extend our code later on. When we don't separate concerns, we can run into problems with the addition of new features.

Ease of Maintenance

Imagine if a new version of Mongo DB comes out, and it changes how a certain very common database call is made. If we don't have an API layer in our code, we have to find every call made to the database scattered throughout the frontend code. On the other hand, having an API layer means that the actual calls to the database are ONLY ever made from within the API layer. This is much easier to test and ensure that we haven't missed a change. Our application is safer to run and less likely to have an error in production because of good design.

Scalability

What happens if we need more database resources? What happens if we need more frontend resources? If we haven't separated our concerns, we're not able to efficiently scale up - we have to do all, or none. With an API layer in place, we get to choose where to allocate our resources.

A brief correction to last week's code

In setting up for this week's episode, I found a couple mistakes I wanted to highlight! First, there were a couple of things wrong with my init_mongo.sh script:

Recently, the mongo command was removed from the runtime and you now use mongosh.
In bash, the if syntax does NOT contain a colon (:)

Here are my updates to that script so that it will run properly:

#!/bin/bash

COLLECTION_EXISTS=$(mongosh --username $MONGO_INITDB_ROOT_USERNAME --password $MONGO_INITDB_ROOT_PASSWORD --authenticationDatabase admin --quiet --eval "db.getSiblingDB('$DB_NAME').getCollectionNames().includes('$COLLECTION_NAME')")

if [ "$COLLECTION_EXISTS" = "false" ]
then
    echo "Collection $COLLECTION_NAME does not exist.  Initializing..."
    mongosh --username $MONGO_INITDB_ROOT_USERNAME --password $MONGO_INITDB_ROOT_PASSWORD --authenticationDatabase admin --eval "db.getSiblingDB('$DB_NAME').createCollection('$COLLECTION_NAME')"
    echo "Collection $COLLECTION_NAME created."
else
    echo "Collection $COLLECTION_NAME already exists."
fi

echo "Mongo initialization script ran" > /tmp/startup_complete.txt

Note that I also added a startup_complete.txt file in /tmp/ in order to have a quick spot-check that things initialized correctly!

Also, there was no need to pass the .env file in the database's Dockerfile. Here's my update for that:

FROM mongo:latest

ENV MONGO_INITDB_ROOT_USERNAME=${MONGO_INITDB_ROOT_USERNAME}
ENV MONGO_INITDB_ROOT_PASSWORD=${MONGO_INITDB_ROOT_PASSWORD}

COPY ./database/init_mongo.sh /docker-entrypoint-initdb.d/init_mongo.sh
RUN chmod -x /docker-entrypoint-initdb.d/init_mongo.sh

EXPOSE 27017

CMD ["mongod"]

Finally, I changed the build context on the mongo service in the docker-compose.yml as follows:

  mongo:
    build:
      context: .
      dockerfile: ./database/Dockerfile
    container_name: mongo_container
    restart: unless-stopped
    env_file:
      - .env
    volumes:
      - ./database/mongo_data:/data/db
    ports:
      - "27017:27017"

This ensures the build context is set correctly for how the mongo database container needs to start. Ok, now let's move on to the new stuff!

Building a Python API

We're going to use Flask, a Python framework that makes it easy to build an API. To get started, we're going to add another folder to our project:

This db-api subfolder will have three files inside:

api.py
Dockerfile
requirements.txt

This will contain the api code, and then a Dockerfile that builds a container to run it. Finally we'll have to list the libraries that the api requires in order to run, so that we can rebuild it in the Docker container. We'll also have to make some modifications to our docker-compose yaml file in the project root... Let's walk through our edits.

Create a Flask API

We're going to have (for starters, at least) two things you can do with the API: Add a new phrase to the collection and Randomly select one of the phrases for use. Here's our code, with comments to explain what's up:

# We have lots of dependencies, but there are two major frameworks 
# in play: flask (for API creation) and pymongo (for database 
# manipulation)
from flask import Flask, jsonify, request
from pymongo import MongoClient
from pymongo.errors import PyMongoError
from bson.objectid import ObjectId
from datetime import datetime
import os

# This defines the Flask object, which initializes the Flask framework
app = Flask(__name__)

# Here are the environment variables (we're storing them in .env with our other constants).  You need the URI, the database name, and the collection name (analogous to a "table" in other databases)
mongo_uri = os.getenv("MONGO_URI_API")
db_name = os.getenv("DB_NAME")
collection_name = os.getenv("COLLECTION_NAME")

# When testing locally, we bypass the .env and load the variables manually.  These are commented out to show you what the content of the variables would look like in .env.
# mongo_uri = "mongodb://blink:theadventuresofblink@localhost:27017/hangman?authSource=admin"
# db_name = "hangman"
# collection_name = "phrases"

# We need an object that allows us to communicate to the database.  Mongo makes this an easy initialization, just pass the URI.  Then we want variables to hold the database and collection info for easy access later.
client = MongoClient(mongo_uri)
db = client[db_name]
collection = db[collection_name]

# A "route" is a path that you add to the end of the URI when you want to access an API method.  If you're running your api on localhost port 5001 like I am, you'd access this method by the following:
# http://localhost:5001/add
# Of course, this is a POST, so you'd be required to create a proper HTTP POST request body with all the stuff - details later in the testing part!  After defining the route, you create the Python method just like any old Python program.
@app.route('/add', methods=['POST'])
def add_item():
    # POST requests come with parameters that we need.  Go get them.
    data = request.get_json()

    # Validate that 'phrase' is one of the parameters provided.
    if 'phrase' not in data:
        return jsonify({"error": "Missing required parameter 'phrase'"}), 400

    # Validate that 'hint' is one of the parameters provided.
    if 'hint' not in data:
        return jsonify({"error": "Missing required parameter 'hint'"}), 400

    # Build the JSON for the new item we're going to add to our collection.  Note that we're auto-creating a couple of parameters here too for our metrics purposes!
    new_item = {
        "phrase": data['phrase'],
        "hint": data['hint'],
        "last_used": None,  # Set to null initially since it hasn't been used yet
        "access_count": 0  # Start the counter at 0
    }

    # Put the new item in the mongo collection.  This is like "adding a row to a table" in a relational DB.
    collection.insert_one(new_item)
    # We need to respond to the POST so that the requester gets feedback on the success of our addition.  HTTP 201 == "Created"
    return jsonify({"status": "Item added", "id": str(new_item['_id'])}), 201


# This is our second route - on this one we're doing an HTTP GET so we don't **require** any parameters.  But we could have them if we needed them!  In this case, we're just asking for a random item from the collection so we don't need any.
@app.route('/random', methods=['GET'])
def get_random_item():
    try:
        # collection.aggregate gives you the ability to select a random item or list of items.  We just want one.
        random_item = list(collection.aggregate([{"$sample": {"size": 1}}]))

        # Did we get one, or was there an error?  If we got one, 
        # we're going to do a few things here:
        # 1 - Update the record's metrics to reflect that it's been
        #     chosen
        # 2 - Return the selected item to the requester
        if random_item:
            random_item = random_item[0]

            # Ensure _id is correctly handled as ObjectId
            record_id = random_item['_id']

            # Update the record with the current timestamp and increment the counter
            collection.update_one(
                {"_id": ObjectId(record_id)},
                {
                    "$set": {"last_used": datetime.now()},
                    "$inc": {"access_count": 1}
                }
            )

            # Return the updated document
            # This _id manipulation here is required because Mongo types the _id field in a collection as an Object rather than a string, and jsonify() won't handle it.  So we convert it here to a string to help it along.
            random_item['_id'] = str(random_item['_id'])
            random_item['last_used'] = datetime.now()
            random_item['access_count'] += 1
            # Here's your HTTP 200 OK that also returns the item.
            return jsonify(random_item), 200

        # This is the 'else' of the 'if random_item' above - if we get to this line we need to let them know we didn't find an item (it's most likely the collection is empty)
        return jsonify({"error": "No items found"}), 404
    # If we get an exception, we need to report it as a HTTP 500 so there's record that something went wrong.
    except PyMongoError as e:
        return jsonify({"error": str(e)}), 500

# This is your python "glue" for running a program
if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5001)

requirements.txt: Managed Dependencies

If you've done any Python coding before, you've probably encountered requirements.txt. This is how you list the libraries your program needs to have present in order to run, so that you can easily share your code with someone else. Here are the contents of our API's requirements.txt file:

Flask==2.3.2
pymongo==4.5.0
python-dotenv==1.0.0

Create a Docker container with our API running in it

Now we need to run our API. Since we wrote our code in Python, we should base our container on a Python container. Here's our Dockerfile, with comments:

# Use an official Python runtime as a parent image.  I selected 3.12 because it's what I have installed locally, so to minimize the risk of compatibility issues, I matched it.
FROM python:3.12-slim

# We're going to put all of our code in /app inside the container.
WORKDIR /app

# Copy the current directory contents (our code) into the container at /app
COPY . /app

# Install the necessary dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Expose the port that Flask will run on
EXPOSE 5001

# Define environment variables for Flask
ENV FLASK_APP=api
ENV FLASK_ENV=development
ENV PYTHONPATH=/app

# Run Flask app when the container launches
CMD ["flask", "run", "--host=0.0.0.0", "--port=5001"]

Update Docker-Compose.yml

We want our API to be orchestrated alongside the mongo database... that is, we know that we want the mongo container to be up and running, and then we want to start up our API container afterward. We do that by adding a new service to our docker-compose.yml file:

  hangman-api:
    # This tells the application that it's going to build the container based on the contents of /db-api
    build:
      context: ./db-api
      dockerfile: Dockerfile
    # Our container will be named hangman-api
    container_name: hangman-api
    # Like the mongo, it will try to auto-restart if it fails
    restart: unless-stopped
    # We're sending the same .env file to both containers
    env_file:
      - .env
    # Port 5001 is where you'll be able to access the API.
    ports:
      - "5001:5001"
    # A VERY important line - this tells docker to only start this container AFTER mongo is running!
    depends_on:
      - mongo
    # This tells the container where to get its source code, and where on the container's file system the code will be stored
    volumes:
      - ./db-api:/app
    # The environment entry defines things that this container needs to access.  Adding the MONGO URI here allows this container to connect to that one as defined, so that the API can call the database.
    environment:
      - MONGO_URI=${MONGO_URI_API}

Time to test

This has been a pretty extensive build, hasn't it? If you stuck with me to the end of this process, I appreciate you! Now that everything's configured, you should be able to start your entire app environment like this:

# Mac
docker compose up --build

# Windows / Linux
docker-compose up --build

Once everything has started, you can see two containers running in your terminal when you use docker ps:

ben@Anduril ~ % docker ps
CONTAINER ID   IMAGE                    COMMAND                  CREATED             STATUS             PORTS                      NAMES
e753dc7d8900   s2-hangman-hangman-api   "flask run --host=0.…"   About an hour ago   Up About an hour   0.0.0.0:5001->5001/tcp     hangman-api
acfc65b3f0e1   s2-hangman-mongo         "docker-entrypoint.s…"   About an hour ago   Up About an hour   0.0.0.0:27017->27017/tcp   mongo_container

The final test is of course being able to call the API and update the database! Since we don't yet have a frontend app to call it from, here's how to test from the terminal:

Add an item to the collection

curl -X POST http://localhost:5000/add \
-H "Content-Type: application/json" \
-d '{"phrase": "hello world", "hint": "programmer quote"}'

Retrieve a random item from the collection

curl http://localhost:5001/random

Try it out and see what you get!

Wrapping up

Episode 3 was intense, friends! We had an empty database, and now we have a fully-functional API in front of it that allows us to add new items and select one at random. Both of these things are running in Docker containers, and the whole thing can be started with a single command. That's what DevOps is all about - making it easy to keep doing awesome stuff. Building from the ground up like this and including our Operational needs in the considerations we have during Development builds better software overall.

I hope you're enjoying this journey along with me! Tune in next week as we begin to create our game!

The Adventures of Blink S2e3: Designing Hangman... and a Flask API