You want to try out feature flags but don’t want to use an external service. Perhaps it’d be better to use an open-source alternative, or even build your own? In this article, we explore the advantages and disadvantages of using either a homegrown, open-source, or hosted feature management platform.

You’re reading this because you want to use feature flags in your current project.

Maybe you used them before and saw how much easier life is when you can separate deployment from release.

Maybe you’ve just heard about feature flags and want to try out this nifty “testing in production” idea. (No, it doesn’t just mean “release bugs to everyone before you find them”.)

Great! Just sign up for a hosted feature flag service—or feature management platform, as we call it. (We’ll use the acronym “FMP” for the rest of this article.) It just so happens I have a lovely one right here! It’s called Lau… no?

Oh, you don’t want to sign up with a commercial service? Don’t worry, you’re not the only one.

There are many reasons why you might prefer to run your own FMP instead of buying one. And some of those reasons are actually good. For example:

• you’re on a very tight budget
• you don’t want to rely on a service that isn’t Free or Open Source Software
• you want absolute control over all the systems that control your code’s execution
• you can’t buy services without a procurement process, which takes so long
• THEY’RE JUST A FEW BOOLEANS! Why on earth would you want someone else to host your booleans for you? Just put them in app configuration!

We sympathize with that last point. Many of us at LaunchDarkly took the same incredulous posture before we joined the company, but oh boy did we soon learn how wrong we’d been. Even a basic FMP does more than just hold some boolean state and share it out when needed.

To demonstrate the ups and downs of creating your own FMP, I’ll walk you through a couple of different scenarios of what that journey might look like. But before we set off, let’s make a quick pit stop to relieve you of a common misconception.

Feature flags != configuration

When deploying a server-side application, the deployment usually includes a set of configuration files. The config and code are bundled together for the deployment, and the config usually doesn’t change until the next deployment. Even if some config is read from somewhere other than the bundled files, the app won’t read it again until it restarts.

In contrast, feature flags can change at any time. Flags are used to control app behavior, and so when the flag changes, the app behavior should change immediately—or, at least, within a few seconds. No redeployment or restart is required. It doesn’t mean that the app code is replaced; rather, that the next time the app evaluates a flag in a conditional, the returned value should be the new one. (If you try to replicate this effect with config, you’ll soon find some problems—such as previously-unknown caching layers—which show why it’s a bad idea. Yes, even with “serverless” code.)

The difference between the behavior of flags and config also shows their different uses. Config should be used for everything that the app needs to start, including connections to essential resources such as the FMP. Feature flags should be used for behavioral decisions that the app makes after it has initialized.

Are you feeling better now that that’s all cleared up? Me too. Back to the journey.

Breaking down feature flags

So, we’re off to see the FMP… but how will we recognize it when we get there?

To work out the qualifications for a feature management platform, we have to reduce it to its simplest possible form. Forget about multivariate flags, experimentation, integrations, or any other high-value features. What’s the most basic set of functionality that still makes feature flags worthwhile?

• flags are evaluated in your app code, which it can’t do without an API
• flag definitions and values should stick, so: a persistent flag store
• team members need a user interface to create and flip flags
• keep track of changes with an audit trail—vital for debugging!
• finally, architect the API protocol to provide those immediate flag updates

For the purposes of this article, that’s our Minimum Viable FMP. Doesn’t sound like much, does it?

Give me a home where the booleans roam

We started your journey at the fork in the road marked  “Build vs. Buy”. You’ve chosen the “Build” path. The next fork comes almost immediately—how are you going to implement your FMP? Your choices are:

1. code it all yourself
2. use an existing Free or Open Source system

Let’s look at each option in more detail, starting with the one that tempts so many engineers: code it all yourself.

Option 1: Code your own

Implementing a Minimum Viable FMP isn’t particularly challenging, especially for an experienced, competent engineer like you. Not only are you accomplished, but you love taking on new side projects. Why not jump into another one?

I’m using an imaginary C-like programming language for the code samples here. Feel free to translate to the language of your choice.

The flag evaluation API can be very simple: it only needs commands to connect to the FMP and, more importantly, evaluate a named flag. Evaluation can just be something like:

bool myFlagValue = eval_flag(“flag_id”);

… and that’s it!

Flag state has to live somewhere persistent, so you need a flag store. Database systems are good for this, and you can build your API on top of the existing database API, and you get your network protocol for free. So that’s nice!

func eval_flag(string flag_id) {
  // Let’s pretend that database queries never fail
  string sqlQuery = “SELECT value FROM flags WHERE flag_id = ?”;
  query flagQuery = DB::exec_query(sqlQuery, flag_id);
  bool flagValue = query.cursor.readNext();
  return flagValue;
}

Ah, but: if you want a clean separation between your new experimental flag store and your app’s precious primary DB, then the flag store should be in its own DB. And the whole thing’s still experimental, and it might go down. As we do in LaunchDarkly’s API, we recommend that your flag evaluation calls include a fallback value to use when the FMP is unavailable:

// Let’s stop pretending that database queries never fail
bool myFlagValue = eval_flag(“flag_id”, false); // See that false?
// It's the fallback value.

Unfortunately, you don’t get a user interface for free. Your organization needs an accessible way to create and flip flags without having to live-code SQL statements. We tend to see this as a web app, but command-line tools may be enough to start. If your app already has an easily-extensible control interface, you can put it there.

[ imagine a pile of web interface code, 
  including a big nasty HTML table ]

And now that you have multiple team members flipping flags (which you should, because you’re too busy to be the sole person with access), you really need an audit trail to keep track of who flipped which flag and when they did it. It could be another database table, maybe a log file. Just make sure it’s easy to access when you’re frantically trying to debug problems.

Those are the essential components, but don’t forget one vital feature I mentioned earlier: flag changes should trigger behavioral changes immediately. If you’re storing the flags in an external database, you could just query the database for each flag evaluation, but that’s going to affect performance and deter flag usage. Instead of making flag evaluation expensive, cache the flags in memory and keep them updated with a separate thread.

Most flag systems check for flag updates with regular background polling. Unfortunately, this forces a trade-off between change latency and load: when you lower one, you raise the other. The alternative, which LaunchDarkly uses, is to hold an open connection between the app and the FMP. The app initiates the connection as a continual stream which triggers update events every time the FMP sends a flag change. This reduces the change latency to a few milliseconds and removes the load caused by continual polling. The downside: it’s more work to implement on the FMP, but these days there are plenty of libraries to help you.

Now that I’ve described all the work you need to do, how long would it take you? Depending on the tools you use, I’d guess at anything between a couple of hours (at the most optimistic) to a few days. That’s just for the first version, with an API client for one language. Once you add in the feature requests and regular tinkering, it still might stay under a week’s work per year…maybe.

And you love taking on new side projects—after all, what are Git repositories for? (“Ah, solving that question / Brings the priest and the doctor / In their long coats / Running over the fields.” So wrote Philip Larkin, who never finished his cool to-do list app.)

So what’s the big deal with commercial alternatives? Why pay for an FMP when you can build one so easily?

Well, there are a couple of reasons.

Firstly, this FMP is so basic that it won’t even let you do testing in production. That’s the nifty maneuver in which you:

1. code an enhancement to your app
2. wrap the enhancement in a feature flag
3. deploy the new code to production with the enhancement disabled: it’s deployed but not released, because nobody can use it yet
4. enable the enhancement but only for the dev and QA teams! It lets you test how your code behaves in the production environment without exposing potential bugs to your users

It’s an incredibly useful trick, but it’s only possible if you’re able to evaluate the same flag differently for different users. That’s a bit more work.

The second reason why one might pay for a commercial FMP is that building an FMP yourself is still expensive. There are some other costs we’ve not factored in yet. We’ll encounter those in a moment, when we look at the other option for our journey: open-source software.

Option 2: Don’t reinvent the wheel

Why bother coding a basic FMP when others have already created better ones and made their code freely available? It’s a far better idea to take advantage of projects that already exist.

If you don’t want to write an FMP from scratch, but you still want full control over the code and how it’s run, then take a look through what’s available as Free and Open Source Software (commonly abbreviated to FOSS).

We love FOSS at LaunchDarkly—our service wouldn’t exist without it. While our core FMP product is our own proprietary creation, the underlying infrastructure is full of FOSS. We proudly contribute back to the open software ecosystem, both financially and with much of our own work.

There are plenty of FMPs out there with FOSS licenses. Most of them are pretty simple; light on features but also limited to a specific language or application framework. That simplicity isn’t necessarily bad; being written for a particular language may mean that the API makes good use of that language’s idioms.

Here are some examples of language/framework-specific feature management platforms:

But if your team uses more than one programming language, you’ll need an FMP that caters to them, and is available as a network service with a web user interface. These systems offer client SDKs for multiple languages as well as network APIs (using REST or GRPC) so they can be accessed by other platforms:

(If you’re wondering about the “Update checks” column: it’s the method that the client SDKs use to check for flag changes. As I discussed earlier, polling causes performance issues but is easier to implement; a persistent stream connection is better.)

Some of these polyglot FMPs are great. They have plenty more features than the minimum viable FMP we constructed earlier.  And they’re free! Well… kind of.

As the popular saying goes, the “Free” in Free Software means “free as in speech, not as in beer”. In other words, the code is free of restrictions for usage. There is a monetary cost, which includes not only the cost of infrastructure used to run it but also something much more expensive: your time.

To paraphrase a certain software-engineer-turned-nightclub-owner: open-source software is only free if your time has no value. The time-consuming activities include:

• installing and configuring the FMP
• getting agreement from the operations team, which has to adopt the FMP as another component of the production system
• ensuring that enough other people know how to maintain the FMP

If you’re not paying a vendor to look after your FMP, then you’re either taking on that responsibility yourself, or giving it to others in your organization. It’s “free as in puppies”.

It adds up to a lot of work, which adds up to a lot of money, because your time is expensive. It’s not just about the financial aspect: think about what you’re paying in opportunity cost, which is all the tasks you don’t complete because you’re busy with this one. Those other tasks are probably more relevant to your organization’s core purpose, and you have more expertise in them.

When you consider those costs, you can see why it’s cheaper to pay people who are already running an FMP service than to spend the time getting expertise in it yourself. Take it from Bevan Blackie, the Development Manager for Jira Cloud at Atlassian (which uses LaunchDarkly): “My teams can focus on things that are really going to add value to our customers, rather than worrying about other things that are coupled but not directly related to a specific feature. This translates to teams spending more time focusing on the items that differentiate us as a company.”

Note that a couple of the major FMPs discussed above, such as Bullet Train and Unleash, offer managed hosting which you can pay for. It’s tempting to start by hosting one of them yourself, and then migrating to paid hosting if needed. The problem is that you’re front-loading that same expertise and time cost, paying for it yourself, and then paying for hosting on top of that. If you’re trying to save money, it’s cheaper to go straight to evaluating those services as commercial options.

Conclusion

A feature flag system sounds simple to implement, and in its most basic form, it is. A good engineer could build a minimal feature management platform in a couple of days.

Even so, you shouldn’t build one from scratch. There are already several feature-rich open-source products that you can use without paying a vendor, and they’re good. Obviously, we at LaunchDarkly think that ours is better, and not just because we made it. Do the math, and remember to factor in time and support costs: you’ll find that running your own flag system is usually far less economical than paying a vendor to do it.

But if you have a reason to not use a commercial, hosted FMP, then we strongly recommend that you go with one of the existing open-source alternatives. They have problems, and they’re certainly not free in terms of time and money.  But they’re better than writing your own, and they’re far better than having no feature flags at all. Once you’ve experienced the profound relief of turning off buggy code by flipping a switch, you’ll never want to go without flags again.

Read Part 2 (“Have You Outgrown Your In-House Feature Flagging Tool?”) and Part 3 (“11 Tips for Migrating to LaunchDarkly”) of this three-part blog series on “Build vs. Buy”.

If you’re interested, learn more about LaunchDarkly’s feature management platform.