In enterprise projects, you would often be thrown to work on a random spot of a very big system rather than building services one by one from ground up like start-ups. Hence, it could be much more difficult to grasp and piece together the different backend concepts when you are not even sure where or what is the head and tail of the architecture. This article aims to help you to get a better footing of the things to look out for in an enterprise backend development project.

Introduction to enterprise backend development

At a high level, Enterprise backend development can be described in terms of 3 concepts - Domain, Framework and Service Types.

Domain:
- Payments is an often cited domain because transaction-related issues such as concurrency & performance become more salient to be addressed.
- Other domains include health data, vehicles etc. Literally anything can be a domain.
Framework:
- Spring Boot is a popular choice of Java-based opinionated framework for creating backend applications.
- Frameworks from other languages include Flask, Django, Express, Ruby on Rails, CakePHP.
Types of Backend Services commonly built:
1. Domain Service: this service that is responsible for most business logic related features
2. Scheduler Job Service: this could be a midnight job running to send out emails or to update database records to invalidate memberships that have reached their expiration date
3. File Ingestion Service: this is a service often found in enterprise projects to support internal operations, such as parsing the excel sheet files generated by the admin department and then helping to save those data into the database.

For an enterprise backend development project that is starting from scratch, even before deciding the framework and types of backend services to build, it is important to evaluate the team compositions available and come up with a proper technical vision for the project. If you're joining an existing enterprise backend development project, then the technical vision would already be determined, but it will still be helpful for you to learn about the process of designing backend services, as it might be your job to enhance existing services or create new services to add to the system.

Service Design

Service Design is a phrase usually used in the field of User Experience (UX), but it is also very applicable for backend services. In the article Service Design 101 by Nielson Norman Group, this is the definition that they give Service Design.

Service design is the activity of planning and organizing a business’s resources (people, props, and processes) in order to (1) directly improve the employee’s experience, and (2) indirectly, the customer’s experience.

In the UX field, we identify and gather such business resources to improve the internal company's operations, which thus improve the experience that we deliver to end users. Likewise in enterprise backend, we perform a similar process of identifying components of the system:

Data entities
Relationships and associations between entities
Features that are required to perform business logic and deliver value
Features & responsibilities to be delegated to specific services
A "code of conduct" to be established between the producer and consumer
Models required to derive the entities
Validations required on the entities and models

Here's a diagram to visualize the steps and the type of design pattern/process involved for each step.

For the difference between entity and model, you can refer to this short article

The list of steps above may not be performed in the exact sequence as listed above when you are work in enterprise backend development.

If you work in an Agile enterprise project, it is more common for your business analysts (BA) to introduce the requirements of the system starting from step 3, and then they will leave it to the technical people (us) to determine how the rest of the service design process are done. In such a scenario, it is no wonder why the microservices architecture approach has become so popular for delivering enterprise backend projects.

The microservices approach:

Services are designed to be loosely coupled. Examples of loosely coupled architecture include Hexagonal Architecture, Onion Architecture.
Services are designed to be backward compatible for services that depend on them to continue using them without a need to upgrade. This can be done with a proper versioning strategy and use of feature toggles.
This approach hence allows different services to be be designed and worked on independently by different teams in parallel.

At the same time, services do not need to know how their respective producers/consumers are implemented internally in code. For example, a Python Django BFF should be able to consume data from a Java Spring Boot domain service without knowing the tech stack of the upstream service that it is calling. But if the tech stack and code implementation is unknown, how does one requests another service to perform necessary computations? That is where the APIs comes in.

API contract

Application programming interfaces (API) allow microservices to interact with one another, and an API contract written by a service dictates the specifications and format of API requests that they accept from the applications that are consuming this API. In some enterprise projects, Contract-first API development is a practiced concept. In my previous projects, we also perform Contract Testing using SwaggerHub and Pact.

With an API contract, it helps to distinguish which services are the producer and the consumer. It also declares that these services interact synchronously. For asynchronous interaction, Kafka stream processing is an option and you could also enforce a data contract with a schema registry (I'll include a resource link later for this, Kafka concepts are beyond the scope of the article)

There are alternative and commonly used phrases to describe the dependency relationship between services aside from consumer & producer - upstream & downstream services.

Upstream and Downstream Services

Architectures are generally easier to comprehend after understanding the flow of data across different parts of the system. However, in a data flow architecture context, the meaning of the terms "upstream" and "downstream" are not as explicit. If you simply treat data flows as streams of water, you would think that the source of the stream is considered "upstream" while the destination of the stream would be considered "downstream". However in a microservices context, services could be both a producer and consumer of data. In that case, how do we distinguish which is the upstream or downstream service?

On a Reflectoring article, the author deducted 2 rules for determining determine which service is the upstream and downstream service.

Dependency Rule: each item depends on all the items upstream from its viewpoint
Value Rule: moving downstream, each step adds more value to the product

In a microservices context, the value here refers to the value to the end users.

In an example enterprise project architecture as shown above, with the knowledge of the 2 rules, it is much easier to deduce that the BFF is considered a downstream service of the domain service, and the Kafka ingestion service is an upstream service of the domain service.

So why does knowing the upstream and downstream services matter? As much as we try to make services loosely coupled, when we improve on an existing functionality of a service, there is still a chance that we would accidentally break other services that are dependent on that service. Hence knowing this dependency and value relationship will help us to look out for such potential problems and actively address them before they even happen.

Exploring further

With this article, you should be more prepared to face the challenges of enterprise backend development regardless of the tech stack. At the minimum, you now know of some considerations that you should be making when you are working to improve or create a new backend service.

If you are interested to learn more, these are some topics that I have explicitly excluded because of their complexity and variation from project to project.

Authentication
Infrastructure e.g. Gateways, Routing
Stream Processing with Kafka

Here are some resources for topics briefly discussed in this article

That's a wrap folks! 🎉

Thank you for reading, hope you enjoyed the article!

If you find the article awesome, hit the reactions 🧡 and share it 🐦~

I'll also be working on a follow up to this article which will include a Spring Boot project example ✨

To stay updated whenever I post new stuff, follow me on Twitter.

Introduction to Enterprise Backend Development: Service Design Considerations