In today's digital economy, most companies express their business strategy through software. They use software to provide customers with great experiences, and drive operational efficiencies and insights. Recent advances in technologies such as in AI only accelerate how businesses will drive innovation and capitalize on new market opportunities through the software they are building.

With the rise of cloud computing, developers are more empowered than ever and hold the keys to any business’ success. Cloud computing created layers of abstractions for compute, storage and networking making it easier for developers to innovate and ship applications faster. However, programming with data is still challenging and time consuming; in a recent survey1, around 75% of developers and decision makers say working with data is the hardest part of developing and running applications.

Why is working with data so challenging for developers?
There are many dependencies between the application and databases that have to be managed across the entire software lifecycle - how developers develop and iterate their application, how the application runs and scales in production and how the many unforeseen requirements from users present themselves as business demands evolve

As developers get started, they must first deal with data modeling, aligning the application’s requirements to data definitions and schema design, integrating database drivers and working with the unique syntax of a database’s query language. Each new feature introduced through the application’s lifecycle demands reassessment and adjustment to the original schema. These in turn trigger migrations across application code, middleware, and data pipelines in all of your dev, test, QA, and production environments.
‍
Releasing to production introduces another set of dependencies that developers have to deal with. These span continuous optimizations to maintain predictable performance as the app scales and new features are added, rightsizing resources to handle traffic peaks, and enforcing multiple levels of data access privacy for different classes of application user. Each of these tasks demands access to skilled and scarce technical expertise.
‍
As the application evolves, a secondary set of requirements emerge. From adding caching layers, distributing data into new geographic regions, and releasing new features such as full-text keyword search to better engage users. And it's not just your users that want access to application data. It's also the data teams for model training and analytics, and many other teams in the organization for additional use cases.

The challenges of this constant care and feeding of your databases are not new; pretty much every engineering team faces them. Most deal with it by adding a lot of disparate tools to handle their applications and databases. As each tool is usually bespoke to specific tasks within the application lifecycle, it’s not unusual to find five or more tools for each database. By our own estimates over 60% of today’s apps require at least two databases. So now your developers are working with more than ten tools just to control two databases for one application. This proliferation of tools drives enormous complexity across your teams, slowing the pace of innovation and missing market opportunities while driving up project risk, cost, and attrition.
‍

Using data abstraction to improve developer experience
A few organizations have taken a different approach to these problems by building an in-house (custom) abstraction tooling layer between their applications and databases. These usually address some of the challenges above but are highly expensive to build and maintain, and take away skilled engineering resources that would be better spent innovating for the business.

Defining data, integrating it, programming it, monitoring/scaling/securing and extending its usage are all interconnected workflows and convergence of these data tools is long due. Every developer must address these problems and consider abstracting dependencies between applications and databases over the entire software development lifecycle to accelerate developer productivity, remove complexities, and reduce cost.

‍Originally published on https://www.neurelo.com/post/database-abstractions-are-long-overdue

1 https://www.mongodb.com/collateral/report-on-data-and-innovation-en