AdelaA few days ago, we posted a question on HN: Ask HN: Is DBA still a good job? The question quickly drew the attention of the HN community and got 236 thought-provoking comments. We share and feel the challenges, the frustration, the fatigue and the expectation.
We want to preserve all these heated discussions and help more people develop a multi-dimensional perspective on DBAs. That's how an idea for a blog series emerges -- D·D·D (DBA, Database, and Developer).
As the debut of this series, we compiled 10 stories about DBAs based on the replies and comments from the HN thread: we adopted the Decameron style, made up the story background, and borrowed the true stories shared by the commenters and turned them over to the storytellers (all credits go to the rightful owners). Some of these storytellers had experience as full-time DBAs; the others may be developers who have worked with DBAs before. You can regard this article as the appetizer of a full course meal. The crew welcomes all kinds of feedback and, last but not least: Bon Appetite!
10 people, consisting of developers and DBAs from different companies, woke up in a locked-up room. Apart from basic furniture, the room seems as normal as it could be except for a wooden board hung above the door with 3 letters carved on it -- DDD.
Nobody knows what happened and how they ended up there. Just when they were about to smash the windows, an electronic screen flashed for a few seconds and displayed their way out:
"One true story about DBAs per day, you'll only get locked up for 10 days."
Somebody sneered at the somehow sentient screen:
"Stupid rule...What could possibly happen if we don't?"
Another sentence appeared on the screen:
"You could leave, but your team's databases will be dropped and won't be recovered."
Dead silence permeats the room. After a while, somebody finally breaks the ice:
"Well, I'll go first."
Day 1 From donatj: Honestly Wish We had a DBA
We had a DBA about 6 years ago, he was one of the smartest guys I've ever worked with, and knew his stuff inside and out. I would recommend a DBA highly to anyone having growth issues. I honestly don't think we would have scaled without him. I learned a ton from him.
He got let go for basically having a really bad temper, and we decided not to look for a new DBA. I was sad he was gone but also glad to not be getting screamed at. It feels silly, but I've got really unresolved and mixed feelings about the whole thing. I genuinely really liked the guy, loved talking music and movies with him, but he would heel turn in an instant and just ruin your day.
Honestly wish we had a DBA, I ended up picking up a lot of his responsibilities. I'd love to have someone I could learn from. Basically, I'm getting real sick of optimizing reports.
The screen really is sentient, even intelligent at some level. It automatically summarized the story:
"Honestly wish we had a DBA"
Day 2 From steve_adams_86: Both Brilliant, both jerks
This is strange and funny to read; the only two DBAs I've worked with matched this description perfectly.
The first guy was so irate and literally yelling about his entitlement to punish us for making bad architectural decisions that I thought he was joking. I just laughed, but he got angrier. Then it clicked after maybe five or ten seconds – he was in the process of verbally punishing us for querying the database too much.
The second guy was similar, but didn't yell. Still super protective of his domain, almost antagonistic about us integrating with the database at all.
Both brilliant, taught me more than I can recount here, hilarious and interesting… But total jerks too, and both let go more or less for that reason.
One they had a great system in place and the product felt mature, the companies axed them.
The screen flashes:
"All brilliant, all jerks."
DBAs in the room frowned a little.
Day 3 From jamal-kumar: Gone DBA
Is this the DBA gone off the rails story thread?
I tried to hire a guy for a postgres DBA at a company I had worked at. I knew him from before as a MS SQL DBA from a previous company and hoped he could fulfill the role, he said he could. The guy saved our asses in the previous case.
We gave him like a month paid time to get acquainted with postgres. Dude DISAPPEARS, gets his vpn keys revoked and everything because we're like what the hell. A month later he sends this big long email saying how he got in some altercation with the police, and landed in jail... it was just way too rambling to take seriously. We ask a mutual friend what the hell was up and he said that since he was working too much his wife CONFISCATED HIS CELL PHONE.
I think we called him 'ballgag ***' (fill in the blank with your least favorite person's name) after that
These guys are worth their weight in gold though, I think I heard he ended up getting a job at Microsoft down the line.
The screen switches to today's theme:
"Gone DBA"
Day 4 From Simon_O_Rourke: He was Supposed to Know Everything
I had eerily similar experiences from an older DBA guy, who was going through college in his 40's while working away. He knew his stuff, helped a lot optimizing the ETL process, but could turn nasty in an instant, and had a major chip on his shoulder about younger analysts with college degrees.
Only thing, was this guy had zero programming experience outside SQL and Bash. Anytime he had a question in a meeting about some calling Java process, he used to wait until the meeting was finished and then grab five minutes with me privately to ask about whatever. So there was this perceived weakness that his professional persona "knew everything", which meant he couldn't ask fundamental questions or try to (publicly) learn things outside his domain.
In the end, we moved to Amazon Redshift, and he was progressively moved into a smaller and smaller role, where now he just curates 2-year old JIRA tickets. Probably knows he can't get a similarly well-paying gig elsewhere now.
The familiar flashes appeared once more:
"He was supposed to 'know everything'"
Day 5 From skc: “UR RUINING MY F****ING DATABASE”
This is eerily similar to my experiences with two very, very good DBAs I worked with.
I'm now wondering if there is something about the role that lends itself to such behavior. One day they are super pleasant, helpful and friendly. The next they are chewing people out for seemingly minor and random transgressions and saying things like "ruining my fucking database".
Today these guys remind me a lot of JK Simmons character in "Whiplash"
The screen displays an image this time:
The DBAs among these 10 people have been humble audiences for 5 days. This picture becomes the last straw to break their nerves. They decided to take the lead now.
Day 6 From forinti: Try to be the person I needed help from before
I was a developer and turned into a DBA by necessity. When I was still just a dev, I couldn't get the support I needed from the DBAs, so now I try to be the person I needed help from before.
But, I now understand the problem. At my organisation we have only 2 DBAs, and we can't give all of our time to this task.
To be a DBA is an endless battle against the entropy some devs try to create. The data model should really always be validated by someone with more knowledge and experience. Also, some young devs come up with crazy ideas (for instance, we should not use foreign keys). Others want to load insane amounts of data to process in the database, instead of preprocessing, or using a staging database. There is a continuous stream of bad ideas being implemented.
And when a bad idea gets implemented, it is really hard to undo it and it tends to just create more chaos: you need materialised views to get around bad modelling, or weird views to compensate for duplicate data, etc.
So, yes, a DBA is a really important role. With regards to it being a good job, it depends if your company takes it seriously and puts you in the development process, otherwise you'll be in for a lot of stress.
Screen displays:
"Try to be the person I needed help from before"
Day 7 From compton93: For that company my role was priceless
I hate to imagine how many startups out there don't even create indexes and just consider it to be a database problem and increase CPU/RAM and look the other way.
I worked for several years at a semi-startup as a Database Engineer where my role was to guide developers in writing performant SQL (and writing indexes for them) and architect their schema to be inline with our future plans and various other database tasks (managing query plans, being an expert on the database feature set etc). I even created an internal course: SQL School, that I included the directors/customer support teams in.
To that specific company my role was priceless, new features could be built in 1/2 the time with an expert writing the queries and handling the database. My programmatic analysis of our 20 year old code base had 5000 unique queries and 3000 more when accounting for dynamic SQL. Not gonna lie it was a complete mess but if I didn't exist they would have needed a lot more database resources. It was a terrible code base and any plan to refactor and avoid this mess would still need my role to do the transition.
The screen flashes and shows:
"For that company my role was priceless"
Day 8 From Twisell: Exhausting yet rewarding
I would argue that it's way more accurate to say this role is actually outsourced in most organizations with all the tradeoffs that come with that strategy. I begged my former employer to subscribe to a good PostgreSQL support company. Guess what the one we picked is constantly looking to hire new DBA to sustain growth...
I was in database engineering/administration position for the last 10 year on a small company. It was exhausting because management wouldn't care to rely one guy to manage the whole data stack. It was rewarding because the CEO saw me as probably smarter than I was because I knew SQL well enough to produce outstanding value as a single individual.
I spent a few years complaining that my colleagues didn't know their database well enough, and it was causing unnecessary bugs and poor performance.
I wanted the company to pay for some basic courses on the databases we were using, especially for the more junior Devs.
Instead we switched database like 11 times and now I don't know the database either.
The bugs and poor performance lingers though.
The screen changed the sentence to:
"Exhausting yet rewarding"
Day 9 From jhugo: Conflict comes from lack of understanding
I think this kind of conflict often comes from frustration at developers not having much understanding of the underlying technology that the network engineer / DBA is responsible for.
Networks & DBs present abstractions, but they're very leaky ones, yet developers often expect to just treat them as magical black boxes about which they don't need to know much of the inner workings, constraints, tradeoffs etc. That makes the interface between the network engineer or DBA and software developers a natural site of conflict.
(You even see this above in this very thread, someone complaining that a DBA was punishing developers for "querying the database too much". Most likely the DBA was actually upset about how they were querying the database, not that they were querying it too much, but the developers want it to "just work" so the distinction isn't important to them.)
Screen flashes:
"Conflict comes from lack of understanding"
Day 10 From niklasrde & datalopers: The lost art -- data modeling and SQL
Same at my previous role - we did have some people who were really good at some point, and into DBs. We had a very solid Redis and PostgreSQL set-up, and some advanced stuff for simple data in Redis - but once that knowledge fades with attrition, making space for continued personal development into how two very different DBMS work for a small team is not easy, and it just stays untouched, disliked, and somewhat abandoned forever more.
I honestly think the thing that costs most companies the most in their hosting is developers not using databases correctly
100%. Data modeling and SQL has become a lost art.
The screen flashes and literally drops a few pixel tears:
"The lost art -- data modeling and SQL"
Finale
While the last story approaches its end, the doors and windows open up widely. 10 people storm out, rush back to their office, and thoroughly check their database. Everything is fine, nothing deleted. They sigh with relief and start to delve into their daily work, as concentrated as ever.
It all starts with one sentence:
"Is DBA still a good job?"
We look forward to your story.
Original post https://www.bytebase.com/blog/dba-database-developer-decameron
