6 ways to work with database admins in the DevOps world

DevOps is defined as "unifying the operations and engineering teams," in order to foster a culture of cross-team collaboration, codify how infrastructure is built, and become a more data-driven organization. But it seems databases and the teams that care for them are treated as an exception to this environment. In most companies, databases are still treated like walled gardens, with the database hosts tended to like delicate flowers and the database administrators (DBAs) guarding any and all access to them.

This walled-garden attitude invariably affects the rest of the organization, from tech ops, to delivery engineering, all the way to product planning, as everyone tries to work around the datastore. Ultimately this reduces the benefits of an agile approach to software development, which is a problem for companies that have been running for a few years and have reached a solid financial footing with loyal paying customers, but are having a hard time shedding that startup skin (the one that flies by the seat of its pants), and are feeling the pressure to achieve a sense of stability in existing and future offerings.

Before we get into how to make DBAs and their traditionally walled gardens part of the DevOps culture, let's inspect the current state of affairs.

In a typical shop, the development cycle of a new feature looks like this:

1. Product asks for new shiny feature.
2. Maybe developers promise a date for new shiny feature.
3. Developers code it.
4. Some functional testing is done in an environment that maybe looks like production (or... maybe not exactly).
5. It's deployed.
6. Profit??

This is all great. But what if your product manager was right about the market need for this shiny, new feature?

• When you wrote the code for this were you testing with 100 fake users?
• Did you test with 100,000 users?
• How about 1 million+ users?
• Did you consider long-term needs? Growth rate? Data retention? Security needs?
• Did you design for failover scenarios when database hosts/stores fail?
• Did you practice those assumptions?

This is where limiting DevOps to operations and engineering falls short. Without getting DBAs' perspective in the planning stage, you may be in trouble if that thing you build succeeds. And who wants to be afraid of success?

DBAs aren't blameless in this either. Many DBAs can be stuck in the old way of not seeing a change until it is in production and, when thing break in production, reacting by being even more protective of their databases against those "pesky developers." 

Try these six strategies to bring DBAs into the DevOps culture to fix these problems and make the products you ship more successful.

1. Maintain an organization tech catalog.

Engineering is a team sport. The entire team needs to be rowing in the same direction. This means that no matter how much this new tech you read about seems like a good fit for the new thing you are building, you need to be aware of the organizational cost of adding new technology to an existing and sprawling product/tech stack.

This doesn't mean you can't add new technology to that catalog when the need arises. But, as a growing business, it is important to recognize the cost and risk that comes with every new technology you add to your stack. Especially when that new tech is going to store any form of application state. More senior members in the engineering org, the operations team, and the DBAs all need to be involved in consciously vetting potential new datastores and defining the specific business need the new datastore technology is expected to address. Details like multi-datacenter replication, availability guarantees, integrity guarantees, and operational needs should all be part of the checklist for any delivery team making the case for adopting a new datastore.

2. Write design blueprints. Keep them updated.

Before you set out to create the new best product ever, write down how this new thing will be built and how its many components will interact with each other. In this new world of microservices and distributed systems, design by watercooler and hearsay can be detrimental to any product's success. Make sure you document your design and keep it in a living document so everyone, from the product manager asking for features to the database team who will be responsible for these new deployments, can get on the same page for every little detail.

Things to track in these documents that can help your product become successful include:

• data-retention needs
• expected availability goals
• expected data consistency
• expected growth rate
• expected latency needs
• upstream and downstream dependencies of the new database

You would be amazed how many times engineers and product managers working on the same project have completely different expectations and design plans. That is a big problem. The engineering team, which is usually responsible for promising the business and customers a delivery date, must ensure that all stakeholders are aware of the details of those promises. Those stakeholders include not just management, but anyone who will help you deploy all the layers of your system.

3. Start an architecture-review process.

If your organization has more than a handful of engineers working on the same thing in the same room, you must accept that not everyone is always aware of all the new things being built. If you are already making significant revenue and have a two- or three-level engineering org with quarterly planning for new products, but still do not have a process for reviewing the architecture of those planned new products, you are in for a bad time.

Just as writing the new thing's architecture is very important for keeping everyone in the loop, so is having a venue for constructive conversations and soliciting feedback about the plan before a single line of code is written. This is a great way to draw on the expertise of people who have built something similar or who have seen a similar architecture used with a larger customer base and can share crucial lessons learned.

4. Leverage your teammates' experience.

Once you have internalized the team sport aspect of designing a new product for success, you may find that your DBA team is not only good at solving issues in production, but also very good at identifying issues before they happen, when the design is still a document in review. As the architecture blueprint and review process become standard in your organization, you'll no longer feel like a detached team is gatekeeping your progress. You can constantly draw on the DBA team's knowledge to inform how you use the datastores in your tech stack, how to scale, and how to own the availability and performance of your new product. This is like a superpower for your team.

5. Fire drills are not only for ops.

Even after you do all the planning and design in architecture review, there is still plenty of value in testing all your assumptions once the thing is built, but before your business revenue is dependent on it. This is even more critical when the success of a product is directly related to the datastore's performance, availability, or ability to scale. Before releasing the new feature to your customers, make sure to schedule a fire drill where you cause failures in a controlled environment and work through the expected scenario for recovery to either prove its accuracy or adjust expectations. Yes, make sure your database team is involved in the planning for the fire drill and the actual exercise. There is a lot of useful muscle to build in an engineering org by having "game days" and making sure the troubleshooting and recovery plans play out as expected.

6. Don't rescrict this to new features.

Finally, as your company continues to grow, you will inevitably find that parts of the product that worked fine a few years ago are now showing their age. When the time comes for rewrites, don't pretend this is an engineering-only endeavor. Rewrites and refactors are a full-organization exercise, just like new features are. They need the buy-in of product managers, the advice of the operations and DBA teams, and the full blueprint and architecture review process.

In organizations that want to practice DevOps, engineering teams have a lot of power in taking a product from conception to production. And with that power comes the ability to draw on more than just the ability to design a stack or write code, to develop not just for features, but also for stability and scalability. Increasing collaboration across your entire technology organization offers long-term value for your business and your customers.

Learn more in Silvia's talk, Working with DBAs in a DevOps World, at LISA17, which will be held October 29-November 3 in San Francisco, California.