Filling the Data Gap in DevOps

Introduction Software is reshaping industries, from healthcare to finance to retail and beyond. The pace of change is accelerating, and companies must quickly deliver applications in order to survive in the digital era. Speed is the new standard in the software economy. To achieve the goal of faster time-to-market, companies continue to adopt faster, more iterative development methodologies. These organizations have replaced age-old waterfall development methodologies with Agile practices. From an infrastructure perspective, they’ve also invested in modern architectures and cloud technologies to achieve higher efficiencies. Some companies have combined these two approaches by adopting DevOps—investing in new tools and processes such as infrastructure automation and continuous integration—to clock even faster speeds. But as companies adopt faster development methodologies, a new constraint has emerged in the journey to digital transformation. Data has long been the neglected discipline—the weakest link in the tool chain—with provisioning times still counted in days, weeks, or even months. Most companies, even those using the latest DevOps automation tools, still manage and deploy database changes manually, further anchoring development teams. Put differently, most organizations attempt to support modern, Agile development environments with half-century-old database and data management processes and procedures—a result akin to mounting a modern Ferrari on spoked Model T tires. It’s a great way to get nowhere fast. 3 The Challenge Today’s development teams are increasingly hamstrung by the the scenes, IT operations teams are constrained by the slow, speed and quality of data. Developers require fast, high-fidelity inefficient process of extracting, copying, and moving data from data, but their requests often go unmet because environments system to system. For systems containing sensitive data that must are expensive and time-consuming to create. As a result, they’re also be secured, environments can be even more complex and forced to work with low-quality, stale, or incomplete data—leading costly to provision. In some cases, constraints in the availability of to adverse consequences such as more time spent analyzing environments and data can lead to delayed releases or production and resolving data-related defects instead of coding. Behind downtime, which can have a material impact on the business. Prod Dev Test Stage Data + Database Code Management Data Database Code Slow, Error-prone SLOW Many admins COSTLY No version control Not repeatable Sensitive data exposed RISKY Syncing both data and database schema changes with application releases has been a challenge. To make an already antiquated process worse, each new version of an application requires structure or logic updates to the database— including adding or changing tables, columns, or stored procedures. This means application developers need a mechanism to update the database schema to align with their version of the application. After provisioning a new environment with production data for an existing development branch, for instance, database teams might need to apply committed database structural changes to the data and also potentially inject any required synthetic data. Deploying database code, however, is often slow and error-prone. Many IT teams struggle to maintain and align database code with the application code because their schema management processes are not repeatable. Even if a change is virtually identical to a thousand previous changes, it must be approached as a new process, which can lead to inconsistent configurations. A single failure can lead to even more failures downstream, resulting in expenditures on the order of hundreds of hours spent troubleshooting. All in, database schema management further hinders development teams from working at an agile pace. Keeping both data and database schema changes up-to-date with application releases has always been a challenging task. In an environment where speed counts every day, it’s no longer adequate to add more people and infrastructure to solve these challenges. There’s an increasingly pressing need for a mechanism that delivers high-fidelity, usable data in a fast and repeatable manner. 4