DataOps: Accelerating Innovation through Data

Data Friction Inhibits Innovation The digital economy has created an unquenchable thirst for data across all aspects of business. Every company is now a software company, and the market shift from complex, feature-heavy products for the masses to highly personalized, on-demand experiences, mandates a sophisticated data strategy. Data is a competitive advantage, providing insight into customer behavior, market dynamics, and untapped opportunity that no competitor has. As companies are increasingly realizing the value of their data, the demands for access to this data is also increasing. But it’s all for naught if users don’t have fast and easy access to the data they need to drive rapid development, analysis and decision making to enable a continuous stream of new product and business opportunities. Those that can leverage data to drive innovation will win; those that can’t, will lose. The core problem is not data itself, but the data friction caused when constraints on data prevent people from meeting the ever-growing demands of the business. On one end are the forces driving that thirst for data. An ever These forces are driving data to be everywhere, for anyone, in growing constituency of Data Consumers that are responsible any form. But the other hand are forces driving to constrain data, for using data to drive new projects and innovation. Includes restricting access and availability: developers, testers, data scientists, analysts, and more. • Increased Cost – Data is doubling in size every two years,11 • More Demand – As DevOps and cloud tear down the barriers requiring storage, network, and compute resources to keep between people and infrastructure, more environments, more pace. Even when technologies can keep up, the rising number automation, and more speed means more demand for data, faster. But of data copies creates an exponential effect that is nearly it’s not just speed; new technology and market trends are continually impossible to overcome. creating new data and data needs – 60% of companies today have a machine learning strategy7 and 43% already have an IoT strategy.8 • More Users – It used to be that developers were responsible for • Increased Complexity – The number of popular data sources has quadrupled in the past five years,12 with more as-a-service and on-premise solutions than ever before. Meanwhile, new feeding data into complex business applications and ETL processes, so technologies like mobile, social, machine learning, and IoT have that business professionals could consume carefully crafted reports created entirely new data sources for the business. As a result, and dashboards. But the world moves too quickly, and everyone data is increasingly generated and consumed in silos, and data must become a data consumer in order to get the data they need, operators are struggling to simply deliver a consistent level of when they need it. While the number of data scientists has more service for all the teams that need data. than doubled in the last four years,9 so too has increased the number of auditors, analysts, developers, testers, and compliance officers that need data. It’s not just quantity of users, but the variety of backgrounds, requirements, and skills spread out across organizations. • More Places – The days of a single datacenter are gone, and virtually no enterprise talks about a journey to a single cloud provider. 85% of enterprises have a multi-cloud strategy, with applications running across an average of 5 public and private clouds.10 And every cloud vendor is desperately trying to deliver unique services to lock in customers, requiring companies to adapt to an ever changing cloud landscape in order to leverage the best possible technology solutions. • Increased Risk – More than 9 billion personal records have been stolen in the last five years,13 escalating security and privacy concerns. Data security is now a top imperative for the business, and responding to regulatory pressure a key hurdle to overcome. But the easiest solution – preventing access – is the opposite of what data consumers need. These forces strain the ability of the Data Operators who are responsible for infrastructure, security, and maintenance of the data. These include DBAs, security and compliance, system administrators, and more. Overcoming Friction with DataOps While Agile, cloud, and DevOps have helped, they are ultimately for the challenge. We need a new approach; one that does for insufficient. Predictable delivery of compute environments has data what DevOps did for infrastructure. That is the promise of gone from weeks to minutes, with automated, elastic, and on- DataOps, a new means to connect people to data, empowering demand infrastructure. But data is unlike compute. It’s expensive them to overcome data friction and achieve the velocity of to maintain, full of sensitive information, difficult to copy, hard innovation demanded by the digital economy. to track over time, and slow to deliver to the teams that need it. In fact, getting to the cloud and eliminating infrastructure as a constraint actually exacerbates the problem of data friction. Data operators are still struggling to manage, secure, and deliver the data environments demanded of them. And users are still struggling to access, manipulate, and share the data they need. And this constant struggle has had a negative impact on the relationship between the data consumers and data operators: less than 37% of business organizations perceive that IT’s digital initiatives are aligned with the business; around 25% view IT as correctly using data; and less than 25% consider IT is using structured approaches to deliver value to customers.5 While it seems likely that IT is more aligned to the business than those numbers suggest, the numbers do reveal a clear disconnect between the two groups. DataOps is the alignment of people, process, and technology to enable the rapid, automated, and secure management of data. Its goal is to improve outcomes by bringing together those that need data with those that provide it, eliminating friction throughout the data lifecycle. Mastering DataOps requires overcoming the organizational and cultural barriers that separate people from data. It starts with bringing two key audiences together as one team: Data Operators and Data Consumers. In order to do this, we must first recognize that this impacts more than just “the business” and IT; afterall, sometimes members of IT are also data consumers. Once we have identified the data operators and consumers, then we can then map out the current process flow to complete a consumer’s request for data. This should include all of the manual steps taken by operations, as well as internal and When data friction becomes the blocker to innovation, customers external constraints in the process. It is important that both the leave, competitors win, and businesses spend more time reacting consumers and operators are involved in this process as a team instead of leading. so that all constraints are correctly identified and quantified. But it doesn’t need to be this way – companies can win against data friction. Data Operations can overcome cost, complexity, and risk to become an enabler for the business. Users can get the data they need to unleash their capacity for innovation. And everyone can work as one team to drive massive outcomes for the business. But today’s methodologies are ill-equipped After the mapping is complete, the most critical constraints should be prioritized to be eliminated; this is typically done through automation and self-service, but could possibly purely be a change in policy, i.e. it may no longer be necessary to require encryption on lower environments once data masking has been implemented in upstream data environments.