Why Covid is the perfect crisis to accelerate your transformation to a data-driven enterprise. Can DataOps help you get there?

The Covid pandemic crisis is transforming the way we live and work. Enterprises are relying more than ever on their digital infrastructure as millions of employees shift from working in offices to working at home. Leaders are simultaneously placing new importance on data-driven insights. Lack of clear and accurate visibility into the health of their supply chain, manufacturing, distribution, aftermarket service, and customer support is more painful now than ever before. There has never been a more important time for enterprises to have the ability to put the right data in the hands of the right people. In addition, your current products may have to change as customer needs change as a result of a more remote workforce.

The current pandemic is the perfect crisis to push organizations to invest in data transformations that work. Traditionally, enterprises have invested in purchasing tools or implementing point solutions (aka another data warehouse). Enterprises have significantly under-invested in changing the data literacy, culture, and processes of their employees to be more data literate and nimble.

The traditional approaches to “baby steps digital transformations” have not resulted in the massive change required to turn enterprises from monolithic ERP heavy organizations struggling to use their siloed data to nimble gazelles that use data to make informed decisions and bring agility to their business.

Can DataOps be the solution to finally bring value through data in a big meaningful and transformational way? Let’s explore DataOps in more detail.

What is DataOps (Data Operations)

There are varied definitions:

  • DataOps is an automated, process-oriented methodology, used by analytic and data teams, to improve the quality and reduce the cycle time of data analytics. Source: Wikipedia
  • DataOps is an emerging discipline that brings together DevOps teams with data engineer and data scientist roles to provide the tools, processes and organizational structures to support the data-focused enterprise. Source: CIO Magazine
  • DataOps is an integrated approach for delivering data analytic solutions that uses automation, testing, orchestration, collaborative development, containerization, and continuous monitoring to continuously accelerate output and improve quality. Source: The Eckerson Group

The more I learn about DataOps, the more I think this holistic approach that aligns people, process, and technology to enable more agility to data management might be the way to go.

DataOps requires a cultural shift. It is not something that can be implemented all at once or in a short period of time. DataOps is a journey.

DataOps approach takes its cues from the agile methodology. The approach values continuous delivery of analytic insights with the primary goal of satisfying the customer. DataOps teams embrace change and seek to constantly understand evolving customer needs.

Impact of DataOps

DataOps can contribute in the following areas:

  • Speed up processes and increase quality. It provides streamlined data analytics pipelines via deep levels of automation and testing.
  • Increase the value proposition of data and analytics by having robust mature processes.
  • Establish a culture of continuous improvement and collaboration.
  • Support the data wrangling of complex data landscapes.
  • Operationalize data science to provide more value to the business. Data Science is thought in production-ready platforms and not one-off science experiments

DataOps Implementation Best Practices:

A successful implementation of DataOps requires the following (source: The Eckerson Group):

Culture:

The core of DataOps is a culture of collaboration and trust. All stakeholders must work together and feel responsible for the entire process. Awareness of the business requirements in all stages is essential. You need to build a culture of continuous improvement and embraces automation.

Processes:

DataOps requires well-defined processes, roles, guidelines, and metrics to reinforce DataOps principles. Consequently, many companies establish testing and certification programs to educate and train staffers. Team leaders need to establish agile and lean processes, augmented by testing and automation tools, to support the fast creation of data analytics pipelines.

Technology:

DataOps requires tools and infrastructure to support automation, testing, and orchestration, as well as collaboration and communication among all stakeholders. DataOps uses technology to help data stakeholders collaborate and align interests. DataOps teams should strive to:

  1. Think of analytics as code and use software engineering tools, such as version control systems, to keep track of change; virtualization to build disposable environments; and test automation to rapidly validate changes.
  2. Use automation tools and scripts whenever possible.
  3. Use tools to support the orchestration of complex analytics pipelines that involve many stakeholders, tools, and technologies.
  4. Establish a good collaboration platform that digitally supports cross-functional communication and knowledge transfer.
  5. Use monitoring tools in production to derive insights for possible improvements.

PayPal Success Story

As an example, I recently watched a video of PayPal’s multi-year cloud transformation using the Google Cloud. Their journey had three areas of focus:

  1. Change in mindset. Lift and shift was not going to work. They wanted to use the best of public cloud and not leverage old paradigms and this required a mindset shift internally.
  2. Change in toolset. If you do not change how you work and how you think about app services, the tools were not going to work.
  3. Change in skillsets. They invested in their people and culture. Cloud, data, and infrastructure literacy. This was a human change journey that required the people and their skills to transform as well.

Summary

In summary, DataOps is a reasonable potential solution for the new role of data in modern organizations. It requires fundamental changes on many levels indeed but eventually pays off by making data and analytics more efficient and paving the way to the next level of maturity.