The new world of DataOps

Businesses today need to quickly deliver data to support more analytic applications, Data Science and AI, and microservices, used by their line of business personnel as well as their customers.

Modern Data Architectures need to be developed to accommodate these new analytic requirements, larger data volumes and take advantage of newer technologies such as cloud, data lakes and data streaming.

Additionally, the concurrency of the data is becoming imperative, e.g. compliance & regulatory data, web data, social media, etc.

Putting DataOps in place along with procedures for sharing and using data across enterprise (by way of Business & Enterprise Architecture and IT Strategy & Planning) will enable every business user to deliver and provision data with ease and speed – to provide the competitive edge many companies seek.

But what is DataOps?

DataOps is a collaborative Data Management practice introduced in an official manner in mid-2018 on the Gartner 2018 Hype Cycle. It is not a technology solution, but a process for managing data, people and technology in a manner that improves efficiency and ways in which data is used across a company.

Put simply, DataOps is the application of DevOps practices to data management and integration combined with AL/ML to reduce the cycle time of data analytics, with a focus on automation, collaboration and monitoring.

It offers a blueprint to solve architectural complexities and data challenges posed by data drift, hybrid multi-cloud architectures and real-time analytics (+ AI and ML).

The DataOps lifecycle is similar to DevOps but is analytics-focused; it is chiefly concerned about accessing & transforming data, modelling & visualising data and reporting insights – via data curation, data manipulation, data pipeline, and general data management methods.

It is underpinned by Data Management practices and the DataOps platform itself is maturing rapidly to include StreamSets, AI/ML and other data-centric technologies that ultimately make accessing the right data, with the right insights available to Data Scientists, business users and increasingly customers (public users).

The DataOps Platform typically lends itself to the world of data lakes and high-intensity GPU based analytics built on solutions including (amongst others) AWS, Apache Hadoop, Microsoft Azure, snowflake, databricks, Solr, Kinetica and many other emerging (data focused) technologies and products.

Points for Reflection

DataOps is enabled by good IT Strategy and Planning and robust Business/Enterprise Architecture initiatives. Without proper design thinking, the data could be rendered worthless (think “garbage-in-garbage-out”) to consumers and analysts who are seeking to exploit insights to better serve the business.

Below are a few points worth reflecting on before you and your organisation embark on embracing and integrating DataOps into your business culture.

• How it can help Data Scientist send more time modelling and less time working with data.
• It is important to have a view of the entire lifecycle of the data from source through the pipelines to operation.
• Some companies are automating data access to provide self-service to analysts.
• Data sources are growing (GSK has 1,800 sources), data is constantly changing, requiring rework or faulty results.
• The concurrency of the data is becoming imperative, e.g. wellhead data, web data, social media, etc.
• Putting DataOps in place can be the competitive edge many companies seek.

Parting Thoughts

The key point to remember is that DataOps is a process – not a solution that can be purchased and installed on the network or in the cloud – and a core component of (future) enterprise Data Architecture.

That said, the supporting technology and product domains are rapidly changing and expanding. This means everyone (especially IT Leaders, CTOs, CIOs, Business & Enterprise Architects) needs to keep up and even get their ‘hands dirty’ to understand the potential of creating scalable (data) architectures and business centric solutions that provide real tangible benefits and ROI.

Naturally, your organisation could engage external consultancies and product vendors to assist but I would strongly urge against outsourcing your IP and engage very closely with all personnel – internal, contingent and external – to ensure all DataOps activities become an inherent part of your culture and business practices.

Arrange a Conversation