Presented by Thoughtworks
There’s hardly a company on the planet that hasn’t recognized the need to make a significant investment in data, insights, and analytics. Many have been early adopters, eager to maintain their competitive edge. Many have invested heavily, motivated by promises of hyper-personalization and acceleration to business outcomes with artificial intelligence (AI) and machine learning (ML). Yet many are still looking to see their investment come back in a meaningful way, particularly as they’re used to a straighter line when they invest in things like infrastructure.
Data mesh is a new approach to distributed data platforms that enables organizations to leverage their data assets — not just having access to all their data, but being able to understand what they have, what it means, and how to use it to meet their current and future goals.
Data mesh is about moving the shaping of the data to the people who are closest to the business use case. Because the analysis and management of the data is moved closer to these subject matter experts, they can do things like create new machine learning models and outputs faster, reuse tailored data for new purposes, and create new use cases for their data when they need them. By giving control of the data to those at the edge, who understand that data best, teams can more rapidly turn it into stronger business value and outcomes for the organization at scale.
A Cambrian explosion for data
Technologies like data lakes and data warehouses have been used to unsilo and centralize data, enabling organizations to access all of their data and serve that data from a centralized platform. But these solutions were designed for limited types of data — mostly for those associated with financial reporting. In today’s data-rich, always-on world, organizations collect a vast variety of data — and those old technologies are running up against their practical limits as the boundaries between operational data and analytical data dissolve.
This is where data mesh turns your data into invaluable, actionable business insight.
Data mesh is about more than just plugging some new technology product into your infrastructure. Zhamak Dehghani, Principal Consultant at Thoughtworks, describes it as “a socio-technical approach to sharing, accessing [and] managing analytic data at scale.”
“That means it focuses both on organizational structure as well as people’s relationships with technology, in addition to technology and infrastructure, [and] most importantly, data platform infrastructure to empower organizations to get value from the data when they’re in a complex, highly-scalable environment,” she says.
Data as a product within domains
Rather than centralizing data, data mesh stresses four key principles:
- domain-oriented decentralized data ownership and architecture
- data as a product
- self-serve data infrastructure as a platform
- federated computational governance
Domain ownership stresses the importance of empowering those people in your organization who are experts. For instance, a media company might organize itself along product lines: TV shows, films, podcasts, and so on, where each unit manages releases, artists, royalties, and the like. Data mesh gives the accountability and serving of analytics data to those domain experts.
This enables any domain team — say, the podcast unit — to gain insights into listening patterns, to build up a rich picture of their audience and to spot opportunities to widen their audience, without having to wait for monthly reports. But at the same time, the data they collect and analyze doesn’t sit at the edge of the enterprise; the owners are responsible for serving that data to the rest of the organization. That means treating data as a product: tackling data quality at the source and ensuring that it’s delivered to the rest of the audience in such a way that they know what they’re getting.
Those teams have to think about how other teams in the company may want or need to access and use that data. It’s product thinking — understanding what problems other teams need to solve and how your data could help them do so in a way that moves the company’s goals forward.
Data mesh moves analysis and management of the data closer to the domain team who best understands the data. In this pragmatic and automated approach, each team owns and is responsible for the data in their domain. When an organization achieves this, it can have a self-serve data infrastructure wherein generalists in every domain, alongside embedded data engineers and data product owners, can share or consume data as it serves their respective purposes.
Driving business value
How does all that roll up to increasing business value? “Data mesh enables you to get value from data as you grow,” says Dehghani. “As the size of the organization grows, as the functions of your organization [and] the numbers of functions you perform increase, hence the sources of data and your aspirations for data grow. It gives you a model that can still give access to data, still give value from the data when you scale. It’s a solution that scales out with the growth of the organization.”
The data mesh paradigm also enables sustained agility in extracting value from data as your organization — and its collection of independent data products — grows. But this approach demands careful management. These independent data products need to adhere to standards in order for them to interoperate. In this way, you should see more value in your data architecture and infrastructure investments.
For a timely example, consider the demand from pharma companies with multiple drug trials, many of which may be outside of a single organization, such as from their own R&D as well as from companies they recently acquired. These pharma companies want to become more adept at moving on from a drug trial that isn’t promising so they can allocate resources to trials with a better outlook. That requires insight into many diverse sets of data coming out of multiple trials. While this is an example specific to the pharma industry, there’s a generalizable takeaway — that “data collection” simply does not scale, yet “data connection” through the mesh has a much better chance.
Essentially, it’s all about figuring out what’s not going to be successful and being able to quickly adjust. That’s how companies can find value with data mesh.
Fundamentally, the paradigm shift towards data mesh can give your business the ability to unlock your data, provide meaningful access and insight to it throughout your organization and extract tremendous business value from it.
Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. Content produced by our editorial team is never influenced by advertisers or sponsors in any way. For more information, contact [email protected]
Source: Read Full Article