This article explores the intersection of The Platform Organization model and Product/Platform Design with Data Mesh, a paradigm shift that addresses how organizations can leverage their data assets in line with their context and organizational structure.

The article connects Platform Design principles with Data Mesh’s decentralized architecture, emphasizing (1) identifying the right organizational players and (2) designing scalable services to maximize data value. By adopting Data Mesh and integrating it with Platform Organization structures, businesses can unlock further AI and innovation applications covered in upcoming installments. This article set the stage for further exploration and practical insights into leveraging data as a strategic asset within modern enterprises.

 

 

Introduction

In 2019, Zhamak Dehghani introduced the concept of Data Mesh, a paradigm shift in how large-scale data management and analytics are organized in modern enterprises. Traditionally, data architecture relied on centralized data lakes and data warehouses, which often resulted in bottlenecks, poor scalability, and challenges in effectively addressing the diverse needs of various business domains. To respond to these limitations, Dehghani proposed Data Mesh, advocating for a decentralized approach, drawing on domain-driven design and Team Topologies.

The Data Mesh paradigm shows its potential in dynamic and complex contexts, where ecosystems of opportunities around data exist, evolving from the “old” concept of Big Data and Data Lakes. It acknowledges that data and their exploitation are the foundation of building market-ready applications of AI in real organizations.

 

We’ll connect Platform Design elements with the Platform Organization’s key artifacts and set the ground for further discussions on vertical applications.

We explore the central thesis:

1. The real value of data within an organization is typically not fully expressed, not due to technical limitations, but because of a lack of company readiness, wrong organization architecture, and lack of capability to envision business cases.
2. When the approach and business culture are right, and there’s a strong focus on identifying key use and business cases, new enablers like Data Mesh can unleash data value.
3. Platform design techniques and the Platform Organization structure, alongside Data Mesh and a correct team topology approach, can synthesize multiple layers: organization architecture, interfaces, processes, and Domain-Driven Design practices. Data Mesh can address opportunities beyond a technological perspective, focusing on ecosystem needs and use cases driven perspective.

This article links to and follows the Summer of Platforms webinar #2 – Data Products and Platform Organizations, with further installments and research.

 

 

 

What is the Data Mesh paradigm?

Data Mesh emphasizes distributing data ownership and responsibility across domain teams, treating data as a product. This allows teams to handle their own data pipelines, ensuring higher agility and scalability. It encourages cross-functional collaboration, with each domain managing its datasets while adhering to federated governance standards for overall data consistency and quality.

Data Mesh offers a way to better support the complex and ever-evolving needs of organizations operating in diverse and dynamic environments by breaking away from the monolithic approach to data architecture.

In a second article later in 2020, named Data Mesh Principles and Logical Architecture, Zhamak also identified the four fundamental principles behind the Data Mesh approach. We suggest the reader check the original article, but we provide a synthetic overview of the four pillars below:

Domain-Oriented Decentralized Data Ownership and Architecture

• Key idea: The responsibility of data ownership should be with the domain teams closest to the data. Instead of a central data team managing all data, each domain (e.g., marketing, finance, operations) is responsible for its data. This approach, inspired by domain-driven design, ensures data is treated as a first-class product by the teams that understand it best.
• Goal: To overcome the bottlenecks and lack of agility in centralized data platforms by enabling domain teams to operate independently while still contributing to the overall data ecosystem.

Data as a Product

• Key Idea: Data should be treated as a product. Domain teams should act as the data product owners. Teams are responsible for providing high-quality, reliable, and consumable data products to the organization. The data products should be discoverable, addressable, trustworthy, self-describing, and interoperable.
• Goal: To shift the mindset from just managing data to treating it with the same level of care and quality as a product, making it valuable and easy for others within the organization to consume.

Self-Serve Data Infrastructure as a Platform

• Key Idea: A self-serve infrastructure platform is essential to empower domain teams to manage their own data. This pillar focuses on providing the necessary tools, infrastructure, and services that allow teams to autonomously create, maintain, and use data products without deep expertise in data engineering. The platform should abstract complexities and provide standardized storage, processing, governance, and monitoring capabilities.
• Goal: To enable domain teams to focus on their core domain logic and data without worrying about the underlying infrastructure, thus reducing dependency on centralized data engineering teams.

Federated Computational Governance

• Key Idea: A governance model is needed for consistency, security, and compliance across the organization, despite Data Mesh promoting decentralization. Federated governance balances local autonomy and global standards. It leverages automation and standardization to enforce governance policies, such as data quality standards, metadata management, and access control, uniformly without hindering individual domain teams’ agility.
• Goal: To balance decentralized innovation and centralized control, ensuring the organization can scale effectively while adhering to regulatory and quality standards.

Let’s see how the Platform Design and Platform Organization principles interact with these principles.

Why and How the Platform Organization Must Adopt Data Mesh

 

 

 

In a recent article about how Organizations have evolved so far, we highlighted the main elements of Platform Organizations. We also explained the foundational beliefs behind those types of organizations in a manifesto. We are not duplicating all definitions here, assuming the reader is familiar with the main concepts of Industry Platforms (IP), MicroEnterprise (ME), Shared Services Platforms (SSP), and their contractual interfaces, like the Value Adjustment Mechanisms (VAMs) contract and Enterprise Micro Communities (EMCs) contract.

 

We can establish parallels between the four pillars of the Data Mesh paradigm and the main elements of the Platform Organizations.

 

The Domain-Oriented Decentralized Data Ownership and Architecture principle resonates with the autonomy principle that Platform Organizations apply to identify Micro Enterprises. Both the Platform Organization and the Data Mesh use domain-driven design  (a significant software design approach introduced by Eric Evans in his book of the same name published in 2003) to identify the boundaries between products, services, value propositions, and data products. In a platform organization, autonomous entities interact dynamically to produce or consume “value”, being Data a specific type of such value. Data can be the end value proposition, or one of the main elements encapsulated – as modules – into a value product or service. The same composability and modularity that exists at the product level in the platform organization need to be reflected in how data – among all the other assets – is sourced, transformed, and elaborated: specifically because data is valuable only if the context and the metadata associated with the “main content type” is described and transferred to the users, and they are enabled to act on data intentionally. So, a framework to exchange the “data objects” without losing all the relevant elements is paramount.

 

One can define the value proposition, key consumers and clients, and product features, including key interfaces, interaction processes, and experiences, considering Data as a product. Coupled with the autonomy of the data product owners, this connects nicely with the Micro Enterprise (ME) frame, a key construct in a Platform Organization. Such a Micro-Entrepreneurial unit can provide data-related products to external customers or internally as a Shared Services Platform (common services to all internal Micro-Enterprises) or a Node Micro-Enterprise (an internal ME serving other MEs in customer-facing services).

 

Creating a Shared Services Platform (SSP) for data access aligns with the Data Mesh principle of self-serving data infrastructure as a platform. This setup reduces transaction costs by standardizing recurring interactions and transforming complex processes into SaaS, resonating with some of the key platform design principles. Direct interaction between data consumers and providers allows easy personalization tailored to user needs. Typically, an SSP would manage this infrastructure, ensuring coherent processes and turning generated value into replicable, easy-to-use solutions. These solutions can even be commercialized if there’s demand and the data isn’t crucial for competitive advantage.

 

The last principle, Federated Computational Governance, resonates with typical patterns in a Platform Organization. In this setting, the organization ensures enabling platforms and infrastructures run with common service-level agreements essential for quality assurance. In a Platform Organization, the units’ autonomy coexists with centralized capital allocation, quality assurance, and compliance, assuring entrepreneurial autonomy and coherence toward the companies’ end users.

 

At this point, it should be clear that adopting the Platform Organization model with a Data Mesh approach is a no-brainer. By design, Platform Organizations are ready to adopt the key elements of Data Mesh.

 

In the modern age, organizations are ecosystem-driven, multi-sided, interacting companies generating massive data. Still, most of them lack a proper way to valorize data in their context and take action on data-driven decisions. Much of this value is lost or not fully exploited.

 

Combining these two patterns offers the flexibility to connect data with products and services managed by the proper stakeholders. Most importantly, the driver behind data exploitation is not “adopting another technology” but from value-driven business cases.

 

To answer “how” organizations can adopt Data Mesh key ideas, we can consider two layers where the Platform Organization approach can help: the Organization Design layer and the Product layer.

 

In a previous post, we explained how one could use Wardley Mapping combined with the identification of bounded contexts to unbundle and re-bundle organizational capabilities and functions in a market-driven way. This resulted in suggestions for Micro-Enterprises and Shared Services Platforms to build. Such analysis can also identify essential interfaces where data can be used as a value enabler. The process we adopted is the following:

• Identify the key interfaces, and add to the Platform Design analysis, the identification of Data relevant exchanges;
• Visualizing the key organizational inflection points “around which data provisioning or consumption happens” can help spot opportunities for building particular services and define which organization artifacts to create to provide such capability;
• Identify how the value generated by the data is requested, consumed, or integrated into final workflows by the end recipients of such value.

 

A deep understanding of the organization domains and data role will provide clear directions about:

• The organizational artifacts responsible for each part of the data product lifecycle (MEs, SSPs, NodeMEs);
• The contracts (VAMs, EMCs) that govern how the capability is provisioned and the value is shared between the data owners and consumers;

 

In this landscape, it will be easier to understand what elements of the data product lifecycle are to be standardized, what must remain custom-built, and what can be turned into products or managed services, or commoditized and externalized, along with the partner or competitor relationship with company data. More soon.

What is a Data Product, and How Can Platform Design be Paramount?

Moving to a Data Product perspective, it’s clear how key tenets of Platform Design can benefit the organization.

Dehghani clearly explains the complexity behind many successful data solutions in her brilliant post on Data Mesh processes and interfaces, as evident in the picture.

 

Users have different needs, data consumption methods, and interaction ways with the data product or its integration into their workflows. These interactions depend not on the “customer unit” nature (marketing, content production, publishing, project management, production, etc.) but on each user’s skills and capabilities. In traditional organizations, those with the idea may lack the skills to validate assumptions and customer needs, needing to interact with a “data department” or IT unit.

 

Recent trends towards simplification, modularity, and self-serving interfaces have driven the emergence and success of no-code and low-code products. With these solutions, product owners can develop and deploy functional MVPs themselves and validate assumptions more easily. Similarly, clear and efficient interfaces should be provided for data-centric workflows to accelerate the adoption of data driven attitudes throughout the organization. Business and product owners should access, understand, and leverage data to make better decisions in a real operating context. Data mesh is the low code for decision makers. 

Conclusion

In conclusion, the synergy between Platform Organizations and Data Mesh approaches offers a powerful framework for modern businesses to harness their data assets. By embracing these complementary models, organizations can create a more flexible, scalable, and value-driven data ecosystem.

 

This integration enables better decision-making, fosters innovation, and empowers teams to leverage data effectively across the enterprise without men in the middle.

 

Adopting Platform Design techniques can help design bespoke, adaptive, and easy-to-integrate Data Mesh products. This can achieve sustainable, large-scale adoption in the organization without burdening certain players.

 

In the coming weeks, more research will be published on our blog on this topic, so… stay tuned.