The Data Product Platform Navigator

Understanding the Maturity Model

The CNCF Platform Engineering Maturity Model is defined by several maturity levels and a set of aspects that evolve as the platform matures.

The maturity levels are:

• Provisional: An initial, ad-hoc platform.
• Operational: A more standardized and centrally managed platform.
• Scalable: A self-service platform that encourages adoption.
• Optimizing: A platform that is continuously improving and deeply integrated into the development workflow.

The aspects of the platform that are assessed at each level include:

• Investment: How staffing and funds are allocated to the platform.
• Adoption: How and why users discover and use the platform.
• Interfaces: How users interact with and consume platform capabilities.
• Operations: How platform capabilities are planned, prioritized, developed, and maintained.
• Measurement: How feedback is gathered and incorporated to improve the platform.

Figure 4: The CNCF Platform Engineering Maturity Model, showing the four levels Provisional, Operational, Scalable, Optimizing and the five aspects (Investment, Adoption, Interfaces, Operations, Measurement).

How to Use the Maturity Model

Figure 5: The three-step process for using the maturity model: 1. Know your current platform maturity, 2. Focus on high friction aspects, 3. Iterate.

Using the maturity model is a straightforward process:

1. Know your current state: The first step is to map your current platform situation against the different aspects and maturity levels. For example, we might have a dedicated team for platform investment (Level 2), but operations and measurement might still be ad-hoc (Level 1).


Figure 6: Mapping the current state of a platform on the maturity model grid, and indicating planned improvements on aspects with high friction.

2. Focus on a specific aspect: Once we have a clear picture of the current state, we can focus on a specific aspect to improve. For instance, if we want to ensure we are building the right features, we might decide to invest in a platform product team and start collecting insights on the biggest pain points for our users.
3. Iterate: The final step is to apply our improvements and continuously adapt.

As we iterate, we must remember to balance the different aspects of the maturity model against each other. Some aspects, such as Investment and Adoption, heavily revolve around people, policies, and processes, while others, like Interfaces and Operations, are more focused on technology. By using the maturity model grid as our compass, we can consciously guide our next steps. For example, we might realize we need to balance our investments by focusing on self-service solutions and automated central enablement (technology) only after we have established a dedicated product team and collected insights (people, policies, and processes). This balance prevents us from over-investing in one area while leaving others behind, and the model helps us clearly communicate these strategic decisions to stakeholders.

Level 1: Provisional Platform - The “No Platform” Platform

A provisional platform for data products and data mesh is characterized by an ad-hoc and chaotic approach to technology and provisioning. To illustrate this, let's use a bit of domain storytelling.

Consider a first data product developer who sets out to create a new data product. Lacking a central self-service platform, they must navigate the organization through personal networks. They start by talking to a DevOps engineer [1] to request a source control repository, and that engineer creates a Git repository [2] on the Git platform. Next, the developer needs a build and deployment automation pipeline, so they talk to a second DevOps engineer [3], who creates a CI pipeline [4] on the CI system. For data storage, the developer talks to a cloud engineer [5], who creates an S3 bucket [6] on AWS Cloud. With the basic infrastructure in place, the developer sets up their own data processing tools: they create an Iceberg database [7] on an Iceberg catalog and query engine, and create a Spark job [8] running on a Spark cluster. By manually assembling these disparate pieces, they produce something that resembles a data product.

Sometime later, a second data product developer is tasked with building another data product. Having no platform documentation or self-service interface to guide them, they talk to the first developer [9] to ask how to get started. The first developer refers them to the DevOps engineer who helped them earlier. The second developer then talks to that same DevOps engineer [10], who creates another Git repository [11] on the Git platform. Next, the developer talks to the second DevOps engineer [12], who creates another CI pipeline [13] on the CI system.

When choosing their analytical technology stack, however, the second developer takes an entirely different direction. Rather than using Iceberg and Spark, they create a dataset [14] on Google BigQuery, run a dbt model [15] on dbt Cloud for transformations, and create a view [16] on a data virtualization tool to serve the data.

This scenario highlights the core failure modes of a provisional platform. Data technology is requested and provisioned entirely ad-hoc, with human bottlenecks at every step where developers must “know someone who knows someone that knows how to do something” to make progress. There is no explicit concept of a data product or data contract. There are only isolated, manually stitched pieces of technical infrastructure. The result is too high a cognitive load, deep technological fragmentation and complexity, and increasing shadow IT across the organization.

Level 2: Operational Platform - The “Fragmented” Platform Experience

At the operational level, there is an effort to standardize technology and track requests centrally. However, the experience for data product developers remains fragmented and complex.

To introduce order into the chaos, organizations typically establish dedicated infrastructure teams (e.g. a DevOps Team, a Cloud Team, and a Data Platform Team). Requests are then channeled through a centralized ticketing system like Jira.

Continuing the story, the data product developer now has to request every component through formal tickets:

The developer creates a Git Repository Ticket [1] in the ticketing system. Sometime later, the DevOps Team reads the ticket [2] and provisions a Git repository [3] on the Git platform. The developer then creates a CI Pipeline Ticket [4], which the DevOps Team reads [5] to provision a CI pipeline [6] on the CI System. For storage, the developer creates an S3 Bucket Ticket [7], which the Cloud Team reads [8] to provision an S3 bucket [9] on AWS Cloud. Turning to data infrastructure, the developer creates an Iceberg Database Ticket [10], which the Data Platform Team reads [11] to provision an Iceberg database [12] on the Iceberg Catalog & Query Engine. Finally, the developer creates a Spark Job Ticket [13], which the Data Platform Team reads [14] to provision a Spark job [15] on the Spark Cluster.

While this introduces central visibility and standardizes the technology catalog, it replaces personal tapping-on-shoulders with queues and handoffs. Waiting for multiple teams to fulfill tickets takes significant time. More critically, once all these individual components are provisioned, the developer is still left entirely alone to glue them together into a functioning data product. Frustrated by the delays and integration friction, developers are often tempted to bypass the platform altogether, managing [16] their own shadow IT solution (“My Data Platform”) to deliver results faster.

At the operational level, automation can be introduced through typical Infrastructure-as-Code (IaC) scripts (e.g. Terraform) to speed up the provisioning of individual components. This will avoid ticketing systems and human bottlenecks.

In this alternative setup, each specialized team provides reusable scripts that developers can execute independently. The data product developer executes the createGitRepository script [1], which automatically provisions a Git repository [2] on Git. The developer then executes the createCIPipeline script [3], which provisions a CI pipeline [4] on the CI System. For cloud storage, the developer executes the createS3Bucket script [5], provisioning an S3 bucket [6] on AWS Cloud. For data assets, the developer executes the createIcebergDatabase script [7], provisioning an Iceberg database [8] on the Iceberg Catalog & Query Engine, and executes the createSparkJob script [9], provisioning a Spark job [10] on the Spark Cluster.

While this script-based automation eliminates ticketing delays, the developer is still left with the complex task of gluing everything together to create a data product. The cognitive load remains high, and there is a knowledge bottleneck, as developers need to know how to compose data products from these different components (e.g. configure and integrate networking, security roles, storage, pipelines, secrets, etc). When errors occur, developers run into knowledge gaps and still need [11] direct human help from the DevOps Team, Cloud Team, and Data Platform Team.

Ultimately, this operational approach leads to a lack of uniformity: organizations end up with “Frankenstein” data products that look similar on the surface because they use the same underlying technologies, but are architecturally inconsistent under the hood. Exhausted by the complexity of stitching everything together, developers frequently retreat to shadow IT, continuing to manage [12] their own local platforms (“My Data Platform”).

Level 3: Scalable Platform - Platform as a Product

A scalable data mesh platform represents a major shift in the developer experience. The interface changes completely, moving from individual technologies to a higher-level, data product-centric abstraction.

In this scenario, the organization establishes a dedicated Data Product & Data Mesh Platform Team that treats the platform as a product. Rather than handling tickets or providing disconnected scripts, this team provides a self-service developer experience consisting of a Data Product Developer Portal and an underlying Data Mesh Platform Backend.

The domain story is much simpler. In summary, a data product developer just clicks 'Create Data Product', and the platform wires the technology and infrastructure together as a Data Product. In the domain storytelling diagram above, the developer creates a Data Product [1] on the Data Product Developer Portal by defining its architectural specification such as input and output ports, data transformation code setup, and data contracts for the structure and guarantees on the data. The portal captures this intent and instructs the Data Mesh Platform Backend to do the provisioning and integration of components.

From this single declaration, the Data Mesh Platform Backend automatically orchestrates the underlying technical resources across all infrastructure domains: it provisions a Git repository [2] on Git, provisions a CI pipeline [3] on the CI System, and provisions an S3 bucket [4] on AWS Cloud. In the data infrastructure domain, the backend provisions an Iceberg database [5] on the Iceberg Catalog & Query Engine, and provisions a Spark job [6] on the Spark Cluster.

Instead of just spinning up isolated components, the backend automatically wires and glues them together into a cohesive, standardized data product.

This integrated self-service tooling transforms the developer experience. The cognitive load on developers drops significantly because they interact with the abstraction of a data product rather than wrestling with underlying technologies. It removes the knowledge bottleneck: developers no longer need to be infrastructure plumbing experts who know how to hand-stitch security permissions, pipelines, and storage configurations. Furthermore, because the platform backend enforces architectural standards, every data product exhibits more technical uniformity, which prevents “Frankenstein” data products. When every data product adheres to the same standardized architectural structure, the platform can enable automated governance by injecting automated policy checks, audit logging, access controls, and data quality verification across all data products. By making data product creation easy, straightforward and frictionless, it encourages voluntary adoption, naturally discouraging shadow IT.

By removing the knowledge bottleneck, the platform empowers data product developers with more autonomy, which is crucial for scaling a data mesh and data product management practice across multiple domain teams.

Level 4: Optimizing Platform - Platform as an Ecosystem

At the optimizing level, the platform becomes deeply integrated into the daily developer workflow. Rather than requiring developers to switch contexts and go to a platform portal, the platform comes directly to the developer's working environment and context.

This shift is achieved in the developer's local context by making the tools they live in every day (e.g. IDE or CLI) “data product aware.” The domain story illustrates how this streamlines the entire data product lifecycle:

The data product developer creates [1] a local Data Product directly from their day-to-day workspace. Through an IDE Plugin or a dedicated CLI, the developer can create, develop, deploy, and monitor the data products from within their familiar environment without disruptive context switching.

The local tools communicate with the platform and create [2] the Data Product on the centralized Data Mesh Platform Backend (which is maintained by the Data Mesh Platform Team).

From there, the Data Mesh Platform Backend automatically provisions and integrates the necessary technologies needed for the data product. A Git Repository is provisioned on Git [3] and a CI Pipeline on the CI System [4]. In AWS Cloud, an S3 Bucket is created [5] to store data in the data product. Structure is given to the data using an Iceberg Database [6] on an Iceberg Catalog & Query Engine. And finally, the data product comes alive when a Spark Job is running in the Spark Cluster [7].

Just as in Level 3, the platform backend automatically handles all the wiring and integration so the provisioned resources are glued together to function as a single, cohesive data product.

Another defining dimension of the optimizing level is the emergence of a federated platform ecosystem. The central platform team no longer acts as a bottleneck for new capabilities. Instead, other teams across the organization can contribute directly to the platform backend:

• An engineer from a Domain Team can contribute a domain-specific extension, such as Platform Capability X (e.g. specialized data product templates), making it available to the entire organization [8].
• An engineer from a Specialized Team (such as security, compliance, or data governance) can contribute platform capabilities, such as Platform Capability Y (e.g. automated PII masking, fine-grained access policies, or audit logging) [9].

An optimizing platform for data products and data mesh minimizes context switching for developers, manages the entire data product lifecycle, and enables an ecosystem of contributors. Managing the entire data product lifecycle empowers the organization to execute migrations of data products when underlying technologies evolve (e.g. upgrading an Iceberg or Spark version, breaking changes in data contract formats, etc), without requiring every team to rewrite their data products. Actively tracking which platform capabilities and which data products are actually consumed allows teams to prune unused data products and decommission obsolete platform features, continually refining and evolving the platform to meet organizational needs.