Data Mesh Cloud Migration Pattern

Original Source Here

Data Mesh Cloud Migration Pattern

Cloud migration in enterprises has been far too slow, costly, and complex. The Cloud Migration Data Mesh pattern shows how to accelerate your cloud migration.

Photo by Glenn Carstens-Peters on Unsplash

Data Mesh Cloud Migration Pattern

Cloud technology is ubiquitous. It is hidden behind each mobile phone app, and it touches and stores every email, message, and social interaction we have. And, the cloud value proposition appears to be clear: according to McKinsey, across all industries, there is “$1 trillion in business value that cloud adoption can unlock.”

Yet cloud adoption in the enterprise — in particular, migration of core applications and data — is slow and fraught with challenges. A16Z, a well-known venture fund, believes it is because “most companies find it hard to justify moving workloads off the cloud given the sheer magnitude of such efforts, and the necessary rewriting seems SO impractical as to be impossible.” So, it is not surprising that most systems in just about every industry still run predominantly on internal data centers.

But new techniques and patterns and even better approaches are now available to move data across the enterprise and into the cloud. That is where Data Mesh comes in.

This article describes the Cloud Migration Data Mesh Pattern and how it addresses fundamental data challenges allowing enterprises to accelerate their cloud migration.

Note that it is important to understand that building a Data Mesh involves much more than adopting technical pattern. It also requires a huge shift in how you think about data culture, organizational behaviour, and data operating model. Thinh Ha, a co-author of this article, has previously discussed cultural and organizational challenges you might encounter when adopting Data Mesh in an outstanding article entitled 10 reasons you are not ready to adopt Data Mesh. Thinh is a Strategic Cloud Engineer in Google’s Professional Services in the UK who has worked with many clients establishing cloud-native Data Mesh environments.

One last point: we assume that you have a high-level understanding of Data Mesh. If you need some background information on Data Mesh, there are a number of great articles are available here (patterns), here (architecture), here(principles) and here (lessons learned). For interested readers, a full description of patterns is available here and here.

Pattern Summary

The Cloud Migration Data Mesh Solution Pattern moves data in near real-time from any type of system (transactional or analytical) to a cloud-native data product within an enterprise Data Mesh while at the same time making data discoverable and easy to consume.

Context and Business Problem

McKinsey says that “75% of cloud budgets are over budget” and “38% of cloud projects are behind schedule.” They go on to say some organizations “are leaking their share of that value instead of capturing it, with inefficiencies in orchestrating cloud migrations adding unexpected cost and delays. Approximately $100 billion of wasted migration spend is expected over the next three years, and most enterprises cite the costs around migration as a major inhibitor to adopting the cloud.”

Echoing this, is recent analysis from Accenture showing “two-thirds of enterprises haven’t fully achieved their expected outcomes, even as the COVID-19 pandemic has turned cloud adoption into a mandate.”

Gartner estimates that cloud computing accounts for about 9.1% of IT spend in 2020. And despite aggressive growth expectations, cloud spend will only be 14.2% of IT spend. So, despite the massive growth of cloud computing, it is still a fraction of total IT spend.

I suppose the real question is why is cloud adoption so darn slow? I would argue that there is one simple, compelling, and practical reason that many enterprises continue to favour on-premises solutions over cloud: because that is where the data is.

Simply put, data begets data. Today’s critical mass of data in enterprise data centers creates a “data gravity well” which pulls all applications, environments, and services into its orbit. And the truth is until we can move the center of gravity out of the data center and into the cloud, adoption will remain slow.

Figure 1, Establishing the Cloud Data Gravity Well

Unfortunately, the corollary is also true: small amounts of data have no “gravity well” and hence attract little new development.

But there is a second reason for the longevity of enterprise data centers. Over many years of development and growth our enterprise data environment has become tangled and interdependent. Most architects can point to their enterprise application architecture that looks more like an integrated circuit diagram from Intel.

But the connections between applications are tightly bound together like a spider web. When we move an application, we tug on one strand of its data “spider web” and the entire architecture moves and deforms. So, in this interconnected data web even small migrations have large consequences leading to long timelines and commensurately huge costs.

Figure 2, Untangling the Data Spider Web


So, the challenge for cloud is simple. First, what is the fastest, most effective, and most efficient way to establish powerful data gravity well on the cloud. And second, how do we do this without wholesale untangling of the enterprise application and data spider web?

This is where Data Mesh and the “Cloud Migration” solution pattern comes in. The Cloud Migration Data Mesh Solution Pattern moves data in near real-time and unobtrusively from any type of system (transactional or analytical) to a secure cloud-native data product within an enterprise.

This pattern creates a “live replica” of the original data and does not require application migrations. This offers several tangible and practical benefits:

  • It is unobtrusive and hence requires zero application changes, thereby alleviating the need to “untangle” the existing application and data spider web.
  • Since no application changes are required, this pattern can be deployed very quickly.
  • With simpler data migrations, this establishes a cloud “data gravity well” required to accelerate cloud migration.
  • As a “live replica”, data is available on the cloud in near real-time providing timely, consistent, and accurate data to newer cloud-native tasks such as AI/ML and advanced analytics.
  • With data available on cloud, new applications can be developed quicker and time to market can be dramatically reduced.

Several Data Mesh Accelerator patterns are used to provide this functionality:

  • Live Replica Pattern, to make a “replica” (same data, and if needed, same structure) in a near-real-time fashion (detailed article for this pattern will be published soon).
  • Data Product Security Pattern, to ensure data managed by data products are safe and secure (detailed article for this pattern will be published soon).
  • Data Product Observability Pattern, to make data movement traceable and visible.
  • Data Product Operability Pattern, to ensure exceptions and errors in a data product are logged and broadcast to appropriate enterprise monitoring systems.

Several Data Mesh Foundational Patterns are used to provide this functionality:

  • Change Data Capture Pattern, to capture changes in data with a data product such that it can be broadcast to interested consumers, but also provide the raw data required to understand data lineage. This pattern unobtrusively captures database changes and publishes them, via an event streaming backbone, to their destination in a cloud-native database. A “Outbox Pattern” is an alternative pattern where CDC is not available.
  • API Pattern, to make data easy to access.
  • Data Product Catalog Pattern, to make data discoverable.
  • Immutable Log Pattern, to provide visibility into the lineage of data managed by the data product.
  • Event Streaming Backbone Pattern, to stream data to data consumers in a near real-time fashion.

How It Works

Figure 3 (below) illustrates how the Data Mesh Cloud Migration Pattern works.

Figure 3, Cloud Migration Solution Pattern
  1. Replication: Data and is replicated in near real-time to a cloud-native data product from transaction, engagement, or analytic systems using the Live Replica Accelerator pattern.
  2. Discovery: Data Product Catalog and Immutable Log foundational patterns provide visibility into data managed by the data product, as well as to the lineage of data within the data product.
  3. Access: JSON schemas and OpenAPI specifications (and supporting registries) enable the API Patterns allow systems to query and update data products using well-known methods.
  4. Streaming: Data is streamed from the cloud-native data product to other systems or data products allowing systems to “listen” for changes within the data product, while also offering sophisticated methods of analysing and processing data in near real-time.
  5. Observability, Security, and Operability: The cloud-native data product is monitored using the Data Product Observability, Data Product Security, and Data Product Operability accelerator patterns ensuing that the cloud-native data product functions in a safe, secure, and reliable manner.

Client Experiences: Cloud Adoption in a Financial Services Organization

I talked with Thinh Ha, about his experiences in driving Cloud Adoption with Data Mesh. Thinh is a Strategic Cloud Engineer in Google’s Professional Services in the UK who has worked with many clients establishing cloud-native Data Mesh environments. According to Ha:

Data Mesh helped us establish a data gravity well that enabled scaled value realization from data and drew in additional use-cases and workloads to the Data Cloud platform.

We chose to build a Data Mesh with a small number of data domain teams to prove a single end-to-end business process for the organization. We set-out to build a Cloud-First Data Platform to empower the Data Domain teams to independently build Data Products to prove the business case, while providing the centralized security and governance guardrails to provide assurances that key controls are being satisfied as we adopt a federated operating model.

Google Cloud has a rich set of technology to help you build a Data Mesh. As part of the Data Platform team, we created self-serviceable and compliant-by-default code templates to deploy fully managed and auto-scaling Google Cloud products such as Cloud Spanner for RDBMS, Cloud Dataflow for streaming processing, and BigQuery for data warehousing. The code templates served as accelerators to help Data Domain teams get started quickly and to simplify operations. As multi-cloud was an important requirement for the customer, we also leveraged services such as Confluent Kafka for event streaming and Collibra for Data Cataloguing.

As we started to prove out the Data Mesh, the existing Data Domain teams started realizing that by owning their own Data Products, they gained significant benefits from being able to locally optimize for agility. As a result, the Data Domain teams independently started building new business-cases to move additional workloads onto the platform.

Word started spreading about Data Products that are available on the platform, which attracted new use-cases to the platform that were previously not feasible due the difficulty of accessing this data in the original systems.

From that point, not only did we have better, faster, cheaper insights. We have also established the foundation to quickly migrate more complex applications to the cloud!

In our discussion, Ha found that Data Mesh allowed his clients to create critical mass of new data on cloud which established a new data gravity well on the cloud. Ha’s clients started small — and picked an important but non-trivial domain to start. Ha says his clients now get the further benefits of Data Mesh:

  • Data is much more self-serve.
  • Data products align accountability — and decision making and funding — with a clear owner.
  • And because we can see what data we have (catalogs etc), we now can establish a much lighter-weight governance process.

Concluding Thoughts

Today, enterprises are faced with a fundamental challenge: Despite the cloud’s obvious benefits (agility, speed, cost) the migration to cloud is slow. Why? Because our data legacy — the vast stores of data that still reside in an enterprise data center — has its own “data gravity well”. And when attempts are made to move applications, we find that our “data spider web” is far too difficult and costly to untangle.

This article has shown how the Cloud Migration Data Mesh Solution Pattern addresses this challenge by making it easy to create a cloud-native data gravity well in a safe, reliable, and near real-time manner. And it shows how to do this without impacting your current applications and data environment.

Hopefully this article gives you the necessary insight to kickstart your own Enterprise Data Mesh and accelerate your enterprise’s cloud adoption!


All images in this document except where otherwise noted have been created by Eric Broda (the co-author of this article). All icons used in the images are stock PowerPoint icons and are free from copyrights.

The opinions expressed in this article by Eric Broda are his alone and do not necessarily reflect the views of his clients.

The opinions expressed in this article by Thinh Ha (co-author of this article) are his alone and do not necessarily reflect the views of his clients or Google.


Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot

%d bloggers like this: