Based on our experiences with clients, we have found that non-technical challenges often end up delaying the cloud journey, frustrating both technical and business teams, and negatively impacting other important projects. 

So, you’re all set to move your data architecture and data assets into the cloud. (Hopefully, you were inspired by reading previous articles, written by my colleagues and me!) Your business case has been approved, and budget has been allocated. The cloud platform has been selected, and you have begun platform design.

Just as success seems around the corner, you encounter some of the non-technical challenges that are critically important in making your journey to the cloud smoother. Based on our experiences with many clients, we have found that these non-technical challenges often end up delaying your cloud journey, frustrating both technical and business teams, and negatively impacting other important projects.

In this article, I describe some of these common challenges, and for each one, provide an approach that we used to help our clients overcome them. 

Challenge 1 – Information Security 

One of the most important steps on your journey to move your data into the cloud is approval from the Information Security (InfoSec) group within your organization. This seems obvious, but many clients overlook the fact that it is often a lengthy process. This is more so if you are a global organization, you have sensitive data such as personal information, or you are subject to other privacy standards like the General Data Protection Regulation (GDPR). In some cases, it takes months to gain InfoSec approval. Meanwhile, very little data of value can be moved to the cloud, and correspondingly, very little business benefit is obtained as discussions stagnate.

Approach – Involve your Information Security group early on

We recommend engaging the InfoSec group as soon as the cloud discussion takes hold. InfoSec needs to be involved from the start so that concerns can be addressed early and collaboratively. It is easy and cost effective to setup a temporary platform with multiple public cloud vendors during this process, in order to test InfoSec criteria using sample scrubbed data. This approach will save considerable time and momentum later.

Challenge 2 – Picking the First Project

Another critical challenge is to pick the right use case, project, or workload for the first implementation. Several criteria need to factor into this decision. A large enterprise program could showcase the immense value that cloud technologies will bring. Yet a large program has greater risks, and will typically take a long time to demonstrate this value. A quick win could, by design, demonstrate value quickly – but will it be enough to bring support from all business areas and stakeholders?

Approach – Build a strategic roadmap

Typically, we recommend that our clients develop a strategic roadmap that has both tactical, short term projects as well as long-term strategic initiatives. Business value is delivered incrementally as the infrastructure and data assets are stood up, thereby (ideally) creating a virtuous cycle. The roadmap lays out the path for identified data subject areas to be moved to the cloud, and how these integrated data sets provide the reporting and analytical capability to the business. Leadership must approve the roadmap, and any significant changes to it.

Challenge 3 – Transitioning Data Feeds to the Cloud

Many clients struggle with orchestrating the movement of data feeds from their current state to the cloud. These feeds include data to/from applications, data warehouses, and external sources/targets. Additionally, organizations have developed Machine Learning (ML) models, usually built by data scientists in business silos, using on-premise tools. These models also ingest data and produce insights that are fed to applications.

As you plan the transition of data feeds to the cloud, all of these (including feeds to/from ML models) need to be analyzed, updated and transitioned to the cloud in a systematic manner. Due to the data dependencies, this sequencing can be very complicated. If this orchestration is not planned well, the result is a mess of old and new data feeds with increased support and maintenance costs.

Approach: Prioritize data feed migrations based on data dependencies and value, considering ML models

• Follow a thoughtfully-developed plan that takes into account the data dependencies and business priorities. This plan should be part of the (previously recommended) roadmap.
• Stay away from in-flight, business-critical initiatives until they are complete. Combining such an initiative with the cloud journey will only complicate matters, and could delay both.
• Start with interfaces that have data or performance issues, so that real business value can be delivered in moving them to the cloud. However, make sure that realistic expectations are set, since the timeline for improving these is likely longer than just moving them.
• For ML models, consider using cloud-based platforms, which have robust features. Data to/from these cloud-based models to on-premise applications should then follow the same patterns as other data paths in new architecture.

Challenge 4 – Current Process Limitations

Businesses are moving their data architectures to the cloud to increase speed, agility, flexibility, and scale of delivering solutions to the business, while containing costs. Yes, cloud technology has the potential to provide these benefits. Businesses often stifle these advantages, however, by layering in their existing project and deployment processes. Not that these aren’t important, but a cloud strategy needs to re-define how current IT processes will evolve to a more efficient future state. For example, project owners need to ask:      

• can data deployments take advantage of DataOps concepts?
• can projects be run using Agile methodologies, to quickly deliver business value in smaller increments?
• can organizations utilize automation and continuous integration / continuous development (CI/CD) tools?
• how can project intake and prioritization evolve to take advantage of rapid delivery cycles?

Approach – Refresh current end-to-end delivery processes for the cloud

Organizations should take a holistic approach to revamping end-to-end data lifecycle processes during their journey to the cloud. These process updates should be put in place in the early stages of the journey so that they are well-defined and understood. A Proof-of-Concept delivered early using these new processes will allow the teams to test and improve them.

Challenge 5 – Data Governance

Data Governance - that function that organizations have been struggling with for years with on-premise data - doesn’t go away with data in the cloud! With the speed, agility, and scale provided by the cloud data platform, you may be tempted to forego or reduce governance processes. However, that will only lead to further deterioration of data quality and trust in the cloud-based data.    

Approach – Be proactive with your Data Governance processes                    

The recommended approach is to update data governance in preparation for your cloud journey, including governance of:

• data moving into the cloud (from on-premise or directly), and the lineage of data across all hops
• data products generated in the cloud (data feeds, data marts, reports, analyses)
• predictive and machine learning models in the cloud

This challenge should be approached head-on, in the early stages of the move to the cloud. It is critical to identify individuals to own, measure and monitor the lineage and quality of data in the cloud. Data governance must be part of the cloud roadmap and aligned to the data sets, applications or projects that are prioritized.

Challenge 6 – Cloud Skills

Cloud platforms feature new tools and technologies, and organizations do not necessarily have the skills required to utilize them. This is an often overlooked challenge. For example, a Microsoft centric company may typically decide on the Azure platform. However, the skills required on-premise do not automatically transfer to Azure, since there are many new, evolving Azure tools built specifically for the cloud.  Ignoring this challenge will result in the company struggling to stay on track on the cloud journey. We’ve found this challenge to be true irrespective of the cloud platform chosen.

Approach – Allocate sufficient resources (time, $) to build the required cloud skills

The right approach to address this challenge is to allocate a sufficient budget for training and potential re-allocation of resources, as well as some net new hires. Using a proven consulting or contract partner during the initial stages of the program can also mitigate this challenge.

Challenge 7 – Project Ownership and Communications

Understanding who will own key steps of the cloud journey is critical to its success. We often see organizations take the “ready, fire, aim” approach, kicking off large-scale projects without establishing foundational steps such as developing a roadmap, delegating ownership or outlining how they will communicate with the rest of the organization.

Approach – Define a Communications Strategy

It is critical to define a regular communications strategy to provide consistent updates to all stakeholders. Designating specific resources to lead communications and the roadmap steps will provide transparency for stakeholders and establish a point-of-contact for issues that require escalation.

Conclusion

Moving data and analytics to the cloud will, indeed, provide the touted benefits of speed-to-market, agility, scale, and cost-effectiveness. There are many challenges along the way, and non-technical challenges tend to get little attention. However, these non-technical challenges often lead to even longer delays and increased frustration. Organizations must take a measured, holistic, and well-planned approach with adequate funding and attention given to both technical and non-technical issues.

Do you agree with these challenges? If you faced them, how did you address them? Were there others? If you are currently facing these challenges, we can provide best practices and other techniques that have helped our clients overcome them. Drop me a note or leave a comment, and let’s discuss.