Data Migration Resources

Learn more about data migration projects. Here you will find information and valuable resources, including details of what’s involved in planning and executing a data migration project and resources to increase your project’s success rate.

 

What is Data Migration?

Data Migration is the process of transferring corporate data from one system or database to a new one. It differs from data integration, ETL, and data replication, which are primarily concerned with moving data between existing environments. However, these technologies may be used to support the migration process. Data migration is often a subset or sub-project of a broader application migration (for example: migrating from InfoLease to Odessa Platform (LeaseWave), consolidating Oracle environments, or migrating from one version of SAP to another). It may also be used to support migration from one database to another or between major upgrades of a database. Data migration is also crucial in the overall process of migrating on-premise IT infrastructure to a Cloud environment.

Data migration is not a technology, but rather a specialized task that needs to be supported by a variety of software applications, tools, and frameworks.

Facts about data migration projects

Data migrations tend to be among the most often underestimated tasks critical to the successful delivery of a new system or platform implementation. Correct and complete data migrations are necessary for successful application or systems implementations.
A report from Bloor Research found that more than 80% of data migration projects run over time and budget, cost overruns average 30%, and time overruns average 41%. Gartner has found that 80% of organizations will underestimate the costs of data acquisition tasks by an average of 50%.

Why is the failure rate so high?

The data migration project is supposed to be a key component in intricate company-wide projects. Still, the reality is that it is often subject to misconceptions that minimize the challenges and complexities that entail the migration process. Due to this, organizations underestimate the effort required to complete the migration tasks on time and budget. Typically, the effort associated with successful data migration is underestimated for a variety of reasons:

Misconceptions and poor assumptions

    • Thinking that it is just about moving data from A to B.
    • Assuming that the existing data will fit into the new system.
    • Limited understanding of the true quality (or lack of quality) of the data and the associated effort required to make the data “fit for purpose.”
    • Focusing exclusively on the business process that needs to be modeled rather than thinking about the data required to support it.
    • Assuming that documentation about the current state landscape is complete, accurate, and well-understood.
    • Thinking that data migration is a one-time event done by IT that needs no business involvement.

Risks not identified early enough.

    • In most cases, the risk usually surfaces very late in the form of load failures in the target system.
    • Often, these load failures are a result of poor data quality and a poor understanding of the nuances contained in the data and how those nuances would impact the load and conversion process.

Lack of formal data governance, methodology, and tools to support the migration

    • Being able to load the data into the target system is not the measure of success because it may be valid and even complete but not correct.
    • Without a formal approach to data governance or data management, you will have a moving target in figuring out how accurate the data should be.
    • A traditional waterfall approach to a data migration project does not fit well with the iterative approach required to get the data to the necessary quality and, particularly, with data migrations to complex business applications. The waterfall project plan works from an overall perspective but does not fit well in data migration projects, considering multiple mock migration runs required in this type of project.
    • Lack of purpose-built data quality technology to assist in data profiling and cleansing means that organizations do not have a proper understanding of the current state of the data early enough in the project nor the ability to clean it properly.

Data migration is not solely an IT problem.
Active participation and direct involvement from business stakeholders are vital to the success of the data migration project.

 

How to improve the rate of success when migrating data to a new system of the environment?

Engage the business

Data migration is not solely an IT problem. Active participation and direct involvement from business stakeholders are vital to the success of the data migration project. Everyone is fully committed to their day jobs, and even though it will be painful, an organization must make these essential resources available during the migration process.

Failure to make business resources available for the current state and to-be discussions will lead to an IT-centric project that involves guesswork and assumptions. Most of these assumptions will remain undetected until very late in the project when the data fails to load correctly or when functional testing fails.

Ongoing involvement from business executives and managers will be required to approve data remediation to manage data quality issues. Also, business approval will be necessary to handle any discrepancies in how the new application or software manages specific business processes and the underlying data, including financial data. Data profiling will provide visibility to remediation needs.

Assign dedicated resources to the data migration team

Don’t think about the data migration project as a one-off event involving shifting data from one system or platform to another.

Instead, think of setting up a data migration team responsible for the delivery of an end-to-end, repeatable process flow from source to target with an associated methodology and purpose-built data quality engine at the core.

Start working on data profiling and quality months before the implementation project starts.

The best approach to uncovering risk early and fully understanding your data is through data profiling. The data profiling results can be used to engage stakeholders in aspects such as:

  • Discussions about the actual and real state of the data.
  • Root cause analysis that can point to business process flaws that create data issues.
  • Discovering and understanding discrepancies between the documented and actual states of the environment is based on the data.
  • Engaging in discussions about how “good” the data needs to be, including prioritization of mandatory vs. ‘nice to have’ requirements.
  • Understanding the level of effort required to make the data “fit for purpose.” Without these types of discussions and associated analyses, any estimate about the actual time and effort needed to migrate the data successfully is simply guesswork.

Iterate frequently and adapt to changes

Each iteration executed by the data migration team reduces risk and improves the final output quality through enhanced visibility into the data and the introduction of additional cleansing rules per iteration. Hundreds of iterations will be required during the migration process. The team members (and executives) will need to remember that change will be the only constant during the migration process.

Factors to consider when formulating a data migration strategy

The more carefully your company and executives plan its data migration, the less likely you will encounter surprise costs, unplanned downtime, and, eventually, exposure to project failure. The less likely it is that your end-users will be frustrated or inconvenienced during and after the migration. You’ll want to establish goals, budget, timeline, and brainstorm with your technical and business teams to anticipate any challenges that you may encounter.

There are five principal determinants you should consider when determining how you’ll approach the data migration project:

Type of application

Specialized applications like complex financial, CRM, ERPs, or Accounting software requires more in-depth planning than applications that are non-essential for the core operations of the business. Any changes in your current environment will need careful planning, but some applications are more critical than others. In some cases, data can be moved with software tools specific to the type of data being migrated. These tools are usually provided by the software vendor or third party companies.

Go-live strategy

There are multiple things to consider when migrating data, but one of the most important is the go-live strategy. You can transfer data in stages while keeping the source and target systems running parallel. Alternatively, you can plan a big-bang approach to migrate the data outside of regular business hours. This second approach involves more risk but is probably the best alternative in most projects.

Amount of data

Even systems with few records require thorough planning. When examining the amount of data, the first factor is volume, but you should also analyze how complex and intertwined these records are. For example, if you need to migrate leasing contracts, you will need to include all the underlying transactions linked to each deal in your analysis. There could be cases where you initially consider the volume of the system’s main entities like customers or vendors but fail to understand the volume of other related entities.

Migration performance

One aspect that if often overseen in most projects but becomes critical on the go-live date is the time to complete the data transfer. There are many considerations to this point, like server memory, optimized SQL queries, and ETL structures. The amount of data being migrated and the speed of the processing tool will determine how long data migration takes. Suppose your go-live strategy considers a 48-hour window (during a weekend) to complete the data migration. In that case, the data processing step can’t take all of these hours since your business and technical teams will require time to complete other tasks related to the project.

Migration of Historical Data

This is probably the most common request in data migration projects. Business users request that all current and historical data should be migrated and available in the new system. While this could be a valid request from a business perspective, you might want to consider migrating transactions and records related only to active customers. Migrating historical data introduces an additional level of complexity to a project that is already complex. Historical data needs to be available for business users, but consider other solutions like a data warehouse, or data lake.

Specialized Services

An option that you should always consider is to manage the data migration aspect of your projects from planning to delivery with a solution provider that has knowledge and experience in similar data migration projects.

If your in-house data migration expertise is limited, and the applications you’re migrating are critical to the business operations, a specialized consultant can help you. These companies have experience designing cost-effective and comprehensive plans to minimize the risk involved in the migration process.

Another critical aspect to consider is that even if your company’s resources have some knowledge about data migration projects, they will be dedicated to their regular job responsibilities. Therefore they won’t have the required commitment to help your migration project move forward successfully.

To learn more about CXO Data Migration and Conversion specialized services, contact us today.

LET’S START A NEW PROJECT TOGETHER

We establish a collaborative relationship with our clients, understand their business needs and challenges, and leverage the extensive experience of our highly skilled professionals to devise the best solution.

LET’S START