What is data migration?
Data migration can be defined as: transfer of data between different types of file formats, databases, and storage systems or as transfer of data from one repository to another.
Requirements to be remembered:
- Data quality must be assessed and documented before migration to ensure implementation is without data loss.
- The data type of the columns in the table must be the same in most cases.
- Emphasis must be given to the primary kind of data during migration.
- During migration, one can increase the no. of the columns as per requirements but are not allowed to remove any column without specific reason to prevent risk of data loss.
- Schema and table name can be modified as per need, but data content must be the same.
- The Table should not be restricted to limited entries (less than the data that is going to be migrated).
- If there exists a correlation between the tables before migration, the same should be maintained correctly after successful migration.
Since a script is run (executed) to migrate data in bulk in a single click, it would be helpful to maintain a log for the input tables configured to store the initial data, to be recoverable in case of data migration failures. Finally, there will need to be a clear count of data after migration.
Types of Data Migration
When an organization switches from one platform or vendor application to another.
Moving data from one storage system to another (e.g. hard disk).
An organization’s complete or partial information asserts, applications, or services are deployed to the cloud.
There are mainly two considerations for data migration: Pre Migration Process and Post Migration Process.
Pre Migration process
This activity considers the prerequisites for the migration such as planning tasks for the pre-migration phase.
The following series of actions are considered in this phase:
Design a Strategy:
The strategy describes the need for data migration. An organization needs to decide whether to go for simple data migration services or use enterprise-grade ETL tools designed to facilitate data migration.
Assess and Analyze:
This phase considers requirements such as how much data requires migration and whether the source is structured or unstructured, dated or recent one, etc.
Collect and Cleanse Data:
In this phase, all sets of duplicate data are identified and cleansed or removed before migration is performed.
Ensuring to route data into the right buckets.
In this phase, data mapping and testing are performed as per structure (always refer to older databases).
This is the final step in which everything falls into place. All the above-mentioned steps help to achieve a clean and well-organized data. Now, the data is ready to be migrated.
Post Migration process
This process involves testing required to ensure that the old data is working as expected on the new platform. In this process, testing is first performed on the new test data and subsequently, the functionality of the application is observed and to determine if it is working as expected – exhibiting the same behavior as with the old data set.
Benefits of data migration
- Improved data consistency across systems, processes and organizations
- Increased responsiveness to the business changes
- Faster, lower-cost data migration possibilities
- Reduced risk of data leakage and compliance slips
Challenges of migrating data
- Older and legacy systems require specialized and costly skills to access the data.
- Systems tend to lack up-to-date documentation, so data content and quality are unknown.
- Often multiple systems are migrated to a single system, creating a need to resolve significant redundancy and inconsistency
Disadvantages of unplanned data migration
Time Consuming: Migration is a continuous process, which must be repeated every time a medium/asset reaches obsolescence, for all data objects stored on a certain media.
Data migration involves planning, executing and testing in equal measure to ensure the desired outcome is achieved – to keep the systems up to date with the current data.
This article was developed by Mithilesh Singh, Automation Engineer.
Write to firstname.lastname@example.org for product testing talent, automation strategies and quality engineering services.