Irrespective of the substantial development in making enterprise data more democratic, aka “easily accessible by all”, it is not quite there yet. And, Oracle GoldenGate (hereon, referred to as OGG) can be considered one of the highest points in the said direction as of today. A revolutionary data mesh platform for enterprises looking to effectively enable high data accessibility and real-time analytics without the need to manage compute environments while designing, implementing, and observing their data replication and stream data processing. However, expert onboarding is necessary for ensuring satisfactory outcomes. This is exactly where Infolob also enters the picture.

After serving over a decade enabling prospective and existing Oracle customers to achieve their data management, application development, performance, availability, security, marketing, and workforce targets via 22 specializations in Oracle’s core technology stack and 250+ Oracle-certified professionals deployed globally – Infolob has created a niche as the digital transformation enablement company. The testimony of which is the ever-expanding list of happy customers; the first-ever delivery of Cloud@Customer by an Oracle partner; and the prestigious title of the Most Valuable Partner held by Infolob in 2018, 2019, and 2020.

Let us go through a recent migration project that Infolob undertook for a massive consumer goods manufacturer, to explore how Oracle GoldenGate revolutionizes the spheres of data replication and real-time analytics while our expertise helps customers to seamlessly migrate per their requirements bypassing all complexities.

The Project: Specifications

The customer environment comprised data interfacing from OTM SaaS to Azure Data Lake (ADL) leveraging Oracle Cloud Infrastructure Market Place (OCI MP), Oracle GoldenGate, and Oracle GoldenGate Big Data Adapter (OGG BDA). And, the solution architecture implemented comprised:

1. Tabular data from OTM SaaS database is extracted to trial files using GoldenGate on OCI Market Place and replicated the data onto the Staging database

2. From the staging database exclusive tables (as per business need) are again extracted using GoldenGate and trail files are staged to the GoldenGate Big Data Adapter server

3. GG Big Data Adapter works as a data integration product, and it converts data from GoldenGate trail files into row formatted parquet files on the target Azure Data Lake using Hadoop client

4. OTM SaaS data interfaced to Azure Data Lake is used by enterprise data lake and downstream applications to perform Data Analytics

The customer reached out to Infolob with enterprise data lake migration project to migrate their existing Oracle GoldenGate Infrastructure and Databases from Customer Compartment (Ashburn Region) to ACS Compartment (London Region) within Oracle Cloud Infrastructure (OCI). There is a total of five Global Regions targeted for this migration – North America (NA), Latin America (LA), Europe (EU), South Asia (SA), APAC | Africa| Middle East | Turkey | Russia (AAR).

The Project: Challenges

Infolob immediately deployed a team of experts to assist the customer with their request. Per the protocol, a thorough assessment was initiated, and the following challenges surfaced:

1. As each region had its own unique and complex architecture it was very difficult to afford the best common solution approach for a seamless migration

2. Proposing and incorporating a GoldenGate Logic during infrastructure migration to avoid downstream data processing impact at enterprise data lake’s part

3. Maintaining Data Integrity (No Data Loss) in every phase of migration

4. Strict timelines due to decommissioning of existing support infrastructure

5. No structural or code changes were allowed and accepted at the enterprise data lake

If an issue arose in any of the phases of migration, the following would be the result:

   → Data integrity is compromised – Loss of data
   → GG Infrastructure has to be rebuilt from the core
   → Rebuilding of Replica Databases (Staging environment between OTM SaaS and UDL)
 → Data reload to be performed at enterprise data lake’ end and multiple other downstream applications

Any of the aforementioned situations could lead to business loss, investment of time, cost, and resources to do a complete re-work, project deadline extension, etc.

The Migration Approach Applied

Firstly, it was necessary to ensure that data processing at enterprise data lake is unimpacted due to infrastructure migration. Enterprise data lake used an external condition in combination with row data to identify the latest version of the record to process them for downstream applications. The existing condition was at the risk of breaking if GoldenGate processes were migrated to another target – impacting enterprise data lake and downstream apps.

Hence, a new GoldenGate Logic—appending of Goldengate Metacolumn (currenttimestamp) which is a support feature of GoldenGate Big Data Adapter (BDA) to each record—was proposed and implemented after comprehensive testing by Infolob’s GoldenGate, data lake & other application teams for each region. This new solution yielded affirmative results and proved that enterprise data lake’s data processing may not be impacted during current migration, and even in future infrastructural changes if required.

Next, owing to architectural complexity and various infrastructure support dependencies, the Migration plan was split into two phases – Downstream Cutover & Upstream Cutover – giving utmost significance to maintaining data integrity.

1. Downstream Cutover: The phase with Interface between Staging (Replica) Database, Big Data Adapter, and enterprise Azure Data Lake involved the migration of Goldengate Infrastructure (GG Processes – Extract, Dist Path, Replicat, etc.) and Staging (Replica) databases from Customer Compartment to Newly provisioned infrastructure in ACS Compartment.

2. Upstream Cutover: The phase for the interface between OTM SaaS Source Database to Staging (Replica) Database involved Staging (Replica) databases migration along with repointing of Goldengate Infrastructure (GG Process – Extract, Dist Path & Replicat) transferring replication files to Customer compartment to be pointed to transfer files to ACS Compartment.

The Project: Takeaways

Upon a successful migration and cutover, the following takeaways were drawn from the engagement.

• Splitting the infrastructure migration into two phases reduced the complexity giving decent scope for Database & GoldenGate migration with minimum downtime at customer end w.r.t data availability

• Constructive planning in every phase of migration for each region helped in completing migration within timelines

• Teams accurately identified the risks involved in tasks and preparedness required for mitigation

• Comprehensive testing of GG Metacolumn solution in UAT enterprise data lake environment for data integrity in collaboration with downstream applications proved fruitful

• Close coordination with other teams (Cloudops, UDL, etc.) contributed to a seamless migration

• Clear communication on the implementation timelines to customers for preparedness was fruitful

• OCI GG MP & Goldengate BDA solution can be used as a data interfacing logic where OTM SaaS or other non-oracle products are used as Data Source