Harbourline Home moves off its on-prem warehouse
Harbourline Home (made up) runs 200 stores and an online shop. Its data platform is an on-prem warehouse fed by nightly jobs and two self-managed Kafka clusters for point-of-sale and inventory events. The plan is a managed cloud warehouse and a managed Kafka service within a year. The risk isn't moving the data; it's breaking a dashboard or a forecast nobody knew depended on a table. Everything here is fictional.
The four steps
1. Map what depends on what: Insight Catalog
Before anything moves, the team connects Insight Catalog to the warehouse, the ETL jobs and the BI tool. It crawls and profiles schemas, tracks lineage through pipelines to dashboards and reports, and classifies sensitive data. The feature that matters most here is impact analysis: for any table or column, the downstream assets that would be affected by changing it. That turns "what will break?" into a list with owners.
2. Know the state of the clusters: InsightKafka
The Kafka clusters get an InsightKafka health check first: an automated assessment of each cluster with findings to fix or carry over deliberately, so the team doesn't faithfully migrate a misconfiguration.
3. Migrate with validation: Insight Automation Hub
Insight Automation Hub plans and runs migrations (databases, warehouses, data-processing workloads, Kafka clusters and BI) as workflows. Its distinguishing part is validation: test cases generated from the schema and data, row-level diffs, statistical drift checks and golden-query comparisons. Every workflow has rollback steps defined next to the forward steps; if validation fails, the run halts, alerts the team and rolls back to the last known good state. Each run records who launched it, which script version ran, every check, and every approval or override.
4. Rebuild the pipelines: Insight Dataflow
The nightly jobs are rebuilt in Insight Dataflow: a visual designer with connectors for the old and new platforms, generating Airflow DAGs that run the same on AWS, Azure and GCP, with lineage and monitoring. Catalog then picks up the new pipelines, so the lineage map stays true after the move.
How Harbourline sequences it
| Wave | What moves | Gate to the next wave |
|---|---|---|
| 1 | Reference data and finance marts (few dependents) | All golden queries match; no rollback |
| 2 | Inventory event stream and its consumers | Kafka findings closed; consumer lag within agreed bounds |
| 3 | Sales history and forecasting inputs | Forecast outputs compared side by side for two weeks |
| 4 | Decommission the old platform | Catalog shows no remaining dependents |
Validation is only as good as its checks. Generated test cases and golden queries need review by the people who own the data. Agree what "matching" means for each dataset (exact, within tolerance, or statistically) before the first run.
What the audit trail is for
A migration of this size gets reviewed: by finance, because the warehouse holds the numbers; by security, because data is moving; and by the business, because forecasts depend on it. Automation Hub's run history answers the reviewers' questions directly: which script version ran against which tables, what every row count, checksum and golden-query comparison showed, and who approved or overrode a step. Catalog's lineage shows that each dependent asset was accounted for. Together they replace the spreadsheet of migration checks that usually goes stale halfway through.
Where it runs
Automation Hub can be used as cloud SaaS for a quick start, or self-hosted on Kubernetes or Docker in your cloud or data centre, with its control plane and workers running alongside your data, which is the usual choice when the data is sensitive.
Related: the reference architecture.