• LinkedIn - White Circle
  • Twitter - White Circle
  • YouTube - White Circle
  • Facebook - White Circle

+353 (0) 21 421 2223

Bluemetrix HQ:

5th Floor, River House, Blackpool Retail Park, Blackpool, Cork, T23 R5TF, Ireland.

Spain Office:

Maudes, 51, planta 8 28003 Madrid

Malaysia office:

The Horizon Phase 2,  Level 8, Tower 8, Avenue 5. No 8, lalan Kerinchi, 59200, Bangsar South, Kuala Lumpur, Malaysia

©2019 Bluemetrix | Sitemap | Privacy Policy | Terms of Use

Schema Evolution

Inform Pipeline Owners of schema changes as they occur in source data sets
 

Challenge: Adopt Consistency at Scale

No matter what industry data professionals are in today, they have to ensure that all people within the organization are working from the same datasets. But, this is easier said than done. The issue is more complex than just keeping a record of the schemas at the data sources and data destinations and keeping them in sync. It is also necessary to understand how these schema changes affect existing pipelines which are expecting to receive a specific data schema and now have a different schema available to them. The easiest way to guarantee consistency is to automate the process, and guarantee that schema changes are informed to pipeline owners as soon as they happen.

 

Impact: Maintains Consistency with Central Record of Schemas

Schema changes are identified by BDM Control at source, and all pipelines that are derived from this data source are notified of the change, ensuring that all destinations are always working with the most up to date data that is available. 

 

Solution: Schema & Pipeline Consistency 

We have developed a solution which delivers on the schema and pipeline consistency issues:​​

  • The schema consistency of the data sources is read several times each day (configurable - with the ability to check schemas on-demand) and if changes have occurred these changes are recorded and stored in the central repository

  • All pipelines are recorded in the central repository and a record is created of all schemas that they are consuming data from

  • As changes are recorded in these schemas, the owner of the pipeline is informed and given the option to ignore these changes or to upgrade their pipeline

  • A version control system is also kept in place for pipelines to record which data source each pipeline is running off

 

Opportunity: Building Up-to-date Pipeline

BDM Schema Evolution guarantees consistency across the data. All users on the data lake will inherit their permissions from the original data sources whereas pipeline owners are now always informed of the changes to their data sources, allowing them to keep their data pipelines up to date. With the enhanced Audit Trail, data consistency is guaranteed between data sources and data destinations. 

What data do you need to control?