BM Ingest

The Data Manager Ingest module provides an automated, fully interactive drag and drop interface to ensure that you can easily ingest from all data sources.

  • No need to develop ingest code.
  • Templates can be customised based on data source.
  • New sources deployed in hours not weeks.
  • Simple drag and drop interface.

hadoop-security-vulnerabilities

Just five data points needed for data ingest

Bluemetrix Data Manager just requires the following basic information to ingest data into Hadoop

With this information, Bluemetrix Data Manager will successfully ingest data from an EDW/File onto HIVE in Hadoop.

 

Some of our clients

Ryanair
Toshiba
Swisscom
Pioneer
Hortonworks
Avon

Ingest task automation

During the ingestion process, Bluemetrix Data Manager automates the following tasks:

Hadoop automation data ingest

  • Database schema is checked and updated on an ongoing basis to ensure data stored in Hadoop is always in sync with the data on the EDW
  • Sqoop code is automatically generated to:
    • Copy the EDW schema onto HDFS
    • Move the data onto HDFS
    • Use the schema in HDFS to create Ext. and Orc. files in HIVE
    • Move the data from HDFS onto HIVE
  • All data types from the original EDW are changed into data types that are compatible in HIVE
  • Data is cleaned and normalised to ensure that all of the data is compatible and works in HIVE

Ingestion connectors are written and available for most data sources, including:

  • EDW – Oracle, Teradata, SQLServer, MySQL
  • Files – Syslogs, JSON, etc.

Where a connector is not available for a particular data source, it can usually be developed and deployed with just two weeks work.

Working with Fortune 500 companies across all industry sectors

Finance, Telecom, Automotive, Retail, Insurance, Pharmaceutical, Industrial & Government – helping them on all aspects of deployment from Architecture, Infrastructure, Security, Operations, Development and Data Science

industries

Next: Hadoop Data Transformation

Request more information

For more information on our Hadoop Data Ingest please leave your details below.