Homepage Banner_600px.png

Automate Data Ingestion at Scale

BDM Ingest automates the ingestion of data at scale while simplifying the creation and operation of your pipelines.

Let BDM Ingest simplify 
your Data Engineering

Most enterprises have multiple different data sources that are created each day. New tables, new files with names you've never heard of - it's a tough job to keep up! 

BDM Ingests source connectors and automation functionality have been developed for your unique needs - which means you can rest easy knowing your pipelines are always up to date and ingesting data even as your data sources change.

From the first login to fully fleshed out data pipelines, source to destination, BDM Ingest allows you create pipelines in minutes to manage all of your data ingestion needs. 

 
Completely Automated

BDM Ingest uses automation to create an intelligent ingestion engine that can simplify the ingestion of any number of databases, files or other data sources at scale while maintaining high throughputs and low latencies. 

Scalability with Incomparable Speed

BDM uses the native Kubernetes as a Service solution available through your cloud provider, to allow multiple instances of Spark VM’s to be spun up at scale allowing any amount of data from multiple sources to be ingested.

Enterprise Level of Ingestion

BDM runs Spark on Kubernetes, allowing it to be deployed on your cloud of choice (AWS, Azure, GCP) using their native Kubernetes as a Service offering. BDM Ingest also integrates with all major on-premise data sources and environments to enable the integration of cloud and on-premise processing environments.

Massive Time Savings and Productivity Gains 

With BDM Ingest, your data engineering teams will be able to migrate their data to the cloud with an intuitive solution. Not only does it have the most complete suite of connectors, automation and orchestration features, but the software is free! 

BDM Ingest Architecture.png
 

Completely Automated 

BDM Ingest allows you to automate the ingestion of complex data sources, ensuring that as your data source changes dynamically, your ingestion pipelines remain stable. It works with all major enterprise data sources, ensuring all data can be simply ingested onto the cloud.

  • Templates: The ability to build and use templates to ingest complex data sources

  • Variables: The ability to use variables in templates to ingest complex data sources

  • Large Scale Ingest: We have custom solutions to work with all file data sources

  • Orchestration: We support multiple scheduler tools to automate the execution of the ingestion

  • Pipelines:  BDM Ingest automates the creation and the management of your pipelines
     

 

Supports all major cloud environments

BDM Ingest use cloud native services on your account

(Kubernetes, AD/ LDAP) to move data from sources to destinations. 

Your Kubernetes clusters can run on on-premises or in  data lakes and warehouses like Azure, Google Cloud, AWS etc., allowing you to query anything at any time, affordably and securely. 

At all stages of the ingestion process, your data will remain in your environment and will only be accessible to your data engineering team. 

 

Frequently Asked Questions

Select from the following list of Product and Technical FAQs. Browse through these FAQs to find answers to commonly raised questions about creating Pipelines, configuring Sources and Destinations, and working with Models. 

How do I learn more about BDM Ingest?


Reach out here to learn more with BDM Ingest for free and automate data pipeline management with a visual low code builder. Alternatively, you can request a personalized demo from our team.




What data sources does BDM Ingest offer connectors for?


Bluemetrix offers a full suite of Connection Profiles for major data sources - Mainframes, Data Warehouse, Files, Streaming Data - and destinations that includes,

  • Databases: JDBC, etc.
  • Files: JSON, CSV, AVRO, EBCDIC, Text, Parquet, ORC, etc
  • Streams: Kafka & Spark Structured Streaming
We also add new connectors based on customer requests. The more requests we get for a source, the higher we prioritize building the new connectors.




How does BDM Ingest automate the ingestion of data?


Bluemetrix has been working with Hadoop and other Data Lake technologies since 2009, and in that time we have built over 400 enterprise Data Lakes. Using this experience we have developed our own proprietary technology to create an intelligent ingestion engine that simplifies the ingestion of data at scale. The functionality includes:

  • Templates: The ability to build and use templates to ingest complex data sources
  • Variables: The ability to use variables in templates to ingest complex data sources
  • Large Scale Ingest: We have custom solutions to work with most data sources
  • Orchestration: We support multiple scheduler tools to automate the execution of the ingestion
  • Pipelines: BDM Ingest automates the creation and the management of your pipelines




How does Bluemetrix handle changes in the source, such as schema or API changes?


Our pipelines are configured to handle new fields or tables added to your source automatically, so you don’t need to make manual adjustments in the UI. As the schema of your data changes at source, we implement these changes at the destination plus we inform all pipeline owners that consume the source of the changes as they happen, so that they can change their pipelines if necessary. We constantly monitor and stay ahead of API changes or deprecations so you don’t need to think about it.




Do I have to do anything if an API endpoint is changed?


No, the Bluemetrix team will update the connector. BDM Ingest is fully managed, including managing your destination schema in addition to staying ahead of API changes for all connectors.





What cloud environment does BDM Ingest support?


Bluemetrix loads data from any of your pipelines to a Destination system of your choice, including Azure, AWS and Google Cloud. Using their native Kubernetes as a Service offering, BDM Ingest will be deployed and operate in your cloud account/environment.




Is my data secure? Does Bluemetrix store my data?


BDM is deployed in your cloud environment. The data will always be stored in your own environment, no copies of the data are ever moved from your environment, and Bluemetrix will never have access to your data. Bluemetrix is fully GDPR compliant.




What scheduler tools does BDM Ingest support?


BDM Ingest by default is deployed with Control-M from BMC, but it can be integrated with most enterprise sechedulers and our customers will have the ability to use multiple schedulers in their final deployment. Schedulers that are supported are as follows:

  • Control-M - Server version and Helix
  • Azure Scheduler
  • Google Scheduler
  • AWS Scheduler
  • Tivoli
  • Airflow




What can I do if I want more functions in BDM Ingest?


You can check out all the tools Bluemetrix offers on this page. Alternatively, you can speak to one of our experts in areas for more details. We are happy to add new functionality to BDM Ingest, so please feel free to contact us with your requests.




How do I contact the Bluemetrix support team?


Bluemetrix has a dedicated, fully trained support team that supports every customer with an email ticketing system and agreed response time - enabling our customers to meet the requirements of their busy workloads. Please feel free to contact us on the following email address and we will be happy to deal with your request: info@bluemetrix.com





Discover What's New

Call to Action Banner Image.png

Load your data from any source into any destination today. FREE!