Search Results

The EU AI Act has emerged as the cornerstone of EU policy on the deployment and adoption of artificial intelligence (AI) by enterprises, where it attempts to mitigate potential risks, safeguard human rights, and build trust in AI development and deployment across the EU. This groundbreaking legislation not only sets rigorous standards for AI development but also champions responsible data governance, data quality, and transparency. In recent years, the landscape of AI technology has undergone remarkable evolution, presenting organisations with unprecedented opportunities and challenges. For organisations venturing into AI initiatives, compliance with the EU AI Act isn’t merely a legal obligation; it’s a strategic imperative, especially when dealing with sensitive data. In this blog, we explore the pivotal highlights of this regulation, its implications and the steps organisation can take to achieve full compliance. Understanding the EU AI Act The EU AI Act stands as a pioneering regulation designed to govern artificial intelligence systems, placing a strong emphasis on upholding fundamental rights, and preventing potential AI-induced harm. Notably, the Act classified AI systems into four risk levels, with the highest category banning systems like mass social scoring and real-time biometric surveillance. It also mandates transparency obligations on AI models, including clear labelling of AI-generated content. While its immediate impact is felt within the European market, the influence of the EU AI Act extends globally, representing a significant milestone in AI regulation. Key Provisions and Their Impact The Act's provisions span various facets of AI development and deployment, each with implications for the AI landscape: Banning Threatening AI Systems: The legislation prohibits AI systems deemed to pose a clear threat to human safety, livelihoods, and rights. This includes imposing stringent regulations on high-risk programs employed in critical infrastructure, law enforcement, and elections. Regulating Government Surveillance: Acknowledging the potential misuse of AI in biometric surveillance and social scoring, the EU AI Act imposes restrictions on intrusive applications. Specifically, real-time facial recognition in public spaces is constrained, with exceptions granted for specific law enforcement purposes. Transparency Requirements: The Act mandates transparency obligations for AI models prior to market entry, particularly focusing on foundation models like ChatGPT. AI-generated content, including manipulated images and videos (such as deepfakes), must be clearly identified as AI-generated, thus mitigating the potential spread of misinformation and manipulation. Risk-Based Approach: One of the core principles of the EU AI Act is its adoption of a risk-based approach to AI regulation. This means that AI systems are classified into four risk levels: unacceptable, high, limited and minimal/none, based on the degree of threat they pose. High-risk AI systems face specific legal requirements that includes: Registering with an EU database Implementing a compliant quality management system Undergoing conformity assessments. Implications and Enforcement The EU AI Act is poised to become law soon, with phased rollout planned across member states over the next three years. Its reach extends beyond European boarders, meaning that AI providers accessing the European market must adhere to its provisions to protect European citizens' rights. While the formal adoption is set for April 2024, businesses are granted a 24-month grace period to achieve full compliance. Non-compliance with the Act can result in hefty penalties, ranging from €35 million or 7% of global revenue to €7.5 million or 1.5% of revenue, depending on the infringement. Impact on the Banking Sector As the EU AI Act lays down the groundwork for responsible AI governance, its implications are expected to transcend mere compliance, gradually evolving into a de facto standard across various sectors, including banking. Institutions operating within this realm, especially those leveraging general-purpose AI system like LLMs, are urged to carefully assess the Act’s key provisions. Whether explicitly for regulatory compliance purposes or implicitly to enhance data governance, systems, process and reporting capabilities. For banks aligning closely with the Act’s directives, the benefits are manifold. They become better equipped to adapt to future changes, more agile in responding to threats and opportunities, and may face less regulatory scrutiny due to higher confidence. Achieving the Full AI Compliance Full AI compliance demands a shift from a fragmented approach to strong integration and cooperation, particularly between the Risk, Governance and Data departments. If any warning signs appear in current data access control system, it's time to take decisive action. Here are some steps to achieve full compliance: Perform a comprehensive compliance health check assessment to identify vulnerabilities and gaps in your existing controls and knowledge base, while identifying clearly the risk levels you are undertaking on your AI projects. Choose an automated data governance engine with patented security feature that seamlessly integrates with data catalogs like Collibra, Atlas, Alation or Informatica. Implement data quality policies and automate the tokenization of PII data for all data sets that are used to train and run AI Models. Get an auditable query log allowing organisation to validate the effectiveness of the data controls. This log should provide insights into general and sensitive data access. Design your engineering environment so that compliance with the AI Act is an output of the process (i.e. logs, audit records, documentation, etc.), which will expedite the compliant usage of AI within your organization. From the design stage of your AI project, ensure that Human oversight is designed into the complete process. Remain updated on emerging threats and regulatory changes to adjust access controls and address new risks and compliance mandates. Bonus Tip: Start with Automation As technological advancements transform our world, navigating the evolving EU AI regulation requires collaborative efforts and ongoing vigilance. Bluemetrix brings deep regulatory expertise and proven solutions that help banks and modern companies build the vision, strategies, and data capabilities needed for compliance. Our AI/Gen AI Health Check approach, complemented by NIST FIPS 140-3 compatible ETL solution, enables organisations to drive more visibility, trust, and automation into their data and AI practices. Discover how Bluemetrix can help your organization chart a successful path forward in the era of responsible AI. Request a free data consultation today and take the first step towards building a future-ready, compliant data ecosystem.

Data tokenization, data masking, and data encryption are three data security techniques that are often confused. Here's how they differ and the challenges you may encounter. Data security involves three crucial pillars of protection: data tokenization, data masking and data encryption. While these terms are often used interchangeably, each plays a distinct role in fortifying data integrity and confidentiality. As cyber threats continue to evolve, organisations must deploy the most appropriate data security solution for each project and use case. This blog aims to demystify these controls and highlight their differences to help you manage your data securely. Jump to: Defining Data Tokenization Defining Data Masking Defining Data Encryption Key Difference between Data Tokenization, Masking and Data Encryption Tools and Technologies Defining Data Tokenization: Data Tokenization is a security measure that replaces sensitive data with surrogate values called tokens. These tokens, acting as references, enable reversible retrieval of the original data through an authorised tokenization system – a process known as de-tokenization. While reversible schemes maintain token-to-data mappings, some implementations omit for irreversibility. Data Tokenization provides high security for data at rest and in motion, often deployed extensively within payment processing services. There are two types of Tokenization techniques: Stateless Tokenization: This method operates without needing a mapping database/table to maintain token-to-value relationships. It's efficient, scalable, and interoperable. It is commonly used for one-time transactions or when consistent anonymisation is required without retrieval. Stateful Tokenization: It involves maintaining a reference or mapping between the token and the original value. Requires a mapping database/table to store token-to-value relationships. Used when there is a need to retrieve the original data from the token Challenges: It relies on a centralised token vault, which can be complex to implement and manage. Advantages: High security as original data remains within the storage premises; performance benefits over encryption; compliance with standards like PCI DSS, GDPR and BCBS 239. Defining Data Masking: Data masking, or data anonymization, refers to obscuring sensitive data with randomised values using various data shuffling and manipulation techniques. Crucial for privacy protection, this technique ensures that confidential information remains hidden from unauthorised access. Whether in testing environments or production systems, the irreversible nature of masking safeguards confidentiality without compromising data utility. Data masking is commonly used to protect PII data and comply with data protection regulations like GDPR, PCI DSS and BCBS 239. Data masking can be performed in various ways: Static Data Masking: It involves altering sensitive data at rest, typically in a non-production environment, to permanently ensure privacy and data protection without altering the original data. The masked data must match the original data for accurate results and will be loaded into separate environments. Dynamic Data Masking: It hides real-time, sensitive data as users access or query it, preserving the original data at rest. This approach is well-suited for role-based data security and is commonly deployed in production systems to avoid separate storage. However, it may face consistency issues with multiple systems. On-the-fly Masking: Obscures sensitive data during its movement or transfer without retaining altered data in the database. This technique proves helpful in scenarios with space constraints or when data must swiftly transition between different source locations, making it ideal for continuous software development, skipping staging delays. Challenges: Masked data is often irreversible, making it unsuitable for scenarios where the original data needs to be retrieved. Advantages: It reduces the risk of data exposure, is easy to implement, and supports compliance with regulations like GDPR. Defining Data Encryption: Data encryption is the most commonly used method of encoding original data (unencrypted plaintext) into unreadable form (encrypted ciphertext) using an algorithm and a cryptographic key. The main difference between tokenization and encryption is that tokenization utilises tokens while encryption employs a 'secret key' for safeguarding data. Despite its reversible and breakable nature, encrypted data is treated as sensitive and considered a strong defence mechanism. Depending on the keys used, there are primarily 2 types of encryption keys: Symmetric Key Schemes – The same key encrypts and decrypts text. Public Key Encryption—Encryption and decryption are performed using different keys, namely a public key (known to everyone) and a private key (secret key). In terms of encryption algorithm: Deterministic: Provide a single outcome for the same input; beneficial when sharing the data. Non-Deterministic: Produces different outcomes for the same plaintext; hardly used in real-life scenarios. Challenges: Key management can be complex; encrypted data can be decrypted with sufficient resources. Advantages: Widely adopted and understood; can secure entire files or databases; supports data sharing. Key Difference between Data Tokenization vs Data Masking vs Encryption Understanding the nuances between Masking, Tokenization, and Encryption is pivotal for crafting data protection strategies that cater to diverse organisational needs and effectively mitigate evolving cyber threats. Here's the comparison of Data Tokenization vs Data Masking vs Data Encryption: Tools for Data Tokenization vs Data Masking vs Data Encryption: Various tools and technologies support both data tokenization and data masking. Among the most used tools that seamlessly integrate with all ETL tools and facilitate the creation of a unified data environment are: Bluemetrix Protegrity Privitar - Informatica Thales IBM Security What's Next? To sum up, the choice between tokenization, masking, and encryption hinges on an organisation's specific needs and context. Factors such as the nature of data, regulatory requirements, and the operational environment all contribute to determining the most appropriate data security method. Masking, for example, is ideal for organisations seeking to balance privacy protection with data utility. On the other hand, Tokenization is better suited for organisations prioritising compliance, particularly with standards like PC1 DSS or GDPR, especially for long-term storage and analytics purposes. Encryption, meanwhile, is more appropriate for facilitating secure remote work scenarios by enabling the safe exchange of sensitive information among authorised users with access to encryption keys. Here's a quick summary of the use cases and suggested approaches. Bluemetrix's latest automation release revolutionizes data governance, security, and LakeHouse integration, ensuring seamless, continuous security management. Trusted by global leaders, Bluemetrix's data tokenization and masking solution, which is NIST—and FIPS 140-3 compliant—empowers organizations to innovate confidently while staying ahead of privacy provisions and penalties. Explore our product pages or connect with our team for firsthand experience implementing data governance and security control using Bluemetrix.

In today's digital age, syncing data governance with evolving pipelines is more than just a best practice – it's a fundamental requirement for ensuring accuracy, compliance, and real-time insights. Data governance, in its essence, demands an unswerving alignment with the data it oversees. The ever-changing data landscape places a significant demand on organizations, requiring not only diligent data management but also adaptability and responsiveness. When we delve into the realm of digital transformation, it's clear that data collection and analysis lie at its core. Data forms the bedrock of this transformative journey. For instance, consider the financial services sector, where data acts as the lifeblood of organizations, offering insights into customer behaviors, financial needs, and culminating in the creation of tailored financial products. Without data, this transformative process would remain an unattainable dream. The Importance of Data Governance Every organisation requires a robust data governance policy that is, most importantly, synchronized with its data pipelines. Maintaining this synchronization isn't a one-time effort but an ongoing process, to ensure that data flowing into the organisation from diverse sources or pipelines is handled uniformly. For instance, the data must undergo rigorous checks for errors, ensure consistency, and align with the organisation's established governance policies and procedures. Gartner's statistics, showing that poor data quality cost organisations an average of $15 million in 2017, underscore the necessity of robust data governance, particularly in today's data-intensive business landscape. If your data governance operates independently of the pipeline, you risk missing crucial data changes, whether they occur in real-time or on a daily basis. This lack of synchronization can significantly impact how you manage real-time data within your overaching data governance policy. Addressing Real-time Data Governance Challenges The demands of real-time data governance present formidable challenges that must be carefully addressed. Tools like Collibra and Alation play a pivotal role in maintaining data consistency; however, ensuring real-time alignment with evolving data is a complex endeavor. This challenge is further compounded by the continually shifting landscape of data generation, storage, and processing methods employed by modern organisations. Today, organisations grapple with data stemming from an array of sources, ranging from conventional on-premises servers to cloud-based solutions, mobile applications, and IoT devices, and advent of AI. Gone are the days when data was predominantly stored on-premises, with manageable volumes. Now, data accumulates at an exponential rate, sourced from various channels - on-premises, cloud-based applications, and mobile devices. It's not solely the magnitude of data that poses a challenge, but the diversity and rapidity of data generation. Data is no longer confined to neatly structured databases; it emerges in unstructured and semi-structured formats, often arriving in real-time. This complex data landscape underscores the utmost importance of staying aligned with data governance. If your data governance falter in its synchronization with the data, you open the door to pitfalls like inaccurate reporting, potential breaches of regulatory frameworks such as GDPR and BCBS 239, and the consequential erosion of your organization's profitability. Embracing Synchronicity in Data Governance Synchronizing data governance with evolving data pipelines is not merely a best practice; it's an absolute necessity. To achieve this, a system-agnostic platform takes center stage, automating governance and engineering tasks, enhancing efficiency, and reducing the potential for human errors. What distinguishes a system-agnostic platform is its remarkable adaptability and versatility. It isn't confined by specific technologies or vendors, enabling it to govern data seamlessly from a myriad of sources. In today's data landscape, marked by its dynamic, real-time flow, such adaptability is paramount. The benefits of keeping your data governance in perfect sync with your data are diverse and profound. It empowers real-time decision-making founded on accurate, up-to-date data. Moreover, it ensures compliance with data protection regulations, safeguarding your organization from potential legal issues. From financial services to healthcare, precision in data governance is the linchpin for success. Inaccuracies can lead to stagnation rather than transformation. Consider healthcare providers leveraging data for enhanced patient care or financial institutions tailoring personalized services through data analysis. The possibilities are boundless when data governance aligns seamlessly with the evolving data landscape. Bluemetrix: Redefining Data Governance Operations True synchronization extends beyond mere data cataloging or metadata management. Bluemetrx's system-agnostic, automated platform operates as the heartbeat of your data landscape, ensuring that every data point stays in perfect harmony with your governance policies. Our platform has the following in store for you! Automation with Precision: We seamlessly automate data governance and engineering tasks, eliminating the potential for human errors while accelerating processes. Real-time Adaptability: Our platform effortlessly adapts to the dynamic data landscape, ensuring your governance policies evolve in real-time. Versatility and Consistency: Designed to be system-agnostic, we're primed to govern data from diverse sources, technologies, and vendors. This versatility guarantees your data remains consistently synchronized with governance policies. Up-to-the-Minute Insights: We capture metadata at the pipeline's creation stage and automatically update data catalogs, ensuring data owners and analysts always have the most current insights at their fingertips. Curious to explore what Bluemetrix has to offer? SIGN UP here for a free demo and experience the feature-rich Bluemetrix. Synchronizing for Success: A Necessity, Not an Option In a data-driven world, synchronization is the key to unlocking data's full potential. Keeping your data governance in sync with your data pipelines is not a mere option; it's a critical requirement. The consequences of falling out of sync are far-reaching and detrimental. Inaccurate data can lead to flawed decision-making, and non-compliance with data protection regulations can plunge orgnisation into legal trouble. It's a risk that no organisation can afford. Investing in a synchronization platform is an investment in the future of your organisation. It's a commitment to accurate, real-time data governance that will propel your business forward in the ever-evolving digital landscape. Data governance is not dynamic, not static. The future belongs to those who sync, not those who sink. For additional insights on capturing data changes, please explore our guide on ''How to Capture Data Changes with Data Governance Automation'.

Navigate the complexities of BCBS 239 compliance and achieve regulatory success with Bluemetrix. This blog post unravels the fundamental aspects of this regulation and provides best practices to expedite implementation for your organisation. In the fast-paced world of global finance, regulatory bodies are increasingly focused on managing risk and ensuring data quality. One of the top priorities for financial services sectors seeking to meet regulatory standards is compliance with BCBS 239. However, risk and governance executives constantly face obstacles in achieving and maintaining compliance due to disparate IT systems and reliance on manual workarounds. Whether you're new to the regulation or looking for a proven solution, this blog post will unravel the fundamental aspect of BCBS 239, common reference architecture patterns, and best practices that help you achieve compliance success quickly. What is BCBS 239? BCBS 239, also known as the Basel Committee on Banking Supervision's Standard 239, is a comprehensive set of 14 principles geared towards improving governance and infrastructure, in addition to their risk data aggregation and reporting capabilities. Initially published in 2013, it applies to Global Systemically Important Banks (G-SIBs) and Domestic Systemically Important Banks (D-SIBs). While BCBS 239 does not impose sanctions for noncompliance, regulatory bodies are still meticulous in assessing an institution's standard implementation. As such, it is critical for banks to incorporate it into their regulatory transformation programs. Failure to comply with BCBS 239 can result in several consequences for banks: Increased regulatory scrutiny from watchdogs Reputational impairment leads to customers' and investors' attrition. Financial losses due to poor risk management and decision-making processes. Legal and regulatory action with potential fines or other penalties Who does BCB239 protect? At a high level, the implementation of BCBS 239 bolsters the safety and stability of the banking sector, shielding its stakeholders – such as shareholders, investors, customers, and counterparts – from various risks. Through enhanced transparency regarding risk profiles, regulators are better equipped to conduct thorough systemic risk assessments which promote the long-term health of the sector. Adherence to BCBS 239 compliance not only offers greater protection for banking stakeholders but also provides financial institutions with multiple competitive advantages. These range from improved efficiency to increased profitability, as well as reduced chances of incurring losses and more effective strategic decision-making processes. All these benefits can contribute significantly towards boosting investor confidence in the banking system and minimising any potential economic losses. BCB2S239 key principles BCBS239 encompasses 14 best practices, with 11 targeting banks and the remaining three principles catering to the national and regional regulatory bodies. Principle for Banks: Principle 1: Governance framework and Robust data architecture and IT infrastructure Principle 2: Robust data architecture and IT infrastructure Principle 3: Accuracy and Integrity Principle 4: Completeness Principle 5: Timeliness Principle 6: Adaptability Principle 7: Accuracy Principle 8: Comprehensiveness Principle 9: Clarity and usefulness Principle 10: Frequency Principle 11: Distribution Principles for National and Regional Regulatory Bodies Principle 12: Supervisory review, tools and cooperation Principle 13: Remedial actions and supervisory measures Principle 14: Continuous improvement Common reference architecture patterns and solutions Incorporating BCBS 239 compliance within a reference architecture stands as a pivotal stride for any financial institution striving to meet the standards outlined by BCBS 239 principles. Despite some progress made since its initial introduction, the BIS revealed in a 2019 compliance assessment that many G-SIBs still lag behind in conforming to BCBS 239's requirements due to their reliance on outdated, manual processes. Bluemetrix presents an improved solution to this problem by offering automated capabilities that facilitate the adherence to the first seven core principles of BCBS 239 – from data and processes to business and regulatory requirements. By leveraging data automation technology, Bluemetrix Data Manager (BDM) can effortlessly collect information from diverse sources, consolidate, cleanse, and transform it into easily digestible reporting insights within a strict timeframe. Principle 1: Governance framework and clear roles and responsibilities The deployment of an automated governance and meta-management tool to capture, store and oversee the business and technical metadata for data assets becomes imperative. This feature facilities data lineage, data quality management, and data traceability, all integral to BCBS 239 compliance. With a repository of pre-configured content encompassing policies, data categories, critical data elements and more, BDM empowers the risk management team to build and operate a solid data governance framework. This entails defining and enforcing data governance policies and standards, enabling stakeholders to respond systematically to and handle data breaches appropriately. Principle 2: Robust data architecture and IT infrastructure Creating a strong data & IT architecture presents a multifaceted challenge. The integration of multiple standalone components often leads to fragmented systems that lack coherence. In response to this concern, Bluemetrix offers a comprehensive solution with over 150 out-of-the-box integrations across key IT systems in the market, including AWS, GCP, Azure, Hadoop, Databricks, Teradata, Collibra and more. By leveraging these integrations, the capacity to aggregate and automate data operations across the necessary infrastructure is made attainable. Principle 3: Accuracy & Integrity of data Data quality forms a foundational pillar of BCBS 239, essential for generating accurate and complete risk data. The provision of visual evidence for auditors and streamlining the error-resolution process during reporting hold significance. Bluemetrix presents an all-encompassing, no-code solution with extensive data lineage and traceability, facilitating the monitoring of data source, transformation and movement throughout the entire data lifecycle. This functionality ensures the integrity and accuracy of reporting. Connections between data elements and business concepts are established by leveraging both business and technical metadata, offering a clear understanding of the relationships within these data. Principle 4: Completeness of data To comprehensively capture and aggregate all risk data across the banking group, considering various aspects like business line, legal entity, assets, industry, region and other groupings. BDM facilitates both business and IT users to precisely define and extract specific data details for analytical reporting and output. It offers granular-level data and summary entities to aid risk data aggregation. Our platform ensures robust and reliable risk data aggregation capabilities by swiftly and accurately tracking data flow from source to target, including all intermediate stops along the way. This seamless tracking ensures the production of data that aligns with stratified requirements, supporting informed decision-making and comprehensive reporting. Principle 5: Timeliness of risk data Bluemetrix understands the importance of collecting, aggregating, and promptly reporting data. Our automation capabilities serve as an accelerator, enabling the data team to efficiently gather and deliver risk data within specified timeframes to fulfil reporting obligations effectively. Manual procedures may introduce potential delays and impede your ability to respond promptly in critical situations. Bluemetrix's automation tools facilitate the implementation of reporting requirements, data aggregation, and timely presentation. By clearly defining reporting requirements and linking them to the relevant IT components, it becomes easier to identify the necessary information and promptly implement it. Principle 6: Adaptability of risk data BDM offers a wealth of content covering a wide array of regulations, including Basel, GDPR, and more. The detailed layers within the content allow users to navigate to their required level easily. This comprehensive coverage caters to both prescribed regulatory requirements and ad-hoc reporting. Additionally, BDM is highly adaptable and can be customised to fit your specific use case within your big data environment. Regular releases are released to reflect new banking industry regulations. If required, the modules of BDM can be further customised upon request. Principle 7: Accuracy of risk data When relying on manual processes, there is a higher likelihood of errors in accuracy and completeness, particularly during times of crisis. As data volumes and users increase, the risk of unauthorised access also grows. Data teams face the challenge of navigating extensive audit data across multiple external systems. Given the complexity of this data landscape, it is vital to have comprehensive data coverage and granularity to maintain accuracy. The ability to drill down into reporting allows teams to define the necessary level of detail. Bluemetrix underscores the indispensable role of automated data lineage tools in fulfilling BCBS 239 compliance. Our solution guarantees that even the largest datasets consistently reach business and reporting applications in the correct format and in a timely manner. This capability ultimately empowers data teams to capture compliance for critical data elements easily. By utilising detailed lineage and reporting, your team can resolve & mitigate risks related to data aggregation and reporting, maintaining dependability even in high-stress scenarios. Why Intelligent Data Automation? Financial institutions striving to achieve and maintain BCBS 239 compliance can benefit from harnessing the power of Bluemetrix's data automation capabilities. Efficient Data Management: Bluemetrix provides a centralised platform that streamlines data collection and distribution, ensuring efficient data management. User-Friendly Interface: The platform is designed to be user-friendly, allowing governance personnel to easily create and implement comprehensive data policies without the need for extensive documentation or coding. Compliance with BCBS 239 Principles: Bluemetrix adheres to the first seven principles of BCBS 239, ensuring compliance with industry regulations and promoting a culture of risk management and data governance. Promotes Growth and Efficiency: By implementing Bluemetrix's solution, businesses can achieve growth and efficiency by effectively managing their data, reducing manual efforts, and ensuring data accuracy and timeliness. Seamless Digital Experiences: Bluemetrix enables businesses to build and secure seamless digital experiences by providing reliable and accurate data for decision-making processes. Bluemetrix Data Manager provides financial institutions with an invaluable tool for meeting BCBS 239 compliance requirements while building capacity for risk management, data governance and digital experience optimisation. With the help of this powerful automation technology, businesses can unlock growth opportunities while ensuring the accuracy, timeliness, safety, security and consistency of their data - all at once. Request a demo of Bluemetrix to see firsthand how seamless your BCBS 239 can be. *Disclaimer: This article is for risk managers, data governance and engineering teams addressing BC239's9’s data-intensive requirements. It showcases the Bluemetrix Data Manager and Control-M as best practices. The patterns presented are options provided by Bluemetrix to aid compliance with the first seven principles of BCBS 239. Note that this article does not provide legal advice. Customers should review applicable privacy requirements and consult internal privacy and legal teams to determine suitable & acceptable data solutions. Bluemetrix does not guarantee compliance with any legal or regulatory requirements before, during, or after an engagement.

In today's digital landscape, data transformation is essential for every industry. Whether you're a financial institution handling massive transactions globally or a non-profit organization integrating data from multiple sources, effectively extracting value from collected data requires efficient transformation. As data experts, Bluemetrix offers a robust and automated ETL/ELT platform that provides seamless data transformation. Our comprehensive BDM Spark Data Transformation guide includes over 250 pre-built data models that streamline data preparation across diverse technical disciplines. For more information, download the guide to learn how our solution can assist you with your data transformation requirements.

Joint customers now have access to Bluemetrix’s data automation solution – Bluemetrix Data Manager (BDM) for the Cloudera Data Platform (CDP). Bluemetrix, the leading provider of cloud data solutions, today announced its certification on the Cloudera Data Platform (CDP). As a Cloudera Certified Information Software Vendor (ISV) partner, the Bluemetrix Data Manager has undergone rigorous testing and validation to ensure seamless integration with the Cloudera Data Platform. This significant certification empowers joint customers to leverage the full potential of the CDP data stack, enabling them to derive valuable insights from their data. It also represents the culmination of Bluemetrix’s longstanding relationship and dedicated effort since 2016 to collaborate with Hortonworks and Cloudera, with successful testing by Cloudera further affirming the software’s capabilities. “Big Data technology has been a major growth catalyst in the Enterprise sector, and for the past 15 years, Bluemetrix has played a pivotal role in delivering top-notch big data services, “said Liam English, CEO of Bluemetrix. “Obtaining the ISV competency is a source of great delight for us. It underscores our commitment to delivering an exceptional experience to our valued customers, enabling them to foster innovation and optimise their data-driven strategies. We remain steadfast in our mission to stay at the forefront of the industry and support our customers with comprehensive technology solutions that maximise their investment in Cloudera’s cloud infrastructure.” CDP is a comprehensive cloud-native data and analytics platform designed to help organisations effectively handle and analyse large amounts of data across public, private and hybrid clouds. With its new certification on CDP, the Bluemetrix Data Manager is empowered to fully utilise the platform’s cutting-edge technology, such as its security framework and scale-out architecture, providing customers with a unified solution for all their data processing needs. In addition, this certification also acknowledges Bluemetrix’s in-depth expertise and proficiency in Cloudera technologies, validating the effectiveness of the Bluemetrix solution in meeting customer requirements within Cloudera environments. Through a strengthened partnership with Cloudera, Bluemetrix expands access to a broader range of tools, resources, training, and direct technical support, resulting in enhanced profitability and exceptional customer satisfaction. Jeremiah Jacquet, CTO at Bluemetrix, highlights the advantages of the combined Bluemetrix and Cloudera technologies, stating, "Our collaboration with Cloudera provides privileged access to the Bluemetrix application, enabling our customers to benefit from more advanced data management and governance capabilities." Businesses rely on Bluemetrix solutions to ensure the completeness, cleanliness, and integrity of their data in Cloudera. With Bluemetrix’s unique combination of ETL and governance automation capabilities, customers can now streamline and accelerate the creation and management of data pipelines at scale – in the cloud and on-premises – making it readily accessible and available to all relevant data stakeholder who needs it throughout an organization. About Bluemetrix Bluemetrix is a leading provider of cloud data solutions, providing innovative solutions in Finance and Healthcare sectors. Our mission is to help enterprises get more value from their data, combining business strategy, innovative approaches, and technology. Since 2009, we have delivered over 400 Hadoop & Big Data Implementations across all major industrial sectors in Europe, APAC and the US. We continue to work closely with our customers to overcome technology gaps or skill set shortages. For a look at our integration in action, visit the Cloudera Partners Listing or schedule a demo.

As the digital world becomes more intricate, with 5G and the Internet of Things (IoT) taking center stage, and the influx of customer data growing exponentially, one constant holds true: Automated data validations integrated into the overall data preparation process are crucial for maintaining accurate and timely data amidst rapid data changes. They're the fundamental processes of any successful organisation, enabling informed decisions, increased efficiency and maximised profits through quality data. Download the guide to learn more.

Certification Underscores Expertise with OMOP Common Data Model (CDM) Ecosystem to Harmonize Real-World Clinical Data for Deeper Analytics Insights Cork, Ireland – October 20, 2022 – Bluemetrix, a leading provider of cloud and big data integration solutions, today announces that it has been certified by European Health Data and Evidence Network (EDHEN) to help EHDEN Data Partners map their health data to the OMOP Common Data Model. Bluemetrix will now work with EHDEN Data Partners to increase adoption of the OMOP Common Data Model at scale, ensuring various data sources are available and usable to the wider research community, with a data platform supporting healthcare service and real-world research needs. EHDEN, an Innovative Medicines Initiative public-private project, was launched in 2018 to leverage and further improve the OMOP CDM ecosystem. EHDEN has built a federated network currently standing at 166 Data Partners for the secure analysis over observational clinical data from millions of European patients in a standardised format. EHDEN is now running its final open call for European Data Partners, which will close on 11 November. “Every day, patients across Europe are not receiving optimal healthcare because we are not maximising the transformational potential of real-world data." In response to this, “EHDEN aspires to be the trusted observational research ecosystem to enable better health decisions, outcomes and care for patients across Europe.” As a newly certified EHDEN SME, Bluemetrix is committed to working closely with EHDEN Data Partners by providing unparalleled OMOP data harmonization service and accelerating healthcare innovation, all of which will ultimately deliver operational healthcare analytics environments. Bluemetrix is the first Irish software vendor to complete this OMOP training, demonstrating our capability and commitment to work with the national and international healthcare sectors in delivering innovative healthcare data solutions. Bluemetrix’s state-of-the-art technical platform will connect major hospitals, primary care networks and regional databases, allowing patient-focused research in a shared care record environment with such services: Consultancy on design, architecture, and test of data infrastructure Mapping of source data to OMOP CDM vocabularies Provision of data integration and ingestion tools to populate the data environment Design, implementation, and deployment of ETL data pipelines Implementation of data governance programs Evaluation of EHDEN ecosystem infrastructure at Data Partner’s site Expand the EHDEN ecosystem and configure OMOP CDM and tools in this ecosystem “Partnering with EDHEN is a significant step for Bluemetrix to deliver more value-added healthcare data for research and operational purposes. The integration of OMOP data harmonization efforts and ETL technologies will help healthcare data users open new routes to data analysis. We are extremely proud to be part of this movement and involved in this flagship programme across Europe and beyond." Liam English, CEO at Bluemetrix. To view the certification in participation of EDHEN program, please visit here. For more information on Bluemetrix and its complete suite of data integration applications, visit www.bluemetrix.com/bdm-health. About Bluemetrix, LTD Bluemetrix is a leading provider of cloud data solutions, providing innovative and solutions in Finance and Healthcare sector. Since 2009, we have delivered over 400 Big Data Implementations across all major European enterprises in all industry sectors. We work closely with our customers and provide data automation solutions to cover any technology gaps or skill set shortages that exist on their side. We have extensive experience of dealing with healthcare data (EHR data, Pathology data, etc.) and have developed and deployed Data Lakes in Europe which combine patient data from multiple sources, which deliver research and operational analytics.

Extracting complex JSON data for analysis like a pro with Bluemetrix, IDERA and Collibra. If you’ve ever worked with data stored in JSON files, you probably know that extracting data from nested JSON files for processing can be daunting and time-consuming. Data stored in JSON files make up a whopping 80% or more of data collected by large-scale enterprises. In addition, standard BI Tools are designed for relational data and struggle to process JSON data. The easiest way to do so is to extract the data from the files and store the data in a relational database like Snowflake, Google BigQuery, Databricks, etc. where it can be easily queried and processed using standard tools. This blog explores an integrated automation solution combining features from ER/Studio, Collibra and Bluemetrix Data Manager, which simplifies this extraction process and allows you to extract data at scale from nested JSON files. Code-Free JSON Data Extraction with Bluemetrix, IDERA and Collibra Reporting on a JSON data is not easy. So, our mission is to take data from a large number of hierarchical JSON files, either in a simple folder structure or on a data lake platform, unpack it and load it into a collection of related tables in a database. Then we can run the BI tools we normally use against the data and even link that data with existing data sets. There are three parts to the challenge: 1. To design the structure of our relational database that will contain our data 2. To unpack the data in our JSON files and load it into those tables 3. To ensure that sensitive or private data is protected along the way Doing this without tools would be time consuming and error prone. The right tools can make this a breeze. Let’s learn a little about those tools. Bluemetrix Data Manager (BDM) is a platform-agnostic Spark-based ETL solution that simplifies the creation of data pipelines for the movement and transformation of data. It works with multiple data sources and targets, including JSON files. ER/Studio is a data modelling tool for the design and documentation of data assets such as databases and JSON files using graphical business-friendly models. Collibra Data Intelligence Cloud is the leading enterprise cloud platform for capturing and recording data governance. Combining BDM with the data modelling functionality available in ER/Studio and the Data Governance features available in Collibra, provides a no-code automation solution that simplifies and expedites the extraction of data from nested JSON files. Let’s examine in more detail how this works. Step 1: Create Target Data Store and Models for Nested JSON Files Our nested JSON files have data stored in a hierarchical structure. ER/Studio will reverse engineer this JSON and create a physical model of the source JSON and a logical model of the information within it. The user will enhance the logical model by adding primary keys and moving it through the process to 3rd Normal Form. Step 2: Create Physical Model Next, we automatically generate a physical model choosing Snowflake as the target technology from the logical model created in Step 1. You can then generate DDL code and deploy this to your Snowflake instance. Step 3: Synchronize the Business Glossaries of Collibra and ER/Studio Team Server ER/Studio Team Server has built in features to synchronize with Collibra. In this example, we have a business glossary term ’patron number’ which has been flagged as PII and has a Sensitivity Level as Critical. These values will have been set by Data Stewards in Collibra. Step 4: Use Data Architect in ER/Studio We map fields in ER/Studio’s physical and logical models to these glossary terms. You can also apply classification properties directly to fields. Step 5: Publish in Collibra The logical model of the information in our JSON along with the physical schemas of the JSON file and the target Snowflake database are now published to Collibra along with mappings to the Business Terms and direct classifications. Step 6: Read These Models from Collibra into Bluemetrix Data Manager (BDM) In BDM we read the schema information for the JSON files from Collibra, for the source and target database. With this information we then read the source and ingest the data into memory. Step 7: Read the JSON files and use the models to explode the data and transform it into relational tables in Snowflake. Once the data is available to us in memory, BDM will then explode this data and write it into a data frame for further processing. Step 8: Identify PII tag in Collibra BDM will read the data policies which apply to this data source from Collibra. In this example, we can see that the column patron number has PII tags, which means the data in this column is sensitive and needs to be tokenized. We know this data needs to be Tokenized because a Ruleset has been created in BDM which states that any data that is tagged PII has to be Tokenized. This means that before the data in this column is written to the table in Snowflake it will be automatically tokenized. Step 9. Run the pipeline Now we will run the pipeline and write the JSON data from the dataframe into a relational table in Snowflake. And we can see here that the data has been converted into a tabular form, and a foreign key relationship has been created between the tables using the ISBN number. This means that any BI tool user can now execute standard queries against this normalized relational data. And per the data policy rules, the patron number data has been tokenized when it was written into the table. Step 10: Capture the Lineage in Collibra And finally, we will look at Collibra once more, where we can see the lineage that has been captured for this pipeline. We can see that the JSON files ingested have been turned into relational tables in Snowflake, while the appropriate fields have been tokenized before the data was written into Snowflake in compliance with the Data Policies in Collibra. How can data-driven organisations benefit from the integration? This integrated solution from Collibra, IDERA and Bluemetrix strives to make the impossible possible and the hard easy. We make extracting data from complex JSON files at scale simple and quick, while doing so in a secure manner which is compliant with Data Governance policies, as outlined in the previous 10 steps. By converting the data into a tabular form, while establishing a foreign key relationship between the tables using the ISBN number, this means that any BI tool user can now execute standard queries against this normalized relational data. All of this allows analysts and business users analyse the data using their existing BI tools and skill sets, allowing them to extract the value from the data that was previously locked in the JSON files. They can carry out all this work in a controlled and compliant manner, ensuring that the data is processed in compliance with the Data Policies in Collibra, where all the lineage is captured ensuring that activities can be tracked and available for auditing for GDPR purposes. Bluemetrix, Collibra and IDERA’s partnership takes a proactive approach to JSON data extraction and simplifies the responsibilities for the Data Engineering and Data Ops teams, while ensuring data is accessible and can be shared seamlessly by the data users. To see how the integration works in action, watch the video here. If you’re ready to take JSON Data Extraction to new heights, contact us here for more information on how we can help you extract value from your JSON data.

Snowflake offers a powerful way to store and process semi-structured data like JSON, allowing data analysts to derive valuable insight from the data using standard SQL. However, for data analysts to access the value in this data, data engineers must first extract the nested JSON data into relational tables to process it. Managing a large amount of JSON data is a manual process that takes time away from data projects, introduces risks, and slows down Snowflake's use and adoption. In addition, with more stringent data privacy regulations arising from the need to ensure every approval for data access and sharing requests adheres to governance policies, the process can be daunting and costly. In this solution brief, we will explore how the combined solution provides the enterprise-class analytic engineering automation that allows your data team export data from nested JSON files into relational tables, simplifying the process and expediting access to your data. Click here to learn more

Bluemetrix leadership in data automation are complemented by Collibra's strengths in data governance, allowing joint customers to automatically enforce and execute the data policies in Collibra, achieving full regulatory compliance and auditing. Bluemetrix, a leading data and governance automation company for the healthcare and financial sectors, is thrilled to announce a new technology integration partnership with Collibra for 2022. This partnership will support global customers with their data governance initiatives, helping to enhance data security and accessibility needs and achieve full regulatory compliance for better governance. "Data governance is a complex process that requires significant resources and time. While most enterprises have their programs in place to govern how their data is accessed and processed, these data policies are generally data source specific, and not always enforced and implemented when data is used daily," said Liam English, CEO, Bluemetrix. "By partnering with Collibra, a pioneer in Data Intelligence, we are excited to provide organisations with an integrated solution that combines best-of-breed software to address these problems." The new collaboration will focus on integrating the Bluemetrix Data Manager (BDM) Full-Service Spark Based ETL solution with Collibra Data Intelligence Cloud, eliminating complexity across the entire data policy enforcement and management process. These solutions allow customers to automatically enforce and execute the Collibra Data Policies associated with each data source in the pipeline. Bluemetrix was also able to bring a robust and scalable approach by leveraging the Collibra-provided lineage feature around these policies, helping with compliance and auditing so that everyone can benefit from the responsible access to the data. With combined Bluemetrix and Collibra solutions, customers can operationalize Collibra Data Intelligence Cloud, guaranteeing that data is consistent, transformed and ready for consumption by the analytics and data science team. "The new integration will deliver an optimal engine that eliminates all potential roadblocks on the path to getting clean, secure and governed data ready," said Liam English. "We are looking forward to leveraging Collibra customers' hard work in data governance creation and achieving faster time to value on their data management investment in Bluemetrix." About Bluemetrix Bluemetrix is a leading provider of cloud data solutions, providing innovative solutions in Finance and Healthcare sectors. Our mission is to help enterprises get more value from their data, combining business strategy, innovative approaches, and technology. Since 2009, we have delivered over 400 Big Data Implementations across all major enterprises in Europe in all industry sectors. We continue to work closely with our customers to overcome technology gaps or skill set shortages. For a look at our integration in action, visit the Collibra Marketplace or schedule a demo here.

As today's enterprises strive to integrate their data assets in a secure and compliant manner while delivering on their digital transformation journey, the combined capabilities of Collibra and Bluemetrix will help global customers eliminate complexity across the entire data policy enforcement and management process. Read our product brief to learn how the platform streamlines and scales the enforcement of data policies enabling data users to create, capture and maintain the data governance state of all data assets processed in the pipeline. Click here to learn more