Today’s supply chains demand more than historical insights–they need real-time intelligence. In this actionable session, discover how leading enterprises are unlocking the full potential of their SAP data by integrating it with Databricks and AI. See how CPG companies are transforming supply chain planning by combining SAP ERP data with external signals like weather and transportation data–enabling them to predict disruptions, optimize inventory, and make faster, smarter decisions. Powered by Databricks, this solution delivers true agility and resilience through a unified data architecture. Join us to learn how: You can eliminate SAP data silos and make them ML and AI-ready at scale External data sources amplify SAP use cases like forecasting and scenario planning AI-driven insights accelerate time-to-action across supply chain operations Whether you're just starting your data modernization journey or seeking ROI from SAP analytics, this session will show you what’s possible.

Managing customer interactions across multiple disconnected platforms creates inefficiencies and delays in resolving support tickets. At WHOOP, support agents had to manually navigate through siloed data across payments, ERP, and ticketing systems, slowing down response times and impacting customer satisfaction.In this session, Matt Luizzi (Director of Business Analytics, WHOOP) and Brendan Farley (Sales Engineer, Snowflake) will showcase how WHOOP:

Consolidated fragmented data from multiple systems into a unified customer support app. Enabled real-time access to customer history, allowing agents to quickly surface relevant insights. Eliminated the need for custom engineering by leveraging Sigma’s no-code interface to build interactive workflows. Accelerated ticket resolution by allowing support teams to take action directly within Sigma, reducing dependency on multiple SaaS tools. Improved forecasting and decision-making by implementing AI-powered analytics on top of Snowflake. Before Sigma, getting a full view of customer issues required navigating across multiple tools—now, WHOOP’s customer support team can access, analyze, and act on real-time data in a single interface. Join us for an inside look at how WHOOP and Snowflake partnered to build a modern customer support data app that enhances efficiency and customer experience.

➡️ Learn more about Data Apps: https://www.sigmacomputing.com/product/data-applications?utm_source=youtube&utm_medium=organic&utm_campaign=data_apps_conference&utm_content=pp_data_apps

➡️ Sign up for your free trial: https://www.sigmacomputing.com/go/free-trial?utm_source=youtube&utm_medium=video&utm_campaign=free_trial&utm_content=free_trial

sigma #sigmacomputing #dataanalytics #dataanalysis #businessintelligence #cloudcomputing #clouddata #datacloud #datastructures #datadriven #datadrivendecisionmaking #datadriveninsights #businessdecisions #datadrivendecisions #embeddedanalytics #cloudcomputing #SigmaAI #AI #AIdataanalytics #AIdataanalysis #GPT #dataprivacy #python #dataintelligence #moderndataarchitecture

Integrating data from Oracle ERP to Google BigQuery? Join this session and discover how to enable seamless data integration, creating a robust data and integration fabric on Google Cloud. This capability enhances data accessibility and analytics, empowering informed business decisions. We also developed an abstraction layer to streamline integrations, fostering synergy across third-party platforms, accelerating time-to-value, and supporting a scalable, data-driven enterprise.

This Session is hosted by a Google Cloud Next Sponsor.
Visit your registration profile at g.co/cloudnext to opt out of sharing your contact information with the sponsor hosting this session.

This talk will demonstrate how the SAP user community can use Looker/Explore Assistant Chatbot to explore data insights into SAP ERP data stored on Google Cloud's BigQuery using natural language prompts. We will discuss the challenge of accessing and analyzing SAP data - ETL, Complex Data Model, introduction to Generative AI and Large Language Models (LLMs), and Looker Explore Assistant and Chatbot This presentation will illustrate how SAP users can leverage Looker and Explore Assistant Chatbot to gain insights into their SAP ERP data residing on Google Cloud's BigQuery, using natural language prompts. We will address common challenges in accessing and analyzing SAP data, such as ETL processes and complex data models. Additionally, we will provide an introduction to Generative AI and Large Language Models (LLMs), as well as an overview of Looker Explore Assistant and Chatbot's capabilities.

🌟 Session Overview 🌟

Session Name: AI Agents in Action: Transforming Enterprise Processes Speaker: Karyna Mihalevich Session Description: Drawing from Karyna's experience in SAP environments and intelligent automation, she will demonstrate how AI agents are transforming the landscape of business operations. The session will begin with an overview of AI agent applications across various business functions, followed by a live demonstration of an AI agent in action.

After the demo, Karyna will share additional use cases, providing attendees with insights into how AI agents are being used across different industries and departments. She will also outline practical steps for implementing AI agents in SAP and other ERP systems.

Attendees will leave this session with:

A clear understanding of the role of AI agents in modern enterprise automation Practical strategies for implementing AI agents in their own organizations

🚀 About Big Data and RPA 2024 🚀

Unlock the future of innovation and automation at Big Data & RPA Conference Europe 2024! 🌟 This unique event brings together the brightest minds in big data, machine learning, AI, and robotic process automation to explore cutting-edge solutions and trends shaping the tech landscape. Perfect for data engineers, analysts, RPA developers, and business leaders, the conference offers dual insights into the power of data-driven strategies and intelligent automation. 🚀 Gain practical knowledge on topics like hyperautomation, AI integration, advanced analytics, and workflow optimization while networking with global experts. Don’t miss this exclusive opportunity to expand your expertise and revolutionize your processes—all from the comfort of your home! 📊🤖✨

📅 Yearly Conferences: Curious about the evolution of QA? Check out our archive of past Big Data & RPA sessions. Watch the strategies and technologies evolve in our videos! 🚀 🔗 Find Other Years' Videos: 2023 Big Data Conference Europe https://www.youtube.com/playlist?list=PLqYhGsQ9iSEpb_oyAsg67PhpbrkCC59_g 2022 Big Data Conference Europe Online https://www.youtube.com/playlist?list=PLqYhGsQ9iSEryAOjmvdiaXTfjCg5j3HhT 2021 Big Data Conference Europe Online https://www.youtube.com/playlist?list=PLqYhGsQ9iSEqHwbQoWEXEJALFLKVDRXiP

💡 Stay Connected & Updated 💡

Don’t miss out on any updates or upcoming event information from Big Data & RPA Conference Europe. Follow us on our social media channels and visit our website to stay in the loop!

🌐 Website: https://bigdataconference.eu/, https://rpaconference.eu/ 👤 Facebook: https://www.facebook.com/bigdataconf, https://www.facebook.com/rpaeurope/ 🐦 Twitter: @BigDataConfEU, @europe_rpa 🔗 LinkedIn: https://www.linkedin.com/company/73234449/admin/dashboard/, https://www.linkedin.com/company/75464753/admin/dashboard/ 🎥 YouTube: http://www.youtube.com/@DATAMINERLT

Unlock the full potential of Microsoft Business Central by integrating it with the Power Platform through this practical and hands-on guide. With step-by-step tutorials, you'll learn how to combine the capabilities of tools like Power Apps, Power Automate, and Dataverse to build scalable and efficient business solutions. By the end of the book, you'll be equipped to streamline business processes and add significant value. What this Book will help me do Effectively deploy Power Platform functionalities for Microsoft Business Central projects. Seamlessly connect Business Central with cloud and on-premises services. Leverage Dataverse and virtual tables to enhance data modeling and accessibility. Build custom applications using Power Apps and automate workflows with Power Automate. Generate advanced visual reports with Power BI directly integrated with Business Central. Author(s) Kim Congleton and Shawn Sissenwein are industry professionals with extensive experience in ERP systems and Microsoft technologies. With a deep knowledge of Business Central and the Power Platform, they bring practical insights into maximizing business value through technological advancements. Their teaching approach focuses on hands-on learning, real-world application, and empowering readers with actionable skills. Who is it for? This book is ideal for Business Central users, consultants, and solution architects aiming to enhance Business Central's capabilities through the Power Platform. If you're familiar with Business Central's basics and seek to optimize and extend its functionality without requiring extensive programming knowledge, then this guide is tailored for you.

Kickstart your journey towards becoming a sustainable and value-driven enterprise with "RISE with SAP" as your guide. This book explains how to optimize your business processes and implement S/4HANA effectively using RISE with SAP, preparing decision-makers and architects with actionable insights and strategic guidance. What this Book will help me do Understand the challenges organizations face when adopting market trends and how to address them effectively. Learn to build a robust business case for transitioning to SAP S/4HANA using RISE with SAP as the foundational framework. Gain insights into process discovery, data migration, and the best practices for the fit-to-standard approach. Develop skills to design optimized enterprise landscapes effectively on the RISE with SAP platform. Master strategies to leverage SAP tools, services, and cloud ecosystems for industry-specific transformation. Author(s) Adil Zafar, Dharma Alturi, Sanket Taur, and Mihir R. Gor bring together years of combined expertise in enterprise architecture and SAP ecosystems. They leverage their hands-on experience to provide readers with practical advice and cutting-edge insights. Their collaborative work aims to demystify complexities and guide professionals toward sustainable practices. Who is it for? This book is ideal for CXOs, enterprise architects, and solution architects operating in SAP ecosystems who seek practical guidance for transitioning to SAP S/4HANA via RISE with SAP. It caters to readers who wish to build business cases effectively and ensure sustainable and optimized implementation. Prior experience with SAP or ERP systems will enhance the learning experience.

Upgrade your knowledge to learn S/4HANA, the latest version of the SAP ERP system, with its built-in intelligent technologies, including AI, machine learning, and advanced analytics. Since the first edition of this book published as SAP ERP Financial and Controlling: Configuration and Use Management, the perspective has changed significantly as S/4HANA now comes with new features, such as FIORI (new GUI), which focuses on flexible app style development and interactivity with mobile phones. It also has a universal journal, which helps in data integration in a single location, such as centralized processing, and is faster than ECC S/3. It merges FI & CO efficiently, which enables document posting in the Controlling area setup. General Ledger Accounts (FI) and Cost Element (CO) are mapped together in a way that cost elements (both primary and secondary) are part of G/L accounts. And a mandatory setup of customer-vendor integration with business partners is included vs the earlier ECC creation with separate vendor master and customer master.This updated edition presents new features in SAP S/4HANA, with in-depth coverage of the FI syllabus in SAP S/4HANA. A practical and hands-on approach includes scenarios with real-life examples and practical illustrations. There is no unnecessary jargon in this configuration and end-user manual. What You Will Learn Configure SAP FI as a pro in S/4 Master core aspects of Financial Accounting and Controlling Integrate SAP Financial with other SAP modules Gain a thorough hands-on experience with IMG (Implementation Guide) Understand and explain the functionalities of SAP FI Who This Book Is For FI consultants, trainers, developers, accountants, and SAP FI support organizations will find the book an excellent reference guide. Beginners without prior FI configuration experience will find the step-by-step illustrations to be practical and great hands-on experience.

In retail, the right quantity at the right time is crucial for success. In this session we share how a demand forecasting solution helped some of our retailers to improve efficiencies and sharpen fresh product production and delivery planning.

With the setup in place we train hundreds of models in parallel, training on various levels including store level, product level and the combination of the two. By leveraging the distributed computation of Spark, we can do all of this in a scalable and fast way. Powered by Delta Lake, feature store and MLFlow this session clarifies how we built a highly reliable ML factory.

We show how this setup runs at various retailers and feeds accurate demand forecasts back to the ERP system, supporting the clients in their production planning and delivery. Through this session we want to inspire retailers & conference attendants to use data & AI to not only gain efficiency but also decrease food waste.

Connect with us: Website: https://databricks.com Facebook: https://www.facebook.com/databricksinc Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/data... Instagram: https://www.instagram.com/databricksinc/

SAP Intelligent RPA for Developers dives deep into the realm of robotic process automation using SAP Intelligent RPA. It provides a comprehensive guide to leveraging RPA for automating repetitive business processes, ensuring a seamless integrated environment for SAP and non-SAP systems. By the end, you'll be equipped to craft, manage, and optimize automated workflows. What this Book will help me do Master the fundamentals of SAP Intelligent RPA and its architecture. Develop and deploy automation bots to streamline business processes. Utilize low-code and pro-code methodologies effectively in project designs. Debug and troubleshoot RPA solutions to ensure operational efficiency. Understand and plan the migration from SAP Intelligent RPA to SAP Process Automation. Author(s) None Madhuvarshi and None Ganugula are experts in SAP Intelligent RPA with years of experience in ERP systems integration and process automation. Together, they offer a practical and comprehensive approach to mastering and implementing SAP RPA solutions effectively. Who is it for? This book is perfect for developers and business analysts eager to explore SAP Intelligent RPA. It caters to those with a basic knowledge of JavaScript who aspire to leverage RPA for automating monotonous workflows. If you're looking to dive into SAP's automation framework and understand its practical applications, this book is a great fit for you.

This book helps SAP architects and SAP Basis administrators deploy and operate SAP S/4HANA systems on the most common public cloud platforms. Market-leading cloud offerings are covered, including Amazon Web Services, Microsoft Azure, and Google Cloud. You will gain an end-to-end understanding of the initial implementation of SAP S/4HANA systems on those platforms. You will learn how to move away from the big monolithic SAP ERP systems and arrive at an environment with a central SAP S/4HANA system as the digital core surrounded by cloud-native services. The book begins by introducing the core concepts of Hyperscaler cloud platforms that are relevant to SAP. You will learn about the architecture of SAP S/4HANA systems on public cloud platforms, with specific content provided for each of the major platforms. The book simplifies the deployment of SAP S/4HANA systems in public clouds by providing step-by-step instructions and helping you deal with thecomplexity of such a deployment. Content in the book is based on best practices, industry lessons learned, and architectural blueprints, helping you develop deep insights into the operations of SAP S/4HANA systems on public cloud platforms. Reading this book enables you to build and operate your own SAP S/4HANA system in the public cloud with a minimum of effort. What You Will Learn Choose the right Hyperscaler platform for your future SAP S/4HANA workloads Start deploying your first SAP S/4HANA system in the public cloud Avoid typical pitfalls during your implementation Apply and leverage cloud-native services for your SAP S/4HANA system Save costs by choosing the right architecture and build a robust architecture for your most critical SAP systems Meet your business’ criteria for availability and performance by having the right sizing in place Identify further use cases whenoperating SAP S/4HANA in the public cloud Who This Book Is For SAP architects looking for an answer on how to move SAP S/4HANA systems from on-premises into the cloud; those planning to deploy to one of the three major platforms from Amazon Web Services, Microsoft Azure, and Google Cloud Platform; and SAP Basis administrators seeking a detailed and realistic description of how to get started on a migration to the cloud and how to drive that cloud implementation to completion

Succeed in your conversion to SAP S/4HANA. This book will help you understand the core aspects and implement a conversion project. You will start with an overview of the SAP S/4HANA conversion tools: Readiness Check, Simplification Item Check report, Maintenance Planner, Custom Code Analysis, SUM (Software Update Manager), and more. You will understand the preparation activities for SAP FI (Finance), SAP CO (Controlling), SAP AA (Asset Accounting), Material Ledger, and COPA (Controlling–Profitability Analysis). And you will find the SAP CVI (Customer/Vendor Integration) steps that can help consultants understand the mandatory activities to be completed as a part of preparation on the SAP ECC (ERP Central Component) system. You will learn the preparation activities for conversion of accounting to SAP S/4HANA, and migration activities: customizing, asset accounting, controlling, and house bank accounts. You will gain knowledge on data migration activities such as the migration of cost elements, technical check of transactional data, material ledger migration enrichment of data, migration of line items, balances, and general ledger allocations to journal entry tables. After reading this book, you will know how to use the Migration Cockpit for data migration and post-conversion activities to successfully execute and implement an SAP S/4 HANA conversion. What You Will Learn Choose an ideal path and planning tools for SAP S/4HANA Start with the preparation step: General Ledger Accounting, Asset Accounting, Controlling, Material Ledger, and so on Use Migration Cockpit for conversion preparation, migration, and post-migration activities Who This Book Is For SAP application consultants, finance consultants, and CVI consultants who need help with SAP S/4HANA conversion

This IBM® Redpaper Redbooks publication provides guidance about a backup and recovery solution for SAP High-performance Analytic Appliance (HANA) running on IBM Power Systems. This publication provides case studies and how-to procedures that show backup and recovery scenarios. This publication provides information about how to protect data in an SAP HANA environment by using IBM Spectrum® Protect and IBM Spectrum Copy Data Manager. This publication focuses on the data protection solution, which is described through several scenarios. The information in this publication is distributed on an as-is basis without any warranty that is either expressed or implied. Support assistance for the use of this material is limited to situations where IBM Spectrum Scale or IBM Spectrum Protect are supported and entitled, and where the issues are specific to a blueprint implementation. The goal of the publication is to describe the best aspects and options for backup, snapshots, and restore of SAP HANA Multitenant Database Container (MDC) single and multi-tenant installations on IBM Power Systems by using theoretical knowledge, hands-on exercises, and documenting the findings through sample scenarios. This document provides resources about the following processes: Describing how to determine the best option, including SAP Landscape aspects to back up, snapshot, and restore of SAP HANA MDC single and multi-tenant installations based on IBM Spectrum Computing Suite, Red Hat Linux Relax and Recover (ReAR), and other products. Documenting key aspects, such as recovery time objective (RTO) and recovery point objective (RPO), backup impact (load, duration, scheduling), quantitative savings (for example, data deduplication), integration and catalog currency, and tips and tricks that are not covered in the product documentation. Using IBM Cloud® Object Storage and documenting how to use IBM Spectrum Protect to back up to the cloud. SAP HANA 2.0 SPS 05 has this feature that is built in natively. IBM Spectrum Protect for Enterprise Resource Planning (ERP) has this feature too. Documenting Linux ReaR to cover operating system (OS) backup because ReAR is used by most backup products, such as IBM Spectrum Protect and Symantec Endpoint Protection (SEP) to back up OSs. This publication targets technical readers including IT specialists, systems architects, brand specialists, sales teams, and anyone looking for a guide about how to implement the best options for SAP HANA backup and recovery on IBM Power Systems. Moreover, this publication provides documentation to transfer the how-to-skills to the technical teams and solution guidance to the sales team. This publication complements the documentation that is available at IBM Knowledge Center, and it aligns with the educational materials that are provided by IBM Garage™ for Systems Technical Education and Training.

This IBM® Redpaper publication documents how to containerize and deploy SAP software into Red Hat OpenShift 4 Kubernetes clusters on IBM Power Systems by using predefined Red Hat Ansible scripts, different configurations, and theoretical knowledge, and it documents the findings through sample scenarios. This paper documents the following topics: Running SAP S/4HANA, SAP HANA, and SAP NetWeaver on-premises software in containers that are deployed in Red Hat OpenShift 4 on IBM Power Systems hardware. Existing SAP systems running on IBM Power Systems can be repackaged at customer sites into containers that use predefined Red Hat Ansible scripts. These containers can be deployed multiple times into Red Hat OpenShift 4 Kubernetes clusters on IBM Power Systems. The target audiences for this paper are Chief Information Officers (CIOs) that are interested in containerized solutions of SAP Enterprise Resource Planning (ERP) systems, developers that need containerized environments, and system administrators that provide and manage the infrastructure with underpinning automation. This paper complements the documentation that is available at IBM Knowledge Center, and it aligns with the educational materials that are provided by IBM Garage™ for Systems Education.

Learn how to fuse today's data science tools and techniques with your SAP enterprise resource planning (ERP) system. With this practical guide, SAP veterans Greg Foss and Paul Modderman demonstrate how to use several data analysis tools to solve interesting problems with your SAP data. Data engineers and scientists will explore ways to add SAP data to their analysis processes, while SAP business analysts will learn practical methods for answering questions about the business. By focusing on grounded explanations of both SAP processes and data science tools, this book gives data scientists and business analysts powerful methods for discovering deep data truths. You'll explore: Examples of how data analysis can help you solve several SAP challenges Natural language processing for unlocking the secrets in text Data science techniques for data clustering and segmentation Methods for detecting anomalies in your SAP data Data visualization techniques for making your data come to life

Why is Data Quality still an issue after all these years? To get an answer to the prevalent question, Wayne Eckerson and Jason Beard engage in a dynamic exchange of questions which lead us to the root cause of data quality and data governance problems. Using examples from his past projects, Jason shows the value of business process mapping and how it exposes the hidden problems which go undetected under the standard IT lens.

In his most recent role as Vice President of Process & Data Management at Wiley, a book publisher, he was responsible for master data setup and governance, process optimization, business continuity planning, and change management for new and emerging business models. Jason has led business intelligence, data governance, master data management, Process Improvement, Business Transformation, and ERP projects in a variety of industries, including Scientific and Trade publishing, Educational Technology, Consumer Goods, Banking, Investments, and Insurance.

Summary Building internal expertise around big data in a large organization is a major competitive advantage. However, it can be a difficult process due to compliance needs and the need to scale globally on day one. In this episode Jesper Søgaard and Keld Antonsen share the story of starting and growing the big data group at LEGO. They discuss the challenges of being at global scale from the start, hiring and training talented engineers, prototyping and deploying new systems in the cloud, and what they have learned in the process. This is a useful conversation for engineers, managers, and leadership who are interested in building enterprise big data systems.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch. To help other people find the show please leave a review on iTunes, or Google Play Music, tell your friends and co-workers, and share it on social media. Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Your host is Tobias Macey and today I’m interviewing Keld Antonsen and Jesper Soegaard about the data infrastructure and analytics that powers LEGO

Interview

Introduction How did you get involved in the area of data management? My understanding is that the big data group at LEGO is a fairly recent development. Can you share the story of how it got started?

What kinds of data practices were in place prior to starting a dedicated group for managing the organization’s data? What was the transition process like, migrating data silos into a uniformly managed platform?

What are the biggest data challenges that you face at LEGO? What are some of the most critical sources and types of data that you are managing? What are the main components of the data infrastructure that you have built to support the organizations analytical needs?

What are some of the technologies that you have found to be most useful? Which have been the most problematic?

What does the team structure look like for the data services at LEGO?

Does that reflect in the types/numbers of systems that you support?

What types of testing, monitoring, and metrics do you use to ensure the health of the systems you support? What have been some of the most interesting, challenging, or useful lessons that you have learned while building and maintaining the data platforms at LEGO? How have the data systems at Lego evolved over recent years as new technologies and techniques have been developed? How does the global nature of the LEGO business influence the design strategies and technology choices for your platform? What are you most excited for in the coming year?

Contact Info

Jesper

LinkedIn

Keld

LinkedIn

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

LEGO Group ERP (Enterprise Resource Planning) Predictive Analytics Prescriptive Analytics Hadoop Center Of Excellence Continuous Integration Spark

Podcast Episode

Apache NiFi

Podcast Episode

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Support Data Engineering Podcast

Power BI Data Analysis and Visualization provides a roadmap to vendor choices and highlights why Microsoft’s Power BI is a very viable, cost effective option for data visualization. The book covers the fundamentals and most commonly used features of Power BI, but also includes an in-depth discussion of advanced Power BI features such as natural language queries; embedding Power BI dashboards; and live streaming data. It discusses real solutions to extract data from the ERP application, Microsoft Dynamics CRM, and also offers ways to host the Power BI Dashboard as an Azure application, extracting data from popular data sources like Microsoft SQL Server and open-source PostgreSQL. Authored by Microsoft experts, this book uses real-world coding samples and screenshots to spotlight how to create reports, embed them in a webpage, view them across multiple platforms, and more. Business owners, IT professionals, data scientists, and analysts will benefit from this thorough presentation of Power BI and its functions.

Summary

With the proliferation of data sources to give a more comprehensive view of the information critical to your business it is even more important to have a canonical view of the entities that you care about. Is customer number 342 in your ERP the same as Bob Smith on Twitter? Using master data management to build a data catalog helps you answer these questions reliably and simplify the process of building your business intelligence reports. In this episode the head of product at Tamr, Mark Marinelli, discusses the challenges of building a master data set, why you should have one, and some of the techniques that modern platforms and systems provide for maintaining it.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $20 credit and launch a new server in under a minute. You work hard to make sure that your data is reliable and accurate, but can you say the same about the deployment of your machine learning models? The Skafos platform from Metis Machine was built to give your data scientists the end-to-end support that they need throughout the machine learning lifecycle. Skafos maximizes interoperability with your existing tools and platforms, and offers real-time insights and the ability to be up and running with cloud-based production scale infrastructure instantaneously. Request a demo at dataengineeringpodcast.com/metis-machine to learn more about how Metis Machine is operationalizing data science. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch. Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Your host is Tobias Macey and today I’m interviewing Mark Marinelli about data mastering for modern platforms

Interview

Introduction How did you get involved in the area of data management? Can you start by establishing a definition of data mastering that we can work from?

How does the master data set get used within the overall analytical and processing systems of an organization?

What is the traditional workflow for creating a master data set?

What has changed in the current landscape of businesses and technology platforms that makes that approach impractical? What are the steps that an organization can take to evolve toward an agile approach to data mastering?

At what scale of company or project does it makes sense to start building a master data set? What are the limitations of using ML/AI to merge data sets? What are the limitations of a golden master data set in practice?

Are there particular formats of data or types of entities that pose a greater challenge when creating a canonical format for them? Are there specific problem domains that are more likely to benefit from a master data set?

Once a golden master has been established, how are changes to that information handled in practice? (e.g. versioning of the data) What storage mechanisms are typically used for managing a master data set?

Are there particular security, auditing, or access concerns that engineers should be considering when managing their golden master that goes beyond the rest of their data infrastructure? How do you manage latency issues when trying to reference the same entities from multiple disparate systems?

What have you found to be the most common stumbling blocks for a group that is implementing a master data platform?

What suggestions do you have to help prevent such a project from being derailed?

What resources do you recommend for someone looking to learn more about the theoretical and practical aspects of

Summary

The information about how data is acquired and processed is often as important as the data itself. For this reason metadata management systems are built to track the journey of your business data to aid in analysis, presentation, and compliance. These systems are frequently cumbersome and difficult to maintain, so Octopai was founded to alleviate that burden. In this episode Amnon Drori, CEO and co-founder of Octopai, discusses the business problems he witnessed that led him to starting the company, how their systems are able to provide valuable tools and insights, and the direction that their product will be taking in the future.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 200Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $20 credit and launch a new server in under a minute. For complete visibility into the health of your pipeline, including deployment tracking, and powerful alerting driven by machine-learning, DataDog has got you covered. With their monitoring, metrics, and log collection agent, including extensive integrations and distributed tracing, you’ll have everything you need to find and fix performance bottlenecks in no time. Go to dataengineeringpodcast.com/datadog today to start your free 14 day trial and get a sweet new T-Shirt. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. Your host is Tobias Macey and today I’m interviewing Amnon Drori about OctopAI and the benefits of metadata management

Interview

Introduction How did you get involved in the area of data management? What is OctopAI and what was your motivation for founding it? What are some of the types of information that you classify and collect as metadata? Can you talk through the architecture of your platform? What are some of the challenges that are typically faced by metadata management systems? What is involved in deploying your metadata collection agents? Once the metadata has been collected what are some of the ways in which it can be used? What mechanisms do you use to ensure that customer data is segregated?

How do you identify and handle sensitive information during the collection step?

What are some of the most challenging aspects of your technical and business platforms that you have faced? What are some of the plans that you have for OctopAI going forward?

Contact Info

Amnon

LinkedIn @octopai_amnon on Twitter

OctopAI

@OctopaiBI on Twitter Website

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

OctopAI Metadata Metadata Management Data Integrity CRM (Customer Relationship Management) ERP (Enterprise Resource Planning) Business Intelligence ETL (Extract, Transform, Load) Informatica SAP Data Governance SSIS (SQL Server Integration Services) Vertica Airflow Luigi Oozie GDPR (General Data Privacy Regulation) Root Cause Analysis

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support Data Engineering Podcast