Summary

If your business metrics looked weird tomorrow, would you know about it first? Anomaly detection is focused on identifying those outliers for you, so that you are the first to know when a business critical dashboard isn't right. Unfortunately, it can often be complex or expensive to incorporate anomaly detection into your data platform. Andrew Maguire got tired of solving that problem for each of the different roles he has ended up in, so he created the open source Anomstack project. In this episode he shares what it is, how it works, and how you can start using it today to get notified when the critical metrics in your business aren't quite right.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management You shouldn't have to throw away the database to build with fast-changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. Whether it’s real-time dashboarding and analytics, personalization and segmentation or automation and alerting, Materialize gives you the ability to work with fresh, correct, and scalable results — all in a familiar SQL interface. Go to dataengineeringpodcast.com/materialize today to get 2 weeks free! Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack Data projects are notoriously complex. With multiple stakeholders to manage across varying backgrounds and toolchains even simple reports can become unwieldy to maintain. Miro is your single pane of glass where everyone can discover, track, and collaborate on your organization's data. I especially like the ability to combine your technical diagrams with data documentation and dependency mapping, allowing your data engineers and data consumers to communicate seamlessly about your projects. Find simplicity in your most complex projects with Miro. Your first three Miro boards are free when you sign up today at dataengineeringpodcast.com/miro. That’s three free boards at dataengineeringpodcast.com/miro. Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics. Trusted by teams of all sizes, including Comcast and Doordash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Want to see Starburst in action? Go to dataengineeringpodcast.com/starburst and get $500 in credits to try Starburst Galaxy today, the easiest and fastest way to get started using Trino. Your host is Tobias Macey and today I'm interviewing Andrew Maguire about his work on the Anomstack project and how you can use it to run your own anomaly detection for your metrics

Interview

Introduction How did you get involved in the area of data management? Can you describe what Anomstack is and the story behind it?

What are your goals for this project? What other tools/products might teams be evaluating while they consider Anom

Today I’m joined by Nick Zervoudis, Data Product Manager at CKDelta. As we dive into his career and background, Nick shares insights into his approach when it comes to developing both internal and external data products. Nick explains why he feels that a software engineering approach is the best way to develop a product that could have multiple applications, as well as the unique way his team is structured to best handle the needs of both internal and external customers. He also talks about the UX design course he took, how that affected his data product work and research with users, and his thoughts on dashboard design. We discuss common themes he’s observed when data product teams get it wrong, and how he manages feelings of imposter syndrome in his career as a DPM. 

Highlights/ Skip to:

I introduce Nick, who is a Data Product Manager at CKDelta (00:35) Nick’s mindset around data products and how his early career in consulting shaped his approach (01:30) How Nick defines a data product and why he focuses more on the process rather than the end product (03:59) The types of data products that Nick has helped design and his work on both internal and external projects at CKDelta (07:57) The similarities and differences of working with internal versus external stakeholders (12:37) Nick dives into the details of the data products he has built and how they feed into complex use cases (14:21) The role that Nick plays in the Delta Power SaaS application and how the CKDelta team is structured around that product (17:14) Where Nick sees data products going wrong and how he’s found value in filling those gaps (23:30) Nick’s view on how a digital-first mindset affects the scalability of data products (26:15) Why Nick is often heavily involved in the design element of data product development and the course he took that helped shape his design work (28:55) The imposter syndrome that Nick has experienced when implementing this new strategy to data product design (36:51) Why Nick feels that figuring things out yourself is an inherent part of the DPM role (44:53) Nick shares the origins and information on the London Data Product Management meetup (46:08)

Quotes from Today’s Episode “What I’m always trying to do is see, how can we best balance the customer’s need to get exactly the data point or insight that they’re after to the business need. ... There’s that constant tug of war between customization and standardization that I have the joy of adjudicating. I think it’s quite fun.” — Nick Zervoudis (16:40)

“I’ve had times where I was hired, told, 'You’re going to be the product manager for this data product that we have,' as if it’s already, to some extent built and maybe the challenge is scaling it or bringing it to more customers or improving it, and then within a couple of weeks of starting to peek under the hood, realizing that this thing that is being branded a product is actually a bunch of projects hiding under a trench coat.” — Nick Zervoudis (24:04)

“If I just speak to five users because they’re the users, they’ll give me the insight I need. […] Even when you have a massive product with a huge user base, people face the same issues.” — Nick Zervoudis (33:49)

“For me, it’s more about making sure that you’re bringing that more software engineering way of building things, but also, before you do that, knowing that your users' needs are going to [be varied]. So, it’s a combination of both, are we building the right thing—in other words, a product that’s flexible enough to meet the different needs of different users—but also, are we building it in the right way?” – Nick Zervoudis (27:51)

“It’s not to say I’m the only person thinking about [UX design], but very often, I’m the one driving it.” – Nick Zervoudis (30:55)

“You’re never going to be as good at the thing your colleague does because their job almost certainly is to be a specialist: they’re an architect, they’re a designer, they’re a developer, they’re a salesperson, whereas your job [as a DPM] is to just understand it enough that you can then pass information across other people.” – Nick Zervoudis (41:12)

“Every time I feel like an imposter, good. I need to embrace that, because I need to be working with people that understand something better than me. If I’m not, then maybe something’s gone wrong there. That’s how I’ve actually embraced impostor syndrome.” – Nick Zervoudis (41:35)

Links CKDelta: https://www.ckdelta.ie LinkedIn: https://www.linkedin.com/in/nzervoudis/

Join host Avery Smith, with Excel expert Matt Brattin about mastering data analytics with Excel in this episode of the Data Career Podcast.

From customizing shortcuts to leveraging advanced functions, Matt shares tips and tricks to take your Excel skills to the next level.

Get insights from an industry pro and boost your chances of landing your dream data job.

Tune in now!

Connect with Matt Brattin:

🤝 Connect on Linkedin

▶️ Subscribe to Youtube Channel

🎒 Learn About the Excel 4 Academy

🤝 Ace your data analyst interview with the interview simulator⁠

📩 Get my weekly email with helpful data career tips

📊 Come to my next free “How to Land Your First Data Job” training

🏫 Check out my 10-week data analytics bootcamp

Timestamps:

(07:53) - 5 Concepts of Excel you should know

(20:35) - Top Excel Shortcuts

(26:54) - Can you make a dashboard in Excel?

(30:20) - Excel is like Thor’s hammer

Connect with Avery:

📺 Subscribe on YouTube

🎙Listen to My Podcast

👔 Connect with me on LinkedIn

📸 Instagram

🎵 TikTok

Mentioned in this episode: Join the last cohort of 2025! The LAST cohort of The Data Analytics Accelerator for 2025 kicks off on Monday, December 8th and enrollment is officially open!

To celebrate the end of the year, we’re running a special End-of-Year Sale, where you’ll get: ✅ A discount on your enrollment 🎁 6 bonus gifts, including job listings, interview prep, AI tools + more

If your goal is to land a data job in 2026, this is your chance to get ahead of the competition and start strong.

👉 Join the December Cohort & Claim Your Bonuses: https://DataCareerJumpstart.com/daa https://www.datacareerjumpstart.com/daa

Advances in LLM technology and the semantic layer have made AI-powered data analytics easy. Nothing special. Par for the course.

Wait, you haven't seen yet? Don't worry, we'll show you. But I gotta warn you, it's pretty boring stuff. You just ask for whatever data you want, and then the computer gets it for you. Yes, seriously!

So what is still impressive in October 2023? Well, good old fashioned human ingenuity and improvisational skill. Computers may have gotten pretty good at crunching the numbers, but you still can't replace good old fashioned meat and bone when it comes time to present those numbers to the big boss.

In this session, Izzy Miller and Matt David put that to the test with a live game of Dashboard Karaoke— with an AI twist. Random audience members take turns using Hex's Magic AI to generate entire analytical reports on datasets of their choosing. Then, our handpicked data practitioners have to improvise a compelling presentation of the results on the spot, explaining the intricacies and trends of data they've never seen. And of course, there will be plenty of time for audience questions ;)

Speakers: Izzy Miller, Developer Advocate, Hex; Matt David, Product Marketing Lead, Hex

Register for Coalesce at https://coalesce.getdbt.com

From the dawn of humanity, decisions, both big and small, have shaped our trajectory. Decisions have built civilizations, forged alliances, and even charted the course of our very evolution. And now, as data & AI become more widespread, the potential upside for better decision making is massive. Yet, like any technology, the true value of data & AI is realized by how we wield it.  We're often drawn to the allure of the latest tools and techniques, but it's crucial to remember that these tools are only as effective as the decisions we make with them. ChatGPT is only as good as the prompt you decide to feed it and what you decide to do with the output. A dashboard is only as good as the decisions that it influences. Even a data science team is only as effective as the value they deliver to the organization.  So in this vast landscape of data and AI, how can we master the art of better decision making? How can we bridge data & AI with better decision intelligence? ​​Cassie Kozyrkov founded the field of Decision Intelligence at Google where, until recently, she served as Chief Decision Scientist, advising leadership on decision process, AI strategy, and building data-driven organizations. Upon leaving Google, Cassie started her own company of which she is the CEO, Data Scientific. In almost 10 years at the company, Cassie personally trained over 20,000 Googlers in data-driven decision-making and AI and has helped over 500 projects implement decision intelligence best practices. Cassie also previously served in Google's Office of the CTO as Chief Data Scientist, and the rest of her 20 years of experience was split between consulting, data science, lecturing, and academia.  Cassie is a top keynote speaker and a beloved personality in the data leadership community, followed by over half a million tech professionals. If you've ever went on a reading spree about AI, statistics, or decision-making, chances are you've encountered her writing, which has reached millions of readers.  In the episode Cassie and Richie explore misconceptions around data science, stereotypes associated with being a data scientist, what the reality of working in data science is, advice for those starting their career in data science, and the challenges of being a data ‘jack-of-all-trades’.  Cassie also shares what decision-science and decision intelligence are, what questions to ask future employers in any data science interview, the importance of collaboration between decision-makers and domain experts, the differences between data science models and their real-world implementations, the pros and cons of generative AI in data science, and much more.  Links mentioned in the Show: Data scientist: The sexiest job of the 22nd centuryThe Netflix PrizeAI Products: Kitchen AnalogyType one, Two & Three Errors in StatisticsCourse: Data-Driven Decision Making for BusinessRadar: Data & AI Literacy...

Gain deep insight into real-time analytics, including the features of these systems and the problems they solve. With this practical book, data engineers at organizations that use event-processing systems such as Kafka, Google Pub/Sub, and AWS Kinesis will learn how to analyze data streams in real time. The faster you derive insights, the quicker you can spot changes in your business and act accordingly. Author Mark Needham from StarTree provides an overview of the real-time analytics space and an understanding of what goes into building real-time applications. The book's second part offers a series of hands-on tutorials that show you how to combine multiple software products to build real-time analytics applications for an imaginary pizza delivery service. You will: Learn common architectures for real-time analytics Discover how event processing differs from real-time analytics Ingest event data from Apache Kafka into Apache Pinot Combine event streams with OLTP data using Debezium and Kafka Streams Write real-time queries against event data stored in Apache Pinot Build a real-time dashboard and order tracking app Learn how Uber, Stripe, and Just Eat use real-time analytics

¿Has dicho erróneamente que vas a invertir en la “BOLSA” de RD? En este episodio nos acompaña Ivan Carvajal y Jaime Lama de la Bolsa de Valores, quiénes nos explican de manera sencilla lo que hace la Bolsa de Valores de RD y en qué nos aporta a nosotros como inversionistas. Como BONUS aprendemos sobre una herramienta que beneficiará mucho al Mercado de Valores: el Dashboard de las Acciones de César Iglesias. Definitivamente un episodio lleno de aprendizaje para los potenciales y actuales inversionistas del Mercado de Valores de RD. Síguenos en: Instagram: https://www.instagram.com/economicsdata/ Tiktok: https://www.tiktok.com/@economicsdata Youtube: https://youtube.com/@economicsdata257

Em um papo empolgante, mergulhamos no universo dos profissionais de dados e suas habilidades essenciais, com um foco especial no poderoso Power BI e demais ferramentas. Descubra, como as ferramentas de analytics, como o Power BI, estão moldando o futuro do campo de dados e análises.

Nste episódio do Data Hackers — a maior comunidade de AI e Data Science do Brasil-, conheçam as apaixonadas pela área de dados e principais referências no assunto: a Karine Lago — especialista em Business Intelligence, Power BI e Excel, premiada pela Microsoft mais de sete vezes e Escritora; e a Letícia Smirelli — Chief Product Officer (CPO), Power BI Specialist, Microsoft Data Analyst Associate e DataViz & Dashboard Design; ambas sócias na Nexos Educação.

Lembrando que você pode encontrar todos os podcasts da comunidade Data Hackers no Spotify, iTunes, Google Podcast, Castbox e muitas outras plataformas. Caso queira, você também pode ouvir o episódio aqui no post mesmo!

Link Medium: https://medium.com/data-hackers/power-bi-dashboards-e-a-carreira-de-analista-de-dados-data-hackers-podcast-72-829986f5f2a1

Falamos no episódio

Conheça nosso convidado:

Karine Lago — especialista em Business Intelligence, Escritora, Power BI e Excel, premiada pela Microsoft mais de sete vezes;  Letícia Smirelli — Chief Product Officer (CPO), Power BI Specialist, Microsoft Data Analyst Associate e DataViz & Dashboard Design.

Bancada Data Hackers:

Paulo Vasconcellos Gabriel Lages Monique Femme

Links de referências:

Tech and Cheers — Meetup ed. Data Connect (São Paulo): https://www.sympla.com.br/evento/tech-and-cheers-meetup-ed-data-connect/2110360 https://towardsdatascience.com/whats-the-difference-between-analytics-and-statistics-cd35d457e17

Tech and Cheers — ed. Mulher.ADA (Blumenau):https://www.sympla.com.br/evento/tech-and-cheers-meetup-ed-mulher-ada/2109236

World Economic Forum (The Future of Jobs Report 2023):https://www.weforum.org/reports/the-future-of-jobs-report-2023/ Canal Karine Lago (Youtube):https://www.youtube.com/@KarineLago Pagina Karine Lago: https://keepo.io/karinedolago/?fbclid=PAAaZ32JXyRtPv7wcHcfaxtKA5TOU9VRaCt_F_nb7zhAptO4AtthorxiHWCdg_aem_Ab53sgYj0AXg1wHrOP9-c_K7pwoMqX0psYWAvNMAanqh5pafTHBFb3bnshKB534J9AA Canal Leticia Smirelli (Youtube): https://www.youtube.com/@LeticiaSmirelli Pagina Leticia Smirelli: https://keepo.io/leticia/?fbclid=PAAabu7cvnFTkkFw1UiJrDMIXiMJ45Av6XKlCXIfWAUiRH2c4kiSZzo7FX6TY_aem_Ab7BHn25MaVK22HFw9zXNfsYv5k5Y5o9WLMGZeFB9wSSSAV3d7EDA0JuGjXWSqd_SEs

Summary

Data pipelines are the core of every data product, ML model, and business intelligence dashboard. If you're not careful you will end up spending all of your time on maintenance and fire-fighting. The folks at Rivery distilled the seven principles of modern data pipelines that will help you stay out of trouble and be productive with your data. In this episode Ariel Pohoryles explains what they are and how they work together to increase your chances of success.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold Your host is Tobias Macey and today I'm interviewing Ariel Pohoryles about the seven principles of modern data pipelines

Interview

Introduction How did you get involved in the area of data management? Can you start by defining what you mean by a "modern" data pipeline? At Rivery you published a white paper identifying seven principles of modern data pipelines:

Zero infrastructure management ELT-first mindset Speaks SQL and Python Dynamic multi-storage layers Reverse ETL & operational analytics Full transparency Faster time to value

What are the applications of data that you focused on while identifying these principles? How do the application of these principles influence the ability of organizations and their data teams to encourage and keep pace with the use of data in the business? What are the technical components of a pipeline infrastructure that are necessary to support a "modern" workflow? How do the technologies involved impact the organizational involvement with how data is applied throughout the business? When using managed services, what are the ways that the pricing model acts to encourage/discourage experimentation/exploration with data? What are the most interesting, innovative, or unexpected ways that you have seen these seven principles implemented/applied? What are the most interesting, unexpected, or challenging lessons that you have learned while working with customers to adapt to these principles? What are the cases where some/all of these principles are undesirable/impractical to implement? What are the opportunities for further advancement/sophistication in the ways that teams work with and gain value from data?

Contact Info

LinkedIn

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.init covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned somethi

As we venture into the future of data engineering, streaming and serverless technologies take center stage. In this fun, hands-on, in-depth and interactive session you can learn about the essence of future data engineering today.

We will tackle the challenge of processing streaming events continuously created by hundreds of sensors in the conference room from a serverless web app (bring your phone and be a part of the demo). The focus is on the system architecture, the involved products and the solution they provide. Which Databricks product, capability and settings will be most useful for our scenario? What does streaming really mean and why does it make our life easier? What are the exact benefits of serverless and how "serverless" is a particular solution?

Leveraging the power of the Databricks Lakehouse Platform, I will demonstrate how to create a streaming data pipeline with Delta Live Tables ingesting data from AWS Kinesis. Further, I’ll utilize advanced Databricks workflows triggers for efficient orchestration and real-time alerts feeding into a real-time dashboard. And since I don’t want you to leave with empty hands - I will use Delta Sharing to share the results of the demo we built with every participant in the room. Join me in this hands-on exploration of cutting-edge data engineering techniques and witness the future in action.

Talk by: Frank Munz

Connect with us: Website: https://databricks.com Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/databricks Instagram: https://www.instagram.com/databricksinc Facebook: https://www.facebook.com/databricksinc

Databricks Workflows provides unified orchestration for the Lakehouse. Since it was first announced last year, thousands of organizations have been leveraging Workflows for orchestrating lakehouse workloads such as ETL, BI dashboard refresh and ML model training.

In this session, the Workflows product team will cover and demo the latest features and capabilities of Databricks Workflows in the areas of workflow authoring, observability and more. This session will also include an outlook for future innovations you can expect to see in the coming months.

Talk by: Muhammad Bilal Aslam

Connect with us: Website: https://databricks.com Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/databricks Instagram: https://www.instagram.com/databricksinc Facebook: https://www.facebook.com/databricksinc

In this session, we will introduce Databricks Asset Bundles, provide a demonstration of how they work for a variety of data products, and how to fit them into an overall CICD strategy for the well-architected Lakehouse.

Data teams produce a variety of assets; datasets, reports and dashboards, ML models, and business applications. These assets depend upon code (notebooks, repos, queries, pipelines), infrastructure (clusters, SQL warehouses, serverless endpoints), and supporting services/resources like Unity Catalog, Databricks Workflows, and DBSQL dashboards. Today, each organization must figure out a deployment strategy for the variety of data products they build on Databricks as there is no consistent way to describe the infrastructure and services associated with project code.

Databricks Asset Bundles is a new capability on Databricks that standardizes and unifies the deployment strategy for all data products developed on the platform. It allows developers to describe the infrastructure and resources of their project through a YAML configuration file, regardless of whether they are producing a report, dashboard, online ML model, or Delta Live Tables pipeline. Behind the scenes, these configuration files use Terraform to manage resources in a Databricks workspace, but knowledge of Terraform is not required to use Databricks Asset Bundles.

Talk by: Rafi Kurlansik and Pieter Noordhuis

Connect with us: Website: https://databricks.com Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/databricks Instagram: https://www.instagram.com/databricksinc Facebook: https://www.facebook.com/databricksinc

System tests are executable DAGs for example and testing purposes. With a simple pytest command, you can run an entire DAG. From a provider point of view, they can be viewed as integration tests for all provider related operators and sensors. Running these system tests frequently and monitoring the results allow us to enforce stability amongst many other benefits. In this presentation we will explore how AWS built their system test environment, from the GitHub fork to the health dashboard that exists today…but more importantly, why you should do it as well!

ABOUT THE TALK: Dashboards sit at the end of a long chain of ever-changing data dependencies. And, it is a very visual process – it is hard to tell if a dashboard is correct without an end user looking at the rendered result. This all adds up to a development process that can be slow and error-prone.

“DataOps” is a new set of code-based patterns and practices that aim to address these challenges. In this talk, Dan Eisenberg does a deep dive on these approaches and demonstrate some ways to integrate DataOps into the BI development lifecycle at Glean.

ABOUT THE SPEAKER: Dan Eisenberg is the VP of Technology at Glean.io, a platform for data visualization and collaboration. Prior to Glean, he was a Senior Director of Engineering at Flatiron Health, where his teams designed and built systems for abstracting data from unstructured medical records at scale.

ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers.

Make sure to subscribe to our channel for the most up-to-date talks from technical professionals on data related topics including data infrastructure, data engineering, ML systems, analytics and AI from top startups and tech companies.

FOLLOW DATA COUNCIL: Twitter: https://twitter.com/DataCouncilAI LinkedIn: https://www.linkedin.com/company/datacouncil-ai/

Welcome to this step-by-step guide for Excel users, data analysts, and finance specialists. It is designed to take you through practical report and development scenarios, including both the approach and the technical challenges. This book will equip you with an understanding of the overall Power Platform use case for addressing common business challenges. While Power BI continues to be an excellent tool of choice in the BI space, Power Platform is the real game changer. Using an integrated architecture, a small team of citizen developers can build solutions for all kinds of business problems. For small businesses, Power Platform can be used to build bespoke CRM, Finance, and Warehouse management tools. For large businesses, it can be used to build an integration point for existing systems to simplify reporting, operation, and approval processes. The author has drawn on his15 years of hands-on analytics experience to help you pivot from the traditional Excel-based reporting environment. By using different business scenarios, this book provides you with clear reasons why a skill is important before you start to dive into the scenarios. You will use a fast prototyping approach to continue to build exciting reporting, automation, and application solutions and improve them while you acquire new skill sets. The book helps you get started quickly with Power BI. It covers data visualization, collaboration, and governance practices. You will learn about the most practical SQL challenges. And you will learn how to build applications in PowerApps and Power Automate. The book ends with an integrated solution framework that can be adapted to solve a wide range of complex business problems. What You Will Learn Develop reporting solutions and business applications Understand the Power Platform licensing and development environment Apply Data ETL and modeling in Power BI Use Data Storytelling and dashboard design to better visualize data Carry out data operations with SQL and SharePoint lists Develop useful applications using Power Apps Develop automated workflows using Power Automate Integrate solutions with Power BI, Power Apps, and Power Automate to build enterprise solutions Who This Book Is For Next-generation data specialists, including Excel-based users who want to learn Power BI and build internal apps; finance specialists who want to take a different approach to traditional accounting reports; and anyone who wants to enhance their skill set for the future job market.

Today I’m chatting with Samir Sharma, CEO of datazuum. Samir is passionate about developing data strategies that drive business outcomes, and shares valuable insights into how problem framing and research can be done effectively from both the data and business side. Samir also provides his definition of a data strategy, and why it can be complicated to uncover whose job it is to create one. Throughout the conversation, Samir and I uncover the value of including different perspectives when implementing a data strategy and discuss solutions to various communication barriers. Of course, dashboards and data products also popped up in this episode as well! 

Highlights/ Skip to:

How Samir defines a data strategy and whose job it is to create one (01:39) The challenges Samir sees when trying to uncover and understand a company’s existing data strategy (03:39) The problem with the problem statements that Samir commonly encounters (08:37) Samir unpacks the communication challenges that lead to negative business outcomes when developing data products (14:05) An example of how improving research and problem framing solved a problem for Samir’s first big client (24:33) How speaking in a language your users understand can open the door to more exciting and valuable projects (31:08)

Quotes from Today’s Episode “I don’t think business teams really care how you do it. If you can get an outcome—even if it’s quick and dirty. We’re not supposed to be doing these things for months on end. We’re supposed to be iterating quickly to start to show that result and add value and then building on top of that to show more value, more results.” — Samir Sharma (07:29)

“Language is so important for business teams and technical teams and data teams to actually be able to speak a common language which has common business constructs. Why are organizations trying to train 20,000 people on data literacy, when they’ve got a ten-person data team? Why not just teach the ten people in the data team business language?” — Samir Sharma (10:52)

“I will continuously talk about processes because there’s not enough done actually understanding processes and how data is an event that occurs when a process is kicked off. … If you don’t understand the process and how data is enabling that process, or how data is being generated and the trigger points, then you’re just building something without really understanding where I need to fit that product in or where I need to fit that workflow in.” – Samir Sharma (11:46)

“But I start with asking clear questions about if I built you this dashboard, what is the decision you’re going to make off the back of it? Nine times out of ten, that question isn’t asked, if I build you this widget on this dashboard, what decision or action are you going to make or take? And how is that going to be linked back to the map that strategic objective? And if you can ask that question, you can build with purpose.” – Samir Sharma (19:27)

“You show [users] a bit of value, you show them what they’ve been dying to have, you give them a little bit extra in that so they can really optimize their decisions, and suddenly, you’ve got both sides now speaking a language that is really based on business outcomes and results.” – Samir Sharma (32:38)

“If the people in that conversation are the developers on one side, the business team, and they’re starting to see a new narrative, even the developers will start to say, “Oh! Now, I know exactly why I’m doing this. Now, I know why I’m building it.” So, they’re also starting to learn about the business, about what impacts sales, and maybe how marketing then intertwines into that. It’s important that that is done, but not enough time has been taken on that approach.” – Samir Sharma (24:05)

The thing for me is, business teams don’t know what they don’t know, right? Most of the time, they’re asking a question. If I was on the data team and I’d already built a dashboard that would [answer that question], then I haven’t built it properly in the first instance. What I’ve done is I’ve built it for the beauty and the visualization instead of the what I would class is the ugliness and impact that I need.” – Samir Sharma (17:05)

Links datazuum: https://datazuum.com/ LinkedIn: https://www.linkedin.com/in/samirsharma1/

Send us a text How to incubate and start a company. Dwight Harris Jr, CEO of DataScalp, joins us to share the online platform that captures consumer experiences and uses consumer data to rank companies in a performance dashboard. 01:31 Dwight Harris Jr, problem solver02:38 DataScalp and airlines10:11 Airlines rebuttal12:41 GTM13:45 Why start this company15:59 Next steps20:14 A day in the life22:25 Leadership advice22:40 "From Good to Great"LinkedIn: linkedin.com/in/dwightharrisjr Website: https://datascalp.com/ Want to be featured as a guest on Making Data Simple? Reach out to us at [email protected] and tell us why you should be next. The Making Data Simple Podcast is hosted by Al Martin, WW VP Technical Sales, IBM, where we explore trending technologies, business innovation, and leadership ...while keeping it simple & fun. Want to be featured as a guest on Making Data Simple? Reach out to us at [email protected] and tell us why you should be next. The Making Data Simple Podcast is hosted by Al Martin, WW VP Technical Sales, IBM, where we explore trending technologies, business innovation, and leadership ... while keeping it simple & fun.

When working with data, it’s easy for us to think about it as a mechanistic process, where data comes in and products come out. But as we’ve explored throughout the show, succeeding in data, whether you’re a data leader looking to build a data culture, a data scientist ascending the ranks, or even a policy maker looking to have an impact with data, the human side is crucial. At the heart of the “human side” is empathy— whether it’s for your stakeholders if you’re a data scientist developing a dashboard for them, empathy for your workforce if you’re a data or learning leader, or empathy for the planet and your citizens if you’re a policy maker.  So how can we all practice better empathy? Specifically, can we all practice better data empathy? Luckily, empathy is a muscle that can be built. It’s not a “you have it, or you don’t” type of skill. So how can individuals and organizations utilize data empathy to improve how they work with data and the success rate of their projects?   Enter Phil Harvey, an Industrial Metaverse Architect in the Industrial Metaverse Core group at Microsoft. He is an expert in Data & AI Technical and Business Strategy & Philosophy. Harvey is also co-author of the book Data: A Guide to Humans, which explores the concept of Data Empathy, and how it can power better use of data through better communication and understanding of stakeholders in the value chain of data. 

Today I’m discussing something we’ve been talking about a lot on the podcast recently - the definition of a “data product.” While my definition is still a work in progress, I think it’s worth putting out into the world at this point to get more feedback. In addition to sharing my definition of data products (as defined the “producty way”), on today’s episode definition, I also discuss some of the non-technical skills that data product managers (DPMs) in the ML and AI space need if they want to achieve good user adoption of their solutions. I’ll also share my thoughts on whether data scientists can make good data product managers, what a DPM can do to better understand your users and stakeholders, and how product and UX design factors into this role. 

Highlights/ Skip to:

I introduce my reasons for sharing my definition of a data product (0:46) My definition of data product (7:26) Thinking the “producty” way (8:14) My thoughts on necessary skills for data PMs (in particular, AI & machine learning product management) (12:21) How data scientists can become good data product managers (DPMs) by taking off the data science hat (13:42) Understanding the role of UX design within the context of DPM (16:37) Crafting your sales and marketing strategies to emphasize the value of your product to the people who can use or purchase it (23:07) How to build a team that will help you increase adoption of your data product (30:01) How to build relationships with stakeholders/customers that allow you to find the right solutions for them (33:47) Letting go of a technical identity to develop a new identity as a DPM who can lead a team to build a product that actually gets used (36:32)

Quotes from Today’s Episode “This is what’s missing in some of the other definitions that I see around data products  [...] they’re not talking about it from the customer of the data product lens. And that orientation sums up all of the work that I’m doing and trying to get you to do as well, which is to put the people at the center of the work that you’re doing and not the data science, engineering, tech, or design. I want you to put the people at the center.” (6:12) “A data product is a data-driven, end-to-end, human-in-the-loop decision support solution that’s so valuable, users would potentially pay to use it.” (7:26) “I want to plunge all the way in and say, ‘if you want to do this kind of work, then you need to be thinking the product-y way.’ And this means inherently letting go of some of the data science-y way of thinking and the data-first kinds of ways of thinking.” (11:46) “I’ve read in a few places that data scientists don’t make for good data product managers. [While it may be true that they’re more introverted,] I don’t think that necessarily means that there’s an inherent problem with data scientists becoming good data product managers. I think the main challenge will be—and this is the same thing for almost any career transitioning into product management—is knowing when to let go of your former identity and wear the right hat at the right time.” (14:24) “Make better things for people that will improve their life and their outcomes and the business value will follow if you’ve properly aligned those two things together.” (17:21) “The big message here is this: there is always a design and experience, whether it is an API, or a platform, a dashboard, a full application, etc. Since there are no null design choices, how much are you going to intentionally shape that UX, or just pray that it comes out good on the other end? Prayer is not really a reliable strategy.  If you want to routinely do this work right, you need to put intention behind it.” (22:33)  “Relationship building is a must, and this is where applying user experience research can be very useful—not just for users, but also with stakeholders. It’s learning how to ask really good questions and learning the feelings, emotions, and reasons why people ask your team to build the thing that they’ve asked for. Learning how to dig into that is really important.” (26:26)

Links Designing for Analytics Community Work With Me Email Record a question

Summary The data ecosystem has been growing rapidly, with new communities joining and bringing their preferred programming languages to the mix. This has led to inefficiencies in how data is stored, accessed, and shared across process and system boundaries. The Arrow project is designed to eliminate wasted effort in translating between languages, and Voltron Data was created to help grow and support its technology and community. In this episode Wes McKinney shares the ways that Arrow and its related projects are improving the efficiency of data systems and driving their next stage of evolution.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show! Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Struggling with broken pipelines? Stale dashboards? Missing data? If this resonates with you, you’re not alone. Data engineers struggling with unreliable data need look no further than Monte Carlo, the leading end-to-end Data Observability Platform! Trusted by the data teams at Fox, JetBlue, and PagerDuty, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, dbt models, Airflow jobs, and business intelligence tools, reducing time to detection and resolution from weeks to just minutes. Monte Carlo also gives you a holistic picture of data health with automatic, end-to-end lineage from ingestion to the BI layer directly out of the box. Start trusting your data with Monte Carlo today! Visit dataengineeringpodcast.com/montecarlo to learn more. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc., are already available as plug-and-play connectors with reliable, intuitive SaaS solutions. Hevo Data is a highly reliable and intuitive data pipeline platform used by data engineers from 40+ countries to set up and run low-latency ELT pipelines with zero maintenance. Boasting more than 150 out-of-the-box connectors that can be set up in minutes, Hevo also allows you to monitor and control your pipelines. You get: real-time data flow visibility, fail-safe mechanisms, and alerts if anything breaks; preload transformations and auto-schema mapping precisely control how data lands in your destination; models and workflows to transform data for analytics; and reverse-ETL capability to move the transformed data back to your business software to inspire timely action. All of this, plus its transparent pricing and 24*7 live support, makes it consistently voted by users as the Leader in the Data Pipeline category on review platforms like G2. Go to dataengineeringpodcast.com/hevodata and sign up for a free 14-day trial that also comes with 24×7 support. Your host is Tobias Macey and today I’m interviewing Wes McKinney about his work at Voltron Data and on the Arrow ecosystem

Interview

Introduction How did you get involved in the area of data management? Can you describe what you are building at Voltron Data and the story behind it? What is the vision for the broader data ecosystem that you are trying to realize through your investment in Arrow and related projects?

How does your work at Voltron Data contribute to the realization of that vision?

What is the impact on engineer productivity and compute efficiency that gets introduced by the impedance mismatches between language and framework representations of data? The scope and capabilities of the Arrow project have grown substantially since it was first introduced. Can you give an overview of the current features and extensions to the project? What are some of the ways that ArrowVe and its related projects can be integrated with or replace the different elements of a data platform? Can you describe how Arrow is implemented?

What are the most complex/challenging aspects of the engineering needed to support interoperable data interchange between language runtimes?

How are you balancing the desire to move quickly and improve the Arrow protocol and implementations, with the need to wait for other players in the ecosystem (e.g. database engines, compute frameworks, etc.) to add support? With the growing application of data formats such as graphs and vectors, what do you see as the role of Arrow and its ideas in those use cases? For workflows that rely on integrating structured and unstructured data, what are the options for interaction with non-tabular data? (e.g. images, documents, etc.) With your support-focused business model, how are you approaching marketing and customer education to make it viable and scalable? What are the most interesting, innovative, or unexpected ways that you have seen Arrow used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on Arrow and its ecosystem? When is Arrow the wrong choice? What do you have planned for the future of Arrow?

Contact Info

Website wesm on GitHub @wesmckinn on Twitter

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don’t forget to check out our other shows. Podcast.init covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story. To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

Links

Voltron Data Pandas

Podcast Episode

Apache Arrow Partial Differential Equation FPGA == Field-Programmable Gate Array GPU == Graphics Processing Unit Ursa Labs Voltron (cartoon) Feature Engineering PySpark Substrait Arrow Flight Acero Arrow Datafusion Velox Ibis SIMD == Single Instruction, Multiple Data Lance DuckDB

Podcast Episode

Data Threads Conference Nano-Arrow Arrow ADBC Protocol Apache Iceberg

Podcast Episode

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Sponsored By: Atlan:

Have you ever woken up to a crisis because a number on a dashboard is broken and no one knows why? Or sent out frustrating slack messages trying to find the right data set? Or tried to understand what a column name means?

Our friends at Atlan started out as a data team themselves and faced all this collaboration chaos themselves, and started building Atlan as an internal tool for themselves. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more.

Go to dataengineeringpodcast.com/atlan and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription.a href="https://dataengineeringpodcast.com/montecarlo"…