Randy Bean and I discuss why generative AI is making companies more data-oriented, the latest Wave Stone Data and AI Leadership Executive Survey, his career and writing process and much more.

We learned so much about generative AI and its impact for people and organizations in 2023, we must anticipate many more innovations in the data and AI space 2024. One of the best places to look for this information is through the wisdom of those that spend their time with the Fortune 1000 leaders that are helping shape data and AI practices. Wavestone’s annual Data and AI Executive Leadership Survey is a great way to gain insight into thoughts in current practices, as well as understand what to expect from business leaders and organizations in the near future. In this episode, we speak to the author of the survey.  Randy Bean is a start-up business founder, CEO, industry thought leader, author, and speaker in the field of data-driven business leadership.  He serves as Innovation Fellow, Data Strategy for Paris-based consultancy Wavestone. Randy is the creator of the Data and AI Leadership Executive Survey discussed in today's episode. He is the author of the bestselling "Fail Fast, Learn Faster: Lessons in Data-Driven Leadership in an Age of Disruption, Big Data, and AI", and a current contributor to Forbes, Harvard Business Review, and MIT Sloan Management Review.   In the episode, Richie and Randy explore the 2024 Data and AI Leadership Executive Survey, the impact of generative AI in 2023 and what to expect from it in 2024, the state of generative AI implementation in organizations, healthcare and AI, including examples of generative AI outperforming human doctors, the evolving responsibilities of CDOs, the increasing importance of data-driven decision-making in organizations, the barriers to becoming data-driven, insights on data skills and the generational shift towards more data-savvy business leaders, as well as much more.  Links Mentioned in the Show: Data and AI Leadership Executive SurveyRandy’s Articles in ForbesAlly FinancialResponsible AI InstituteCourse: Implementing AI Solutions in Business

We’re definitely in AI hype mode at the moment largely driven by the evolution in generative AI. However, it seems like this progress is not necessarily driving lots of data-related innovation inside organisations that are not AI-first tech companies. A recent survey published by Randy Bean’s company, NewVantage Partners, confirms this. Here are the main findings compared to when the survey was last run 4 years ago: 59.5% of executives say their companies use data for business innovation – the same as four years ago.A drop from 47.6% to 40.8% of executives say their companies compete using data and analytics.Fewer executives (39.5% down from 46.9%) say their companies manage data as a business asset.Only 23.9% of executives now say their companies are data-driven, compared to 31% before.Just 20.6% of executives report having a data culture in their companies, down 27% from 28.3% in 2019.These numbers spell regression, not progress. Why is it so hard to become a truly data-driven organisation? In this episode, Randy and I explore the challenges facing Chief Data & Analytics Officers and their teams, including: How organizations can create an environment that encourages innovation in data-driven initiativesExamples of organisations doing data well, and whyHow to set clear expectations around the responsibilities of CDAOsThe most important qualities for someone in the CDAO role, and much more.Randy on LinkedIn: https://www.linkedin.com/in/randybeannvp/ Randy's website and book, 'Fail Fast, Learn Faster': https://www.randybeandata.com/book

Summary Organizations of all sizes are striving to become data driven, starting in earnest with the rise of big data a decade ago. With the never-ending growth in data sources and methods for aggregating and analyzing them, the use of data to direct the business has become a requirement. Randy Bean has been helping enterprise organizations define and execute their data strategies since before the age of big data. In this episode he discusses his experiences and how he approached the work of distilling them for his book "Fail Fast, Learn Faster". This is an entertaining and enlightening exploration of the business side of data with an industry veteran.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show! Struggling with broken pipelines? Stale dashboards? Missing data? If this resonates with you, you’re not alone. Data engineers struggling with unreliable data need look no further than Monte Carlo, the world’s first end-to-end, fully automated Data Observability Platform! In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, ETL, and business intelligence, reducing time to detection and resolution from weeks or days to just minutes. Start trusting your data with Monte Carlo today! Visit dataengineeringpodcast.com/impact today to save your spot at IMPACT: The Data Observability Summit a half-day virtual event featuring the first U.S. Chief Data Scientist, founder of the Data Mesh, Creator of Apache Airflow, and more data pioneers spearheading some of the biggest movements in data. The first 50 to RSVP with this link will be entered to win an Oculus Quest 2 — Advanced All-In-One Virtual Reality Headset. RSVP today – you don’t want to miss it! Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription Your host is Tobias Macey and today I’m interviewing Randy Bean about his recent book focusing on the use of big data and AI for informing data driven business leadership

Interview

Introduction How did you get involved in the area of data management? Can you start by discussing the focus of the book and what motivated you to write it?

Who is the intended audience, and how did that inform the tone and content?

Businesses and their officers have been aiming to be "data driven" for years. In your experience, what are the concrete goals that are implied by that term?

What are the barriers that organizations encounter in the pursuit of those goals? How have the success rates (real and imagined) shifted in recent years as the level of sophisticatio