This IBM® Redbooks® publication describes the features and functions the latest member of the IBM Z® platform, the IBM z15™ Model T02 (machine type 8562). It includes information about the IBM z15 processor design, I/O innovations, security features, and supported operating systems. The z15 is a state-of-the-art data and transaction system that delivers advanced capabilities, which are vital to any digital transformation. The z15 is designed for enhanced modularity, which is in an industry standard footprint. This system excels at the following tasks: Making use of multicloud integration services Securing data with pervasive encryption Accelerating digital transformation with agile service delivery Transforming a transactional platform into a data powerhouse Getting more out of the platform with IT Operational Analytics Accelerating digital transformation with agile service delivery Revolutionizing business processes Blending open source and Z technologies This book explains how this system uses new innovations and traditional Z strengths to satisfy growing demand for cloud, analytics, and open source technologies. With the z15 as the base, applications can run in a trusted, reliable, and secure environment that improves operations and lessens business risk.

In this talk, Yassine will dive deeper into the transformative impact of AI on Agile methodologies, focusing on software development.

Our panel of experts are well versed in the dynamic realm of data stack evolution. We sit them down for a deep dive on the dichotomy between open-source and proprietary platforms, unraveling the "Occam's Razor" for making the best choice. Discover the strategies they've used for agile experimentation with diverse data architectures, adeptly maneuvering through privacy and scalability constraints. This is a must see panel that will give you the tools and tips needed to navigate the ever changing Data Stack.

Yello is currently embarking on a journey to modernize because their existing platform inhibits their ability to provide speed to insights for internal and external clients. Yello needed a solution that not only improved our ability to extract insights, but also enables the team to establish a single source of truth and enhance their level of data stewardship. Yello's new data architecture needed to be nimble, flexible, and agile - developing a solution that not only works for their clients, but also works internally for downstream consumers. Hear from Shawn Crenshaw and Peter Lim as they share insights from this moderinzation journey, and discuss how to develop and implement a data lakehouse as part of the journey. This final data lakehouse architecture will satisfy client needs and accomplish the mission of the Yello Data Services team, which is to improve the health and accessibility of data at Yello.

Keith Belanger is an OG data modeling practitioner, having been in the game for decades.

We chat about a wide range of data modeling topics.

What's changed and what's stayed the same? How to model data to fit the business's needs. Agile data modeling. When it works, when it doesn't. Data modeling for data mesh and decentralization. The art of data modeling How to teach conceptual data modeling to new practitioners

Keith brings a wealth of experience and a practical, no-nonsense perspective. If you're interested in data modeling, don't miss this!

LinkedIn: https://www.linkedin.com/in/krbelanger/

Alejandro Matamala Ortiz is the Co-Founder and Chief Design Officer of Runway, an applied AI research company shaping the next era of art, entertainment, and human creativity. Runway has pioneered research in multi-modal AI systems by applying deep learning techniques to audiovisual content, building a future where content creation is accessible, controllable and empowering for creatives. Join us as we speak to Alejandro about the intersection of design, technology and creativity as generative AI becomes ubiquitous. Learn his perspective on the importance of designing flexible products and solutions, to deliver value for users today and enable an agile business model for the future.

Click the blue “Learn more” button above to tap into special offers designed to help you implement what you are learning at Google Cloud Next 25.

Conventional data governance conflicts with today’s world of self-service analytics and agile projects. Published at: https://www.eckerson.com/articles/modern-data-governance-problems

Dive into a thought-provoking Panel Discussion on 'How AI Is Rewriting the Creative Rulebook: Ethics, Innovation, and the Future of Art.' 🎨🤖 Explore the intersection of AI, ethics, and art innovation, and gain insights into the exciting future of creative expression. 🗣️🖼️ #AI #Art #ethics

✨ H I G H L I G H T S ✨

🙌 A huge shoutout to all the incredible participants who made Big Data Conference Europe 2023 in Vilnius, Lithuania, from November 21-24, an absolute triumph! 🎉 Your attendance and active participation were instrumental in making this event so special. 🌍

Don't forget to check out the session recordings from the conference to relive the valuable insights and knowledge shared! 📽️

Once again, THANK YOU for playing a pivotal role in the success of Big Data Conference Europe 2023. 🚀 See you next year for another unforgettable conference! 📅 #BigDataConference #SeeYouNextYear

Join Agnė Kelminskienė, Gabrielė Adomonytė, and Vaiva Mikelevičienė in a lively panel discussion as they explore the application of Agile principles and practices in the world of analytics. 📊🔄 Gain insights from experts in this engaging session! #AgileAnalytics #PanelDiscussion 🗨️

✨ H I G H L I G H T S ✨

🙌 A huge shoutout to all the incredible participants who made Big Data Conference Europe 2023 in Vilnius, Lithuania, from November 21-24, an absolute triumph! 🎉 Your attendance and active participation were instrumental in making this event so special. 🌍

Don't forget to check out the session recordings from the conference to relive the valuable insights and knowledge shared! 📽️

Once again, THANK YOU for playing a pivotal role in the success of Big Data Conference Europe 2023. 🚀 See you next year for another unforgettable conference! 📅 #BigDataConference #SeeYouNextYear

We talked about:

Lera’s background Lera’s move from Ukraine to Germany The transition from Marketing to Product Ownership The importance of communication and one-on-ones The role of Product Owner Utilizing Scrum as a Product Owner Building teams and cross-functionality Lera’s experience learning about search The importance of having both technical knowledge and business context Open developer positions at AUTODOC What experience Lera came to AUTODOC with How marketing skills helped Lera in her current role Lera’s resource recommendations Everything is possible

Links:

Post: https://www.linkedin.com/posts/leracaiman_elasticsearch-ecommerce-activity-7106615081588674560-5WQO

Free ML Engineering course: http://mlzoomcamp.com Join DataTalks.Club: https://datatalks.club/slack.html Our events: https://datatalks.club/events.html

Behind any good DataOps within a Modern Data Stack (MDS) architecture is a solid DevOps design! This is particularly pressing when building an MDS solution at scale, as reliability, quality and availability of data requires a very high degree of process automation while remaining fast, agile and resilient to change when addressing business needs.

While DevOps in Data Engineering is nothing new - for a broad-spectrum solution that includes data warehouse, BI, etc seemed either a bit out of reach due to overall complexity and cost - or simply overlooked due to perceived issues around scaling often attributed to the challenges of automation in CI/CD processes. However, this has been fast changing with tools such as dbt having super cool features which allow a very high degree of autonomy in the CI/CD processes with relative ease, with flexible and cutting edge features around pre-commits, Slim CI, etc.

In this session, Datatonic covers the challenges around building and deploying enterprise-grade MDS solutions for analytics at scale and how they have used dbt to address those - especially around near-complete autonomy to the CI/CD processes!

Speaker: Ash Sultan, Lead Data Architect, Datatonic

Register for Coalesce at https://coalesce.getdbt.com

The Guild team migrated to Snowflake and dbt for their data warehousing needs and immediately saw the benefits of standardizing model structure, DRYer logic, data lineage ,and automated testing on Pull Requests.

But leveraging dbt didn’t solve everything. Pain points around maintaining model logic, handling historical data, and integrating data from over 60 source systems meant that analysts still struggled to provide a unified view of the business. The team knew that they needed to level up their processes and modeling again, and chose to adopt Data Vault (DV).

Brandon and Rebecca take you behind the scenes of this decision to explain the benefits of Data Vault. They highlight DV’s ability to handle complex data integration requirements while remaining agile and demonstrate that it complements other modern data concepts like domain-driven design and data mesh.

Attendees learn what Data Vault is, when it can be a key component of a successful data strategy, and instances where it’s not the right fit. Walk away with practical tips to successfully transition based on a real-world implementation.

Guild transformed their data warehouse; you can too!

Speakers: Brandon Taylor, Senior Data Architect, Guild; Rebecca Di Bari Staff Data Engineer , Guild

Register for Coalesce at https://coalesce.getdbt.com

In the rapidly evolving landscape of data technology, the integration of Snowflake and dbt has revolutionized the creation and management of data applications. Now, developers can harness their combined capabilities to build superior, scalable, and sophisticated data applications.

With Snowflake’s cloud-based architecture, developers can access boundless storage, computing, and seamless data sharing. Additionally, Snowpark Python enables the performance of data transformation, analytics, and algorithmic functions within Snowflake, presenting developers with a new realm of opportunities. Incorporating dbt further enhances the synergy, allowing developers to streamline data workflows in an agile, model-driven environment.

This session covers how the Snowflake and dbt partnership can pave the way toward building better, future-proof data applications that cater to the dynamic needs of businesses in the digital era.

Speaker: Tarik Dwiek, Head of Technology and Application Partners, Snowflake

Register for Coalesce at https://coalesce.getdbt.com

Data engineers build critical pipelines that underpin the computational infrastructure that powers our global economy. Analytics engineers translate impossibly complex business semantics into precise frameworks that give modern companies digital eyes and ears, making them the most agile and responsive organizations ever created. Data scientists find revolutionary truths among vast expanses of noise and distraction, unearthing tiny diamonds in endless mines of numerical dirt.

This talk isn’t about any of that.

This talk is about commas. It’s about capitalization. It’s about indentation. It’s about the tedious arguments that no sensible person should ever care about, but we, as data people, can’t seem to resist. It’s about indulging in our unhinged debates, analyzing data on billions of queries, and giving us all a reason to bury our flag a bit deeper into whatever petty hill we’ve each chosen to die on. Let the good times roll.

Will it tell us anything useful? No. Will there be slides that you can take pictures of to show your leadership team why your role is valuable? Absolutely not. But there will be lots of charts , graphs , and misplaced commas.

Speaker: Benn Stancil, CTO, Mode

Register for Coalesce at https://coalesce.getdbt.com

We talked about:

Loïc's background Data management Loïc's transition to data engineer Challenges in the transition to data engineering What is a data architect? The output of a data architect's work Establishing metrics and dimensions The importance of communication Setting up best practices for the team Staying relevant and tech-watching Setting up specifications for a pipeline Be agile, create a POC, iterate ASAP, and build reusable templates Reaching out to Loïc for questions

Links:

Loiic LinkedIn: https://www.linkedin.com/in/loicmagnien/

Free ML Engineering course: http://mlzoomcamp.com

Join DataTalks.Club: https://datatalks.club/slack.html

Our events: https://datatalks.club/events.html

By choosing the right database, you can maximize your business potential, improve performance, increase efficiency, and gain a competitive edge. This insightful report examines the benefits of using a simplified data architecture containing cloud-based HTAP (hybrid transactional and analytical processing) database capabilities. You'll learn how this data architecture can help data engineers and data decision makers focus on what matters most: growing your business. Authors Joe McKendrick and Ed Huang explain how cloud native infrastructure supports enterprise businesses and operations with a much more agile foundation. Just one layer up from the infrastructure, cloud-based databases are a crucial part of data management and analytics. Learn how distributed SQL databases containing HTAP capabilities provide more efficient and streamlined data processing to improve cost efficiency and expedite business operations and decision making. This report helps you: Explore industry trends in database development Learn the benefits of a simplified data architecture Comb through the complex and crowded database choices on the market Examine the process of selecting the right database for your business Learn the latest innovations database for improving your company's efficiency and performance

Are you looking for an introduction to the Lakehouse and what the related technology is all about? This session is for you. This session explains the value that lakehouses bring to the table using examples of companies that are actually modernizing their data, showing demos throughout. The data lakehouse is the future for modern data teams that want to simplify data workloads, ease collaboration, and maintain the flexibility and openness to stay agile as a company scales.

Come to this session and learn about the full stack, including data engineering, data warehousing in a lakehouse, data streaming, governance, and data science and AI. Learn how you can create modern data solutions of your own.

Talk by: Ari Kaplan and Pearl Ubaru

Connect with us: Website: https://databricks.com Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/databricks Instagram: https://www.instagram.com/databricksinc Facebook: https://www.facebook.com/databricksinc

How can you maximize data collaboration across your organization without having to build integrations between individual applications, systems, and other data sources? Data collaboration architectures that don't depend on integrations aren't a new idea, but they've assumed greater urgency as organizations increasingly struggle to manage the ever-growing numbers of data sources that exist inside their IT estates. In this report, Cinchy cofounders Dan DeMers and Karanjot Jaswal show CIOs, CTOs, CDOs, and other IT leaders how to rethink their organization's approach to data architectures, data management, and data governance. You'll learn about different approaches to creating data platforms that liberate and autonomize data, enable agile data management, apply consistent data access controls, and maximize visibility without requiring application-specific integrations. With this report, you'll discover: Why data integration is often handled piecemeal—combining one app with another rather than integrating all apps together How data collaboration platforms enable data sharing across all apps, systems, and sources without application-specific integrations Four major platforms you can use to make data available to all applications and services: Cinchy, K2View, Microsoft Dataverse, and The Modern Data Company Principles and practices for deploying the data collaboration platform of your choice Dan DeMers is the CEO and cofounder of Cinchy. Karanjot Jaswal is cofounder and CTO of Cinchy.

DATA WRANGLING Written and edited by some of the world’s top experts in the field, this exciting new volume provides state-of-the-art research and latest technological breakthroughs in data wrangling, its theoretical concepts, practical applications, and tools for solving everyday problems. Data wrangling is the process of cleaning and unifying messy and complex data sets for easy access and analysis. This process typically includes manually converting and mapping data from one raw form into another format to allow for more convenient consumption and organization of the data. Data wrangling is increasingly ubiquitous at today’s top firms. Data cleaning focuses on removing inaccurate data from your data set whereas data wrangling focuses on transforming the data’s format, typically by converting “raw” data into another format more suitable for use. Data wrangling is a necessary component of any business. Data wrangling solutions are specifically designed and architected to handle diverse, complex data at any scale, including many applications, such as Datameer, Infogix, Paxata, Talend, Tamr, TMMData, and Trifacta. This book synthesizes the processes of data wrangling into a comprehensive overview, with a strong focus on recent and rapidly evolving agile analytic processes in data-driven enterprises, for businesses and other enterprises to use to find solutions for their everyday problems and practical applications. Whether for the veteran engineer, scientist, or other industry professional, this book is a must have for any library.

Summary

A significant portion of the time spent by data engineering teams is on managing the workflows and operations of their pipelines. DataOps has arisen as a parallel set of practices to that of DevOps teams as a means of reducing wasted effort. Agile Data Engine is a platform designed to handle the infrastructure side of the DataOps equation, as well as providing the insights that you need to manage the human side of the workflow. In this episode Tevje Olin explains how the platform is implemented, the features that it provides to reduce the amount of effort required to keep your pipelines running, and how you can start using it in your own team.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudderstack Your host is Tobias Macey and today I'm interviewing Tevje Olin about Agile Data Engine, a platform that combines data modeling, transformations, continuous delivery and workload orchestration to help you manage your data products and the whole lifecycle of your warehouse

Interview

Introduction How did you get involved in the area of data management? Can you describe what Agile Data Engine is and the story behind it? What are some of the tools and architectures that an organization might be able to replace with Agile Data Engine?

How does the unified experience of Agile Data Engine change the way that teams think about the lifecycle of their data? What are some of the types of experiments that are enabled by reduced operational overhead?

What does CI/CD look like for a data warehouse?

How is it different from CI/CD for software applications?

Can you describe how Agile Data Engine is architected?

How have the design and goals of the system changed since you first started working on it? What are the components that you needed to develop in-house to enable your platform goals?

What are the changes in the broader data ecosystem that have had the most influence on your product goals and customer adoption? Can you describe the workflow for a team that is using Agile Data Engine to power their business analytics?

What are some of the insights that you generate to help your customers understand how to improve their processes or identify new opportunities?

In your "about" page it mentions the unique approaches that you take for warehouse automation. How do your practices differ from the rest of the industry? How have changes in the adoption/implementation of ML and AI impacted the ways that your customers exercise your platform? What are the most interesting, innovative, or unexpected ways that you have seen the Agile Data Engine platform used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on Agile Data Engine? When is Agile Data Engine the wrong choice? What do you have planned for the future of Agile Data Engine?

Guest Contact Info

LinkedIn

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

About Agile Data Engine

Agile Data Engine unlocks the potential of your data to drive business value - in a rapidly changing world. Agile Data Engine is a DataOps Management platform for designing, deploying, operating and managing data products, and managing the whole lifecycle of a data warehouse. It combines data modeling, transformations, continuous delivery and workload orchestration into the same platform.

Links

Agile Data Engine Bill Inmon Ralph Kimball Snowflake Redshift BigQuery Azure Synapse Airflow

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Sponsored By: Rudderstack:

RudderStack provides all your customer data pipelines in one platform. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines.

RudderStack’s warehouse-first approach means it does not store sensitive information, and it allows you to leverage your existing data warehouse/data lake infrastructure to build a single source of truth for every team.

RudderStack also supports real-time use cases. You can Implement RudderStack SDKs once, then automatically send events to your warehouse and 150+ business tools, and you’ll never have to worry about API changes again.

Visit dataengineeringpodcast.com/rudderstack to sign up for free today, and snag a free T-Shirt just for being a Data Engineering Podcast listener.Support Data Engineering Podcast