In this podcast, Dewayne Washington speaks the unadulterated truth about the role of the CIO and discusses keys to success and common pitfalls. Washington is a senior consultant with 20+ years of experience in BI and Analytics in over two dozen verticals. He is the former BI manager at Dallas/Fortworth International Airport and the current CIO at The Business of Intelligence. Washington is also a sought-after speaker and mentor for organizations striving to leverage BI and Analytics to meet business goals, thus earning him the title, BI Pharaoh.

Mike Masciandaro is a veteran business intelligence practitioner who recently retired from an illustrious career at Dow Chemical. Mike has seen and done just about everything there is to do in the world of BI, data, and analytics. He is now intent on sharing his hard-won knowledge with others. You will learn the definition and purpose of a BI program, the role of subject matter experts, how to hire and retain talent, keys to delivering value as a BI program, and more.

Perform Interactive Data Analysis with Smarter Visualizations and Support your Enterprise-wide Analytical Needs About This Book Get a practical demonstration of discovering data for sales, human resources, and more using Qlik Sense Create dynamic dashboards for business intelligence and predictive analytics Create and collaborate comprehensive analytical solutions using Rattle and Qlik Sense Who This Book Is For This course is for anyone who wishes to understand and utilize the various new approaches to business intelligence actively in their business practice. Knowing the basics of business intelligence concepts would be helpful when picking up this course, but is not mandatory. What You Will Learn Build simple visualization models with Rattle and Qlik Sense Desktop Get to grips with the life cycle and new visualization functions of a Qlik Sense application Discover simple ways to examine data and get it ready for analysis Visualize your data with Qlik Sense's engaging and informative graphs Build efficient and responsive Associative Models Optimize Qlik Sense for sales, human resources, and demographic data discovery Explore various tips and tricks of navigation for the Qlik Sense® front end Develop creative extensions for your Qlik Sense® dashboard In Detail Qlik Sense is powerful and creative visual analytics software that allows users to discover data, explore it, and dig out meaningful insights in order to make a profit and make decisions for your business. This course begins by introducing you to the features and functions of the most modern edition of Qlik Sense so you get to grips with the application. The course will teach you how to administer the data architecture in Qlik Sense, enabling you to customize your own Qlik Sense application for your business intelligence needs. It also contains numerous recipes to help you overcome challenging situations while creating fully featured desktop applications in Qlik Sense. It explains how to combine Rattle and Qlik Sense Desktop to apply predictive analytics to your data to develop real-world interactive data applications. The course includes premium content from three of our most popular books: Learning Qlik Sense: The Official Guide Second Edition Qlik Sense Cookbook Predictive Analytics using Rattle and Qlik Sense On completion of this course, you will be self-sufficient in improving your data analysis and will know how to apply predictive analytics to your datasets. Through this course, you will be able to create predictive models and data applications, allowing you to explore your data insights much deeper. Style and approach The course will follow a practical approach with rich set of examples through which it will demonstrate its concepts, features and its implementation. The course will also feature numerous solutions which will cover entire spectrum of BI use cases.

Learning Alteryx introduces you to using the powerful Alteryx platform for self-service analytics, helping you master key features like data preparation and predictive analytics without needing to code. With this book, you'll gain the skills to create workflows that generate actionable insights, empowering your business to make data-driven decisions. What this Book will help me do Master creating and optimizing workflows in Alteryx to address complex analytical problems. Learn how to clean, prepare, and blend data from various sources efficiently. Understand advanced Alteryx expressions for processing large datasets effectively. Develop meaningful reports and visualizations to communicate insights clearly. Leverage predictive analytics capabilities in Alteryx to make informed decisions. Author(s) The authors of Learning Alteryx collectively bring years of expertise in data analytics and business intelligence. Having worked on diverse projects across multiple industries, they understand the challenges faced by data professionals and are skilled in simplifying complex concepts. They focus on providing practical insights and step-by-step guides to empower learners. Who is it for? Learning Alteryx is ideal for professionals aspiring to enhance their data analytics capabilities or explore self-service analytics. It caters to beginners unfamiliar with analytics platforms, as well as intermediate users seeking to deepen their Alteryx knowledge. Readers should have a basic understanding of data analysis principles.

Written for students in undergraduate and graduate statistics courses, as well as for the practitioner who wants to make better decisions from data and models, this updated and expanded second edition of Fundamentals of Predictive Analytics with JMP(R) bridges the gap between courses on basic statistics, which focus on univariate and bivariate analysis, and courses on data mining and predictive analytics. Going beyond the theoretical foundation, this book gives you the technical knowledge and problem-solving skills that you need to perform real-world multivariate data analysis. First, this book teaches you to recognize when it is appropriate to use a tool, what variables and data are required, and what the results might be. Second, it teaches you how to interpret the results and then, step-by-step, how and where to perform and evaluate the analysis in JMP . Using JMP 13 and JMP 13 Pro, this book offers the following new and enhanced features in an example-driven format: an add-in for Microsoft Excel Graph Builder dirty data visualization regression ANOVA logistic regression principal component analysis LASSO elastic net cluster analysis decision trees k-nearest neighbors neural networks bootstrap forests boosted trees text mining association rules model comparison With today’s emphasis on business intelligence, business analytics, and predictive analytics, this second edition is invaluable to anyone who needs to expand his or her knowledge of statistics and to apply real-world, problem-solving analysis. This book is part of the SAS Press program.

Use machine learning and Oracle Business Intelligence Enterprise Edition (OBIEE) as a comprehensive BI solution. This book follows a when-to, why-to, and how-to approach to explain the key steps involved in utilizing the artificial intelligence components now available for a successful OBIEE implementation. Oracle Business Intelligence with Machine Learning covers various technologies including using Oracle OBIEE, R Enterprise, Spatial Maps, and machine learning for advanced visualization and analytics. The machine learning material focuses on learning representations of input data suitable for a given prediction problem. This book focuses on the practical aspects of implementing machine learning solutions using the rich Oracle BI ecosystem. The primary objective of this book is to bridge the gap between the academic state-of-the-art and the industry state-of-the-practice by introducing you to machine learning with OBIEE. What You Will Learn See machine learning in OBIEE Master the fundamentals of machine learning and how it pertains to BI and advanced analytics Gain an introduction to Oracle R Enterprise Discover the practical considerations of implementing machine learning with OBIEE Who This Book Is For Analytics managers, BI architects and developers, and data scientists.

Quickly find solutions to dozens of common problems encountered while using XML and JSON features that are built into SQL Server. Content is presented in the popular problem-solution format. Look up the problem that you want to solve. Read the solution. Apply the solution directly in your own code. Problem solved! This book shows how to take advantage of XML and JSON to share data and automate tasks. JSON is commonly used to move data back and forth between the database and front-end applications, often running in a browser. This book shows all you need to know about transforming query results into JSON format, and back again. Also covered are the processes and techniques for moving data into and out of XML format for business intelligence and other purposes, such as when transferring data from a reporting system into a data warehouse, or between different database brands such as between SQL Server and Oracle. Microsoft intensively implements XML in SQL Server, and in many related products. Execution plans are generated in XML format, and this book shows you how to parse those plans and automate the detection of performance problems. The relatively new Extended Events feature writes tracing data into XML files, and the recipes in this book help in parsing those files. XML is also used in SQL Server's BI tool set, including in SSIS, SSR, and SSAS. XML is used in many configuration files, and is even behind the construction of DDL triggers. In reading this book you’ll dive deeply into the features that allow you to build and parse XML, and also JSON, which is a specific format of XML used to transmit objects in a web-friendly format between a database and its front-end applications. What You Will Learn Build XML and JSON objects in support of automation and data transfer Import and parse XML and JSON from operating system files Build appropriate indexes on XML objects to improve query performance Move data from query result sets into JSON format, and back again Automate the detection of database performance problems by querying and parsing the database’s own execution plans Replace external and manual JSON processes with SQL Server's internal, JSON functionality Who This Book Is For Database administrators, .NET developers, business intelligence developers, and other professionals who want a deep and detailed skill set around working with XML and JSON in a SQL Server database environment. Web developers will particularly find the book useful for its coverage of transforming database result sets into JSON text that can be transmitted to front-end web applications.

Deliver eye-catching Business Intelligence with Microsoft Power BI Desktop. This new edition has been updated to cover all the latest features, including combo charts, Cartesian charts, trend lines, use of gauges, and more. Also covered are Top-N features, the ability to bin data into groupings and chart the groupings, and new techniques for detecting and handling outlier data points. You can take data from virtually any source and use it to produce stunning dashboards and compelling reports that will seize your audience’s attention. Slice and dice the data with remarkable ease and then add metrics and KPIs to project the insights that create your competitive advantage. Make raw data into clear, accurate, and interactive information with Microsoft’s free self-service business intelligence tool. Pro Power BI Desktop shows you how to choose from a wide range of built-in and third-party visualization types so that your message is always enhanced. You’ll be able to deliver those results on the PC, tablets, and smartphones, as well as share results via the cloud. This book helps you save time by preparing the underlying data correctly without needing an IT department to prepare it for you. What You'll Learn Deliver attention-grabbing information, turning data into insight Mash up data from multiple sources into a cleansed and coherent data model Create dashboards that help in monitoring key performance indicators of your business Build interdependent charts, maps, and tables to deliver visually stunning information Share business intelligence in the cloud without involving IT Deliver visually stunning and interactive charts, maps, and tables Find new insights as you chop and tweak your data as never before Adapt delivery to mobile devices such as phones and tablets Who This Book Is For Everyone from CEOs and Business Intelligence developers to power users and IT managers

"Learning Pentaho Data Integration 8 CE" is your comprehensive guide to mastering data manipulation and integration using Pentaho Data Integration (PDI) 8 Community Edition. Through step-by-step instructions and practical examples, you'll learn to explore, transform, validate, and integrate data from multiple sources, equipping you to handle real-world data challenges efficiently. What this Book will help me do Effectively install and understand the foundational concepts of Pentaho Data Integration 8 Community Edition. Efficiently organize, clean, and transform raw data from various sources into useful formats. Perform advanced data operations like metadata injection, managing relational databases, and implementing ETL solutions. Design, create, and deploy comprehensive data warehouse solutions using modern best practices. Streamline daily data processing tasks with flexibility and accuracy while handling errors gracefully. Author(s) The author, Carina Roldán, is an experienced professional in the field of data science and ETL (Extract, Transform, Load) development. Her expertise in leveraging tools like Pentaho Data Integration has allowed her to contribute significantly to BI and data management projects. Her approach in writing this book reflects her commitment to simplifying complex topics for aspiring professionals. Who is it for? This book is ideal for software developers, data analysts, business intelligence professionals, and IT students aiming to enhance their skills in ETL processes using Pentaho Data Integration. Beginners who wish to learn PDI comprehensively and professionals looking to deepen their expertise will both find value in this resource. It's also suitable for individuals involved in data warehouse design and implementation. This book will equip you with the skills to handle diverse data transformation tasks effectively.

Prepare for Microsoft Exam 70-767–and help demonstrate your real-world mastery of skills for managing data warehouses. This exam is intended for Extract, Transform, Load (ETL) data warehouse developers who create business intelligence (BI) solutions. Their responsibilities include data cleansing as well as ETL and data warehouse implementation. The reader should have experience installing and implementing a Master Data Services (MDS) model, using MDS tools, and creating a Master Data Manager database and web application. The reader should understand how to design and implement ETL control flow elements and work with a SQL Service Integration Services package. Focus on the expertise measured by these objectives: • Design, and implement, and maintain a data warehouse • Extract, transform, and load data • Build data quality solutionsThis Microsoft Exam Ref: • Organizes its coverage by exam objectives • Features strategic, what-if scenarios to challenge you • Assumes you have working knowledge of relational database technology and incremental database extraction, as well as experience with designing ETL control flows, using and debugging SSIS packages, accessing and importing or exporting data from multiple sources, and managing a SQL data warehouse. Implementing a SQL Data Warehouse About the Exam Exam 70-767 focuses on skills and knowledge required for working with relational database technology. About Microsoft Certification Passing this exam earns you credit toward a Microsoft Certified Professional (MCP) or Microsoft Certified Solutions Associate (MCSA) certification that demonstrates your mastery of data warehouse management Passing this exam as well as Exam 70-768 (Developing SQL Data Models) earns you credit toward a Microsoft Certified Solutions Associate (MCSA) SQL 2016 Business Intelligence (BI) Development certification. See full details at: microsoft.com/learning

Walk away from old-fashioned and cumbersome query approaches and answer your business intelligence questions through simple and powerful queries built on common table expressions (CTEs) and window functions. These new features in MariaDB and MySQL help you to write queries without having to wade through a quagmire of brittle self-joins and other crazy techniques from the past. Your queries will generate correct results, be more readable and less brittle in the face of unexpected data, and you’ll be able to adapt them quickly in the face of changing business requirements. MariaDB and MySQL Common Table Expressions and Window Functions Revealed introduces and explains CTEs and window functions, newly available in MariaDB 10.2 and MySQL 8.0, and helps you understand why and how every MariaDB and MySQL database programmer should learn and apply these features in their daily work. CTEs and especially window functions enable easy solutions to many query challenges that in prior releases have been difficult and sometimes impossible to surmount. Mastering these features opens the door to query solutions that are more robust, execute faster, and are easier to maintain over time than prior solutions using older techniques. The book: Takes you step-by-step through the workings of common table expressions and window functions Provides easy-to-follow examples of the new syntax Helps you answer business questions faster and easier than ever What You'll Learn Answer business questions using simple queries that don’t break in the face of unexpected data Avoid writing queries that are a difficult-to-maintain quagmire of self-joins and nested subqueries Recognize situations that call for window functions, and learn when to use these features Reduce the need for performance-robbing self-joins Simplify and speed the execution of analytical queries Create queries that finish in seconds instead of hours Who This Book Is For Database administrators and application developers who want to quickly get up to speed on important features in MariaDB and MySQL for writing business intelligence queries. Any developer writing SQL against MariaDB and MySQL databases will benefit tremendously from the knowledge and techniques this book provides.

Learn Business Intelligence Markup Language (Biml) for automating much of the repetitive, manual labor involved in data integration. We teach you how to build frameworks and use advanced Biml features to get more out of SQL Server Integration Services (SSIS), Transact-SQL (T-SQL), and SQL Server Analysis Services (SSAS) than you ever thought possible. The first part of the book starts with the basics—getting your development environment configured, Biml syntax, and scripting essentials. Whether a beginner or a seasoned Biml expert, the next part of the book guides you through the process of using Biml to build a framework that captures both your design patterns and execution management. Design patterns are reusable code blocks that standardize the approach you use to perform certain types of data integration, logging, and other key data functions. Design patterns solve common problems encountered when developing data integration solutions. Because you do not have to build the code from scratch each time, design patterns improve your efficiency as a Biml developer. In addition to leveraging design patterns in your framework, you will learn how to build a robust metadata store and how to package your framework into Biml bundles for deployment within your enterprise. In the last part of the book, we teach you more advanced Biml features and capabilities, such as SSAS development, T-SQL recipes, documentation autogeneration, and Biml troubleshooting. The Biml Book: Provides practical and applicable examples Teaches you how to use Biml to reduce development time while improving quality Takes you through solutions to common data integration and BI challenges What You'll Learn Master the basics of Business Intelligence Markup Language (Biml) Study patterns for automating SSIS package generation Build a Biml Framework Import and transform database schemas Automate generation of scripts and projects Who This Book Is For BI developers wishing to quickly locate previously tested solutions, Microsoft BI specialists, those seeking more information about solution automation and code generation, and practitioners of Data Integration Lifecycle Management (DILM) in the DevOps enterprise

"Implementing Qlik Sense" is your practical guide to becoming a proficient Qlik Sense consultant. By exploring detailed examples and real-world scenarios, you will learn how to effectively design and develop BI solutions tailored to meet and exceed client expectations. This book equips you with the skills and strategies needed to excel in the ever-evolving world of business intelligence. What this Book will help me do Understand the role and responsibilities of a Qlik Sense consultant and the value they provide. Engage effectively with clients to gather requirements and align BI solutions with their strategic goals. Design and implement architectural frameworks for robust and scalable Qlik Sense solutions. Adopt best practices for project management, from requirement gathering to post-deployment support. Gain practical insight into all stages of a Qlik project using realistic case studies and scenarios. Author(s) None Hegde and None Solanki are experienced professionals in Qlik Sense technologies, with a consultancy background in delivering impactful BI solutions. Known for their hands-on expertise, practical teaching, and collaborative approach to problem-solving, their insights make this book an indispensable resource. Who is it for? This book is tailored for BI developers and analysts who have basic knowledge of Qlik Sense and aspire to elevate their expertise into full-fledged consultancy roles. It's ideal for those seeking to provide value-driven insights through Qlik implementations, improving customer engagement and satisfaction.

Nearly 7,000 new mobile applications appear every day, and a constant stream of data gives them life. Many organizations rely on a predictive analytics model to turn data into useful business information and ensure the predictions remain accurate as data changes. It can be a complex, time-consuming process. This book shows how to automate and accelerate that process using machine learning (ML) on a modern data warehouse that runs on any cloud. Product specialists from MemSQL explain how today’s modern data warehouses provide the foundations to implement ML algorithms that run efficiently. Through several real-time use cases, you’ll learn how to quickly identify the right metrics to make actionable business decisions. This book explores foundational ML and artificial intelligence concepts to help you understand: How data warehouses accelerate deployment and simplify manageability How companies make a choice between cloud and on-premises deployments for building data processing applications Ways to build analytics and visualizations for business intelligence on historical data The technologies and architecture for building and deploying real-time data pipelines This book demonstrates specific models and examples for building supervised and unsupervised real-time ML applications, and gives practical advice on how to make the choice between building an ML pipeline or buying an existing solution. If you need to use data accurately and efficiently, a real-time data warehouse is a critical business tool.

This comprehensive guide dives deep into the Power BI platform, teaching you how to create insightful data models, reports, and dashboards that drive business decisions. With hands-on recipes and real-world examples, this book is a practical resource for mastering the full range of Power BI's capabilities. What this Book will help me do Understand and apply data cleansing and transformation using Power BI tools. Create and utilize intuitive data models for business intelligence reporting. Leverage DAX and M languages for advanced data analysis and custom solutions. Build dynamic, user-specific dashboards and visualizations for impactful insights. Integrate Power BI with Microsoft Excel, SQL Server, and cloud services for extended functionality. Author(s) None Powell is an experienced data analyst and Microsoft BI solution architect with extensive expertise in Power BI. He has worked on numerous BI projects, providing practical solutions using Microsoft's data platform technologies. Through detailed, scenario-based writing, he shares his knowledge to help readers excel in their BI endeavors. Who is it for? This book is perfect for business intelligence professionals and analysts seeking to expand their skills in Power BI. Ideal readers may have foundational Power BI knowledge and look to master advanced techniques. If you aim to build impactful BI solutions and are motivated to handle complex data integrations, this book will be instrumental.

Relational databases haven’t gone away, but they are evolving to integrate messy, disjointed unstructured data into a cleansed repository for analytics. With the execution of massively parallel processing (MPP), the latest generation of analytic data warehouses is helping organizations move beyond business intelligence to processing a variety of advanced analytic workloads. These MPP databases expose their power with the familiarity of SQL. This report introduces the Greenplum Database, recently released as an open source project by Pivotal Software. Lead author Marshall Presser of Pivotal Data Engineering takes you through the Greenplum approach to data analytics and data-driven decisions, beginning with Greenplum’s shared-nothing architecture. You’ll explore data organization and storage, data loading, running queries, as well as performing analytics in the database. You’ll learn: How each networked node in Greenplum’s architecture features an independent operating system, memory, and storage Four deployment options to help you balance security, cost, and time to usability Ways to organize data, including distribution, storage, partitioning, and loading How to use Apache MADlib for in-database analytics, and GPText to process and analyze free-form text Tools for monitoring, managing, securing, and optimizing query responses available in the Pivotal Greenplum commercial database

Book + Content Update Program SQL Server 2016 High Availability Unleashed provides start-to-finish coverage of SQL Server’s powerful high availability (HA) solutions for your traditional on-premise databases, cloud-based databases (Azure or AWS), hybrid databases (on-premise coupled with the cloud), and your emerging Big Data solutions. This complete guide introduces an easy-to-follow, formal HA methodology that has been refined over the past several years and helps you identity the right HA solution for your needs. There is also additional coverage of both disaster recovery and business continuity architectures and considerations. You are provided with step-by-step guides, examples, and sample code to help you set up, manage, and administer these highly available solutions. All examples are based on existing production deployments at major Fortune 500 companies around the globe. This book is for all intermediate-to-advanced SQL Server and Big Data professionals, but is also organized so that the first few chapters are great foundation reading for CIOs, CTOs, and even some tech-savvy CFOs. Learn a formal, high availability methodology for understanding and selecting the right HA solution for your needs Deep dive into Microsoft Cluster Services Use selective data replication topologies Explore thorough details on AlwaysOn and availability groups Learn about HA options with log shipping and database mirroring/ snapshots Get details on Microsoft Azure for Big Data and Azure SQL Explore business continuity and disaster recovery Learn about on-premise, cloud, and hybrid deployments Provide all types of database needs, including online transaction processing, data warehouse and business intelligence, and Big Data Explore the future of HA and disaster recovery In addition, this book is part of InformIT’s exciting Content Update Program, which provides content updates for major technology improvements! As significant updates are made to SQL Server, sections of this book will be updated or new sections will be added to match the updates to the technologies. As updates become available, they will be delivered to you via a free Web Edition of this book, which can be accessed with any Internet connection. To learn more, visit informit.com/cup. How to access the Web Edition: Follow the instructions inside to learn how to register your book to access the FREE Web Edition. * The companion material is not available with the online edition on O'Reilly Learning

Discover the power of SAP Analytics Cloud in solving business intelligence challenges through concise and clear instruction. This book is the essential guide for beginners, providing you a comprehensive understanding of the platform's features and capabilities. By the end, you'll master creating reports, models, and dashboards, making data-driven decisions with confidence. What this Book will help me do Learn how to navigate and utilize the SAP Analytics Cloud interface effectively. Create data models using various sources like Excel or text files for comprehensive insights. Design and compile visually engaging stories, reports, and dashboards effortlessly. Master collaborative and presentation tools inside SAP Digital Boardroom. Understand how to plan, predict, and analyze seamlessly within a single platform. Author(s) None Ahmed is an experienced SAP consultant and analytics professional, bringing years of practical experience in BI tools and enterprise analytics. As an expert in SAP Analytics Cloud, None has guided numerous teams in deploying effective analytics solutions. Their writing aims to demystify complex tools for learners. Who is it for? This book is ideal for IT professionals, business analysts, and newcomers eager to understand SAP Analytics Cloud. Beginner-level BI developers and managers seeking guided steps for mastering this platform will find it invaluable. If you aim to enhance your career in cloud-based analytics, this book is tailored for you.

"QlikView for Developers" is a comprehensive guide to mastering QlikView, a powerful business intelligence tool. This book takes you on a journey from understanding the basics to building scalable and maintainable QlikView applications. Designed to provide practical methods, real-world scenarios, and valuable tips, it is ideal for anyone wanting to learn and effectively use QlikView for BI solutions. What this Book will help me do Understand the key features and architecture of QlikView and what has changed in QlikView 12. Learn to transform, model, and organize data in QlikView to effectively support business processes. Master best practices for creating interactive dashboards using charts, tables, and visualization objects. Discover techniques to optimize data architecture for scalable deployments and ensure data consistency. Implement advanced scripting and calculation methods, such as Set Analysis, to handle complex analytical requirements. Author(s) Miguel Angel Garcia and Barry Harmsen bring years of professional expertise in business intelligence and QlikView application development. Both authors have contributed significantly to the BI community and have extensive experience teaching and consulting on QlikView solutions. Their goal with this book is to provide a resource that is both informative and practical for QlikView developers. Who is it for? This book is intended for developers and analysts looking to harness the capabilities of QlikView for business intelligence purposes. It is suitable for beginners with minimal experience in QlikView, as well as for experienced practitioners wanting to deepen their knowledge and skills. The book provides a balanced approach that caters to various skill levels, ensuring accessible and actionable content for all readers.

Summary

Yelp needs to be able to consume and process all of the user interactions that happen in their platform in as close to real-time as possible. To achieve that goal they embarked on a journey to refactor their monolithic architecture to be more modular and modern, and then they open sourced it! In this episode Justin Cunningham joins me to discuss the decisions they made and the lessons they learned in the process, including what worked, what didn’t, and what he would do differently if he was starting over today.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at www.dataengineeringpodcast.com/linode?utm_source=rss&utm_medium=rss and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. You can help support the show by checking out the Patreon page which is linked from the site. To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers Your host is Tobias Macey and today I’m interviewing Justin Cunningham about Yelp’s data pipeline

Interview with Justin Cunningham

Introduction How did you get involved in the area of data engineering? Can you start by giving an overview of your pipeline and the type of workload that you are optimizing for? What are some of the dead ends that you experienced while designing and implementing your pipeline? As you were picking the components for your pipeline, how did you prioritize the build vs buy decisions and what are the pieces that you ended up building in-house? What are some of the failure modes that you have experienced in the various parts of your pipeline and how have you engineered around them? What are you using to automate deployment and maintenance of your various components and how do you monitor them for availability and accuracy? While you were re-architecting your monolithic application into a service oriented architecture and defining the flows of data, how were you able to make the switch while verifying that you were not introducing unintended mutations into the data being produced? Did you plan to open-source the work that you were doing from the start, or was that decision made after the project was completed? What were some of the challenges associated with making sure that it was properly structured to be amenable to making it public? What advice would you give to anyone who is starting a brand new project and how would that advice differ for someone who is trying to retrofit a data management architecture onto an existing project?

Keep in touch

Yelp Engineering Blog Email

Links

Kafka Redshift ETL Business Intelligence Change Data Capture LinkedIn Data Bus Apache Storm Apache Flink Confluent Apache Avro Game Days Chaos Monkey Simian Army PaaSta Apache Mesos Marathon SignalFX Sensu Thrift Protocol Buffers JSON Schema Debezium Kafka Connect Apache Beam

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support Data Engineering Podcast