Summary Python has grown to be one of the top languages used for all aspects of data, from collection and cleaning, to analysis and machine learning. Along with that growth has come an explosion of tools and engines that help power these workflows, which introduces a great deal of complexity when scaling from single machines and exploratory development to massively parallel distributed computation. In answer to that challenge the Fugue project offers an interface to automatically translate across Pandas, Spark, and Dask execution environments without having to modify your logic. In this episode core contributor Kevin Kho explains how the slight differences in the underlying engines can lead to big problems, how Fugue works to hide those differences from the developer, and how you can start using it in your own work today.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show! Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription The only thing worse than having bad data is not knowing that you have it. With Bigeye’s data observability platform, if there is an issue with your data or data pipelines you’ll know right away and can get it fixed before the business is impacted. Bigeye let’s data teams measure, improve, and communicate the quality of your data to company stakeholders. With complete API access, a user-friendly interface, and automated yet flexible alerting, you’ve got everything you need to establish and maintain trust in your data. Go to dataengineeringpodcast.com/bigeye today to sign up and start trusting your analyses. Every data project starts with collecting the information that will provide answers to your questions or inputs to your models. The web is the largest trove of information on the planet and Oxylabs helps you unlock its potential. With the Oxylabs scraper APIs you can extract data from even javascript heavy websites. Combined with their residential proxies you can be sure that you’ll have reliable and high quality data whenever you need it. Go to dataengineeringpodcast.com/oxylabs today and use code DEP25 to get your special discount on residential proxies. Your host is Tobias Macey and today I’m interviewing Kevin Kho about Fugue, a library that offers a unified interface for distributed computing that lets users execute Python, pandas, and SQL code on Spark and Dask without rewrites

Interview

Introduction How did you get involved in the area of data management? Can you describe what Fugue is and the story behind it? What are the core goals of the Fugue project? Who are the target users for Fugue and how does that influence the feature priorities and API design? How does Fugue compare to projects such as Modin, etc. for abst

Summary One of the perennial challenges of data analytics is having a consistent set of definitions, along with a flexible and performant API endpoint for querying them. In this episode Artom Keydunov and Pavel Tiunov share their work on Cube.js and the various ways that it is being used in the open source community.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show! Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Datafold helps Data teams gain visibility and confidence in the quality of their analytical data through data profiling, column-level lineage and intelligent anomaly detection. Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Go to dataengineeringpodcast.com/datafold today to start a 30-day trial of Datafold. Your host is Tobias Macey and today I’m interviewing Artyom Keydunov and Pavel Tiunov about Cube.js a framework for building analytics APIs to power your applications and BI dashboards

Interview

Introduction How did you get involved in the area of data management? Can you describe what Cube is and the story behind it? What are the main use cases and platform architectures that you are focused on?

Who are the target personas that will be using and managing Cube.js?

The name comes from the concept of an OLAP cube. Can you discuss the applications of OLAP cubes and their role in the current state of the data ecosystem?

How does the idea of an OLAP cube compare to the recent focus on a dedicated metrics layer?

What are the pieces of a data platform that might be replaced by Cube.js? Can you describe the design and architecture of the Cube platform?

How has the focus and target use case for the Cube platform evolved since you first started working on it?

One of the perpetually hard problems in computer science is cache management. How have you approached that challenge in the pre-aggregation layer of the Cube framework? What is your overarching design philosophy for the API of the Cube system? Can you talk through the workflow of someone building a cube and querying it from a downstream system?

What do the iteration cycles look like as you go from initial proof of concept to a more sophisticated usage of Cube.js

Use R 4, RStudio, Tidyverse, and Shiny to interrogate and analyze your data, and then use the D3 JavaScript library to format and display that data in an elegant, informative, and interactive way. You will learn how to gather data effectively, and also how to understand the philosophy and implementation of each type of chart, so as to be able to represent the results visually. With the popularity of the R language, the art and practice of creating data visualizations is no longer the preserve of mathematicians, statisticians, or cartographers. As technology leaders, we can gather metrics around what we do and use data visualizations to communicate that information. Pro Data Visualization Using R and JavaScript combines the power of the R language with the simplicity and familiarity of JavaScript to display clear and informative data visualizations. Gathering and analyzing empirical data is the key to truly understanding anything. We can track operational metrics to quantify the health of our products in production. We can track quality metrics of our projects, and even use our data to identify bad code. Visualizing this data allows anyone to read our analysis and easily get a deep understanding of the story the data tells. This book makes the R language approachable, and promotes the idea of data gathering and analysis mostly using web interfaces. What You Will Learn Carry out data visualization using R and JavaScript Use RStudio for data visualization Harness Tidyverse data pipelines Apply D3 and R Notebooks towards your data Work with the R Plumber API generator, Shiny, and more Who This Book Is For Programmers and data scientists/analysts who have some prior experience with R and JavaScript.

Build interactive, data-driven websites with the potent combination of open source technologies and web standards, even if you have only basic HTML knowledge. With the latest edition of this popular hands-on guide, you'll tackle dynamic web programming using the most recent versions of today's core technologies: PHP, MySQL, JavaScript, CSS, HTML5, jQuery, and the powerful React library. Web designers will learn how to use these technologies together while picking up valuable web programming practices along the way, including how to optimize websites for mobile devices. You'll put everything together to build a fully functional social networking site suitable for both desktop and mobile browsers. Explore MySQL from database structure to complex queries Use the MySQL PDO extension, PHP's improved MySQL interface Create dynamic PHP web pages that tailor themselves to the user Manage cookies and sessions and maintain a high level of security Enhance JavaScript with the React library Use Ajax calls for background browser-server communication Style your web pages by acquiring CSS skills Implement HTML5 features, including geolocation, audio, video, and the canvas element Reformat your websites into mobile web apps

Integrate D3.js into a React TypeScript project and create a chart component working in harmony with React. This book will show you how utilize D3 with React to bring life to your charts. Seasoned author Elad Elrom will show you how to create simple charts such as line, bar, donut, scatter, histogram and others, and advanced charts such as a world map and force charts. You'll also learn to share the data across your components and charts using React Recoil state management. Then integrate third-party chart libraries that are built on D3 such as Rechart, Visx, Nivo, React-vi, and Victory and in the end deploy your chart as a server or serverless app on popular platforms. React and D3 are two of the most popular frameworks in their respective areas – learn to bring them together and take your storytelling to the next level. What You'll Learn Set up your project with React, TypeScript and D3.js Create simple and advanced D3.js charts Work with complex charts such as world and force charts Integrate D3 data with React state management Improve the performance of your D3 components Deploy as a server or serverless app and debug test Who This Book Is ForReaders that already have basic knowledge of React, HTML, CSS and JavaScript.

Learn to operate at a professional level with HTML, CSS, DOM, JavaScript, PERL and the MySQL database. With plain language explanations and step-by-step examples, you will understand the key facets of web development that today’s employers are looking for. Encapsulating knowledge that is usually found in many books rather than one, this is your one-stop tutorial to becoming a web professional. You will learn how to use the PERL scripting language and the MySQL database to create powerful web applications. Each chapter will become progressively more challenging as you progress through experimentation and ultimately master database-driven web development via the web applications studied in the last chapters. Including practical tips and guidance gleaned from 20+ years of working as a web developer, Thomas Valentine provides you with all the information you need to prosper as a professional database-driven web professional. What You'll Learn Leverage standard web technologies to benefit a database-driven approach Create an effective web development workstation with databases in mind Use the PERL scripting language and the MySQL database effectively Maximize the Apache Web Server Who This Book Is For The primary audience for this book are those who know already know web development basics and web developers who want to master database driven web development. The skills required to understand the concepts put forth are a working knowledge of PERL and basic MySQL.

Use JSON theme files to standardize the look of Power BI dashboards and reports. This book shows how you can create theme files using the Power BI Desktop application to define high-level formatting attributes for dashboards as well as how to tailor detailed formatting specifications for individual dashboard elements in JSON files. Standardize the look of your dashboards and apply formatting consistently over all your reports. The techniques in this book provide you with tight control over the presentation of all aspects of the Power BI dashboards and reports that you create. Power BI theme files use JSON (JavaScript Object Notation) as their structure, so the book includes a brief introduction to JSON as well as how it applies to Power BI themes. The book further includes a complete reference to all the current formatting definitions and JSON structures that are at your disposal for creating JSON theme files. Finally, the book includes dozens of theme files, from the simple to the most complex, that you can adopt and adapt to suit your own requirements. What You Will Learn Produce designer output without manually formatting every individual visual in a Power BI dashboard Standardize presentation for families of dashboard types Switch presentation styles in a couple of clicks Save dozens, or hundreds, of hours laboriously formatting dashboards Define enterprise-wide presentation standards Retroactively apply standard styles to existing dashboards Who This Book Is For Power BI users who want to save time by defining standardized formatting for their dashboards and reports, IT professionals who want to create corporate standards of dashboard presentation, and marketing and communication specialists who want to set organizational standards for dashboard delivery

This book, "Interactive Dashboards and Data Apps with Plotly and Dash", is a practical guide to building dynamic dashboards and applications using the Dash Python framework. It covers creating visualizations, integrating interactive controls, and deploying the apps, all without requiring JavaScript expertise. What this Book will help me do Master creating interactive data dashboards using Dash and Plotly. Understand how to integrate controls such as sliders and dropdowns into apps. Learn to use Plotly Express for visually representing data with ease. Develop capabilities to deploy a fully functional web app for data interaction. Understand how to use multi-page configurations and URLs for advanced apps. Author(s) None Dabbas is a seasoned Python developer with extensive expertise in data visualization and full-stack development. Drawing from real-world experience, None brings a practical approach to teaching, ensuring that learners understand not only how to build applications but why the approach works. Who is it for? This book is ideal for data analysts, engineers, and developers looking to enhance their visualization capabilities. If you are familiar with Python and have basic HTML skills, you will find this book accessible and rewarding. Beginners looking to explore advanced dashboard creation without JavaScript will also appreciate the clear approach.

Master the Shiny web framework—and take your R skills to a whole new level. By letting you move beyond static reports, Shiny helps you create fully interactive web apps for data analyses. Users will be able to jump between datasets, explore different subsets or facets of the data, run models with parameter values of their choosing, customize visualizations, and much more. Hadley Wickham from RStudio shows data scientists, data analysts, statisticians, and scientific researchers with no knowledge of HTML, CSS, or JavaScript how to create rich web apps from R. This in-depth guide provides a learning path that you can follow with confidence, as you go from a Shiny beginner to an expert developer who can write large, complex apps that are maintainable and performant. Get started: Discover how the major pieces of a Shiny app fit together Put Shiny in action: Explore Shiny functionality with a focus on code samples, example apps, and useful techniques Master reactivity: Go deep into the theory and practice of reactive programming and examine reactive graph components Apply best practices: Examine useful techniques for making your Shiny apps work well in production

Tell your story and show it with data, using free and easy-to-learn tools on the web. This introductory book teaches you how to design interactive charts and customized maps for your website, beginning with simple drag-and-drop tools such as Google Sheets, Datawrapper, and Tableau Public. You'll also gradually learn how to edit open source code templates like Chart.js, Highcharts, and Leaflet on GitHub. Hands-On Data Visualization takes you step-by-step through tutorials, real-world examples, and online resources. This practical guide is ideal for students, nonprofit organizations, small business owners, local governments, journalists, academics, and anyone who wants to take data out of spreadsheets and turn it into lively interactive stories. No coding experience is required. Build interactive charts and maps and embed them in your website Understand the principles for designing effective charts and maps Learn key data visualization concepts to help you choose the right tools Convert and transform tabular and spatial data to tell your data story Edit and host Chart.js, Highcharts, and Leaflet map code templates on GitHub Learn how to detect bias in charts and maps produced by others

In "Effortless App Development with Oracle Visual Builder," you will explore how to quickly design, develop, and deploy robust web and mobile applications using Oracle Visual Builder's intuitive drag-and-drop features. This book equips you with the know-how to simplify application development tasks, making it perfect for professionals looking to boost productivity. What this Book will help me do Master the core architecture and features of Oracle Visual Builder to develop real-world applications effectively. Learn to create, manage, and leverage business objects and connect to various SaaS APIs within your applications. Build scalable and secure web and mobile applications using practical examples and clear implementation guidelines. Discover best practices for application lifecycle management, debugging, and troubleshooting VB applications. Extend Oracle and non-Oracle SaaS applications through hands-on knowledge tailored to real-world scenarios. Author(s) None Jain is an experienced developer and technical writer specializing in Oracle Visual Builder and cloud-based application development. With years of hands-on experience building and deploying cloud applications, they bring expertise and a practical approach to education. Their engaging writing style focuses on enabling readers to learn and apply new skills confidently. Who is it for? This book is perfectly suited for developers, UI designers, and IT professionals who want to master Oracle Visual Builder for developing web and mobile applications. If you already have experience with technologies like JavaScript, UI frameworks, and REST APIs, and seek to create intuitive applications using a simplified interface, this book is for you. Whether you're in the early stages of learning VB or looking to refine your skills, this book serves as a valuable guide.

Transform the way your business works with easy-to-build apps. With this updated and expanded second edition, you can build business apps that work with your company's systems and databases, without having to enlist the expertise of costly, professionally trained software developers. In this new edition, business applications expert Tim Leung offers step-by-step guidance on how you can improve all areas of your business. He shows how you can replace manual or paper processes with modern apps that run on phone or tablet devices. For administrative and back-office operations, he covers how to build apps with workflow and dashboard capabilities. To facilitate collaboration with customers and clients, you’ll learn how to build secure web portals with data entry capabilities, including how to customize those portals with code. This hands-on new edition has 10 new chapters—including coverage on model-driven and portal apps, artificial intelligence, building components using the Power Apps Component Framework, using PowerShell for administration, and more—complete with context, explanatory screenshots, and non-technical terminology. What You Will Learn Create offline capable mobile apps and responsive web apps Carry out logic, data access, and data entry through formulas Embellish apps with charting, file handling, photo, barcode, and location features Set up Common Data Service, SharePoint, and SQL data sources Use AI to predict outcomes, recognize images, and analyze sentiment Integrate apps with external web services and automate tasks with Power Automate Build reusable code and canvas components, make customizations with JavaScript Transfer apps and data, and secure, administer, and monitor Power Apps environments Who This Book Is For Beginners and non-developers, and assumes no prior knowledge of Power Apps

The dbt project at tails.com has over 600 models and 66k lines of code. With multiple contributors to a project and varying SQL backgrounds, it's really difficult to maintain consistent readability and comprehension across a codebase like that by hand.

Python has flake8, Javascript has JSLint, but SQL...?

Listen to this talk from Alan Cruickshank to find out whether SQLFluff might help your teams be more productive with SQL.

In this episode, Bryce and Conor talk about each of their favorite data structures. Date Recorded: 2020-11-28 Date Released: 2020-12-04 C++ | Containers OCaml | Containers Java | Collections Python | Collections Kotlin | Collections Scala | Collections Rust | Collections Go | Collections Haskell | Collections TS | Collections Ruby | Collections JS | Collections F# | Collection Types Racket | Data Structures Clojure | Data Structures What do you mean by “cache friendly”? - Björn Fahller - code::dive 2019Alan J. Perlis’ Epigrams on Programmingstd::vectorP1072 basic_string::resize_default_initstd::arraystd::unique_ptr (Array Specialization)P0316 allocate_unique and allocator_deletethurst::allocate_uniqueIntro Song Info Miss You by Sarah Jansen https://soundcloud.com/sarahjansenmusic Creative Commons — Attribution 3.0 Unported — CC BY 3.0 Free Download / Stream: http://bit.ly/l-miss-you Music promoted by Audio Library https://youtu.be/iYYxnasvfx8

Get started building custom Fiori applications for your enterprise. This book teaches you how to design, build, and deploy enterprise-ready, custom Fiori applications in SAP HANA. Tips and tricks collected from projects using Fiori applications (built consuming OData models and REST APIs) and integrating third-party JS libraries are presented. Also included are examples using Fiori templates from different tools such as the SAP Web IDE and the new Visual Studio Code extensions. This book explains the 5 design principles that all Fiori applications are built upon: Role-based, Responsive, Coherent, Simple, and Delightful. The book expands on consuming OData services and REST APIs internal and external to SAP HANA. The Fiori application exercise demonstrates the use of the MVC pattern, JavaScript modularization, reuse of SAP UI5 controls, debugging, and the tools required for a complete scenario. The book closes with an exercise showcasing a finished single page application with multiple views and layouts, navigation between the views, and deployment of the application to AWS. This book is simple enough for entry-level developers getting started in web frameworks but also highlights integration points from the data models being consumed from the application, and shows how the application communicates with back-end services, resulting in a complete front-end custom Fiori application. What You Will Learn Know the 5 Fiori design principles Understand how to consume OData and REST API models Apply the MVC pattern using XML views and the SAP UI5 controls along with controller behavior in JavaScript Debug and deploy the application Who This Book is For Web developers and application leads who have some experience in JavaScript frameworks and web development and understand web protocol communication

Build enterprise-grade microservices in the SAP HANA Advanced Model (XSA). This book explains building scalable APIs in XSA and the benefits of building microservices with SAP HANA XSA. This book covers the cloud foundry (CF) architecture and how SAP HANA XSA follows the model. It begins with the details of the different architectural layers of applications hosted in XSA (specifically, microservices). Everything you need to know is presented, including analyzing requests, modularization, database ingestion, building JSON responses, and scaling your microservices. You will learn to use developmental tools such as the SAP WEB IDE, POSTMAN, and the SAP HANA Cockpit for XSA, including debugging examples on SAP HANA XSA with code snippets showing how microservices can be developed, debugged, scaled, and deployed on SAP HANA XSA. Microservices are divided into security and authentication, request handling, modularization of Node.js, and interaction with the SAP HANA database containers and response formatting. An end-to-end scenario is presented of a Node.js REST API that uses HTTP methods, concluding with deploying an SAP HANA XSA project to a production environment. This book is simple enough to help you implement a Node.js module in order to understand the development of microservices, and complex enough for architects to design their next business-ready solution integrating UAA security, application modularization, and an end-to-end REST API on SAP HANA XSA. What You Will Learn Know the definition and architecture of cloud foundry and its application on SAP HANA XSA Understand REST principles and different HTTP methods Explore microservices (Node.js) development Database interaction from Node (executing SQL statements and stored procedures) Who This Book Is For Architects designing business-ready solutions that integrate UAA security, application modularization, and an end-to-end REST API on SAP HANA XSA

Customize the SAS Stored Process web application to create amazing tools for end users. This book shows you how to use stored processes—SAS programs stored on a server and executed as required by requesting applications. Never before have there been so many ways to turn data into information and build applications with SAS. This book teaches you how to use the web technologies that you frequently see used on impressive websites. By using SAS Stored Processes, you will be able to build applications that exploit CSS, JavaScript, and HTML libraries and enable you to build powerful and impressive web applications using SAS as the backend.While this approach is not common with SAS users, some have had amazing results. People who have SAS skills usually do not have web development skills, and those with web development skills usually do not have SAS skills. Some people have both skills but are unaware of how to connect them with the SAS Stored Process web application. This book shows you how to leverage your skills for success. What You Will Learn Know the benefits of stored processes Write your own tools in SAS Make a stored process generate its own HTML menu Pass data between stored processes Use stored processes to generate pure JavaScript Utilize data generated by SAS Convert a SAS program into a stored process Who This Book Is For SAS programmers looking to improve their existing programming skills to develop web applications, and programming managers who want to make better use of the SAS software they already license

Imagine what you could do if scalability wasn't a problem. With this hands-on guide, you’ll learn how the Cassandra database management system handles hundreds of terabytes of data while remaining highly available across multiple data centers. This third edition—updated for Cassandra 4.0—provides the technical details and practical examples you need to put this database to work in a production environment. Authors Jeff Carpenter and Eben Hewitt demonstrate the advantages of Cassandra’s nonrelational design, with special attention to data modeling. If you’re a developer, DBA, or application architect looking to solve a database scaling issue or future-proof your application, this guide helps you harness Cassandra’s speed and flexibility. Understand Cassandra’s distributed and decentralized structure Use the Cassandra Query Language (CQL) and cqlsh—the CQL shell Create a working data model and compare it with an equivalent relational model Develop sample applications using client drivers for languages including Java, Python, and Node.js Explore cluster topology and learn how nodes exchange data

Learn to create stunning animated and interactive charts using Highcharts and Angular. Use and build on your existing knowledge of HTML, CSS, and JavaScript to develop impressive dashboards that will work in all modern browsers. You will learn how to use Highcharts, call backend services for data, and easily construct real-time data dashboards. You'll also learn how you can club your code with jQuery and Angular. This book provides the best solutions for real-time challenges and covers a wide range of charts including line, area, maps, plot, different types of pie chart, Gauge, heat map, Histogram, stacked bar, scatter plot and 3d charts. After reading this book, you'll be able to export your charts in different formats for project-based learning. Highcharts is one the most useful products worldwide for develop charting on the web, and Angular is well known for speed. Using Highcharts with Angular, developers can build fast, interactive dashboards. Get up to speed using this book today. What You’ll Learn How to develop interactive, animated dashboards How you can implement Highcharts using Angular How to develop a real-time application with the use of WebAPI, Angular, and Highcharts How to create interactive styling themes and colors for a dashboard Who This Book Is For This book is aimed at developers, dev leads, software architects, students or enthusiasts who are already familiar with HTML, CSS, and JavaScript.

Once upon a time, website behavioral data was extracted from the log files of web servers. That data was messy to work with and missing some information that analysts really wanted. This was the OG "server-side" data collection. Then, the JavaScript page tag arrived on the scene, and the data became richer and cleaner and easier to implement. That data was collected by tags firing in the user's browser (which was called "client-side" data collection). But then ad blockers and browser quirks and cross-device behavior turned out to introduce pockets of unreliability into THAT data. And now here we are. What was old is now somewhat new again, and there is a lot to be unpacked with the ins and outs and tradeoffs of client-side vs. server-side data collection. On this episode, Mike Robins from Poplin Data joined the gang to explore the topic from various angles. For complete show notes, including links to items mentioned in this episode and a transcript of the show, visit the show page.