Making a decision is a complex task. Today's guest Dongho Kim discusses how he and his team at Prowler has been building a platform that will be accessible by way of APIs and a set of pre-made scripts for autonomous decision making based on probabilistic modeling, reinforcement learning, and game theory. The aim is so that an AI system could make decisions just as good as humans can.

Learn how to access analytics from SAS Cloud Analytic Services (CAS) using Python and the SAS Viya platform. SAS Viya : The Python Perspective is an introduction to using the Python client on the SAS Viya platform. SAS Viya is a high-performance, fault-tolerant analytics architecture that can be deployed on both public and private cloud infrastructures. While SAS Viya can be used by various SAS applications, it also enables you to access analytic methods from SAS, Python, Lua, and Java, as well as through a REST interface using HTTP or HTTPS. This book focuses on the perspective of SAS Viya from Python. SAS Viya is made up of multiple components. The central piece of this ecosystem is SAS Cloud Analytic Services (CAS). CAS is the cloud-based server that all clients communicate with to run analytical methods. The Python client is used to drive the CAS component directly using objects and constructs that are familiar to Python programmers. Some knowledge of Python would be helpful before using this book; however, there is an appendix that covers the features of Python that are used in the CAS Python client. Knowledge of CAS is not required to use this book. However, you will need to have a CAS server set up and running to execute the examples in this book. With this book, you will learn how to: Install the required components for accessing CAS from Python Connect to CAS, load data, and run simple analyses Work with CAS using APIs familiar to Python users Grasp general CAS workflows and advanced features of the CAS Python client SAS Viya : The Python Perspective covers topics that will be useful to beginners as well as experienced CAS users. It includes examples from creating connections to CAS all the way to simple statistics and machine learning, but it is also useful as a desktop reference.

Summary

One of the critical components for modern data infrastructure is a scalable and reliable messaging system. Publish-subscribe systems have been popular for many years, and recently stream oriented systems such as Kafka have been rising in prominence. This week Rajan Dhabalia and Matteo Merli discuss the work they have done on Pulsar, which supports both options, in addition to being globally scalable and fast. They explain how Pulsar is architected, how to scale it, and how it fits into your existing infrastructure.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at dataengineeringpodcast.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. You can help support the show by checking out the Patreon page which is linked from the site. To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers A few announcements:

There is still time to register for the O’Reilly Strata Conference in San Jose, CA March 5th-8th. Use the link dataengineeringpodcast.com/strata-san-jose to register and save 20% The O’Reilly AI Conference is also coming up. Happening April 29th to the 30th in New York it will give you a solid understanding of the latest breakthroughs and best practices in AI for business. Go to dataengineeringpodcast.com/aicon-new-york to register and save 20% If you work with data or want to learn more about how the projects you have heard about on the show get used in the real world then join me at the Open Data Science Conference in Boston from May 1st through the 4th. It has become one of the largest events for data scientists, data engineers, and data driven businesses to get together and learn how to be more effective. To save 60% off your tickets go to dataengineeringpodcast.com/odsc-east-2018 and register.

Your host is Tobias Macey and today I’m interviewing Rajan Dhabalia and Matteo Merli about Pulsar, a distributed open source pub-sub messaging system

Interview

Introduction How did you get involved in the area of data management? Can you start by explaining what Pulsar is and what the original inspiration for the project was? What have been some of the most challenging aspects of building and promoting Pulsar? For someone who wants to run Pulsar, what are the infrastructure and network requirements that they should be considering and what is involved in deploying the various components? What are the scaling factors for Pulsar and what aspects of deployment and administration should users pay special attention to? What projects or services do you consider to be competitors to Pulsar and what makes it stand out in comparison? The documentation mentions that there is an API layer that provides drop-in compatibility with Kafka. Does that extend to also supporting some of the plugins that have developed on top of Kafka? One of the popular aspects of Kafka is the persistence of the message log, so I’m curious how Pulsar manages long-term storage and reprocessing of messages that have already been acknowledged? When is Pulsar the wrong tool to use? What are some of the improvements or new features that you have planned for the future of Pulsar?

Contact Info

Matteo

merlimat on GitHub @merlimat on Twitter

Rajan

@dhabaliaraj on Twitter rhabalia on GitHub

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

Pulsar Publish-Subscribe Yahoo Streamlio ActiveMQ Kafka Bookkeeper SLA (Service Level Agreement) Write-Ahead Log Ansible Zookeeper Pulsar Deployme

Abstract This IBM® Redbooks® publication is intended for IBM CICS® system programmers and IBM Z architects. It describes how to deploy and manage Java EE 7 web-based applications in an IBM CICS Liberty JVM server and access data on IBM Db2® for IBM z/OS® and IBM MQ for z/OS sub systems. In this book, we describe the key steps to create and install a Liberty JVM server within a CICS region. We then describe how to best use the different deployment techniques for Java EE applications and the specific considerations when deploying applications that use JDBC, JMS, and the new CICS link to Liberty API. Finally, we describe how to secure web applications in CICS Liberty, including transport-level security and request authentication and authorization by using IBM RACF® and LDAP registries. Information is also provided about how to build a high availability infrastructure and how to use the logging and monitoring functions that are available in the CICS Liberty environment. This book is based on IBM CICS Transaction Server (CICS TS) V5.4 that uses the embedded IBM WebSphere® Application Server Liberty technology. It is also applicable to CICS TS V5.3 with the fixes for the continuous delivery APAR PI77502 applied. Sample applications are used throughout this publication and are freely available for download from the IBM CICSDev GitHub organization along with detailed deployment instructions.

Abstract This IBM® Redbooks® publication covers the background and implementation of the IBM CICS® asynchronous API, which is a simple, accessible API that is designed to enable CICS application developers to create efficient asynchronous programs in all CICS-supported languages. Using the API, application developers can eliminate the overhead that is involved in coding and managing homegrown asynchronous solutions, instead using a set of CICS-supported API commands to underpin CICS applications, which are more responsive and robust than ever. Initially, the book reviews the history and motivations of asynchronous processing in computing and the benefits involved when calling external services. It then introduces the asynchronous API itself and its commands. It also provides a range of scenarios, including sample code, that cover everything from the basics of making an asynchronous request to updating existing synchronous program calls, with the goal of illustrating how to harness the CICS asynchronous API to solve real business problems. Later chapters take a deeper dive into the capabilities of the asynchronous API for advanced use cases. Beyond application development, CICS provides a complete solution for system programmers to manage and monitor asynchronous business logic. Thus, the final chapters of this book cover enhancements to CICS monitoring, statistics, trace, and dumps. Using supporting CICS tooling, system programmers have greater insight than ever, with improved transaction tracking capabilities and CICS policies to provide maximum control and optimization of asynchronous processing in CICS environments.

If you're ready to untap the potential of data analytics in the cloud, 'Learning Google BigQuery' will take you from understanding foundational concepts to mastering advanced techniques of this powerful platform. Through hands-on examples, you'll learn how to query and analyze massive datasets efficiently, develop custom applications, and integrate your results seamlessly with other tools. What this Book will help me do Understand the fundamentals of Google Cloud Platform and how BigQuery operates within it. Migrate enterprise-scale data seamlessly into BigQuery for further analytics. Master SQL techniques for querying large-scale datasets in BigQuery. Enable real-time data analytics and visualization with tools like Tableau and Python. Learn to create dynamic datasets, manage partition tables and use BigQuery APIs effectively. Author(s) None Berlyant, None Haridass, and None Brown are specialists with years of experience in data science, big data platforms, and cloud technologies. They bring their expertise in data analytics and teaching to make advanced concepts accessible. Their hands-on approach and real-world examples ensure readers can directly apply the skills they acquire to practical scenarios. Who is it for? This book is tailored for developers, analysts, and data scientists eager to leverage cloud-based tools for handling and analyzing large-scale datasets. If you seek to gain hands-on proficiency in working with BigQuery or want to enhance your organization's data capabilities, this book is a fit. No prior BigQuery knowledge is needed, just a willingness to learn.

"R Programming By Example" serves as an engaging and practical introduction to the R programming language for data analysis and visualization. Through step-by-step examples and comprehensive guides, this book builds your understanding from foundational knowledge to advanced applications in R. You will master programming practices while analyzing real-world scenarios. What this Book will help me do Gain proficiency in leveraging R's versatile features and package ecosystem to tackle data analysis tasks. Learn to create and customize high-quality visualizations, including 3D graphs, for enhanced data presentation. Understand statistical modeling and descriptive analysis techniques for extracting insights from data. Discover efficient programming strategies in R, including code profiling and parallelization, to optimize performance. Acquire the skills to interface R with databases and RESTful APIs for robust data integration. Author(s) The authors, None Trejo Navarro and Omar Trejo Navarro, bring a wealth of experience in statistical programming and data analysis. Having worked extensively with R, they focus on practical and results-driven teaching. They have a passion for making complex topics accessible to learners. Who is it for? This book is aimed at aspiring data scientists, statisticians, or analysts looking to learn R. It is particularly suitable for readers familiar with basic programming concepts and who wish to apply R in practical scenarios. Whether you're analyzing data, building models, or creating visualizations, this book will guide you effectively. If you're eager to advance your R skills through hands-on projects, this is for you.

D3.js in Action, Second Edition is completely revised and updated for D3 v4 and ES6. It's a practical tutorial for creating interactive graphics and data-driven applications using D3. About the Technology Visualizing complex data is hard. Visualizing complex data on the web is darn near impossible without D3.js. D3 is a JavaScript library that provides a simple but powerful data visualization API over HTML, CSS, and SVG. Start with a structure, dataset, or algorithm; mix in D3; and you can programmatically generate static, animated, or interactive images that scale to any screen or browser. It's easy, and after a little practice, you'll be blown away by how beautiful your results can be! About the Book D3.js in Action, Second Edition is a completely updated revision of Manning's bestselling guide to data visualization with D3. You'll explore dozens of real-world examples in full-color, including force and network diagrams, workflow illustrations, geospatial constructions, and more! Along the way, you'll pick up best practices for building interactive graphics, animations, and live data representations. You'll also step through a fully interactive application created with D3 and React. What's Inside Rich full-color diagrams and illustrations Updated for D3 v4 and ES6 Reusable layouts and components Geospatial data visualizations Mixed-mode rendering About the Reader Suitable for web developers with HTML, CSS, and JavaScript skills. No specialized data science skills required. About the Author Elijah Meeks is a senior data visualization engineer at Netflix. Quotes From basic to complex, this book gives you the tools to create beautiful data visualizations. - Claudio Rodriguez, Cox Media Group The best reference for one of the most useful DataViz tools. - Jonathan Rioux, TD Insurance From toy examples to techniques for real projects. Shows how all the pieces fit together. - Scott McKissock, USAID A clever way to immerse yourself in the D3.js world. - Felipe Vildoso Castillo, University of Chile

Learn to use the ArcGIS API 4 for JavaScript to build custom web mapping applications. This book teaches you to easily create interactive displays of geographic information that you can use to tell stories and answer questions. Version 4 of the ArcGIS API for JavaScript introduces new patterns and fundamental concepts, including 3D mapping capabilities. You will learn the fundamentals of using the API in order to get the most out of it. Covering key concepts and how different components work together, you will also learn how to take advantage of the Widget framework built into the API to build your own reusable widgets for your own ArcGIS JSAPI applications. Including a series of samples you can use to leverage the API for your own applications, Introducing ArcGIS API 4 for JavaScript helps you take your existing knowledge of JavaScript to a new level, and add new features to your app libraries. What You'll Learn Create both 2D and 3D custom web mapping applications Work with popups and custom widgets Leverage the ArcGIS platform in your applications Utilize custom visualizations Who This Book Is For Developers who need to learn the ArcGIS JSAPI for work or school. Those with some JavaScript experience; GIS or mapping experience is not required.

Master the basics of XML as well as the namespaces and objects you need to know in order to work efficiently with XML. You’ll learn extensive support for XML in everything from data access to configuration, from raw parsing to code documentation. You will see clear, practical examples that illustrate best practices in implementing XML APIs and services as part of your C#-based Windows 10 applications. Beginning XML with C# 7 is completely revised to cover the XML features of .NET Framework 4.7 using C# 7 programming language. In this update, you’ll discover the tight integration of XML with ADO.NET and LINQ as well as additional .NET support for today’s RESTful web services and Web API. Written by a Microsoft Most Valuable Professional and developer, this book demystifies everything to do with XML and C# 7. What You Will Learn: Discover how XML works with the .NET Framework Read, write, access, validate, and manipulate XML documents Transform XML with XSLT Use XML serialization and web services Combine XML in ADO.NET and SQL Server Create services using Windows Communication Foundation Work with LINQ Use XML with Web API and more Who This Book Is For : Those with experience in C# and .NET new to the nuances of using XML. Some XML experience is helpful.

Create and run a real-time, highly-available, and high-redundancy version of the world's most popular open-source database, MySQL. You will understand the advantages and disadvantages of the MySQL NDB Cluster solution, and when MySQL NDB Cluster is the right choice. Pro MySQL NDB Cluster walks you through the full lifecycle of a MySQL Cluster installation: starting with the installation and initial configuration, moving through online configuration and schema changes, and completing with online upgrades. Along the way, you will learn to monitor your cluster, make decisions about schema design, implement geographic replication, troubleshoot and optimize performance, and much more. This book covers the many programming APIs that are supported by MySQL NDB Cluster. There's also robust coverage of connecting to MySQL NDB Cluster from Java, SQL, memcached, and even from C++. From any of these languages, you'll be able to connect and store and retrieve data as your applications demand. The book: Covers MySQL NDB Cluster concepts and architecture Takes you through the MySQL NDB Cluster lifecycle from installation to upgrades Guides you through DBA and Developer decisions when working with MySQL NDB Cluster What You'll Learn Understand the shared-nothing architecture behind MySQL NDB Cluster Plan, install, and configure a MySQL NDB Cluster environment Perform everyday tasks such as backing up, restoring, and upgrading Develop applications from Java, memcached, C++, and SQL Troubleshoot and resolve application performance problems Master enterprise-level features such the MySQL NDB Cluster Manager Who This Book Is For Database administrators and developers who are looking into deploying MySQL NDB Cluster, or who already have a cluster in production and want to increase their knowledge and ability to handle routine administrative tasks and troubleshooting. The book also is for those developers wanting to employ MySQL NDB Cluster as their chosen storage engine from Java, memcached, and C++ applications.

Mastering ArcGIS Enterprise Administration teaches you how to install, configure, and manage ArcGIS Enterprise, guiding you through publishing, optimizing, and securing GIS services for your organizational needs. With this book, you will build a robust GIS infrastructure and gain solutions to common administration challenges. What this Book will help me do Install and configure ArcGIS Enterprise, including its enterprise geodatabase and web services components. Utilize administrative tools like user interfaces, REST API, and Python for system management. Securely publish and manage GIS services, optimizing performance for end users. Apply best practices for securing GIS resources and enabling efficient access. Troubleshoot issues and implement automation to maintain system reliability. Author(s) None Cooper is a seasoned GIS professional with extensive experience in deploying and managing ArcGIS systems for various organizational workflows. With a meticulous approach, they have assisted numerous teams in leveraging GIS technologies to accomplish their objectives. None's commitment to teaching is reflected in this comprehensive guide, which simplifies complex topics to foster effective learning. Who is it for? This book is perfect for GIS analysts, managers, and administrators who aim to learn ArcGIS Enterprise installation and administration. If you already work with ArcGIS or similar GIS platforms, this book can enhance your understanding. It's suitable for those with a basic understanding of geospatial concepts looking to specialize in Enterprise administration. Whether you work as a GIS engineer or database administrator, this guide helps you establish a secure and functional ArcGIS infrastructure.

GeoServer Beginner's Guide is your starting point for mastering GeoServer, a powerful open-source tool for serving geospatial data online. This book makes it easy to create, manage, and share maps and geographic information systems (GIS) even if you don't have advanced technical experience. With step-by-step guidance, you'll leverage GeoServer's full capabilities. What this Book will help me do Configure and install GeoServer to publish your geospatial data quickly and efficiently, making it available online. Create interactive and visually appealing maps by styling points, lines, and polygons using GeoServer's tools. Learn how to connect GeoServer with back-end databases like PostGIS for advanced data management and functionalities. Optimize GeoServer for performance and prepare for production-ready deployments, ensuring a seamless user experience. Use GeoServer's REST API to automate tasks and integrate with other applications for enhanced workflows. Author(s) None Iacovella has extensive experience in GIS and web technologies, specializing in open-source solutions. With a passion for teaching, he has authored several books and tutorials that make technical topics accessible to developers and enthusiasts. His approachable writing style ensures that complex concepts are broken down into understandable steps. Who is it for? This book is designed for web developers and technical users who are new to GeoServer or open-source GIS tools. Ideal readers are those with basic server-side scripting knowledge and an interest in publishing dynamic, interactive maps. If you're looking to enhance your website with geospatial data, this guide will provide the step-by-step instructions you need.

Every enterprise application creates data, whether it’s log messages, metrics, user activity, outgoing messages, or something else. And how to move all of this data becomes nearly as important as the data itself. If you’re an application architect, developer, or production engineer new to Apache Kafka, this practical guide shows you how to use this open source streaming platform to handle real-time data feeds. Engineers from Confluent and LinkedIn who are responsible for developing Kafka explain how to deploy production Kafka clusters, write reliable event-driven microservices, and build scalable stream-processing applications with this platform. Through detailed examples, you’ll learn Kafka’s design principles, reliability guarantees, key APIs, and architecture details, including the replication protocol, the controller, and the storage layer. Understand publish-subscribe messaging and how it fits in the big data ecosystem. Explore Kafka producers and consumers for writing and reading messages Understand Kafka patterns and use-case requirements to ensure reliable data delivery Get best practices for building data pipelines and applications with Kafka Manage Kafka in production, and learn to perform monitoring, tuning, and maintenance tasks Learn the most critical metrics among Kafka’s operational measurements Explore how Kafka’s stream delivery capabilities make it a perfect source for stream processing systems

"Learning Spark SQL" takes you from data exploration to designing scalable applications with Apache Spark SQL. Through hands-on examples, you will comprehend real-world use cases and gain practical skills crucial for working with Spark SQL APIs, data frames, streaming data, and optimizing Spark applications. What this Book will help me do Understand the principles of Spark SQL and its APIs for building scalable distributed applications. Gain hands-on experience performing data wrangling and visualization using Spark SQL and real-world datasets. Learn how to design and optimize applications for performance and scalability with Spark SQL. Develop the skills to integrate Spark SQL with other frameworks like Apache Kafka for streaming analytics. Master the techniques required to architect machine learning and deep learning solutions using Spark SQL. Author(s) None Sarkar is an experienced technologist and trainer specializing in big data, streaming analytics, and scalable architectures using Apache Spark. With years of practical experience in implementing Spark solutions, Sarkar draws from real-world projects to provide readers with valuable insights. Sarkar's approachable and detailed writing style ensures readers grasp both the theory and the practice of Spark SQL. Who is it for? This book is ideal for software developers, data engineers, and architects aspiring to harness Apache Spark for robust, scalable applications. It suits readers with some SQL querying experience and a basic knowledge of programming in languages like Scala, Java, or Python. Whether you're a Spark newcomer or advancing your capabilities in scalable data processing, this resource will accelerate your learning journey.

Abstract This IBM® Redbooks® publication is based on the Presentations Guide of the course Essentials of Cloud Application Development on IBM Bluemix that was developed by the IBM Redbooks team in partnership with IBM Skills Academy Program. This course is designed to teach university students the basic skills that are required to develop, deploy, and test cloud-based applications that use the IBM Bluemix® cloud services. The primary target audience for this course is university students in undergraduate computer science and computer engineer programs with no previous experience working in cloud environments. However, anyone new to cloud computing can also benefit from this course. After completing this course, you should be able to accomplish the following tasks: Define cloud computing Describe the factors that lead to the adoption of cloud computing Describe the choices that developers have when creating cloud applications Describe infrastructure as a service, platform as a service, and software as a service Describe IBM Bluemix and its architecture Identify the runtimes and services that IBM Bluemix offers Describe IBM Bluemix infrastructure types Create an application in IBM Bluemix Describe the IBM Bluemix dashboard, catalog, and documentation features Explain how the application route is used to test an application from the browser Create services in IBM Bluemix Describe how to bind services to an application in IBM Bluemix Describe the environment variables that are used with IBM Bluemix services Explain what are IBM Bluemix organizations, domains, spaces, and users Describe how to create an IBM SDK for Node.js application that runs on IBM Bluemix Explain how to manage your IBM Bluemix account with the Cloud Foundry CLI Describe how to set up and use the IBM Bluemix plug-in for Eclipse Describe the role of Node.js for server-side scripting Describe IBM Bluemix DevOps Services and the capabilities of IBM DevOps Services Identify the Web IDE features in IBM Bluemix DevOps Describe how to connect a Git repository client to Bluemix DevOps Services project Explain the pipeline build and deploy processes that IBM Bluemix DevOps Services use Describe how IBM Bluemix DevOps Services integrate with the IBM Bluemix cloud Describe the agile planning tools in IBM Bluemix Describe the characteristics of REST APIs Explain the advantages of the JSON data format Describe an example of REST APIs using Watson Describe the main types of data services in IBM Bluemix Describe the benefits of IBM Cloudant® Explain how Cloudant databases and documents are accessed from IBM Bluemix Describe how to use REST APIs to interact with Cloudant database Describe Bluemix mobile backend as a service (MBaaS) and the MBaaS architecture Describe the Push Notifications service Describe the App ID service Describe the Kinetise service Describe how to create Bluemix Mobile applications by using MobileFirst Services Starter Boilerplate The workshop materials were created in June 2017. Therefore, all IBM Bluemix features that are described in this Presentations Guide and IBM Bluemix user interfaces that are used in the examples are current as of June 2017.

Summary

Building a data pipeline that is reliable and flexible is a difficult task, especially when you have a small team. Astronomer is a platform that lets you skip straight to processing your valuable business data. Ry Walker, the CEO of Astronomer, explains how the company got started, how the platform works, and their commitment to open source.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at www.dataengineeringpodcast.com/linode?utm_source=rss&utm_medium=rss and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. You can help support the show by checking out the Patreon page which is linked from the site. To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers This is your host Tobias Macey and today I’m interviewing Ry Walker, CEO of Astronomer, the platform for data engineering.

Interview

Introduction How did you first get involved in the area of data management? What is Astronomer and how did it get started? Regulatory challenges of processing other people’s data What does your data pipelining architecture look like? What are the most challenging aspects of building a general purpose data management environment? What are some of the most significant sources of technical debt in your platform? Can you share some of the failures that you have encountered while architecting or building your platform and company and how you overcame them? There are certain areas of the overall data engineering workflow that are well defined and have numerous tools to choose from. What are some of the unsolved problems in data management? What are some of the most interesting or unexpected uses of your platform that you are aware of?

Contact Information

Email @rywalker on Twitter

Links

Astronomer Kiss Metrics Segment Marketing tools chart Clickstream HIPAA FERPA PCI Mesos Mesos DC/OS Airflow SSIS Marathon Prometheus Grafana Terraform Kafka Spark ELK Stack React GraphQL PostGreSQL MongoDB Ceph Druid Aries Vault Adapter Pattern Docker Kinesis API Gateway Kong AWS Lambda Flink Redshift NOAA Informatica SnapLogic Meteor

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support Data Engineering Podcast

Delve into mastering big data processing with 'Apache Spark 2.x for Java Developers.' This book provides a practical guide to implementing Apache Spark using the Java APIs, offering a unique opportunity for Java developers to leverage Spark's powerful framework without transitioning to Scala. What this Book will help me do Learn how to process data from formats like XML, JSON, CSV using Spark Core. Implement real-time analytics using Spark Streaming and third-party tools like Kafka. Understand data querying with Spark SQL and master SQL schema processing. Apply machine learning techniques with Spark MLlib to real-world scenarios. Explore graph processing and analytics using Spark GraphX. Author(s) None Kumar and None Gulati, experienced professionals in Java development and big data, bring their wealth of practical experience and passion for teaching to this book. With a clear and concise writing style, they aim to simplify Spark for Java developers, making big data approachable. Who is it for? This book is perfect for Java developers who are eager to expand their skillset into big data processing with Apache Spark. Whether you are a seasoned Spark user or first diving into big data concepts, this book meets you at your level. With practical examples and straightforward explanations, you can unlock the potential of Spark in real-world scenarios.

This episode collects interviews from my recent trip to Microsoft Build where I had the opportunity to speak with Dharma Shukla and Syam Nair about the recently announced CosmosDB. CosmosDB is a globally consistent, distributed datastore that supports all the popular persistent storage formats (relational, key/value pair, document database, and graph) under a single streamlined API. The system provides tunable consistency, allowing the user to make choices about how consistency trade-offs are managed under the hood, if a consumer wants to go beyond the selected defaults.

In many organizations today, businesspeople are busy requesting unified views of data stored across multiple sources within their organizations. But integrating multiple data types from multiple data stores is a complex, error-prone, and time-consuming process of cobbling everything together manually. This concise book examines how multi-model databases can help you integrate data storage and access across your organization in a seamless and elegant way. Author Pete Aven and Diane Burley from MarkLogic explain how this latest evolution in data management naturally accepts heterogeneous data, enabling you to eventually phase out technical data silos. Through several case studies, you’ll discover how organizations use multi-model databases to reduce complexity, save money, take advantage of opportunities, lessen risk, and shorten time to value. Get unified views across disparate data models and formats within a single database Learn how multi-model databases leverage the inherent structure of the data being stored Load and use unstructured and semi-structured data (such as documents and text) as is Provide agility in data access and delivery through APIs, interfaces, and indexes Learn how to scale a multi-model database, and provide ACID capabilities and security Examine how a multi-model database would fit into your existing architecture