Dive into the world of distributed data processing with the 'Apache Hadoop 3 Quick Start Guide.' This comprehensive resource equips you with the knowledge needed to handle large datasets effectively using Apache Hadoop. Learn how to set up and configure Hadoop, work with its core components, and explore its powerful ecosystem tools. What this Book will help me do Understand the fundamental concepts of Apache Hadoop, including HDFS, MapReduce, and YARN, and use them to store and process large datasets. Set up and configure Hadoop 3 in both developer and production environments to suit various deployment needs. Gain hands-on experience with Hadoop ecosystem tools like Hive, Kafka, and Spark to enhance your big data processing capabilities. Learn to manage, monitor, and troubleshoot Hadoop clusters efficiently to ensure smooth operations. Analyze real-time streaming data with tools like Apache Storm and perform advanced data analytics using Apache Spark. Author(s) The author of this guide, Vijay Karambelkar, brings years of experience working with big data technologies and Apache Hadoop in real-world applications. With a passion for teaching and simplifying complex topics, Vijay has compiled his expertise to help learners confidently approach Hadoop 3. His detailed, example-driven approach makes this book a practical resource for aspiring data professionals. Who is it for? This book is ideal for software developers, data engineers, and IT professionals who aspire to dive into the field of big data. If you're new to Apache Hadoop or looking to upgrade your skills to include version 3, this guide is for you. A basic understanding of Java programming is recommended to make the most of the topics covered. Embark on this journey to enhance your career in data-intensive industries.

This expert guide, "Mastering Apache Cassandra 3.x," is designed for individuals looking to achieve scalable and fault-tolerant database deployment using Apache Cassandra. From mastering the foundational components of Cassandra architecture to advanced topics like clustering and analytics integration with Apache Spark, this book equips readers with practical, actionable skills. What this Book will help me do Understand and deploy Apache Cassandra clusters for fault-tolerant and scalable databases. Use advanced features of CQL3 to streamline database queries and operations. Optimize and configure Cassandra nodes to improve performance for demanding applications. Monitor and manage Cassandra clusters effectively using best practices. Combine Cassandra with Apache Spark to build robust data analytics pipelines. Author(s) None Ploetz and None Malepati are experienced technologists and software professionals with extensive expertise in distributed database systems and big data algorithms. They've combined their industry knowledge and teaching backgrounds to create accessible and practical guides for learners worldwide. Their collaborative work is focused on demystifying complex systems for maximum learning impact. Who is it for? This book is ideal for database administrators, software developers, and big data specialists seeking to expand their skill set into scalable data storage using Cassandra. Readers should have a basic understanding of database concepts and some programming experience. If you're looking to design robust databases optimized for modern big data use-cases, this book will serve as a valuable resource.

Use data analytics to drive innovation and value throughout your network infrastructure Network and IT professionals capture immense amounts of data from their networks. Buried in this data are multiple opportunities to solve and avoid problems, strengthen security, and improve network performance. To achieve these goals, IT networking experts need a solid understanding of data science, and data scientists need a firm grasp of modern networking concepts. Data Analytics for IT Networks fills these knowledge gaps, allowing both groups to drive unprecedented value from telemetry, event analytics, network infrastructure metadata, and other network data sources. Drawing on his pioneering experience applying data science to large-scale Cisco networks, John Garrett introduces the specific data science methodologies and algorithms network and IT professionals need, and helps data scientists understand contemporary network technologies, applications, and data sources. After establishing this shared understanding, Garrett shows how to uncover innovative use cases that integrate data science algorithms with network data. He concludes with several hands-on, Python-based case studies reflecting Cisco Customer Experience (CX) engineers’ supporting its largest customers. These are designed to serve as templates for developing custom solutions ranging from advanced troubleshooting to service assurance. Understand the data analytics landscape and its opportunities in Networking See how elements of an analytics solution come together in the practical use cases Explore and access network data sources, and choose the right data for your problem Innovate more successfully by understanding mental models and cognitive biases Walk through common analytics use cases from many industries, and adapt them to your environment Uncover new data science use cases for optimizing large networks Master proven algorithms, models, and methodologies for solving network problems Adapt use cases built with traditional statistical methods Use data science to improve network infrastructure analysisAnalyze control and data planes with greater sophistication Fully leverage your existing Cisco tools to collect, analyze, and visualize data

In this podcast, John Busby(@johnmbusby), Chief Analytics Officer @CenterfieldUSA, talks about his journey leading the data analytics practice of a digital marketing agency. He sheds light on some methodologies for building a sound data science practice. He sheds light on the future of digital marketing and shared some big opportunities ripe for disruption in the digital space.

Timeline: 0:28 John's journey. 4:26 Introduction to Centerfield. 6:00 John's role. 6:50 Designing a common platform for customers. 9:15 Analytics in Amazon. 11:02 Data science and marketing. 18:02 Importance of understanding the product for marketing. 21:44 AI in the marketing business. 25:26 Making sense of customer behavior. 27:50 End to end consumer behavior. 31:05 Editing and calibrating KPIs. 32:53 Creating an inside driven organization. 35:35 Recipe for a successful chief analytic officer. 37:46 On data bias. 39:12 Hiring the right people. 41:33 Big opportunities in digital marketing. 44:15 Future of digital marketing. 45:27 John's recipe for success. 48:52 John's favorite reads. 50:35 Key takeaways.

John's Recommended Read: Secrets of Professional Tournament Poker (D&B Poker) by Jonathan Little amzn.to/2MNKjN3

Podcast Link: https://futureofdata.org/data-today-shaping-digital-marketing-of-tomorrow-johnmbusby-centerfieldusa/

John's BIO: John Busby serves as Centerfield’s Chief Analytics Officer. A seasoned digital marketing executive, John leads the company’s data science, analytics and insights teams. Before joining Centerfield, John was Head of Analytics for Amazon’s grocery delivery service and responsible for business intelligence, data science and automated reporting. Prior to Amazon, John was Senior Vice President of Analytics and Marketing at Marchex. John began his career in product management for InfoSpace, Go2net and IQ Chart. He holds a Bachelor of Science from Northwestern University. Outside of work, John coaches youth hockey, and enjoys sports, poker and hanging out with his wife and two children.

About #Podcast:

FutureOfData podcast is a conversation starter to bring leaders, influencers and lead practitioners to come on show and discuss their journey in creating the data driven future.

Wanna Join? If you or any you know wants to join in, Register your interest by mailing us @ [email protected]

Want to sponsor? Email us @ [email protected]

Keywords: FutureOfData,

DataAnalytics,

Leadership,

Futurist,

Podcast,

BigData,

Strategy

Send us a text Making Data Simple host Al Martin has a chance to discuss all thing data with Laura Ellis, also known as Little Miss Data. Laura is an analytics architect for IBM Cloud as well as a frequent blogger. Together, they talk about how critical it is to understand your data in order create specific calls to action, and what it means to build a data democracy. Show Notes 00:00 - Follow @IBMAnalyticsSupport on Twitter. 00:22 - Check out our YouTube channel. We're posting full episodes weekly. 00:24 - Connect with Al Martin on LinkedIn and Twitter. 01:20 - Check out littlemissdata.com. 01:22 - Connect with Laura Ellis on Twitter, Instagram, and LinkedIn. 02:20 - Curious to know more about analytics architecture? Check out this IBM article on the topic. 03:52 - Check out the Little Miss Data article Al referenced here. 04:45 - Learn more about Data Democracy here in Laura's blog post. 05:31 - Understand more about the importance of data for your business in this article. 09:11 - Find out more about the challenges of being a data scientist here. 12:45 - Working with good quality data is crucial. Check out this article for more details. 16:12 - Simple data can provide the most effective returns. Learn more here. 21:15 - Choosing the right, supportive environment for your data science journey will make sure you don't get burnt out. This article examines your options. 21:35 - Data is a fundamental step when working with AI. But do you know the difference between data analytics, AI and machine learning? This Forbes article walks you through it. 22:42 - Need to brush up on what a data dashboard is? Learn more here. Want to be featured as a guest on Making Data Simple? Reach out to us at [email protected] and tell us why you should be next. The Making Data Simple Podcast is hosted by Al Martin, WW VP Technical Sales, IBM, where we explore trending technologies, business innovation, and leadership ... while keeping it simple & fun.

In this episode, Wayne Eckerson and Rich Galan discuss the obstacles to delivering timely analysis, the problems that large volumes of data create, solutions to those issues, and where BI is headed in the near future. Rich is a veteran data analytics leader with 20 years of experience in a variety of data-driven organizations.

Explore the latest Python tools and techniques to help you tackle the world of data acquisition and analysis. You'll review scientific computing with NumPy, visualization with matplotlib, and machine learning with scikit-learn. This revision is fully updated with new content on social media data analysis, image analysis with OpenCV, and deep learning libraries. Each chapter includes multiple examples demonstrating how to work with each library. At its heart lies the coverage of pandas, for high-performance, easy-to-use data structures and tools for data manipulation Author Fabio Nelli expertly demonstrates using Python for data processing, management, and information retrieval. Later chapters apply what you've learned to handwriting recognition and extending graphical capabilities with the JavaScript D3 library. Whether you are dealing with sales data, investment data, medical data, web page usage, or other data sets, Python Data Analytics, Second Edition is an invaluable reference with its examples of storing, accessing, and analyzing data. What You'll Learn Understand the core concepts of data analysis and the Python ecosystem Go in depth with pandas for reading, writing, and processing data Use tools and techniques for data visualization and image analysis Examine popular deep learning libraries Keras, Theano,TensorFlow, and PyTorch Who This Book Is For Experienced Python developers who need to learn about Pythonic tools for data analysis

In this podcast Jason Carmel(@defenestrate99) Chief Data Officer @ POSSIBLE talks about his journey leading data analytics practice of digital marketing agency. He sheds light on some methodologies for building a sound data science practice. He sheds light on using data science chops for doing some good while creating traditional value. He shared his perspective on keeping team-high on creativity to keep creating innovative solutions. This is a great podcast for anyone looking to understanding the digital marketing landscape and how to create a sound data science practice.

Timelines: 0:29 Jason's journey. 6:40 Advantage of having a legal background for a data scientist. 9:15 Understanding emotions based on data. 13:54 The empathy model. 14:53 From idea to inception to execution. 23:40 The role of digital agencies. 30:20 Measuring the right amount of data. 32:40 Management in a creative agency. 34:40 Leadership qualities that promote creativity. 38:14 Leader's playbook in a digital agency. 40:50 Qualities of a great data science team in the digital agency. 44:30 Leadership's role in data creativity. 47:00 Opportunites as a data scientist in the digital agency. 49:18 Future of data in digital media. 51:38 Jason's success mantra. 53:30 Jason's favorite reads. 57:11 Key takeaways.

Jason's Recommended Read: Trendology: Building an Advantage through Data-Driven Real-Time Marketing by Chris Kerns amzn.to/2zMhYkV Venomous: How Earth's Deadliest Creatures Mastered Biochemistry by Christie Wilcox amzn.to/2LhqI76

Podcast Link: https://futureofdata.org/jason-carmel-defenestrate99-possible-leading-analytics-data-digital-marketing/

Jason's BIO: Jason Carmel is Chief Data Officer at Possible. With nearly 20 years of digital data and marketing experience, Jason has worked with clients such as Coca Cola, Ford, and Microsoft to evolve digital experiences based on real-time feedback and behavioral data. Jason manages a global team of 100 digital analysts across POSSIBLE, a digital advertising agency that uses traditional and unconventional data sets and models to help brands connect more effectively with their customers.

Of particular interest is Jason’s work using data and machine learning to define and understand the emotional components of human conversation. Jason spearheaded the creation of POSSIBLE’s Empathy Model, with translates the raw, unstructured content of social media into a quantitative understanding of what customers are actually feeling about a given topic, event, or brand.

About #Podcast:

FutureOfData podcast is a conversation starter to bring leaders, influencers and lead practitioners to come on show and discuss their journey in creating the data driven future.

Wanna Join? If you or any you know wants to join in, Register your interest by mailing us @ [email protected]

Want to sponsor? Email us @ [email protected]

Keywords: FutureOfData,

DataAnalytics,

Leadership,

Futurist,

Podcast,

BigData,

Strategy

With "Qlik Sense Cookbook," you will gain practical knowledge to harness the capabilities of Qlik Sense for effective business intelligence. This book is packed with step-by-step recipes that guide you in leveraging this powerful tool's data analytics features to create intuitive interactive dashboards and derive actionable insights. What this Book will help me do Master the process of sourcing, previewing, and distributing data through efficient interactive dashboards. Utilize the latest visualization options and learn best practices for creating impactful visuals. Develop scripts for automation and customize functionality using Qlik Sense subroutines. Enhance your Qlik Sense dashboard with advanced UI customizations and interactive elements. Leverage Qlik Sense's advanced aggregation functions like AGGR to perform multidimensional insights. Author(s) The authors of "Qlik Sense Cookbook" bring years of professional expertise in business intelligence and analytics. They have extensive experience working with Qlik platforms and have authored numerous industry-relevant resources. With a practical and accessible writing style, they thrive in breaking down complex concepts into manageable, actionable knowledge. Who is it for? This book is perfect for data analysts, business intelligence specialists, and Qlik Sense practitioners who want to advance their skills. It's suitable for beginners aiming to develop proficiency in Qlik Sense, as well as for professionals experienced with other tools like QlikView. Basic business intelligence knowledge is recommended for getting the most out of this book.

Business Analytics: A Data-Driven Decision Making Approach for Business-Part I,/i> provides an overview of business analytics (BA), business intelligence (BI), and the role and importance of these in the modern business decision-making. The book discusses all these areas along with three main analytics categories: (1) descriptive, (2) predictive, and (3) prescriptive analytics with their tools and applications in business. This volume focuses on descriptive analytics that involves the use of descriptive and visual or graphical methods, numerical methods, as well as data analysis tools, big data applications, and the use of data dashboards to understand business performance. The highlights of this volume are: Business analytics at a glance; Business intelligence (BI), data analytics; Data, data types, descriptive analytics; Data visualization tools; Data visualization with big data; Descriptive analytics-numerical methods; Case analysis with computer applications.

Navigate the fascinating intersection of healthcare and data science with the book "Healthcare Analytics Made Simple." This comprehensive guide empowers you to use Python and machine learning techniques to analyze and improve real healthcare systems. Demystify intricate concepts with Python code and SQL to gain actionable insights and build predictive models for healthcare. What this Book will help me do Understand healthcare incentives, policies, and datasets to ground your analysis in practical knowledge. Master the use of Python libraries and SQL for healthcare data analysis and visualization. Develop skills to apply machine learning for predictive and descriptive analytics in healthcare. Learn to assess quality metrics and evaluate provider performance using robust tools. Get acquainted with upcoming trends and future applications in healthcare analytics. Author(s) The authors, None Kumar and None Khader, are experts in data science and healthcare informatics. They bring years of experience teaching, researching, and applying data analytics in healthcare. Their approach is hands-on and clear, aiming to make complex topics accessible and engaging for their audience. Who is it for? This book is perfect for data science professionals eager to specialize in healthcare analytics. Additionally, clinicians aiming to leverage computing and data analytics in improving healthcare processes will find valuable insights. Programming enthusiasts and students keen to enter healthcare analytics will also greatly benefit. Tailored for beginners in this field, it is an educational yet robust resource.

Mastering Kibana 6.x is your guide to leveraging Kibana for creating impactful data visualizations and insightful dashboards. From setting up basic visualizations to exploring advanced analytics and machine learning integrations, this book equips you with the necessary skills to dive deep into your data and gain actionable insights at scale. You'll also learn to effectively manage and monitor data with powerful tools such as X-Pack and Beats. What this Book will help me do Build sophisticated dashboards to visualize elastic stack data effectively. Understand and utilize Timelion expressions for analyzing time series data. Incorporate X-Pack capabilities to enhance security and monitoring in Kibana. Extract, analyze, and visualize data from Elasticsearch for advanced analytics. Set up monitoring and alerting using Beats components for reliable data operations. Author(s) With extensive experience in big data technologies, the author brings a practical approach to teaching advanced Kibana topics. Having worked on real-world data analytics projects, their aim is to make complex concepts accessible while showing how to tackle analytics challenges using Kibana. Who is it for? This book is ideal for data engineers, DevOps professionals, and data scientists who want to optimize large-scale data visualizations. If you're looking to manage Elasticsearch data through insightful dashboards and visual analytics, or enhance your data operations with features like machine learning, then this book is perfect for you. A basic understanding of the Elastic Stack is helpful, though not required.

In this podcast @DanDeGrazia from @IBM spoke with @Vishaltx from @AnalyticsWeek to discuss the mingling of chief data scientist with open sources. He sheds light into some of the big opportunities in open source and how businesses could work together to achieve progress in data science. Dan also shared the importance of smooth communication for success as a data scientist.

TIMELINE: 0:29 Dan's journey. 9:40 Dan's role in IBM. 11:26 Tips on staying consistent while creating a database. 16:23 Chief data scientist and open-source put together. 20:28 The state of open source when it comes to data. 23:50 Evaluating the market to understand business requirements. 29:19 Future of data and open-source market. 33:23 Exciting opportunities in data. 37:06 Data scientist's role in integrating business and data. 49:41 Ingredients of a successful data scientist. 53:04 Data science and trust issues. 59:35 Human element behind data. 1:01:20 Dan's success mantra. 1:06:52 Key takeaways.

Dan's Recommended Read: The Five Temptations of a CEO, Anniversary Edition: A Leadership Fable by Patrick Lencioni https://amzn.to/2Jcm5do What Every BODY is Saying: An Ex-FBI Agent8217;s Guide to Speed-Reading People by Joe Navarro, Marvin Karlins https://amzn.to/2J1RXxO

Podcast Link: https://futureofdata.org/where-chief-data-scientist-open-source-meets-dandegrazia-futureofdata-podcast/

Dan's BIO: Dan has almost 30 years of experience working with large data sets. Starting with the unusual work of analyzing potential jury pools in the 1980s, Dan also did some of the first PC based voter registration analytics in the Chicago area, including putting the first complete list of registered voters on a PC (as hard as that is to imagine today a 50-megabyte hard drive on DOS systems was staggering). Interested in almost anything new and technical, he worked at The Chicago Board of Trade. He taught himself BASIC to write algorithms while working as an Arbitrager in financial futures. After the military, Dan moved to San Francisco. He worked with several small companies and startups designing and implementing some of the first PC-based fax systems (who cares now!), enterprise accounting software, and early middleware connections using the early 3GL/4GL languages. Always perusing the technical edge cases, Dan worked for InfoBright, a Column store Database startup in the US and EMEA, at Lingotek, an In-Q-Tel funded company working in large data set translations and big data analytics companies like Datameer and his current position as a Chief Data Scientist for Open Source in the IBM Channels organization. Dan's current just for fun Project is working to create an app that will record and analyze bird songs and provide the user with information on the bird and the specifics of the current song.

About #Podcast:

FutureOfData podcast is a conversation starter to bring leaders, influencers, and lead practitioners to discuss their journey to create the data-driven future.

Want to sponsor? Email us @ [email protected]

Keywords:

FutureOfData #DataAnalytics #Leadership #Podcast #BigData #Strategy

In this podcast, @BillFranksGA talks about the ingredients of a successful analytics ecosystem. He shared his analytics journey, his perspective on how other businesses are engaging in data analytics practice. He also sheds some light on best practices that businesses could adopt to execute a successful data strategy.

Timeline: 0:28 Bill's journey. 4:00 Bill's journey as an analyst. 9:29 Maturity of the analytics market. 11:56 Business, IT, and Data. 16:18 Introducing centralized analytics practice in an enterprise. 19:50 Tips and strategies for chief data officers to deliver the goods. 26:07 What do businesses don't get about data analytics? 29:40 Is the future aligned with data or analytics. 34:25 Importance for leadership to understand analytics. 36:35 The role of analytics professionals in the age of AI. 41:42 Upgrading analytics models. 47:50 How much should a business experiment on AI. 55:25 Evaluating blockchain. 59:50 Bill's success mantra. 1:05:25 Bill's favorite reads. 1:07:17 Key takeaway.

Podcast Link: https://futureofdata.org/billfranksga-on-the-ingredients-of-successful-analytics-ecosystem-futureofdata-podcast/

Bill's BIO: Bill Franks is Chief Analytics Officer for The International Institute For Analytics (IIA), where he provides perspective on trends in the analytics and big data space and helps clients understand how IIA can support their efforts to improve analytic performance. He also serves on the advisory boards of multiple university and professional analytic programs. He has held a range of executive positions in the analytics space in the past, including several years as Chief Analytics Officer for Teradata (NYSE: TDC).

Bill is the author of the book Taming The Big Data Tidal Wave (John Wiley & Sons). In the book, he applies his two decades of experience working with clients on large-scale analytics initiatives to outline what it takes to succeed in today’s world of big data and analytics. The book made Tom Peter’s list of 2014 “Must Read” books and also the Top 10 Most Influential Translated Technology Books list from CSDN in China.

His focus has always been to help translate complex analytics into terms that business users can understand and to then help an organization implement the results effectively within their processes. His work has spanned clients in a variety of industries for companies ranging in size from Fortune 100 companies to small non-profit organizations.

He earned a Bachelor’s degree in Applied Statistics from Virginia Tech and a Master’s degree in Applied Statistics from North Carolina State University.

About #Podcast:

FutureOfData podcast is a conversation starter to bring leaders, influencers and lead practitioners to come on show and discuss their journey in creating the data driven future.

Want to sponsor? Email us @ [email protected]

Keywords:

FutureOfData #DataAnalytics #Leadership #Podcast #BigData #Strategy

Spark for Data Professionals introduces and solidifies the concepts behind Spark 2.x, teaching working developers, architects, and data professionals exactly how to build practical Spark solutions. Jeffrey Aven covers all aspects of Spark development, including basic programming to SparkSQL, SparkR, Spark Streaming, Messaging, NoSQL and Hadoop integration. Each chapter presents practical exercises deploying Spark to your local or cloud environment, plus programming exercises for building real applications. Unlike other Spark guides, Spark for Data Professionals explains crucial concepts step-by-step, assuming no extensive background as an open source developer. It provides a complete foundation for quickly progressing to more advanced data science and machine learning topics. This guide will help you: Understand Spark basics that will make you a better programmer and cluster “citizen” Master Spark programming techniques that maximize your productivity Choose the right approach for each problem Make the most of built-in platform constructs, including broadcast variables, accumulators, effective partitioning, caching, and checkpointing Leverage powerful tools for managing streaming, structured, semi-structured, and unstructured data

Big Data Analytics with Hadoop 3 is your comprehensive guide to understanding and leveraging the power of Apache Hadoop for large-scale data processing and analytics. Through practical examples, it introduces the tools and techniques necessary to integrate Hadoop with other popular frameworks, enabling efficient data handling, processing, and visualization. What this Book will help me do Understand the foundational components and features of Apache Hadoop 3 such as HDFS, YARN, and MapReduce. Gain the ability to integrate Hadoop with programming languages like Python and R for data analysis. Learn the skills to utilize tools such as Apache Spark and Apache Flink for real-time data analytics within the Hadoop ecosystem. Develop expertise in setting up a Hadoop cluster and performing analytics in cloud environments such as AWS. Master the process of building practical big data analytics pipelines for end-to-end data processing. Author(s) Sridhar Alla is a seasoned big data professional with extensive industry experience in building and deploying scalable big data analytics solutions. Known for his expertise in Hadoop and related ecosystems, Sridhar combines technical depth with clear communication in his writing, providing practical insights and hands-on knowledge. Who is it for? This book is tailored for data professionals, software engineers, and data scientists looking to expand their expertise in big data analytics using Hadoop 3. Whether you're an experienced developer or new to the big data ecosystem, this book provides the step-by-step guidance and practical examples needed to advance your skills and achieve your analytical goals.

Analytics is the new force driving business. Tools have been created to measure program impacts and ROI, visualize data and business processes, and uncover the relationship between key performance indicators, many using the unprecedented amount of data now flowing into organizations. Featuring updated examples and surveys, this dynamic book covers leading-edge topics in analytics and finance. It is packed with useful tips and practical guidance you can apply immediately. This book prepares accountants to: Deal with major trends in predictive analytics, optimization, correlation of metrics, and big data. Interpret and manage new trends in analytics techniques affecting your organization. Use new tools for data analytics. Critically interpret analytics reports and advise decision makers.

A Deep Dive into NoSQL Databases: The Use Cases and Applications, Volume 109, the latest release in the Advances in Computers series first published in 1960, presents detailed coverage of innovations in computer hardware, software, theory, design and applications. In addition, it provides contributors with a medium in which they can explore their subjects in greater depth and breadth. This update includes sections on NoSQL and NewSQL databases for big data analytics and distributed computing, NewSQL databases and scalable in-memory analytics, NoSQL web crawler application, NoSQL Security, a Comparative Study of different In-Memory (No/New)SQL Databases, NoSQL Hands On-4 NoSQLs, the Hadoop Ecosystem, and more. Provides a very comprehensive, yet compact, book on the popular domain of NoSQL databases for IT professionals, practitioners and professors Articulates and accentuates big data analytics and how it gets simplified and streamlined by NoSQL database systems Sets a stimulating foundation with all the relevant details for NoSQL database researchers, developers and administrators

Summary

The rate of change in the data engineering industry is alternately exciting and exhausting. Joe Crobak found his way into the work of data management by accident as so many of us do. After being engrossed with researching the details of distributed systems and big data management for his work he began sharing his findings with friends. This led to his creation of the Hadoop Weekly newsletter, which he recently rebranded as the Data Engineering Weekly newsletter. In this episode he discusses his experiences working as a data engineer in industry and at the USDS, his motivations and methods for creating a newsleteter, and the insights that he has gleaned from it.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $20 credit and launch a new server in under a minute. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. Your host is Tobias Macey and today I’m interviewing Joe Crobak about his work maintaining the Data Engineering Weekly newsletter, and the challenges of keeping up with the data engineering industry.

Interview

Introduction How did you get involved in the area of data management? What are some of the projects that you have been involved in that were most personally fulfilling?

As an engineer at the USDS working on the healthcare.gov and medicare systems, what were some of the approaches that you used to manage sensitive data? Healthcare.gov has a storied history, how did the systems for processing and managing the data get architected to handle the amount of load that it was subjected to?

What was your motivation for starting a newsletter about the Hadoop space?

Can you speak to your reasoning for the recent rebranding of the newsletter?

How much of the content that you surface in your newsletter is found during your day-to-day work, versus explicitly searching for it? After over 5 years of following the trends in data analytics and data infrastructure what are some of the most interesting or surprising developments?

What have you found to be the fundamental skills or areas of experience that have maintained relevance as new technologies in data engineering have emerged?

What is your workflow for finding and curating the content that goes into your newsletter? What is your personal algorithm for filtering which articles, tools, or commentary gets added to the final newsletter? How has your experience managing the newsletter influenced your areas of focus in your work and vice-versa? What are your plans going forward?

Contact Info

Data Eng Weekly Email Twitter – @joecrobak Twitter – @dataengweekly

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

USDS National Labs Cray Amazon EMR (Elastic Map-Reduce) Recommendation Engine Netflix Prize Hadoop Cloudera Puppet healthcare.gov Medicare Quality Payment Program HIPAA NIST National Institute of Standards and Technology PII (Personally Identifiable Information) Threat Modeling Apache JBoss Apache Web Server MarkLogic JMS (Java Message Service) Load Balancer COBOL Hadoop Weekly Data Engineering Weekly Foursquare NiFi Kubernetes Spark Flink Stream Processing DataStax RSS The Flavors of Data Science and Engineering CQRS Change Data Capture Jay Kreps

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support Data Engineering Podcast

Summary

Managing an analytics project can be difficult due to the number of systems involved and the need to ensure that new information can be delivered quickly and reliably. That challenge can be met by adopting practices and principles from lean manufacturing and agile software development, and the cross-functional collaboration, feedback loops, and focus on automation in the DevOps movement. In this episode Christopher Bergh discusses ways that you can start adding reliability and speed to your workflow to deliver results with confidence and consistency.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $20 credit and launch a new server in under a minute. For complete visibility into the health of your pipeline, including deployment tracking, and powerful alerting driven by machine-learning, DataDog has got you covered. With their monitoring, metrics, and log collection agent, including extensive integrations and distributed tracing, you’ll have everything you need to find and fix performance bottlenecks in no time. Go to dataengineeringpodcast.com/datadog today to start your free 14 day trial and get a sweet new T-Shirt. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. Your host is Tobias Macey and today I’m interviewing Christopher Bergh about DataKitchen and the rise of DataOps

Interview

Introduction How did you get involved in the area of data management? How do you define DataOps?

How does it compare to the practices encouraged by the DevOps movement? How does it relate to or influence the role of a data engineer?

How does a DataOps oriented workflow differ from other existing approaches for building data platforms? One of the aspects of DataOps that you call out is the practice of providing multiple environments to provide a platform for testing the various aspects of the analytics workflow in a non-production context. What are some of the techniques that are available for managing data in appropriate volumes across those deployments? The practice of testing logic as code is fairly well understood and has a large set of existing tools. What have you found to be some of the most effective methods for testing data as it flows through a system? One of the practices of DevOps is to create feedback loops that can be used to ensure that business needs are being met. What are the metrics that you track in your platform to define the value that is being created and how the various steps in the workflow are proceeding toward that goal?

In order to keep feedback loops fast it is necessary for tests to run quickly. How do you balance the need for larger quantities of data to be used for verifying scalability/performance against optimizing for cost and speed in non-production environments?

How does the DataKitchen platform simplify the process of operationalizing a data analytics workflow? As the need for rapid iteration and deployment of systems to capture, store, process, and analyze data becomes more prevalent how do you foresee that feeding back into the ways that the landscape of data tools are designed and developed?

Contact Info

LinkedIn @ChrisBergh on Twitter Email

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

DataOps Manifesto DataKitchen 2017: The Year Of DataOps Air Traffic Control Chief Data Officer (CDO) Gartner W. Edwards Deming DevOps Total Quality Management (TQM) Informatica Talend Agile Development Cattle Not Pets IDE (Integrated Devel