Every December, Eckerson Group fulfills its industry obligation to summon its collective knowledge and insights about data and analytics and speculate about what might happen in the coming year. The diversity of predictions from our research analysts and consultants exemplifies the breadth of their research and consulting experiences and the depth of their thinking. Predictions from Kevin Petrie, Joe Hilleary, Dave Wells, Andrew Sohn, and Sean Hewitt range from data and privacy governance to artificial intelligence with stops along the way for DataOps, data observability, data ethics, cloud platforms, and intelligent robotic automation.

This is an audio blog about the perplexities of the Data Lakehouse and if it is, indeed, the "paradigm of the decade". To hear more of Eckerson Group perspectives on the data lakehouse be sure to check out the blogs from colleagues, Wayne Eckerson and Kevin Petrie, and the recording of our recent Shop Talk discussion.

Originally published at: https://www.eckerson.com/articles/an-architect-s-view-of-the-data-lakehouse-perplexity-and-perspective

Recent technology developments are driving urgency to modernize data management. What do you do about architecture, modeling, quality, and governance to keep up with big data, cloud, self-service, and other trends in data and technology? Examining some best practices can spark ideas of where to begin.

Originally published at https://www.eckerson.com/articles/stepping-up-to-modern-data-management

Data catalogs have become the centerpiece of modern data management. They are the means to connect self-service analysts with the right data, the “go to” technology for data curation, and the new gold standard for metadata management. With many data catalog tools available, choosing the right data catalog is an important decision.

Data pipelines become chaotic with pressures of agile, democratization, self-service, and organizational “pockets” of analytics. From enterprise BI to self-service analysis, data pipeline management should ensure analysis results are traceable, reproducible, and of production strength. Robust data pipelines rely on eight critical components.

Originally published at https://www.eckerson.com/articles/the-complexities-of-modern-data-pipelines

Ethics is challenging because right and good are not always clear. More data, more kinds of data, and advanced analysis of data often conflict with concerns of data privacy, security, anonymity, and ownership. Resolving these conflicts requires acknowledgment, discussion, and the hard work of defining ethics-based policies and creating a culture of ethical conduct.

Originally published at https://www.eckerson.com/articles/data-ethics-the-new-data-governance-challenge

Data engineering is one of the hottest and most difficult jobs to fill in the field of analytics. Breadth and depth of required skills limits the number of people qualified to work as data engineers. If you’re seeking to hire data engineers, consider the 24 skill areas identified here as guidance to shape job descriptions and to screen candidates. If you’re seeking to become a data engineer, take the skills assessment to highlight your strengths and identify your gaps.

Originally published at https://www.eckerson.com/articles/data-engineering-coming-of-age

In this podcast, Henry Eckerson interviews Dave Wells on the current health and future of the data warehouse. Wells acknowledges that data warehouses are struggling, but argues they are still necessary and cannot be replaced by data lakes. He then explains what the role of the modern data warehouse should be, practical steps forward for evolving the data warehouse, and much more.

Wells is an advisory consultant, educator, and industry analyst dedicated to building meaningful connections throughout the path from data to business value. He works at the intersection of information management and business management, driving business impact through analytics, business intelligence, and active data management. More than forty years of information systems experience combined with over ten years of business management give him a unique perspective about the connections among business, information, data, and technology. Knowledge sharing and skill building are Dave’s passions, carried out through consulting, speaking, teaching, and writing.

He is now the practice director of data management at Eckerson Group, cofounder and director of education at eLearningCurve, and a faculty member at The Data Warehousing Institute.