This blog explores three criteria to evaluate tools that manage unstructured data pipelines for GenAI. Published at: https://www.eckerson.com/articles/data-engineering-for-genai-three-criteria-to-evaluate-pipeline-tools

Data engineering is now considered a crucial job in IT as Generative AI, the hottest technology of this decade, relies on data engineers to provide accurate inputs. Published at: https://www.eckerson.com/articles/data-engineering-for-genai-how-to-optimize-data-pipelines-and-governance

GenAI can help data engineers become more productive, and data engineering can help GenAI drive new levels of innovation. Published at: https://www.eckerson.com/articles/achieving-fusion-how-genai-and-data-engineering-help-one-another

Data modeling is a core skill of data engineering, but it is missing or inadequate in many data engineering teams. These teams focus on moving data with little attention to shaping the data. They engineer processes, not products. Full data engineering is both process and product engineering, and that calls for data modeling. Published at: https://www.eckerson.com/articles/a-fresh-look-at-data-modeling-part-2-rediscovering-the-lost-art-of-data-modeling

The unbundling of the data ecosystem is causing organizations to “duct tape” products and frameworks together to build their solutions and data delivery processes. Organizations fail to build and deploy end-to-end, automated, repeatable data-driven systems, ignoring data engineering & dataops principles as well as best practices. Published at: https://www.eckerson.com/articles/dataops-in-data-engineering

This blog recommends guiding principles for successful implementation of language models to assist data engineering. Published at: https://www.eckerson.com/articles/should-ai-bots-build-your-data-pipelines-part-iv-guiding-principles-for-success-with-language-models-and-data-engineering

An emerging approach to generative AI will help data engineering teams achieve much-needed productivity gains while controlling risk. Published at: https://www.eckerson.com/articles/should-ai-bots-build-your-data-pipelines-part-iii-the-emergence-of-small-language-models-for-data-engineering

Many data engineers already use large language models to assist data ingestion, transformation, DataOps, and orchestration. This blog commences a series that explores the emergence of ChatGPT, Bard, and LLM tools from data pipeline vendors, and their implications for the discipline of data engineering. Published at: https://www.eckerson.com/articles/should-ai-bots-build-your-data-pipelines-examining-the-role-of-chatgpt-and-large-language-models-in-data-engineering

Design patterns have proven valuable in many endeavors. Can data pipeline design patterns help to break the data engineering logjam? Published at: https://www.eckerson.com/articles/data-pipeline-design-patterns

In this episode, Wayne Eckerson and Shakeeb Ahkter dive into DataOps. They discuss what DataOps is, the goals and principles of DataOps, and reasons to adopt a DataOps strategy. Shakeeb also reveals the benefits gained from DataOps and what tools he uses. He is the Director of Enterprise Data Warehouse at Northwestern Medicine and is responsible for direction and oversight of data management, data engineering, and analytics.

Data engineering is one of the hottest and most difficult jobs to fill in the field of analytics. Breadth and depth of required skills limits the number of people qualified to work as data engineers. If you’re seeking to hire data engineers, consider the 24 skill areas identified here as guidance to shape job descriptions and to screen candidates. If you’re seeking to become a data engineer, take the skills assessment to highlight your strengths and identify your gaps.

Originally published at https://www.eckerson.com/articles/data-engineering-coming-of-age