A complex data flow is a set of operations to extract data from multiple sources, write to multiple targets, and refine the results using extract, transform, join, filter, and sort. Chaos Engineering involves experimenting with a distributed system to test its ability to withstand turbulent conditions in production. But, what about data? How confident are we that the complex data system will be safe once it is in production? The key is to experiment in production and automate while minimizing customer pain and protecting data from getting corrupted or accidentally deleted. In this session, you will discover how chaos engineering principles apply to distributed data systems and the tools that enable us to make our data workloads more resilient. We will also show you how to leverage lakeFS to recover from deploying code that resulted in corrupted data, which can easily happen with many moving parts.

Connect with us: Website: https://databricks.com Facebook: https://www.facebook.com/databricksinc Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/data... Instagram: https://www.instagram.com/databricksinc/