We’re excited to be back at Big Data LDN this year—huge thanks to the organisers for hosting Databricks London once more!

Join us for an evening of insights, networking, and community with the Databricks Team and Advancing Analytics!

🎤 Agenda:

6:00 PM – 6:10 PM | Kickoff & Warm Welcome

Grab a drink, say hi, and get the lowdown on what’s coming up. We’ll set the scene for an evening of learning and laughs.

6:10 PM – 6:50 PM | The Metadata Marathon: How three projects are racing forward – Holly Smith (Staff Developer Advocate, Databricks)

With the enormous amount of discussion about open storage formats between nerds and even not-nerds, it can be hard to keep track of who’s doing what and how this actually makes any impact on day to day data projects.

Holly will take a closer look at the three big projects in this space; Delta, Hudi and Iceberg. They’re all trying to solve for similar data problems and have tackled the various challenges in different ways. Her talk will start with the very basics of how we got here, what the history is before diving deep into the underlying tech, their roadmaps, and their impacts on the data landscape as a whole.

6:50 PM – 7:10 PM | What’s New in Databricks & Databricks AI – Simon Whiteley & Gavi Regunath

Hot off the press! Simon and Gavi will walk you through the latest and greatest from Databricks, including shiny new AI features and platform updates you’ll want to try ASAP.

7:10 PM onwards | Q&A Panel + Networking

Your chance to ask the experts anything—then stick around for drinks, snacks, and some good old-fashioned data geekery.

Join us in the heart of New York City for a free ML/AI mega-meetup featuring an incredible lineup of ML experts, data scientists, and DevOps professionals. Dive into scaling AI in production, optimizing ML workflows, and more in data science. This is a great opportunity to mingle with the best tech minds NYC has to offer over some good food. Register now on LUMA and save your spot!

Featuring:

• AIOps at Reasonable Scale by Ciro Greco, Founder and CEO @ Bauplan
• This session explores the unique security challenges of ML systems in the GenAI era and provides actionable strategies to safeguard them. Learn why traditional approaches fall short and how to fortify your ML lifecycle to stay ahead in an evolving threat landscape.

• We have pioneered the concept of “ML for the 99%” with the ML at reasonable scale series, and recently discussed what changed with the new AI wave (spoiler: not much, the fundamentals stay!). In the talk, we review the basics of ML in production and stress what changed and what didn’t in the era of LLMs.

• Protecting ML Systems in the GenAI Era by Yuval Fernbach, VP, CTO MLOps @ JFrog

• Generative AI and machine learning systems are reshaping industries but also introducing new security risks. The reliance on vast data, rapid deployment cycles, and automated pipelines in MLOps has expanded the attack surface, exposing vulnerabilities to data poisoning, adversarial inputs, and pipeline exploitation.

• This session explores the unique security challenges of ML systems in the GenAI era and provides actionable strategies to safeguard them. Learn why traditional approaches fall short and how to fortify your ML lifecycle to stay ahead in an evolving threat landscape.

• Integrating Tech to Unlock Generative AI by Liron Freind Saadon, Head of Dev Rel @ NVIDIA

• An Ideal MLOps platform in the Generative Al era is a comprehensive solution that supports the entire machine learning lifecycle, from data preparation and model development to model deployment and monitoring. It should provide seamless integration of tools and technologies that enable organizations to build, deploy, and manage machine learning models with ease. In this talk, we'll review some of the tools and solutions that exist to build, deploy, and scale GenAl workloads across different environments successfully.

• Human-in-the-loop feedback, agentic systems by Ari Kaplan, Head of Evangelism, Databricks Discover how incorporating continuous human guidance ensures ethical, accurate, and context-aware outputs, while autonomous agentic systems push the boundaries of Al, enabling proactive decision-making and complex problem-solving.

• Networking/Happy Hour

Register now on LUMA and save your spot!

• 4:00 PM - 4:15 PM Arrival and Sign in
• 4:15 PM - 4:30 PM AIOps at Reasonable Scale
• 4:50 PM - 5:30 PM Protecting ML Systems in the GenAI Era
• 5:30 PM - 6:00 PM Fireside Chat with Industry leaders
• 6:00 PM - 6:45 PM Integrating Tech to Unlock

Brought to you by JFrog!

Register now on LUMA and save your spot!

Join us on January 29 for a chance to connect, learn, and get inspired by the latest and greatest in AI. Whether you're a pro at AI or just getting started, this event is all about building our community and shaping the future of AI development. Get ready for an evening packed with awesome sessions, discussions, and networking opportunities!

Let's dive into the exciting world of AI together!

Agenda

• 6:00 PM ET: Welcome and introduction
• 6:05 PM ET to 8:00 PM ET: Program
• 8:00 PM ET to 9:00 PM ET: Networking

Program Sessions

| Session Title | Session Description | Speaker | | :------------- |:-------------|:-------------| | Empowering Enterprises with LLM Agents: Assistants for Data-Driven Insights | Large Language Model (LLM) agents are emerging as powerful tools that can revolutionize how organizations interact with their data. This session will delve into the concept of LLM agents and their potential to transform internal operations by creating intelligent virtual assistants on top of your datawarehouse. We will explore how these agents can be leveraged to generate SQL query statements dynamically, enabling seamless access to data stored in various data warehouses such as Databricks, Microsoft Fabric. Attendees will gain insights into the architecture and implementation of these virtual assistants, and discover how they can enhance productivity, streamline decision-making processes, and unlock valuable insights from their data. | Sammy Deprez | | Simplifying AI in your .NET Application | Discover how to effortlessly integrate AI into your .NET applications with the Microsoft.Extensions.AI library. Learn to add chat features, embedding generation, and tool calling seamlessly. Plus, explore innovative techniques for real-time prompt management, allowing you to test and refine AI prompts live without stopping your application. Join us for a fast-paced session packed with practical insights and cutting-edge strategies to optimize your development workflow. | Maria Naggaga Nakanwagi | | AutoGen 0.4: A Programming Framework for Agentic AI reimagined | AutoGen is an open-source framework for building AI agent systems. It simplifies the creation of event-driven, distributed, scalable, and resilient agentic applications using multi-agent architectures. In this talk, Jack will provide an overview of the AutoGen framework and dig into 0.4, a reimagining of the framework. | Jack Gerrits | | AI in Action: Lessons and Success Stories | Join us for an engaging panel discussion featuring practitioners of artificial intelligence. Our panelists will share their firsthand experiences, challenges, and successes in implementing solutions across various industries. Learn about innovative use cases and discover best practices from experts who are at the forefront of AI technology.| Lisa Qu, Vivian Lei, Samir Mahmoud, Justin Trugman |

🌟 Session Overview 🌟

Session Name: Supernovas, Black Holes, and Streaming Data: A Journey in Space with Apache Kafka data streams from NASA Speaker: Frank Munz Session Description: In this fun, hands-on, and in-depth How-To, we explore NASA's GCN project, which publishes various events in space as Kafka topics.

The focus of my talk is on end-to-end data engineering, from consuming the data and ELT-ing the stream, to using generative AI tools for analytics.

We will analyze GCN data in real time, specifically targeting the data stream from exploding supernovas. This data triggers dozens of terrestrial telescopes to potentially reposition and point toward the event.

The speaker will kick off the session by contrasting various ways of ingesting and transforming the data, discussing their trade-offs: Should you use a declarative data pipeline, or can a data analyst manage with SQL only? Alternatively, when would it be better to follow the classic approach of orchestrating Spark notebooks to get the data ingested?

He will answer the question: Does a data engineer working with streaming data benefit from generative AI-based tools and assistants today? Is it worth it, or is it just hype?

The demo is easy to replicate at home, and Frank will share the notebooks in a GitHub repository so you can analyze real NASA data yourself!

This session is ideal for data engineers, data architects who enjoy some coding, generative AI enthusiasts, or anyone fascinated by technology and the sparkling stars in the night sky.

While the focus is clearly on tech, the demo will run on the open-source and open-standards-based Databricks Intelligence Platform (so inevitably, you'll get a high-level overview here too).

🚀 About Big Data and RPA 2024 🚀

Unlock the future of innovation and automation at Big Data & RPA Conference Europe 2024! 🌟 This unique event brings together the brightest minds in big data, machine learning, AI, and robotic process automation to explore cutting-edge solutions and trends shaping the tech landscape. Perfect for data engineers, analysts, RPA developers, and business leaders, the conference offers dual insights into the power of data-driven strategies and intelligent automation. 🚀 Gain practical knowledge on topics like hyperautomation, AI integration, advanced analytics, and workflow optimization while networking with global experts. Don’t miss this exclusive opportunity to expand your expertise and revolutionize your processes—all from the comfort of your home! 📊🤖✨

📅 Yearly Conferences: Curious about the evolution of QA? Check out our archive of past Big Data & RPA sessions. Watch the strategies and technologies evolve in our videos! 🚀 🔗 Find Other Years' Videos: 2023 Big Data Conference Europe https://www.youtube.com/playlist?list=PLqYhGsQ9iSEpb_oyAsg67PhpbrkCC59_g 2022 Big Data Conference Europe Online https://www.youtube.com/playlist?list=PLqYhGsQ9iSEryAOjmvdiaXTfjCg5j3HhT 2021 Big Data Conference Europe Online https://www.youtube.com/playlist?list=PLqYhGsQ9iSEqHwbQoWEXEJALFLKVDRXiP

💡 Stay Connected & Updated 💡

Don’t miss out on any updates or upcoming event information from Big Data & RPA Conference Europe. Follow us on our social media channels and visit our website to stay in the loop!

🌐 Website: https://bigdataconference.eu/, https://rpaconference.eu/ 👤 Facebook: https://www.facebook.com/bigdataconf, https://www.facebook.com/rpaeurope/ 🐦 Twitter: @BigDataConfEU, @europe_rpa 🔗 LinkedIn: https://www.linkedin.com/company/73234449/admin/dashboard/, https://www.linkedin.com/company/75464753/admin/dashboard/ 🎥 YouTube: http://www.youtube.com/@DATAMINERLT

Hello hello everyone! Thanks to all who attended the last meet up on Data & AI - The product Perspective. It was a full house and hope you all enjoyed the slightly different panel discussion style. This time we are back to the norm... 2 fantastic speakers from more of an engineering focus. 25-30 minute talks followed by an open Q&A. Drinks, networking and of course tons of pizza.. Look forward to seeing you all then!

Holly Smith \| Staff Developer Advocate @ Databricks Data Engineers AI Survival Guide I have a hypothesis, that 90% of people doing Gen AI today weren’t doing it two years ago. The landscape is full of people stumbling their way through it, from the AI academics learning that code for papers is not software development ready, all the way to data experts suddenly needing to learn a new skill. In this talk, we'll go through what data engineers need to know to help get those AI projects off the ground. Starting with picking the right projects, execution plans, through to toolsets and skills that will make you shine.

About me Holly Smith is a multi award winning Data and AI expert who has over a decade of experience working with Data and AI teams in a variety of capacities from individual contributors all the way up to leadership. In her role at Databricks she has worked with many multi national companies as they embark on their journey to the cutting edge of data. She is a renowned public speaker, teacher and minority in tech advocate.

Zack Akil \| Senior ML Engineer & Developer Advocate @ Google The "IVO" design pattern for pragmatic GenAI Come learn about the "IVO" design pattern with live demos and cautionary tales. This pragmatic pattern serves as a foundation for building effective tools with generative AI. Whether you're a developer or simply a user, IVO will guide you in crafting the right questions and tasks to get the most out of generative AI.

About me Zack Akil (@ZackAkil) is a senior machine learning engineer & developer advocate at Google specialising in practical AI/ML with a background in full stack application development and data science. He spends most of his time building end-to-end ML applications (often inspired by his many random hobbies) that help to inspire developers on how they can solve problems with ML.

Please aim to arrive at 6pm. The talks will start around 6.30pm (give or take a few mins) and there will be pizza and refreshments as always on arrival :)

Look forward to seeing you all soon!

Event Overview: We are thrilled to invite you to London Data Engineering Coffee Chat. This time we will invite two speakers from Databricks to share their data career. Also, an exciting gathering dedicated to exploring the latest advancements and insights in Databricks. This event will bring together industry leaders, experts, and enthusiasts for an engaging series of discussions, presentations, and networking opportunities.

Speakers: Liping Huang, Data Architect in Databricks and founder of Youtube channel Data Leaps. Dustin Smith, Data Architect in Databricks with 10+ years in Big Data Engineering and Machines Learning.

Why Attend?

• Learn from the Best: Gain insights from top experts and innovators.
• Network: Connect with professionals and peers in the industry.
• Innovate: Discover new ideas and approaches that can drive your work forward.

Engage: Participate in Q&A sessions, panel discussions, and networking.

Date set!

Join us once again for the next PyData Bristol Meetup! We’re meeting in a lovely venue Amdaris!! Big thank you for hosting our event this time. Massive thanks to our other sponsor - Adlib for the scrumptious pizza, and thirst-quenching refreshments. Here is an intro from out new hosts!

Hi, We are Amdaris, an Insight company, and at our core we specialise in extending teams with highly skilled software experts. With bespoke Software Development, Product Design, Strategy and Consultation, Managed Services and Data Solutions, we seamlessly integrate into your business and culture, bringing passion, care, and technical proficiency directly to you.

Agenda for the evening: 🚪 6:00 pm - Doors open 🕡 6:30 pm - Talks commence (sharp!) 📚 Deep Dive 50 Minute Talk:

• Better ETL with Managed Airflow in ADF by Niall Langley

📢 Community announcements 🤝 Relaxed networking over beers and soft drinks Interested in sharing your knowledge or experience at this or a future event? Fill out this form to submit your talk proposal: PyData Bristol Talk Proposal We look forward to seeing you there for another fantastic evening of Python, Data Science, and camaraderie! ‌ ‌

Talks

Better ETL with Managed Airflow in ADF by Niall Langley Building complex data workflows using Azure Data Factory can get a little clunky - as you orchestration needs get more complex you hit limitations like not being able to nest loops or conditionals, running simple Python, bash or PowerShell scripts is difficult, and costs can grow quickly as you are charged per task execution. Recently another option become available, Managed Airflow in ADF. Apace Airflow is a code-centric open-source platform for developing, scheduling and monitoring batch-based data workflows, built using the python language Data Engineers know and love. But until Managed Airflow, getting it working in Azure was a complex task for customers more used to PaaS services such as ADF, Databricks and Fabric. It is also an important ETL orchestrator on AWS and GCP, so cross cloud compatibility becomes simpler to achieve. In this session we’ll look at what Airflow is, how it’s different from ADF, and what advantages Managed Airflow in ADF gives us. We talk about the idea of a DAG for building the workflow, and then work through some demos to show just how easy it is to use Python to write an Airflow DAG’s and import them into the Managed Airflow Environment as pipelines. We then dive into the excellent monitoring UI and find out just how easy is it to trigger a pipeline, view it to see the dependencies between tasks, and monitor runs. By the end of the session attendees will have a good understanding of what Airflow is, when to use it, and how it fits into the Azure Data Platform.

‌ 🕖 LOGISTICS Talks kick off at 18:30 sharp; then networking in Left Handed Giant from 20:40. If you realise you can't make it, please un-RSVP in good time to free up your place for your fellow community members. Follow @‌pydatabristol (PyData Bristol (@PyDataBristol) / X) for updates on this and future events, as well as news from the global PyData community. 📜 CODE OF CONDUCT The PyData Code of Conduct governs this meetup (Code of Conduct ). To discuss any issues or concerns relating to the code of conduct or behaviour of anyone at the PyData meetup, please contact the PyData Bristol organisers, or you can submit a report of any potential Code of Conduct violation directly to NumFOCUS (NumFOCUS Code of Conduct Report Form).

Speakers

Niall Langley

Sponsored by Amdaris - https://amdaris.com and Advancing Analytics - https://www.advancinganalytics.co.uk/

Amdaris is your trusted partner for high velocity extended delivery teams.

Advancing Analytics specialises in building cutting-edge Data Platforms for Data Science and Data Engineering.

Location: Amdaris, Finzels Reach, Aurora, Bristol BS1 6BX

AGENDA

18.00 – 18:30 Meet & Greet

--------------

18:30 - 19:15 Introduction to Tuning Spark on Databricks and Fabric by Niall Langley

More and more organisations are building data platforms in the cloud, often utilising Spark with tools like Databricks, and the recently announced Fabric to build data processing pipelines. These distributed computing tools can take a while to learn, and often teams migrating older premises data warehouses to cloud solutions like the lake house concentrate on getting good data over getting the best performance.

However in the cloud, any performance improvements can have a big impact on monthly cost, making it much easier to to justify spending time getting things running faster and more efficiently. This talk aims to help point you at common pain points when working with Spark using Databricks or Fabric, showing you where to look, and what to look for, and what can be done to improve things.

--------------

19:15 - 19:45 Pizza and Networking

--------------

19:45 - 20:30 Want End-to-End MLOps? Look no further than Databricks! by Tori Tompkins

Arguably the largest challenge in ML today is effectively deploying reliable and efficient models into production, with experts quoting that as many as 90% of model created never make it to production. MLOps streamlines the process of taking machine learning models to production, and then maintaining and monitoring them. With new MLOps micro-venders popping up every day, is there a tool that does everything?

In this session, we will consider Databricks as an end-to-end MLOps tool, exploring collaborative workspaces, feature stores, model registries and model serving. We will also touch upon other critical MLOps practices such as model fairness, explainability and monitoring.

Including practical demos of Databricks Feature Store, MLFlow and real-time Model Serving, this session is suitable for Data Scientists and Machine Learning Engineers of all levels.

https://www.linkedin.com/in/tompkinstori/

--------------

20:30 - Pub

-------------- Notes

• Photos: We, and potentially others, take photos of our meetups which are shared (e.g. on our meetup group, print publicity; social media, etc.). Please speak to one of the organisers in advance of the meetup if you need to discuss this
• Bicycle parking: We do not have arrangements with the venue for secure storage. There are public bicycle stands by Spicer and Cole/Premier Inn (to the left of Amdaris). Bicycles are left at your own risk

-------------- About Amdaris

Of course, we’re obsessed with cutting-edge technology but it’s how much we care about people that sets us apart. Whether that’s looking after our clients, our staff or the next generation of tech talent, we know that exceptional software development is only possible with exceptional teamwork.

If you need help extending your team, building your big idea or application support, we offer a better way to do software.