talk-data.com

Join fellow Airflow enthusiasts and industry leaders at the Marriott Downtown for an evening of insightful talks, great food and drinks, and exclusive swag!

This event brings together seasoned pros to explore cutting-edge strategies for managing complex DAG codebases, orchestrating large-scale data workflows, and unlocking the powerful new capabilities of Apache Airflow 3. Don't miss this opportunity to connect, learn, and level up your Airflow expertise!

PRESENTATIONS

Contending with Complex DAG Codebases - a Codemodding Approach

• Speaker: Alex Tomic, Data Engineering Independent Contractor
• As Airflow adoption continues to increase, organizations are faced with a mounting challenge of maintaining projects over the long-term, while meeting the demand for new project onboarding. Codemodding, a process to automatically modify code, is a valuable new tool to empower project and platform teams to develop, manage and support ever more complex dag codebases and workflows.

Harmonizing Music Rights at Scale: Airflow for Ownership Resolution and Fair Royalties

• Speaker: Pei-Chi Chen, Lead Data Engineer, Create Music Group
• In this session, we’ll share how Create Music Group leverages Apache Airflow to resolve music rights ownership conflicts—an ongoing challenge that increases costs and disrupts operations. We’ll dive into our Airflow-driven approach for large-scale, vital data collection and discuss best practices we’ve adopted from Airflow Summit 2024 to combat zombie tasks, out-of-memory issues, and worker configuration tuning. We'll also showcase how these techniques have helped us distribute revenue more accurately and streamline daily operations. This session provides a real-world, production-level example of Airflow’s power in action.

Unlocking the Future of Data Orchestration: Introducing Apache Airflow® 3

• Speaker: Constance Martineau, Staff Product Manager, Astronomer
• Airflow 3 is here, bringing a new era of flexibility, scalability, and security to data orchestration. This release introduces powerful new capabilities to simplify pipeline management, enhance security, and enable seamless execution across diverse environments. In this session, we’ll explore key innovations, including an upgraded UI, DAG Versioning, scheduler-integrated backfills, stronger task isolation, and support for remote execution. Join us to discover how Airflow 3 is redefining orchestration and empowering teams to build and scale workflows like never before!

AGENDA

• 5:30-6PM: Arrivals, eat, drink, and network
• 6-7:40PM: Presentations
• 7:40-8PM: Networking

The Future of Big Data with Gen AI: Accelerating System Migration and Boosting Efficiency

Description

In the fast-evolving landscape of big data, frequent system migrations pose significant challenges, particularly in code conversion and adoption of new data processing engines. This presentation addresses the inefficiencies of manual code migration processes faced by big data teams, emphasizing the need for scalability and faster adoption rates. We propose an innovative approach that utilizes Gen AI and cloud services such as AWS Bedrock, Redshift, Glue, Athena, and S3, coupled with robust automated testing mechanisms. Our methodology has proven effective, demonstrating a 75% reduction in the time and effort required for code conversion and testing during migrations. We will also explore further applications of this strategy across the big data domain, highlighting its potential to significantly enhance operational efficiency for big data teams.

Register for Future Data Driven Summit 2024

Transforming Data Science and PowerBI with Fabric Semantic Link

Description

This session is for Data Scientists, Data Engineers, and PowerBI Developers who want to fully use Semantic Link in their projects.

We'll delve into how Semantic Link is revolutionizing data analysis and management, offering valuable insights for different roles. You'll learn about the functionalities of Semantic Link, including its Data Frame and Spark-based features, and how to apply these tools in real-world scenarios.

We'll also cover advanced techniques such as relationship analysis and dependency management, showcasing how Semantic Link can elevate your data projects. Whether you're aiming to enhance data quality, streamline workflows, or explore new data strategies, this session will provide you with the knowledge and skills to effectively leverage Semantic Link.

Join me to unlock the full potential of your data with Semantic Link in Microsoft Fabric.

Register for Future Data Driven Summit 2024

We cordially invite you to join us at the Future Data Driven Summit 2024, a vibrant confluence of thought leaders and innovators in the realms of Data, AI, and Visualization.

With a lineup of over 60 distinguished speakers covering a diverse array of topics, this event promises to be a treasure trove of insights and knowledge. We encourage you to register and seize this opportunity to engage, learn, and grow. But the experience shouldn’t stop with you.

Extend this invitation to your peers, share the excitement, and let’s together create a ripple effect of learning and innovation. Looking forward to seeing you there!

RSVP: Future Data Driven summit 2024: Registration (datadrivencommunity.com)

Hello,

data folks, please read a message from Future Data Driven Summit 2024 organisers.

Please use SotonDataCloud as a promo code - the event is free, but it will help us identify how many of our members registered.

We have some exciting news for you! The 'Future Data Driven' Summit is back and ready to blow your mind! This year the conference is scheduled for Sep-25-2024. It's a full day virtual event of free content for you to enjoy.

The registration is opened now. Secure your spot today! https://datadrivencommunity.com/fdd2024registration.html

We have an incredible lineup of speakers who submitted sessions for this year to share their knowledge, experience, and passion with you. You'll get to hear from the top experts in the industry, connect with other data lovers, and explore new possibilities and solutions.

This is an awesome opportunity to dive into the world of data & AI and get motivated by the most cutting-edge trends and innovations. You don't want to miss this!

We continuously update information on the conference page. Therefore, we highly recommend that you visit frequently: https://datadrivencommunity.com/FutureDataDriven2024.html

We invite you to save your spot and share this summit with your peers.

So don't delay and register for the event today!

Sincerely, Mikey

Advanced Techniques in SQL Server Performance & Optimization

Description

Join this session to explore advanced techniques for optimizing SQL Server performance. Learn how to fine-tune queries, utilize indexing strategies effectively, and optimize database schema designs for maximum efficiency. Discover insights into query execution plans and performance monitoring tools to identify and resolve bottlenecks. Gain practical tips on memory management, caching strategies, and best practices for maintaining high-performance databases. Whether you're managing large-scale data or enhancing application responsiveness, this session equips you with essential skills to elevate SQL Server performance and optimize database operations.

Register for Future Data Driven Summit 2024

ML Prompt Flow: The CoPilot's King - Dennes Torres

Description

There are many different architectures being implemented to produce RAG, an LLM capable to analyse the private data for one company.

However, ML Prompt Flow is for sure above all of them, because it's capable to create an orchestration between different architectures resulting in a single co-pilot capable to get every piece of data your company has, on every format.

Register for Future Data Driven Summit 2024

Reynold Xin, Co-founder and Chief Architect at Databricks, presented during Data + AI Summit 2024 on Databricks SQL and its advancements and how to drive performance improvements with the Databricks Data Intelligence Platform.

Speakers: Reynold Xin, Co-founder and Chief Architect, Databricks Pearl Ubaru, Technical Product Engineer, Databricks

Main Points and Key Takeaways (AI-generated summary)

Introduction of Databricks SQL: - Databricks SQL was announced four years ago and has become the fastest-growing product in Databricks history. - Over 7,000 customers, including Shell, AT&T, and Adobe, use Databricks SQL for data warehousing.

Evolution from Data Warehouses to Lakehouses: - Traditional data architectures involved separate data warehouses (for business intelligence) and data lakes (for machine learning and AI). - The lakehouse concept combines the best aspects of data warehouses and data lakes into a single package, addressing issues of governance, storage formats, and data silos.

Technological Foundations: - To support the lakehouse, Databricks developed Delta Lake (storage layer) and Unity Catalog (governance layer). - Over time, lakehouses have been recognized as the future of data architecture.

Core Data Warehousing Capabilities: - Databricks SQL has evolved to support essential data warehousing functionalities like full SQL support, materialized views, and role-based access control. - Integration with major BI tools like Tableau, Power BI, and Looker is available out-of-the-box, reducing migration costs.

Price Performance: - Databricks SQL offers significant improvements in price performance, which is crucial given the high costs associated with data warehouses. - Databricks SQL scales more efficiently compared to traditional data warehouses, which struggle with larger data sets.

Incorporation of AI Systems: - Databricks has integrated AI systems at every layer of their engine, improving performance significantly. - AI systems automate data clustering, query optimization, and predictive indexing, enhancing efficiency and speed.

Benchmarks and Performance Improvements: - Databricks SQL has seen dramatic improvements, with some benchmarks showing a 60% increase in speed compared to 2022. - Real-world benchmarks indicate that Databricks SQL can handle high concurrency loads with consistent low latency.

User Experience Enhancements: - Significant efforts have been made to improve the user experience, making Databricks SQL more accessible to analysts and business users, not just data scientists and engineers. - New features include visual data lineage, simplified error messages, and AI-driven recommendations for error fixes.

AI and SQL Integration: - Databricks SQL now supports AI functions and vector searches, allowing users to perform advanced analysis and query optimizations with ease. - The platform enables seamless integration with AI models, which can be published and accessed through the Unity Catalog.

Conclusion: - Databricks SQL has transformed into a comprehensive data warehousing solution that is powerful, cost-effective, and user-friendly. - The lakehouse approach is presented as a superior alternative to traditional data warehouses, offering better performance and lower costs.

Please join us on 17th April 2024 to listen to the topic: All About SQL Injection

What \~ Toronto Data Professionals Community (Virtual) When \~ Wednesday 17th April 2024

Agenda:

• 6:00 PM Networking and Introduction
• 6:15 PM Topic: All About SQL Injection with Ed Pollack
• 7:30 PM End

Where: Online via Microsoft team

Session Details: SQL injection has consistently ranked as one of the top security threats in software development. Businesses have experienced massive data breaches and have even been forced to close as a result of security holes related to SQL injection. Proactive solutions are key to preventing SQL injection and ensuring that hackers don't see your application as an easy target. We will demonstrate common mistakes that lead to SQL injection and a variety of ways which can safeguard our data against attacks. While illustrating the ways in which hackers probe and attack our systems, we will show that SQL injection is not only the result of bad TSQL, but also bad code. By securing our data from the application and database tiers, we can greatly decrease their attractiveness to hackers and prevent attacks that could cripple a business permanently.

Speaker Bio: Ed Pollack is a Microsoft Data Platform MVP with a passion for learning how the Microsoft Data Platform works and sharing that knowledge with the community. His experiences in data architecture, database design, performance optimization, and data security are motivation for public speaking, writing, coding, and other community activities. Ed has spoken at SQL Saturday events, SQL Bits, PASS Summit, EightKB, and many other regional and international events. Ed is the organizer of the Capital Area SQL Server Group and SQL Saturday Albany, as well as a co-organizer of SQL Saturday New York City, and Future Data Driven. He has published a number of books, including "Dynamic SQL: Applications, Performance, and Security in Microsoft SQL Server", "Expert Performance Indexing in Azure SQL and SQL Server 2022", and "Analytics Optimization with Columnstore Indexes in Microsoft SQL Server: Optimizing OLAP Workloads". Ed is also an active contributor of content to SimpleTalk. In his free time, Ed enjoys video games, traveling, cooking exceptionally spicy foods, and hanging out with his amazing wife and sons.