talk-data.com
Topic
12
tagged
Whether you have basic or mature cloud cost management practices in your organization, AI workloads add specific considerations. Learn how AI usage correlates to cost, what to monitor, and how to drive accountability for your AI investment, with Azure tools and industry best practices.
Come learn about the latest tools to keep your team under budget and optimize your data ROI. Managing the costs of data consumption, attributing to projects, and understanding TCO is top of mind for customers. We will discuss implementing FinOps, observability around cost management. Workloads, consumption, users, what they worked on - and how to optimize it all.
The session will cover how to use Unity Catalog governed system tables to understand what is happening in Databricks. We will touch on key scenarios for FinOps, DevOps and SecOps to ensure you have a well-observed Data Intelligence Platform. Learn about new developments in system tables and other features that will help you observe your Databricks instance.
Westat, a leader in data-driven research for 60 years+, has implemented a centralized Databricks platform to support hundreds of research projects for government, foundations, and private clients. This initiative modernizes Westat’s technical infrastructure while maintaining rigorous statistical standards and streamlining data science. The platform enables isolated project environments with strict data boundaries, centralized oversight, and regulatory compliance. It allows project-specific customization of compute and analytics, and delivers scalable computing for complex analyses. Key features include config-driven Infrastructure as Code (IaC) with Terragrunt, custom tagging and AWS cost integration for ROI tracking, budget policies with alerts for proactive cost management, and a centralized dashboard with row-level security for self-service cost analytics. This unified approach provides full financial visibility and governance while empowering data teams to deliver value. Audio for this session is delivered in the conference mobile app, you must bring your own headphones to listen.
This session is repeated. You’ve seen your usage grow on Databricks, across departments, use cases, product lines and users. What can you do to ensure your end-users (data practitioners) of the platform remain cost-efficient and productive, while staying accountable to your budget? We’ll discuss spend monitoring, chargeback models and developing a culture of cost efficiency by using Databricks tools.
We will explore how leveraging Databricks' Unity Catalog has accelerated our FinOps maturity, enabling us to optimize platform utilization and achieve significant cost reductions. By implementing Unity Catalog, we've gained comprehensive visibility and governance over our data assets, leading to more informed decision-making and efficient resource allocation. Learn how Corning discovered actionable insights and leveraged best practices on utilizing Unity Catalog to streamline data management, enhance financial operations and drive substantial savings within your organization.
At Kaizen Gaming, data drives our decision-making, but rapid growth exposed inefficiencies in our legacy cloud setup — escalating costs, delayed insights and scalability limits. Operating in 18 countries with 350M daily transactions (1PB+), shared quotas and limited cost transparency hindered efficiency. To address this, we redesigned our cloud architecture with Data Landing Zones, a modular framework that decouples resources, enabling independent scaling and cost accountability. Automation streamlined infrastructure, reduced overhead and enhanced FinOps visibility, while Unity Catalog ensured governance and security. Migration challenges included maintaining stability, managing costs and minimizing latency. A phased approach, Delta Sharing, and DBx Asset Bundles simplified transitions. The result: faster insights, improved cost control and reduced onboarding time, fostering innovation and efficiency. We share our transformation, offering insights for modern cloud optimization.
Managing cloud costs requires accurate resource tagging, but maintaining completeness and accuracy is a challenge. In this session, Mitchell Ertle (Senior Partner Solutions Architect) and Josue Bogran (Data & AI Architect) demonstrate how Sigma and Databricks combine to streamline FinOps and resource management with AI-driven cost attribution and workflow automation.
Through a practical demonstration, you'll see:
Identify and classify untagged Databricks pipelines with a cost attribution app Use GenAI from Databricks to suggest tags with human-in-the-loop approval Enable bidirectional data flow between Sigma and Databricks for real-time updates Automate workflows with Sigma’s actions framework Ensure security and governance by inheriting Unity Catalog permissions Discover why this combination is powerful—Sigma provides intuitive application building while Databricks delivers computation, AI/ML capabilities, and data storage. These platforms create solutions business users can interact with directly, without technical expertise.
Whether in data engineering, finance, or operations, learn how Sigma + Databricks can automate workflows, optimize costs, and drive business impact.
➡️ Learn more about Data Apps: https://www.sigmacomputing.com/product/data-applications?utm_source=youtube&utm_medium=organic&utm_campaign=data_apps_conference&utm_content=pp_data_apps
➡️ Sign up for your free trial: https://www.sigmacomputing.com/go/free-trial?utm_source=youtube&utm_medium=video&utm_campaign=free_trial&utm_content=free_trial
Eric will share the team's experience with dbt and tell the development story of bringing Workday FinOps cost analysis to cloud engineers and stakeholders. He will describe how the team is using dbt, Trino and Lightdash to build a new data platform that is now a key part of their data-driven business decision process in multiple organizations within Workday. Plus, he'll show how they created a secure, efficient, and scalable platform — through dbt governance features — to drive those successful data projects.
Speakers: Eric Pu Senior Software Engineer Workday
Pattabhi Nanduri FinOps Data Engineer Workday
Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements
This talk was recorded at Crunch Conference 2022. Zoltán and Gergely from Aliz.ai company spoke about FinOps examples using Google Cloud BigQuery.
"In this talk we will talk about the basics of FinOps concept and going to make an introduction to it through real life examples using BigQuery."
The event was organized by Crafthub.
You can watch the rest of the conference talks on our channel.
If you are interested in more speakers, tickets and details of the conference, check out our website: https://crunchconf.com/ If you are interested in more events from our company: https://crafthub.events/
This session begins with data warehouse trivia and lessons learned from production implementations of multicloud data architecture. You will learn to design future-proof low latency data systems that focus on openness and interoperability. You will also gain a gentle introduction to Cloud FinOps principles that can help your organization reduce compute spend and increase efficiency.
Most enterprises today are multicloud. While an assortment of low-code connectors boasts the ability to make data available for analytics in real time, they post long-lasting challenges:
The alternative is an open data lake that unifies batch and streaming workloads. Bronze landing zones in open format eliminate the data extraction costs required by proprietary EDW. Apache Spark™ Structured Streaming provides a unified ingestion interface. Streaming triggers allow us to switch back and forth between batch and stream with one-line code changes. Streaming aggregation enables us to incrementally compute on data that arrives near each other.
Specific examples are given on how to use Autoloader to discover newly arrived data and ensure exactly once, incremental processing. How DLT can be configured effectively to further simplify streaming jobs and accelerate the development cycle. How to apply SWE best practices to Workflows and integrate with popular Git providers, either using the Databricks Project or Databricks Terraform provider.
Talk by: Christina Taylor
Here’s more to explore: Big Book of Data Engineering: 2nd Edition: https://dbricks.co/3XpPgNV The Data Team's Guide to the Databricks Lakehouse Platform: https://dbricks.co/46nuDpI
Connect with us: Website: https://databricks.com Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/databricks Instagram: https://www.instagram.com/databricksinc Facebook: https://www.facebook.com/databricksinc
This demo showcases Finout’s ability to manage and optimize cloud spend across Azure and services like Kubernetes, Datadog, and OpenAI. It highlights Finout’s unified “MegaBill” view for exploring Azure resources, subscriptions, and tags. The session introduces Virtual Tags for dynamic, rules-based cost allocation and covers shared cost distribution, dashboards, anomaly detection, and alerting—empowering teams to improve Azure cost efficiency.