Improving your security posture requires continuous evaluation. This lightning talk demonstrates how generative AI can analyze architectural diagrams to identify AWS security best practices from the AWS Well-Architected Framework. Elements like identity providers or secret management within architectural diagrams can provide insights into your architectural design and help you identify areas for improvement. Additionally, see how to apply this concept to infrastructure as code for deeper inspection of your cloud architecture against AWS security best practices.

Learn more about AWS re:Inforce at https://go.aws/reinforce.

Subscribe: More AWS videos: http://bit.ly/2O3zS75 More AWS events videos: http://bit.ly/316g9t4

ABOUT AWS Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts.

AWS is the world's most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

reInforce2024 #CloudSecurity #AWS #AmazonWebServices #CloudComputing

Learn the proven strategies and approaches to creating scalable infrastructure, as code, with HashiCorp Terraform. We'll cover how large organizations find success in structuring projects across teams, architect globally available systems, securely share sensitive information between environments, set up developer-friendly workflows, and more. This session will discuss these topics in-depth and show them in action with a live demo and codebase that deploys many services across multiple teams.

Click the blue “Learn more” button above to tap into special offers designed to help you implement what you are learning at Google Cloud Next 25.

Learn how to streamline application development using Infrastructure as Code (IaC) principles. This session explores building data APIs with declarative configurations and deploying them globally for seamless scalability. GraphQL empowers your teams to work independently within their domains, guided by a robust type system. Break down data silos, enable self-service data access across your enterprise, and watch your developers create exceptional user experiences. By attending this session, your contact information may be shared with the sponsor for relevant follow up for this event only.

Click the blue “Learn more” button above to tap into special offers designed to help you implement what you are learning at Google Cloud Next 25.

You’ve provisioned your cloud resources with infrastructure as code. A year later, you find yourself refactoring it to better manage resource usage and dependencies and add a new region, runtime, or project. In this session, you’ll learn the patterns and practices to go from zero to the multi-region, multi-runtime, multi-project, multicloud scale. By establishing the foundation for scale in the beginning, you can lessen your refactoring effort and add new runtimes, regions, projects, and even clouds to your system.

Click the blue “Learn more” button above to tap into special offers designed to help you implement what you are learning at Google Cloud Next 25.

Platform/ IT admins: Ever experienced delays setting up a foundational infrastructure on Google Cloud? Look no further than Google's new solution for setting up a foundational infrastructure -- a mix of UI and Infrastructure as Code. Now available to all customers globally in the console, the Google Cloud setup product will guide you through setting up an organization resource, integrating with a 3p identity provider, setting up initial folder/project structure, IAM, Shared VPCs, hybrid networking and more. You can deploy directly from the console or download as Terraform. In this session will demo the new guided product and share how you can test and provide feedback.

Click the blue “Learn more” button above to tap into special offers designed to help you implement what you are learning at Google Cloud Next 25.

The best way to fix security issues is to prevent them in the first place. Join this technical session and learn how to protect cloud workloads with new security posture controls that define and monitor operating guardrails, and new infrastructure as code (IaC) scanning that can identify security gaps before cloud environments go live. Learn how Lloyds Banking Group meets stringent governance and compliance requirements with Security Command Center, while supporting accelerated change.

Click the blue “Learn more” button above to tap into special offers designed to help you implement what you are learning at Google Cloud Next 25.

Learn how Landis+Gyr, a leading manufacturer and operator of smart meters, modernized their legacy systems and moved to a cloud-first architecture using infrastructure as code. The session reflects on how Landis+Gyr embarked on their journey from manually provisioning hardware and compute environments for individual smart metering customer wins to harnessing the power of cloud and infrastructure as code to support a large-scale modernization and replatforming project of Landis+Gyr’s backend for smart meter data and data-driven services.

Click the blue “Learn more” button above to tap into special offers designed to help you implement what you are learning at Google Cloud Next 25.

Learn about Wells Fargo’s journey to next-generation logging, with focus on the architecture that handles logging at scale for the third-largest U.S. Bank. We'll explore topics such as: architecture supporting infrastructure serving over 70 million customers; multicloud integration across on-premises, software as a service, and public cloud; programmatic and self-service migration of dashboards, alerts, and queries; logging infrastructure as code; sensitive data handling; Gemini accelerating troubleshooting and usability, and much more.

Click the blue “Learn more” button above to tap into special offers designed to help you implement what you are learning at Google Cloud Next 25.

Learn how to deploy your first summarization Gen AI application easily with new Jump Start Solution in 7 minutes. Then we will show how to customize your deployment (e.g. change prompt to auto customer compliant reply) with terraform in order to achieve Gen AI application Infrastructure as Code for better automation, security, efficiency, reliability, consistency and speed to market.

Click the blue “Learn more” button above to tap into special offers designed to help you implement what you are learning at Google Cloud Next 25.

Explore the depths of subsurface analysis with Maciej Marek as he unveils 'Peer Beneath the Surface of the Earth.' 🌍📡 Discover Widmo's innovative Spectral Ground Penetrating Radar and their journey from single to multi-cloud solutions, facing challenges along the way. Join the session to learn about data warehousing, Infrastructure as Code, and the role of AI in automation! 💡🛠️ #SubsurfaceAnalysis #Innovation #MultiCloud

Join Angelika Postaremczak in an enlightening session on 'Best Practices for Storing Data in BigQuery' and discover the keys to optimizing data storage for lightning-fast queries without breaking the bank! 🚀💾 Explore table design strategies, data partitioning, clustering, and resource management through Infrastructure as Code for maximizing the potential of cloud data storage. ☁️📊 #BigQuery #datastorage

✨ H I G H L I G H T S ✨

🙌 A huge shoutout to all the incredible participants who made Big Data Conference Europe 2023 in Vilnius, Lithuania, from November 21-24, an absolute triumph! 🎉 Your attendance and active participation were instrumental in making this event so special. 🌍

Don't forget to check out the session recordings from the conference to relive the valuable insights and knowledge shared! 📽️

Once again, THANK YOU for playing a pivotal role in the success of Big Data Conference Europe 2023. 🚀 See you next year for another unforgettable conference! 📅 #BigDataConference #SeeYouNextYear

In today's cloud ecosystem, many laud the visible pillars of AWS's Well-Architected Framework, yet an essential component often remains in the shadows: Infrastructure as Code (IAC). Elizabeth Adeotun Adegbaju, a DevOps Engineer with a rich history in AWS cloud infrastructure, unravels the indispensable role of IAC in fortifying each of the renowned AWS pillars. Through this illuminating talk, attendees will gain insights into the intricate interplay between IAC and AWS's principles of operational excellence, cost optimization, reliability, performance efficiency, security, and sustainability. Dive deep into real-world examples, understand the potential pitfalls of overlooking IAC, and emerge with a renewed appreciation for its foundational significance in cloud architecture. This session is a clarion call for organizations to recognize and harness the power of IAC, positioning it not just as an option but as an imperative in achieving success in the cloud.

This talk discusses “software-defined assets”, a declarative approach to orchestration and data management that makes it drastically easier to trust and evolve datasets and ML models. Dagster is an open source orchestrator built for maintaining software-defined assets.

In traditional data platforms, code and data are only loosely coupled. As a consequence, deploying changes to data feels dangerous, backfills are error-prone and irreversible, and it’s difficult to trust data, because you don’t know where it comes from or how it’s intended to be maintained. Each time you run a job that mutates a data asset, you add a new variable to account for when debugging problems.

Dagster proposes an alternative approach to data management that tightly couples data assets to code - each table or ML model corresponds to the function that’s responsible for generating it. This results in a “Data as Code” approach that mimics the “Infrastructure as Code” approach that’s central to modern DevOps. Your git repo becomes your source of truth on your data, so pushing data changes feels as safe as pushing code changes. Backfills become easy to reason about. You trust your data assets because you know how they’re computed and can reproduce them at any time. The role of the orchestrator is to ensure that physical assets in the data warehouse match the logical assets that are defined in code, so each job run is a step towards order.

Software-defined assets is a natural approach to orchestration for the modern data stack, in part because dbt models are a type of software-defined asset.

Attendees of this session will learn how to build and maintain lakehouses of software-defined assets with Dagster.

Connect with us: Website: https://databricks.com Facebook: https://www.facebook.com/databricksinc Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/data... Instagram: https://www.instagram.com/databricksinc/

Profiling small molecules in human blood across global populations gives rise to a greater understanding of the varied biological pathways and processes that contribute to human health and diseases. Herein, we describe the development of a comprehensive Human Biology Database, derived from nontargeted molecular profiling of over 300,000 human blood samples from individuals across diverse backgrounds, demographics, geographical locations, lifestyles, diseases, and medication regimens, and its applications to inform drug development.

Approximately 11,000 circulating molecules have been captured and measured per sample using Sapient’s high-throughput, high-specificity rapid liquid chromatography-mass spectrometry (rLC-MS) platform. The samples come from cohorts with adjudicated clinical outcomes from prospective studies lasting 10-25 years, as well as data on individuals’ diet, nutrition, physical exercise, and mental health. Genetic information for a subset of subjects is also included and we have added microbiome sequencing data from over 150,000 human samples in diverse diseases.

An efficient data science environment is established to enable effective health insight mining across this vast database. Built on a customized AWS and Databricks “infrastructure-as-code” Terraform configuration, we employ streamlined data ETL and machine learning-based approaches for rapid rLC-MS data extraction. In mining the database, we have been able to identify circulating molecules potentially causal to disease; illuminate the impact of human exposures like diet and environment on disease development, aging, and mortality over decades of time; and support drug development efforts through identification of biomarkers of target engagement, pharmacodynamics, safety, efficacy, and more.

Connect with us: Website: https://databricks.com Facebook: https://www.facebook.com/databricksinc Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/data... Instagram: https://www.instagram.com/databricksinc/

Summary Building and maintaining reliable data assets is the prime directive for data engineers. While it is easy to say, it is endlessly complex to implement, requiring data professionals to be experts in a wide range of disparate topics while designing and implementing complex topologies of information workflows. In order to make this a tractable problem it is essential that engineers embrace automation at every opportunity. In this episode Chris Riccomini shares his experiences building and scaling data operations at WePay and LinkedIn, as well as the lessons he has learned working with other teams as they automated their own systems.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show! RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder. Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer. Your host is Tobias Macey and today I’m interviewing Chris Riccomini about building awareness of data usage into CI/CD pipelines for application development

Interview

Introduction How did you get involved in the area of data management? What are the pieces of data platforms and processing that have been most difficult to scale in an organizational sense? What are the opportunities for automation to alleviate some of the toil that data and analytics engineers get caught up in? The application delivery ecosystem has been going through ongoing transformation in the form of CI/CD, infrastructure as code, etc. What are the parallels in the data ecosystem that are still nascent? What are the principles that still need to be translated for data practitioners? Which are subject to impedance mismatch and may never make sense to translate? As someone with a software engineering background and extensive e

Learn how to seamlessly modernize a Linux application stack in Azure. Participants will be able to step through the migration process with a Linux/Postgres/Java application and utilize infrastructure as a code (IaC) for scale migrations.