- Vancouver Airflow Meetup at Amazon!AWS Offices, Vancouver, BC
Join Airflow experts from Stripe, AWS, and Astronomer for an evening of talks, networking, and delicious food and drinks at Amazon's Offices!
Join our three speakers as they delve into the exciting new features of the upcoming Airflow 3 release and share best practices for safely testing your Airflow pipelines—all while you enjoy delicious food and drinks!
WHAT WE'LL COVER:
What's Coming in Airflow 3
- Speaker: Kenten Danas, Developer Relations Manager at Astronomer
- Apache Airflow has become the de facto standard for data orchestration, with growing adoption in machine learning and AI use cases over recent years. Nearly four years have passed since the release of Airflow 2, and as a community, we've reached consensus that it’s time for a major, foundational update: Airflow 3—now just around the corner. In this talk, we’ll introduce the vision behind Airflow 3, highlighting key industry trends and how Airflow will adapt to meet these needs. We’ll cover the architectural enhancements designed to support emerging use cases and distributed data infrastructure models. Attendees will gain insight into the major new features, the intended outcomes of the release, and the foundational concepts of Airflow 3 that will continue evolving in subsequent 3.x releases.
Stress-Free Airflow development: From Dev to Prod at Stripe
- Speaker: Nick Bilozerov, Sr. Software Engineer at Stripe
- At Stripe, compliance with regulations is of utmost importance, and ensuring the integrity of production data is crucial. To address this challenge, Stripe developed a powerful system called User Scope Mode (USM), which allows users to safely and efficiently test new or existing Airflow pipelines without the risk of corrupting production data. USM takes care of automatically overwriting the necessary configurations for Airflow pipelines, enabling users to test their production-ready pipelines locally with ease. This approach empowers Stripe’s teams to iterate and refine their workflows without the burden of manual setup or the fear of disrupting live operations.
In this talk, we’ll dive into the inner workings of USM and explore how it has transformed Stripe’s development and testing processes. Discover how this system seamlessly integrates with Airflow, allowing users to validate their pipelines with confidence and agility, all while maintaining the highest standards of compliance and data integrity.
Beyond SLAs: A Deep Dive into Airflow 3's New Deadline Alerts
- Speaker: Dennis Ferruzzi, Software Developer at Amazon & Apache Airflow Committer
- Do you have a DAG that needs to be done by a certain time? Have you tried to use Airflow's existing SLA feature and found it restrictive or complicated? In this talk we'll take a look at the all-new Deadline Alerts feature coming in Airflow 3 which replaces and revolutionizes SLA. We'll discuss how Deadline Alerts work and how they're different than the previous SLA feature. Then we'll look at some examples of workflows you can build with the new feature. Finally, we'll walk through some future usecases of using Deadlines for Tasks and even Assets.
AGENDA
- 5:00 PM-5:30 PM: Arrivals, Eat, Drink and Network!
- 5:30 PM-7:00 PM: Presentations!
- 7:00 PM- 8:00 PM: Eat more, drink more, and network!
