The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI Tue, 13 Oct 2026 16:35:52 -0400 https://airflow.apache.org/ en © 2026 The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI yes 50fd8116-be73-4852-be37-94d8bc0082fa kenten@astronomer.io Astronomer episodic false Buzzsprout (https://www.buzzsprout.com) Astronomer kenten@astronomer.io https://storage.buzzsprout.com/n014xqg93jtsv98pzl2uxwbo21vc?.jpg The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI https://airflow.apache.org/ Scaling to 1,000+ Airflow Dags at Descartes / Idelic Scaling to 1,000+ Airflow Dags at Descartes / Idelic Scaling Airflow to over 1,000 active DAGs on a team of 3-4 engineers takes structure. In this episode, Kenten Danas talks with Matt Stavinga, Data Engineer at Idelic (now part of Descartes), about how his team standardized DAG creation and logging, built a Jenkins-driven testing workflow with per-PR Airflow environments, and is opening up DAG configuration and triggering to their own customers through the Safety Suite UI.


Key Takeaways:

00:00 Introduction.

01:40 Idelic's mission: collecting data from North American trucking fleets and using it to identify risky drivers so fleets can intervene before accidents happen.

02:59 Where Airflow fits: Idelic runs on Astro, with each customer/integration combination as its own DAG.

03:56 The reality of vendor integrations: APIs, SFTP, email, screen scraping, and even executables installed on customer servers to push data out.

06:11 How the DAG count grew and where things started to hurt as a small team tried to keep up.

08:49 The standardized framework: abstract methods that wrap common Airflow functions and generate task groups for each ETL phase, plus standardized logging across vendors.

11:15 CI/CD with Jenkins: per-PR Airflow deployments that QA can spin up on demand through a Jira hook and tear back down when done.

13:31 Letting customers create their own DAGs through a wizard in Safety Suite, with plans for customer-triggered runs, filtered result review, and human-in-the-loop approval.

15:37 Using the Airflow API to trigger customer DAGs, and why Idelic pushes run data into its own database instead of relying on the Airflow logs API.

18:58 What Matt wants next from Airflow: task and DAG level trend tracking, and more from Otto on self-healing DAGs.


Resources Mentioned:

Airflow Survey: astronomer.typeform.com/airflowsurvey26

Astro by Astronomer: astronomer.io

Apache Airflow: airflow.apache.org

Otto (Astronomer's data engineering agent): https://www.astronomer.io/product/otto/


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#ai #orchestration #airflow



]]> Scaling Airflow to over 1,000 active DAGs on a team of 3-4 engineers takes structure. In this episode, Kenten Danas talks with Matt Stavinga, Data Engineer at Idelic (now part of Descartes), about how his team standardized DAG creation and logging, built a Jenkins-driven testing workflow with per-PR Airflow environments, and is opening up DAG configuration and triggering to their own customers through the Safety Suite UI.


Key Takeaways:

00:00 Introduction.

01:40 Idelic's mission: collecting data from North American trucking fleets and using it to identify risky drivers so fleets can intervene before accidents happen.

02:59 Where Airflow fits: Idelic runs on Astro, with each customer/integration combination as its own DAG.

03:56 The reality of vendor integrations: APIs, SFTP, email, screen scraping, and even executables installed on customer servers to push data out.

06:11 How the DAG count grew and where things started to hurt as a small team tried to keep up.

08:49 The standardized framework: abstract methods that wrap common Airflow functions and generate task groups for each ETL phase, plus standardized logging across vendors.

11:15 CI/CD with Jenkins: per-PR Airflow deployments that QA can spin up on demand through a Jira hook and tear back down when done.

13:31 Letting customers create their own DAGs through a wizard in Safety Suite, with plans for customer-triggered runs, filtered result review, and human-in-the-loop approval.

15:37 Using the Airflow API to trigger customer DAGs, and why Idelic pushes run data into its own database instead of relying on the Airflow logs API.

18:58 What Matt wants next from Airflow: task and DAG level trend tracking, and more from Otto on self-healing DAGs.


Resources Mentioned:

Airflow Survey: astronomer.typeform.com/airflowsurvey26

Astro by Astronomer: astronomer.io

Apache Airflow: airflow.apache.org

Otto (Astronomer's data engineering agent): https://www.astronomer.io/product/otto/


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#ai #orchestration #airflow



]]> briana.okyere@astronomer.io (Astronomer) 746abb61-fc4f-4ae6-987b-70300e53c649 Thu, 08 Oct 2026 08:00:00 -0400 1375 1 98 false Orchestrating local LLMs for sensitive data with Airflow Orchestrating local LLMs for sensitive data with Airflow Running LLMs over sensitive data means external APIs are off the table, but open-weight models still need reproducibility, validation, and a paper trail before downstream systems can depend on their output. In this episode, Kenten is joined by Chhayank Jain, Senior ML Engineer at Visa, to talk through how Airflow orchestrates local LLM pipelines end to end: nightly inference over millions of records, silent failure handling with Pydantic and distribution checks, and using a separate DAG to manage LoRA fine-tuning cycles.


Key Takeaways:


00:00 Introduction.


01:47 From modeling to pipelines. Chhayank shares how leading a readmission risk model across hospitals with different EHR systems turned him into a "pipeline person."


03:14 Day-to-day as an LLM engineer. Serving with ONNX Runtime, failure design, and agentic tooling with LangGraph. The model is only about one-fifth of the work.


05:03 The sensitive data problem. Why hosted LLM APIs are off the table for PHI and payments data, and why the real challenge is running Llama 3 nightly with enough checks that downstream models can trust the output.


07:34 Where Airflow fits. Sensors per site, dependency ordering, retries and backfills, and run records as the "paper trail" for audits.


10:03 DAG structure. Four stages: ingest and normalize, inference on GPU pods via the KubernetesPodOperator, validation, and load to the feature store.


12:32 Loud vs silent failures. OOM kills and timeouts are easy. Silent failures where the output looks clean but is wrong are the scary category, handled with Pydantic schemas, cross-checks against structured records, and distribution monitoring.


16:45 LoRA fine-tuning in its own DAG. A separate monthly or quarterly DAG pulls labeled data, fine-tunes with Hugging Face PEFT on top of Llama 3, and only writes a new adapter if it beats the frozen holdout.


20:33 Hardest production lessons. Pin everything (model version, temperature, features, seed), match parallelism to GPU capacity, and invest in traceability so you can answer "which adapter produced this row" six months later.


23:00 Airflow wishlist. First-class GPU awareness so teams stop reinventing it through pools and queues, and treating model versions as Airflow assets.


Resources Mentioned:

- Annual Airflow survey

- Airflow Summit talk: From Hours to Minutes: Orchestrating Local LLMs for Sensitive Data Pipelines with Apache Airflow

- Llama 3

- Hugging Face PEFT

- ONNX Runtime

- LangGraph

- Pydantic


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Orchestration #Airflow



]]> Running LLMs over sensitive data means external APIs are off the table, but open-weight models still need reproducibility, validation, and a paper trail before downstream systems can depend on their output. In this episode, Kenten is joined by Chhayank Jain, Senior ML Engineer at Visa, to talk through how Airflow orchestrates local LLM pipelines end to end: nightly inference over millions of records, silent failure handling with Pydantic and distribution checks, and using a separate DAG to manage LoRA fine-tuning cycles.


Key Takeaways:


00:00 Introduction.


01:47 From modeling to pipelines. Chhayank shares how leading a readmission risk model across hospitals with different EHR systems turned him into a "pipeline person."


03:14 Day-to-day as an LLM engineer. Serving with ONNX Runtime, failure design, and agentic tooling with LangGraph. The model is only about one-fifth of the work.


05:03 The sensitive data problem. Why hosted LLM APIs are off the table for PHI and payments data, and why the real challenge is running Llama 3 nightly with enough checks that downstream models can trust the output.


07:34 Where Airflow fits. Sensors per site, dependency ordering, retries and backfills, and run records as the "paper trail" for audits.


10:03 DAG structure. Four stages: ingest and normalize, inference on GPU pods via the KubernetesPodOperator, validation, and load to the feature store.


12:32 Loud vs silent failures. OOM kills and timeouts are easy. Silent failures where the output looks clean but is wrong are the scary category, handled with Pydantic schemas, cross-checks against structured records, and distribution monitoring.


16:45 LoRA fine-tuning in its own DAG. A separate monthly or quarterly DAG pulls labeled data, fine-tunes with Hugging Face PEFT on top of Llama 3, and only writes a new adapter if it beats the frozen holdout.


20:33 Hardest production lessons. Pin everything (model version, temperature, features, seed), match parallelism to GPU capacity, and invest in traceability so you can answer "which adapter produced this row" six months later.


23:00 Airflow wishlist. First-class GPU awareness so teams stop reinventing it through pools and queues, and treating model versions as Airflow assets.


Resources Mentioned:

- Annual Airflow survey

- Airflow Summit talk: From Hours to Minutes: Orchestrating Local LLMs for Sensitive Data Pipelines with Apache Airflow

- Llama 3

- Hugging Face PEFT

- ONNX Runtime

- LangGraph

- Pydantic


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Orchestration #Airflow



]]> briana.okyere@astronomer.io (Astronomer) b4ad5901-99a9-4f8d-8e4e-27f64908874c Thu, 01 Oct 2026 08:00:00 -0400 1612 1 97 false Improving security in Airflow 3 Improving security in Airflow 3 Security is one of the least visible parts of Airflow for most users, but it shapes what features teams can turn on in production. In this episode, Kenten Danas talks with Anish Giri, Senior Software Engineer and active Apache Airflow contributor focused on security, about three recent contributions that touch nearly every Airflow 3 deployment: hardening connection testing, adding JWT revocation, and splitting execution tokens by scope.


Key Takeaways:


00:00 Introduction.

01:16 Anish's inside-out path into Airflow, starting from source code and docs rather than running pipelines in production.

03:13 Why security engineering flips the usual software question from "what can this do?" to "what can anyone make this to?"

04:49 How connection testing worked before, why it was disabled in Airflow 2.7 after a CVE, and how the redesign moves execution off the API server.

06:26 Running connection tests on the worker in Airflow 3 so credentials never leave the API server boundary.

07:50 What the mailing list and architecture review looked like, including a first-round rejection and eventual consensus.

10:00 The JWT logout problem in Airflow 3 and the revocation table that makes logout an actual logout.

13:12 A production incident with a GPU pipeline task hitting 403 after 15 minutes, and why it required splitting the token in two.

15:41 How the workload token and the auth token now have different lifetimes and scopes, tied to the task queue timeout.

19:53 Advice for new Airflow contributors: start small, don't fear the PR review, and treat feedback as mentorship.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Orchestration #Airflow



]]> Security is one of the least visible parts of Airflow for most users, but it shapes what features teams can turn on in production. In this episode, Kenten Danas talks with Anish Giri, Senior Software Engineer and active Apache Airflow contributor focused on security, about three recent contributions that touch nearly every Airflow 3 deployment: hardening connection testing, adding JWT revocation, and splitting execution tokens by scope.


Key Takeaways:


00:00 Introduction.

01:16 Anish's inside-out path into Airflow, starting from source code and docs rather than running pipelines in production.

03:13 Why security engineering flips the usual software question from "what can this do?" to "what can anyone make this to?"

04:49 How connection testing worked before, why it was disabled in Airflow 2.7 after a CVE, and how the redesign moves execution off the API server.

06:26 Running connection tests on the worker in Airflow 3 so credentials never leave the API server boundary.

07:50 What the mailing list and architecture review looked like, including a first-round rejection and eventual consensus.

10:00 The JWT logout problem in Airflow 3 and the revocation table that makes logout an actual logout.

13:12 A production incident with a GPU pipeline task hitting 403 after 15 minutes, and why it required splitting the token in two.

15:41 How the workload token and the auth token now have different lifetimes and scopes, tied to the task queue timeout.

19:53 Advice for new Airflow contributors: start small, don't fear the PR review, and treat feedback as mentorship.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Orchestration #Airflow



]]> briana.okyere@astronomer.io (Astronomer) 0201addb-e9f9-4513-b318-8f7f85a71815 Thu, 24 Sep 2026 08:00:00 -0400 1337 1 96 false Orchestrating data across 30 companies at itti Orchestrating data across 30 companies at itti Orchestrating data across more than 30 companies means most Airflow users on the platform aren't data engineers. In this episode, Kenten Danas talks with Lucas Trubiano, Data Engineer at itti, the technology company within Grupo Vázquez in Paraguay. Lucas walks through the custom YAML framework his Center of Excellence built on top of Airflow, how they baked data quality and custom operators into it, and how a spec-driven AI workflow now lets product and business users contribute to templates without knowing Python.


Key Takeaways:

00:00 Introduction.

01:47 What itti and Grupo Vázquez do, and the Data Engineering Center of Excellence's mandate to build a 360-degree view of the customer across more than 30 companies.

02:56 How Airflow fits in as a central task orchestrator (not a processing engine) across around 300 production DAGs.

04:25 Managing enterprise-scale Airflow: preferring Airflow-as-a-service, plus enabling self-service for non-technical users through YAML.

05:44 Why itti built a second, more opinionated YAML framework after DAG Factory-style customization created a code review bottleneck.

07:14 More than 80% of new DAGs are now created with the new framework because it's simply faster.

07:54 How the framework works end to end: Python DAGs, Jinja templates, YAML configs, and CI/CD compilation.

09:10 A Google Sheets ingestion example that shows how prevalidation, download, and processing tasks are hidden behind a simple YAML config to preserve reliability.

10:14 Building an in-house data quality tool that tests per partition instead of full-scanning tables, triggered via custom Airflow operators.

11:17 Custom operators for dbt, the in-house data quality tool, and AWS services like QuickSight dashboard refreshes, and how OSS Airflow makes them portable across instances.

15:35 Spec-driven development with a fork of GitHub Spec Kit so business users can describe what they want and let agents generate DAGs against certified templates.

17:24 Slack-native error routing: every DAG has an owner team, common errors ship with explanations, and only deep issues escalate to the central team.

19:29 Where they're heading next: agent-triggered pull requests for self-healing pipelines.

20:22 Airflow 3 wishlist: backfill improvements, event-driven orchestration for streaming pipelines, and Human in the Loop for generative AI DAGs.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow



]]> Orchestrating data across more than 30 companies means most Airflow users on the platform aren't data engineers. In this episode, Kenten Danas talks with Lucas Trubiano, Data Engineer at itti, the technology company within Grupo Vázquez in Paraguay. Lucas walks through the custom YAML framework his Center of Excellence built on top of Airflow, how they baked data quality and custom operators into it, and how a spec-driven AI workflow now lets product and business users contribute to templates without knowing Python.


Key Takeaways:

00:00 Introduction.

01:47 What itti and Grupo Vázquez do, and the Data Engineering Center of Excellence's mandate to build a 360-degree view of the customer across more than 30 companies.

02:56 How Airflow fits in as a central task orchestrator (not a processing engine) across around 300 production DAGs.

04:25 Managing enterprise-scale Airflow: preferring Airflow-as-a-service, plus enabling self-service for non-technical users through YAML.

05:44 Why itti built a second, more opinionated YAML framework after DAG Factory-style customization created a code review bottleneck.

07:14 More than 80% of new DAGs are now created with the new framework because it's simply faster.

07:54 How the framework works end to end: Python DAGs, Jinja templates, YAML configs, and CI/CD compilation.

09:10 A Google Sheets ingestion example that shows how prevalidation, download, and processing tasks are hidden behind a simple YAML config to preserve reliability.

10:14 Building an in-house data quality tool that tests per partition instead of full-scanning tables, triggered via custom Airflow operators.

11:17 Custom operators for dbt, the in-house data quality tool, and AWS services like QuickSight dashboard refreshes, and how OSS Airflow makes them portable across instances.

15:35 Spec-driven development with a fork of GitHub Spec Kit so business users can describe what they want and let agents generate DAGs against certified templates.

17:24 Slack-native error routing: every DAG has an owner team, common errors ship with explanations, and only deep issues escalate to the central team.

19:29 Where they're heading next: agent-triggered pull requests for self-healing pipelines.

20:22 Airflow 3 wishlist: backfill improvements, event-driven orchestration for streaming pipelines, and Human in the Loop for generative AI DAGs.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow



]]> briana.okyere@astronomer.io (Astronomer) 7bc60f90-fe5e-4f07-94ba-59ddc2f34ac3 Thu, 17 Sep 2026 08:00:00 -0400 1170 1 95 false Building self-healing Airflow pipelines at ATC Drivetrain Building self-healing Airflow pipelines at ATC Drivetrain Manufacturing data pipelines can't afford silent failures. When quality decisions and shop-floor visibility depend on Airflow, a broken DAG at midnight can cost real money. In this episode, Kenten Danas talks with Kumuda Sreenivasa, Founder of Receitly and Senior Data Architect at ATC Drivetrain, about how her team built self-healing Airflow pipelines, where AI fits into the recovery loop, and how they orchestrate AI agents as governed workflow components.


Key Takeaways:

00:00 Introduction.

02:29 How ATC Drivetrain uses Airflow across thousands of DAGs to orchestrate ETL/ELT jobs, data quality checks, and production reporting for a complex automotive remanufacturing environment.

04:08 Defining self-healing: pipelines that identify a known failure, decide whether they can recover safely, execute an approved action, and validate the result, all without paging an engineer.

07:03 The five-layer self-healing architecture: observe, classify, policy, recover, and validate.

10:00 Where AI fits in the recovery loop: classification, context gathering, and recommendations, but never bypassing operational policies or approval steps.

13:00 A concrete before/after: a currency-exchange failure caught overnight by AI-assisted recovery that saved four hours of downtime and roughly 120K.

14:27 Confidence levels and success rates: about 95% of small failure modes recover on their own.

15:34 AI-assisted troubleshooting at scale: how contextual log analysis and recommended actions save engineers from digging through thousands of log lines.

20:13 Orchestrating AI agents through Airflow: treating agents as bounded, governed workflow components with human approval and confidence-based stop conditions.

23:45 What Kumuda wants next from Airflow: stronger AI agent governance, standardized tracking of prompts and tool calls, and more flexible event-driven execution.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow



]]> Manufacturing data pipelines can't afford silent failures. When quality decisions and shop-floor visibility depend on Airflow, a broken DAG at midnight can cost real money. In this episode, Kenten Danas talks with Kumuda Sreenivasa, Founder of Receitly and Senior Data Architect at ATC Drivetrain, about how her team built self-healing Airflow pipelines, where AI fits into the recovery loop, and how they orchestrate AI agents as governed workflow components.


Key Takeaways:

00:00 Introduction.

02:29 How ATC Drivetrain uses Airflow across thousands of DAGs to orchestrate ETL/ELT jobs, data quality checks, and production reporting for a complex automotive remanufacturing environment.

04:08 Defining self-healing: pipelines that identify a known failure, decide whether they can recover safely, execute an approved action, and validate the result, all without paging an engineer.

07:03 The five-layer self-healing architecture: observe, classify, policy, recover, and validate.

10:00 Where AI fits in the recovery loop: classification, context gathering, and recommendations, but never bypassing operational policies or approval steps.

13:00 A concrete before/after: a currency-exchange failure caught overnight by AI-assisted recovery that saved four hours of downtime and roughly 120K.

14:27 Confidence levels and success rates: about 95% of small failure modes recover on their own.

15:34 AI-assisted troubleshooting at scale: how contextual log analysis and recommended actions save engineers from digging through thousands of log lines.

20:13 Orchestrating AI agents through Airflow: treating agents as bounded, governed workflow components with human approval and confidence-based stop conditions.

23:45 What Kumuda wants next from Airflow: stronger AI agent governance, standardized tracking of prompts and tool calls, and more flexible event-driven execution.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow



]]> briana.okyere@astronomer.io (Astronomer) ca1ae71d-da2f-49dc-923b-96731b0113f7 Thu, 10 Sep 2026 04:00:00 -0400 1622 1 94 false Orchestrating strictly sequential ETL pipelines at Synechron Orchestrating strictly sequential ETL pipelines at Synechron Strict sequential execution across DAGs sounds simple until you have scheduled pipelines and event-driven pipelines writing to the same MongoDB collections. Ivana Isailovic, Senior Big Data Engineer at Synechron, joins Marc Lamberti to walk through the three-layer DAG architecture her team built to solve exactly that, plus how they generate 200-task DAGs and how they rebuilt subdag-style group retries in Airflow 3.


Key Takeaways:

(00:00) Introduction.

(02:16) The stack: Snowflake source, MongoDB with a medallion (bronze, silver, gold) layout, Spark for processing, Airflow for orchestration, Elasticsearch for reports.

(05:29) Why standard Airflow options (max active runs, pools, dependency setups) each solved only part of the problem.

(06:40) Data consistency across bronze, silver, and gold layers is what forced strict sequential execution.

(09:11) Scheduled DAGs versus event-driven DAGs triggered at any moment from the application side.

(10:04) The three-layer architecture: trigger DAGs, a single proxy DAG that controls the queue, and main ETL DAGs.

(12:00) The queue is literally another DAG. The proxy DAG allows only one active run and serializes everything behind it.

(15:13) 200-task DAGs generated from nested task groups and YAML configuration files, with DAG versions tied to release numbers.

(17:38) How the layers talk to each other: sensors and TriggerDagRunOperator.

(19:44) Migrating from subdags to task groups without losing the ability to retry a whole group.

(21:49) Airflow 2.9 approach: reset task instance state via the metadata DB, keyed off the task group identifier.

(23:13) Airflow 3 approach: move the retry logic onto the official REST API for stability, security, and maintainability.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow



]]> Strict sequential execution across DAGs sounds simple until you have scheduled pipelines and event-driven pipelines writing to the same MongoDB collections. Ivana Isailovic, Senior Big Data Engineer at Synechron, joins Marc Lamberti to walk through the three-layer DAG architecture her team built to solve exactly that, plus how they generate 200-task DAGs and how they rebuilt subdag-style group retries in Airflow 3.


Key Takeaways:

(00:00) Introduction.

(02:16) The stack: Snowflake source, MongoDB with a medallion (bronze, silver, gold) layout, Spark for processing, Airflow for orchestration, Elasticsearch for reports.

(05:29) Why standard Airflow options (max active runs, pools, dependency setups) each solved only part of the problem.

(06:40) Data consistency across bronze, silver, and gold layers is what forced strict sequential execution.

(09:11) Scheduled DAGs versus event-driven DAGs triggered at any moment from the application side.

(10:04) The three-layer architecture: trigger DAGs, a single proxy DAG that controls the queue, and main ETL DAGs.

(12:00) The queue is literally another DAG. The proxy DAG allows only one active run and serializes everything behind it.

(15:13) 200-task DAGs generated from nested task groups and YAML configuration files, with DAG versions tied to release numbers.

(17:38) How the layers talk to each other: sensors and TriggerDagRunOperator.

(19:44) Migrating from subdags to task groups without losing the ability to retry a whole group.

(21:49) Airflow 2.9 approach: reset task instance state via the metadata DB, keyed off the task group identifier.

(23:13) Airflow 3 approach: move the retry logic onto the official REST API for stability, security, and maintainability.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow



]]> briana.okyere@astronomer.io (Astronomer) 3eceb662-7da7-40a6-82ad-10b5fdf34c71 Thu, 03 Sep 2026 12:00:00 -0400 1468 1 93 false Managing financial datasets with Airflow at Wise Managing financial datasets with Airflow at Wise Financial data pipelines have to be right the first time. On this episode, Kenten sits down with [Antonello Benedetto](linkedin.com/in/anbento4), Staff Data Engineer at Wise, to talk about how the central data and analytics engineering team runs Airflow for critical financial datasets, tiers pipelines by reliability, and orchestrates LLM-enabled workflows with validation layers and agent-checking-agent patterns.


Key Takeaways:

  • 00:00 Introduction.
  • 01:47 What Wise does and Antonello's role in the central data and analytics engineering team, a hybrid platform-plus-analytics team that owns dbt infrastructure, BI, and Analytics MCPs as a service.
  • 05:53 Three principles that guide Airflow pipeline design at Wise: a clean separation between orchestration and computation logic, computational awareness (offloading memory-intensive tasks to EMR or SageMaker), and standardized deployments.
  • 07:21 Why Wise treats Airflow as a pure orchestration layer and pushes memory-intensive work to external workers.
  • 08:45 Moving to the Python Virtual Environment Operator to standardize Airflow deployments across the org while giving analysts and data scientists per-job Python environments.
  • 10:50 The tiering system for pipelines, how it distinguishes highly controlled, well-documented, well-observed pipelines from newer ones, and how requirements from downstream drive tier promotion.
  • 17:18 Where LLM-enabled workflows differ from standard pipelines: validation layers for specific use cases, plus observability and evaluation platforms that track model performance across executions.
  • 19:08 Using Airflow to orchestrate LLM generation of monthly variance commentary for analysts.
  • 21:10 Handling non-idempotent LLM outputs with multi-layer validation against source-of-truth data, and using a second agent (CI/CD style) to validate the first agent's output.
  • 23:04 How AI-enabled workflow orchestration differs from batch ETL, and why teams should start small before building fully agentic pipelines.
  • 25:25 What Antonello would most like to see from Airflow next: native support for agentic workflows and better local development that mirrors production.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

]]> Financial data pipelines have to be right the first time. On this episode, Kenten sits down with [Antonello Benedetto](linkedin.com/in/anbento4), Staff Data Engineer at Wise, to talk about how the central data and analytics engineering team runs Airflow for critical financial datasets, tiers pipelines by reliability, and orchestrates LLM-enabled workflows with validation layers and agent-checking-agent patterns.


Key Takeaways:

  • 00:00 Introduction.
  • 01:47 What Wise does and Antonello's role in the central data and analytics engineering team, a hybrid platform-plus-analytics team that owns dbt infrastructure, BI, and Analytics MCPs as a service.
  • 05:53 Three principles that guide Airflow pipeline design at Wise: a clean separation between orchestration and computation logic, computational awareness (offloading memory-intensive tasks to EMR or SageMaker), and standardized deployments.
  • 07:21 Why Wise treats Airflow as a pure orchestration layer and pushes memory-intensive work to external workers.
  • 08:45 Moving to the Python Virtual Environment Operator to standardize Airflow deployments across the org while giving analysts and data scientists per-job Python environments.
  • 10:50 The tiering system for pipelines, how it distinguishes highly controlled, well-documented, well-observed pipelines from newer ones, and how requirements from downstream drive tier promotion.
  • 17:18 Where LLM-enabled workflows differ from standard pipelines: validation layers for specific use cases, plus observability and evaluation platforms that track model performance across executions.
  • 19:08 Using Airflow to orchestrate LLM generation of monthly variance commentary for analysts.
  • 21:10 Handling non-idempotent LLM outputs with multi-layer validation against source-of-truth data, and using a second agent (CI/CD style) to validate the first agent's output.
  • 23:04 How AI-enabled workflow orchestration differs from batch ETL, and why teams should start small before building fully agentic pipelines.
  • 25:25 What Antonello would most like to see from Airflow next: native support for agentic workflows and better local development that mirrors production.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 7576a518-263a-4310-b413-a13da92fc7d5 Thu, 27 Aug 2026 08:00:00 -0400 1735 1 92 false Managing travel platform data with Airflow at Headout Managing travel platform data with Airflow at Headout Running a travel platform means dealing with fast-moving inventory, real-time fraud detection, and heavy performance marketing attribution, all while keeping data trustworthy for a data-driven org. In this episode, Mrinalini Singh, Data Platform Engineer at [Headout](headout.com), walks through how her team uses Airflow as the nervous system of their stack: orchestrating dbt with a write-audit-publish pattern, running ML training and inference, and wiring up alerting that points to the exact commit that broke a DAG.


Key Takeaways:

  • 00:00 Introduction.
  • 01:05 What a data platform engineer does at Headout, and the hub-and-spokes model where analysts and scientists write their own dbt models.
  • 04:11 The specific data challenges of a travel platform: fast-changing inventory, real-time fraud analytics, and performance marketing attribution.
  • 05:40 Where Airflow sits in the stack, from ingestion to transformation to serving.
  • 07:01 The write-audit-publish dbt pattern and why slightly stale data beats wrong data.
  • 09:55 Why Headout uses a custom Python operator instead of the dbt provider or Cosmos, reading the dbt manifest to build task groups per model.
  • 13:45 ML use cases on Airflow: Feast feature store, model training, inference, and data/feature drift tracking.
  • 17:00 Custom Slack failure hooks that stitch together Airflow logs, GitHub commit URLs, and teammate Slack IDs.
  • 20:04 A zombie task incident that filled the metadata DB, caused locking issues, and drove the move to Grafana-based monitoring.
  • 22:26 Using AI to generate Airflow code, encoding internal patterns as a skill file, and running an AI reviewer bot on every PR.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

]]> Running a travel platform means dealing with fast-moving inventory, real-time fraud detection, and heavy performance marketing attribution, all while keeping data trustworthy for a data-driven org. In this episode, Mrinalini Singh, Data Platform Engineer at [Headout](headout.com), walks through how her team uses Airflow as the nervous system of their stack: orchestrating dbt with a write-audit-publish pattern, running ML training and inference, and wiring up alerting that points to the exact commit that broke a DAG.


Key Takeaways:

  • 00:00 Introduction.
  • 01:05 What a data platform engineer does at Headout, and the hub-and-spokes model where analysts and scientists write their own dbt models.
  • 04:11 The specific data challenges of a travel platform: fast-changing inventory, real-time fraud analytics, and performance marketing attribution.
  • 05:40 Where Airflow sits in the stack, from ingestion to transformation to serving.
  • 07:01 The write-audit-publish dbt pattern and why slightly stale data beats wrong data.
  • 09:55 Why Headout uses a custom Python operator instead of the dbt provider or Cosmos, reading the dbt manifest to build task groups per model.
  • 13:45 ML use cases on Airflow: Feast feature store, model training, inference, and data/feature drift tracking.
  • 17:00 Custom Slack failure hooks that stitch together Airflow logs, GitHub commit URLs, and teammate Slack IDs.
  • 20:04 A zombie task incident that filled the metadata DB, caused locking issues, and drove the move to Grafana-based monitoring.
  • 22:26 Using AI to generate Airflow code, encoding internal patterns as a skill file, and running an AI reviewer bot on every PR.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 55d84eb8-ee99-4d2e-be58-9a344cc28888 Thu, 20 Aug 2026 08:00:00 -0400 1728 1 91 false How Airflow orchestration decisions impact Spark performance How Airflow orchestration decisions impact Spark performance Airflow and Spark are one of the most common combinations in modern data platforms, but the orchestration decisions made on the Airflow side often determine whether Spark jobs run fast and cheap or slow and expensive. In this episode, Kenten is joined by [Meni Shmueli](linkedin.com/in/meni-shmueli-dataflint), Co-Founder and CEO at [DataFlint](dataflint.io), to dig into how Airflow and Spark fit together, where teams go wrong, and how AI is changing the way they reason about cost and performance.


Key Takeaways:

  • 00:00 Introduction.
  • 01:40 Meni's background as a data engineer and what led him to start DataFlint, a production observability agent for Apache Spark.
  • 02:53 Why the Airflow plus Spark combination is so common, and how each tool plays to its strengths.
  • 04:23 How orchestration decisions in Airflow directly impact Spark performance and cost.
  • 04:44 A customer story where parallelizing Airflow tasks made Spark jobs slower, less stable, and more expensive, and the opposite case where sequential runs left compute on the table.
  • 07:02 The number one mistake teams make benchmarking pipelines: only looking at the Airflow side and ignoring underlying Spark cost and resource usage.
  • 08:13 What DataFlint's Airflow and Astro integration gives teams, and how it brings production context into AI agents.
  • 10:46 How AI is changing pipeline optimization, including holistic scheduling across hundreds of pipelines and connecting context from Spark, FinOps, and cloud.
  • 12:21 A customer migration from Databricks to EMR that cut workflow costs by 80%, with examples of up to 100x optimizations.
  • 14:23 Where the Airflow and Spark story could be better, including Spark Declarative Pipelines and tighter feedback between the two projects.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #ApacheAirflow

]]> Airflow and Spark are one of the most common combinations in modern data platforms, but the orchestration decisions made on the Airflow side often determine whether Spark jobs run fast and cheap or slow and expensive. In this episode, Kenten is joined by [Meni Shmueli](linkedin.com/in/meni-shmueli-dataflint), Co-Founder and CEO at [DataFlint](dataflint.io), to dig into how Airflow and Spark fit together, where teams go wrong, and how AI is changing the way they reason about cost and performance.


Key Takeaways:

  • 00:00 Introduction.
  • 01:40 Meni's background as a data engineer and what led him to start DataFlint, a production observability agent for Apache Spark.
  • 02:53 Why the Airflow plus Spark combination is so common, and how each tool plays to its strengths.
  • 04:23 How orchestration decisions in Airflow directly impact Spark performance and cost.
  • 04:44 A customer story where parallelizing Airflow tasks made Spark jobs slower, less stable, and more expensive, and the opposite case where sequential runs left compute on the table.
  • 07:02 The number one mistake teams make benchmarking pipelines: only looking at the Airflow side and ignoring underlying Spark cost and resource usage.
  • 08:13 What DataFlint's Airflow and Astro integration gives teams, and how it brings production context into AI agents.
  • 10:46 How AI is changing pipeline optimization, including holistic scheduling across hundreds of pipelines and connecting context from Spark, FinOps, and cloud.
  • 12:21 A customer migration from Databricks to EMR that cut workflow costs by 80%, with examples of up to 100x optimizations.
  • 14:23 Where the Airflow and Spark story could be better, including Spark Declarative Pipelines and tighter feedback between the two projects.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #ApacheAirflow

]]> briana.okyere@astronomer.io (Astronomer) 6ba029e4-264f-410b-a443-8813a79e3cb3 Thu, 13 Aug 2026 08:00:00 -0400 1142 1 90 false Using Airflow for diverse client projects at Accion Labs Using Airflow for diverse client projects at Accion Labs When a real-time loan eligibility scoring pipeline is built on cron jobs, midnight pages are inevitable. In this episode, [Chandan Gowda](linkedin.com/in/chandan-gowda-a-h-744908195), Data Engineer at [Accion Labs](accionlabs.com), joins Kenten to discuss how his team uses Airflow across client projects, including a financial services scoring use case and a POC applying production-grade orchestration to RAG and GenAI data pipelines.


Key Takeaways:

  • 00:00 Introduction.
  • 01:00 What Accion Labs does as a technology consulting and services firm working across BFSI, healthcare, and retail.
  • 02:00 Chandan's role at the intersection of data engineering and GenAI, building pipelines one week and RAG-based agents the next.
  • 04:20 Why Airflow tends to win client evaluations: infrastructure agnostic, no cloud lock-in, fine-grained control over pipeline logic.
  • 06:00 Containerizing Airflow on Kubernetes or VMs so migrations between clouds don't require a rewrite.
  • 08:14 The loan eligibility scoring use case for a financial services client, and replacing fragile cron jobs with a single Airflow DAG end to end, cutting effort by about 25%.
  • 10:42 Triggering strategy: S3 file sensors as the primary trigger handling 90% of runs, plus a scheduled fallback as a safety net.
  • 12:53 End-to-end flow inside Airflow: ingestion into the data lake, validation and transformation with credit bureau joins, containerized model inference, and writeback to the loan management system.
  • 15:56 The AI orchestration POC and why the data feeding GenAI models needs the same rigor as any production pipeline.
  • 18:09 Using Airflow to detect document changes and re-chunk and re-embed only what changed, with quality thresholds and rollback before promoting to live.
  • 20:23 The roadmap: model evaluation pipelines, multi-agent orchestration, and provider packages for LangChain, OpenAI, and Hugging Face.
  • 22:32 Wishlist for Airflow: native event-driven triggers beyond polling sensors, first-class observability for AI workloads, and better dynamic DAG generation at scale.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

]]> When a real-time loan eligibility scoring pipeline is built on cron jobs, midnight pages are inevitable. In this episode, [Chandan Gowda](linkedin.com/in/chandan-gowda-a-h-744908195), Data Engineer at [Accion Labs](accionlabs.com), joins Kenten to discuss how his team uses Airflow across client projects, including a financial services scoring use case and a POC applying production-grade orchestration to RAG and GenAI data pipelines.


Key Takeaways:

  • 00:00 Introduction.
  • 01:00 What Accion Labs does as a technology consulting and services firm working across BFSI, healthcare, and retail.
  • 02:00 Chandan's role at the intersection of data engineering and GenAI, building pipelines one week and RAG-based agents the next.
  • 04:20 Why Airflow tends to win client evaluations: infrastructure agnostic, no cloud lock-in, fine-grained control over pipeline logic.
  • 06:00 Containerizing Airflow on Kubernetes or VMs so migrations between clouds don't require a rewrite.
  • 08:14 The loan eligibility scoring use case for a financial services client, and replacing fragile cron jobs with a single Airflow DAG end to end, cutting effort by about 25%.
  • 10:42 Triggering strategy: S3 file sensors as the primary trigger handling 90% of runs, plus a scheduled fallback as a safety net.
  • 12:53 End-to-end flow inside Airflow: ingestion into the data lake, validation and transformation with credit bureau joins, containerized model inference, and writeback to the loan management system.
  • 15:56 The AI orchestration POC and why the data feeding GenAI models needs the same rigor as any production pipeline.
  • 18:09 Using Airflow to detect document changes and re-chunk and re-embed only what changed, with quality thresholds and rollback before promoting to live.
  • 20:23 The roadmap: model evaluation pipelines, multi-agent orchestration, and provider packages for LangChain, OpenAI, and Hugging Face.
  • 22:32 Wishlist for Airflow: native event-driven triggers beyond polling sensors, first-class observability for AI workloads, and better dynamic DAG generation at scale.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 20d0a509-15cf-4811-8dcb-a4c8a7dfe939 Thu, 06 Aug 2026 10:03:06 -0400 1656 apache airflow,automation,artificial intelligence 1 89 false Consolidating legacy job schedulers with Airflow at Trading Technologies Consolidating legacy job schedulers with Airflow at Trading Technologies Migrating two decades of legacy job orchestration is the kind of project most teams quietly avoid. In this episode, Marc Lamberti is joined by [Sanket Patel](linkedin.com/in/sanket-patel-a2a57b3a), Director of Engineering at Trading Technologies, to talk through how his team is consolidating 20 years of C#/.NET-based scheduling and a recently acquired company's stack onto Airflow, Astronomer, and Snowflake. The conversation covers orchestrator selection, observability, the operational economics of managed Airflow, the human side of migrations, and where AI fits into the new data platform.


Key Takeaways:

  • 00:00 Introduction.
  • 02:11 Sanket's background and role leading data platform initiatives at Trading Technologies.
  • 04:32 The data landscape at TT: silos, OLTP-driven reporting, and an acquired company running orchestration on a 17-18 year old ASP.NET stack.
  • 10:53 Choosing Airflow and Snowflake as the foundation of the new data platform.
  • 11:36 Why Airflow won over legacy schedulers like Autosys plus Informatica, especially past the 2,000-3,000 job mark.
  • 16:34 Collapsing thousands of legacy jobs into 15-20 Airflow DAGs.
  • 17:38 Argo vs Airflow: community support and ecosystem as the deciding factors.
  • 18:20 Why Astronomer over self-managing Airflow on MWAA or Cloud Composer: CI/CD, observability, SLAs, PagerDuty, and billing alerts.
  • 24:07 The operational math: why running Airflow globally needs dedicated specialists, and why a generalist cannot handle upgrades.
  • 27:00 Migration strategy and the human side: addressing the "why" so teams move off legacy tools willingly.
  • 30:31 Where AI fits in: letting customers dialogue with their own data, and using cloud skills with co-pilots to generate boilerplate DAGs.
  • 36:00 The honest take on AI coding tools: accelerators, not replacements for engineering judgment.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #ApacheAirflow

]]> Migrating two decades of legacy job orchestration is the kind of project most teams quietly avoid. In this episode, Marc Lamberti is joined by [Sanket Patel](linkedin.com/in/sanket-patel-a2a57b3a), Director of Engineering at Trading Technologies, to talk through how his team is consolidating 20 years of C#/.NET-based scheduling and a recently acquired company's stack onto Airflow, Astronomer, and Snowflake. The conversation covers orchestrator selection, observability, the operational economics of managed Airflow, the human side of migrations, and where AI fits into the new data platform.


Key Takeaways:

  • 00:00 Introduction.
  • 02:11 Sanket's background and role leading data platform initiatives at Trading Technologies.
  • 04:32 The data landscape at TT: silos, OLTP-driven reporting, and an acquired company running orchestration on a 17-18 year old ASP.NET stack.
  • 10:53 Choosing Airflow and Snowflake as the foundation of the new data platform.
  • 11:36 Why Airflow won over legacy schedulers like Autosys plus Informatica, especially past the 2,000-3,000 job mark.
  • 16:34 Collapsing thousands of legacy jobs into 15-20 Airflow DAGs.
  • 17:38 Argo vs Airflow: community support and ecosystem as the deciding factors.
  • 18:20 Why Astronomer over self-managing Airflow on MWAA or Cloud Composer: CI/CD, observability, SLAs, PagerDuty, and billing alerts.
  • 24:07 The operational math: why running Airflow globally needs dedicated specialists, and why a generalist cannot handle upgrades.
  • 27:00 Migration strategy and the human side: addressing the "why" so teams move off legacy tools willingly.
  • 30:31 Where AI fits in: letting customers dialogue with their own data, and using cloud skills with co-pilots to generate boilerplate DAGs.
  • 36:00 The honest take on AI coding tools: accelerators, not replacements for engineering judgment.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #ApacheAirflow

]]> briana.okyere@astronomer.io (Astronomer) 1aac4b5a-a8de-4497-9b87-ed059662a805 Thu, 30 Jul 2026 08:00:00 -0400 1887 1 88 false Orchestrating AI video intelligence and evaluation pipelines at Firework Orchestrating AI video intelligence and evaluation pipelines at Firework How do you orchestrate AI workflows when LLM outputs are non-deterministic and evaluation costs can quietly exceed compute costs? Shawn Feng, Head of Data at [Firework](firework.com), joins the show to walk through how his team uses Airflow to coordinate ingestion, embeddings, and evaluation pipelines for AI features in a video commerce platform. The conversation covers the technical stack, the challenges of validating LLM output, cost guardrails, and what Shawn wants to see next from the Airflow project.


Key Takeaways:

  • 00:00 Introduction.
  • 00:48 What Firework does: a video commerce platform bringing short-form shoppable video and livestream experiences directly onto brand websites and apps.
  • 02:13 Shawn's team owns the full data and AI platform stack at Firework, from ingestion through BI and applied AI.
  • 02:59 Airflow has been the main orchestration layer for batch transformations since the early open source days, and now powers AI workflows like conversational insights, knowledge-base building, and automated eval pipelines.
  • 06:45 The AI workflow stack: ingestion of user interaction and content data, transformation into personalized profiles and embeddings, separate eval pipelines dispatched from third-party systems, all on Snowflake with Cortex AI and coordinated by Airflow.
  • 08:55 Why Airflow stuck: flexibility, reliability, and clear visibility into dependencies across SQL, Python, and AI tasks.
  • 11:27 The deterministic pipeline problem. Airflow assumes predictable input and output. LLM workflows break that assumption, which forces evaluation layers, traceability, regression testing, and feedback loops into the pipeline itself.
  • 13:07 Evaluation cost can exceed compute cost. Validating a single output can mean running four or five parallel eval jobs across prompts and configurations, so caching, batching, and selective evaluation become essential.
  • 15:58 Building safeguards: alerting and aborting jobs when cost exceeds thresholds, so the platform stays operationally stable while other improvements catch up.
  • 18:30 Wishlist: stronger first-class support for AI workflow patterns (evaluation tracking, prompt experimentation, model observability, event-driven AI orchestration) and continued UI/UX improvement in Airflow 3.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #ApacheAirflow

]]> How do you orchestrate AI workflows when LLM outputs are non-deterministic and evaluation costs can quietly exceed compute costs? Shawn Feng, Head of Data at [Firework](firework.com), joins the show to walk through how his team uses Airflow to coordinate ingestion, embeddings, and evaluation pipelines for AI features in a video commerce platform. The conversation covers the technical stack, the challenges of validating LLM output, cost guardrails, and what Shawn wants to see next from the Airflow project.


Key Takeaways:

  • 00:00 Introduction.
  • 00:48 What Firework does: a video commerce platform bringing short-form shoppable video and livestream experiences directly onto brand websites and apps.
  • 02:13 Shawn's team owns the full data and AI platform stack at Firework, from ingestion through BI and applied AI.
  • 02:59 Airflow has been the main orchestration layer for batch transformations since the early open source days, and now powers AI workflows like conversational insights, knowledge-base building, and automated eval pipelines.
  • 06:45 The AI workflow stack: ingestion of user interaction and content data, transformation into personalized profiles and embeddings, separate eval pipelines dispatched from third-party systems, all on Snowflake with Cortex AI and coordinated by Airflow.
  • 08:55 Why Airflow stuck: flexibility, reliability, and clear visibility into dependencies across SQL, Python, and AI tasks.
  • 11:27 The deterministic pipeline problem. Airflow assumes predictable input and output. LLM workflows break that assumption, which forces evaluation layers, traceability, regression testing, and feedback loops into the pipeline itself.
  • 13:07 Evaluation cost can exceed compute cost. Validating a single output can mean running four or five parallel eval jobs across prompts and configurations, so caching, batching, and selective evaluation become essential.
  • 15:58 Building safeguards: alerting and aborting jobs when cost exceeds thresholds, so the platform stays operationally stable while other improvements catch up.
  • 18:30 Wishlist: stronger first-class support for AI workflow patterns (evaluation tracking, prompt experimentation, model observability, event-driven AI orchestration) and continued UI/UX improvement in Airflow 3.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #ApacheAirflow

]]> briana.okyere@astronomer.io (Astronomer) cd0b7ff2-4f08-4e74-b3af-fda7f934040e Thu, 23 Jul 2026 08:00:00 -0400 1351 1 87 false Orchestrating Retail Data Pipelines at Saks Global Orchestrating Retail Data Pipelines at Saks Global Saks Global runs one of the largest retail data operations in the US, with around 8 million SKUs flowing across point-of-sale, e-commerce, catalog, and fraud detection systems into Snowflake. In this episode, [Shailesh Kadam](linkedin.com), Architect at [Saks Global](saks.com), joins Kenten to walk through how Airflow acts as the nervous system tying it all together, why they moved from self-managed Kubernetes to Astro, what is driving their Airflow 3 upgrade, and how they are approaching agentic AI, MCP, and credential security.


Key Takeaways:

  • 00:00 Introduction.
  • 01:31 Saks Global today. Shailesh describes the business after separating e-commerce from brick and mortar and acquiring Neiman Marcus, and the modern cloud-native stack on AWS, Snowflake, and Airflow.
  • 02:50 8 million SKUs in motion. Why every name, image, inventory, and price change has to flow in near real time across operational systems.
  • 04:50 What the pipelines look like. Point-of-sale ingestion, fraud signals to third parties like Fiserv, and hourly product catalog feeds out to Meta and Google.
  • 07:30 Moving off self-managed Kubernetes to Astro. Shailesh contrasts past experience with Kubernetes, IBM Tivoli, and Control-M against running on Astro.
  • 09:35 Upgrading to Airflow 3. Event and asset-based scheduling, DAG versioning, task isolation, and using Otto to convert DAGs in a phased rollout.
  • 13:13 Agentic AI and MCP on the roadmap. How Saks plans to use Airflow's MCP for LLM-driven product classification and to feed Snowflake analyses like churn and spend.
  • 18:01 Securing PII and credentials. Secrets backends, cloud secret manager integration, key rotation, and keeping credentials out of DAG code.
  • 21:02 Wishlist for Airflow. Interactive data lineage across DAGs and a UI-based debugging interface for support teams.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

]]> Saks Global runs one of the largest retail data operations in the US, with around 8 million SKUs flowing across point-of-sale, e-commerce, catalog, and fraud detection systems into Snowflake. In this episode, [Shailesh Kadam](linkedin.com), Architect at [Saks Global](saks.com), joins Kenten to walk through how Airflow acts as the nervous system tying it all together, why they moved from self-managed Kubernetes to Astro, what is driving their Airflow 3 upgrade, and how they are approaching agentic AI, MCP, and credential security.


Key Takeaways:

  • 00:00 Introduction.
  • 01:31 Saks Global today. Shailesh describes the business after separating e-commerce from brick and mortar and acquiring Neiman Marcus, and the modern cloud-native stack on AWS, Snowflake, and Airflow.
  • 02:50 8 million SKUs in motion. Why every name, image, inventory, and price change has to flow in near real time across operational systems.
  • 04:50 What the pipelines look like. Point-of-sale ingestion, fraud signals to third parties like Fiserv, and hourly product catalog feeds out to Meta and Google.
  • 07:30 Moving off self-managed Kubernetes to Astro. Shailesh contrasts past experience with Kubernetes, IBM Tivoli, and Control-M against running on Astro.
  • 09:35 Upgrading to Airflow 3. Event and asset-based scheduling, DAG versioning, task isolation, and using Otto to convert DAGs in a phased rollout.
  • 13:13 Agentic AI and MCP on the roadmap. How Saks plans to use Airflow's MCP for LLM-driven product classification and to feed Snowflake analyses like churn and spend.
  • 18:01 Securing PII and credentials. Secrets backends, cloud secret manager integration, key rotation, and keeping credentials out of DAG code.
  • 21:02 Wishlist for Airflow. Interactive data lineage across DAGs and a UI-based debugging interface for support teams.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 3300e415-5cbb-4d58-a27b-908cabf90b9c Thu, 16 Jul 2026 08:00:00 -0400 1520 1 86 false What's New in Apache Airflow® 3.3 What's New in Apache Airflow® 3.3 Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode, Marc Lamberti, Education Content Lead at [Astronomer](astronomer.io), joins Kenten Danas to walk through what's new in the release and where each feature actually pays off.


Key Takeaways:

  • 00:00 Introduction.
  • 01:46 The new task state store (AIP-103) lets tasks persist state across retries, so a long-running Spark job can be reattached after a worker failure instead of being duplicated on retry.
  • 03:46 The asset state store enables watermarking patterns: persist the last processed date or offset to an asset and resume from there on the next run.
  • 05:33 Why this matters for agentic workflows: resume an agent from where it left off rather than replaying every action.
  • 06:58 Why XComs don't solve this problem: they get reinitialized on every retry.
  • 09:27 Pluggable retries let you attach a retry policy to a task that branches on the exception type. Retry on transient errors, stop immediately on a 403.
  • 11:42 Subclassing the retry rule for more complex logic, including dynamic retry counts that used to require hacking the metadatabase.
  • 15:52 Updates to asset partitions in 3.3: segment-based partitioning with fan-out and roll-up mappers for downstream DAGs.
  • 21:42 Running tasks in Java and Go, moving Airflow toward a multi-language orchestrator.
  • 23:33 DAG versioning improvement: choose whether a manual rerun uses the most recent DAG version or the original version from that run.
  • 25:03 Advice for teams still on Airflow 2: use the upgrade ebook and Astro's AI migration tooling to handle the undifferentiated heavy lifting.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

]]> Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode, Marc Lamberti, Education Content Lead at [Astronomer](astronomer.io), joins Kenten Danas to walk through what's new in the release and where each feature actually pays off.


Key Takeaways:

  • 00:00 Introduction.
  • 01:46 The new task state store (AIP-103) lets tasks persist state across retries, so a long-running Spark job can be reattached after a worker failure instead of being duplicated on retry.
  • 03:46 The asset state store enables watermarking patterns: persist the last processed date or offset to an asset and resume from there on the next run.
  • 05:33 Why this matters for agentic workflows: resume an agent from where it left off rather than replaying every action.
  • 06:58 Why XComs don't solve this problem: they get reinitialized on every retry.
  • 09:27 Pluggable retries let you attach a retry policy to a task that branches on the exception type. Retry on transient errors, stop immediately on a 403.
  • 11:42 Subclassing the retry rule for more complex logic, including dynamic retry counts that used to require hacking the metadatabase.
  • 15:52 Updates to asset partitions in 3.3: segment-based partitioning with fan-out and roll-up mappers for downstream DAGs.
  • 21:42 Running tasks in Java and Go, moving Airflow toward a multi-language orchestrator.
  • 23:33 DAG versioning improvement: choose whether a manual rerun uses the most recent DAG version or the original version from that run.
  • 25:03 Advice for teams still on Airflow 2: use the upgrade ebook and Astro's AI migration tooling to handle the undifferentiated heavy lifting.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 04fab02f-9ccf-4042-a787-da2010dbfca1 Thu, 09 Jul 2026 08:00:00 -0400 1647 1 85 false Running Airflow 3 in a regulated environment at OTPP Running Airflow 3 in a regulated environment at OTPP Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan, Cloud Data Platform Lead, Data Engineering at [Ontario Teachers' Pension Plan](otpp.com), joins host Kenten Danas to walk through OTPP's cloud migration, their move to Airflow 3, and going fully live on remote execution.


Key Takeaways:

  • 00:00 Introduction.
  • 01:18 Inside the OTPP data platform team and what they're responsible for across cloud migration, standards, and enablement.
  • 02:33 What's driving OTPP's multi-year move off on-prem to a cloud architecture built around scalability and resilience.
  • 02:57 The new stack: Snowflake as the enterprise data platform, dbt for transformation, and Airflow as the orchestrator in the middle.
  • 04:15 Why OTPP chose Astronomer: active contributions to the Airflow OSS project, fast runtime releases, and built-in monitoring, observability, and RBAC.
  • 05:50 Evolving from dbt core with Bash operators to dbt Cosmos for model-level granularity, lineage, and precise failure recovery, plus a performance boost from watcher mode.
  • 08:00 Upgrading from Airflow 2.9 to Airflow 3, using the Astro CLI and linters to catch deprecations quickly.
  • 09:32 The drivers behind adopting remote execution: keeping data inside the security perimeter and scaling workloads on their own Kubernetes cluster.
  • 11:35 How remote execution replaced a complex network architecture of VPN tunnels and firewall rules, removing latency along the way.
  • 12:53 The POV process, success criteria, and a six week timebox to validate remote execution before going to production.
  • 14:14 Going fully live: OTPP's last hosted deployment was sunset just before recording.
  • 15:06 What Kowsy wants next from Airflow: AI orchestration capabilities and continued maturation of remote execution.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

]]> Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan, Cloud Data Platform Lead, Data Engineering at [Ontario Teachers' Pension Plan](otpp.com), joins host Kenten Danas to walk through OTPP's cloud migration, their move to Airflow 3, and going fully live on remote execution.


Key Takeaways:

  • 00:00 Introduction.
  • 01:18 Inside the OTPP data platform team and what they're responsible for across cloud migration, standards, and enablement.
  • 02:33 What's driving OTPP's multi-year move off on-prem to a cloud architecture built around scalability and resilience.
  • 02:57 The new stack: Snowflake as the enterprise data platform, dbt for transformation, and Airflow as the orchestrator in the middle.
  • 04:15 Why OTPP chose Astronomer: active contributions to the Airflow OSS project, fast runtime releases, and built-in monitoring, observability, and RBAC.
  • 05:50 Evolving from dbt core with Bash operators to dbt Cosmos for model-level granularity, lineage, and precise failure recovery, plus a performance boost from watcher mode.
  • 08:00 Upgrading from Airflow 2.9 to Airflow 3, using the Astro CLI and linters to catch deprecations quickly.
  • 09:32 The drivers behind adopting remote execution: keeping data inside the security perimeter and scaling workloads on their own Kubernetes cluster.
  • 11:35 How remote execution replaced a complex network architecture of VPN tunnels and firewall rules, removing latency along the way.
  • 12:53 The POV process, success criteria, and a six week timebox to validate remote execution before going to production.
  • 14:14 Going fully live: OTPP's last hosted deployment was sunset just before recording.
  • 15:06 What Kowsy wants next from Airflow: AI orchestration capabilities and continued maturation of remote execution.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 883d798b-7c91-4c13-874a-491ef706d213 Thu, 25 Jun 2026 08:00:00 -0400 1129 artifical intelligence,apache airflow,open source,data management,data engineer 1 84 false Managing a Customer Analytics Platform with Airflow at Skimlinks Managing a Customer Analytics Platform with Airflow at Skimlinks Skimlinks runs a reporting platform that serves around 2,000 weekly publisher users, and the data infrastructure behind it runs on Airflow. In this episode, Julian Larralde, Director of Data Engineering at Skimlinks, walks through the stack, the migration from external task sensors to event-driven Assets, and a YAML-based DAG factory the team built to onboard new publishers without rewriting Python.



Key Takeaways

  • 00:00 Introduction.
  • 00:45 What Skimlinks does and how it operates as an affiliate marketing network aggregator for publishers.
  • 02:12 Julian's team and the data platform they own: a reporting portal that serves ~2,000 weekly publisher users.
  • 03:07 The stack: real-time ingestion into BigQuery, Airflow as the orchestrator, raw / silver / gold layers, and Apache Druid as the serving database for sub-second BI queries.
  • 04:50 Reusing the same data marts for ~100 internal customers across marketing, finance, operations, and account management.
  • 06:25 Airflow as the single orchestrator: BigQuery operators for SQL business logic, plus raw file exports for the largest publishers.
  • 08:08 Moving from external task sensors to datasets (now Assets) and what the migration actually solved.
  • 09:18 Why sensor polling created scheduler load and worker overload, and how event-driven Assets fixed both.
  • 10:15 The lineage view in the Airflow UI that came as a bonus after the Assets migration.
  • 10:49 The vision for multi-tenant Airflow inside Skimlinks: replacing cron, Rundeck, and team-local Airflow instances with a shared platform.
  • 14:31 Building a custom DAG factory with YAML configuration for onboarding new publishers.
  • 17:33 Breaking a single Python class into single-responsibility components for the DataPipe project.
  • 19:07 Adding a Pydantic layer so misconfigured YAML fails at DAG parse time instead of run time.
  • 20:31 Using AI assistance to guide refactoring decisions and generate tests across the new class structure.
  • 22:34 What Julian wants from Airflow next: asset watchers paired with data contracts.


Resources Mentioned


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow



]]> Skimlinks runs a reporting platform that serves around 2,000 weekly publisher users, and the data infrastructure behind it runs on Airflow. In this episode, Julian Larralde, Director of Data Engineering at Skimlinks, walks through the stack, the migration from external task sensors to event-driven Assets, and a YAML-based DAG factory the team built to onboard new publishers without rewriting Python.



Key Takeaways

  • 00:00 Introduction.
  • 00:45 What Skimlinks does and how it operates as an affiliate marketing network aggregator for publishers.
  • 02:12 Julian's team and the data platform they own: a reporting portal that serves ~2,000 weekly publisher users.
  • 03:07 The stack: real-time ingestion into BigQuery, Airflow as the orchestrator, raw / silver / gold layers, and Apache Druid as the serving database for sub-second BI queries.
  • 04:50 Reusing the same data marts for ~100 internal customers across marketing, finance, operations, and account management.
  • 06:25 Airflow as the single orchestrator: BigQuery operators for SQL business logic, plus raw file exports for the largest publishers.
  • 08:08 Moving from external task sensors to datasets (now Assets) and what the migration actually solved.
  • 09:18 Why sensor polling created scheduler load and worker overload, and how event-driven Assets fixed both.
  • 10:15 The lineage view in the Airflow UI that came as a bonus after the Assets migration.
  • 10:49 The vision for multi-tenant Airflow inside Skimlinks: replacing cron, Rundeck, and team-local Airflow instances with a shared platform.
  • 14:31 Building a custom DAG factory with YAML configuration for onboarding new publishers.
  • 17:33 Breaking a single Python class into single-responsibility components for the DataPipe project.
  • 19:07 Adding a Pydantic layer so misconfigured YAML fails at DAG parse time instead of run time.
  • 20:31 Using AI assistance to guide refactoring decisions and generate tests across the new class structure.
  • 22:34 What Julian wants from Airflow next: asset watchers paired with data contracts.


Resources Mentioned


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow



]]> briana.okyere@astronomer.io (Astronomer) 53b24480-9757-4c9d-b677-b221e8ca5b82 Thu, 11 Jun 2026 08:00:00 -0400 1360 1 83 false Building a custom Tableau provider for Airflow at JLR Building a custom Tableau provider for Airflow at JLR JLR is the UK's largest automotive manufacturer, behind brands like Range Rover, Jaguar, Defender, and Discovery. In this episode, Najeeb Sulaiman, Senior Data Engineer at JLR, walks through how Airflow orchestrates data across manufacturing, supply chain, and finance — including a custom Tableau provider his team built (after the community version dropped PAT authentication) and a CI/CD pipeline that validates DAGs before they reach production.


Key Takeaways:

  • 00:00 Introduction.
  • 00:48 What JLR makes: luxury vehicles under the Range Rover, Jaguar, Defender, and Discovery brands.
  • 01:42 Najeeb's team in the Data and AI Office, supporting manufacturing, supply chain, finance, and commerce analytics.
  • 03:25 Airflow as the central nervous system of the JLR data stack — the orchestrator that connects every source and downstream system.
  • 05:01 How JLR uses Tableau, and the two modes for getting data in: live connection and scheduled extract refresh.
  • 06:24 Why scheduled Tableau refreshes go stale: they aren't aware of when the data pipeline actually finished.
  • 08:09 First attempt at solving it: Python scripts calling the Tableau REST API directly.
  • 08:47 Why the script approach didn't scale across teams — code duplication and version drift.
  • 10:00 Trying the community Airflow Tableau provider and hitting the PAT authentication roadblock.
  • 12:21 Building a custom provider on top of the community one to keep PAT auth.
  • 13:30 Treating CI/CD as a deployment gate for Airflow DAGs at JLR's scale.
  • 15:23 What the CI/CD pipeline actually catches: top-level code making external calls, import errors, and Airflow 3 compatibility.
  • 17:47 How the gate blocks broken DAGs from reaching production.
  • 18:30 What Najeeb wants from Airflow next: native integration testing, better OpenTelemetry support, and built-in lineage.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow



]]> JLR is the UK's largest automotive manufacturer, behind brands like Range Rover, Jaguar, Defender, and Discovery. In this episode, Najeeb Sulaiman, Senior Data Engineer at JLR, walks through how Airflow orchestrates data across manufacturing, supply chain, and finance — including a custom Tableau provider his team built (after the community version dropped PAT authentication) and a CI/CD pipeline that validates DAGs before they reach production.


Key Takeaways:

  • 00:00 Introduction.
  • 00:48 What JLR makes: luxury vehicles under the Range Rover, Jaguar, Defender, and Discovery brands.
  • 01:42 Najeeb's team in the Data and AI Office, supporting manufacturing, supply chain, finance, and commerce analytics.
  • 03:25 Airflow as the central nervous system of the JLR data stack — the orchestrator that connects every source and downstream system.
  • 05:01 How JLR uses Tableau, and the two modes for getting data in: live connection and scheduled extract refresh.
  • 06:24 Why scheduled Tableau refreshes go stale: they aren't aware of when the data pipeline actually finished.
  • 08:09 First attempt at solving it: Python scripts calling the Tableau REST API directly.
  • 08:47 Why the script approach didn't scale across teams — code duplication and version drift.
  • 10:00 Trying the community Airflow Tableau provider and hitting the PAT authentication roadblock.
  • 12:21 Building a custom provider on top of the community one to keep PAT auth.
  • 13:30 Treating CI/CD as a deployment gate for Airflow DAGs at JLR's scale.
  • 15:23 What the CI/CD pipeline actually catches: top-level code making external calls, import errors, and Airflow 3 compatibility.
  • 17:47 How the gate blocks broken DAGs from reaching production.
  • 18:30 What Najeeb wants from Airflow next: native integration testing, better OpenTelemetry support, and built-in lineage.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow



]]> briana.okyere@astronomer.io (Astronomer) d3dea324-23a6-434a-ab1c-b51f23c69bc1 Thu, 04 Jun 2026 08:00:00 -0400 1278 AI,Automation,Apache Airflow,Data Engineering 1 82 false Orchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira Orchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira Running Airflow at the scale of a national retailer means more than just scheduling. It means giving non-engineers a path to ship DAGs, and classifying thousands of runs to know which ones need attention. In this episode, Mateus Ferreira, Senior Data Engineer at Luiza Labs (the technology arm of Magazine Luiza, one of Brazil's largest retailers), joins Marc to talk about the patterns his team uses to run 2,000+ Airflow pipelines across more than four petabytes of data.


Key Takeaways:

  • 00:00 Introduction
  • 01:11 Mateus introduces himself and Luiza Labs, the technology arm of Magazine Luiza (Magalu), one of Brazil's largest retailers (founded 1957). 1,000+ physical stores, multi-region operations, and a data team that has to handle the variability that comes with all of it.
  • 04:33 Lu Brain, Magalu's AI initiative built around their character Lu, and how AI fits into the data work.
  • 06:47 The data reliability engineering channel where AI summarizes Airflow errors with confidence scores and posts a suggested fix in chat.
  • 08:30 How Airflow became the heart of orchestration. Coming from Control-M in banking, then GCP, then consolidating on Cloud Composer to centralize roughly 2,000 pipelines.
  • 14:23 The YAML wrapper that lets non-engineers ship DAGs. Reads namespace, tables, and Spark options. Handles CDC, JDBC full, and JDBC incremental collection types with checkpoints. All changes go through data reliability engineering.
  • 17:20 Why metadata is the most valuable asset in the AI era, and how the wrapper makes data lineage observable across 2,000 pipelines.
  • 18:26 The Data Reliability Engineering team. A 10-person group that is the window to the company, handling maintenance, validation, corrections, and optimization for the business unit pipelines.
  • 20:09 Operating at four petabytes of data.
  • 21:24 Why they built custom Spark operators. Cost drove the move off the DataprocOperator. The custom operator exposes Spark driver and executor sizing as Airflow parameters and generates the Kubernetes manifest.
  • 24:36 The monitoring dashboard built on the Airflow metadata DB. A timeline view that shows how many DAGs run each hour, used to spread scheduling across the day.
  • 26:37 Classifying DAGs by their last five runs: success, partially correct, intermittent, total failure. A reusable observability pattern.
  • 29:57 How to reach Mateus, and a closing thought in Portuguese on appreciating the good old times while you are living them.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow



]]> Running Airflow at the scale of a national retailer means more than just scheduling. It means giving non-engineers a path to ship DAGs, and classifying thousands of runs to know which ones need attention. In this episode, Mateus Ferreira, Senior Data Engineer at Luiza Labs (the technology arm of Magazine Luiza, one of Brazil's largest retailers), joins Marc to talk about the patterns his team uses to run 2,000+ Airflow pipelines across more than four petabytes of data.


Key Takeaways:

  • 00:00 Introduction
  • 01:11 Mateus introduces himself and Luiza Labs, the technology arm of Magazine Luiza (Magalu), one of Brazil's largest retailers (founded 1957). 1,000+ physical stores, multi-region operations, and a data team that has to handle the variability that comes with all of it.
  • 04:33 Lu Brain, Magalu's AI initiative built around their character Lu, and how AI fits into the data work.
  • 06:47 The data reliability engineering channel where AI summarizes Airflow errors with confidence scores and posts a suggested fix in chat.
  • 08:30 How Airflow became the heart of orchestration. Coming from Control-M in banking, then GCP, then consolidating on Cloud Composer to centralize roughly 2,000 pipelines.
  • 14:23 The YAML wrapper that lets non-engineers ship DAGs. Reads namespace, tables, and Spark options. Handles CDC, JDBC full, and JDBC incremental collection types with checkpoints. All changes go through data reliability engineering.
  • 17:20 Why metadata is the most valuable asset in the AI era, and how the wrapper makes data lineage observable across 2,000 pipelines.
  • 18:26 The Data Reliability Engineering team. A 10-person group that is the window to the company, handling maintenance, validation, corrections, and optimization for the business unit pipelines.
  • 20:09 Operating at four petabytes of data.
  • 21:24 Why they built custom Spark operators. Cost drove the move off the DataprocOperator. The custom operator exposes Spark driver and executor sizing as Airflow parameters and generates the Kubernetes manifest.
  • 24:36 The monitoring dashboard built on the Airflow metadata DB. A timeline view that shows how many DAGs run each hour, used to spread scheduling across the day.
  • 26:37 Classifying DAGs by their last five runs: success, partially correct, intermittent, total failure. A reusable observability pattern.
  • 29:57 How to reach Mateus, and a closing thought in Portuguese on appreciating the good old times while you are living them.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow



]]> briana.okyere@astronomer.io (Astronomer) 4ee70cc0-dba2-4de2-ab1b-170b6c00a997 Thu, 28 May 2026 08:00:00 -0400 1957 1 81 false Enhancing DAGs for Data Processing with William Orgertrice III at Cargill Enhancing DAGs for Data Processing with William Orgertrice III at Cargill In the data engineering world, the difference between a pipeline that works and one that's truly production-ready often comes down to a handful of deliberate decisions. William Orgertrice III, Data Engineer at Cargill, joins us to share the DAG design and monitoring practices he presented at Airflow Summit 2025 and how his team is rolling out Airflow across 60+ internal teams as part of Cargill's new Minerva data platform.


Key Takeaways:


00:00 Introduction.

01:45 Cargill is one of the largest privately owned companies in the US, operating across 70 countries and serving 125+ markets.

03:45 William's team on the Cargill Data Platform supports 60+ internal teams, providing data products that drive decisions across finance, inventory and operations.

05:10 Cargill chose Airflow as a core component of its new Minerva data platform to replace older ETL tooling with a more supportable, observable stack.

06:26 Native SLA sensors and dependency management were specific features that made Airflow the right fit for Cargill's batch ingestion pipelines.

09:00 Cargill is running Airflow through Astronomer as their managed solution, with some teams already in production.

13:22 Every task in a DAG should have a single, documented purpose — one task doing everything makes troubleshooting significantly harder.

14:40 A DAG that never enters a failed state but keeps running indefinitely will spend compute budget without alerting anyone.

15:25 In shared Airflow environments, embedding contact information and owner tags in DAGs ensures the right team is reached when something breaks upstream.

21:00 William flags connection testing as a friction point in pipeline development — verifying a connection string before building the full job would reduce iteration time.



Resources Mentioned:


Cargill | Website

https://www.cargill.com/food-beverage


Airflow Community on Slack 

https://airflow.apache.org/community/



Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> In the data engineering world, the difference between a pipeline that works and one that's truly production-ready often comes down to a handful of deliberate decisions. William Orgertrice III, Data Engineer at Cargill, joins us to share the DAG design and monitoring practices he presented at Airflow Summit 2025 and how his team is rolling out Airflow across 60+ internal teams as part of Cargill's new Minerva data platform.


Key Takeaways:


00:00 Introduction.

01:45 Cargill is one of the largest privately owned companies in the US, operating across 70 countries and serving 125+ markets.

03:45 William's team on the Cargill Data Platform supports 60+ internal teams, providing data products that drive decisions across finance, inventory and operations.

05:10 Cargill chose Airflow as a core component of its new Minerva data platform to replace older ETL tooling with a more supportable, observable stack.

06:26 Native SLA sensors and dependency management were specific features that made Airflow the right fit for Cargill's batch ingestion pipelines.

09:00 Cargill is running Airflow through Astronomer as their managed solution, with some teams already in production.

13:22 Every task in a DAG should have a single, documented purpose — one task doing everything makes troubleshooting significantly harder.

14:40 A DAG that never enters a failed state but keeps running indefinitely will spend compute budget without alerting anyone.

15:25 In shared Airflow environments, embedding contact information and owner tags in DAGs ensures the right team is reached when something breaks upstream.

21:00 William flags connection testing as a friction point in pipeline development — verifying a connection string before building the full job would reduce iteration time.



Resources Mentioned:


Cargill | Website

https://www.cargill.com/food-beverage


Airflow Community on Slack 

https://airflow.apache.org/community/



Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 9a0839d8-6ce9-4bd3-a725-dc569f329aa4 Thu, 21 May 2026 09:52:00 -0400 1575 1 80 false Getting Into Data Engineering with Shrividya Hegde, Data and AI Engineer Getting Into Data Engineering with Shrividya Hegde, Data and AI Engineer In this episode, we take a step back from implementation-specific topics to explore what it actually takes to build a career in data engineering — and how AI is reshaping that path.

Shrividya Hegde,  a data and AI engineer and an Airflow champion in Astronomer’s Champions program, joins us to discuss getting into data engineering, contributing to open source and why good data engineering should make AI output trustworthy rather than confidently wrong.


Key Takeaways:


00:00 Introduction.

04:08 Build fundamentals before chasing trending tools — understanding what a tool does, why it exists and what problem it solves has to come first. 

07:19 Data engineering fundamentals mean SQL query performance under joins and aggregations, how data moves between pipelines, DAG failure recovery and idempotency — not just writing queries. 

08:10 The most common mistake newer data engineers make is skipping fundamentals to chase trends — it is a sequencing problem, not a talent problem. 

13:15 AI creates more opportunity for data engineers because AI output quality is directly determined by the quality of the data pipeline feeding it — confidently wrong output is harder to catch than obviously wrong output. 

15:06 Airflow's supporting operators make AI outputs production-ready — orchestration is what converts experimental AI into something reliable. 

17:14 AI-generated DAGs help newer engineers understand underlying concepts rather than just producing working code. 

23:12 The Airflow open source community is more welcoming than most people expect for a project of its size — raising issues and reviewing PRs are viable entry points for first contributions.


Resources Mentioned:


Shrividya Hegde

https://www.linkedin.com/in/shrividya-hegde-shri-91562365/


Astronomer | LinkedIn

https://www.linkedin.com/company/astronomer/


Astronomer | Website

https://www.astronomer.io


Women in Data | Website

https://womenindata.mn.co/landing


Apache Airflow Slack 

https://airflow.apache.org/


Shrividya's Medium writing

https://medium.com/@shrihegde


Shrividya’ Substack writing

https://substack.com/@shrividyahegde




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> In this episode, we take a step back from implementation-specific topics to explore what it actually takes to build a career in data engineering — and how AI is reshaping that path.

Shrividya Hegde,  a data and AI engineer and an Airflow champion in Astronomer’s Champions program, joins us to discuss getting into data engineering, contributing to open source and why good data engineering should make AI output trustworthy rather than confidently wrong.


Key Takeaways:


00:00 Introduction.

04:08 Build fundamentals before chasing trending tools — understanding what a tool does, why it exists and what problem it solves has to come first. 

07:19 Data engineering fundamentals mean SQL query performance under joins and aggregations, how data moves between pipelines, DAG failure recovery and idempotency — not just writing queries. 

08:10 The most common mistake newer data engineers make is skipping fundamentals to chase trends — it is a sequencing problem, not a talent problem. 

13:15 AI creates more opportunity for data engineers because AI output quality is directly determined by the quality of the data pipeline feeding it — confidently wrong output is harder to catch than obviously wrong output. 

15:06 Airflow's supporting operators make AI outputs production-ready — orchestration is what converts experimental AI into something reliable. 

17:14 AI-generated DAGs help newer engineers understand underlying concepts rather than just producing working code. 

23:12 The Airflow open source community is more welcoming than most people expect for a project of its size — raising issues and reviewing PRs are viable entry points for first contributions.


Resources Mentioned:


Shrividya Hegde

https://www.linkedin.com/in/shrividya-hegde-shri-91562365/


Astronomer | LinkedIn

https://www.linkedin.com/company/astronomer/


Astronomer | Website

https://www.astronomer.io


Women in Data | Website

https://womenindata.mn.co/landing


Apache Airflow Slack 

https://airflow.apache.org/


Shrividya's Medium writing

https://medium.com/@shrihegde


Shrividya’ Substack writing

https://substack.com/@shrividyahegde




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 0e1bf992-36b8-4402-89f6-7c017753cf4d Thu, 14 May 2026 05:45:33 -0400 1654 1 79 false Orchestrating DBT With Cosmos and Airflow with Filip Kunčar at ShipMonk Product Development Orchestrating DBT With Cosmos and Airflow with Filip Kunčar at ShipMonk Product Development We explore how a third-party logistics platform built its entire data orchestration layer on Airflow, and what that makes possible for developer teams and merchant-facing products alike.


Filip Kunčar, Platform Director at ShipMonk Product Development, discusses migrating from a closed source tool to Airflow, orchestrating dbt with both Cosmos and the BashOperator and using Airflow to power customer-facing data delivery.


Key Takeaways:


00:00 Introduction.

01:07 ShipMonk is a third-party logistics company guaranteeing two-day delivery across the US. The data platform team's mission is to lower cognitive load for developers working with data. 

05:13 ShipMonk migrated to Airflow in 2022, moving away from a closed-source UI-based tool, driven by the need for a code-first approach, open source extensibility and broad cloud provider support. 

10:02 The team uses Cosmos for developer-facing visibility and lineage and BashOperator for internal pipelines where runtime performance matters. 

12:20 Switching from Cosmos to the BashOperator for a frequently running pipeline reduced runtime from over 15 minutes to three minutes. 

13:14 Because the full dbt chain runs inside Airflow, a configurable downstream DAG can deliver processed data directly to each merchant's preferred destination, with secrets management and SLA tracking already handled. 

15:03 Per-team alerting is hooked to each DAG by owner and severity, so teams can react to SLA breaches immediately. 

18:09 ShipMonk uses Airflow in three ways for AI: authoring DAGs faster with skills, orchestrating AI workloads in Lambda and containers and using Astronomer's skills repo to simplify Airflow version upgrades.


Resources Mentioned:


Filip Kunčar

https://www.linkedin.com/in/filipkuncar/


ShipMonk Product Development

https://www.linkedin.com/company/shipmonk-product-development/


ShipMonk | Website

http://www.shipmonk.com


Astronomer Cosmos

http://www.astronomer.io/cosmos


Astronomer AI Skills Repo

http://www.github.com/astronomer/airflow-llm-providers-demo


Datadog

http://www.datadoghq.com




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> We explore how a third-party logistics platform built its entire data orchestration layer on Airflow, and what that makes possible for developer teams and merchant-facing products alike.


Filip Kunčar, Platform Director at ShipMonk Product Development, discusses migrating from a closed source tool to Airflow, orchestrating dbt with both Cosmos and the BashOperator and using Airflow to power customer-facing data delivery.


Key Takeaways:


00:00 Introduction.

01:07 ShipMonk is a third-party logistics company guaranteeing two-day delivery across the US. The data platform team's mission is to lower cognitive load for developers working with data. 

05:13 ShipMonk migrated to Airflow in 2022, moving away from a closed-source UI-based tool, driven by the need for a code-first approach, open source extensibility and broad cloud provider support. 

10:02 The team uses Cosmos for developer-facing visibility and lineage and BashOperator for internal pipelines where runtime performance matters. 

12:20 Switching from Cosmos to the BashOperator for a frequently running pipeline reduced runtime from over 15 minutes to three minutes. 

13:14 Because the full dbt chain runs inside Airflow, a configurable downstream DAG can deliver processed data directly to each merchant's preferred destination, with secrets management and SLA tracking already handled. 

15:03 Per-team alerting is hooked to each DAG by owner and severity, so teams can react to SLA breaches immediately. 

18:09 ShipMonk uses Airflow in three ways for AI: authoring DAGs faster with skills, orchestrating AI workloads in Lambda and containers and using Astronomer's skills repo to simplify Airflow version upgrades.


Resources Mentioned:


Filip Kunčar

https://www.linkedin.com/in/filipkuncar/


ShipMonk Product Development

https://www.linkedin.com/company/shipmonk-product-development/


ShipMonk | Website

http://www.shipmonk.com


Astronomer Cosmos

http://www.astronomer.io/cosmos


Astronomer AI Skills Repo

http://www.github.com/astronomer/airflow-llm-providers-demo


Datadog

http://www.datadoghq.com




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 20728bb8-75b0-4ded-ac50-f104e6131651 Thu, 07 May 2026 07:03:41 -0400 1497 1 80 false Building Airflow CTL with Buğra Öztürk at Mollie Building Airflow CTL with Buğra Öztürk at Mollie Buğra Öztürk, Senior Data Engineer at Mollie and Committer and PMC member on the Apache Airflow project, joins us to walk through Airflow CTL — what it is, how it differs from the existing Airflow CLI and where it is headed under AIP-94.


Key Takeaways:


00:00 Introduction.

03:10 Buğra has contributed to Airflow since 2022, from docs changes up to Committer and PMC member — a path he hopes inspires others to start small and contribute. 

04:05 Airflow CTL solves secure user interaction by abstracting database credentials behind the public core API. 

05:13 Airflow CLI and Airflow CTL are complementary — CLI handles administration and database management while CTL handles secure user interactions via the API. 

07:08 Airflow CTL authenticates via the API, acquires a JWT token and stores it securely in the OS keyring — running on the user's machine and never requiring direct database access.

08:21 Concrete use cases include local DAG development without the UI and CI/CD automation using headless mode with short-lived JWT tokens.

10:08 AIP-94 describes the long-term vision — decoupling all remote commands from the Airflow CLI and routing them through Airflow CTL. 

13:12 Airflow CTL is currently at 0.X and already being used in CI and deployment automations. The move to 1.0 with full CLI parity is the next milestone under AIP-94.  

16:09 Multi-team deployment becoming generally available in a future Airflow release is Buğra's most-anticipated upcoming feature beyond Airflow CTL.


Resources Mentioned:


Buğra Öztürk

https://www.linkedin.com/in/bugraozturk93/


Mollie

https://www.linkedin.com/company/mollie/


Mollie | Website

https://www.mollie.com/


Apache Airflow CTL 

https://airflow.apache.org/


AIP-94 on Airflow Confluence

https://lists.apache.org/thread/d2o1pr78wxdp1wozq519stp0pkcv6k6c


Apache Airflow GitHub

https://www.github.com/apache/airflow




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Buğra Öztürk, Senior Data Engineer at Mollie and Committer and PMC member on the Apache Airflow project, joins us to walk through Airflow CTL — what it is, how it differs from the existing Airflow CLI and where it is headed under AIP-94.


Key Takeaways:


00:00 Introduction.

03:10 Buğra has contributed to Airflow since 2022, from docs changes up to Committer and PMC member — a path he hopes inspires others to start small and contribute. 

04:05 Airflow CTL solves secure user interaction by abstracting database credentials behind the public core API. 

05:13 Airflow CLI and Airflow CTL are complementary — CLI handles administration and database management while CTL handles secure user interactions via the API. 

07:08 Airflow CTL authenticates via the API, acquires a JWT token and stores it securely in the OS keyring — running on the user's machine and never requiring direct database access.

08:21 Concrete use cases include local DAG development without the UI and CI/CD automation using headless mode with short-lived JWT tokens.

10:08 AIP-94 describes the long-term vision — decoupling all remote commands from the Airflow CLI and routing them through Airflow CTL. 

13:12 Airflow CTL is currently at 0.X and already being used in CI and deployment automations. The move to 1.0 with full CLI parity is the next milestone under AIP-94.  

16:09 Multi-team deployment becoming generally available in a future Airflow release is Buğra's most-anticipated upcoming feature beyond Airflow CTL.


Resources Mentioned:


Buğra Öztürk

https://www.linkedin.com/in/bugraozturk93/


Mollie

https://www.linkedin.com/company/mollie/


Mollie | Website

https://www.mollie.com/


Apache Airflow CTL 

https://airflow.apache.org/


AIP-94 on Airflow Confluence

https://lists.apache.org/thread/d2o1pr78wxdp1wozq519stp0pkcv6k6c


Apache Airflow GitHub

https://www.github.com/apache/airflow




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 1bef6309-fff6-437c-9093-e982250cc514 Thu, 30 Apr 2026 00:00:00 -0400 1182 1 79 false Introducing Airflow’s Common AI Provider with Pavan Kumar Gopidesu and Kaxil Naik Introducing Airflow’s Common AI Provider with Pavan Kumar Gopidesu and Kaxil Naik In this episode, we explore the newly released Apache Airflow common AI provider — what problem it solves, how it was built and what's coming next.


Kaxil Naik, Senior Director of Engineering at Astronomer and Apache Airflow PMC member, and Pavan Kumar Gopidesu, Lead Data Engineer at Experian and Apache Airflow PMC member, join us to walk through the provider's first release and the technical decisions behind it.


Key Takeaways:


00:00 Introduction.

04:05 The common AI provider was born from a real production problem.

07:10 Airflow already had the primitives needed for durable agent execution, making it the natural foundation for AI orchestration. 

09:15 The LLM schema compare operator uses Apache DataFusion to fetch source schemas.

11:07 Apache DataFusion was chosen for its speed.

13:09 Hook tool sets expose Airflow's provider hooks to agents with an allowed methods list that blocks destructive operations.

15:20 Passing durable=True to an LLM operator caches tool calls and LLM outputs mid-task. 

18:13 The provider offers three abstraction levels. 

21:20 The provider currently requires Airflow 3 — the team is open to adding Airflow 2.11 support if demand is high enough. 

24:10 MCP server configs can be stored as Airflow connections.



Resources Mentioned:


Kaxil Naik

https://www.linkedin.com/in/kaxil/


Pavan Kumar Gopidesu

https://www.linkedin.com/in/pavan-kumar-gopidesu/


Astronomer | LinkedIn

https://www.linkedin.com/company/astronomer/


Astronomer | Website

https://www.astronomer.io


Experian

https://www.linkedin.com/company/experian/


Apache Airflow

https://www.linkedin.com/company/apache-airflow


Apache Airflow common AI provider docs

https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/commits.html


Apache DataFusion

https://datafusion.apache.org/


Pydantic AI

https://pydantic.dev/docs/ai/overview/


Airflow Slack

https://airflow.apache.org/docs/apache-airflow-providers-slack/stable/index.html


Introducing the Common AI Provider: LLM and AI Agent Support for Apache Airflow

https://airflow.apache.org/blog/common-ai-provider/





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#Automation #Airflow #MachineLearning

]]> In this episode, we explore the newly released Apache Airflow common AI provider — what problem it solves, how it was built and what's coming next.


Kaxil Naik, Senior Director of Engineering at Astronomer and Apache Airflow PMC member, and Pavan Kumar Gopidesu, Lead Data Engineer at Experian and Apache Airflow PMC member, join us to walk through the provider's first release and the technical decisions behind it.


Key Takeaways:


00:00 Introduction.

04:05 The common AI provider was born from a real production problem.

07:10 Airflow already had the primitives needed for durable agent execution, making it the natural foundation for AI orchestration. 

09:15 The LLM schema compare operator uses Apache DataFusion to fetch source schemas.

11:07 Apache DataFusion was chosen for its speed.

13:09 Hook tool sets expose Airflow's provider hooks to agents with an allowed methods list that blocks destructive operations.

15:20 Passing durable=True to an LLM operator caches tool calls and LLM outputs mid-task. 

18:13 The provider offers three abstraction levels. 

21:20 The provider currently requires Airflow 3 — the team is open to adding Airflow 2.11 support if demand is high enough. 

24:10 MCP server configs can be stored as Airflow connections.



Resources Mentioned:


Kaxil Naik

https://www.linkedin.com/in/kaxil/


Pavan Kumar Gopidesu

https://www.linkedin.com/in/pavan-kumar-gopidesu/


Astronomer | LinkedIn

https://www.linkedin.com/company/astronomer/


Astronomer | Website

https://www.astronomer.io


Experian

https://www.linkedin.com/company/experian/


Apache Airflow

https://www.linkedin.com/company/apache-airflow


Apache Airflow common AI provider docs

https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/commits.html


Apache DataFusion

https://datafusion.apache.org/


Pydantic AI

https://pydantic.dev/docs/ai/overview/


Airflow Slack

https://airflow.apache.org/docs/apache-airflow-providers-slack/stable/index.html


Introducing the Common AI Provider: LLM and AI Agent Support for Apache Airflow

https://airflow.apache.org/blog/common-ai-provider/





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) bc8f7e4d-1edb-45eb-99ba-b7938c6e05d5 Thu, 23 Apr 2026 10:25:00 -0400 1716 1 78 false Building AI Debugging Agents Into Airflow DAGs at Jeppesen ForeFlight with Samantha Blaney Cuevas Building AI Debugging Agents Into Airflow DAGs at Jeppesen ForeFlight with Samantha Blaney Cuevas Aviation data pipelines run on strict 28-day publication cycles, and the margin for error is zero. In this episode, we're joined by Samantha Blaney Cuevas, Software Engineer at Jeppesen ForeFlight, to explore how her team orchestrates a complex, time-sensitive data pipeline with Airflow and where AI is starting to fit into that picture.


Key Takeaways:


00:00 Introduction.

04:05 Airflow orchestrates almost all business logic and data transformations across the cycle, with custom timetables built to track busy and slow periods programmatically.

06:10 Cycle-aware sensing tasks handle irregular source deliveries, including duplicates and early or late arrivals, without disrupting the pipeline.

08:07 The two main AI use cases are pipeline debugging and cycle awareness — both designed to reduce the manual overhead of monitoring a complex DAG dependency graph.

09:03 The Data Port agent is a two-task DAG that routes Slack pipeline alerts to either a predefined command list or an AI token, depending on whether the fix is already known.

13:10 AI is still in development at Jeppesen ForeFlight — the team is focused on token efficiency and scoping how much autonomy to give agents across different environments.

15:04 Airflow setup and MCP configuration were straightforward — the harder design work was deciding which environments agents could access across QA staging and production.

17:06 Airflow's skills repo and agent tooling are helping onboard new developers and extend pipeline awareness to analysts who work alongside engineers on the cycle.

19:10 Samantha would like to see single-task retries with different parameters in Airflow — resetting one task without clearing the full pipeline run.

21:05 A future AI use case under consideration is live DAG editing and re-upload within Airflow to make one-off fixes without halting pipeline progress.


Resources Mentioned:


Samantha Blaney Cuevas

https://www.linkedin.com/in/samantha-blaney/


Jeppesen ForeFlight | LinkedIn

https://www.linkedin.com/company/jeppesen-foreflight/


Jeppesen ForeFlight | Website

http://www.foreflight.com


Astronomer Airflow Skills Repo

http://www.github.com/astronomer/airflow-llm-providers-demo


Apache Airflow 

https://airflow.apache.org/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> Aviation data pipelines run on strict 28-day publication cycles, and the margin for error is zero. In this episode, we're joined by Samantha Blaney Cuevas, Software Engineer at Jeppesen ForeFlight, to explore how her team orchestrates a complex, time-sensitive data pipeline with Airflow and where AI is starting to fit into that picture.


Key Takeaways:


00:00 Introduction.

04:05 Airflow orchestrates almost all business logic and data transformations across the cycle, with custom timetables built to track busy and slow periods programmatically.

06:10 Cycle-aware sensing tasks handle irregular source deliveries, including duplicates and early or late arrivals, without disrupting the pipeline.

08:07 The two main AI use cases are pipeline debugging and cycle awareness — both designed to reduce the manual overhead of monitoring a complex DAG dependency graph.

09:03 The Data Port agent is a two-task DAG that routes Slack pipeline alerts to either a predefined command list or an AI token, depending on whether the fix is already known.

13:10 AI is still in development at Jeppesen ForeFlight — the team is focused on token efficiency and scoping how much autonomy to give agents across different environments.

15:04 Airflow setup and MCP configuration were straightforward — the harder design work was deciding which environments agents could access across QA staging and production.

17:06 Airflow's skills repo and agent tooling are helping onboard new developers and extend pipeline awareness to analysts who work alongside engineers on the cycle.

19:10 Samantha would like to see single-task retries with different parameters in Airflow — resetting one task without clearing the full pipeline run.

21:05 A future AI use case under consideration is live DAG editing and re-upload within Airflow to make one-off fixes without halting pipeline progress.


Resources Mentioned:


Samantha Blaney Cuevas

https://www.linkedin.com/in/samantha-blaney/


Jeppesen ForeFlight | LinkedIn

https://www.linkedin.com/company/jeppesen-foreflight/


Jeppesen ForeFlight | Website

http://www.foreflight.com


Astronomer Airflow Skills Repo

http://www.github.com/astronomer/airflow-llm-providers-demo


Apache Airflow 

https://airflow.apache.org/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 58235e2f-edef-47ab-b968-2a4b90b2d50f Thu, 16 Apr 2026 00:00:00 -0400 1337 1 77 false Introducing Airflow 3.2 Introducing Airflow 3.2 We introduce Airflow 3.2 and its updates for teams that build and operate data pipelines.

Astronomer’s Head of Customer Education, Marc Lamberti, and Senior Manager of Developer Relations, Kenten Danas, break down what’s new, from asset partitioning to Async Python tasks and DAG versioning. They explore how these updates improve scheduling, performance and observability in production workflows.


Key Takeaways:


00:00 Introduction.

02:10 Airflow 3 architecture separates workers from the metadata database.

03:05 Plugin versioning and UI-based backfills simplify operations.

06:20 Asset partitioning enables granular, partition-level scheduling.

07:15 Triggering DAGs on partitions instead of full datasets.

11:05 Deferrable operators reduce worker slot usage.

12:00 Async operators reduce database pressure and overhead.

14:10 Async improves throughput, not single task speed.

22:20 Inlets and outlets improve asset lineage visibility.

23:00 DAG version markers show changes directly in the UI.


Resources Mentioned:


Marc Lamberti

https://www.linkedin.com/in/marclamberti/


Apache Airflow 

https://airflow.apache.org/


Astronomer | LinkedIn

https://www.linkedin.com/company/astronomer/


Astronomer | Website

https://www.astronomer.io/


3.2 Webinar

https://www.astronomer.io/events/webinars/introducing-airflow-3-2-video


Asset Partitioning Guide

https://www.astronomer.io/docs/learn/airflow-partitioned-runs


Asynchronous Processes Guide

https://www.astronomer.io/docs/learn/deferrable-operators


Release Notes

https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#airflow-3-2-0-2026-04-07


Provider Registry

https://airflow.apache.org/registry/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> We introduce Airflow 3.2 and its updates for teams that build and operate data pipelines.

Astronomer’s Head of Customer Education, Marc Lamberti, and Senior Manager of Developer Relations, Kenten Danas, break down what’s new, from asset partitioning to Async Python tasks and DAG versioning. They explore how these updates improve scheduling, performance and observability in production workflows.


Key Takeaways:


00:00 Introduction.

02:10 Airflow 3 architecture separates workers from the metadata database.

03:05 Plugin versioning and UI-based backfills simplify operations.

06:20 Asset partitioning enables granular, partition-level scheduling.

07:15 Triggering DAGs on partitions instead of full datasets.

11:05 Deferrable operators reduce worker slot usage.

12:00 Async operators reduce database pressure and overhead.

14:10 Async improves throughput, not single task speed.

22:20 Inlets and outlets improve asset lineage visibility.

23:00 DAG version markers show changes directly in the UI.


Resources Mentioned:


Marc Lamberti

https://www.linkedin.com/in/marclamberti/


Apache Airflow 

https://airflow.apache.org/


Astronomer | LinkedIn

https://www.linkedin.com/company/astronomer/


Astronomer | Website

https://www.astronomer.io/


3.2 Webinar

https://www.astronomer.io/events/webinars/introducing-airflow-3-2-video


Asset Partitioning Guide

https://www.astronomer.io/docs/learn/airflow-partitioned-runs


Asynchronous Processes Guide

https://www.astronomer.io/docs/learn/deferrable-operators


Release Notes

https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#airflow-3-2-0-2026-04-07


Provider Registry

https://airflow.apache.org/registry/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 87849d21-4021-45d4-9836-741e29fe6088 Thu, 09 Apr 2026 00:00:00 -0400 1582 1 76 false Reflections on a Decade of Data Engineering at Seattle Data Guy Reflections on a Decade of Data Engineering at Seattle Data Guy Lessons from the past decade of data engineering reveal how much the ecosystem has changed and what has stayed surprisingly consistent.


In this episode, Benjamin Rogojan, Owner and Data Consultant at Seattle Data Guy, joins us to reflect on how the data engineering landscape has evolved alongside Apache Airflow. We explore when Airflow makes sense as an orchestrator, why batch processing is still dominant and how AI is reshaping the workflows and responsibilities of modern data engineers.


Key Takeaways:


00:00 Introduction.

03:00 Airflow becomes valuable when workflows involve many pipelines, teams and dependencies.

05:00 Data engineers are still focused on making data accessible and aligning work with business needs.

05:30 Batch pipelines remain the most common approach even as real-time use cases grow.

07:45 Many “real-time” requests are actually event-driven batch workflows.

09:00 Airflow replaced many custom-built pipeline systems with built-in dependency management.

11:00 Modern orchestration tools often build on Airflow concepts or differentiate from them.

14:00 AI can assist with writing SQL and pipelines but still requires experienced engineers.

15:30 Organizations are collecting increasingly granular data creating more engineering demand.

19:00 The data stack has shifted rapidly from Hadoop-era systems to modern cloud platforms.


Resources Mentioned:


Benjamin Rogojan

https://www.linkedin.com/in/benjaminrogojan/


Seattle Data Guy

https://www.linkedin.com/company/seattle-data-guy/


Apache Airflow

https://airflow.apache.org


Airflow Summit / Airflow Conference

https://airflowsummit.org


Snowflake

https://www.snowflake.com


HubSpot Data Sharing / APIs

https://developers.hubspot.com


MLflow

https://mlflow.org




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> Lessons from the past decade of data engineering reveal how much the ecosystem has changed and what has stayed surprisingly consistent.


In this episode, Benjamin Rogojan, Owner and Data Consultant at Seattle Data Guy, joins us to reflect on how the data engineering landscape has evolved alongside Apache Airflow. We explore when Airflow makes sense as an orchestrator, why batch processing is still dominant and how AI is reshaping the workflows and responsibilities of modern data engineers.


Key Takeaways:


00:00 Introduction.

03:00 Airflow becomes valuable when workflows involve many pipelines, teams and dependencies.

05:00 Data engineers are still focused on making data accessible and aligning work with business needs.

05:30 Batch pipelines remain the most common approach even as real-time use cases grow.

07:45 Many “real-time” requests are actually event-driven batch workflows.

09:00 Airflow replaced many custom-built pipeline systems with built-in dependency management.

11:00 Modern orchestration tools often build on Airflow concepts or differentiate from them.

14:00 AI can assist with writing SQL and pipelines but still requires experienced engineers.

15:30 Organizations are collecting increasingly granular data creating more engineering demand.

19:00 The data stack has shifted rapidly from Hadoop-era systems to modern cloud platforms.


Resources Mentioned:


Benjamin Rogojan

https://www.linkedin.com/in/benjaminrogojan/


Seattle Data Guy

https://www.linkedin.com/company/seattle-data-guy/


Apache Airflow

https://airflow.apache.org


Airflow Summit / Airflow Conference

https://airflowsummit.org


Snowflake

https://www.snowflake.com


HubSpot Data Sharing / APIs

https://developers.hubspot.com


MLflow

https://mlflow.org




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 0b9f65a4-85da-4b01-8dc0-975611636cf7 Fri, 03 Apr 2026 02:06:00 -0400 1572 1 75 false Managing Data Quality and Governance With Airflow at Credit Karma with Ashir Alam Managing Data Quality and Governance With Airflow at Credit Karma with Ashir Alam Data quality is not optional when you manage credit data at scale.


In this episode, Ashir Alam, Senior Data Engineer at Credit Karma, joins us to share how his team acts as the gatekeeper for credit data ingestion, how they standardize data quality with Airflow and DAG Factory and how they scale safely across thousands of DAGs. We explore how governance, PII protection and orchestration come together inside a modern data platform.



Key Takeaways:


00:00 Introduction.

01:00 Overview of Credit Karma’s products and financial data ecosystem.

02:00 The team acts as gatekeepers for ingesting data from TransUnion and Equifax.

03:00 Why PII handling and controlled downstream access led to adopting Airflow.

04:00 BigQuery as the warehouse and Airflow as the primary orchestrator.

05:00 Why data quality and governance are critical in financial systems.

07:00 Why Airflow was selected: ease of use and unified ETL plus data quality.

09:00 Introduction to DAG Factory and YAML-based DAG generation.

10:00 GitHub executor creates PR-driven DAG workflows with CI checks.

12:00 BigQuery operators, structured checks and custom Slack and PagerDuty alerts.

13:00 Failed checks stop ETL pipelines and trigger notifications.

17:00 Scaling DAG Factory across thousands of DAGs and runtime vs compile-time concerns.

19:00 Future improvements: better defaults, retries and GenAI workflows in Airflow.


Resources Mentioned:


Ashir Alam

https://www.linkedin.com/in/ashir-alam/


Credit Karma

https://www.linkedin.com/company/intuit-credit-karma/


Apache Airflow

https://airflow.apache.org/


DAG Factory

https://github.com/astronomer/dag-factory


BigQuery (Google Cloud)

https://cloud.google.com/bigquery


GitHub

https://github.com/


Slack

https://slack.com/


PagerDuty

https://www.pagerduty.com/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> Data quality is not optional when you manage credit data at scale.


In this episode, Ashir Alam, Senior Data Engineer at Credit Karma, joins us to share how his team acts as the gatekeeper for credit data ingestion, how they standardize data quality with Airflow and DAG Factory and how they scale safely across thousands of DAGs. We explore how governance, PII protection and orchestration come together inside a modern data platform.



Key Takeaways:


00:00 Introduction.

01:00 Overview of Credit Karma’s products and financial data ecosystem.

02:00 The team acts as gatekeepers for ingesting data from TransUnion and Equifax.

03:00 Why PII handling and controlled downstream access led to adopting Airflow.

04:00 BigQuery as the warehouse and Airflow as the primary orchestrator.

05:00 Why data quality and governance are critical in financial systems.

07:00 Why Airflow was selected: ease of use and unified ETL plus data quality.

09:00 Introduction to DAG Factory and YAML-based DAG generation.

10:00 GitHub executor creates PR-driven DAG workflows with CI checks.

12:00 BigQuery operators, structured checks and custom Slack and PagerDuty alerts.

13:00 Failed checks stop ETL pipelines and trigger notifications.

17:00 Scaling DAG Factory across thousands of DAGs and runtime vs compile-time concerns.

19:00 Future improvements: better defaults, retries and GenAI workflows in Airflow.


Resources Mentioned:


Ashir Alam

https://www.linkedin.com/in/ashir-alam/


Credit Karma

https://www.linkedin.com/company/intuit-credit-karma/


Apache Airflow

https://airflow.apache.org/


DAG Factory

https://github.com/astronomer/dag-factory


BigQuery (Google Cloud)

https://cloud.google.com/bigquery


GitHub

https://github.com/


Slack

https://slack.com/


PagerDuty

https://www.pagerduty.com/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 5694749b-ef0f-484f-893b-f86bc31dcea2 Thu, 26 Mar 2026 05:33:00 -0400 1324 1 74 false Open Source Airflow Contributions and Performance Improvements at G-Research with Christos Bisias Open Source Airflow Contributions and Performance Improvements at G-Research with Christos Bisias Modern Airflow isn’t just orchestration. It's a contribution.



In this episode, we explore how open source investment drives real performance gains and deeper observability.


We’re joined by Christos Bisias, Open Source Software Engineer, Apache Airflow at G-Research, to discuss how his team uses Airflow for large-scale data transformations, contributes upstream and improves scheduler throughput and OpenTelemetry support. From trace-level observability to CI-enforced metrics governance and a major scheduler optimization, this conversation spans strategy, engineering and community impact.


Key Takeaways:


00:00 Introduction.

01:20 How G-Research applies machine learning and big data to predict financial market movements.

02:15 Contributing to open source is a business decision.

03:10 Maintaining a fork is costly.

04:30 OpenTelemetry collects metrics, logs and traces to provide deep system visibility.

06:10 Custom spans help identify bottlenecks inside tasks and enable performance optimization.

08:05 OpenTelemetry integration works properly in Airflow 3.0 and above.

10:00 A YAML-based metrics registry with CI enforcement ensures consistency between docs and exported metrics.

12:10 Scheduler throughput improved significantly by applying concurrency limits earlier in the database query. 

15:20 Future Task SDK changes may enable language-agnostic DAG authoring beyond Python.


Resources Mentioned:


Christos Bisias

https://www.linkedin.com/in/xbis/


G-Research

https://www.linkedin.com/company/g-research/


Apache Airflow

https://airflow.apache.org/


OpenTelemetry

https://opentelemetry.io/


Prometheus

https://prometheus.io/


Grafana

https://grafana.com/


Jaeger

https://www.jaegertracing.io/





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> Modern Airflow isn’t just orchestration. It's a contribution.



In this episode, we explore how open source investment drives real performance gains and deeper observability.


We’re joined by Christos Bisias, Open Source Software Engineer, Apache Airflow at G-Research, to discuss how his team uses Airflow for large-scale data transformations, contributes upstream and improves scheduler throughput and OpenTelemetry support. From trace-level observability to CI-enforced metrics governance and a major scheduler optimization, this conversation spans strategy, engineering and community impact.


Key Takeaways:


00:00 Introduction.

01:20 How G-Research applies machine learning and big data to predict financial market movements.

02:15 Contributing to open source is a business decision.

03:10 Maintaining a fork is costly.

04:30 OpenTelemetry collects metrics, logs and traces to provide deep system visibility.

06:10 Custom spans help identify bottlenecks inside tasks and enable performance optimization.

08:05 OpenTelemetry integration works properly in Airflow 3.0 and above.

10:00 A YAML-based metrics registry with CI enforcement ensures consistency between docs and exported metrics.

12:10 Scheduler throughput improved significantly by applying concurrency limits earlier in the database query. 

15:20 Future Task SDK changes may enable language-agnostic DAG authoring beyond Python.


Resources Mentioned:


Christos Bisias

https://www.linkedin.com/in/xbis/


G-Research

https://www.linkedin.com/company/g-research/


Apache Airflow

https://airflow.apache.org/


OpenTelemetry

https://opentelemetry.io/


Prometheus

https://prometheus.io/


Grafana

https://grafana.com/


Jaeger

https://www.jaegertracing.io/





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) fca554c0-14c3-482a-a1f6-9c426cda6c45 Thu, 19 Mar 2026 00:00:00 -0400 1063 1 73 false Automating Threat Intelligence Using Airflow with Karan Alang Automating Threat Intelligence Using Airflow with Karan Alang In this episode, Karan Alang, Principal Software Engineer at Versa Networks, joins the conversation to discuss how Airflow can be used to automate threat intelligence in modern cybersecurity environments. He explains the growing scale of cloud computing, the profitability of hacking and the shortage of SOC analysts. Karan also outlines a novel architecture that combines Airflow, XDR, graph databases and LLMs to orchestrate automated threat detection and response.


Key Takeaways:


00:00 Introduction.

05:00 Organizations face massive log volumes and a shortage of SOC analysts.

07:00 The solution integrates Airflow, XDR, Neo4j graph databases and LLMs into one architecture.

08:00 MITRE ATT&CK provides a global framework for mapping tactics and techniques.

11:00 Airflow acts as the orchestration backbone for ingestion graph transformation and LLM workflows.

13:00 Graph databases provide a full relationship view of attackers’ systems and entities.

14:00 LLMs automate mapping activity to MITRE ATT&CK and assign explainable risk scores.

17:00 Traditional signature-based detection allows lateral movement and exfiltration before teams can react.

18:00 End-to-end automation is essential to mitigating modern cybersecurity threats.

20:00 Future opportunities include deeper LLM integration as first-class citizens within Airflow.


Resources Mentioned:


Karan Alang

https://www.linkedin.com/in/karan-alang-4173437


Versa Networks | LinkedIn

https://www.linkedin.com/company/versa-networks


Versa Networks | Website

https://versa-networks.com


Google Cloud Composer (Managed Airflow on GCP)

https://cloud.google.com/composer


Microsoft Defender XDR 

https://www.microsoft.com/es-es/security/business/siem-and-xdr/microsoft-defender-xdr


Neo4j (Graph Database)

https://neo4j.com


MITRE ATT&CK Framework

https://attack.mitre.org




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> In this episode, Karan Alang, Principal Software Engineer at Versa Networks, joins the conversation to discuss how Airflow can be used to automate threat intelligence in modern cybersecurity environments. He explains the growing scale of cloud computing, the profitability of hacking and the shortage of SOC analysts. Karan also outlines a novel architecture that combines Airflow, XDR, graph databases and LLMs to orchestrate automated threat detection and response.


Key Takeaways:


00:00 Introduction.

05:00 Organizations face massive log volumes and a shortage of SOC analysts.

07:00 The solution integrates Airflow, XDR, Neo4j graph databases and LLMs into one architecture.

08:00 MITRE ATT&CK provides a global framework for mapping tactics and techniques.

11:00 Airflow acts as the orchestration backbone for ingestion graph transformation and LLM workflows.

13:00 Graph databases provide a full relationship view of attackers’ systems and entities.

14:00 LLMs automate mapping activity to MITRE ATT&CK and assign explainable risk scores.

17:00 Traditional signature-based detection allows lateral movement and exfiltration before teams can react.

18:00 End-to-end automation is essential to mitigating modern cybersecurity threats.

20:00 Future opportunities include deeper LLM integration as first-class citizens within Airflow.


Resources Mentioned:


Karan Alang

https://www.linkedin.com/in/karan-alang-4173437


Versa Networks | LinkedIn

https://www.linkedin.com/company/versa-networks


Versa Networks | Website

https://versa-networks.com


Google Cloud Composer (Managed Airflow on GCP)

https://cloud.google.com/composer


Microsoft Defender XDR 

https://www.microsoft.com/es-es/security/business/siem-and-xdr/microsoft-defender-xdr


Neo4j (Graph Database)

https://neo4j.com


MITRE ATT&CK Framework

https://attack.mitre.org




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 6f656ed3-a808-4318-aa10-2e7c0b4e15f5 Thu, 12 Mar 2026 02:58:00 -0400 1334 1 72 false Using Plugins To Customize Airflow at Ponder Labs with Egor Tarasenko Using Plugins To Customize Airflow at Ponder Labs with Egor Tarasenko In this episode, we explore how teams scale Apache Airflow in complex environments and what it takes to make orchestration work across many stakeholders. We look at real-world challenges around visibility, ownership and predictability as data platforms grow.


Egor Tarasenko, Data and AI Engineer at Ponder Labs, joins us to share how Ponder Labs customizes Airflow for education organizations using plugins, event-driven architectures and AI-powered tooling. He explains how his team supports large charter school networks and why structure, consistency and extensibility become critical at scale.


Key Takeaways:


00:00 Introduction.

01:21 Ponder Labs helps education organizations bring data from many systems together so it becomes useful for teachers, school leaders and administrators.

03:10 Airflow serves as the backbone for orchestrating ingestion, transformation and reverse ETL across client data platforms.

05:43 Everything is triggered from Airflow to maintain dependency, visibility and a single operational picture.

09:05 Managing hundreds of DAGs requires a focus on structure, visibility and consistency across teams.

09:51 Treating DAGs like APIs helps teams scale without needing deep knowledge of upstream logic.

12:00 Custom plugins like schedule insights help predict DAG run times across layered dependencies.

15:00 AI-powered Airflow chat enables non-technical stakeholders to understand DAG ownership dependencies and cluster activity.

22:06 Migrating plugins to Airflow 3 improves developer experience through cleaner APIs and faster extensibility.


Resources Mentioned:


Egor Tarasenko

https://www.linkedin.com/in/egorseno/


Apache Airflow

https://airflow.apache.org


dbt

https://www.getdbt.com


Astronomer Astro Platform

https://www.astronomer.io


Egor Tarasenko on Substack 

https://egortarasenko.substack.com




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> In this episode, we explore how teams scale Apache Airflow in complex environments and what it takes to make orchestration work across many stakeholders. We look at real-world challenges around visibility, ownership and predictability as data platforms grow.


Egor Tarasenko, Data and AI Engineer at Ponder Labs, joins us to share how Ponder Labs customizes Airflow for education organizations using plugins, event-driven architectures and AI-powered tooling. He explains how his team supports large charter school networks and why structure, consistency and extensibility become critical at scale.


Key Takeaways:


00:00 Introduction.

01:21 Ponder Labs helps education organizations bring data from many systems together so it becomes useful for teachers, school leaders and administrators.

03:10 Airflow serves as the backbone for orchestrating ingestion, transformation and reverse ETL across client data platforms.

05:43 Everything is triggered from Airflow to maintain dependency, visibility and a single operational picture.

09:05 Managing hundreds of DAGs requires a focus on structure, visibility and consistency across teams.

09:51 Treating DAGs like APIs helps teams scale without needing deep knowledge of upstream logic.

12:00 Custom plugins like schedule insights help predict DAG run times across layered dependencies.

15:00 AI-powered Airflow chat enables non-technical stakeholders to understand DAG ownership dependencies and cluster activity.

22:06 Migrating plugins to Airflow 3 improves developer experience through cleaner APIs and faster extensibility.


Resources Mentioned:


Egor Tarasenko

https://www.linkedin.com/in/egorseno/


Apache Airflow

https://airflow.apache.org


dbt

https://www.getdbt.com


Astronomer Astro Platform

https://www.astronomer.io


Egor Tarasenko on Substack 

https://egortarasenko.substack.com




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 3fc501fc-ae6a-4b8d-a7e0-1a9c0e6c721a Thu, 05 Mar 2026 00:15:00 -0500 1665 1 71 false Scaling Airflow at Wix for Analytics and AI with Ethan Shalev Scaling Airflow at Wix for Analytics and AI with Ethan Shalev Modern data orchestration at scale demands reliability, speed and thoughtful adoption of new tooling. As organizations grow, keeping pipelines efficient while supporting more teams becomes a critical challenge.


In this episode, we’re joined by Ethan Shalev, Data Engineer at Wix, to discuss how Wix operates Airflow at massive scale, migrates to Airflow 3 and uses AI to accelerate development.


Key Takeaways:


00:00 Introduction.

02:13 Wix structures data engineering across multiple product-focused organizations.

03:40 Migrating nearly 8,000 DAGs to Airflow 3 requires careful planning.

04:31 Migration creates an opportunity to remove long-standing legacy Airflow code.

05:32 Internal playbooks and Cursor rules standardize and speed up DAG migrations.

07:39 Airflow 3 introduces backfills, DAG versioning and asset-aware scheduling.

09:16 Deferrable operators reduce scheduler congestion in large Airflow environments.

12:54 AI-generated code still requires review and strong testing practices.

14:52 Moving to managed Airflow reduces operational burden on internal platform teams.

15:57 Improving multi-tenancy and UI personalization remains a key Airflow need.


Resources Mentioned:


Ethan Shalev

https://www.linkedin.com/in/eshalev/


Wix | LinkedIn

https://www.linkedin.com/company/wix-com/


Wix | Website

https://www.wix.com/


Apache Airflow

https://airflow.apache.org/


Astronomer

https://www.astronomer.io/


Trino

https://trino.io/


Apache Iceberg

https://iceberg.apache.org/


Cursor

https://cursor.sh/


Airflow Summit

https://airflowsummit.org/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> Modern data orchestration at scale demands reliability, speed and thoughtful adoption of new tooling. As organizations grow, keeping pipelines efficient while supporting more teams becomes a critical challenge.


In this episode, we’re joined by Ethan Shalev, Data Engineer at Wix, to discuss how Wix operates Airflow at massive scale, migrates to Airflow 3 and uses AI to accelerate development.


Key Takeaways:


00:00 Introduction.

02:13 Wix structures data engineering across multiple product-focused organizations.

03:40 Migrating nearly 8,000 DAGs to Airflow 3 requires careful planning.

04:31 Migration creates an opportunity to remove long-standing legacy Airflow code.

05:32 Internal playbooks and Cursor rules standardize and speed up DAG migrations.

07:39 Airflow 3 introduces backfills, DAG versioning and asset-aware scheduling.

09:16 Deferrable operators reduce scheduler congestion in large Airflow environments.

12:54 AI-generated code still requires review and strong testing practices.

14:52 Moving to managed Airflow reduces operational burden on internal platform teams.

15:57 Improving multi-tenancy and UI personalization remains a key Airflow need.


Resources Mentioned:


Ethan Shalev

https://www.linkedin.com/in/eshalev/


Wix | LinkedIn

https://www.linkedin.com/company/wix-com/


Wix | Website

https://www.wix.com/


Apache Airflow

https://airflow.apache.org/


Astronomer

https://www.astronomer.io/


Trino

https://trino.io/


Apache Iceberg

https://iceberg.apache.org/


Cursor

https://cursor.sh/


Airflow Summit

https://airflowsummit.org/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 67f5eff3-53f4-4032-8b6a-14a68d1befca Thu, 26 Feb 2026 00:00:00 -0500 1080 1 70 false Using Airflow To Orchestrate Billions of Events at Addi with Carlos Daniel Puerto Niño Using Airflow To Orchestrate Billions of Events at Addi with Carlos Daniel Puerto Niño Strong data orchestration is as much about culture and visibility as it is about technology. As data platforms scale, teams need systems that reduce cognitive load while increasing reliability and observability.


In this episode, Carlos Daniel Puerto Niño, Senior Analytics Engineer and Data Analyst at Addi, joins us to share how Addi uses Airflow to support batch orchestration, manage organizational complexity and improve monitoring across its data platform.


Key Takeaways:


00:00 Introduction.

01:25 Changes in company strategy increase data platform complexity over time.

04:00 Centralized data teams help manage organizational and technical change.

06:08 Scalable architectures support growing data volumes and use cases.

09:10 Adopting orchestration tools introduces operational and maintenance challenges.

14:43 Abstraction layers lower technical barriers for onboarding new team members.

15:36 Modularity and visibility improve the reliability of data pipelines.

18:14 Integrated monitoring supports faster incident response and resolution.

22:19 Limited access to orchestration metadata constrains proactive analysis.


Resources Mentioned:


Carlos Daniel Puerto Niño

https://www.linkedin.com/in/carlospuertoni%C3%B1o/


Addi | LinkedIn

https://www.linkedin.com/company/addicol/


Addi | Website

https://www.addi.com


Apache Airflow

https://airflow.apache.org/


Astronomer

https://www.astronomer.io/


Databricks

https://www.databricks.com/


dbt

https://www.getdbt.com/


Grafana

https://grafana.com/


Slack

https://slack.com/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> Strong data orchestration is as much about culture and visibility as it is about technology. As data platforms scale, teams need systems that reduce cognitive load while increasing reliability and observability.


In this episode, Carlos Daniel Puerto Niño, Senior Analytics Engineer and Data Analyst at Addi, joins us to share how Addi uses Airflow to support batch orchestration, manage organizational complexity and improve monitoring across its data platform.


Key Takeaways:


00:00 Introduction.

01:25 Changes in company strategy increase data platform complexity over time.

04:00 Centralized data teams help manage organizational and technical change.

06:08 Scalable architectures support growing data volumes and use cases.

09:10 Adopting orchestration tools introduces operational and maintenance challenges.

14:43 Abstraction layers lower technical barriers for onboarding new team members.

15:36 Modularity and visibility improve the reliability of data pipelines.

18:14 Integrated monitoring supports faster incident response and resolution.

22:19 Limited access to orchestration metadata constrains proactive analysis.


Resources Mentioned:


Carlos Daniel Puerto Niño

https://www.linkedin.com/in/carlospuertoni%C3%B1o/


Addi | LinkedIn

https://www.linkedin.com/company/addicol/


Addi | Website

https://www.addi.com


Apache Airflow

https://airflow.apache.org/


Astronomer

https://www.astronomer.io/


Databricks

https://www.databricks.com/


dbt

https://www.getdbt.com/


Grafana

https://grafana.com/


Slack

https://slack.com/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) b2966f43-e21c-49b8-8f26-48adaeea7c33 Thu, 19 Feb 2026 00:10:00 -0500 1489 1 69 false Building Event-Driven Data Pipelines With Airflow 3 at Astrafy with Andrea Bombino Building Event-Driven Data Pipelines With Airflow 3 at Astrafy with Andrea Bombino Real-time data expectations are reshaping how modern data teams think about orchestration and dependencies. As event-driven architectures become more common, teams need to rethink how pipelines react to data changes, rather than schedules.


In this episode, Andrea Bombino, Co-Founder and Head of Analytics Engineering at Astrafy, joins us to discuss how event-driven scheduling in Airflow is evolving and how Astrafy applies it to deliver faster, more responsive data pipelines.


Key Takeaways:


00:00 Introduction.

02:02 Astrafy’s role in guiding clients across the modern data stack.

03:15 Strong DAG dependencies create challenges for time-based scheduling.

04:48 Event-driven pipelines respond to increasing real-time data demands.

05:30 Airflow 3 introduces native support for event-driven orchestration.

06:27 Sensor-based workflows reveal scalability and efficiency limitations.

11:32 Event-driven assets improve efficiency and pipeline elegance.

14:45 Governance and cross-instance coordination emerge as ongoing challenges.


Resources Mentioned:


Andrea Bombino

https://www.linkedin.com/in/andrea-bombino/


Astrafy | LinkedIn

https://www.linkedin.com/company/astrafy/


Astrafy | Website

https://www.astrafy.io


Apache Airflow

https://airflow.apache.org/


Google Cloud

https://cloud.google.com/


Google Pub/Sub

https://cloud.google.com/pubsub


Google BigQuery

https://cloud.google.com/bigquery




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> Real-time data expectations are reshaping how modern data teams think about orchestration and dependencies. As event-driven architectures become more common, teams need to rethink how pipelines react to data changes, rather than schedules.


In this episode, Andrea Bombino, Co-Founder and Head of Analytics Engineering at Astrafy, joins us to discuss how event-driven scheduling in Airflow is evolving and how Astrafy applies it to deliver faster, more responsive data pipelines.


Key Takeaways:


00:00 Introduction.

02:02 Astrafy’s role in guiding clients across the modern data stack.

03:15 Strong DAG dependencies create challenges for time-based scheduling.

04:48 Event-driven pipelines respond to increasing real-time data demands.

05:30 Airflow 3 introduces native support for event-driven orchestration.

06:27 Sensor-based workflows reveal scalability and efficiency limitations.

11:32 Event-driven assets improve efficiency and pipeline elegance.

14:45 Governance and cross-instance coordination emerge as ongoing challenges.


Resources Mentioned:


Andrea Bombino

https://www.linkedin.com/in/andrea-bombino/


Astrafy | LinkedIn

https://www.linkedin.com/company/astrafy/


Astrafy | Website

https://www.astrafy.io


Apache Airflow

https://airflow.apache.org/


Google Cloud

https://cloud.google.com/


Google Pub/Sub

https://cloud.google.com/pubsub


Google BigQuery

https://cloud.google.com/bigquery




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 8d359b8f-dc6c-4850-9c81-05546a615c68 Thu, 12 Feb 2026 08:34:03 -0500 1123 1 70 false Uphold’s Approach to Orchestrating Modern Data Workflows with Jaime Oliveira Uphold’s Approach to Orchestrating Modern Data Workflows with Jaime Oliveira A strong data-driven mindset underpins how fintech teams scale analytics, infrastructure and decision-making across the business.


In this episode, Jaime Oliveira, Lead Data Engineer at Uphold, joins us to discuss how Uphold structures its data organization and orchestration strategy. Jaime shares how the team uses Airflow and dbt to support analytics, reporting and data activation while evolving their approach as the stack grows.


Key Takeaways:


00:00 Introduction.

01:23 A data-driven mindset supports product development and business decisions.

02:55 Diverse ingestion pipelines enable scalable analytics.

04:18 A single orchestration platform simplifies analytics workflows.

05:17 Early experience with orchestration tools shapes engineering practices.

08:16 Analytics orchestration works best when aligned with transformation workflows.

09:25 Infrastructure choices involve tradeoffs in testing, visibility and overhead.

16:39 More collaborative workflow tools could improve accessibility and autonomy.


Resources Mentioned:


Jaime Oliveira

https://www.linkedin.com/in/jaime-oliveira-b075855a/


Uphold | LinkedIn

https://www.linkedin.com/company/upholdinc/


Uphold | Website

https://uphold.com


Apache Airflow

https://airflow.apache.org


dbt

https://www.getdbt.com


Snowflake

https://www.snowflake.com


Kubernetes

https://kubernetes.io


Astronomer Cosmos

https://astronomer.github.io/astronomer-cosmos


Cosmos e-book

https://www.astronomer.io/ebooks/orchestrating-dbt-with-airflow-using-cosmos/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> A strong data-driven mindset underpins how fintech teams scale analytics, infrastructure and decision-making across the business.


In this episode, Jaime Oliveira, Lead Data Engineer at Uphold, joins us to discuss how Uphold structures its data organization and orchestration strategy. Jaime shares how the team uses Airflow and dbt to support analytics, reporting and data activation while evolving their approach as the stack grows.


Key Takeaways:


00:00 Introduction.

01:23 A data-driven mindset supports product development and business decisions.

02:55 Diverse ingestion pipelines enable scalable analytics.

04:18 A single orchestration platform simplifies analytics workflows.

05:17 Early experience with orchestration tools shapes engineering practices.

08:16 Analytics orchestration works best when aligned with transformation workflows.

09:25 Infrastructure choices involve tradeoffs in testing, visibility and overhead.

16:39 More collaborative workflow tools could improve accessibility and autonomy.


Resources Mentioned:


Jaime Oliveira

https://www.linkedin.com/in/jaime-oliveira-b075855a/


Uphold | LinkedIn

https://www.linkedin.com/company/upholdinc/


Uphold | Website

https://uphold.com


Apache Airflow

https://airflow.apache.org


dbt

https://www.getdbt.com


Snowflake

https://www.snowflake.com


Kubernetes

https://kubernetes.io


Astronomer Cosmos

https://astronomer.github.io/astronomer-cosmos


Cosmos e-book

https://www.astronomer.io/ebooks/orchestrating-dbt-with-airflow-using-cosmos/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 61fa3c01-76b7-4f1a-b598-d81ea9270386 Thu, 05 Feb 2026 00:10:00 -0500 1135 1 69 false Modern Airflow Best Practices for Scalable Data Pipelines with Bhavani Ravi Modern Airflow Best Practices for Scalable Data Pipelines with Bhavani Ravi Building reliable data pipelines at scale requires more than writing code. It depends on thoughtful design, infrastructure trade-offs and an understanding of how orchestration platforms evolve over time.


In this episode, Airflow best practices shaped by real-world implementation are examined. Bhavani Ravi, Independent Software Consultant and Apache Airflow Champion, shares lessons on pipeline design, architectural decisions and the evolution of the Airflow ecosystem in modern data environments.


Key Takeaways:


00:00 Introduction.

01:30 Independent consulting supports effective Airflow adoption.

02:38 Early challenges shaped modern Airflow practices.

03:21 Airflow setup has become significantly simpler.

04:30 New features expanded workflow capabilities.

06:03 Frequent releases support long-term sustainability.

07:34 Community and providers strengthen the ecosystem.

10:03 Pipeline design should come before coding.

10:55 Decoupling logic requires careful trade-offs.

13:30 Plugins extend Airflow into new use cases.


Resources Mentioned:


Bhavani Ravi

https://www.linkedin.com/in/bhavanicodes/


Apache Airflow

https://airflow.apache.org/


Kubernetes

https://kubernetes.io/


Azure Fabric

https://learn.microsoft.com/en-us/fabric/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> Building reliable data pipelines at scale requires more than writing code. It depends on thoughtful design, infrastructure trade-offs and an understanding of how orchestration platforms evolve over time.


In this episode, Airflow best practices shaped by real-world implementation are examined. Bhavani Ravi, Independent Software Consultant and Apache Airflow Champion, shares lessons on pipeline design, architectural decisions and the evolution of the Airflow ecosystem in modern data environments.


Key Takeaways:


00:00 Introduction.

01:30 Independent consulting supports effective Airflow adoption.

02:38 Early challenges shaped modern Airflow practices.

03:21 Airflow setup has become significantly simpler.

04:30 New features expanded workflow capabilities.

06:03 Frequent releases support long-term sustainability.

07:34 Community and providers strengthen the ecosystem.

10:03 Pipeline design should come before coding.

10:55 Decoupling logic requires careful trade-offs.

13:30 Plugins extend Airflow into new use cases.


Resources Mentioned:


Bhavani Ravi

https://www.linkedin.com/in/bhavanicodes/


Apache Airflow

https://airflow.apache.org/


Kubernetes

https://kubernetes.io/


Azure Fabric

https://learn.microsoft.com/en-us/fabric/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 98be3cf3-6873-4417-97da-8e4271032592 Thu, 29 Jan 2026 00:05:00 -0500 1030 1 68 false Inside Conviva’s Decision To Power Its Data Platform With Airflow with Han Zhang Inside Conviva’s Decision To Power Its Data Platform With Airflow with Han Zhang Conviva operates at a massive scale, delivering outcome-based intelligence for digital businesses through real-time and batch data processing. As new use cases emerged, the team needed a way to extend a streaming-first architecture without rebuilding core systems.


In this episode, Han Zhang joins us to explain how Conviva uses Apache Airflow as the orchestration backbone for its batch workloads, how the control plane is designed and what trade-offs shaped their platform decisions.


Key Takeaways:


00:00 Introduction.

01:17 Large-scale data platforms require low-latency processing capabilities.

02:08 Batch workloads can complement streaming pipelines for additional use cases.

03:45 An orchestration framework can act as the core coordination layer.

06:12 Batch processing enables workloads that streaming alone cannot support.

08:50 Ecosystem maturity and observability are key orchestration considerations.

10:15 Built-in run history and logs make failures easier to diagnose.

14:20 Platform users can monitor workflows without managing orchestration logic.

17:08 Identity, secrets and scheduling present ongoing optimization challenges.

19:59 Configuration history and change visibility improve operational reliability.


Resources Mentioned:


Han Zhang

https://www.linkedin.com/in/zhanghan177


Conviva | Website

http://www.conviva.com


Apache Airflow

https://airflow.apache.org/


Celery

https://docs.celeryq.dev/


Temporal

https://temporal.io/


Kubernetes

https://kubernetes.io/


LDAP

https://ldap.com/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> Conviva operates at a massive scale, delivering outcome-based intelligence for digital businesses through real-time and batch data processing. As new use cases emerged, the team needed a way to extend a streaming-first architecture without rebuilding core systems.


In this episode, Han Zhang joins us to explain how Conviva uses Apache Airflow as the orchestration backbone for its batch workloads, how the control plane is designed and what trade-offs shaped their platform decisions.


Key Takeaways:


00:00 Introduction.

01:17 Large-scale data platforms require low-latency processing capabilities.

02:08 Batch workloads can complement streaming pipelines for additional use cases.

03:45 An orchestration framework can act as the core coordination layer.

06:12 Batch processing enables workloads that streaming alone cannot support.

08:50 Ecosystem maturity and observability are key orchestration considerations.

10:15 Built-in run history and logs make failures easier to diagnose.

14:20 Platform users can monitor workflows without managing orchestration logic.

17:08 Identity, secrets and scheduling present ongoing optimization challenges.

19:59 Configuration history and change visibility improve operational reliability.


Resources Mentioned:


Han Zhang

https://www.linkedin.com/in/zhanghan177


Conviva | Website

http://www.conviva.com


Apache Airflow

https://airflow.apache.org/


Celery

https://docs.celeryq.dev/


Temporal

https://temporal.io/


Kubernetes

https://kubernetes.io/


LDAP

https://ldap.com/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 57af11f5-cab8-4bf5-8d25-bbe9ddd8d3b5 Thu, 22 Jan 2026 07:05:57 -0500 1305 1 67 false Why Airflow Became the Scheduling Backbone at Condé Nast Technology Lab with Arun Karthik Why Airflow Became the Scheduling Backbone at Condé Nast Technology Lab with Arun Karthik Data platforms are moving from batch-first pipelines to near real-time systems where orchestration, observability, scalability and governance all have to work together.


In this episode, Arun Karthik, Director, Data Solutions Engineering at Condé Nast Technology Lab, joins us to share how data engineering evolves from relational databases and ETL into distributed processing, modern orchestration with Apache Airflow and managed Airflow with Astronomer.


Key Takeaways:


00:00 Introduction.

02:13 Early data systems rely heavily on relational databases and batch-oriented processing models.

07:01 Scheduling requirements evolve beyond fixed time windows as dependencies increase.

10:14 Ease of use and developer experience influence adoption of orchestration frameworks.

13:22 Operating open source orchestration tools requires ongoing engineering effort.

14:45 Managed services help teams reduce infrastructure and maintenance responsibilities.

17:27 Observability improves confidence in pipeline execution and system health.

19:12 Governance considerations grow in importance as data platforms mature.

20:46 Building data systems requires balancing speed, reliability and long-term sustainability.


Resources Mentioned:


Arun Karthik

https://www.linkedin.com/in/earunkarthik/


Condé Nast Technology Lab | LinkedIn

https://www.linkedin.com/company/conde-nast-technology-lab/


Condé Nast Technology Lab | Website

https://www.condenast.com/


Apache Airflow

https://airflow.apache.org/


Astronomer

https://www.astronomer.io/


Apache Spark

https://spark.apache.org/


Apache Hadoop

https://hadoop.apache.org/


Jenkins

https://www.jenkins.io/


dbt Labs

https://www.getdbt.com/product/what-is-dbt


Amazon Web Services

https://aws.amazon.com/free/?trk=54026797-7540-48d8-9f6b-0db2c3a0040c&sc_channel=ps&trk=54026797-7540-48d8-9f6b-0db2c3a0040c&sc_channel=ps&ef_id=CjwKCAiAmp3LBhAkEiwAJM2JUKIc3E2I-hDlF6fRWgZn5n2-RWX-kEDAVApJYd88wwlsiyosV71VixoCmRoQAvD_BwE:G:s&s_kwcid=AL!4422!3!785574063524!e!!g!!amazon%20web%20services!23291338728!189486861095&gad_campaignid=23291338728&gbraid=0AAAAADjHtp813XNbg7azDj5QMwJPbGNqZ&gclid=CjwKCAiAmp3LBhAkEiwAJM2JUKIc3E2I-hDlF6fRWgZn5n2-RWX-kEDAVApJYd88wwlsiyosV71VixoCmRoQAvD_BwE




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> Data platforms are moving from batch-first pipelines to near real-time systems where orchestration, observability, scalability and governance all have to work together.


In this episode, Arun Karthik, Director, Data Solutions Engineering at Condé Nast Technology Lab, joins us to share how data engineering evolves from relational databases and ETL into distributed processing, modern orchestration with Apache Airflow and managed Airflow with Astronomer.


Key Takeaways:


00:00 Introduction.

02:13 Early data systems rely heavily on relational databases and batch-oriented processing models.

07:01 Scheduling requirements evolve beyond fixed time windows as dependencies increase.

10:14 Ease of use and developer experience influence adoption of orchestration frameworks.

13:22 Operating open source orchestration tools requires ongoing engineering effort.

14:45 Managed services help teams reduce infrastructure and maintenance responsibilities.

17:27 Observability improves confidence in pipeline execution and system health.

19:12 Governance considerations grow in importance as data platforms mature.

20:46 Building data systems requires balancing speed, reliability and long-term sustainability.


Resources Mentioned:


Arun Karthik

https://www.linkedin.com/in/earunkarthik/


Condé Nast Technology Lab | LinkedIn

https://www.linkedin.com/company/conde-nast-technology-lab/


Condé Nast Technology Lab | Website

https://www.condenast.com/


Apache Airflow

https://airflow.apache.org/


Astronomer

https://www.astronomer.io/


Apache Spark

https://spark.apache.org/


Apache Hadoop

https://hadoop.apache.org/


Jenkins

https://www.jenkins.io/


dbt Labs

https://www.getdbt.com/product/what-is-dbt


Amazon Web Services

https://aws.amazon.com/free/?trk=54026797-7540-48d8-9f6b-0db2c3a0040c&sc_channel=ps&trk=54026797-7540-48d8-9f6b-0db2c3a0040c&sc_channel=ps&ef_id=CjwKCAiAmp3LBhAkEiwAJM2JUKIc3E2I-hDlF6fRWgZn5n2-RWX-kEDAVApJYd88wwlsiyosV71VixoCmRoQAvD_BwE:G:s&s_kwcid=AL!4422!3!785574063524!e!!g!!amazon%20web%20services!23291338728!189486861095&gad_campaignid=23291338728&gbraid=0AAAAADjHtp813XNbg7azDj5QMwJPbGNqZ&gclid=CjwKCAiAmp3LBhAkEiwAJM2JUKIc3E2I-hDlF6fRWgZn5n2-RWX-kEDAVApJYd88wwlsiyosV71VixoCmRoQAvD_BwE




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) c06b5dc2-ae81-418c-af79-cebf6fa1e81a Thu, 15 Jan 2026 00:00:00 -0500 1456 1 66 false The Role of Airflow in Building Smarter ML Pipelines at Vivian Health with Max Calehuff The Role of Airflow in Building Smarter ML Pipelines at Vivian Health with Max Calehuff The integration of data orchestration and machine learning is critical to operational efficiency in healthcare tech. Vivian Health leverages Airflow to power both its ETL pipelines and ML workflows while maintaining strict compliance standards.


Max Calehuff, Lead Data Engineer at Vivian Health, joins us to discuss how his team uses Airflow for ML ops, regulatory compliance and large-scale data orchestration. He also shares insights into upgrading to Airflow 3 and the importance of balancing flexibility with security in a healthcare environment.


Key Takeaways:


00:00 Introduction.

04:21 The role of Airflow in managing ETL pipelines and ML retraining.

06:23 Using AWS SageMaker for ML training and deployment.

07:47 Why Airflow’s versatility makes it ideal for MLOps.

10:50 The importance of documentation and best practices for engineering teams.

13:44 Automating anonymization of user data for compliance.

15:30 The benefits of remote execution in Airflow 3 for regulated industries.

18:16 Quality-of-life improvements and desired features in future Airflow versions.


Resources Mentioned:


Max Calehuff

https://www.linkedin.com/in/maxwell-calehuff/


Vivian Health | LinkedIn

https://www.linkedin.com/company/vivianhealth/


Vivian Health | Website

https://www.vivian.com


Apache Airflow

https://airflow.apache.org/


Astronomer

https://www.astronomer.io/


AWS SageMaker

https://www.google.com/aclk?sa=L&ai=DChsSEwj3-fbz1tiQAxWXlKYDHXUBBVoYACICCAEQABoCdGI&ae=2&aspm=1&co=1&ase=2&gclid=Cj0KCQiA5abIBhCaARIsAM3-zFWbfj2olUvX4dqoiYNaE3q2fMf_ZifRjmbKNQCVX7D6ZMClaUXUkFkaAuwmEALw_wcB&cid=CAASQuRoMccxWhBvMq-1Uez3XOZti1ul7mTDotKvSMoDHv0q2xCsyS2FzMptO5dJf3tmfkLRu22TtD8ChTmdjvs6YetTjQ&cce=2&category=acrcp_v1_35&sig=AOD64_2xE2xolEEVbpDb56qXQluxTzs-Aw&q&nis=4&adurl&ved=2ahUKEwj7le3z1tiQAxWXcvUHHfZePbAQ0Qx6BAgUEAE


dbtLabs

https://www.getdbt.com/


Cosmos

https://github.com/astronomer/astronomer-cosmos


Split

https://www.split.io/


Snowflake

https://www.snowflake.com/en/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> The integration of data orchestration and machine learning is critical to operational efficiency in healthcare tech. Vivian Health leverages Airflow to power both its ETL pipelines and ML workflows while maintaining strict compliance standards.


Max Calehuff, Lead Data Engineer at Vivian Health, joins us to discuss how his team uses Airflow for ML ops, regulatory compliance and large-scale data orchestration. He also shares insights into upgrading to Airflow 3 and the importance of balancing flexibility with security in a healthcare environment.


Key Takeaways:


00:00 Introduction.

04:21 The role of Airflow in managing ETL pipelines and ML retraining.

06:23 Using AWS SageMaker for ML training and deployment.

07:47 Why Airflow’s versatility makes it ideal for MLOps.

10:50 The importance of documentation and best practices for engineering teams.

13:44 Automating anonymization of user data for compliance.

15:30 The benefits of remote execution in Airflow 3 for regulated industries.

18:16 Quality-of-life improvements and desired features in future Airflow versions.


Resources Mentioned:


Max Calehuff

https://www.linkedin.com/in/maxwell-calehuff/


Vivian Health | LinkedIn

https://www.linkedin.com/company/vivianhealth/


Vivian Health | Website

https://www.vivian.com


Apache Airflow

https://airflow.apache.org/


Astronomer

https://www.astronomer.io/


AWS SageMaker

https://www.google.com/aclk?sa=L&ai=DChsSEwj3-fbz1tiQAxWXlKYDHXUBBVoYACICCAEQABoCdGI&ae=2&aspm=1&co=1&ase=2&gclid=Cj0KCQiA5abIBhCaARIsAM3-zFWbfj2olUvX4dqoiYNaE3q2fMf_ZifRjmbKNQCVX7D6ZMClaUXUkFkaAuwmEALw_wcB&cid=CAASQuRoMccxWhBvMq-1Uez3XOZti1ul7mTDotKvSMoDHv0q2xCsyS2FzMptO5dJf3tmfkLRu22TtD8ChTmdjvs6YetTjQ&cce=2&category=acrcp_v1_35&sig=AOD64_2xE2xolEEVbpDb56qXQluxTzs-Aw&q&nis=4&adurl&ved=2ahUKEwj7le3z1tiQAxWXcvUHHfZePbAQ0Qx6BAgUEAE


dbtLabs

https://www.getdbt.com/


Cosmos

https://github.com/astronomer/astronomer-cosmos


Split

https://www.split.io/


Snowflake

https://www.snowflake.com/en/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) f813c79f-715b-4899-9132-1510e2fbee95 Thu, 11 Dec 2025 04:02:06 -0500 1170 1 65 false Scaling Airflow to 11,000 DAGs Across Three Regions at Intercom with András Gombosi and Paul Vickers Scaling Airflow to 11,000 DAGs Across Three Regions at Intercom with András Gombosi and Paul Vickers The evolution of Intercom’s data infrastructure reveals how a well-built orchestration system can scale to serve global needs. With thousands of DAGs powering analytics, AI and customer operations, the team’s approach combines technical depth with organizational insight.


In this episode, András Gombosi, Senior Engineering Manager of Data Infra and Analytics Engineering, and Paul Vickers, Principal Engineer, both at Intercom, share how they built one of the largest Airflow deployments in production and enabled self-serve data platforms across teams.


Key Takeaways:


00:00 Introduction.

04:24 Community input encourages confident adoption of a common platform.

08:50 Self-serve workflows require consistent guardrails and review.

09:25 Internal infrastructure support accelerates scalable deployments.

13:26 Batch LLM processing benefits from a configuration-driven design.

15:20 Standardized development environments enable effective AI-assisted work.

19:58 Applied AI enhances internal analysis and operational enablement.

27:27 Strong test coverage and staged upgrades protect stability.

30:36 Proactive observability and on-call ownership improve outcomes.


Resources Mentioned:


András Gombosi

https://www.linkedin.com/in/andrasgombosi/


Paul Vickers

https://www.linkedin.com/in/paul-vickers-a22b76a3/


Intercom | LinkedIn

https://www.linkedin.com/company/intercom/


Intercom | Website

https://www.intercom.com


Apache Airflow

https://airflow.apache.org/


dbtLabs

https://www.getdbt.com/


Snowflake Cortex AI

https://www.snowflake.com/en/product/features/cortex/


Datadog

https://www.datadoghq.com/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> The evolution of Intercom’s data infrastructure reveals how a well-built orchestration system can scale to serve global needs. With thousands of DAGs powering analytics, AI and customer operations, the team’s approach combines technical depth with organizational insight.


In this episode, András Gombosi, Senior Engineering Manager of Data Infra and Analytics Engineering, and Paul Vickers, Principal Engineer, both at Intercom, share how they built one of the largest Airflow deployments in production and enabled self-serve data platforms across teams.


Key Takeaways:


00:00 Introduction.

04:24 Community input encourages confident adoption of a common platform.

08:50 Self-serve workflows require consistent guardrails and review.

09:25 Internal infrastructure support accelerates scalable deployments.

13:26 Batch LLM processing benefits from a configuration-driven design.

15:20 Standardized development environments enable effective AI-assisted work.

19:58 Applied AI enhances internal analysis and operational enablement.

27:27 Strong test coverage and staged upgrades protect stability.

30:36 Proactive observability and on-call ownership improve outcomes.


Resources Mentioned:


András Gombosi

https://www.linkedin.com/in/andrasgombosi/


Paul Vickers

https://www.linkedin.com/in/paul-vickers-a22b76a3/


Intercom | LinkedIn

https://www.linkedin.com/company/intercom/


Intercom | Website

https://www.intercom.com


Apache Airflow

https://airflow.apache.org/


dbtLabs

https://www.getdbt.com/


Snowflake Cortex AI

https://www.snowflake.com/en/product/features/cortex/


Datadog

https://www.datadoghq.com/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) e7711863-7224-41af-95c5-e27b21bb1ce6 Thu, 04 Dec 2025 03:33:35 -0500 2064 1 65 false How Covestro Turns Airflow Into a Simulation Toolbox with Anja Mackenzie How Covestro Turns Airflow Into a Simulation Toolbox with Anja Mackenzie Building scalable, reproducible workflows for scientific computing often requires bridging the gap between research flexibility and enterprise reliability.


In this episode, Anja MacKenzie, Expert for Cheminformatics at Covestro, explains how her team uses Airflow and Kubernetes to create a shared, self-service platform for computational chemistry.


Key Takeaways:


00:00 Introduction.

06:19 Custom scripts made sharing and reuse difficult.

09:29 Workflows are manually triggered with user traceability.

10:38 Customization supports varied compute requirements.

12:48 Persistent volumes allow tasks to share large amounts of data.

14:25 Custom operators separate logic from infrastructure.

16:43 Modified triggers connect dependent workflows.

18:36 UI plugins enable file uploads and secure access.


Resources Mentioned:


Anja MacKenzie

https://www.linkedin.com/in/anja-mackenzie/


Covestro | LinkedIn

https://www.linkedin.com/company/covestro/


Covestro | Website

https://www.covestro.com


Apache Airflow

https://airflow.apache.org/


Kubernetes

https://kubernetes.io/


Airflow KubernetesPodOperator

https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html


Astronomer

https://www.astronomer.io/


Airflow Academy by Marc Lamberti

https://www.udemy.com/user/lockgfg/?utm_source=adwords&utm_medium=udemyads&utm_campaign=Search_DSA_GammaCatchall_NonP_la.EN_cc.ROW-English&campaigntype=Search&portfolio=ROW-English&language=EN&product=Course&test=&audience=DSA&topic=&priority=Gamma&utm_content=deal4584&utm_term=_._ag_169801645584_._ad_700876640602_._kw__._de_c_._dm__._pl__._ti_dsa-1456167871416_._li_9061346_._pd__._&matchtype=&gad_source=1&gad_campaignid=21341313808&gbraid=0AAAAADROdO1_-I2TMcVyU8F3i1jRXJ24K&gclid=Cj0KCQjwvJHIBhCgARIsAEQnWlC1uYHIRm3y9Q8rPNSuVPNivsxogqfczpKHwhmNho2uKZYC-y0taNQaApU2EALw_wcB


Airflow Documentation

https://airflow.apache.org/docs/


Airflow Plugins

https://airflow.apache.org/docs/apache-airflow/1.10.9/plugins.html




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> Building scalable, reproducible workflows for scientific computing often requires bridging the gap between research flexibility and enterprise reliability.


In this episode, Anja MacKenzie, Expert for Cheminformatics at Covestro, explains how her team uses Airflow and Kubernetes to create a shared, self-service platform for computational chemistry.


Key Takeaways:


00:00 Introduction.

06:19 Custom scripts made sharing and reuse difficult.

09:29 Workflows are manually triggered with user traceability.

10:38 Customization supports varied compute requirements.

12:48 Persistent volumes allow tasks to share large amounts of data.

14:25 Custom operators separate logic from infrastructure.

16:43 Modified triggers connect dependent workflows.

18:36 UI plugins enable file uploads and secure access.


Resources Mentioned:


Anja MacKenzie

https://www.linkedin.com/in/anja-mackenzie/


Covestro | LinkedIn

https://www.linkedin.com/company/covestro/


Covestro | Website

https://www.covestro.com


Apache Airflow

https://airflow.apache.org/


Kubernetes

https://kubernetes.io/


Airflow KubernetesPodOperator

https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html


Astronomer

https://www.astronomer.io/


Airflow Academy by Marc Lamberti

https://www.udemy.com/user/lockgfg/?utm_source=adwords&utm_medium=udemyads&utm_campaign=Search_DSA_GammaCatchall_NonP_la.EN_cc.ROW-English&campaigntype=Search&portfolio=ROW-English&language=EN&product=Course&test=&audience=DSA&topic=&priority=Gamma&utm_content=deal4584&utm_term=_._ag_169801645584_._ad_700876640602_._kw__._de_c_._dm__._pl__._ti_dsa-1456167871416_._li_9061346_._pd__._&matchtype=&gad_source=1&gad_campaignid=21341313808&gbraid=0AAAAADROdO1_-I2TMcVyU8F3i1jRXJ24K&gclid=Cj0KCQjwvJHIBhCgARIsAEQnWlC1uYHIRm3y9Q8rPNSuVPNivsxogqfczpKHwhmNho2uKZYC-y0taNQaApU2EALw_wcB


Airflow Documentation

https://airflow.apache.org/docs/


Airflow Plugins

https://airflow.apache.org/docs/apache-airflow/1.10.9/plugins.html




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow

]]> briana.okyere@astronomer.io (Astronomer) 776278b4-5f92-49e5-8bc5-d46cb7595caf Thu, 20 Nov 2025 00:50:13 -0500 1390 1 64 false Building Secure Financial Data Platforms at AgileEngine with Valentyn Druzhynin Building Secure Financial Data Platforms at AgileEngine with Valentyn Druzhynin The use of Apache Airflow in financial services demands a balance between innovation and compliance. Agile Engine’s approach to orchestration showcases how secure, auditable workflows can scale even within the constraints of regulatory environments.


In this episode, Valentyn Druzhynin, Senior Data Engineer at AgileEngine, discusses how his team leverages Airflow for ETF calculations, data validation and workflow reliability within tightly controlled release cycles.


Key Takeaways:


00:00 Introduction.

03:24 The orchestrator ensures secure and auditable workflows.

05:13 Validations before and after computation prevent errors.

08:24 Release freezes shape prioritization and delivery plans.

11:14 Migration plans must respect managed service constraints.

13:04 Versioning, backfills and event triggers increase reliability.

15:08 UI and integration improvements simplify operations.

18:05 New contributors should start small and seek help.


Resources Mentioned:


Valentyn Druzhynin

https://www.linkedin.com/in/valentyn-druzhynin/


AgileEngine | LinkedIn

https://www.linkedin.com/company/agileengine/


AgileEngine | Website

https://agileengine.com/


Apache Airflow

https://airflow.apache.org/


Astronomer

https://www.astronomer.io/


AWS Managed Airflow

https://aws.amazon.com/managed-workflows-for-apache-airflow/


Google Cloud Composer (Managed Airflow)

https://cloud.google.com/composer


Airflow Summit

https://airflowsummit.org/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> The use of Apache Airflow in financial services demands a balance between innovation and compliance. Agile Engine’s approach to orchestration showcases how secure, auditable workflows can scale even within the constraints of regulatory environments.


In this episode, Valentyn Druzhynin, Senior Data Engineer at AgileEngine, discusses how his team leverages Airflow for ETF calculations, data validation and workflow reliability within tightly controlled release cycles.


Key Takeaways:


00:00 Introduction.

03:24 The orchestrator ensures secure and auditable workflows.

05:13 Validations before and after computation prevent errors.

08:24 Release freezes shape prioritization and delivery plans.

11:14 Migration plans must respect managed service constraints.

13:04 Versioning, backfills and event triggers increase reliability.

15:08 UI and integration improvements simplify operations.

18:05 New contributors should start small and seek help.


Resources Mentioned:


Valentyn Druzhynin

https://www.linkedin.com/in/valentyn-druzhynin/


AgileEngine | LinkedIn

https://www.linkedin.com/company/agileengine/


AgileEngine | Website

https://agileengine.com/


Apache Airflow

https://airflow.apache.org/


Astronomer

https://www.astronomer.io/


AWS Managed Airflow

https://aws.amazon.com/managed-workflows-for-apache-airflow/


Google Cloud Composer (Managed Airflow)

https://cloud.google.com/composer


Airflow Summit

https://airflowsummit.org/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) d3ddc0ca-6c19-4a25-955d-b86a848ccc33 Thu, 13 Nov 2025 06:24:37 -0500 1276 1 63 false How Redica Transformed Their Data With Airflow and Snowflake with Shankar Mahindar How Redica Transformed Their Data With Airflow and Snowflake with Shankar Mahindar The life sciences industry relies on data accuracy, regulatory insight and quality intelligence. Building a unified system that keeps these elements aligned is no small feat.


In this episode, we welcome Shankar Mahindar, Senior Data Engineer II at Redica Systems. We discuss how the team restructures its data platform with Airflow to strengthen governance, reduce compliance risk and improve customer experience.


Key Takeaways:


00:00 Introduction.

01:53 A focused analytics platform reduces compliance risk in life sciences.

07:31 A centralized warehouse orchestrated by Airflow strengthens governance.

09:12 Managed orchestration keeps attention on analytics and outcomes.

10:32 A modern transformation stack enables scalable modeling and operations.

11:51 Event-driven pipelines improve data freshness and responsiveness.

14:13 Asset-oriented scheduling and versioning enhance reliability and change control.

16:53 Observability and SLAs build confidence in data quality and freshness.

21:04 Priorities include partitioned assets and streamlined developer tooling.


Resources Mentioned:


Shankar Mahindar

https://www.linkedin.com/in/shankar-mahindar-83a61b137/


Redica Systems | LinkedIn

https://www.linkedin.com/company/redicasystems/


Redica Systems | Website

https://redica.com


Apache Airflow

https://airflow.apache.org/


Astronomer

https://www.astronomer.io/


Snowflake

https://www.snowflake.com/


AWS

https://aws.amazon.com/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> The life sciences industry relies on data accuracy, regulatory insight and quality intelligence. Building a unified system that keeps these elements aligned is no small feat.


In this episode, we welcome Shankar Mahindar, Senior Data Engineer II at Redica Systems. We discuss how the team restructures its data platform with Airflow to strengthen governance, reduce compliance risk and improve customer experience.


Key Takeaways:


00:00 Introduction.

01:53 A focused analytics platform reduces compliance risk in life sciences.

07:31 A centralized warehouse orchestrated by Airflow strengthens governance.

09:12 Managed orchestration keeps attention on analytics and outcomes.

10:32 A modern transformation stack enables scalable modeling and operations.

11:51 Event-driven pipelines improve data freshness and responsiveness.

14:13 Asset-oriented scheduling and versioning enhance reliability and change control.

16:53 Observability and SLAs build confidence in data quality and freshness.

21:04 Priorities include partitioned assets and streamlined developer tooling.


Resources Mentioned:


Shankar Mahindar

https://www.linkedin.com/in/shankar-mahindar-83a61b137/


Redica Systems | LinkedIn

https://www.linkedin.com/company/redicasystems/


Redica Systems | Website

https://redica.com


Apache Airflow

https://airflow.apache.org/


Astronomer

https://www.astronomer.io/


Snowflake

https://www.snowflake.com/


AWS

https://aws.amazon.com/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 335bfd00-bf0c-41b9-ab69-2178b2ca088f Thu, 06 Nov 2025 04:23:27 -0500 1428 1 62 false How Airflow and AI Power Investigative Journalism at the Financial Times with Zdravko Hvarlingov How Airflow and AI Power Investigative Journalism at the Financial Times with Zdravko Hvarlingov The Financial Times leverages Airflow and AI to uncover powerful stories hidden within vast, unstructured data.


In this episode, Zdravko Hvarlingov, Senior Software Engineer at the Financial Times, discusses building multi-tenant Airflow systems and AI-driven pipelines that surface stories that might otherwise be missed. Zdravko walks through entity extraction and fuzzy matching, linking the UK Register of Members’ Financial Interests with Companies House, and how this work cuts weeks of manual analysis to minutes.


Key Takeaways:


00:00 Introduction.

02:12 What computational journalism means for day-to-day newsroom work.

05:22 Why a shared orchestration platform supports consistent, scalable workflows.

08:30 Tradeoffs of one centralized platform versus many separate instances.

11:52 Using pipelines to structure messy sources for faster analysis.

14:14 Turning recurring disclosures into usable data for investigations.

16:03 Applying lightweight ML and matching to reveal entities and links.

18:46 How automation reduces manual effort and shortens time to insight.

20:41 Practical improvements that make backfilling and reliability easier.


Resources Mentioned:


Zdravko Hvarlingov

https://www.linkedin.com/in/zdravko-hvarlingov-3aa36016b/


Financial Times | LinkedIn

https://www.linkedin.com/company/financial-times/


Financial Times | Website

https://www.ft.com/


Apache Airflow

https://airflow.apache.org/


UK Register of Members’ Financial Interests

https://www.parliament.uk/mps-lords-and-offices/standards-and-financial-interests/parliamentary-commissioner-for-standards/registers-of-interests/register-of-members-financial-interests/


UK Companies House

https://www.gov.uk/government/organisations/companies-house


Doppler

https://www.doppler.com/


Kubernetes

https://kubernetes.io/


Airflow Kubernetes Executor

https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html


GitHub

https://github.com/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> The Financial Times leverages Airflow and AI to uncover powerful stories hidden within vast, unstructured data.


In this episode, Zdravko Hvarlingov, Senior Software Engineer at the Financial Times, discusses building multi-tenant Airflow systems and AI-driven pipelines that surface stories that might otherwise be missed. Zdravko walks through entity extraction and fuzzy matching, linking the UK Register of Members’ Financial Interests with Companies House, and how this work cuts weeks of manual analysis to minutes.


Key Takeaways:


00:00 Introduction.

02:12 What computational journalism means for day-to-day newsroom work.

05:22 Why a shared orchestration platform supports consistent, scalable workflows.

08:30 Tradeoffs of one centralized platform versus many separate instances.

11:52 Using pipelines to structure messy sources for faster analysis.

14:14 Turning recurring disclosures into usable data for investigations.

16:03 Applying lightweight ML and matching to reveal entities and links.

18:46 How automation reduces manual effort and shortens time to insight.

20:41 Practical improvements that make backfilling and reliability easier.


Resources Mentioned:


Zdravko Hvarlingov

https://www.linkedin.com/in/zdravko-hvarlingov-3aa36016b/


Financial Times | LinkedIn

https://www.linkedin.com/company/financial-times/


Financial Times | Website

https://www.ft.com/


Apache Airflow

https://airflow.apache.org/


UK Register of Members’ Financial Interests

https://www.parliament.uk/mps-lords-and-offices/standards-and-financial-interests/parliamentary-commissioner-for-standards/registers-of-interests/register-of-members-financial-interests/


UK Companies House

https://www.gov.uk/government/organisations/companies-house


Doppler

https://www.doppler.com/


Kubernetes

https://kubernetes.io/


Airflow Kubernetes Executor

https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html


GitHub

https://github.com/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 6015cd4b-53b7-49d3-98f8-28906da9ea59 Thu, 30 Oct 2025 05:48:07 -0400 1468 1 61 false Inside Vinted’s Code-Generated Airflow Pipelines with Oscar Ligthart and Rodrigo Loredo Inside Vinted’s Code-Generated Airflow Pipelines with Oscar Ligthart and Rodrigo Loredo The shift from monolithic to decentralized data workflows changes how teams build, connect and scale pipelines.


In this episode, we feature Oscar Ligthart, Lead Data Engineer, and Rodrigo Loredo, Lead Analytics Engineer, both at Vinted, as we unpack their YAML-driven abstraction that generates Airflow DAGs and standardizes cross-team orchestration.


Key Takeaways:


00:00 Introduction.

05:28 Challenges of decentralization.

06:45 YAML-based generator standardizes pipelines and dependencies.

12:28 Declarative assets and sensors align cross-DAG dependencies.

17:29 Task-level callbacks enable auto-recovery and clear ownership.

21:39 Standardized building blocks simplify upgrades and maintenance.

24:52 Platform focus frees domain work.

26:49 Container-only standardization prevents sprawl.


Resources Mentioned:


Oscar Ligthart

https://www.linkedin.com/in/oscar-ligthart/


Rodrigo Loredo

https://www.linkedin.com/in/rodrigo-loredo-410a16134/


Vinted | LinkedIn

https://www.linkedin.com/company/vinted/


Vinted | Website

https://www.vinted.com/?srsltid=AfmBOor87MGR_eLOauCO93V9A-aLDaAhGYx9cnu_oN8s1SAXMlCRuhW7


Apache Airflow

https://airflow.apache.org/


Kubernetes

https://kubernetes.io/


dbt

https://www.getdbt.com/


Google Cloud Vertex AI

https://cloud.google.com/vertex-ai


Airflow Datasets & Assets (concepts)

https://www.astronomer.io/docs/learn/airflow-datasets


Airflow Summit

https://airflowsummit.org/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> The shift from monolithic to decentralized data workflows changes how teams build, connect and scale pipelines.


In this episode, we feature Oscar Ligthart, Lead Data Engineer, and Rodrigo Loredo, Lead Analytics Engineer, both at Vinted, as we unpack their YAML-driven abstraction that generates Airflow DAGs and standardizes cross-team orchestration.


Key Takeaways:


00:00 Introduction.

05:28 Challenges of decentralization.

06:45 YAML-based generator standardizes pipelines and dependencies.

12:28 Declarative assets and sensors align cross-DAG dependencies.

17:29 Task-level callbacks enable auto-recovery and clear ownership.

21:39 Standardized building blocks simplify upgrades and maintenance.

24:52 Platform focus frees domain work.

26:49 Container-only standardization prevents sprawl.


Resources Mentioned:


Oscar Ligthart

https://www.linkedin.com/in/oscar-ligthart/


Rodrigo Loredo

https://www.linkedin.com/in/rodrigo-loredo-410a16134/


Vinted | LinkedIn

https://www.linkedin.com/company/vinted/


Vinted | Website

https://www.vinted.com/?srsltid=AfmBOor87MGR_eLOauCO93V9A-aLDaAhGYx9cnu_oN8s1SAXMlCRuhW7


Apache Airflow

https://airflow.apache.org/


Kubernetes

https://kubernetes.io/


dbt

https://www.getdbt.com/


Google Cloud Vertex AI

https://cloud.google.com/vertex-ai


Airflow Datasets & Assets (concepts)

https://www.astronomer.io/docs/learn/airflow-datasets


Airflow Summit

https://airflowsummit.org/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) f22b61aa-674e-4767-b810-d5fd5a84c86c Thu, 23 Oct 2025 01:25:27 -0400 1776 1 60 false Transforming Data Pipelines at XENA Intelligence with Naseem Shah Transforming Data Pipelines at XENA Intelligence with Naseem Shah The shift from simple cron jobs to orchestrated AI-powered workflows is reshaping how startups scale. For a small team, these transitions come with unique challenges and big opportunities.


In this episode, Naseem Shah, Head of Engineering at Xena Intelligence, shares how he built data pipelines from scratch, adopted Apache Airflow and transformed Amazon review analysis with LLMs.


Key Takeaways:


00:00 Introduction.

03:28 The importance of building initial products that support growth and investment.

06:16 The process of adopting new tools to improve reliability and efficiency.

09:29 Approaches to learning complex technologies through practice and fundamentals.

13:57 Trade-offs small teams face when balancing performance and costs.

18:40 Using AI-driven approaches to generate insights from large datasets.

22:38 How unstructured data can be transformed into actionable information.

25:55 Moving from manual tasks to fully automated workflows.

28:05 Orchestration as a foundation for scaling advanced use cases.


Resources Mentioned:


Naseem Shah

https://www.linkedin.com/in/naseemshah/


Xena Intelligence | LinkedIn

https://www.linkedin.com/company/xena-intelligence/


Xena Intelligence | Website

https://xenaintelligence.com/


Apache Airflow

https://airflow.apache.org/


Google Cloud Composer

https://cloud.google.com/composer


Techstars

https://www.techstars.com/


Docker

https://www.docker.com/


AWS SQS

https://aws.amazon.com/sqs/


PostgreSQL

https://www.postgresql.org/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> The shift from simple cron jobs to orchestrated AI-powered workflows is reshaping how startups scale. For a small team, these transitions come with unique challenges and big opportunities.


In this episode, Naseem Shah, Head of Engineering at Xena Intelligence, shares how he built data pipelines from scratch, adopted Apache Airflow and transformed Amazon review analysis with LLMs.


Key Takeaways:


00:00 Introduction.

03:28 The importance of building initial products that support growth and investment.

06:16 The process of adopting new tools to improve reliability and efficiency.

09:29 Approaches to learning complex technologies through practice and fundamentals.

13:57 Trade-offs small teams face when balancing performance and costs.

18:40 Using AI-driven approaches to generate insights from large datasets.

22:38 How unstructured data can be transformed into actionable information.

25:55 Moving from manual tasks to fully automated workflows.

28:05 Orchestration as a foundation for scaling advanced use cases.


Resources Mentioned:


Naseem Shah

https://www.linkedin.com/in/naseemshah/


Xena Intelligence | LinkedIn

https://www.linkedin.com/company/xena-intelligence/


Xena Intelligence | Website

https://xenaintelligence.com/


Apache Airflow

https://airflow.apache.org/


Google Cloud Composer

https://cloud.google.com/composer


Techstars

https://www.techstars.com/


Docker

https://www.docker.com/


AWS SQS

https://aws.amazon.com/sqs/


PostgreSQL

https://www.postgresql.org/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 84d317ee-d936-4e33-bcf0-8cf29c6810af Thu, 16 Oct 2025 05:36:36 -0400 1712 1 59 false Scaling Geospatial Workflows With Airflow at Overture Maps Foundation and Wherobots with Alex Iannicelli and Daniel Smith Scaling Geospatial Workflows With Airflow at Overture Maps Foundation and Wherobots with Alex Iannicelli and Daniel Smith Using Airflow to orchestrate geospatial data pipelines unlocks powerful efficiencies for data teams. The combination of scalable processing and visual observability streamlines workflows, reduces costs and improves iteration speed.


In this episode, Alex Iannicelli, Staff Software Engineer at Overture Maps Foundation, and Daniel Smith, Senior Solutions Architect at Wherobots, join us to discuss leveraging Apache Airflow and Apache Sedona to process massive geospatial datasets, build reproducible pipelines and orchestrate complex workflows across platforms.


Key Takeaways:


00:00 Introduction.

03:22 How merging multiple data sources supports comprehensive datasets.

04:20 The value of flexible configurations for running pipelines on different platforms.

06:35 Why orchestration tools are essential for handling continuous data streams.

09:45 The importance of observability for monitoring progress and troubleshooting issues.

11:30 Strategies for processing large, complex datasets efficiently.

13:27 Expanding orchestration beyond core pipelines to automate frequent tasks.

17:02 Advantages of using open-source operators to simplify integration and deployment.

20:32 Desired improvements in orchestration tools for usability and workflow management.


Resources Mentioned:


Alex Iannicelli

https://www.linkedin.com/in/atiannicelli/


Overture Maps Foundation | LinkedIn

https://www.linkedin.com/company/overture-maps-foundation/


Overture Maps Foundation | Website

https://overturemaps.org


Daniel Smith

https://www.linkedin.com/in/daniel-smith-analyst/


Wherobots | LinkedIn

https://www.linkedin.com/company/wherobots


Wherobots | Website

https://www.wherobots.com


Apache Airflow

https://airflow.apache.org/


Apache Sedona

https://sedona.apache.org/


Github repo

https://github.com/wherobots/airflow-providers-wherobots




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Using Airflow to orchestrate geospatial data pipelines unlocks powerful efficiencies for data teams. The combination of scalable processing and visual observability streamlines workflows, reduces costs and improves iteration speed.


In this episode, Alex Iannicelli, Staff Software Engineer at Overture Maps Foundation, and Daniel Smith, Senior Solutions Architect at Wherobots, join us to discuss leveraging Apache Airflow and Apache Sedona to process massive geospatial datasets, build reproducible pipelines and orchestrate complex workflows across platforms.


Key Takeaways:


00:00 Introduction.

03:22 How merging multiple data sources supports comprehensive datasets.

04:20 The value of flexible configurations for running pipelines on different platforms.

06:35 Why orchestration tools are essential for handling continuous data streams.

09:45 The importance of observability for monitoring progress and troubleshooting issues.

11:30 Strategies for processing large, complex datasets efficiently.

13:27 Expanding orchestration beyond core pipelines to automate frequent tasks.

17:02 Advantages of using open-source operators to simplify integration and deployment.

20:32 Desired improvements in orchestration tools for usability and workflow management.


Resources Mentioned:


Alex Iannicelli

https://www.linkedin.com/in/atiannicelli/


Overture Maps Foundation | LinkedIn

https://www.linkedin.com/company/overture-maps-foundation/


Overture Maps Foundation | Website

https://overturemaps.org


Daniel Smith

https://www.linkedin.com/in/daniel-smith-analyst/


Wherobots | LinkedIn

https://www.linkedin.com/company/wherobots


Wherobots | Website

https://www.wherobots.com


Apache Airflow

https://airflow.apache.org/


Apache Sedona

https://sedona.apache.org/


Github repo

https://github.com/wherobots/airflow-providers-wherobots




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 1c54c88a-ca83-4431-b6a4-6bda51130ea6 Thu, 09 Oct 2025 00:00:00 -0400 1443 1 58 false Scaling Airflow for Enterprise Data Platforms at PepsiCo with Kunal Bhattacharya Scaling Airflow for Enterprise Data Platforms at PepsiCo with Kunal Bhattacharya PepsiCo’s data platform drives insights across finance, marketing and data science. Delivering stability, scalability and developer delight is central to its success, and engineering leadership plays a key role in making this possible.


In this episode, Kunal Bhattacharya, Senior Manager of Data Platform Engineering at PepsiCo, shares how his team manages Airflow at scale while ensuring security, performance and cost efficiency.


Key Takeaways:


00:00 Introduction.

02:31 Enabling developer delight by extending platform capabilities.

03:56 Role of Snowflake, dbt and Airflow in PepsiCo’s data stack.

06:10 Local developer environments built using official Airflow Helm charts.

07:13 Pre-staging and PR environments as testing playgrounds.

08:08 Automating labeling and resource allocation via DAG factories.

12:16 Cost optimization through pod labeling and Datadog insights.

14:01 Isolating dbt engines to improve performance across teams.

16:12 Wishlist for Airflow 3: Improved role-based grants and database modeling.


Resources Mentioned:


Kunal Bhattacharya

https://www.linkedin.com/in/kunaljubce/


PepsiCo | LinkedIn

https://www.linkedin.com/company/pepsico/


PepsiCo | Website

https://www.pepsico.com


Apache Airflow

https://airflow.apache.org/


Snowflake

https://www.snowflake.com


dbt

https://www.getdbt.com


Kubernetes

https://kubernetes.io


Great Expectations

https://greatexpectations.io


Monte Carlo

https://www.montecarlodata.com




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> PepsiCo’s data platform drives insights across finance, marketing and data science. Delivering stability, scalability and developer delight is central to its success, and engineering leadership plays a key role in making this possible.


In this episode, Kunal Bhattacharya, Senior Manager of Data Platform Engineering at PepsiCo, shares how his team manages Airflow at scale while ensuring security, performance and cost efficiency.


Key Takeaways:


00:00 Introduction.

02:31 Enabling developer delight by extending platform capabilities.

03:56 Role of Snowflake, dbt and Airflow in PepsiCo’s data stack.

06:10 Local developer environments built using official Airflow Helm charts.

07:13 Pre-staging and PR environments as testing playgrounds.

08:08 Automating labeling and resource allocation via DAG factories.

12:16 Cost optimization through pod labeling and Datadog insights.

14:01 Isolating dbt engines to improve performance across teams.

16:12 Wishlist for Airflow 3: Improved role-based grants and database modeling.


Resources Mentioned:


Kunal Bhattacharya

https://www.linkedin.com/in/kunaljubce/


PepsiCo | LinkedIn

https://www.linkedin.com/company/pepsico/


PepsiCo | Website

https://www.pepsico.com


Apache Airflow

https://airflow.apache.org/


Snowflake

https://www.snowflake.com


dbt

https://www.getdbt.com


Kubernetes

https://kubernetes.io


Great Expectations

https://greatexpectations.io


Monte Carlo

https://www.montecarlodata.com




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 05c8a950-867a-4eaf-9301-e74cc65f00fc Thu, 02 Oct 2025 00:00:00 -0400 1144 1 57 false Building a Unified Data Platform at Pattern with William Graham Building a Unified Data Platform at Pattern with William Graham The orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data teams manage large-scale pipelines.


In this episode, we are joined by William Graham, Senior Data Engineer at Pattern, who explains how his team leverages Apache Airflow alongside their open-source tool Heimdall to streamline scheduling, orchestration and access management.


Key Takeaways:


00:00 Introduction.

02:44 Structure of Pattern’s data teams across acquisition, engineering and platform.

04:27 How Airflow became the central scheduler for batch jobs.

08:57 Credential management challenges that led to decoupling scheduling and orchestration.

12:21 Heimdall simplifies multi-application access through a unified interface.

13:15 Standardized operators in Airflow using Heimdall integration.

17:13 Open-source contributions and early adoption of Heimdall within Pattern.

21:01 Community support for Airflow and satisfaction with scheduling flexibility.


Resources Mentioned:


William Graham

https://www.linkedin.com/in/willgraham2/


Pattern | LinkedIn

https://www.linkedin.com/company/pattern-hq/


Pattern | Website

https://pattern.com


Apache Airflow

https://airflow.apache.org


Heimdall on GitHub

https://github.com/patterninc/heimdall


Netflix Genie

https://netflix.github.io/genie/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> The orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data teams manage large-scale pipelines.


In this episode, we are joined by William Graham, Senior Data Engineer at Pattern, who explains how his team leverages Apache Airflow alongside their open-source tool Heimdall to streamline scheduling, orchestration and access management.


Key Takeaways:


00:00 Introduction.

02:44 Structure of Pattern’s data teams across acquisition, engineering and platform.

04:27 How Airflow became the central scheduler for batch jobs.

08:57 Credential management challenges that led to decoupling scheduling and orchestration.

12:21 Heimdall simplifies multi-application access through a unified interface.

13:15 Standardized operators in Airflow using Heimdall integration.

17:13 Open-source contributions and early adoption of Heimdall within Pattern.

21:01 Community support for Airflow and satisfaction with scheduling flexibility.


Resources Mentioned:


William Graham

https://www.linkedin.com/in/willgraham2/


Pattern | LinkedIn

https://www.linkedin.com/company/pattern-hq/


Pattern | Website

https://pattern.com


Apache Airflow

https://airflow.apache.org


Heimdall on GitHub

https://github.com/patterninc/heimdall


Netflix Genie

https://netflix.github.io/genie/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) ceaaa1c3-3790-4e24-b9da-fe56a3a119ad Thu, 25 Sep 2025 00:00:00 -0400 1449 1 56 false How Astronomer Turns Proactive Monitoring Into Customer Success with Collin McNulty How Astronomer Turns Proactive Monitoring Into Customer Success with Collin McNulty The evolution of Airflow continues to shape data orchestration and monitoring strategies. Leveraging it beyond traditional ETL use cases opens powerful new possibilities for proactive support and internal operations.


In this episode, we are joined by Collin McNulty, Sr. Director of Global Support at Astronomer, who shares insights from his journey into data engineering and the lessons learned from leading Astronomer’s Customer Reliability Engineering (CRE) team.


Key Takeaways:


00:00 Introduction.

03:07 Lessons learned in adapting to major platform transitions.

05:18 How proactive monitoring improves reliability and customer experience.

08:10 Using automation to enhance internal support processes.

12:09 Why keeping systems current helps avoid unnecessary issues.

15:14 Approaches that strengthen system reliability and efficiency.

18:46 Best practices for simplifying complex orchestration dependencies.

23:24 Anticipated innovations that expand orchestration capabilities.


Resources Mentioned:


Collin McNulty

https://www.linkedin.com/in/collin-mcnulty/


Astronomer | LinkedIn

https://www.linkedin.com/company/astronomer/


Astronomer | Website

https://www.astronomer.io


Apache Airflow

https://airflow.apache.org/


Prometheus

https://prometheus.io/


Splunk

https://www.splunk.com/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> The evolution of Airflow continues to shape data orchestration and monitoring strategies. Leveraging it beyond traditional ETL use cases opens powerful new possibilities for proactive support and internal operations.


In this episode, we are joined by Collin McNulty, Sr. Director of Global Support at Astronomer, who shares insights from his journey into data engineering and the lessons learned from leading Astronomer’s Customer Reliability Engineering (CRE) team.


Key Takeaways:


00:00 Introduction.

03:07 Lessons learned in adapting to major platform transitions.

05:18 How proactive monitoring improves reliability and customer experience.

08:10 Using automation to enhance internal support processes.

12:09 Why keeping systems current helps avoid unnecessary issues.

15:14 Approaches that strengthen system reliability and efficiency.

18:46 Best practices for simplifying complex orchestration dependencies.

23:24 Anticipated innovations that expand orchestration capabilities.


Resources Mentioned:


Collin McNulty

https://www.linkedin.com/in/collin-mcnulty/


Astronomer | LinkedIn

https://www.linkedin.com/company/astronomer/


Astronomer | Website

https://www.astronomer.io


Apache Airflow

https://airflow.apache.org/


Prometheus

https://prometheus.io/


Splunk

https://www.splunk.com/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) b509a1de-383c-4b68-b2e0-c0753df241f0 Thu, 18 Sep 2025 00:00:00 -0400 1534 1 55 false Overcoming Data Engineering Challenges at Daiichi Sankyo Europe GmbH with Evgenii Prusov Overcoming Data Engineering Challenges at Daiichi Sankyo Europe GmbH with Evgenii Prusov The shift to a unified data platform is reshaping how pharmaceutical companies manage and orchestrate data. Establishing standards across regions and teams ensures scalability and efficiency in handling large-scale analytics.


In this episode, Evgenii Prusov, Senior Data Platform Engineer of Daiichi Sankyo Europe GmbH, joins us to discuss building and scaling a centralized data platform with Airflow and Astronomer.


Key Takeaways:


00:00 Introduction.

02:49 Building a centralized data platform for 15 European countries.

05:19 Adopting SaaS to manage Airflow from day one.

07:01 Leveraging Airflow for data orchestration across products.

08:16 Teaching non-Python users how to work with Airflow is challenging.

12:25 Creating a global data community across Europe, the US and Japan.

14:04 Monthly calls help share knowledge and align regional teams.

15:47 Contributing to the open-source Airflow project as a way to deepen expertise.

16:32 Desire for more guidelines, debugging tutorials and testing best practices in Airflow.


Resources Mentioned: 


Evgenii Prusov

https://www.linkedin.com/in/prusov/


Daiichi Sankyo Europe GmbH | LinkedIn

https://www.linkedin.com/company/daiichi-sankyo-europe-gmbh/


Daiichi Sankyo Europe GmbH | Website

https://www.daiichi-sankyo.eu


Apache Airflow

https://airflow.apache.org/


Astronomer

https://www.astronomer.io/


Snowflake

https://www.snowflake.com/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> The shift to a unified data platform is reshaping how pharmaceutical companies manage and orchestrate data. Establishing standards across regions and teams ensures scalability and efficiency in handling large-scale analytics.


In this episode, Evgenii Prusov, Senior Data Platform Engineer of Daiichi Sankyo Europe GmbH, joins us to discuss building and scaling a centralized data platform with Airflow and Astronomer.


Key Takeaways:


00:00 Introduction.

02:49 Building a centralized data platform for 15 European countries.

05:19 Adopting SaaS to manage Airflow from day one.

07:01 Leveraging Airflow for data orchestration across products.

08:16 Teaching non-Python users how to work with Airflow is challenging.

12:25 Creating a global data community across Europe, the US and Japan.

14:04 Monthly calls help share knowledge and align regional teams.

15:47 Contributing to the open-source Airflow project as a way to deepen expertise.

16:32 Desire for more guidelines, debugging tutorials and testing best practices in Airflow.


Resources Mentioned: 


Evgenii Prusov

https://www.linkedin.com/in/prusov/


Daiichi Sankyo Europe GmbH | LinkedIn

https://www.linkedin.com/company/daiichi-sankyo-europe-gmbh/


Daiichi Sankyo Europe GmbH | Website

https://www.daiichi-sankyo.eu


Apache Airflow

https://airflow.apache.org/


Astronomer

https://www.astronomer.io/


Snowflake

https://www.snowflake.com/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 0eb6542e-4481-4bad-bf4a-90005cfe9cac Thu, 11 Sep 2025 06:58:49 -0400 1166 1 54 false Building a Data-Driven Beauty and Wellness Marketplace at StyleSeat with Paschal Onuorah Building a Data-Driven Beauty and Wellness Marketplace at StyleSeat with Paschal Onuorah StyleSeat is revolutionizing how beauty and wellness professionals grow their businesses through data-driven tools. From streamlining scheduling to optimizing marketing, their platform empowers professionals to focus on their craft while expanding their client base.


In this episode, Paschal Onuorah, Senior Data Engineer at StyleSeat, shares how the company leverages Airflow, dbt, and Cosmos to drive marketplace intelligence, improve client connections and deliver measurable growth for professionals.


Key Takeaways:


00:00 Introduction.

05:44 The role of the data engineering team in driving business success.

08:52 Leveraging technology for real-time business intelligence.

10:52 Data-driven strategies for improving marketing outcomes.

13:05 How adopting the right tools can increase revenue growth.

14:25 Advantages of simplifying and integrating technical workflows.

18:45 Benefits of multi-environment configurations for development and production.

20:17 Foundational skills and best practices for learning Airflow effectively.

22:33 Opportunities for deeper tool integration and improved data visualization.


Resources Mentioned:


Paschal Onuorah

https://www.linkedin.com/in/onuorah-paschal/


StyleSeat | LinkedIn

https://www.linkedin.com/company/styleseat/


StyleSeat | Website

https://www.styleseat.com


Apache Airflow

https://airflow.apache.org/


dbt

https://www.getdbt.com/


Astronomer Cosmos

https://www.astronomer.io/cosmos/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> StyleSeat is revolutionizing how beauty and wellness professionals grow their businesses through data-driven tools. From streamlining scheduling to optimizing marketing, their platform empowers professionals to focus on their craft while expanding their client base.


In this episode, Paschal Onuorah, Senior Data Engineer at StyleSeat, shares how the company leverages Airflow, dbt, and Cosmos to drive marketplace intelligence, improve client connections and deliver measurable growth for professionals.


Key Takeaways:


00:00 Introduction.

05:44 The role of the data engineering team in driving business success.

08:52 Leveraging technology for real-time business intelligence.

10:52 Data-driven strategies for improving marketing outcomes.

13:05 How adopting the right tools can increase revenue growth.

14:25 Advantages of simplifying and integrating technical workflows.

18:45 Benefits of multi-environment configurations for development and production.

20:17 Foundational skills and best practices for learning Airflow effectively.

22:33 Opportunities for deeper tool integration and improved data visualization.


Resources Mentioned:


Paschal Onuorah

https://www.linkedin.com/in/onuorah-paschal/


StyleSeat | LinkedIn

https://www.linkedin.com/company/styleseat/


StyleSeat | Website

https://www.styleseat.com


Apache Airflow

https://airflow.apache.org/


dbt

https://www.getdbt.com/


Astronomer Cosmos

https://www.astronomer.io/cosmos/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 9c27584d-e162-4cca-9145-55a4fc628363 Thu, 04 Sep 2025 05:54:20 -0400 1385 1 53 false Building the Future of Airflow Execution at Astronomer with Ian Buss and Piotr Chomiak Building the Future of Airflow Execution at Astronomer with Ian Buss and Piotr Chomiak The evolution of orchestration in Airflow continues with innovations that address both scalability and security. From improving executor reliability to enabling remote execution, these advancements reshape how organizations manage data pipelines.


In this episode, we’re joined by Ian Buss, Principal Software Engineer at Astronomer, and Piotr Chomiak, Principal Product Manager at Astronomer, who share insights into the Astro Executor and remote execution.


Key Takeaways:


00:00 Introduction.

04:13 How product leadership drives scalability for enterprise needs.

08:23 Architectural changes that improve reliability and remove bottlenecks.

10:15 Metrics that enhance visibility into system performance.

12:54 The role of remote execution in addressing security requirements.

15:56 Differences between open-source solutions and managed offerings.

19:04 Broad industry adoption and applicability of remote execution.

20:39 Future advancements in language support and multi-tenancy.


Resources Mentioned:


Ian Buss

https://www.linkedin.com/in/ian-buss/


Piotr Chomiak

https://www.linkedin.com/in/piotr-chomiak-b1955624/


Astronomer | Website

https://www.astronomer.io


Apache Airflow

https://airflow.apache.org/


Airflow Slack Community

https://airflow.apache.org/community/


Beyond Analytics conference

https://astronomer.io/beyond/dataflowcast




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> The evolution of orchestration in Airflow continues with innovations that address both scalability and security. From improving executor reliability to enabling remote execution, these advancements reshape how organizations manage data pipelines.


In this episode, we’re joined by Ian Buss, Principal Software Engineer at Astronomer, and Piotr Chomiak, Principal Product Manager at Astronomer, who share insights into the Astro Executor and remote execution.


Key Takeaways:


00:00 Introduction.

04:13 How product leadership drives scalability for enterprise needs.

08:23 Architectural changes that improve reliability and remove bottlenecks.

10:15 Metrics that enhance visibility into system performance.

12:54 The role of remote execution in addressing security requirements.

15:56 Differences between open-source solutions and managed offerings.

19:04 Broad industry adoption and applicability of remote execution.

20:39 Future advancements in language support and multi-tenancy.


Resources Mentioned:


Ian Buss

https://www.linkedin.com/in/ian-buss/


Piotr Chomiak

https://www.linkedin.com/in/piotr-chomiak-b1955624/


Astronomer | Website

https://www.astronomer.io


Apache Airflow

https://airflow.apache.org/


Airflow Slack Community

https://airflow.apache.org/community/


Beyond Analytics conference

https://astronomer.io/beyond/dataflowcast




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) db4b7301-bab7-4734-b4ec-fa1aee1f8da5 Thu, 28 Aug 2025 00:00:00 -0400 1345 1 54 false Scaling On-Prem Airflow With 2,000 DAGs at Numberly with Sébastien Crocquevieille Scaling On-Prem Airflow With 2,000 DAGs at Numberly with Sébastien Crocquevieille Scaling 2,000+ data pipelines isn’t easy. But with the right tools and a self-hosted mindset, it becomes achievable.


In this episode, Sébastien Crocquevieille, Data Engineer at Numberly, unpacks how the team scaled their on-prem Airflow setup using open-source tooling and Kubernetes. We explore orchestration strategies, UI-driven stakeholder access and Airflow’s evolving features.


Key Takeaways:


00:00 Introduction.

02:13 Overview of the company’s operations and global presence.

04:00 The tech stack and structure of the data engineering team.

04:24 Running nearly 2,000 DAGs in production using Airflow.

05:42 How Airflow’s UI empowers stakeholders to self-serve and troubleshoot.

07:05 Details on the Kubernetes-based Airflow setup using Helm charts.

09:31 Transition from GitSync to NFS for DAG syncing due to performance issues.

14:11 Making every team member Airflow-literate through local installation.

17:56 Using custom libraries and plugins to extend Airflow functionality.


Resources Mentioned:


Sébastien Crocquevieille

https://www.linkedin.com/in/scroc/


Numberly | LinkedIn

https://www.linkedin.com/company/numberly/


Numberly | Website

https://numberly.com/


Apache Airflow

https://airflow.apache.org/


Grafana

https://grafana.com/


Apache Kafka

https://kafka.apache.org/


Helm Chart for Apache Airflow

https://airflow.apache.org/docs/helm-chart/stable/index.html


Kubernetes

https://kubernetes.io/


GitLab

https://about.gitlab.com/


KubernetesPodOperator – Airflow

https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html


Beyond Analytics Conference

https://astronomer.io/beyond/dataflowcast




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Scaling 2,000+ data pipelines isn’t easy. But with the right tools and a self-hosted mindset, it becomes achievable.


In this episode, Sébastien Crocquevieille, Data Engineer at Numberly, unpacks how the team scaled their on-prem Airflow setup using open-source tooling and Kubernetes. We explore orchestration strategies, UI-driven stakeholder access and Airflow’s evolving features.


Key Takeaways:


00:00 Introduction.

02:13 Overview of the company’s operations and global presence.

04:00 The tech stack and structure of the data engineering team.

04:24 Running nearly 2,000 DAGs in production using Airflow.

05:42 How Airflow’s UI empowers stakeholders to self-serve and troubleshoot.

07:05 Details on the Kubernetes-based Airflow setup using Helm charts.

09:31 Transition from GitSync to NFS for DAG syncing due to performance issues.

14:11 Making every team member Airflow-literate through local installation.

17:56 Using custom libraries and plugins to extend Airflow functionality.


Resources Mentioned:


Sébastien Crocquevieille

https://www.linkedin.com/in/scroc/


Numberly | LinkedIn

https://www.linkedin.com/company/numberly/


Numberly | Website

https://numberly.com/


Apache Airflow

https://airflow.apache.org/


Grafana

https://grafana.com/


Apache Kafka

https://kafka.apache.org/


Helm Chart for Apache Airflow

https://airflow.apache.org/docs/helm-chart/stable/index.html


Kubernetes

https://kubernetes.io/


GitLab

https://about.gitlab.com/


KubernetesPodOperator – Airflow

https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html


Beyond Analytics Conference

https://astronomer.io/beyond/dataflowcast




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 96848ae0-def1-441b-9ee0-dccb68b2f6bf Thu, 21 Aug 2025 00:00:00 -0400 1457 1 53 false How Moniepoint Group Uses Airflow for Exposure Monitoring with Adeolu Adegboye How Moniepoint Group Uses Airflow for Exposure Monitoring with Adeolu Adegboye Managing financial data at scale requires precise orchestration and proactive monitoring to maintain operational efficiency.


In this episode, we are joined by Adeolu Adegboye, Data Engineer at Moniepoint Group, who shares how his team uses data pipelines and workflow automation to manage high volumes of transactions, ensure timely alerts and support diverse stakeholders across the business.


Key Takeaways:


(00:00) Introduction. 

(02:48) The role of data engineering in supporting all business operations.

(04:17) Leveraging workflow orchestration to manage daily processes.

(05:20) Proactively monitoring for anomalies to prevent potential issues.

(08:12) Simplifying complex insights for non-technical teams.

(13:01) Improving efficiency through dynamic and parallel workflows.

(14:19) Optimizing system performance to handle large-scale operations.

(17:19) Exploring creative and innovative uses for workflow automation.


Resources Mentioned:


Adeolu Adegboye

https://www.linkedin.com/in/adeolu-adegboye/


Moniepoint Group | LinkedIn

https://www.linkedin.com/company/moniepoint-inc/


Moniepoint Group | Website

https://www.moniepoint.com


Apache Airflow

https://airflow.apache.org/


ClickHouse

https://clickhouse.com/


Grafana

https://grafana.com/


Beyond Analytics Conference

https://astronomer.io/beyond/dataflowcast



Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Managing financial data at scale requires precise orchestration and proactive monitoring to maintain operational efficiency.


In this episode, we are joined by Adeolu Adegboye, Data Engineer at Moniepoint Group, who shares how his team uses data pipelines and workflow automation to manage high volumes of transactions, ensure timely alerts and support diverse stakeholders across the business.


Key Takeaways:


(00:00) Introduction. 

(02:48) The role of data engineering in supporting all business operations.

(04:17) Leveraging workflow orchestration to manage daily processes.

(05:20) Proactively monitoring for anomalies to prevent potential issues.

(08:12) Simplifying complex insights for non-technical teams.

(13:01) Improving efficiency through dynamic and parallel workflows.

(14:19) Optimizing system performance to handle large-scale operations.

(17:19) Exploring creative and innovative uses for workflow automation.


Resources Mentioned:


Adeolu Adegboye

https://www.linkedin.com/in/adeolu-adegboye/


Moniepoint Group | LinkedIn

https://www.linkedin.com/company/moniepoint-inc/


Moniepoint Group | Website

https://www.moniepoint.com


Apache Airflow

https://airflow.apache.org/


ClickHouse

https://clickhouse.com/


Grafana

https://grafana.com/


Beyond Analytics Conference

https://astronomer.io/beyond/dataflowcast



Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 0a3621da-e19b-4dc3-b362-a6c184ed90c3 Thu, 14 Aug 2025 04:50:00 -0400 1292 1 52 false Inside Bosch’s Airflow 3 Revolution: Remote Execution with Jens Scheffler Inside Bosch’s Airflow 3 Revolution: Remote Execution with Jens Scheffler The evolution of Airflow has reached a milestone with the introduction of remote execution in Airflow 3, enabling flexible orchestration across distributed environments.


In this episode, Jens Scheffler, Test Execution Cluster Technical Architect at Bosch, shares insights on how his team’s need for large-scale, cross-environment testing influenced the development of the Edge Executor and shaped this major release.


Key Takeaways:


(02:39) The role of remote execution in supporting large-scale testing needs.

(04:44) How community support contributed to the Edge Executor’s development.

(08:41) Navigating network and infrastructure limitations within secure environments.

(13:25) Transitioning from database-heavy processes to an API-driven model.

(14:16) How the new task SDK in Airflow 3 improves distributed task execution.

(16:54) What is required to set up and configure the Edge Executor.

(19:36) Managing multiple queues to optimize tasks across different environments.

(23:30) Examples of extreme distance use cases for edge execution.


Resources Mentioned:


Jens Scheffler

https://www.linkedin.com/in/jens-scheffler/


Bosch | LinkedIn

https://www.linkedin.com/company/bosch/


Bosch | Website

https://www.bosch.com/


Apache Airflow

https://airflow.apache.org/


Edge Executor (Edge3 Provider Package)

https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html


Astronomer’s Astro Executor

https://www.astronomer.io/docs/astro/astro-executor/


Beyond Analytics Conference

https://astronomer.io/beyond/dataflowcast





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> The evolution of Airflow has reached a milestone with the introduction of remote execution in Airflow 3, enabling flexible orchestration across distributed environments.


In this episode, Jens Scheffler, Test Execution Cluster Technical Architect at Bosch, shares insights on how his team’s need for large-scale, cross-environment testing influenced the development of the Edge Executor and shaped this major release.


Key Takeaways:


(02:39) The role of remote execution in supporting large-scale testing needs.

(04:44) How community support contributed to the Edge Executor’s development.

(08:41) Navigating network and infrastructure limitations within secure environments.

(13:25) Transitioning from database-heavy processes to an API-driven model.

(14:16) How the new task SDK in Airflow 3 improves distributed task execution.

(16:54) What is required to set up and configure the Edge Executor.

(19:36) Managing multiple queues to optimize tasks across different environments.

(23:30) Examples of extreme distance use cases for edge execution.


Resources Mentioned:


Jens Scheffler

https://www.linkedin.com/in/jens-scheffler/


Bosch | LinkedIn

https://www.linkedin.com/company/bosch/


Bosch | Website

https://www.bosch.com/


Apache Airflow

https://airflow.apache.org/


Edge Executor (Edge3 Provider Package)

https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html


Astronomer’s Astro Executor

https://www.astronomer.io/docs/astro/astro-executor/


Beyond Analytics Conference

https://astronomer.io/beyond/dataflowcast





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 4bdd3ba5-5975-4a3b-9ecc-e777078b11f4 Thu, 07 Aug 2025 00:00:00 -0400 1682 1 51 false Inside Modern Data Infrastructure at Massdriver with Cory O’Daniel and Jake Ferriero Inside Modern Data Infrastructure at Massdriver with Cory O’Daniel and Jake Ferriero Managing modern data platforms means navigating a web of complex infrastructure, competing team needs and evolving security standards. For data teams to truly thrive, infrastructure must become both accessible and compliant without sacrificing velocity or reliability.


In this episode, we’re joined by Cory O’Daniel, CEO and Co-Founder at Massdriver, and Jacob Ferriero, Senior Software Engineer at Astronomer, to unpack what it takes to make data platform engineering scalable, sustainable and secure. They share lessons from years of experience working with DevOps, ML teams and platform engineers and discuss how Airflow fits into the orchestration layer of today’s data stacks.


Key Takeaways:


(03:27) Making infrastructure accessible without deep ops knowledge.

(07:23) Distinct personas and responsibilities across data teams.

(09:53) Infrastructure hurdles specific to ML workloads.

(11:13) Compliance and governance shaping platform design.

(13:27) Tooling mismatches between teams cause friction.

(15:13) Airflow’s orchestration role within broader system architecture.

(22:10) Creating reusable infrastructure patterns for consistency.

(24:13) Enabling secure access without slowing down development.

(26:55) Opportunities to improve Airflow with event-driven and reliability tooling.


Resources Mentioned:


Cory O’Daniel

https://www.linkedin.com/in/coryodaniel/


Massdriver | LinkedIn

https://www.linkedin.com/company/massdriver/


Massdriver | Website

https://www.massdriver.cloud/


Jacob Ferriero

https://www.linkedin.com/in/jacob-ferriero/


Astronomer

https://www.linkedin.com/company/astronomer/


Apache Airflow

https://airflow.apache.org/


Prequel

https://www.prequel.co/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Managing modern data platforms means navigating a web of complex infrastructure, competing team needs and evolving security standards. For data teams to truly thrive, infrastructure must become both accessible and compliant without sacrificing velocity or reliability.


In this episode, we’re joined by Cory O’Daniel, CEO and Co-Founder at Massdriver, and Jacob Ferriero, Senior Software Engineer at Astronomer, to unpack what it takes to make data platform engineering scalable, sustainable and secure. They share lessons from years of experience working with DevOps, ML teams and platform engineers and discuss how Airflow fits into the orchestration layer of today’s data stacks.


Key Takeaways:


(03:27) Making infrastructure accessible without deep ops knowledge.

(07:23) Distinct personas and responsibilities across data teams.

(09:53) Infrastructure hurdles specific to ML workloads.

(11:13) Compliance and governance shaping platform design.

(13:27) Tooling mismatches between teams cause friction.

(15:13) Airflow’s orchestration role within broader system architecture.

(22:10) Creating reusable infrastructure patterns for consistency.

(24:13) Enabling secure access without slowing down development.

(26:55) Opportunities to improve Airflow with event-driven and reliability tooling.


Resources Mentioned:


Cory O’Daniel

https://www.linkedin.com/in/coryodaniel/


Massdriver | LinkedIn

https://www.linkedin.com/company/massdriver/


Massdriver | Website

https://www.massdriver.cloud/


Jacob Ferriero

https://www.linkedin.com/in/jacob-ferriero/


Astronomer

https://www.linkedin.com/company/astronomer/


Apache Airflow

https://airflow.apache.org/


Prequel

https://www.prequel.co/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) e74573a6-59e3-4529-a456-76b950228544 Thu, 31 Jul 2025 04:50:00 -0400 1884 1 50 false The Future of Airflow Telemetry with Bolke de Bruin The Future of Airflow Telemetry with Bolke de Bruin Telemetry has the potential to guide the future of Airflow, but only if it’s implemented transparently and with community trust. 


In this episode, we’re joined by Bolke de Bruin, Director at Metyis and a long-time Airflow PMC member. Bolke discusses how telemetry has been handled in the past, why it matters now and what it will take to get it right.


Key Takeaways:


(03:20) The role of foundations in establishing credibility and sustainability.

(04:52) Why data collection is critical to open-source project direction.

(07:24) Lessons learned from previous approaches to user data collection.

(10:23) The current state of telemetry in the project.

(10:53) Community trust as a prerequisite for technical implementation.

(12:54) The importance of managing sensitive data within trusted ecosystems.

(16:37) Ethical considerations in balancing participation and access.

(18:45) Forward-looking ideas for improving workflow design and usability.


Resources Mentioned:


Bolke de Bruin

https://www.linkedin.com/in/bolke/


Metyis | LinkedIn

https://www.linkedin.com/company/metyis/


Metyis | Website

http://www.metyis.com


Apache Airflow

https://airflow.apache.org/


Airflow Summit

https://airflowsummit.org/


Airflow Dev List

https://lists.apache.org/list.html?dev@airflow.apache.org


https://www.astronomer.io/events/roadshow/london/

   

https://www.astronomer.io/events/roadshow/new-york/ 

  

https://www.astronomer.io/events/roadshow/sydney/   


https://www.astronomer.io/events/roadshow/san-francisco/  

 

https://www.astronomer.io/events/roadshow/chicago/ 




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Telemetry has the potential to guide the future of Airflow, but only if it’s implemented transparently and with community trust. 


In this episode, we’re joined by Bolke de Bruin, Director at Metyis and a long-time Airflow PMC member. Bolke discusses how telemetry has been handled in the past, why it matters now and what it will take to get it right.


Key Takeaways:


(03:20) The role of foundations in establishing credibility and sustainability.

(04:52) Why data collection is critical to open-source project direction.

(07:24) Lessons learned from previous approaches to user data collection.

(10:23) The current state of telemetry in the project.

(10:53) Community trust as a prerequisite for technical implementation.

(12:54) The importance of managing sensitive data within trusted ecosystems.

(16:37) Ethical considerations in balancing participation and access.

(18:45) Forward-looking ideas for improving workflow design and usability.


Resources Mentioned:


Bolke de Bruin

https://www.linkedin.com/in/bolke/


Metyis | LinkedIn

https://www.linkedin.com/company/metyis/


Metyis | Website

http://www.metyis.com


Apache Airflow

https://airflow.apache.org/


Airflow Summit

https://airflowsummit.org/


Airflow Dev List

https://lists.apache.org/list.html?dev@airflow.apache.org


https://www.astronomer.io/events/roadshow/london/

   

https://www.astronomer.io/events/roadshow/new-york/ 

  

https://www.astronomer.io/events/roadshow/sydney/   


https://www.astronomer.io/events/roadshow/san-francisco/  

 

https://www.astronomer.io/events/roadshow/chicago/ 




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 6f0d9bb3-11de-4166-b7ef-d92a2a82ccdd Thu, 17 Jul 2025 07:40:00 -0400 1315 1 49 false Transforming the Airflow UI for Cloudera’s Users with Shubham Raj Transforming the Airflow UI for Cloudera’s Users with Shubham Raj Contributing to open-source projects can be daunting, but it can also unlock unexpected innovation. This episode showcases how one engineer’s journey with Apache Airflow led to impactful UI enhancements and infrastructure solutions at scale. Shubham Raj, Software Engineer II at Cloudera, shares how his team built a drag-and-drop DAG editor for non-coders, contributions which helped shape the Airflow 3.0 Ul and introduced features like external XCom control and bulk APls.



Key Takeaways:


(02:30) Day-to-day responsibilities building platforms that simplify orchestration.

(05:27) Factors that make onboarding into large open-source projects accessible.

(07:35) The value of improved user interfaces for task state visibility and control.

(09:49) Enabling faster debugging by exposing internal data through APIs.

(13:00) Balancing frontend design goals with backend functionality.

(14:19) Creating workflow editors that lower the barrier to entry.

(16:54) Supporting a variety of task types within a visual DAG builder.

(19:32) Common infrastructure challenges faced by orchestration users.

(20:37) Addressing dependency management across distributed environments.


Resources Mentioned:


Shubham Raj

https://www.linkedin.com/in/shubhamrajofficial/


Cloudera | LinkedIn

https://www.linkedin.com/company/cloudera/


Cloudera | Website

https://www.cloudera.com/


Apache Airflow

https://airflow.apache.org/


2023 Airflow Summit

https://airflowsummit.org/


https://www.astronomer.io/events/roadshow/london/  


https://www.astronomer.io/events/roadshow/new-york/  


https://www.astronomer.io/events/roadshow/sydney/  


https://www.astronomer.io/events/roadshow/san-francisco/  


https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Contributing to open-source projects can be daunting, but it can also unlock unexpected innovation. This episode showcases how one engineer’s journey with Apache Airflow led to impactful UI enhancements and infrastructure solutions at scale. Shubham Raj, Software Engineer II at Cloudera, shares how his team built a drag-and-drop DAG editor for non-coders, contributions which helped shape the Airflow 3.0 Ul and introduced features like external XCom control and bulk APls.



Key Takeaways:


(02:30) Day-to-day responsibilities building platforms that simplify orchestration.

(05:27) Factors that make onboarding into large open-source projects accessible.

(07:35) The value of improved user interfaces for task state visibility and control.

(09:49) Enabling faster debugging by exposing internal data through APIs.

(13:00) Balancing frontend design goals with backend functionality.

(14:19) Creating workflow editors that lower the barrier to entry.

(16:54) Supporting a variety of task types within a visual DAG builder.

(19:32) Common infrastructure challenges faced by orchestration users.

(20:37) Addressing dependency management across distributed environments.


Resources Mentioned:


Shubham Raj

https://www.linkedin.com/in/shubhamrajofficial/


Cloudera | LinkedIn

https://www.linkedin.com/company/cloudera/


Cloudera | Website

https://www.cloudera.com/


Apache Airflow

https://airflow.apache.org/


2023 Airflow Summit

https://airflowsummit.org/


https://www.astronomer.io/events/roadshow/london/  


https://www.astronomer.io/events/roadshow/new-york/  


https://www.astronomer.io/events/roadshow/sydney/  


https://www.astronomer.io/events/roadshow/san-francisco/  


https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) a27bf596-e73f-49ac-82d8-f0cae4167aab Thu, 10 Jul 2025 00:00:00 -0400 1348 1 48 false Streamlining Thousands of Data Pipelines at Lyft with Yunhao Qing Streamlining Thousands of Data Pipelines at Lyft with Yunhao Qing Managing data pipelines at scale is not just a technical challenge. It is also an organizational one. At Lyft, success means empowering dozens of teams to build with autonomy while enforcing governance and best practices across thousands of workflows.


In this episode, we speak with Yunhao Qing, Software Engineer at Lyft, about building a governed data-engineering platform powered by Airflow that balances flexibility, standardization and scale.


Key Takeaways:


(03:17) Supporting internal teams with a centralized orchestration platform.

(04:54) Migrating to a managed service to reduce infrastructure overhead.

(06:04) Embedding platform-level governance into custom components.

(08:02) Consolidating and regulating the creation of custom code.

(09:48) Identifying and correcting inefficient workflow patterns.

(11:17) Replacing manual workarounds with native platform features.

(14:32) Preparing teams for major version upgrades.

(16:03) Leveraging asset-based scheduling for smarter triggers.

(18:13) Envisioning GenAI and semantic search for future productivity.


Resources Mentioned:


Yunhao Qing

https://www.linkedin.com/in/yunhao-qing


Lyft | LinkedIn

https://www.linkedin.com/company/lyft/


Lyft | Website

https://www.lyft.com/


Apache Airflow

https://airflow.apache.org/


Astronomer

https://www.astronomer.io/


Kubernetes

https://kubernetes.io/


https://www.astronomer.io/events/roadshow/london/

  

https://www.astronomer.io/events/roadshow/new-york/  


https://www.astronomer.io/events/roadshow/sydney/  


https://www.astronomer.io/events/roadshow/san-francisco/  


https://www.astronomer.io/events/roadshow/chicago/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> Managing data pipelines at scale is not just a technical challenge. It is also an organizational one. At Lyft, success means empowering dozens of teams to build with autonomy while enforcing governance and best practices across thousands of workflows.


In this episode, we speak with Yunhao Qing, Software Engineer at Lyft, about building a governed data-engineering platform powered by Airflow that balances flexibility, standardization and scale.


Key Takeaways:


(03:17) Supporting internal teams with a centralized orchestration platform.

(04:54) Migrating to a managed service to reduce infrastructure overhead.

(06:04) Embedding platform-level governance into custom components.

(08:02) Consolidating and regulating the creation of custom code.

(09:48) Identifying and correcting inefficient workflow patterns.

(11:17) Replacing manual workarounds with native platform features.

(14:32) Preparing teams for major version upgrades.

(16:03) Leveraging asset-based scheduling for smarter triggers.

(18:13) Envisioning GenAI and semantic search for future productivity.


Resources Mentioned:


Yunhao Qing

https://www.linkedin.com/in/yunhao-qing


Lyft | LinkedIn

https://www.linkedin.com/company/lyft/


Lyft | Website

https://www.lyft.com/


Apache Airflow

https://airflow.apache.org/


Astronomer

https://www.astronomer.io/


Kubernetes

https://kubernetes.io/


https://www.astronomer.io/events/roadshow/london/

  

https://www.astronomer.io/events/roadshow/new-york/  


https://www.astronomer.io/events/roadshow/sydney/  


https://www.astronomer.io/events/roadshow/san-francisco/  


https://www.astronomer.io/events/roadshow/chicago/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 74dbd93c-1be1-48af-8121-63059da869e3 Mon, 07 Jul 2025 00:00:00 -0400 1174 1 47 false Transforming Customer Education in Data Engineering at Astronomer with Marc Lamberti Transforming Customer Education in Data Engineering at Astronomer with Marc Lamberti Understanding the complexities of Apache Airflow can be daunting for newcomers and seasoned data engineers. But with the right guidance, mastering the tool becomes an achievable milestone.


In this episode, Marc Lamberti, Head of Customer Education at Astronomer, joins us to share his journey from Udemy instructor to driving education at Astronomer, and how he's helping over 100,000 learners demystify Airflow.


Key Takeaways:


(02:36) Early exposure to Airflow while addressing inefficiencies in data workflows.

(04:10) Common barriers to implementing open source tools in enterprise settings.

(06:18) The shift from part-time teaching to a full-time focus on Airflow education.

(07:53) A modular, guided approach to structuring educational content.

(09:57) The value of highlighting underused Airflow features for broader adoption.

(12:35) Certifications as a method to assess readiness and uncover knowledge gaps.

(13:25) Coverage of essential Airflow concepts in the Fundamentals exam.

(16:07) The DAG Authoring exam’s emphasis on practical, advanced features.

(20:08) A call for more visible integration of Airflow with AI workflows.


Resources Mentioned:


Marc Lamberti

https://www.linkedin.com/in/marclamberti/


Astronomer | LinkedIn

https://www.linkedin.com/company/astronomer/


Astronomer Academy

https://academy.astronomer.io/


Airflow Fundamentals Certification

https://www.astronomer.io/certification/


DAG Authoring Certification

https://academy.astronomer.io/plan/astronomer-certification-dag-authoring-for-apache-airflow-exam


The Complete Hands-On Introduction to Airflow

https://www.udemy.com/course/the-complete-hands-on-course-to-master-apache-airflow/?utm_source=adwords&utm_medium=udemyads&utm_campaign=Search_DSA_Beta_Prof_la.EN_cc.ROW-English&campaigntype=Search&portfolio=ROW-English&language=EN&product=Course&test=&audience=DSA&topic=&priority=Beta&utm_content=deal4584&utm_term=_._ag_162511579404_._ad_696197165418_._kw__._de_c_._dm__._pl__._ti_dsa-1677053911088_._li_9061346_._pd__._&matchtype=&gad_source=1&gad_campaignid=21168154305&gbraid=0AAAAADROdO3MpljfP-gssiYSmDEPdhZV9&gclid=Cj0KCQjw097CBhDIARIsAJ3-nxdjZA6G5-Y0-akk6Huksy2PLb04t92J4iNfUSIbMdrSAla_tb-o2N8aArOeEALw_wcB&couponCode=PMNVD3025


https://www.astronomer.io/events/roadshow/london/  


https://www.astronomer.io/events/roadshow/new-york/  


https://www.astronomer.io/events/roadshow/sydney/  


https://www.astronomer.io/events/roadshow/san-francisco/  


https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Understanding the complexities of Apache Airflow can be daunting for newcomers and seasoned data engineers. But with the right guidance, mastering the tool becomes an achievable milestone.


In this episode, Marc Lamberti, Head of Customer Education at Astronomer, joins us to share his journey from Udemy instructor to driving education at Astronomer, and how he's helping over 100,000 learners demystify Airflow.


Key Takeaways:


(02:36) Early exposure to Airflow while addressing inefficiencies in data workflows.

(04:10) Common barriers to implementing open source tools in enterprise settings.

(06:18) The shift from part-time teaching to a full-time focus on Airflow education.

(07:53) A modular, guided approach to structuring educational content.

(09:57) The value of highlighting underused Airflow features for broader adoption.

(12:35) Certifications as a method to assess readiness and uncover knowledge gaps.

(13:25) Coverage of essential Airflow concepts in the Fundamentals exam.

(16:07) The DAG Authoring exam’s emphasis on practical, advanced features.

(20:08) A call for more visible integration of Airflow with AI workflows.


Resources Mentioned:


Marc Lamberti

https://www.linkedin.com/in/marclamberti/


Astronomer | LinkedIn

https://www.linkedin.com/company/astronomer/


Astronomer Academy

https://academy.astronomer.io/


Airflow Fundamentals Certification

https://www.astronomer.io/certification/


DAG Authoring Certification

https://academy.astronomer.io/plan/astronomer-certification-dag-authoring-for-apache-airflow-exam


The Complete Hands-On Introduction to Airflow

https://www.udemy.com/course/the-complete-hands-on-course-to-master-apache-airflow/?utm_source=adwords&utm_medium=udemyads&utm_campaign=Search_DSA_Beta_Prof_la.EN_cc.ROW-English&campaigntype=Search&portfolio=ROW-English&language=EN&product=Course&test=&audience=DSA&topic=&priority=Beta&utm_content=deal4584&utm_term=_._ag_162511579404_._ad_696197165418_._kw__._de_c_._dm__._pl__._ti_dsa-1677053911088_._li_9061346_._pd__._&matchtype=&gad_source=1&gad_campaignid=21168154305&gbraid=0AAAAADROdO3MpljfP-gssiYSmDEPdhZV9&gclid=Cj0KCQjw097CBhDIARIsAJ3-nxdjZA6G5-Y0-akk6Huksy2PLb04t92J4iNfUSIbMdrSAla_tb-o2N8aArOeEALw_wcB&couponCode=PMNVD3025


https://www.astronomer.io/events/roadshow/london/  


https://www.astronomer.io/events/roadshow/new-york/  


https://www.astronomer.io/events/roadshow/sydney/  


https://www.astronomer.io/events/roadshow/san-francisco/  


https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) f9a2d7bf-b988-4651-8e81-3b9804163263 Thu, 26 Jun 2025 03:44:47 -0400 1339 1 46 false Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi The flexibility of Airflow plays a pivotal role in enabling decentralized data architectures and empowering cross-functional teams.


In this episode, we speak with Alberto Crespi, Data Architect at lastminute.com, who shares how his team scales Airflow across 12 teams while supporting both vertical and horizontal structures under a data mesh approach.


Key Takeaways:


(02:17) Defining responsibilities within data architecture teams.

(04:15) Consolidating multiple orchestrators into a single solution.

(07:00) Scaling Airflow environments with shared infrastructure and DevOps practices.

(10:59) Managing dependencies and readiness using SQL sensors.

(14:23) Enhancing visibility and response through Slack-integrated monitoring.

(19:28) Extending Airflow’s flexibility to run legacy systems.

(22:28) Integrating transformation tools into orchestrated pipelines.

(25:54) Enabling non-engineers to contribute to pipeline development.

(27:33) Fostering adoption through collaboration and communication.


Resources Mentioned:


Alberto Crespi

https://www.linkedin.com/in/crespialberto/


lastminute.com | Website

https://lastminute.com


Apache Airflow

https://airflow.apache.org/


dbt Labs

https://www.getdbt.com/


Astronomer Cosmos

https://github.com/astronomer/astronomer-cosmos


GitLabSlack

https://slack.com/


Kubernetes

https://kubernetes.io/


Confluence

https://www.atlassian.com/software/confluence


Slack

https://slack.com/


https://www.astronomer.io/events/roadshow/london/

   

https://www.astronomer.io/events/roadshow/new-york/

  

https://www.astronomer.io/events/roadshow/sydney/

  

https://www.astronomer.io/events/roadshow/san-francisco/

  

https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> The flexibility of Airflow plays a pivotal role in enabling decentralized data architectures and empowering cross-functional teams.


In this episode, we speak with Alberto Crespi, Data Architect at lastminute.com, who shares how his team scales Airflow across 12 teams while supporting both vertical and horizontal structures under a data mesh approach.


Key Takeaways:


(02:17) Defining responsibilities within data architecture teams.

(04:15) Consolidating multiple orchestrators into a single solution.

(07:00) Scaling Airflow environments with shared infrastructure and DevOps practices.

(10:59) Managing dependencies and readiness using SQL sensors.

(14:23) Enhancing visibility and response through Slack-integrated monitoring.

(19:28) Extending Airflow’s flexibility to run legacy systems.

(22:28) Integrating transformation tools into orchestrated pipelines.

(25:54) Enabling non-engineers to contribute to pipeline development.

(27:33) Fostering adoption through collaboration and communication.


Resources Mentioned:


Alberto Crespi

https://www.linkedin.com/in/crespialberto/


lastminute.com | Website

https://lastminute.com


Apache Airflow

https://airflow.apache.org/


dbt Labs

https://www.getdbt.com/


Astronomer Cosmos

https://github.com/astronomer/astronomer-cosmos


GitLabSlack

https://slack.com/


Kubernetes

https://kubernetes.io/


Confluence

https://www.atlassian.com/software/confluence


Slack

https://slack.com/


https://www.astronomer.io/events/roadshow/london/

   

https://www.astronomer.io/events/roadshow/new-york/

  

https://www.astronomer.io/events/roadshow/sydney/

  

https://www.astronomer.io/events/roadshow/san-francisco/

  

https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 1b224376-6342-4b56-9318-8ac164287a2f Fri, 20 Jun 2025 02:18:19 -0400 1809 1 45 false The AI-Ready Pipeline: Reimagining Airflow at Veyer® Logistics with Anu Pabla The AI-Ready Pipeline: Reimagining Airflow at Veyer® Logistics with Anu Pabla Innovation in orchestration is redefining how engineers approach both traditional ETL pipelines and emerging AI workloads. Understanding how to harness Airflow’s flexibility and observability is essential for teams navigating today’s evolving data landscape.


In this episode, Anu Pabla, Principal Engineer at The ODP Corporation, joins us to discuss her journey from legacy orchestration patterns to AI-native pipelines and why she sees Airflow as the future of AI workload orchestration.


Key Takeaways:


(03:43) Engaging with external technology communities fosters innovation.

(05:05) Mentoring early-career engineers builds confidence in a complex tech landscape.

(07:51) Orchestration patterns continue to evolve with modern data needs.

(08:41) Managing AI workflows requires structured and flexible orchestration.

(10:35) High-quality, meaningful data remains foundational across use cases.

(15:08) Community-driven open source tools offer lasting value.

(16:59) Self-healing systems support both legacy and AI pipelines.

(20:20) Orchestration platforms can drive future AI-native workloads.


Resources Mentioned:


Anu Pabla

https://www.linkedin.com/in/atomicap/


The ODP Corporation

https://www.linkedin.com/company/the-odp-corporation/


The ODP Corporation | Website

https://www.theodpcorp.com/homepage


Apache Airflow

https://airflow.apache.org/


LlamaIndex

https://www.llamaindex.ai/


https://www.astronomer.io/events/roadshow/london/  


https://www.astronomer.io/events/roadshow/new-york/  


https://www.astronomer.io/events/roadshow/sydney/  


https://www.astronomer.io/events/roadshow/san-francisco/  


https://www.astronomer.io/events/roadshow/chicago/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Innovation in orchestration is redefining how engineers approach both traditional ETL pipelines and emerging AI workloads. Understanding how to harness Airflow’s flexibility and observability is essential for teams navigating today’s evolving data landscape.


In this episode, Anu Pabla, Principal Engineer at The ODP Corporation, joins us to discuss her journey from legacy orchestration patterns to AI-native pipelines and why she sees Airflow as the future of AI workload orchestration.


Key Takeaways:


(03:43) Engaging with external technology communities fosters innovation.

(05:05) Mentoring early-career engineers builds confidence in a complex tech landscape.

(07:51) Orchestration patterns continue to evolve with modern data needs.

(08:41) Managing AI workflows requires structured and flexible orchestration.

(10:35) High-quality, meaningful data remains foundational across use cases.

(15:08) Community-driven open source tools offer lasting value.

(16:59) Self-healing systems support both legacy and AI pipelines.

(20:20) Orchestration platforms can drive future AI-native workloads.


Resources Mentioned:


Anu Pabla

https://www.linkedin.com/in/atomicap/


The ODP Corporation

https://www.linkedin.com/company/the-odp-corporation/


The ODP Corporation | Website

https://www.theodpcorp.com/homepage


Apache Airflow

https://airflow.apache.org/


LlamaIndex

https://www.llamaindex.ai/


https://www.astronomer.io/events/roadshow/london/  


https://www.astronomer.io/events/roadshow/new-york/  


https://www.astronomer.io/events/roadshow/sydney/  


https://www.astronomer.io/events/roadshow/san-francisco/  


https://www.astronomer.io/events/roadshow/chicago/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 69a6710a-747d-48ba-adf8-413c5a4b523a Thu, 12 Jun 2025 06:48:03 -0400 1401 1 44 false Streamlining AI and ML Operations at IBM with BJ Adesoji and Ryan Yackel Streamlining AI and ML Operations at IBM with BJ Adesoji and Ryan Yackel The orchestration layer is foundational to building robust AI- and ML-powered data pipelines, especially in complex hybrid enterprise environments. IBM’s partnership with Astronomer reflects a strategic alignment to simplify and scale Airflow-based workflows across industries.


In this episode, we’re joined by IBM’s Senior Product Manager, BJ Adesoji, and GTM PM and Growth Leader, Ryan Yackel. We discuss how IBM customers are using Airflow in production, the challenges they face at scale and what the new IBM–Astronomer collaboration unlocks.


Key Takeaways:


(03:09) The growing importance of orchestration tools in enterprise environments.

(04:48) How organizations are expanding orchestration beyond traditional use cases.

(05:24) Common patterns across industries adopting orchestration platforms.

(07:16) Why orchestration is essential for supporting business-critical workloads.

(10:00) The role of orchestration in compliance and regulatory processes.

(13:02) Challenges enterprises face when managing orchestration infrastructure.

(14:58) Opportunities to simplify and centralize orchestration at scale.

(19:11) The value of integrating orchestration with broader data toolchains.

(20:54) How AI is shaping the future of orchestrated data workflows.


Resources Mentioned:


BJ Adesoji

https://www.linkedin.com/in/bj-soji/


Ryan Yackel

https://www.linkedin.com/in/ryanyackel/


IBM | LinkedIn

https://www.linkedin.com/company/databand-ai/


IBM Databand

https://www.ibm.com/products/databand


IBM DataStage

https://www.ibm.com/products/datastage


IBM watsonx.governance

https://www.ibm.com/products/watsonx-governance


IBM Knowledge Catalog

https://www.ibm.com/products/knowledge-catalog


Apache Airflow

https://airflow.apache.org/


watsonx Orchestrate

https://www.ibm.com/products/watsonx-orchestrate


Domino

https://domino.ai/


Astronomer

https://www.astronomer.io/


Snowflake

https://www.snowflake.com/en/


dbt Labs

https://www.getdbt.com/


Amazon SageMaker

https://aws.amazon.com/sagemaker/


Cloudera

https://www.cloudera.com/


MongoDB

https://www.mongodb.com/


https://www.astronomer.io/events/roadshow/london/ 


https://www.astronomer.io/events/roadshow/new-york/ 


https://www.astronomer.io/events/roadshow/sydney/ 


https://www.astronomer.io/events/roadshow/san-francisco/ 


https://www.astronomer.io/events/roadshow/chicago/ 





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> The orchestration layer is foundational to building robust AI- and ML-powered data pipelines, especially in complex hybrid enterprise environments. IBM’s partnership with Astronomer reflects a strategic alignment to simplify and scale Airflow-based workflows across industries.


In this episode, we’re joined by IBM’s Senior Product Manager, BJ Adesoji, and GTM PM and Growth Leader, Ryan Yackel. We discuss how IBM customers are using Airflow in production, the challenges they face at scale and what the new IBM–Astronomer collaboration unlocks.


Key Takeaways:


(03:09) The growing importance of orchestration tools in enterprise environments.

(04:48) How organizations are expanding orchestration beyond traditional use cases.

(05:24) Common patterns across industries adopting orchestration platforms.

(07:16) Why orchestration is essential for supporting business-critical workloads.

(10:00) The role of orchestration in compliance and regulatory processes.

(13:02) Challenges enterprises face when managing orchestration infrastructure.

(14:58) Opportunities to simplify and centralize orchestration at scale.

(19:11) The value of integrating orchestration with broader data toolchains.

(20:54) How AI is shaping the future of orchestrated data workflows.


Resources Mentioned:


BJ Adesoji

https://www.linkedin.com/in/bj-soji/


Ryan Yackel

https://www.linkedin.com/in/ryanyackel/


IBM | LinkedIn

https://www.linkedin.com/company/databand-ai/


IBM Databand

https://www.ibm.com/products/databand


IBM DataStage

https://www.ibm.com/products/datastage


IBM watsonx.governance

https://www.ibm.com/products/watsonx-governance


IBM Knowledge Catalog

https://www.ibm.com/products/knowledge-catalog


Apache Airflow

https://airflow.apache.org/


watsonx Orchestrate

https://www.ibm.com/products/watsonx-orchestrate


Domino

https://domino.ai/


Astronomer

https://www.astronomer.io/


Snowflake

https://www.snowflake.com/en/


dbt Labs

https://www.getdbt.com/


Amazon SageMaker

https://aws.amazon.com/sagemaker/


Cloudera

https://www.cloudera.com/


MongoDB

https://www.mongodb.com/


https://www.astronomer.io/events/roadshow/london/ 


https://www.astronomer.io/events/roadshow/new-york/ 


https://www.astronomer.io/events/roadshow/sydney/ 


https://www.astronomer.io/events/roadshow/san-francisco/ 


https://www.astronomer.io/events/roadshow/chicago/ 





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) ec1ab1c3-97e7-4840-8e8c-ba5f6561d4ee Thu, 05 Jun 2025 00:00:00 -0400 1484 1 43 false Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich Efficient orchestration and maintainability are crucial for data engineering at scale. Gil Reich, Data Developer for Data Science at Wix, shares how his team reduced code duplication, standardized pipelines, and improved Airflow task orchestration using a Python-based framework built within the data science team.


In this episode, Gil explains how this internal framework simplifies DAG creation, improves documentation accuracy, and enables consistent task generation for machine learning pipelines. He also shares lessons from complex DAG optimization and maintaining testable code.


Key Takeaways:


(03:23) Code duplication creates long-term problems.

(08:16) Frameworks bring order to complex pipelines.

(09:41) Shared functions cut down repetitive code.

(17:18) Auto-generated docs stay accurate by design.

(22:40) On-demand DAGs support real-time workflows.

(25:08) Task-level sensors improve run efficiency.

(27:40) Combine local runs with automated tests.

(30:09) Clean code helps teams scale faster.


Resources Mentioned:


Gil Reich

https://www.linkedin.com/in/gilreich/


Wix | LinkedIn

https://www.linkedin.com/company/wix-com/


Wix | Website

https://www.wix.com/


DS DAG Framework

https://airflowsummit.org/slides/2024/92-refactoring-dags.pdf


Apache Airflow

https://airflow.apache.org/


https://www.astronomer.io/events/roadshow/london/  


https://www.astronomer.io/events/roadshow/new-york/  


https://www.astronomer.io/events/roadshow/sydney/  


https://www.astronomer.io/events/roadshow/san-francisco/  


https://www.astronomer.io/events/roadshow/chicago/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Efficient orchestration and maintainability are crucial for data engineering at scale. Gil Reich, Data Developer for Data Science at Wix, shares how his team reduced code duplication, standardized pipelines, and improved Airflow task orchestration using a Python-based framework built within the data science team.


In this episode, Gil explains how this internal framework simplifies DAG creation, improves documentation accuracy, and enables consistent task generation for machine learning pipelines. He also shares lessons from complex DAG optimization and maintaining testable code.


Key Takeaways:


(03:23) Code duplication creates long-term problems.

(08:16) Frameworks bring order to complex pipelines.

(09:41) Shared functions cut down repetitive code.

(17:18) Auto-generated docs stay accurate by design.

(22:40) On-demand DAGs support real-time workflows.

(25:08) Task-level sensors improve run efficiency.

(27:40) Combine local runs with automated tests.

(30:09) Clean code helps teams scale faster.


Resources Mentioned:


Gil Reich

https://www.linkedin.com/in/gilreich/


Wix | LinkedIn

https://www.linkedin.com/company/wix-com/


Wix | Website

https://www.wix.com/


DS DAG Framework

https://airflowsummit.org/slides/2024/92-refactoring-dags.pdf


Apache Airflow

https://airflow.apache.org/


https://www.astronomer.io/events/roadshow/london/  


https://www.astronomer.io/events/roadshow/new-york/  


https://www.astronomer.io/events/roadshow/sydney/  


https://www.astronomer.io/events/roadshow/san-francisco/  


https://www.astronomer.io/events/roadshow/chicago/




Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) e9412b45-e1f5-4f3d-8f5f-d4ce478b8379 Thu, 29 May 2025 00:00:00 -0400 1862 1 42 false Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero Legacy architecture and AI workloads pose unique challenges at scale, especially in a global enterprise with complex data systems. In this episode, we explore strategies to proactively monitor and optimize pipelines while minimizing downstream failures.


Adonis Castillo Cordero, Senior Automation Manager at Procter & Gamble, joins us to share actionable best practices for dependency mapping, anomaly detection and architecture simplification using Apache Airflow.


Key Takeaways:


(03:13) Integrating legacy data systems into modern architecture.

(05:51) Designing workflows for real-time data processing.

(07:57) Mapping dependencies early to avoid pipeline failures.

(09:02) Building automated monitoring into orchestration frameworks.

(12:09) Detecting anomalies to prevent performance bottlenecks.

(15:24) Monitoring data quality to catch silent failures.

(17:02) Prioritizing responses based on impact severity.

(18:55) Simplifying dashboards to highlight critical metrics.


Resources Mentioned:


Adonis Castillo Cordero

https://www.linkedin.com/in/adoniscc/


Procter & Gamble | LinkedIn

https://www.linkedin.com/company/procter-and-gamble/


Procter & Gamble | Website

http://www.pg.com


Apache Airflow

https://airflow.apache.org/


OpenLineage

https://openlineage.io/


Azure Monitor

https://azure.microsoft.com/en-us/products/monitor/


AWS Lookout for Metrics

https://aws.amazon.com/lookout-for-metrics/


Monte Carlo

https://www.montecarlodata.com/


Great Expectations

https://greatexpectations.io/


https://www.astronomer.io/events/roadshow/london/

  

https://www.astronomer.io/events/roadshow/new-york/

  

https://www.astronomer.io/events/roadshow/sydney/ 

 

https://www.astronomer.io/events/roadshow/san-francisco/ 

 

https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Legacy architecture and AI workloads pose unique challenges at scale, especially in a global enterprise with complex data systems. In this episode, we explore strategies to proactively monitor and optimize pipelines while minimizing downstream failures.


Adonis Castillo Cordero, Senior Automation Manager at Procter & Gamble, joins us to share actionable best practices for dependency mapping, anomaly detection and architecture simplification using Apache Airflow.


Key Takeaways:


(03:13) Integrating legacy data systems into modern architecture.

(05:51) Designing workflows for real-time data processing.

(07:57) Mapping dependencies early to avoid pipeline failures.

(09:02) Building automated monitoring into orchestration frameworks.

(12:09) Detecting anomalies to prevent performance bottlenecks.

(15:24) Monitoring data quality to catch silent failures.

(17:02) Prioritizing responses based on impact severity.

(18:55) Simplifying dashboards to highlight critical metrics.


Resources Mentioned:


Adonis Castillo Cordero

https://www.linkedin.com/in/adoniscc/


Procter & Gamble | LinkedIn

https://www.linkedin.com/company/procter-and-gamble/


Procter & Gamble | Website

http://www.pg.com


Apache Airflow

https://airflow.apache.org/


OpenLineage

https://openlineage.io/


Azure Monitor

https://azure.microsoft.com/en-us/products/monitor/


AWS Lookout for Metrics

https://aws.amazon.com/lookout-for-metrics/


Monte Carlo

https://www.montecarlodata.com/


Great Expectations

https://greatexpectations.io/


https://www.astronomer.io/events/roadshow/london/

  

https://www.astronomer.io/events/roadshow/new-york/

  

https://www.astronomer.io/events/roadshow/sydney/ 

 

https://www.astronomer.io/events/roadshow/san-francisco/ 

 

https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 569743f1-2c08-4c37-a04e-5b186a5b7de3 Thu, 22 May 2025 00:00:00 -0400 1333 1 41 false Building an End-to-End Data Observability System at Netflix with Joseph Machado Building an End-to-End Data Observability System at Netflix with Joseph Machado Building reliable data pipelines starts with maintaining strong data quality standards and creating efficient systems for auditing, publishing and monitoring. In this episode, we explore the real-world patterns and best practices for ensuring data pipelines stay accurate, scalable and trustworthy.


Joseph Machado, Senior Data Engineer at Netflix, joins us to share practical insights gleaned from supporting Netflix’s Ads business as well as over a decade of experience in the data engineering space. He discusses implementing audit publish patterns, building observability dashboards, defining in-band and separate data quality checks, and optimizing data validation across large-scale systems.


Key Takeaways:

.

(03:14) Supporting data privacy and engineering efficiency within data systems.

(10:41) Validating outputs with reconciliation checks to catch transformation issues.

(16:06) Applying standardized patterns for auditing, validating and publishing data.

(19:28) Capturing historical check results to monitor system health and improvements.

(21:29) Treating data quality and availability as separate monitoring concerns.

(26:26) Using containerization strategies to streamline pipeline executions.

(29:47) Leveraging orchestration platforms for better visibility and retry capability.

(31:59) Managing business pressure without sacrificing data quality practices.

(35:46) Starting simple with quality checks and evolving toward more complex frameworks.


Resources Mentioned:


Joseph Machado

https://www.linkedin.com/in/josephmachado1991/


Netflix | LinkedIn

https://www.linkedin.com/company/netflix/


Netflix | Website

https://www.netflix.com/browse


Start Data Engineering

https://www.startdataengineering.com/


Apache Airflow

https://airflow.apache.org/


dbt Labs

https://www.getdbt.com/


Great Expectations

https://greatexpectations.io/


https://www.astronomer.io/events/roadshow/london/


https://www.astronomer.io/events/roadshow/new-york/

 

https://www.astronomer.io/events/roadshow/sydney/


https://www.astronomer.io/events/roadshow/san-francisco/


https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> Building reliable data pipelines starts with maintaining strong data quality standards and creating efficient systems for auditing, publishing and monitoring. In this episode, we explore the real-world patterns and best practices for ensuring data pipelines stay accurate, scalable and trustworthy.


Joseph Machado, Senior Data Engineer at Netflix, joins us to share practical insights gleaned from supporting Netflix’s Ads business as well as over a decade of experience in the data engineering space. He discusses implementing audit publish patterns, building observability dashboards, defining in-band and separate data quality checks, and optimizing data validation across large-scale systems.


Key Takeaways:

.

(03:14) Supporting data privacy and engineering efficiency within data systems.

(10:41) Validating outputs with reconciliation checks to catch transformation issues.

(16:06) Applying standardized patterns for auditing, validating and publishing data.

(19:28) Capturing historical check results to monitor system health and improvements.

(21:29) Treating data quality and availability as separate monitoring concerns.

(26:26) Using containerization strategies to streamline pipeline executions.

(29:47) Leveraging orchestration platforms for better visibility and retry capability.

(31:59) Managing business pressure without sacrificing data quality practices.

(35:46) Starting simple with quality checks and evolving toward more complex frameworks.


Resources Mentioned:


Joseph Machado

https://www.linkedin.com/in/josephmachado1991/


Netflix | LinkedIn

https://www.linkedin.com/company/netflix/


Netflix | Website

https://www.netflix.com/browse


Start Data Engineering

https://www.startdataengineering.com/


Apache Airflow

https://airflow.apache.org/


dbt Labs

https://www.getdbt.com/


Great Expectations

https://greatexpectations.io/


https://www.astronomer.io/events/roadshow/london/


https://www.astronomer.io/events/roadshow/new-york/

 

https://www.astronomer.io/events/roadshow/sydney/


https://www.astronomer.io/events/roadshow/san-francisco/


https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) ecd018eb-1b18-4c9e-9e95-0c99214770b6 Thu, 15 May 2025 00:00:00 -0400 2334 1 40 false Why Developer Experience Shapes Data Pipeline Standards at Next Insurance with Snir Israeli Why Developer Experience Shapes Data Pipeline Standards at Next Insurance with Snir Israeli Creating consistency across data pipelines is critical for scaling engineering teams and ensuring long-term maintainability.


In this episode, Snir Israeli, Senior Data Engineer at Next Insurance, shares how enforcing coding standards and investing in developer experience transformed their approach to data engineering. He explains how implementing automated code checks, clear documentation practices and a scoring system helped drive alignment across teams, improve collaboration and reduce technical debt in a fast-growing data environment.


Key Takeaways:


(02:59) Inconsistencies in code style create challenges for collaboration and maintenance.

(04:22) Programmatically enforcing rules helps teams scale their best practices.

(08:55) Performance improvements in data pipelines lead to infrastructure cost savings.

(13:22) Developer experience is essential for driving adoption of internal tools.

(19:44) Dashboards can operationalize standards enforcement and track progress over time.

(22:49) Standardization accelerates onboarding and reduces friction in code reviews.

(25:39) Linting rules require ongoing maintenance as tools and platforms evolve.

(27:47) Starting small and involving the team leads to better adoption and long-term success.


Resources Mentioned:


Snir Israeli

https://www.linkedin.com/in/snir-israeli/


Next Insurance | LinkedIn

https://www.linkedin.com/company/nextinsurance/


Next Insurance | Website

https://www.nextinsurance.com/


Apache Airflow

https://airflow.apache.org/


https://www.astronomer.io/events/roadshow/london/

   

https://www.astronomer.io/events/roadshow/new-york/   

 

https://www.astronomer.io/events/roadshow/sydney/ 

  

https://www.astronomer.io/events/roadshow/san-francisco/ 

  

https://www.astronomer.io/events/roadshow/chicago/ 





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> Creating consistency across data pipelines is critical for scaling engineering teams and ensuring long-term maintainability.


In this episode, Snir Israeli, Senior Data Engineer at Next Insurance, shares how enforcing coding standards and investing in developer experience transformed their approach to data engineering. He explains how implementing automated code checks, clear documentation practices and a scoring system helped drive alignment across teams, improve collaboration and reduce technical debt in a fast-growing data environment.


Key Takeaways:


(02:59) Inconsistencies in code style create challenges for collaboration and maintenance.

(04:22) Programmatically enforcing rules helps teams scale their best practices.

(08:55) Performance improvements in data pipelines lead to infrastructure cost savings.

(13:22) Developer experience is essential for driving adoption of internal tools.

(19:44) Dashboards can operationalize standards enforcement and track progress over time.

(22:49) Standardization accelerates onboarding and reduces friction in code reviews.

(25:39) Linting rules require ongoing maintenance as tools and platforms evolve.

(27:47) Starting small and involving the team leads to better adoption and long-term success.


Resources Mentioned:


Snir Israeli

https://www.linkedin.com/in/snir-israeli/


Next Insurance | LinkedIn

https://www.linkedin.com/company/nextinsurance/


Next Insurance | Website

https://www.nextinsurance.com/


Apache Airflow

https://airflow.apache.org/


https://www.astronomer.io/events/roadshow/london/

   

https://www.astronomer.io/events/roadshow/new-york/   

 

https://www.astronomer.io/events/roadshow/sydney/ 

  

https://www.astronomer.io/events/roadshow/san-francisco/ 

  

https://www.astronomer.io/events/roadshow/chicago/ 





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 02e6e034-6ffe-489f-8109-dc83387c30b0 Thu, 08 May 2025 01:50:06 -0400 1828 1 39 false Data Quality and Observability at Tekmetric with Ipsa Trivedi Data Quality and Observability at Tekmetric with Ipsa Trivedi Airflow’s adaptability is driving Tekmetric’s ability to unify complex data workflows, deliver accurate insights and support both internal operations and customer-facing services — all within a rapidly growing startup environment.


In this episode, Ipsa Trivedi, Lead Data Engineer at Tekmetric, shares how her team is standardizing pipelines while supporting unique customer needs. She explains how Airflow enables end-to-end data services, simplifies orchestration across varied sources and supports scalable customization. Ipsa also highlights early wins with Airflow, its intuitive UI and the team's roadmap toward data quality, observability and a future self-serve data platform.


Key Takeaways:


(02:26) Powering auto shops nationwide with a unified platform.

(05:17) A new data team was formed to centralize and scale insights.

(07:23) Flexible, open source and made to fit — Airflow wins.

(10:42) Pipelines handle anything from email to AWS.

(12:15) Custom DAGs fit every team’s unique needs.

(17:01) Data quality checks are built into the plan.

(18:17) Self-serve data mesh is the end goal.

(19:59) Airflow now fits so well, there's nothing left on the wishlist.


Resources Mentioned:


Ipsa Trivedi

https://www.linkedin.com/in/ipsatrivedi/


Tekmetric | LinkedIn

https://www.linkedin.com/company/tekmetric/


Tekmetric | Website

https://www.tekmetric.com/


Apache Airflow

https://airflow.apache.org/


AWS RDS

https://aws.amazon.com/free/database/?trk=fc551e06-56b0-418c-9ddd-5c9dba18569b&sc_channel=ps&ef_id=CjwKCAjwzMi_BhACEiwAX4YZULS4jV2Xpnpcac_Q3eS9BAg-klKUDyCt6XSdOul8BLHkmWzFFh4NXRoCGhQQAvD_BwE:G:s&s_kwcid=AL!4422!3!548989592596!e!!g!!amazon%20sql%20database!11543056228!112002958549&gclid=CjwKCAjwzMi_BhACEiwAX4YZULS4jV2Xpnpcac_Q3eS9BAg-klKUDyCt6XSdOul8BLHkmWzFFh4NXRoCGhQQAvD_BwE


Astro by Astronomer

https://www.astronomer.io/product/


https://www.astronomer.io/events/roadshow/london/

   

https://www.astronomer.io/events/roadshow/new-york/ 

  

https://www.astronomer.io/events/roadshow/sydney/   


https://www.astronomer.io/events/roadshow/san-francisco/ 

  

https://www.astronomer.io/events/roadshow/chicago/ 





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Airflow’s adaptability is driving Tekmetric’s ability to unify complex data workflows, deliver accurate insights and support both internal operations and customer-facing services — all within a rapidly growing startup environment.


In this episode, Ipsa Trivedi, Lead Data Engineer at Tekmetric, shares how her team is standardizing pipelines while supporting unique customer needs. She explains how Airflow enables end-to-end data services, simplifies orchestration across varied sources and supports scalable customization. Ipsa also highlights early wins with Airflow, its intuitive UI and the team's roadmap toward data quality, observability and a future self-serve data platform.


Key Takeaways:


(02:26) Powering auto shops nationwide with a unified platform.

(05:17) A new data team was formed to centralize and scale insights.

(07:23) Flexible, open source and made to fit — Airflow wins.

(10:42) Pipelines handle anything from email to AWS.

(12:15) Custom DAGs fit every team’s unique needs.

(17:01) Data quality checks are built into the plan.

(18:17) Self-serve data mesh is the end goal.

(19:59) Airflow now fits so well, there's nothing left on the wishlist.


Resources Mentioned:


Ipsa Trivedi

https://www.linkedin.com/in/ipsatrivedi/


Tekmetric | LinkedIn

https://www.linkedin.com/company/tekmetric/


Tekmetric | Website

https://www.tekmetric.com/


Apache Airflow

https://airflow.apache.org/


AWS RDS

https://aws.amazon.com/free/database/?trk=fc551e06-56b0-418c-9ddd-5c9dba18569b&sc_channel=ps&ef_id=CjwKCAjwzMi_BhACEiwAX4YZULS4jV2Xpnpcac_Q3eS9BAg-klKUDyCt6XSdOul8BLHkmWzFFh4NXRoCGhQQAvD_BwE:G:s&s_kwcid=AL!4422!3!548989592596!e!!g!!amazon%20sql%20database!11543056228!112002958549&gclid=CjwKCAjwzMi_BhACEiwAX4YZULS4jV2Xpnpcac_Q3eS9BAg-klKUDyCt6XSdOul8BLHkmWzFFh4NXRoCGhQQAvD_BwE


Astro by Astronomer

https://www.astronomer.io/product/


https://www.astronomer.io/events/roadshow/london/

   

https://www.astronomer.io/events/roadshow/new-york/ 

  

https://www.astronomer.io/events/roadshow/sydney/   


https://www.astronomer.io/events/roadshow/san-francisco/ 

  

https://www.astronomer.io/events/roadshow/chicago/ 





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) e00b0c49-324f-4328-aa8f-23a070950cf1 Thu, 01 May 2025 00:00:00 -0400 1369 1 38 false Introducing Apache Airflow® 3 with Vikram Koka and Jed Cunningham Introducing Apache Airflow® 3 with Vikram Koka and Jed Cunningham The Airflow 3.0 release marks a significant leap forward in modern data orchestration, introducing architectural upgrades that improve scalability, flexibility and long-term maintainability.


In this episode, we welcome Vikram Koka, Chief Strategy Officer at Astronomer, and Jed Cunningham, Principal Software Engineer at Astronomer, to discuss the architectural foundations, new features and future implications of this milestone release. They unpack the rationale behind DAG versioning and task execution interface, explain how Airflow now integrates more seamlessly within broader data ecosystems and share how these changes lay the groundwork for multi-cloud deployments, language-agnostic workflows and stronger enterprise security.


Key Takeaways:


(02:28) Modern orchestration demands new infrastructure approaches.

(05:02) Removing legacy components strengthens system stability.

(06:26) Major releases provide the opportunity to reduce technical debt.

(08:31) Frontend and API modernization enable long-term adaptability.

(09:36) Event-based triggers expand integration possibilities.

(11:54) Version control improves visibility and execution reliability.

(14:57) Centralized access to workflow definitions increases flexibility.

(21:49) Decoupled architecture supports distributed and secure deployments.

(26:17) Community collaboration is essential for sustainable growth.


Resources Mentioned:


Astronomer Website

https://www.astronomer.io


Apache Airflow

https://airflow.apache.org/


Git Bundle

https://git-scm.com/book/en/v2/Git-Tools-Bundling


FastAPI

https://fastapi.tiangolo.com/


React

https://react.dev/


https://www.astronomer.io/events/roadshow/london/


https://www.astronomer.io/events/roadshow/new-york/


https://www.astronomer.io/events/roadshow/sydney/


https://www.astronomer.io/events/roadshow/san-francisco/


https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> The Airflow 3.0 release marks a significant leap forward in modern data orchestration, introducing architectural upgrades that improve scalability, flexibility and long-term maintainability.


In this episode, we welcome Vikram Koka, Chief Strategy Officer at Astronomer, and Jed Cunningham, Principal Software Engineer at Astronomer, to discuss the architectural foundations, new features and future implications of this milestone release. They unpack the rationale behind DAG versioning and task execution interface, explain how Airflow now integrates more seamlessly within broader data ecosystems and share how these changes lay the groundwork for multi-cloud deployments, language-agnostic workflows and stronger enterprise security.


Key Takeaways:


(02:28) Modern orchestration demands new infrastructure approaches.

(05:02) Removing legacy components strengthens system stability.

(06:26) Major releases provide the opportunity to reduce technical debt.

(08:31) Frontend and API modernization enable long-term adaptability.

(09:36) Event-based triggers expand integration possibilities.

(11:54) Version control improves visibility and execution reliability.

(14:57) Centralized access to workflow definitions increases flexibility.

(21:49) Decoupled architecture supports distributed and secure deployments.

(26:17) Community collaboration is essential for sustainable growth.


Resources Mentioned:


Astronomer Website

https://www.astronomer.io


Apache Airflow

https://airflow.apache.org/


Git Bundle

https://git-scm.com/book/en/v2/Git-Tools-Bundling


FastAPI

https://fastapi.tiangolo.com/


React

https://react.dev/


https://www.astronomer.io/events/roadshow/london/


https://www.astronomer.io/events/roadshow/new-york/


https://www.astronomer.io/events/roadshow/sydney/


https://www.astronomer.io/events/roadshow/san-francisco/


https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) ac88168d-9a56-4590-9f58-2016ae6dcaf2 Thu, 24 Apr 2025 00:00:00 -0400 1648 1 37 false Airflow in Action: Powering Instacart's Complex Ecosystem Airflow in Action: Powering Instacart's Complex Ecosystem The evolution of data orchestration at Instacart highlights the journey from fragmented systems to robust, standardized infrastructure. This transformation has enabled scalability, reliability and democratization of tools for diverse user personas.


In this episode, we’re joined by Anant Agarwal, Software Engineer at Instacart, who shares insights into Instacart's Airflow journey, from its early adoption in 2019 to the present-day centralized cluster approach. Anant discusses the challenges of managing disparate clusters, the implementation of remote executors, and the strategic standardization of infrastructure and DAG patterns to streamline workflows.


Key Takeaways:


(03:49) The impact of external events on business growth and technological evolution.

(04:31) Challenges of managing decentralized systems across multiple teams.

(06:14) The importance of standardizing infrastructure and processes for scalability.

(09:51) Strategies for implementing efficient and repeatable deployment practices.

(12:17) Addressing diverse user personas with tailored solutions.

(14:47) Leveraging remote execution to enhance flexibility and scalability.

(18:36) Benefits of transitioning to a centralized system for organization-wide use.

(20:57) Maintaining an upgrade cadence to stay aligned with the latest advancements.

(23:35) Anticipation for new features and improvements in upcoming software versions.


Resources Mentioned:


Anant Agarwal

https://www.linkedin.com/in/anantag/


Instacart | LinkedIn

https://www.linkedin.com/company/instacart/


Instacart | Website

https://www.instacart.com


Apache Airflow

https://airflow.apache.org/


AWS Amazon

https://aws.amazon.com/ecs/


Terraform

https://www.terraform.io/


https://www.astronomer.io/events/roadshow/london/

https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/

https://www.astronomer.io/events/roadshow/san-francisco/

https://www.astronomer.io/events/roadshow/chicago/



Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow #MachineLearning

]]> The evolution of data orchestration at Instacart highlights the journey from fragmented systems to robust, standardized infrastructure. This transformation has enabled scalability, reliability and democratization of tools for diverse user personas.


In this episode, we’re joined by Anant Agarwal, Software Engineer at Instacart, who shares insights into Instacart's Airflow journey, from its early adoption in 2019 to the present-day centralized cluster approach. Anant discusses the challenges of managing disparate clusters, the implementation of remote executors, and the strategic standardization of infrastructure and DAG patterns to streamline workflows.


Key Takeaways:


(03:49) The impact of external events on business growth and technological evolution.

(04:31) Challenges of managing decentralized systems across multiple teams.

(06:14) The importance of standardizing infrastructure and processes for scalability.

(09:51) Strategies for implementing efficient and repeatable deployment practices.

(12:17) Addressing diverse user personas with tailored solutions.

(14:47) Leveraging remote execution to enhance flexibility and scalability.

(18:36) Benefits of transitioning to a centralized system for organization-wide use.

(20:57) Maintaining an upgrade cadence to stay aligned with the latest advancements.

(23:35) Anticipation for new features and improvements in upcoming software versions.


Resources Mentioned:


Anant Agarwal

https://www.linkedin.com/in/anantag/


Instacart | LinkedIn

https://www.linkedin.com/company/instacart/


Instacart | Website

https://www.instacart.com


Apache Airflow

https://airflow.apache.org/


AWS Amazon

https://aws.amazon.com/ecs/


Terraform

https://www.terraform.io/


https://www.astronomer.io/events/roadshow/london/

https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/

https://www.astronomer.io/events/roadshow/san-francisco/

https://www.astronomer.io/events/roadshow/chicago/



Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 73a928f0-df79-4f7f-96a4-7b8e06e73033 Thu, 17 Apr 2025 00:00:00 -0400 1514 1 36 false From ETL to Airflow: Transforming Data Engineering at Deloitte Digital with Raviteja Tholupunoori From ETL to Airflow: Transforming Data Engineering at Deloitte Digital with Raviteja Tholupunoori Data orchestration at scale presents unique challenges, especially when aiming for flexibility and efficiency across cloud environments. Choosing the right tools and frameworks can make all the difference. 

In this episode, Raviteja Tholupunoori, Senior Engineer at Deloitte Digital, joins us to explore how Airflow enhances orchestration, scalability and cost efficiency in enterprise data workflows.


Key Takeaways:


(01:45) Early challenges in data orchestration before implementing Airflow.

(02:42) Comparing Airflow with ETL tools like Talend and why flexibility matters.

(04:24) The role of Airflow in enabling cloud-agnostic data processing.

(05:45) Key lessons from managing dynamic DAGs at scale.

(13:15) How hybrid executors improve performance and efficiency.

(14:13) Best practices for testing and monitoring workflows with Airflow.

(15:13) The importance of mocking mechanisms when testing DAGs.

(17:57) How Prometheus, Grafana and Loki support Airflow monitoring.

(22:03) Cost considerations when running Airflow on self-managed infrastructure.

(23:14) Airflow’s latest features, including hybrid executors and dark mode.


Resources Mentioned:


Raviteja Tholupunoori

https://www.linkedin.com/in/raviteja0096/?originalSubdomain=in


Deloitte Digital

https://www.linkedin.com/company/deloitte-digital/


Apache Airflow

https://airflow.apache.org/


Grafana

https://grafana.com/solutions/apache-airflow/monitor/


Astronomer Presents: Exploring Apache Airflow® 3 Roadshows

https://www.astronomer.io/events/roadshow/


https://www.astronomer.io/events/roadshow/london/


https://www.astronomer.io/events/roadshow/new-york/


https://www.astronomer.io/events/roadshow/sydney/


https://www.astronomer.io/events/roadshow/san-francisco/


https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Data orchestration at scale presents unique challenges, especially when aiming for flexibility and efficiency across cloud environments. Choosing the right tools and frameworks can make all the difference. 

In this episode, Raviteja Tholupunoori, Senior Engineer at Deloitte Digital, joins us to explore how Airflow enhances orchestration, scalability and cost efficiency in enterprise data workflows.


Key Takeaways:


(01:45) Early challenges in data orchestration before implementing Airflow.

(02:42) Comparing Airflow with ETL tools like Talend and why flexibility matters.

(04:24) The role of Airflow in enabling cloud-agnostic data processing.

(05:45) Key lessons from managing dynamic DAGs at scale.

(13:15) How hybrid executors improve performance and efficiency.

(14:13) Best practices for testing and monitoring workflows with Airflow.

(15:13) The importance of mocking mechanisms when testing DAGs.

(17:57) How Prometheus, Grafana and Loki support Airflow monitoring.

(22:03) Cost considerations when running Airflow on self-managed infrastructure.

(23:14) Airflow’s latest features, including hybrid executors and dark mode.


Resources Mentioned:


Raviteja Tholupunoori

https://www.linkedin.com/in/raviteja0096/?originalSubdomain=in


Deloitte Digital

https://www.linkedin.com/company/deloitte-digital/


Apache Airflow

https://airflow.apache.org/


Grafana

https://grafana.com/solutions/apache-airflow/monitor/


Astronomer Presents: Exploring Apache Airflow® 3 Roadshows

https://www.astronomer.io/events/roadshow/


https://www.astronomer.io/events/roadshow/london/


https://www.astronomer.io/events/roadshow/new-york/


https://www.astronomer.io/events/roadshow/sydney/


https://www.astronomer.io/events/roadshow/san-francisco/


https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) abcecdbb-2db1-4db2-b800-5b68c33da9df Thu, 10 Apr 2025 00:00:00 -0400 1662 1 35 false A Deep Dive Into the 2025 State of Airflow Survey Results with Tamara Fingerlin of Astronomer A Deep Dive Into the 2025 State of Airflow Survey Results with Tamara Fingerlin of Astronomer The 2025 State of Airflow report sheds light on how global users are adopting, evolving and innovating with Apache Airflow. With over 5,000 responses from 116 countries, the survey reveals critical insights into Airflows’ role in business operations, new use cases and what’s ahead for the community.


In this episode, Tamara Fingerlin, Developer Advocate at Astronomer, walks us through her process of analyzing survey data, key trends from the report and what to expect from Airflow 3.0.


Key Takeaways:


(02:14) The State of Airflow report combines anonymized telemetry and survey results.

(03:25) The survey received thousands of responses from many countries, showcasing global reach.

(04:49) The survey process involves multiple steps, from question selection to report creation.

(09:00) Many users expect to increase Airflow usage for revenue-generating or external use cases.

(11:04) Experienced users tend to utilize Airflow more for advanced use cases like MLOps.

(15:13) UI improvements offer enhanced navigation and error visibility.

(18:15) Architectural changes enable new capabilities like remote execution and language support.

(19:40) Long-requested features will be available in the new major release.

(21:00) Future aspirations include integrating data visualization capabilities into the UI.


Resources Mentioned:


Tamara Fingerlin

https://www.linkedin.com/in/tamara-janina-fingerlin/


Astronomer | LinkedIn

https://www.linkedin.com/company/astronomer/


Astronomer | Website

https://www.astronomer.io


Apache Airflow

https://airflow.apache.org/


2025 State of Airflow Webinar

https://www.astronomer.io/airflow/state-of-airflow/


Airflow Slack

https://apache-airflow-slack.herokuapp.com/


Astronomer Presents: Exploring Apache Airflow® 3 Roadshows

https://www.astronomer.io/events/roadshow/


https://www.astronomer.io/events/roadshow/london/


https://www.astronomer.io/events/roadshow/new-york/


https://www.astronomer.io/events/roadshow/sydney/


https://www.astronomer.io/events/roadshow/san-francisco/


https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> The 2025 State of Airflow report sheds light on how global users are adopting, evolving and innovating with Apache Airflow. With over 5,000 responses from 116 countries, the survey reveals critical insights into Airflows’ role in business operations, new use cases and what’s ahead for the community.


In this episode, Tamara Fingerlin, Developer Advocate at Astronomer, walks us through her process of analyzing survey data, key trends from the report and what to expect from Airflow 3.0.


Key Takeaways:


(02:14) The State of Airflow report combines anonymized telemetry and survey results.

(03:25) The survey received thousands of responses from many countries, showcasing global reach.

(04:49) The survey process involves multiple steps, from question selection to report creation.

(09:00) Many users expect to increase Airflow usage for revenue-generating or external use cases.

(11:04) Experienced users tend to utilize Airflow more for advanced use cases like MLOps.

(15:13) UI improvements offer enhanced navigation and error visibility.

(18:15) Architectural changes enable new capabilities like remote execution and language support.

(19:40) Long-requested features will be available in the new major release.

(21:00) Future aspirations include integrating data visualization capabilities into the UI.


Resources Mentioned:


Tamara Fingerlin

https://www.linkedin.com/in/tamara-janina-fingerlin/


Astronomer | LinkedIn

https://www.linkedin.com/company/astronomer/


Astronomer | Website

https://www.astronomer.io


Apache Airflow

https://airflow.apache.org/


2025 State of Airflow Webinar

https://www.astronomer.io/airflow/state-of-airflow/


Airflow Slack

https://apache-airflow-slack.herokuapp.com/


Astronomer Presents: Exploring Apache Airflow® 3 Roadshows

https://www.astronomer.io/events/roadshow/


https://www.astronomer.io/events/roadshow/london/


https://www.astronomer.io/events/roadshow/new-york/


https://www.astronomer.io/events/roadshow/sydney/


https://www.astronomer.io/events/roadshow/san-francisco/


https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 1798eebf-31fd-48e2-9436-02099a76742a Thu, 03 Apr 2025 01:24:00 -0400 1406 1 34 false The Software Risk That Affects Everyone and How To Address It with Michael Winser and Jarek Potiuk The Software Risk That Affects Everyone and How To Address It with Michael Winser and Jarek Potiuk The security of open-source software is a growing concern, especially as dependencies and regulations become more complex, making it essential to understand how to manage software supply chains effectively. 

In this episode, we sit down with Michael Winser, Co-Founder at Alpha-Omega and Security Strategy Ambassador at Eclipse Foundation, and Jarek Potiuk, Member of the Security Committee at the Apache Software Foundation, to discuss the challenges of securing Airflow’s dependencies, the evolving landscape of open-source security and how contributors can help strengthen the ecosystem.

 

Key Takeaways:


(02:43) Jarek quit his full-time engineer position and uses Airflow as a freelancer. 

(04:32) Michael finds happiness in having meaningful work with open-source security.

(07:01) Software supply chain security focuses on correctness, integrity and availability.

(08:44) Airflow’s 790 dependencies present a unique security challenge.

(09:43) Airflow’s security team has significantly improved its vulnerability response.

(10:22) The transition to Airflow 3 emphasizes enterprise security readiness.

(16:20) The ‘Three Fs’ approach: fix it, fork it, or forget it.

(18:45) Dependency health is often more critical than fixing known vulnerabilities.

(23:32) The ‘Three Fs’ in action. 

(26:26) Open-source contributors play a key role in supply chain security.



Resources Mentioned:


Michael Winser -

https://www.linkedin.com/in/michaelw/


Jarek Potiuk -

https://www.linkedin.com/in/jarekpotiuk/


Apache Airflow -

https://airflow.apache.org/


Apache Software Foundation | LinkedIn -

https://www.linkedin.com/company/the-apache-software-foundation/


Apache Software Foundation | Website -

https://www.apache.org/


Eclipse Foundation | LinkedIn -

https://www.linkedin.com/company/eclipse-foundation/


Eclipse Foundation | Website -

https://www.eclipse.org/org/foundation/


OpenSSF Working Groups -

https://openssf.org/community/openssf-working-groups/


Astronomer Roadshow: Exploring Apache Airflow 3 | London

https://www.astronomer.io/events/roadshow/london/


Astronomer Roadshow: Exploring Apache Airflow 3 | New York

https://www.astronomer.io/events/roadshow/new-york/


Astronomer Roadshow: Exploring Apache Airflow 3 | Sydney

https://www.astronomer.io/events/roadshow/sydney/


Astronomer Roadshow: Exploring Apache Airflow 3 | San Francisco

https://www.astronomer.io/events/roadshow/san-francisco/


Astronomer Roadshow: Exploring Apache Airflow 3 | Chicago

https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> The security of open-source software is a growing concern, especially as dependencies and regulations become more complex, making it essential to understand how to manage software supply chains effectively. 

In this episode, we sit down with Michael Winser, Co-Founder at Alpha-Omega and Security Strategy Ambassador at Eclipse Foundation, and Jarek Potiuk, Member of the Security Committee at the Apache Software Foundation, to discuss the challenges of securing Airflow’s dependencies, the evolving landscape of open-source security and how contributors can help strengthen the ecosystem.

 

Key Takeaways:


(02:43) Jarek quit his full-time engineer position and uses Airflow as a freelancer. 

(04:32) Michael finds happiness in having meaningful work with open-source security.

(07:01) Software supply chain security focuses on correctness, integrity and availability.

(08:44) Airflow’s 790 dependencies present a unique security challenge.

(09:43) Airflow’s security team has significantly improved its vulnerability response.

(10:22) The transition to Airflow 3 emphasizes enterprise security readiness.

(16:20) The ‘Three Fs’ approach: fix it, fork it, or forget it.

(18:45) Dependency health is often more critical than fixing known vulnerabilities.

(23:32) The ‘Three Fs’ in action. 

(26:26) Open-source contributors play a key role in supply chain security.



Resources Mentioned:


Michael Winser -

https://www.linkedin.com/in/michaelw/


Jarek Potiuk -

https://www.linkedin.com/in/jarekpotiuk/


Apache Airflow -

https://airflow.apache.org/


Apache Software Foundation | LinkedIn -

https://www.linkedin.com/company/the-apache-software-foundation/


Apache Software Foundation | Website -

https://www.apache.org/


Eclipse Foundation | LinkedIn -

https://www.linkedin.com/company/eclipse-foundation/


Eclipse Foundation | Website -

https://www.eclipse.org/org/foundation/


OpenSSF Working Groups -

https://openssf.org/community/openssf-working-groups/


Astronomer Roadshow: Exploring Apache Airflow 3 | London

https://www.astronomer.io/events/roadshow/london/


Astronomer Roadshow: Exploring Apache Airflow 3 | New York

https://www.astronomer.io/events/roadshow/new-york/


Astronomer Roadshow: Exploring Apache Airflow 3 | Sydney

https://www.astronomer.io/events/roadshow/sydney/


Astronomer Roadshow: Exploring Apache Airflow 3 | San Francisco

https://www.astronomer.io/events/roadshow/san-francisco/


Astronomer Roadshow: Exploring Apache Airflow 3 | Chicago

https://www.astronomer.io/events/roadshow/chicago/





Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 0e220ca3-25da-4723-9adc-6ffb3ef49521 Thu, 20 Mar 2025 00:37:00 -0400 1707 1 32 false Building Scalable ML Infrastructure at Outerbounds with Savin Goyal Building Scalable ML Infrastructure at Outerbounds with Savin Goyal Machine learning is changing fast, and companies need better tools to handle AI workloads. The right infrastructure helps data scientists focus on solving problems instead of managing complex systems. In this episode, we talk with Savin Goyal, Co-Founder and CTO at Outerbounds, about building ML infrastructure, how orchestration makes workflows easier and how Metaflow and Airflow work together to simplify data science. 


Key Takeaways:


(02:02) Savin spent years building AI and ML infrastructure, including at Netflix.

(04:05) ML engineering was not a defined role a decade ago.

(08:17) Modernizing AI and ML requires balancing new tools with existing strengths.

(10:28) ML workloads can be long-running or require heavy computation.

(15:29) Different teams at Netflix used multiple orchestration systems for specific needs.

(20:10) Stable APIs prevent rework and keep projects moving.

(21:07) Metaflow simplifies ML workflows by optimizing data and compute interactions.

(25:53) Limited local computing power makes running ML workloads challenging.

(27:43) Airflow UI monitors pipelines, while Metaflow UI gives ML insights.

(33:13) The most successful data professionals focus on business impact, not just technology.



Resources Mentioned:


Savin Goyal -

https://www.linkedin.com/in/savingoyal/


Outerbounds -

https://www.linkedin.com/company/outerbounds/


Apache Airflow -

https://airflow.apache.org/


Metaflow -

https://metaflow.org/


Netflix’s Maestro Orchestration System -

https://netflixtechblog.com/maestro-netflixs-workflow-orchestrator-ee13a06f9c78?gi=8e6a067a92e9#:~:text=Maestro%20is%20a%20fully%20managed,data%20between%20different%20storages%2C%20etc.


TensorFlow -

https://www.tensorflow.org/


PyTorch -

https://pytorch.org/





Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Machine learning is changing fast, and companies need better tools to handle AI workloads. The right infrastructure helps data scientists focus on solving problems instead of managing complex systems. In this episode, we talk with Savin Goyal, Co-Founder and CTO at Outerbounds, about building ML infrastructure, how orchestration makes workflows easier and how Metaflow and Airflow work together to simplify data science. 


Key Takeaways:


(02:02) Savin spent years building AI and ML infrastructure, including at Netflix.

(04:05) ML engineering was not a defined role a decade ago.

(08:17) Modernizing AI and ML requires balancing new tools with existing strengths.

(10:28) ML workloads can be long-running or require heavy computation.

(15:29) Different teams at Netflix used multiple orchestration systems for specific needs.

(20:10) Stable APIs prevent rework and keep projects moving.

(21:07) Metaflow simplifies ML workflows by optimizing data and compute interactions.

(25:53) Limited local computing power makes running ML workloads challenging.

(27:43) Airflow UI monitors pipelines, while Metaflow UI gives ML insights.

(33:13) The most successful data professionals focus on business impact, not just technology.



Resources Mentioned:


Savin Goyal -

https://www.linkedin.com/in/savingoyal/


Outerbounds -

https://www.linkedin.com/company/outerbounds/


Apache Airflow -

https://airflow.apache.org/


Metaflow -

https://metaflow.org/


Netflix’s Maestro Orchestration System -

https://netflixtechblog.com/maestro-netflixs-workflow-orchestrator-ee13a06f9c78?gi=8e6a067a92e9#:~:text=Maestro%20is%20a%20fully%20managed,data%20between%20different%20storages%2C%20etc.


TensorFlow -

https://www.tensorflow.org/


PyTorch -

https://pytorch.org/





Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) d817db27-4730-453f-92a1-5eaad6c9f72d Thu, 13 Mar 2025 00:37:34 -0400 2206 1 31 false Customizing Airflow for Complex Data Environments at Stripe with Nick Bilozerov and Sharadh Krishnamurthy Customizing Airflow for Complex Data Environments at Stripe with Nick Bilozerov and Sharadh Krishnamurthy Keeping data pipelines reliable at scale requires more than just the right tools — it demands constant innovation. In this episode, Nick Bilozerov, Senior Data Engineer at Stripe, and Sharadh Krishnamurthy, Engineering Manager at Stripe, discuss how Stripe customizes Airflow for its needs, the evolution of its data orchestration framework and the transition to Airflow 2. They also share insights on scaling data workflows while maintaining performance, reliability and developer experience. 


Key Takeaways:



(02:04) Stripe’s mission is to grow the GDP of the internet by supporting businesses with payments and data.

(05:08) 80% of Stripe engineers use data orchestration, making scalability critical.

(06:06) Airflow powers business reports, regulatory needs and ML workflows.

(08:02) Custom task frameworks improve dependencies and validation.

(08:50) "User scope mode" enables local testing without production impact.

(10:39) Migrating to Airflow 2 improves isolation, safety and scalability.

(16:40) Monolithic DAGs caused database issues, prompting a service-based shift.

(19:24) Frequent Airflow upgrades ensure stability and access to new features.

(21:38) DAG versioning and backfill improvements enhance developer experience.

(23:38) Greater UI customization would offer more flexibility.



Resources Mentioned:


Nick Bilozerov -

https://www.linkedin.com/in/nick-bilozerov/


Sharadh Krishnamurthy -

https://www.linkedin.com/in/sharadhk/


Apache Airflow -

https://airflow.apache.org/


Stripe | LinkedIn -

https://www.linkedin.com/company/stripe/


Stripe | Website -

https://stripe.com/




Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> Keeping data pipelines reliable at scale requires more than just the right tools — it demands constant innovation. In this episode, Nick Bilozerov, Senior Data Engineer at Stripe, and Sharadh Krishnamurthy, Engineering Manager at Stripe, discuss how Stripe customizes Airflow for its needs, the evolution of its data orchestration framework and the transition to Airflow 2. They also share insights on scaling data workflows while maintaining performance, reliability and developer experience. 


Key Takeaways:



(02:04) Stripe’s mission is to grow the GDP of the internet by supporting businesses with payments and data.

(05:08) 80% of Stripe engineers use data orchestration, making scalability critical.

(06:06) Airflow powers business reports, regulatory needs and ML workflows.

(08:02) Custom task frameworks improve dependencies and validation.

(08:50) "User scope mode" enables local testing without production impact.

(10:39) Migrating to Airflow 2 improves isolation, safety and scalability.

(16:40) Monolithic DAGs caused database issues, prompting a service-based shift.

(19:24) Frequent Airflow upgrades ensure stability and access to new features.

(21:38) DAG versioning and backfill improvements enhance developer experience.

(23:38) Greater UI customization would offer more flexibility.



Resources Mentioned:


Nick Bilozerov -

https://www.linkedin.com/in/nick-bilozerov/


Sharadh Krishnamurthy -

https://www.linkedin.com/in/sharadhk/


Apache Airflow -

https://airflow.apache.org/


Stripe | LinkedIn -

https://www.linkedin.com/company/stripe/


Stripe | Website -

https://stripe.com/




Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 42bfe9b3-f256-4f08-a4ad-92efcc0bea64 Thu, 06 Mar 2025 02:20:37 -0500 1660 1 30 false Harnessing Airflow for Data-Driven Policy Research at CSET with Jennifer Melot Harnessing Airflow for Data-Driven Policy Research at CSET with Jennifer Melot Turning complex datasets into meaningful analysis requires robust data infrastructure and seamless orchestration. In this episode, we’re joined by Jennifer Melot, Technical Lead at the Center for Security and Emerging Technology (CSET) at Georgetown University, to explore how Airflow powers data-driven insights in technology policy research. Jennifer shares how her team automates workflows to support analysts in navigating complex datasets. 


Key Takeaways:



(02:04) CSET provides data-driven analysis to inform government decision-makers.

(03:54) ETL pipelines merge multiple data sources for more comprehensive insights.

(04:20) Airflow is central to automating and streamlining large-scale data ingestion.

(05:11) Larger-scale databases create challenges that require scalable solutions.

(07:20) Dynamic DAG generation simplifies Airflow adoption for non-engineers.

(12:13) DAG Factory and dynamic task mapping can improve workflow efficiency.

(15:46) Tracking data lineage helps teams understand dependencies across DAGs.

(16:14) New Airflow features enhance visibility and debugging for complex pipelines.


Resources Mentioned:


Jennifer Melot -

https://www.linkedin.com/in/jennifer-melot-aa710144/


Center for Security and Emerging Technology (CSET) -

https://www.linkedin.com/company/georgetown-cset/


Apache Airflow -

https://airflow.apache.org/


Zenodo -

https://zenodo.org/


OpenLineage -

https://openlineage.io/


Cloud Dataplex -

https://cloud.google.com/dataplex




Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> Turning complex datasets into meaningful analysis requires robust data infrastructure and seamless orchestration. In this episode, we’re joined by Jennifer Melot, Technical Lead at the Center for Security and Emerging Technology (CSET) at Georgetown University, to explore how Airflow powers data-driven insights in technology policy research. Jennifer shares how her team automates workflows to support analysts in navigating complex datasets. 


Key Takeaways:



(02:04) CSET provides data-driven analysis to inform government decision-makers.

(03:54) ETL pipelines merge multiple data sources for more comprehensive insights.

(04:20) Airflow is central to automating and streamlining large-scale data ingestion.

(05:11) Larger-scale databases create challenges that require scalable solutions.

(07:20) Dynamic DAG generation simplifies Airflow adoption for non-engineers.

(12:13) DAG Factory and dynamic task mapping can improve workflow efficiency.

(15:46) Tracking data lineage helps teams understand dependencies across DAGs.

(16:14) New Airflow features enhance visibility and debugging for complex pipelines.


Resources Mentioned:


Jennifer Melot -

https://www.linkedin.com/in/jennifer-melot-aa710144/


Center for Security and Emerging Technology (CSET) -

https://www.linkedin.com/company/georgetown-cset/


Apache Airflow -

https://airflow.apache.org/


Zenodo -

https://zenodo.org/


OpenLineage -

https://openlineage.io/


Cloud Dataplex -

https://cloud.google.com/dataplex




Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 22aa5591-5434-40ab-8fee-738a1a773500 Thu, 27 Feb 2025 13:15:00 -0500 1074 1 29 false Hybrid Testing Solutions for Autonomous Driving at Bosch with Jens Scheffler and Christian Schilling Hybrid Testing Solutions for Autonomous Driving at Bosch with Jens Scheffler and Christian Schilling Testing autonomous vehicles demands precision, scalability and powerful orchestration tools — enter Apache Airflow, a key component of Bosch’s cutting-edge testing framework. In this episode, we sit down with Jens Scheffler, Test Execution Cluster Technical Architect, and Christian Schilling, Product Owner Open Loop Testing Automated Driving, both at Bosch, to explore how Bosch harnesses Airflow to streamline complex testing scenarios. They share insights on scaling workflows, integrating hybrid infrastructures and ensuring vehicle safety through rigorous automated testing.


Key Takeaways:


(01:35) Airflow orchestrates millions of test hours for autonomous systems.

(03:15) Jens scales distributed systems with Kubernetes for job orchestration.

(06:02) Airflow runs hundreds of tests simultaneously.

(06:44) Virtual testing reduces costs and on-road trials.

(12:19) Unified APIs and GUIs streamline operations.

(15:05) Self-service setups empower Bosch teams.

(18:00) Physical hardware integration ensures real-world timing.

(20:30) Dynamic task mapping scales workflows efficiently.

(25:22) Open-source contributions improve stability.

(31:06) Edge and Celery executors power Bosch's hybrid scheduling.



Resources Mentioned:


Jens Scheffler -

https://www.linkedin.com/in/jens-scheffler/


Christian Schilling -

https://www.linkedin.com/in/christian-schilling-a5078831a/


Bosch -

https://www.linkedin.com/company/bosch/


Apache Airflow -

https://airflow.apache.org/


Kubernetes -

https://kubernetes.io


GitHub -

https://github.com


Edge Executor -

https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html




Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Testing autonomous vehicles demands precision, scalability and powerful orchestration tools — enter Apache Airflow, a key component of Bosch’s cutting-edge testing framework. In this episode, we sit down with Jens Scheffler, Test Execution Cluster Technical Architect, and Christian Schilling, Product Owner Open Loop Testing Automated Driving, both at Bosch, to explore how Bosch harnesses Airflow to streamline complex testing scenarios. They share insights on scaling workflows, integrating hybrid infrastructures and ensuring vehicle safety through rigorous automated testing.


Key Takeaways:


(01:35) Airflow orchestrates millions of test hours for autonomous systems.

(03:15) Jens scales distributed systems with Kubernetes for job orchestration.

(06:02) Airflow runs hundreds of tests simultaneously.

(06:44) Virtual testing reduces costs and on-road trials.

(12:19) Unified APIs and GUIs streamline operations.

(15:05) Self-service setups empower Bosch teams.

(18:00) Physical hardware integration ensures real-world timing.

(20:30) Dynamic task mapping scales workflows efficiently.

(25:22) Open-source contributions improve stability.

(31:06) Edge and Celery executors power Bosch's hybrid scheduling.



Resources Mentioned:


Jens Scheffler -

https://www.linkedin.com/in/jens-scheffler/


Christian Schilling -

https://www.linkedin.com/in/christian-schilling-a5078831a/


Bosch -

https://www.linkedin.com/company/bosch/


Apache Airflow -

https://airflow.apache.org/


Kubernetes -

https://kubernetes.io


GitHub -

https://github.com


Edge Executor -

https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html




Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) b37f3abe-5376-41a7-9481-7fcbdb361bd4 Thu, 13 Feb 2025 00:10:00 -0500 2025 1 27 false Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer Scaling a data orchestration platform to manage thousands of tasks daily demands innovative solutions and strategic problem-solving. In this episode, we explore the complexities of scaling Airflow and the challenges of orchestrating thousands of tasks in dynamic data environments. Jonathan Rainer, Former Platform Engineer at Monzo Bank, joins us to share his journey optimizing data pipelines, overcoming UI limitations and ensuring DAG consistency in high-stakes scenarios. 


Key Takeaways:

(03:11) Using Airflow to schedule computation in BigQuery.

(07:02) How DAGs with 8,000+ tasks were managed nightly.

(08:18) Ensuring accuracy in regulatory reporting for banking.

(11:35) Handling task inconsistency and DAG failures with automation.

(16:09) Building a service to resolve DAG consistency issues in Airflow.

(25:05) Challenges with scaling the Airflow UI for thousands of tasks.

(27:03) The role of upstream and downstream task management in Airflow.

(37:33) The importance of operational metrics for monitoring Airflow health.

(39:19) Balancing new tools with root cause analysis to address scaling issues.

(41:35) Why scaling solutions require both technical and leadership buy-in



Resources Mentioned:


Jonathan Rainer -

https://www.linkedin.com/in/jonathan-rainer/


Monzo Bank -

https://www.linkedin.com/company/monzo-bank/


Apache Airflow -

https://airflow.apache.org/


BigQuery -

https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/bigquery.html


Kubernetes -

https://kubernetes.io/




Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> Scaling a data orchestration platform to manage thousands of tasks daily demands innovative solutions and strategic problem-solving. In this episode, we explore the complexities of scaling Airflow and the challenges of orchestrating thousands of tasks in dynamic data environments. Jonathan Rainer, Former Platform Engineer at Monzo Bank, joins us to share his journey optimizing data pipelines, overcoming UI limitations and ensuring DAG consistency in high-stakes scenarios. 


Key Takeaways:

(03:11) Using Airflow to schedule computation in BigQuery.

(07:02) How DAGs with 8,000+ tasks were managed nightly.

(08:18) Ensuring accuracy in regulatory reporting for banking.

(11:35) Handling task inconsistency and DAG failures with automation.

(16:09) Building a service to resolve DAG consistency issues in Airflow.

(25:05) Challenges with scaling the Airflow UI for thousands of tasks.

(27:03) The role of upstream and downstream task management in Airflow.

(37:33) The importance of operational metrics for monitoring Airflow health.

(39:19) Balancing new tools with root cause analysis to address scaling issues.

(41:35) Why scaling solutions require both technical and leadership buy-in



Resources Mentioned:


Jonathan Rainer -

https://www.linkedin.com/in/jonathan-rainer/


Monzo Bank -

https://www.linkedin.com/company/monzo-bank/


Apache Airflow -

https://airflow.apache.org/


BigQuery -

https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/bigquery.html


Kubernetes -

https://kubernetes.io/




Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) b078bf1e-b83f-446a-b16d-926628ba0cdd Thu, 06 Feb 2025 22:09:00 -0500 2619 1 26 false Orchestrating Analytics and AI Workflows at Telia with Arjun Anandkumar Orchestrating Analytics and AI Workflows at Telia with Arjun Anandkumar The future of data engineering lies in seamless orchestration and automation. In this episode, Arjun Anandkumar, Data Engineer at Telia, shares how his team uses Airflow to drive analytics and AI workflows. He highlights the challenges of scaling data platforms and how adopting best practices can simplify complex processes for teams across the organization. Arjun also discusses the transformative role of tools like Cosmos and Terraform in enhancing efficiency and collaboration. 


Key Takeaways:


(02:16) Telia operates across the Nordics and Baltics, focusing on telecom and energy services.

(03:45) Airflow runs dbt models seamlessly with Cosmos on AWS MWAA.

(05:47) Cosmos improves visibility and orchestration in Airflow.

(07:00) Medallion Architecture organizes data into bronze, silver and gold layers.

(08:34) Task group challenges highlight the need for adaptable workflows.

(15:04) Scaling managed services requires trial, error and tailored tweaks.

(19:46) Terraform scales infrastructure, while YAML templates manage DAGs efficiently.

(20:00) Templated DAGs and robust testing enhance platform management.

(24:15) Open-source resources drive innovation in Airflow practices.


Resources Mentioned:


Arjun Anandkumar -

https://www.linkedin.com/in/arjunanand1/?originalSubdomain=dk


Telia -

https://www.linkedin.com/company/teliacompany/


Apache Airflow -

https://airflow.apache.org/


Cosmos by Astronomer -

https://www.astronomer.io/cosmos/


Terraform -

https://www.terraform.io/


Medallion Architecture by Databricks -

https://www.databricks.com/glossary/medallion-architecture





Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> The future of data engineering lies in seamless orchestration and automation. In this episode, Arjun Anandkumar, Data Engineer at Telia, shares how his team uses Airflow to drive analytics and AI workflows. He highlights the challenges of scaling data platforms and how adopting best practices can simplify complex processes for teams across the organization. Arjun also discusses the transformative role of tools like Cosmos and Terraform in enhancing efficiency and collaboration. 


Key Takeaways:


(02:16) Telia operates across the Nordics and Baltics, focusing on telecom and energy services.

(03:45) Airflow runs dbt models seamlessly with Cosmos on AWS MWAA.

(05:47) Cosmos improves visibility and orchestration in Airflow.

(07:00) Medallion Architecture organizes data into bronze, silver and gold layers.

(08:34) Task group challenges highlight the need for adaptable workflows.

(15:04) Scaling managed services requires trial, error and tailored tweaks.

(19:46) Terraform scales infrastructure, while YAML templates manage DAGs efficiently.

(20:00) Templated DAGs and robust testing enhance platform management.

(24:15) Open-source resources drive innovation in Airflow practices.


Resources Mentioned:


Arjun Anandkumar -

https://www.linkedin.com/in/arjunanand1/?originalSubdomain=dk


Telia -

https://www.linkedin.com/company/teliacompany/


Apache Airflow -

https://airflow.apache.org/


Cosmos by Astronomer -

https://www.astronomer.io/cosmos/


Terraform -

https://www.terraform.io/


Medallion Architecture by Databricks -

https://www.databricks.com/glossary/medallion-architecture





Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 7f2c816f-d12e-4539-a733-0133824a2844 Thu, 30 Jan 2025 00:00:00 -0500 1560 1 25 false The Role of Airflow in Finance Transformation at Etraveli Group with Mihir Samant The Role of Airflow in Finance Transformation at Etraveli Group with Mihir Samant Transforming bottlenecked finance processes into streamlined, automated systems requires the right tools and a forward-thinking approach. In this episode, Mihir Samant, Senior Data Analyst at Etraveli Group, joins us to share how his team leverages Airflow to revolutionize finance automation. With extensive experience in data workflows and a passion for open-source tools, Mihir provides valuable insights into building efficient, scalable systems. We explore the transformative power of Airflow in automating workflows and enhancing data orchestration within the finance domain. 


Key Takeaways:


(02:14) Etraveli Group specializes in selling affordable flight tickets and ancillary services.

(03:56) Mihir’s finance automation team uses Airflow to tackle month-end bottlenecks.

(06:00) Airflow's flexibility enables end-to-end automation for finance workflows.

(07:00) Open-source Airflow tools offer cost-effective solutions for new teams.

(08:46) Sensors and dynamic DAGs are pivotal features for optimizing tasks.

(13:30) GitSync simplifies development by syncing environments seamlessly.

(16:27) Plans include integrating Databricks for more advanced data handling.

(17:58) Airflow and Databricks offer multiple flexible methods to trigger workflows and execute SQL queries seamlessly.



Resources Mentioned:


Mihir Samant -

https://www.linkedin.com/in/misamant/?originalSubdomain=ca


Etraveli Group -

https://www.linkedin.com/company/etraveli-group/


Apache Airflow -

https://airflow.apache.org/


Docker -

https://www.docker.com/


Databricks -

https://www.databricks.com/





Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> Transforming bottlenecked finance processes into streamlined, automated systems requires the right tools and a forward-thinking approach. In this episode, Mihir Samant, Senior Data Analyst at Etraveli Group, joins us to share how his team leverages Airflow to revolutionize finance automation. With extensive experience in data workflows and a passion for open-source tools, Mihir provides valuable insights into building efficient, scalable systems. We explore the transformative power of Airflow in automating workflows and enhancing data orchestration within the finance domain. 


Key Takeaways:


(02:14) Etraveli Group specializes in selling affordable flight tickets and ancillary services.

(03:56) Mihir’s finance automation team uses Airflow to tackle month-end bottlenecks.

(06:00) Airflow's flexibility enables end-to-end automation for finance workflows.

(07:00) Open-source Airflow tools offer cost-effective solutions for new teams.

(08:46) Sensors and dynamic DAGs are pivotal features for optimizing tasks.

(13:30) GitSync simplifies development by syncing environments seamlessly.

(16:27) Plans include integrating Databricks for more advanced data handling.

(17:58) Airflow and Databricks offer multiple flexible methods to trigger workflows and execute SQL queries seamlessly.



Resources Mentioned:


Mihir Samant -

https://www.linkedin.com/in/misamant/?originalSubdomain=ca


Etraveli Group -

https://www.linkedin.com/company/etraveli-group/


Apache Airflow -

https://airflow.apache.org/


Docker -

https://www.docker.com/


Databricks -

https://www.databricks.com/





Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 2d7d6ae7-5732-4d28-9923-65ec37898aa6 Thu, 23 Jan 2025 05:47:09 -0500 1279 1 24 false Inside Ford’s Data Transformation: Advanced Orchestration Strategies with Vasantha Kosuri-Marshall Inside Ford’s Data Transformation: Advanced Orchestration Strategies with Vasantha Kosuri-Marshall Data engineering is entering a new era, where orchestration and automation are redefining how large-scale projects operate. This episode features Vasantha Kosuri-Marshall, Data and ML Ops Engineer at Ford Motor Company. Vasantha shares her expertise in managing complex data pipelines. She takes us through Ford's transition to cloud platforms, the adoption of Airflow and the intricate challenges of orchestrating data in a diverse environment.



Key Takeaways:


(03:10) Vasantha’s transition to the Advanced Driving Assist Systems team at Ford.

(05:42) Early adoption of Airflow to orchestrate complex data pipelines.

(09:29) Ford's move from on-premise data solutions to Google Cloud Platform.

(12:03) The importance of Airflow's scheduling capabilities for efficient data management.

(16:12) Using Kubernetes to scale Airflow for large-scale data processing.

(19:59) Vasantha’s experience in overcoming challenges with legacy orchestration tools.

(22:22) Integration of data engineering and data science pipelines at Ford.

(28:03) How deferrable operators in Airflow improve performance and save costs.

(32:12) Vasantha’s insights into tuning Airflow properties for thousands of DAGs.

(36:09) The significance of monitoring and observability in managing Airflow instances.



Resources Mentioned:


Vasantha Kosuri-Marshall -

https://www.linkedin.com/in/vasantha-kosuri-marshall-0b0aab188/


Apache Airflow -

https://airflow.apache.org/


Google Cloud Platform (GCP) -

https://cloud.google.com/


Ford Motor Company | LinkedIn -

https://www.linkedin.com/company/ford-motor-company/


Ford Motor Company | Website -

https://www.ford.com/


Astronomer -

https://www.astronomer.io/




Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> Data engineering is entering a new era, where orchestration and automation are redefining how large-scale projects operate. This episode features Vasantha Kosuri-Marshall, Data and ML Ops Engineer at Ford Motor Company. Vasantha shares her expertise in managing complex data pipelines. She takes us through Ford's transition to cloud platforms, the adoption of Airflow and the intricate challenges of orchestrating data in a diverse environment.



Key Takeaways:


(03:10) Vasantha’s transition to the Advanced Driving Assist Systems team at Ford.

(05:42) Early adoption of Airflow to orchestrate complex data pipelines.

(09:29) Ford's move from on-premise data solutions to Google Cloud Platform.

(12:03) The importance of Airflow's scheduling capabilities for efficient data management.

(16:12) Using Kubernetes to scale Airflow for large-scale data processing.

(19:59) Vasantha’s experience in overcoming challenges with legacy orchestration tools.

(22:22) Integration of data engineering and data science pipelines at Ford.

(28:03) How deferrable operators in Airflow improve performance and save costs.

(32:12) Vasantha’s insights into tuning Airflow properties for thousands of DAGs.

(36:09) The significance of monitoring and observability in managing Airflow instances.



Resources Mentioned:


Vasantha Kosuri-Marshall -

https://www.linkedin.com/in/vasantha-kosuri-marshall-0b0aab188/


Apache Airflow -

https://airflow.apache.org/


Google Cloud Platform (GCP) -

https://cloud.google.com/


Ford Motor Company | LinkedIn -

https://www.linkedin.com/company/ford-motor-company/


Ford Motor Company | Website -

https://www.ford.com/


Astronomer -

https://www.astronomer.io/




Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 97312390-383c-4762-8f01-65efd22a9fe3 Thu, 16 Jan 2025 00:15:00 -0500 2334 1 23 false Powering Finance With Advanced Data Solutions at Ramp with Ryan Delgado Powering Finance With Advanced Data Solutions at Ramp with Ryan Delgado Data is the backbone of every modern business, but unlocking its full potential requires the right tools and strategies. In this episode, Ryan Delgado, Director of Engineering at Ramp, joins us to explore how innovative data platforms can transform business operations and fuel growth. He shares insights on integrating Apache Airflow, optimizing data workflows and leveraging analytics to enhance customer experiences.


Key Takeaways:



(01:52) Data is the lifeblood of Ramp, touching every vertical in the company.

(03:18) Ramp’s data platform team enables high-velocity scaling through tailored tools.

(05:27) Airflow powers Ramp’s enterprise data warehouse integrations for advanced analytics.

(07:55) Centralizing data in Snowflake simplifies storage and analytics pipelines.

(12:08) Machine learning models at Ramp integrate seamlessly with Airflow for operational excellence.

(14:11) Leveraging Airflow datasets eliminates inefficiencies in DAG dependencies.

(17:22) Platforms evolve from solving narrow business problems to scaling organizationally.

(18:55) ClickHouse enhances Ramp’s OLAP capabilities with 100x performance improvements.

(19:47) Ramp’s OLAP platform improves performance by reducing joins and leveraging ClickHouse.

(21:46) Ryan envisions a lighter-weight, more Python-native future for Airflow.


Resources Mentioned:


Ryan Delgado -

https://www.linkedin.com/in/ryan-delgado-69544568/


Ramp -

https://www.linkedin.com/company/ramp/


Apache Airflow -

https://airflow.apache.org/


Snowflake -

https://www.snowflake.com/


ClickHouse -

https://clickhouse.com/


dbt -

https://www.getdbt.com/




Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> Data is the backbone of every modern business, but unlocking its full potential requires the right tools and strategies. In this episode, Ryan Delgado, Director of Engineering at Ramp, joins us to explore how innovative data platforms can transform business operations and fuel growth. He shares insights on integrating Apache Airflow, optimizing data workflows and leveraging analytics to enhance customer experiences.


Key Takeaways:



(01:52) Data is the lifeblood of Ramp, touching every vertical in the company.

(03:18) Ramp’s data platform team enables high-velocity scaling through tailored tools.

(05:27) Airflow powers Ramp’s enterprise data warehouse integrations for advanced analytics.

(07:55) Centralizing data in Snowflake simplifies storage and analytics pipelines.

(12:08) Machine learning models at Ramp integrate seamlessly with Airflow for operational excellence.

(14:11) Leveraging Airflow datasets eliminates inefficiencies in DAG dependencies.

(17:22) Platforms evolve from solving narrow business problems to scaling organizationally.

(18:55) ClickHouse enhances Ramp’s OLAP capabilities with 100x performance improvements.

(19:47) Ramp’s OLAP platform improves performance by reducing joins and leveraging ClickHouse.

(21:46) Ryan envisions a lighter-weight, more Python-native future for Airflow.


Resources Mentioned:


Ryan Delgado -

https://www.linkedin.com/in/ryan-delgado-69544568/


Ramp -

https://www.linkedin.com/company/ramp/


Apache Airflow -

https://airflow.apache.org/


Snowflake -

https://www.snowflake.com/


ClickHouse -

https://clickhouse.com/


dbt -

https://www.getdbt.com/




Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 5921f932-6c74-4dcb-9722-e0dc62c08ec6 Fri, 10 Jan 2025 11:15:02 -0500 1475 1 22 false Exploring the Power of Airflow 3 at Astronomer with Amogh Desai Exploring the Power of Airflow 3 at Astronomer with Amogh Desai What does it take to go from fixing a broken link to becoming a committer for one of the world’s leading open-source projects? 


Amogh Desai, Senior Software Engineer at Astronomer, takes us through his journey with Apache Airflow. From small contributions to building meaningful connections in the open-source community, Amogh’s story provides actionable insights for anyone on the cusp of their open-source journey.


Key Takeaways:


(02:09) Building data engineering platforms at Cloudera with Kubernetes.

(04:00) Brainstorming led to contributing to Apache Airflow.

(05:17) Starting small with link fixes, progressing to Breeze development.

(07:00) Becoming a committer for Apache Airflow in September 2023.

(09:51) The steep learning curve for contributing to Airflow.

(16:30) Using GitHub’s “good-first-issue” label to get started.

(18:15) Setting up a development environment with Breeze.

(22:00) Open-source contributions enhance your resume and career.

(24:51) Amogh’s advice: Start small and stay consistent.

(28:12) Engage with the community via Slack, email lists and meetups.


Resources Mentioned:


Amogh Desai -

https://www.linkedin.com/in/amogh-desai-385141157/?originalSubdomain=in%20%20https://www.linkedin.com/company/astronomer/

Astronomer -

https://www.linkedin.com/company/astronomer/

Apache Airflow GitHub Repository -

https://github.com/apache/airflow

Contributors Quick Guide -

https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst

Breeze Development Tool -

https://github.com/apache/airflow/tree/main/dev/breeze


Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> What does it take to go from fixing a broken link to becoming a committer for one of the world’s leading open-source projects? 


Amogh Desai, Senior Software Engineer at Astronomer, takes us through his journey with Apache Airflow. From small contributions to building meaningful connections in the open-source community, Amogh’s story provides actionable insights for anyone on the cusp of their open-source journey.


Key Takeaways:


(02:09) Building data engineering platforms at Cloudera with Kubernetes.

(04:00) Brainstorming led to contributing to Apache Airflow.

(05:17) Starting small with link fixes, progressing to Breeze development.

(07:00) Becoming a committer for Apache Airflow in September 2023.

(09:51) The steep learning curve for contributing to Airflow.

(16:30) Using GitHub’s “good-first-issue” label to get started.

(18:15) Setting up a development environment with Breeze.

(22:00) Open-source contributions enhance your resume and career.

(24:51) Amogh’s advice: Start small and stay consistent.

(28:12) Engage with the community via Slack, email lists and meetups.


Resources Mentioned:


Amogh Desai -

https://www.linkedin.com/in/amogh-desai-385141157/?originalSubdomain=in%20%20https://www.linkedin.com/company/astronomer/

Astronomer -

https://www.linkedin.com/company/astronomer/

Apache Airflow GitHub Repository -

https://github.com/apache/airflow

Contributors Quick Guide -

https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst

Breeze Development Tool -

https://github.com/apache/airflow/tree/main/dev/breeze


Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) cd004074-02f2-4816-b187-596a495ed8c1 Fri, 20 Dec 2024 13:41:00 -0500 1824 1 21 false Using Airflow To Power Machine Learning Pipelines at Optimove with Vasyl Vasyuta Using Airflow To Power Machine Learning Pipelines at Optimove with Vasyl Vasyuta Data orchestration and machine learning are shaping how organizations handle massive datasets and drive customer-focused strategies. Tools like Apache Airflow are central to this transformation. In this episode, Vasyl Vasyuta, R&D Team Leader at Optimove, joins us to discuss how his team leverages Airflow to optimize data processing, orchestrate machine learning models and create personalized customer experiences.


Key Takeaways:



(01:59) Optimove tailors marketing notifications with personalized customer journeys.

(04:25) Airflow orchestrates Snowflake procedures for massive datasets.

(05:11) DAGs manage workflows with branching and replay plugins.

(05:41) The "Joystick" plugin enables seamless data replays.

(09:33) Airflow supports MLOps for customer data grouping.

(11:15) Machine learning predicts customer behavior for better campaigns.

(13:20) Thousands of DAGs run every five minutes for data processing.

(15:36) Custom versioning allows rollbacks and gradual rollouts.

(18:00) Airflow logs enhance operational observability.

(23:00) DAG versioning in Airflow 3.0 could boost efficiency.



Resources Mentioned:


Vasyl Vasyuta -

https://www.linkedin.com/in/vasyl-vasyuta-3270b54a/


Optimove -

https://www.linkedin.com/company/optimove/


Apache Airflow -

https://airflow.apache.org/


Snowflake -

https://www.snowflake.com/


Datadog -

https://www.datadoghq.com/


Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24




Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> Data orchestration and machine learning are shaping how organizations handle massive datasets and drive customer-focused strategies. Tools like Apache Airflow are central to this transformation. In this episode, Vasyl Vasyuta, R&D Team Leader at Optimove, joins us to discuss how his team leverages Airflow to optimize data processing, orchestrate machine learning models and create personalized customer experiences.


Key Takeaways:



(01:59) Optimove tailors marketing notifications with personalized customer journeys.

(04:25) Airflow orchestrates Snowflake procedures for massive datasets.

(05:11) DAGs manage workflows with branching and replay plugins.

(05:41) The "Joystick" plugin enables seamless data replays.

(09:33) Airflow supports MLOps for customer data grouping.

(11:15) Machine learning predicts customer behavior for better campaigns.

(13:20) Thousands of DAGs run every five minutes for data processing.

(15:36) Custom versioning allows rollbacks and gradual rollouts.

(18:00) Airflow logs enhance operational observability.

(23:00) DAG versioning in Airflow 3.0 could boost efficiency.



Resources Mentioned:


Vasyl Vasyuta -

https://www.linkedin.com/in/vasyl-vasyuta-3270b54a/


Optimove -

https://www.linkedin.com/company/optimove/


Apache Airflow -

https://airflow.apache.org/


Snowflake -

https://www.snowflake.com/


Datadog -

https://www.datadoghq.com/


Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24




Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.




#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 5a3328c2-a882-4c84-b0a1-406515373bb3 Thu, 12 Dec 2024 11:28:03 -0500 1451 1 20 false Maximizing Business Impact Through Data at GlossGenius with Katie Bauer Maximizing Business Impact Through Data at GlossGenius with Katie Bauer Bridging the gap between data teams and business priorities is essential for maximizing impact and building value-driven workflows. Katie Bauer, Senior Director of Data at GlossGenius, joins us to share her principles for creating effective, aligned data teams. In this episode, Katie draws from her experience at GlossGenius, Reddit and Twitter to highlight the common pitfalls data teams face and how to overcome them. She offers practical strategies for aligning team efforts with organizational goals and fostering collaboration with stakeholders.


 Key Takeaways:


(02:36) GlossGenius provides an all-in-one platform for beauty professionals.

(03:59) Airflow orchestrates data and MLOps workflows at GlossGenius.

(04:41) Focusing on value helps data teams achieve greater impact.

(06:23) Aligning team priorities with company goals minimizes friction.

(08:44) Building strong stakeholder relationships requires curiosity.

(12:46) Treating roles as flexible fosters team innovation.

(13:21) Adapting to new technologies improves effectiveness.

(18:28) Acting like your time is valuable earns respect.

(23:38) Proactive data initiatives drive strategic value.

(24:20) Usage data offers critical insights into tool effectiveness.


Resources Mentioned:


Katie Bauer -

https://www.linkedin.com/in/mkatiebauer/

GlossGenius -

https://www.linkedin.com/company/glossgenius/

Apache Airflow -

https://airflow.apache.org/

DBT -

https://www.getdbt.com/

Cosmos -

https://cosmos.apache.org/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24


Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Bridging the gap between data teams and business priorities is essential for maximizing impact and building value-driven workflows. Katie Bauer, Senior Director of Data at GlossGenius, joins us to share her principles for creating effective, aligned data teams. In this episode, Katie draws from her experience at GlossGenius, Reddit and Twitter to highlight the common pitfalls data teams face and how to overcome them. She offers practical strategies for aligning team efforts with organizational goals and fostering collaboration with stakeholders.


 Key Takeaways:


(02:36) GlossGenius provides an all-in-one platform for beauty professionals.

(03:59) Airflow orchestrates data and MLOps workflows at GlossGenius.

(04:41) Focusing on value helps data teams achieve greater impact.

(06:23) Aligning team priorities with company goals minimizes friction.

(08:44) Building strong stakeholder relationships requires curiosity.

(12:46) Treating roles as flexible fosters team innovation.

(13:21) Adapting to new technologies improves effectiveness.

(18:28) Acting like your time is valuable earns respect.

(23:38) Proactive data initiatives drive strategic value.

(24:20) Usage data offers critical insights into tool effectiveness.


Resources Mentioned:


Katie Bauer -

https://www.linkedin.com/in/mkatiebauer/

GlossGenius -

https://www.linkedin.com/company/glossgenius/

Apache Airflow -

https://airflow.apache.org/

DBT -

https://www.getdbt.com/

Cosmos -

https://cosmos.apache.org/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24


Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) e8ea0543-ae88-4981-bdd7-6992f4a87401 Thu, 05 Dec 2024 04:19:38 -0500 1549 1 19 false Optimizing Large-Scale Deployments at LinkedIn with Rahul Gade Optimizing Large-Scale Deployments at LinkedIn with Rahul Gade Scaling deployments for a billion users demands innovation, precision and resilience. In this episode, we dive into how LinkedIn optimizes its continuous deployment process using Apache Airflow. Rahul Gade, Staff Software Engineer at LinkedIn, shares his insights on building scalable systems and democratizing deployments for over 10,000 engineers. 

Rahul discusses the challenges of managing large-scale deployments across 6,000 services and how his team leverages Airflow to enhance efficiency, reliability and user accessibility.


Key Takeaways:


(01:36) LinkedIn minimizes human involvement in production to reduce errors.

(02:00) Airflow powers LinkedIn’s Continuous Deployment platform.

(05:43) Continuous deployment adoption grew from 8% to a targeted 80%.

(11:25) Kubernetes ensures scalability and flexibility for deployments.

(12:04) A custom UI offers real-time deployment transparency.

(16:23) No-code YAML workflows simplify deployment tasks.

(17:18) Canaries and metrics ensure safe deployments across fabrics.

(20:45) A gateway service ensures redundancy across Airflow clusters.

(24:22) Abstractions let engineers focus on development, not logistics.

(25:20) Multi-language support in Airflow 3.0 simplifies adoption.


Resources Mentioned:


Rahul Gade -

https://www.linkedin.com/in/rahul-gade-68666818/

LinkedIn -

https://www.linkedin.com/company/linkedin/

Apache Airflow -

https://airflow.apache.org/

Kubernetes -

https://kubernetes.io/

Open Policy Agent (OPA) -

https://www.openpolicyagent.org/

Backstage -

https://backstage.io/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24



Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Scaling deployments for a billion users demands innovation, precision and resilience. In this episode, we dive into how LinkedIn optimizes its continuous deployment process using Apache Airflow. Rahul Gade, Staff Software Engineer at LinkedIn, shares his insights on building scalable systems and democratizing deployments for over 10,000 engineers. 

Rahul discusses the challenges of managing large-scale deployments across 6,000 services and how his team leverages Airflow to enhance efficiency, reliability and user accessibility.


Key Takeaways:


(01:36) LinkedIn minimizes human involvement in production to reduce errors.

(02:00) Airflow powers LinkedIn’s Continuous Deployment platform.

(05:43) Continuous deployment adoption grew from 8% to a targeted 80%.

(11:25) Kubernetes ensures scalability and flexibility for deployments.

(12:04) A custom UI offers real-time deployment transparency.

(16:23) No-code YAML workflows simplify deployment tasks.

(17:18) Canaries and metrics ensure safe deployments across fabrics.

(20:45) A gateway service ensures redundancy across Airflow clusters.

(24:22) Abstractions let engineers focus on development, not logistics.

(25:20) Multi-language support in Airflow 3.0 simplifies adoption.


Resources Mentioned:


Rahul Gade -

https://www.linkedin.com/in/rahul-gade-68666818/

LinkedIn -

https://www.linkedin.com/company/linkedin/

Apache Airflow -

https://airflow.apache.org/

Kubernetes -

https://kubernetes.io/

Open Policy Agent (OPA) -

https://www.openpolicyagent.org/

Backstage -

https://backstage.io/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24



Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 6739664a-6543-4340-9a94-e344d67b5405 Mon, 02 Dec 2024 13:03:19 -0500 1667 1 18 false How Uber Manages 1 Million Daily Tasks Using Airflow, with Shobhit Shah and Sumit Maheshwari How Uber Manages 1 Million Daily Tasks Using Airflow, with Shobhit Shah and Sumit Maheshwari When data orchestration reaches Uber’s scale, innovation becomes a necessity, not a luxury. In this episode, we discuss the innovations behind Uber’s unique Airflow setup. With our guests Shobhit Shah and Sumit Maheshwari, both Staff Software Engineers at Uber, we explore how their team manages one of the largest data workflow systems in the world. Shobhit and Sumit walk us through the evolution of Uber’s Airflow implementation, detailing the custom solutions that support 200,000 daily pipelines. They discuss Uber's approach to tackling complex challenges in data orchestration, disaster recovery and scaling to meet the company’s extensive data needs.


Key Takeaways:

(02:03) Airflow as a service streamlines Uber’s data workflows.

(06:16) Serialization boosts security and reduces errors.

(10:05) Java-based scheduler improves system reliability.

(13:40) Custom recovery model supports emergency pipeline switching.

(15:58) No-code UI allows easy pipeline creation for non-coders.

(18:12) Backfill feature enables historical data processing.

(22:06) Regular updates keep Uber aligned with Airflow advancements.

(26:07) Plans to leverage Airflow’s latest features.



Resources Mentioned:


Shobhit Shah -

https://www.linkedin.com/in/shahshobhit/

Sumit Maheshwar -

https://www.linkedin.com/in/maheshwarisumit/

Uber -

https://www.linkedin.com/company/uber-com/

Apache Airflow -

https://airflow.apache.org/

Airflow Summit -

https://airflowsummit.org/

Uber -

https://www.uber.com/tw/en/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24



Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> When data orchestration reaches Uber’s scale, innovation becomes a necessity, not a luxury. In this episode, we discuss the innovations behind Uber’s unique Airflow setup. With our guests Shobhit Shah and Sumit Maheshwari, both Staff Software Engineers at Uber, we explore how their team manages one of the largest data workflow systems in the world. Shobhit and Sumit walk us through the evolution of Uber’s Airflow implementation, detailing the custom solutions that support 200,000 daily pipelines. They discuss Uber's approach to tackling complex challenges in data orchestration, disaster recovery and scaling to meet the company’s extensive data needs.


Key Takeaways:

(02:03) Airflow as a service streamlines Uber’s data workflows.

(06:16) Serialization boosts security and reduces errors.

(10:05) Java-based scheduler improves system reliability.

(13:40) Custom recovery model supports emergency pipeline switching.

(15:58) No-code UI allows easy pipeline creation for non-coders.

(18:12) Backfill feature enables historical data processing.

(22:06) Regular updates keep Uber aligned with Airflow advancements.

(26:07) Plans to leverage Airflow’s latest features.



Resources Mentioned:


Shobhit Shah -

https://www.linkedin.com/in/shahshobhit/

Sumit Maheshwar -

https://www.linkedin.com/in/maheshwarisumit/

Uber -

https://www.linkedin.com/company/uber-com/

Apache Airflow -

https://airflow.apache.org/

Airflow Summit -

https://airflowsummit.org/

Uber -

https://www.uber.com/tw/en/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24



Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 9ce80b41-338d-4c5e-bcc7-a182b9eb479f Thu, 14 Nov 2024 01:00:00 -0500 1724 1 17 false Building Resilient Data Systems for Modern Enterprises at Astrafy with Andrea Bombino Building Resilient Data Systems for Modern Enterprises at Astrafy with Andrea Bombino Efficient data orchestration is the backbone of modern analytics and AI-driven workflows. Without the right tools, even the best data can fall short of its potential. In this episode, Andrea Bombino, Co-Founder and Head of Analytics Engineering at Astrafy, shares insights into his team’s approach to optimizing data transformation and orchestration using tools like datasets and Pub/Sub to drive real-time processing. Andrea explains how they leverage Apache Airflow and Google Cloud to power dynamic data workflows. 


Key Takeaways:


(01:55) Astrafy helps companies manage data using Google Cloud.

(04:36) Airflow is central to Astrafy’s data engineering efforts.

(07:17) Datasets and Pub/Sub are used for real-time workflows.

(09:59) Pub/Sub links multiple Airflow environments.

(12:40) Datasets eliminate the need for constant monitoring.

(15:22) Airflow updates have improved large-scale data operations.

(18:03) New Airflow API features make dataset updates easier.

(20:45) Real-time orchestration speeds up data processing for clients.

(23:26) Pub/Sub enhances flexibility across cloud environments.

(26:08) Future Airflow features will offer more control over data workflows.



Resources Mentioned:


Andrea Bombino -

https://www.linkedin.com/in/andrea-bombino/

Astrafy -

https://www.linkedin.com/company/astrafy/

Apache Airflow -

https://airflow.apache.org/

Google Cloud -

https://cloud.google.com/

dbt -

https://www.getdbt.com/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24



Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Efficient data orchestration is the backbone of modern analytics and AI-driven workflows. Without the right tools, even the best data can fall short of its potential. In this episode, Andrea Bombino, Co-Founder and Head of Analytics Engineering at Astrafy, shares insights into his team’s approach to optimizing data transformation and orchestration using tools like datasets and Pub/Sub to drive real-time processing. Andrea explains how they leverage Apache Airflow and Google Cloud to power dynamic data workflows. 


Key Takeaways:


(01:55) Astrafy helps companies manage data using Google Cloud.

(04:36) Airflow is central to Astrafy’s data engineering efforts.

(07:17) Datasets and Pub/Sub are used for real-time workflows.

(09:59) Pub/Sub links multiple Airflow environments.

(12:40) Datasets eliminate the need for constant monitoring.

(15:22) Airflow updates have improved large-scale data operations.

(18:03) New Airflow API features make dataset updates easier.

(20:45) Real-time orchestration speeds up data processing for clients.

(23:26) Pub/Sub enhances flexibility across cloud environments.

(26:08) Future Airflow features will offer more control over data workflows.



Resources Mentioned:


Andrea Bombino -

https://www.linkedin.com/in/andrea-bombino/

Astrafy -

https://www.linkedin.com/company/astrafy/

Apache Airflow -

https://airflow.apache.org/

Google Cloud -

https://cloud.google.com/

dbt -

https://www.getdbt.com/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24



Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 141a5166-e67c-4d6f-a98a-6b5f527f0f22 Thu, 07 Nov 2024 03:27:47 -0500 1709 1 16 false Inside Airflow 3: Redefining Data Engineering with Vikram Koka Inside Airflow 3: Redefining Data Engineering with Vikram Koka Data orchestration is evolving faster than ever and Apache Airflow 3 is set to revolutionize how enterprises handle complex workflows. In this episode, we dive into the exciting advancements with Vikram Koka, Chief Strategy Officer at Astronomer and PMC Member at The Apache Software Foundation. Vikram shares his insights on the evolution of Airflow and its pivotal role in shaping modern data-driven workflows, particularly with the upcoming release of Airflow 3.


Key Takeaways:


(02:36) Vikram leads Astronomer’s engineering and open-source teams for Airflow.

(05:26) Airflow enables reliable data ingestion and curation.

(08:17) Enterprises use Airflow for mission-critical data pipelines.

(11:08) Airflow 3 introduces major architectural updates.

(13:58) Multi-cloud and edge deployments are supported in Airflow 3.

(16:49) Event-driven scheduling makes Airflow more dynamic.

(19:40) Tasks in Airflow 3 can run in any language.

(22:30) Multilingual task support is crucial for enterprises.

(25:21) Data assets and event-based integration enhance orchestration.

(28:12) Community feedback plays a vital role in Airflow 3.



Resources Mentioned:


Vikram Koka -

https://www.linkedin.com/in/vikramkoka/

Astronomer -

https://www.linkedin.com/company/astronomer/

The Apache Software Foundation LinkedIn -

https://www.linkedin.com/company/the-apache-software-foundation/

Apache Airflow LinkedIn -

https://www.linkedin.com/company/apache-airflow/

Apache Airflow -

https://airflow.apache.org/

Astronomer -

https://www.astronomer.io/

The Apache Software Foundation -

https://www.apache.org/

Join the Airflow slack and/or Dev list -

https://airflow.apache.org/community/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24



Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Data orchestration is evolving faster than ever and Apache Airflow 3 is set to revolutionize how enterprises handle complex workflows. In this episode, we dive into the exciting advancements with Vikram Koka, Chief Strategy Officer at Astronomer and PMC Member at The Apache Software Foundation. Vikram shares his insights on the evolution of Airflow and its pivotal role in shaping modern data-driven workflows, particularly with the upcoming release of Airflow 3.


Key Takeaways:


(02:36) Vikram leads Astronomer’s engineering and open-source teams for Airflow.

(05:26) Airflow enables reliable data ingestion and curation.

(08:17) Enterprises use Airflow for mission-critical data pipelines.

(11:08) Airflow 3 introduces major architectural updates.

(13:58) Multi-cloud and edge deployments are supported in Airflow 3.

(16:49) Event-driven scheduling makes Airflow more dynamic.

(19:40) Tasks in Airflow 3 can run in any language.

(22:30) Multilingual task support is crucial for enterprises.

(25:21) Data assets and event-based integration enhance orchestration.

(28:12) Community feedback plays a vital role in Airflow 3.



Resources Mentioned:


Vikram Koka -

https://www.linkedin.com/in/vikramkoka/

Astronomer -

https://www.linkedin.com/company/astronomer/

The Apache Software Foundation LinkedIn -

https://www.linkedin.com/company/the-apache-software-foundation/

Apache Airflow LinkedIn -

https://www.linkedin.com/company/apache-airflow/

Apache Airflow -

https://airflow.apache.org/

Astronomer -

https://www.astronomer.io/

The Apache Software Foundation -

https://www.apache.org/

Join the Airflow slack and/or Dev list -

https://airflow.apache.org/community/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24



Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 0c2ed4c0-a230-4e83-9b02-c7cd87dca2b0 Thu, 31 Oct 2024 06:01:00 -0400 1808 1 15 false Building a Data-Driven HR Platform at 15Five with Guy Dassa Building a Data-Driven HR Platform at 15Five with Guy Dassa Data and AI are revolutionizing HR, empowering leaders to measure performance and drive strategic decisions like never before. 

In this episode, we explore the transformation of HR technology with Guy Dassa, Chief Technology Officer at 15Five, as he shares insights into their evolving data platform. Guy discusses how 15Five equips HR leaders with tools to measure and take action on team performance, engagement and retention. He explains their data-driven approach, highlighting how Apache Airflow supports their data ingestion, transformation, and AI integration.


Key Takeaways:


(01:54) 15Five acts as a command center for HR leaders.

(03:40) Tools like performance reviews, engagement surveys, and an insights dashboard guide actionable HR steps.

(05:33) Data visualization, insights, and action recommendations enhance HR effectiveness to improve their people's outcomes.

(07:08) Strict data confidentiality and sanitized AI model training.

(09:21) Airflow is central to data transformation and enrichment.

(11:15) Airflow enrichment DAGs integrate AI models.

(13:33) Integration of Airflow and DBT enables efficient data transformation.

(15:28) Synchronization challenges arise with reverse ETL processes.

(17:10) Future plans include deeper Airflow integration with AI.

(19:31) Emphasizing the need for DAG versioning and improved dependency visibility.



Resources Mentioned:


Guy Dassa -

https://www.linkedin.com/in/guydassa/

15Five -

https://www.linkedin.com/company/15five/

Apache Airflow -

https://airflow.apache.org/

MLflow -

https://mlflow.org/

DBT -

https://www.getdbt.com/

Kubernetes -

https://kubernetes.io/

RedShift -

https://aws.amazon.com/redshift/

15Five -

https://www.15five.com/



Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> Data and AI are revolutionizing HR, empowering leaders to measure performance and drive strategic decisions like never before. 

In this episode, we explore the transformation of HR technology with Guy Dassa, Chief Technology Officer at 15Five, as he shares insights into their evolving data platform. Guy discusses how 15Five equips HR leaders with tools to measure and take action on team performance, engagement and retention. He explains their data-driven approach, highlighting how Apache Airflow supports their data ingestion, transformation, and AI integration.


Key Takeaways:


(01:54) 15Five acts as a command center for HR leaders.

(03:40) Tools like performance reviews, engagement surveys, and an insights dashboard guide actionable HR steps.

(05:33) Data visualization, insights, and action recommendations enhance HR effectiveness to improve their people's outcomes.

(07:08) Strict data confidentiality and sanitized AI model training.

(09:21) Airflow is central to data transformation and enrichment.

(11:15) Airflow enrichment DAGs integrate AI models.

(13:33) Integration of Airflow and DBT enables efficient data transformation.

(15:28) Synchronization challenges arise with reverse ETL processes.

(17:10) Future plans include deeper Airflow integration with AI.

(19:31) Emphasizing the need for DAG versioning and improved dependency visibility.



Resources Mentioned:


Guy Dassa -

https://www.linkedin.com/in/guydassa/

15Five -

https://www.linkedin.com/company/15five/

Apache Airflow -

https://airflow.apache.org/

MLflow -

https://mlflow.org/

DBT -

https://www.getdbt.com/

Kubernetes -

https://kubernetes.io/

RedShift -

https://aws.amazon.com/redshift/

15Five -

https://www.15five.com/



Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.



#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) a0f4a479-f143-4230-9db0-f0f25ed889dd Thu, 24 Oct 2024 06:00:00 -0400 1225 1 14 false The Intersection of AI and Data Management at Dosu with Devin Stein The Intersection of AI and Data Management at Dosu with Devin Stein Unlocking engineering productivity goes beyond coding — it’s about managing knowledge efficiently. In this episode, we explore the innovative ways in which Dosu leverages Airflow for data orchestration and supports the Airflow project. 


Devin Stein, Founder of Dosu, shares his insights on how engineering teams can focus on value-added work by automating knowledge management. Devin dives into Dosu’s purpose, the significance of AI in their product, and why they chose Airflow as the backbone for scheduling and data management. 



Key Takeaways:


(01:33) Dosu's mission to democratize engineering knowledge.

(05:00) AI is central to Dosu's product for structuring engineering knowledge.

(06:23) The importance of maintaining up-to-date data for AI effectiveness.

(07:55) How Airflow supports Dosu’s data ingestion and automation processes.

(08:45) The reasoning behind choosing Airflow over other orchestrators.

(11:00) Airflow enables Dosu to manage both traditional ETL and dynamic workflows.

(13:04) Dosu assists the Airflow project by auto-labeling issues and discussions.

(14:56) Thoughtful collaboration with the Airflow community to introduce AI tools.

(16:37) The potential of Airflow to handle more dynamic, scheduled workflows in the future.

(18:00) Challenges and custom solutions for implementing dynamic workflows in Airflow.



Resources Mentioned:


Apache Airflow - https://airflow.apache.org/

Dosu Website - https://dosu.dev/



Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow #MachineLearning

]]> Unlocking engineering productivity goes beyond coding — it’s about managing knowledge efficiently. In this episode, we explore the innovative ways in which Dosu leverages Airflow for data orchestration and supports the Airflow project. 


Devin Stein, Founder of Dosu, shares his insights on how engineering teams can focus on value-added work by automating knowledge management. Devin dives into Dosu’s purpose, the significance of AI in their product, and why they chose Airflow as the backbone for scheduling and data management. 



Key Takeaways:


(01:33) Dosu's mission to democratize engineering knowledge.

(05:00) AI is central to Dosu's product for structuring engineering knowledge.

(06:23) The importance of maintaining up-to-date data for AI effectiveness.

(07:55) How Airflow supports Dosu’s data ingestion and automation processes.

(08:45) The reasoning behind choosing Airflow over other orchestrators.

(11:00) Airflow enables Dosu to manage both traditional ETL and dynamic workflows.

(13:04) Dosu assists the Airflow project by auto-labeling issues and discussions.

(14:56) Thoughtful collaboration with the Airflow community to introduce AI tools.

(16:37) The potential of Airflow to handle more dynamic, scheduled workflows in the future.

(18:00) Challenges and custom solutions for implementing dynamic workflows in Airflow.



Resources Mentioned:


Apache Airflow - https://airflow.apache.org/

Dosu Website - https://dosu.dev/



Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow #MachineLearning

]]> briana.okyere@astronomer.io (Astronomer) 829c9df0-718a-4a0a-acd3-83d6d9bf903e Fri, 04 Oct 2024 06:53:05 -0400 1218 1 13 false AI-Powered Vehicle Automation at Ford Motor Company with Serjesh Sharma AI-Powered Vehicle Automation at Ford Motor Company with Serjesh Sharma briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/1915402430 Thu, 12 Sep 2024 16:49:35 -0400 1571 false From Task Failures to Operational Excellence at GumGum with Brendan Frick From Task Failures to Operational Excellence at GumGum with Brendan Frick briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/1911310895 Fri, 06 Sep 2024 06:32:20 -0400 1446 false From Sensors to Datasets: Enhancing Airflow at Astronomer with Maggie Stark and Marion Azoulai From Sensors to Datasets: Enhancing Airflow at Astronomer with Maggie Stark and Marion Azoulai briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/1906370054 Thu, 29 Aug 2024 15:44:33 -0400 1345 false Mastering Data Orchestration with Airflow at M Science with Ben Tallman Mastering Data Orchestration with Airflow at M Science with Ben Tallman briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/1904353025 Mon, 26 Aug 2024 16:18:40 -0400 1476 false Welcome to The Data Flowcast Welcome to The Data Flowcast briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/1900165410 Mon, 19 Aug 2024 16:19:41 -0400 121 false Enhancing Business Metrics With Airflow at Artlist with Hannan Kravitz Enhancing Business Metrics With Airflow at Artlist with Hannan Kravitz briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/1897164294 Thu, 15 Aug 2024 10:00:27 -0400 1431 false Cutting-Edge Data Engineering at Teya with Alexandre Magno Lima Martins Cutting-Edge Data Engineering at Teya with Alexandre Magno Lima Martins briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/1891199046 Thu, 08 Aug 2024 10:00:10 -0400 1426 false Airflow Strategies for Business Efficiency at Campbell with Larry Komenda Airflow Strategies for Business Efficiency at Campbell with Larry Komenda briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/1880844150 Thu, 25 Jul 2024 20:11:35 -0400 1570 false How Laurel Uses Airflow To Enhance Machine Learning Pipelines with Vincent La and Jim Howard How Laurel Uses Airflow To Enhance Machine Learning Pipelines with Vincent La and Jim Howard briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/1874450295 Thu, 18 Jul 2024 07:20:44 -0400 1438 false How Vibrant Planet's Self-Healing Pipelines Revolutionize Data Processing How Vibrant Planet's Self-Healing Pipelines Revolutionize Data Processing briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/1857855858 Thu, 27 Jun 2024 15:29:16 -0400 1432 false The Future of AI in Data Engineering With Astronomer’s Julian LaNeve and David Xue The Future of AI in Data Engineering With Astronomer’s Julian LaNeve and David Xue briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/1833134607 Wed, 29 May 2024 19:49:32 -0400 1416 false The Power of Airflow in Modern Data Environments at Wynn Las Vegas with Siva Krishna Yetukuri The Power of Airflow in Modern Data Environments at Wynn Las Vegas with Siva Krishna Yetukuri briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/1832448933 Wed, 29 May 2024 18:30:43 -0400 1471 false Powering the Texas Rangers World Series Win With AI on Airflow with Alexander Booth Powering the Texas Rangers World Series Win With AI on Airflow with Alexander Booth briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/1832444358 Wed, 29 May 2024 18:28:37 -0400 1418 false Expanding the Data Engineering Toolkit at Reddit Expanding the Data Engineering Toolkit at Reddit briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/979278949 Thu, 04 Feb 2021 21:48:17 -0500 2748 false GDPR, Self-Service Data, and Infrastructure Automation with Typeform GDPR, Self-Service Data, and Infrastructure Automation with Typeform briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/931274455 Wed, 18 Nov 2020 02:57:42 -0500 1886 false Adopting Airflow at Netlify Adopting Airflow at Netlify briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/917635843 Mon, 26 Oct 2020 03:42:01 -0400 1715 false The Road to Airflow 2.0 The Road to Airflow 2.0 briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/716910052 Fri, 22 Nov 2019 15:17:19 -0500 1948 false Airflow Breeze Airflow Breeze briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/697700581 Thu, 17 Oct 2019 13:04:56 -0400 2817 false Open Source and Airflow at Google Open Source and Airflow at Google briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/666160796 Thu, 15 Aug 2019 14:22:53 -0400 2343 false Airflow 1.10 Release Airflow 1.10 Release briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/546216096 Mon, 17 Dec 2018 14:46:16 -0500 1743 false Role-Based Access Control (RBAC) Role-Based Access Control (RBAC) briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/485906751 Wed, 15 Aug 2018 15:06:32 -0400 2248 false Airflow on Kubernetes Airflow on Kubernetes briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/465172242 Fri, 29 Jun 2018 17:00:24 -0400 3311 false Pain Points Pain Points briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/451774209 Thu, 31 May 2018 15:45:13 -0400 2337 false Competitors: Luigi Competitors: Luigi briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/429177501 Fri, 13 Apr 2018 14:41:01 -0400 1659 false Best Practices Best Practices briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/417039171 Wed, 21 Mar 2018 13:33:17 -0400 3511 false Use Cases Use Cases briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/403744467 Thu, 22 Feb 2018 22:41:25 -0500 4547 false The Origins of Airflow The Origins of Airflow briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/395373555 Tue, 06 Feb 2018 16:15:58 -0500 2709 false Season One Teaser Season One Teaser briana.okyere@astronomer.io (Astronomer) tag:soundcloud,2010:tracks/385626638 Thu, 18 Jan 2018 20:25:34 -0500 182 false