[Airflow] - what is Airflow

Airflow

---

Airflow is a platform to programmatically author, schedule and monitor workflows.

• Use Airflow to author workflows as Directed Acyclic Graphs (DAGs) of tasks
• The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies.
• Airflow is not a data streaming(data load like image,youtube ) solution.

 

Principles

---

1. Dynamic

• Airflow pipelines are configuration as code (Python)
• Python allow for dynamic pipeline generation
• This allows for writing code that instantiates pipelines.

 

2. Extensible

• user define Operator , Executor
• extend the library

 

3. Elegant

• Airflow pipelines are lean and explicit. using by jinja template

 

4. Scalable

• Airflow has a modular architecture( not one cunstructed app, multiple small module app )
• Airflow use a message queue to orchestrate an arbitrary number of workers

 

*message queue = message middleware to connect app,system,service / ussing ansync message queue , reduce bottlenecks

*container orchestrate = optimize multiple container deployee process

E.g) k8s  / Docker swarm

1. automatic deployment

2. container automatic placement and duplicate

3. container group load balancing

4. container auto-recovery

5. expand and collapse by adding or removing container

 

Reference

https://airflow.apache.org/docs/apache-airflow/stable/index.html