Apache Airflow is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows. Airflow’s extensible Python framework enables you to build workflows connecting with virtually any technology. A web interface helps manage the state of your workflows. Airflow is deployable in many ways, varying from a single process on your laptop to a distributed setup to support even the biggest workflows.
he main characteristic of Airflow workflows is that all workflows are defined in Python code. “Workflows as code” serves several purposes:
Dynamic: Airflow pipelines are configured as Python code, allowing for dynamic pipeline generation.
Extensible: The Airflow framework contains operators to connect with numerous technologies. All Airflow components are extensible to easily adjust to your environment.
Flexible: Workflow parameterization is built-in leveraging the Jinja templating engine.
Airflow is a batch workflow orchestration platform. The Airflow framework contains operators to connect with many technologies and is easily extensible to connect with a new technology. If your workflows have a clear start and end, and run at regular intervals, they can be programmed as an Airflow DAG.
If you prefer coding over clicking, Airflow is the tool for you. Workflows are defined as Python code which means:
Workflows can be stored in version control so that you can roll back to previous versions
Workflows can be developed by multiple people simultaneously
Tests can be written to validate functionality
Components are extensible and you can build on a wide collection of existing components
https://airflow.apache.org/docs/apache-airflow/stable/
No comments:
Post a Comment