Sematic vs. Airflow

Iterative development and experimentation

Without local execution and proper dependency packaging, Airflow makes it very challenging to quickly iterate on pipelines and debug issues. Machine Learning work is inherently iterative: make changes (code, configurations, data), run, evaluate results, repeat.

Sematic lets users quickly iterate and debug pipelines thanks to local execution, dependency packaging, and flexible caching. Users can make changes to their local code, run locally on a small amount of data, then run at full scale in their cloud Kubernetes cluster. Sematic takes care of packaging user code, dependency, static libraries and ships them to the cluster at runtime for execution.

Exhaustive lineage tracking

Lineage tracking is the systematic and exhaustive tracking of all artifacts involved in the production of your final assets (e.g. trained models). This includes code, data, configuration, resources, and so on.

Airflow features certain basic tracking functionalities but they cannot be used as strong guarantees for reproducibility and traceability.

Sematic guarantees automatic and versioned lineage tracking of all assets by default: inputs and outputs of all pipeline steps, code, resources, etc.

Artifact visualization

In order to iterate quickly, ML Engineers need to visualize metrics, plots, configurations and results immediately after running their workloads.

Airflow does not track inputs and outputs of pipeline steps and offers no visualizations. Users will need to persist and track plots and results by themselves, then download and extract them to their local machine for visualization.

Sematic serializes and persists all inputs and outputs of all pipelines steps and displays them in its dashboard UI. Users can quickly visualize results of their workloads – sometimes even during execution – and decide on the next course of action.