HomePhabricator

Initial k8s operator and split out of shared functionality

Authored by James Meickle <eronarn@gmail.com> on Sat, Jul 27, 9:13 PM.

Description

Initial k8s operator and split out of shared functionality

Summary:

  1. Moving shit around: move our Python and Docker operator into two files
  2. Vendor Airflow k8s operator
  3. Add James' DagsterKubernetesPodOperator

As a follow-up, will need to figure out how we add tests for (3).

Test Plan:
tested manually in local airflow + minikube with:

'''
The airflow DAG scaffold for dagster_examples.toys.optional_outputs.optional_outputs

Note that this docstring must contain the strings "airflow" and "DAG" for
Airflow to properly detect it as a DAG
See: http://bit.ly/307VMum
'''
import datetime
import yaml

from dagster_airflow.factory import make_airflow_dag_kubernetized

ENVIRONMENT = '''
loggers:
  console:
    config:
      log_level: DEBUG

solids:
  multiply_the_word:
    inputs:
      word:
        value: bar
    config:
      factor: 2

storage:
  s3:
    config:
      s3_bucket: dagster-airflow-scratch

'''

DEFAULT_ARGS = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime.datetime(2019, 5, 7),
    'email': ['airflow@example.com'],
    'email_on_failure': False,
    'email_on_retry': False,
}

dag, tasks = make_airflow_dag_kubernetized(
    module_name='dagster_airflow_tests.test_project.dagster_airflow_demo',
    pipeline_name='demo_pipeline',
    environment_dict=yaml.load(ENVIRONMENT),
    dag_kwargs={'default_args': DEFAULT_ARGS, 'max_active_runs': 1},
    image='dagster-airflow-demo',
    namespace='default',
)

Reviewers: #ft, schrockn

Reviewed By: #ft, schrockn

Subscribers: max, schrockn

Differential Revision: https://dagster.phacility.com/D801

Details

Committed
Nate Kupp <nate@elementl.com>Tue, Aug 13, 8:43 PM
Reviewer
Restricted Project
Differential Revision
D801: Initial k8s operator and split out of shared functionality
Parents
R1:a8adcc7d7e40: Add Scheduler to dagit and DagsterGraphQLContext
Branches
Unknown
Tags
Unknown