Page MenuHomeElementl

[crag] execute_job API
AbandonedPublic

Authored by cdecarolis on Jun 24 2021, 10:30 PM.

Details

Summary

Potential route for an execute_job API. This one directly expects a code pointer. Provided under test are two possible ways of constructing pointers to jobs.

todos:
support reconstructing from file instead of module
Default to instance.get()
Re-orient tests to use fs io and multiproc to get the real experience
Change name to execute_job
Targeting using the job definition

Test Plan

will expand unit tests

Diff Detail

Repository
R1 dagster
Branch
execute_job
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

cdecarolis retitled this revision from feat: add an execute_job API to [crag] add an execute_job API.Jun 25 2021, 11:50 PM

Add possible methods of reconstructable targeting

cdecarolis retitled this revision from [crag] add an execute_job API to [crag] execute_job API.Jun 29 2021, 10:01 PM
cdecarolis edited the summary of this revision. (Show Details)
python_modules/dagster/dagster/core/definitions/decorators/repository.py
87

maybe just build the pointer here if its possible instead of splitting between here and [1]

python_modules/dagster/dagster/core/definitions/reconstructable.py
530–554

this feels a bit rough to me - i bet we can come up with something more clever

python_modules/dagster/dagster/core/definitions/repository.py
754–757

[1]

python_modules/dagster/dagster/core/definitions/reconstructable.py
530–554

what about this feels particularly rough to you / what would a more clever solution look like?

python_modules/dagster/dagster/core/definitions/reconstructable.py
530–554

API surface area cost of having to introduce a function and class

for example I think static methods on whatever the type execute / execute_job expects with the impl hidden (and changable) would feel better

JobPointer.repo_relative(repo_fn=build_repo, job_name='my_job')

like ^ but with better name

python_modules/dagster/dagster/core/definitions/reconstructable.py
530–554

makes sense. Will give that a spin.

python_modules/dagster/dagster/core/definitions/reconstructable.py
199–211

@alangenfeld what are your thoughts on this? Another option is that this could always be the entrypoint - implementation is so similar we could just accept a union of possible args or something.

JobPointer.from_repo(repo_def, target)
JobPointer.from_repo(repo_factory, target)
JobPointer.from_repo(RepositoryDefinition.from_file(path), target)

etc

python_modules/dagster/dagster/core/definitions/reconstructable.py
199–211

ya i like not having a factory specific api and just being flexible on the input

in light of https://dagster.phacility.com/D8557 it might also be worth looking at those older investigations around a workspace oriented approach since thats what submit / launch python apis work with

tho i think you may also be able to derive a IWorkspace from a JobPointer

q mgmt - I think its good if we can avoid a factory api

This revision now requires changes to proceed.Jul 2 2021, 9:39 PM

launch API attempt up next