Allow users to launch a backfill of the last pipeline partition set that generated an asset #3874
AbandonedPublic
Actions

Authored by • bengotow on May 19 2021, 3:55 AM.

Details

Reviewers

• dish
• sandyryza
• prha

Summary

https://github.com/dagster-io/dagster/issues/3874

This diff adds a "Launch Backfill" button to the "Last materialization from" line of the asset details header, making it easy to launch a backfill from the asset page.

There's a bit of a subtle distinction here - while assets have partitions they do not conform to a partition set and are not bound to a specific pipeline. You can't backfill an asset, but you can backfill (one of the) partition sets which generates an asset. For this UI, if the last materialization was from a partitioned pipeline run we pull the context from that run and offer to backfill more runs of that pipeline + partition set. To make this a bit more clear I put the "Launch Backfill" button next to the pipeline reference instead of next to the list of partitions. If you do have two pipelines generating an asset, the launch backfill button will vary in behavior depending on which materialized the asset last. (eek)

Test Plan

Run tests, give it a spin!

Diff Detail

Repository

R1 dagster

Branch

abg/3874

Lint

Lint Passed

Unit

No Test Coverage

Event Timeline

• bengotow created this revision.May 19 2021, 3:55 AM

• bengotow edited the summary of this revision. (Show Details)May 19 2021, 3:55 AM

• bengotow edited the summary of this revision. (Show Details)

Harbormaster completed remote builds in B30798: Diff 37881.May 19 2021, 4:06 AM

• bengotow requested review of this revision.May 19 2021, 4:06 AM

• sandyryza added a reviewer: • prha.May 19 2021, 9:22 PM

I have some reservations about this, but would not block this if this is a deal-breaker for Prezi.

Here are some of my reservations:

It's a little weird to have this action floating around on a partitioned asset. We don't for example have a 'Launch Run` button that would regenerate non-partitioned assets (even though we could). Should all the actions in the menu that are available for a run also be available here? Re-execution? Open in Playground?

It's also a little weird to talk about backfills, which I think of as running computation, from the assets page to begin with. For example, a single asset could be produced by multiple pipelines. Is it the right thing to just pick the last pipeline to materialize that asset? That might be simple enough when you have a 1-to-1 mapping of asset to pipeline, but it feels like that would be really confusing if you don't. Would you know which partition to run the backfill over? Could that be different than the asset partition value displayed in the asset details (there are no guarantees that those are the same thing). It could be that forcing users to click through to the run view from the asset view is actually providing a lot of value in displaying useful context.

Maybe a useful interim step is to require fewer clicks to get to the backfill page. Like, we could display the run's partition information in the materialization details (although that presents an interesting challenge if there is also an asset partition and their values are different).

I'm in agreement with Phil on the above. Apologies for leaving out the context on the issue that, even though this was a user request worth tracking, I thought it would be a little dicey to actually implement it without making some deeper changes to how we model the relationship between assets and pipelines. I don't think it's a deal breaker.

For what it's worth, going from the assets page to the partition backfill page is a whole hot mess.

We make it easy to go from the asset to the run, but we don't even show the partition information on the run unless you first go to the View tags and configuration and then you see which partition it's for, then go to the pipeline page, click on Partitions, and then find the partition matching the run.

Even making that process a little easier by direct linking would be useful, without necessarily having an action button.

Hey folks! I chatted with Sandy about this one and it sounds like we're all mostly in agreement that this doesn't reinforce a correct mental model of the system and we should probably omit this button and consider other ways to improve the linkage between assets and partition sets. Going to abandon this and follow up on the GH issue with the user!