is there a reason we don't model this on IOManager directly, as an overridable property?

I'm fine with this bool in a world where we don't persist StepOutput, but feel much less good about it as a persisted thing. Feels like we have an immature modeling of the situation currently.

Since I've joined we have tried a few different times to capture these conditions and error out ASAP are :

• will this output be accessible from another process (outputs_persisted ?)
• will this output be accessible from another machine (outputs_persisted_remotely ?)

Not catching these early can lead to pretty confusing errors. Some of the checks around this are still around (leading to this diff) but we have had to remove others for reasons such as "filesystem actually means remote accessible for me because I use a network mounted file system". I think we can do better with IOManager since it's much more reasonable to for example make a nfs_io_manager by inheriting the filesystem one in userland and just overriding an hypothetical "outputs_persisted_remotely" property.

I don't think it's a hard blocker to get this right before we lock something in to persistence that if we need it to move forward on host-mode run worker, but I think we should take some time to think about it and try.

honestly this should be addressed as well before we persist StepOutput before we go down that road - does Optional[bool] really make sense? seems like we should just coerce the bool since i am guessing thats what we do at the callsite

grepping it looks like we currently only use this at resource init time - does output materialization actually work still or do we have multiple sources of truth?

are we handling the None case correctly? given the previous logic it feels like we bias towards assuming output will be persistent

just to throw an idea out

io_manager -> None if no io manager, otherwise

{
  outputs_persisted: bool,
  # room for outputs_persisted_remotely or other things we may want to capture and know in host land
}

could also consider an io_manager_key here and storing these IOManagerProperties/ IOManagerSnapshot else where in the structure

Here was my reasoning:

• https://github.com/dagster-io/dagster/issues/2132#issuecomment-733399883
• https://github.com/dagster-io/dagster/issues/3250

Now that we have IOManagerDefinition, it _would_ be possible to add this. IMO, it's still a weird thing to add for the reasons discussed in the first link, but I don't feel super strongly.