Right now the compute log manager has an API that's tied to a pipeline run. This feels a little awkward, and closes us off from capturing stdout/stderr for other things (like daemon processes).
This diff introduces a handful of more generic APIs (capture_logs, get_logs) that the existing APIs (watch, read_logs_file) could be based on.
This diff also adds some call sites / flags to the API to toggle between step-based capture and process-based capture. The default implementation maintains existing behavior, which captures stdout/stderr for each individual step. To fully support process-based capture, we would need to modify all the executors and all the existing compute log managers.
I could decouple the two changes (step vs process based run capture, run-based API vs generic API), but wanted to get early thoughts.