@themissinghlink have you worked with pyspark much? Do you have thoughts here?

Nice, this reminds me of some of the lakehouse stuff - I sense a pattern will continue emerging around these higher-level solid/resource relationship setups.

I think a context manager that alters the composition context that:

• ensures the first invocation is fully resolved and has a single output
• ensures that subsequent invocations have a single unspecified input and single output
• wires everything together

do we want to dupe all of this as opposed to import it from the other files? I worry about desync

Will let @natekupp pass the sentence to swing the sword on this one

• fixed linter bug

• redesigned abstract bucket resource to transformaer resource to more cleanly match the semantics of what it was doing

not sure about this bucket api, please see comments. why not set_object(key, obj) and get_object(key)? maybe there's a missing upload_file api -- rn there's this odd asymmetry where we have set_object: str -> None and get_object: str -> blob (rather than set_object taking a blob and get_object giving one back)

I believe this is the right course of action until we get this under some form of test, will resign to @natekupp

up

up