bk, touched demo
This makes sense to me. @schrockn did you want to do a pass here as well
since there are two kinds of keys here (run keys and s3 keys) it might be worth being more explicit about which are which (which probably involves prefixing all the s3 keys with s3_)
this seems more like an exception than a SkipReason?
is there a weird edge case here when it returns exactly MAX_KEYS because that's exactly how many there are? Or does that just make it do one more loop with 0 returned and then we're fine
i do think we will evitably attach resources to this stuff btw but no need to address it now :-)
how is last_completion_time computed. I was imagining querying the runs database here and getting the last run_key directly. That would seem to be a nice generalizable pattern where you can implement reconciliation logic
consider passing in s3_session so this can be tested easily
believe last_completion_time is the time we ran the last successful loop - so if you fail partway through that would not affect last_completion_time
at one point we had a 'cursor' field on context which was exactly that (the last run_key), not sure where exactly we landed there. It's also nice to expose the time as well for the poor souls who don't use run keys I think?