Page MenuHomeElementl

Maintain backcompat in date_partition_range
ClosedPublic

Authored by dgibson on Oct 19 2020, 7:31 PM.

Details

Summary

I did not realize that this was part of the public API that we expose to users.

Shockingly, it does not seem to be possible to use timedelta objects (or relativedelta objects) in a timezone-safe way:

>>> dt = pendulum.create(2020, 3, 8, tz="US/Central")
>>> dt + timedelta(days=1)
<Pendulum [2020-03-09T01:00:00-05:00]> # wtf, 1 AM
>>> dt + relativedelta(days=1)
<Pendulum [2020-03-09T01:00:00-05:00]> # wtf, 1 AM
>>> dt + pendulum.interval(days=1)
<Pendulum [2020-03-09T01:00:00-05:00]> # wtf, 1 AM
>>> dt.add(days=1)
<Pendulum [2020-03-09T00:00:00-05:00]> # Seems to be the only way to actually add a day correctly

So I removed the timedelta argument to date_partition_range in https://dagster.phacility.com/D4725. But users use that as well.

This does not cover every conceivable input that might have previously gone into this function, but I will hazard a guess without data that 99% if not 100% of all previous usage was timedelta(days=1). The function is also a bit hacky but we should rmeove the delta parameter altogether (will add an issue for that), at which point we can remove it.

Test Plan

BK coverage in test_utils modified to include both delta and delta_range params (will beef it up a bit more now).

Diff Detail

Repository
R1 dagster
Branch
backcompatdaterange (branched from master)
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

python_modules/dagster/dagster/utils/partitions.py
83–89

should we add warnings about this? is the plan to remove the delta completely for 0.10.0? Might want to add a 0.10.0 tracking task to remove it completely.

python_modules/dagster/dagster/utils/partitions.py
83–89

add warning (filed an issue as well for the breaking/deprecation changes to make before the 0.10.0 release)

This revision is now accepted and ready to land.Oct 20 2020, 6:17 PM
This revision was landed with ongoing or failed builds.Oct 20 2020, 6:32 PM
This revision was automatically updated to reflect the committed changes.