Page MenuHomePhabricator

Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types REDUX
ClosedPublic

Authored by themissinghlink on Feb 19 2020, 11:42 PM.

Details

Summary

This is a continuation of https://dagster.phacility.com/D2077. I fucked up and forgot to checkout a new branch.
This revision just moves it to a fresh branch and implements the last round of feedback.

Test Plan

unit.

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

schrockn requested changes to this revision.Feb 20 2020, 12:00 AM

req'ing changes based on name being optional

python_modules/libraries/dagster-pandas/dagster_pandas/data_frame.py
162

this needs to be required

169

dataframe-level

170–176

I would recommend describing the purpose here

This revision now requires changes to proceed.Feb 20 2020, 12:00 AM
python_modules/libraries/dagster-pandas/dagster_pandas/data_frame.py
170–176

The purpose of the hydro/mat param clobbering? Wasn't it just for pluggability reasons? Maybe I am missing something here.

  • made docs fixes and added comment
schrockn requested changes to this revision.Feb 20 2020, 12:23 AM

still need to make name required. name=None will throw. a test should confirm this

This revision now requires changes to proceed.Feb 20 2020, 12:23 AM
>>> import dagster_pandas
from >>> from dagster_pandas import create_dagster_pandas_dataframe_type
>>> create_dagster_pandas_dataframe_type(name=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/schrockn/code/dagster/python_modules/libraries/dagster-pandas/dagster_pandas/data_frame.py", line 186, in create_dagster_pandas_dataframe_type
    description=description,
  File "/Users/schrockn/code/dagster/python_modules/dagster/dagster/core/types/dagster_type.py", line 86, in __init__
    check.invariant(not (name is None and key is None), 'Must set key or name')
  File "/Users/schrockn/code/dagster/python_modules/dagster/dagster/check/__init__.py", line 166, in invariant
    CheckError('Invariant failed. Description: {desc}'.format(desc=desc))
  File "/Users/schrockn/venvs/dagster-dev-3.7.4/lib/python3.7/site-packages/future/utils/__init__.py", line 419, in raise_with_traceback
    raise exc.with_traceback(traceback)
dagster.check.CheckError: Invariant failed. Description: Must set key or name
>>>
  • got rid of silly name keyword param and made it required
This revision is now accepted and ready to land.Feb 20 2020, 5:58 PM