Page MenuHomeElementl
Differential D4920 Diff 28837 CHANGES.md

Changeset View

Standalone View

View Options

CHANGES.md

Show All 12 Lines
- Eliminated `input_hydration_config` and `output_materialization_config` arguments on type constructors. - Eliminated `input_hydration_config` and `output_materialization_config` arguments on type constructors.
- Changed `ExecuteStepArgs` parameters. Replaced `execute_step_with_structured_logs` CLI command - Changed `ExecuteStepArgs` parameters. Replaced `execute_step_with_structured_logs` CLI command
with `execute_step`, and `execute_run_with_structured_logs` with `execute_run`. with `execute_step`, and `execute_run_with_structured_logs` with `execute_run`.
- The `.compute` suffix is no longer applied to all step keys. - The `.compute` suffix is no longer applied to all step keys.
- `SolidExecutionResult.compute_output_event_dict` changed to `SolidExecutionResult.compute_output_events_dict` - `SolidExecutionResult.compute_output_event_dict` changed to `SolidExecutionResult.compute_output_events_dict`
- Eliminated `pipeline_def` property from `InitResourceContext`. - Eliminated `pipeline_def` property from `InitResourceContext`.
- Removed the argument `step_keys_to_execute` to the `reexecute_pipeline` and `reexecute_pipeline_iterator`. - Removed the argument `step_keys_to_execute` to the `reexecute_pipeline` and `reexecute_pipeline_iterator`.
- [Helm/K8s]
- The schema for the `scheduler` values in the helm chart has been rewritten.
## 0.9.22.post0 ## 0.9.22.post0
**Bugfixes** **Bugfixes**
- [Dask] Pin dask[dataframe] to <=2.30.0 and distributed to <=2.30.1 - [Dask] Pin dask[dataframe] to <=2.30.0 and distributed to <=2.30.1
## 0.9.22 ## 0.9.22
▲ Show 20 Lines • Show All 352 Lines • ▼ Show 20 Lines
**Community contributions** **Community contributions**
- Re-enabled dagster-azure integration tests in dagster-databricks tests (Thanks @sd2k!) - Re-enabled dagster-azure integration tests in dagster-databricks tests (Thanks @sd2k!)
- Moved dict_without_keys from dagster-pandas into dagster.utils (Thanks @DavidKatz-il) - Moved dict_without_keys from dagster-pandas into dagster.utils (Thanks @DavidKatz-il)
- Moved Dask DataFrame read/to options under read/to keys (Thanks @kinghuang) - Moved Dask DataFrame read/to options under read/to keys (Thanks @kinghuang)
**Bugfixes** **Bugfixes**
- Fixed helper for importing data from GCS paths into Bigquery (Thanks @grabangomb (https://github.com/grabangomb)!) - Fixed helper for importing data from GCS paths into Bigquery (Thanks @grabangomb (<https://github.com/grabangomb>)!)
- Postgres event storage now waits to open a thread to watch runs until it is needed - Postgres event storage now waits to open a thread to watch runs until it is needed
**Experimental** **Experimental**
- Added version computation function for DagsterTypeLoader. (Actual versioning will be supported in 0.10.0) - Added version computation function for DagsterTypeLoader. (Actual versioning will be supported in 0.10.0)
- Added version attribute to solid and SolidDefinition. (Actual versioning will be supported in 0.10.0) - Added version attribute to solid and SolidDefinition. (Actual versioning will be supported in 0.10.0)
## 0.9.5 ## 0.9.5
▲ Show 20 Lines • Show All 61 Lines • ▼ Show 20 Lines
- Fixes the GraphQL playground hosted by dagit - Fixes the GraphQL playground hosted by dagit
- Fixes a bug where K8s CronJobs were stopped unnecessarily during schedule reconciliation - Fixes a bug where K8s CronJobs were stopped unnecessarily during schedule reconciliation
**Experimental** **Experimental**
- New `dagster-k8s/config` tag that lets users pass in custom configuration to the Kubernetes `Job`, `Job` metadata, `JobSpec`, `PodSpec`, and `PodTemplateSpec` metadata. - New `dagster-k8s/config` tag that lets users pass in custom configuration to the Kubernetes `Job`, `Job` metadata, `JobSpec`, `PodSpec`, and `PodTemplateSpec` metadata.
- This allows users to specify settings like eviction policy annotations and node affinities. - This allows users to specify settings like eviction policy annotations and node affinities.
- Example: - Example:
```python ```python
@solid( @solid(
tags = { tags = {
'dagster-k8s/config': { 'dagster-k8s/config': {
'container_config': { 'container_config': {
'resources': { 'resources': {
'requests': { 'cpu': '250m', 'memory': '64Mi' }, 'requests': { 'cpu': '250m', 'memory': '64Mi' },
'limits': { 'cpu': '500m', 'memory': '2560Mi' }, 'limits': { 'cpu': '500m', 'memory': '2560Mi' },
▲ Show 20 Lines • Show All 65 Lines • ▼ Show 20 Lines
**New** **New**
- Python 3.8 is now fully supported. - Python 3.8 is now fully supported.
- `-d` or `--working-directory` can be used to specify a working directory in any command that - `-d` or `--working-directory` can be used to specify a working directory in any command that
takes in a `-f` or `--python_file` argument. takes in a `-f` or `--python_file` argument.
- Removed the deprecation of `create_dagster_pandas_dataframe_type`. This is the currently - Removed the deprecation of `create_dagster_pandas_dataframe_type`. This is the currently
supported API for custom pandas data frame type creation. supported API for custom pandas data frame type creation.
- Removed gevent dependency from dagster - Removed gevent dependency from dagster
- New `configured` API for predefining configuration for various definitions: https://docs.dagster.io/overview/configuration/#configured - New `configured` API for predefining configuration for various definitions: <https://docs.dagster.io/overview/configuration/#configured>
- Added hooks to enable success and failure handling policies on pipelines. This enables users to set up policies on all solids within a pipeline or on a per solid basis. Example usage can be found [here](https://docs.dagster.io/examples/hooks) - Added hooks to enable success and failure handling policies on pipelines. This enables users to set up policies on all solids within a pipeline or on a per solid basis. Example usage can be found [here](https://docs.dagster.io/examples/hooks)
- New instance level view of Scheduler and running schedules - New instance level view of Scheduler and running schedules
- dagster-graphql is now only required in dagit images. - dagster-graphql is now only required in dagit images.
## 0.8.11 ## 0.8.11
**Breaking Changes** **Breaking Changes**
▲ Show 20 Lines • Show All 61 Lines • ▼ Show 20 Lines - We have added an `intermediate_storage_defs` argument to `ModeDefinition`, which accepts a
storage. storage.
- We have deprecated `system_storage_defs` argument to `ModeDefinition` in favor of - We have deprecated `system_storage_defs` argument to `ModeDefinition` in favor of
`intermediate_storage_defs`. `system_storage_defs` will be removed in 0.10.0 at the earliest. `intermediate_storage_defs`. `system_storage_defs` will be removed in 0.10.0 at the earliest.
- We have added an `@intermediate_storage` decorator, which makes it easy to define intermediate - We have added an `@intermediate_storage` decorator, which makes it easy to define intermediate
storages. storages.
- We have added `s3_file_manager` and `local_file_manager` resources to replace the file managers - We have added `s3_file_manager` and `local_file_manager` resources to replace the file managers
that previously lived inside system storages. The airline demo has been updated to include that previously lived inside system storages. The airline demo has been updated to include
an example of how to do this: an example of how to do this:
https://github.com/dagster-io/dagster/blob/0.8.8/examples/airline_demo/airline_demo/solids.py#L171. <https://github.com/dagster-io/dagster/blob/0.8.8/examples/airline_demo/airline_demo/solids.py#L171>.
- The help panel in the dagit config editor can now be resized and toggled open or closed, to - The help panel in the dagit config editor can now be resized and toggled open or closed, to
enable easier editing on smaller screens. enable easier editing on smaller screens.
**Bugfixes** **Bugfixes**
- Opening new Dagit browser windows maintains your current repository selection. #2722 - Opening new Dagit browser windows maintains your current repository selection. #2722
- Pipelines with the same name in different repositories no longer incorrectly share playground state. #2720 - Pipelines with the same name in different repositories no longer incorrectly share playground state. #2720
- Setting `default_value` config on a field now works as expected. #2725 - Setting `default_value` config on a field now works as expected. #2725
▲ Show 20 Lines • Show All 615 Lines • ▼ Show 20 Lines
**Bugfix** **Bugfix**
- No longer manually dispose sqlalchemy engine in dagster-postgres - No longer manually dispose sqlalchemy engine in dagster-postgres
- Made boto3 dependency in dagster-aws more flexible (#2418) - Made boto3 dependency in dagster-aws more flexible (#2418)
- Fixed tooltip UI cleanup in partitioned schedule view - Fixed tooltip UI cleanup in partitioned schedule view
**Documentation** **Documentation**
- Brand new documentation site, available at https://docs.dagster.io - Brand new documentation site, available at <https://docs.dagster.io>
- The tutorial has been restructured to multiple sections, and the examples in intro_tutorial have - The tutorial has been restructured to multiple sections, and the examples in intro_tutorial have
been rearranged to separate folders to reflect this. been rearranged to separate folders to reflect this.
## 0.7.8 ## 0.7.8
**Breaking Changes** **Breaking Changes**
- The `execute_pipeline_with_mode` and `execute_pipeline_with_preset` APIs have been dropped in - The `execute_pipeline_with_mode` and `execute_pipeline_with_preset` APIs have been dropped in
▲ Show 20 Lines • Show All 811 Lines • ▼ Show 20 Lines- Simplified graphql API arguments to single-step re-execution to use `retryRunId`, `stepKeys`
execution parameters instead of a `reexecutionConfig` input object execution parameters instead of a `reexecutionConfig` input object
- Fixes missing step-level stdout/stderr from dagster CLI - Fixes missing step-level stdout/stderr from dagster CLI
## 0.6.3 ## 0.6.3
- Adds a `type_check` parameter to `PythonObjectType`, `as_dagster_type`, and `@as_dagster_type` to - Adds a `type_check` parameter to `PythonObjectType`, `as_dagster_type`, and `@as_dagster_type` to
enable custom type checks in place of default `isinstance` checks. enable custom type checks in place of default `isinstance` checks.
See documentation here: See documentation here:
https://dagster.readthedocs.io/en/latest/sections/learn/tutorial/types.html#custom-type-checks <https://dagster.readthedocs.io/en/latest/sections/learn/tutorial/types.html#custom-type-checks>
- Improved the type inference experience by automatically wrapping bare python types as dagster - Improved the type inference experience by automatically wrapping bare python types as dagster
types. types.
- Reworked our tutorial (now with more compelling/scary breakfast cereal examples) and public API - Reworked our tutorial (now with more compelling/scary breakfast cereal examples) and public API
documentation. documentation.
See the new tutorial here: See the new tutorial here:
https://dagster.readthedocs.io/en/latest/sections/learn/tutorial/index.html <https://dagster.readthedocs.io/en/latest/sections/learn/tutorial/index.html>
- New solids explorer in Dagit allows you to browse and search for solids used across the - New solids explorer in Dagit allows you to browse and search for solids used across the
repository. repository.
![Solid Explorer](./screenshots/solid_explorer.png) ![Solid Explorer](./screenshots/solid_explorer.png)
![Solid Explorer](./screenshots/solid_explorer_input.png) ![Solid Explorer](./screenshots/solid_explorer_input.png)
- Enabled solid dependency selection in the Dagit search filter. - Enabled solid dependency selection in the Dagit search filter.
Show All 16 Lines
## 0.6.2 ## 0.6.2
- Changed composition functions `@pipeline` and `@composite_solid` to automatically give solids - Changed composition functions `@pipeline` and `@composite_solid` to automatically give solids
aliases with an incrementing integer suffix when there are conflicts. This removes to the need aliases with an incrementing integer suffix when there are conflicts. This removes to the need
to manually alias solid definitions that are used multiple times. to manually alias solid definitions that are used multiple times.
- Add `dagster schedule wipe` command to delete all schedules and remove all schedule cron jobs - Add `dagster schedule wipe` command to delete all schedules and remove all schedule cron jobs
- `execute_solid` test util now works on composite solids. - `execute_solid` test util now works on composite solids.
- Docs and example improvements: https://dagster.readthedocs.io/ - Docs and example improvements: <https://dagster.readthedocs.io/>
- Added `--remote` flag to `dagster-graphql` for querying remote Dagit servers. - Added `--remote` flag to `dagster-graphql` for querying remote Dagit servers.
- Fixed issue with duplicate run tag autocomplete suggestions in Dagit (#1839) - Fixed issue with duplicate run tag autocomplete suggestions in Dagit (#1839)
- Fixed Windows 10 / py3.6+ bug causing pipeline execution failures - Fixed Windows 10 / py3.6+ bug causing pipeline execution failures
## 0.6.1 ## 0.6.1
- Fixed an issue where Dagster public images tagged `latest` on Docker Hub were erroneously - Fixed an issue where Dagster public images tagged `latest` on Docker Hub were erroneously
published with an older version of Dagster (#1814) published with an older version of Dagster (#1814)
▲ Show 20 Lines • Show All 117 Lines • Show Last 20 Lines