Changelog#

0.8.11#

Breaking Changes

  • AssetMaterializations no longer accepts a dagster_type argument. This reverts the change billed as "AssetMaterializations can now have type information attached as metadata." in the previous release.

0.8.10#

New

  • Added new GCS and Azure file manager resources
  • AssetMaterializations can now have type information attached as metadata. See the materializations tutorial for more
  • Added verification for resource arguments (previously only validated at runtime)

Bugfixes

  • Fixed bug with order-dependent python module resolution seen with some packages (e.g. numpy)
  • Fixed bug where Airflow's context['ts'] was not passed properly
  • Fixed a bug in celery-k8s when using task_acks_late: true that resulted in a 409 Conflict error from Kubernetes. The creation of a Kubernetes Job will now be aborted if another Job with the same name exists
  • Fixed a bug with composite solid output results when solids are skipped
  • Hide the re-execution button in Dagit when the pipeline is not re-executable in the currently loaded repository

Docs

  • Fixed code example in the advanced scheduling doc (Thanks @wingyplus!)
  • Various other improvements

0.8.9#

New

  • CeleryK8sRunLauncher supports termination of pipeline runs. This can be accessed via the “Terminate” button in Dagit’s Pipeline Run view or via “Cancel” in Dagit’s All Runs page. This will terminate the run master K8s Job along with all running step job K8s Jobs; steps that are still in the Celery queue will not create K8s Jobs. The pipeline and all impacted steps will be marked as failed. We recommend implementing resources as context managers and we will execute the finally block upon termination.
  • K8sRunLauncher supports termination of pipeline runs.
  • AssetMaterialization events display the asset key in the Runs view.
  • Added a new "Actions" button in Dagit to allow to cancel or delete mulitple runs.

Bugfixes

  • Fixed an issue where DagsterInstance was leaving database connections open due to not being garbage collected.
  • Fixed an issue with fan-in inputs skipping when upstream solids have skipped.
  • Fixed an issue with getting results from composites with skippable outputs in python API.
  • Fixed an issue where using Enum in resource config schemas resulted in an error.

0.8.8#

New

  • The new configured API makes it easy to create configured versions of resources.
  • Deprecated the Materialization event type in favor of the new AssetMaterialization event type, which requires the asset_key parameter. Solids yielding Materialization events will continue to work as before, though the Materialization event will be removed in a future release.
  • We are starting to deprecate "system storages" - instead of pipelines having a system storage definition which creates an intermediate storage, pipelines now directly have an intermediate storage definition.
    • We have added an intermediate_storage_defs argument to ModeDefinition, which accepts a list of IntermediateStorageDefinitions, e.g. s3_plus_default_intermediate_storage_defs. As before, the default includes an in-memory intermediate and a local filesystem intermediate storage.
    • We have deprecated system_storage_defs argument to ModeDefinition in favor of intermediate_storage_defs. system_storage_defs will be removed in 0.10.0 at the earliest.
    • We have added an @intermediate_storage decorator, which makes it easy to define intermediate storages.
    • We have added s3_file_manager and local_file_manager resources to replace the file managers that previously lived inside system storages. The airline demo has been updated to include an example of how to do this: https://github.com/dagster-io/dagster/blob/0.8.8/examples/airline_demo/airline_demo/solids.py#L171.
  • The help panel in the dagit config editor can now be resized and toggled open or closed, to enable easier editing on smaller screens.

Bugfixes

  • Opening new Dagit browser windows maintains your current repository selection. #2722
  • Pipelines with the same name in different repositories no longer incorrectly share playground state. #2720
  • Setting default_value config on a field now works as expected. #2725
  • Fixed rendering bug in the dagit run reviewer where yet-to-be executed execution steps were rendered on left-hand side instead of the right.

0.8.7#

Breaking Changes

  • Loading python modules reliant on the working directory being on the PYTHONPATH is no longer supported. The dagster and dagit CLI commands no longer add the working directory to the PYTHONPATH when resolving modules, which may break some imports. Explicitly installed python packages can be specified in workspaces using the python_package workspace yaml config option. The python_module config option is deprecated and will be removed in a future release.

New

  • Dagit can be hosted on a sub-path by passing --path-prefix to the dagit CLI. #2073
  • The date_partition_range util function now accepts an optional inclusive boolean argument. By default, the function does not return include the partition for which the end time of the date range is greater than the current time. If inclusive=True, then the list of partitions returned will include the extra partition.
  • MultiDependency or fan-in inputs will now only cause the solid step to skip if all of the fanned-in inputs upstream outputs were skipped

Bugfixes

  • Fixed accidental breaking change with input_hydration_config arguments
  • Fixed an issue with yaml merging (thanks @shasha79!)
  • Invoking alias on a solid output will produce a useful error message (thanks @iKintosh!)
  • Restored missing run pagination controls
  • Fixed error resolving partition-based schedules created via dagster schedule decorators (e.g. daily_schedule) for certain workspace.yaml formats