Changelog#

1.4.10 / 0.20.10 (libraries)#

A new @partitioned_config decorator has been added for defined configuration for partitioned jobs. Thanks @danielgafni!
[dagster-aws] The ConfigurablePickledObjectS3IOManager has been renamed S3PickleIOManager for simplicity. The ConfigurablePickledObjecS3IOManager will continue to be available but is considered deprecated in favor of S3PickleIOManager. There is no change in the functionality of the I/O manager.
[dagster-azure] The ConfigurablePickledObjectADLS2IOManager has been renamed ADLS2PickleIOManager for simplicity. The ConfigurablePickledObjectADLS2IOManager will continue to be available but is considered deprecated in favor of ADLS2PickleIOManager. There is no change in the functionality of the I/O manager.
[dagster-dbt] When an exception is raised when invoking a dbt command using DbtCliResource, the exception message now includes a link to the dbt.log produced. This log file can be inspected for debugging.
[dagster-gcp] The ConfigurablePickledObjectGCSIOManager has been renamed GCSPickleIOManager for simplicity. The ConfigurablePickledObjecGCSIOManager will continue to be available but is considered deprecated in favor of GCSPickleIOManager. There is no change in the functionality of the I/O manager.

Fixed a bug that caused a DagsterInvariantViolationError when executing a multi-asset where both assets have self-dependencies on earlier partitions.
Fixed an asset backfill issue where some runs continue to be submitted after a backfill is requested for cancellation.
[dagster-dbt] Fixed an issue where using the --debug flag raised an exception in the Dagster framework.
[ui] “Launched run” and “Launched backfill” toasts in the Dagster UI behave the same way. To open in a new tab, hold the cmd/ctrl key when clicking “View”
[ui] When opening step compute logs, the view defaults to stderr which aligns with Python’s logging defaults.
[ui] When viewing a global asset graph with more than 100 assets, the “choose a subset to display” prompt is correctly aligned to the query input.

[dagster-graphql] The Dagster GraphQL Python client now includes a default timeout of 300 seconds for each query, to ensure that GraphQL requests don’t hang and never return a response. If you are running a query that is expected to take longer than 300 seconds, you can set the timeout argument when constructing a DagsterGraphQLClient.
[ui] We are continuing to improve the new horizontal rendering of the asset graph, which you can enable in Settings. This release increases spacing between nodes and improves the traceability of arrows on the graph.

Several Pythonic resources and I/O managers now have API docs entries.
Updated the tutorial’s example project and content to be more explicit about resources.
[dagster-dbt] Added API docs examples for DbtCliResource and DbtCliResource.cli(...).
Some code samples in API docs for InputContext and OutputContext have been fixed. Thanks @Sergey Mezentsev!

When setting up a new organization by importing a dbt project, using GitLab is now supported.

Added a respect_materialization_data_versions option to auto materialization. It can enabled in dagster.yaml with


auto_materialize:
  respect_materialization_data_versions: True

This flag may be changed or removed in the near future.

ops or assets with multiple outputs that are all required and return type None/ Nothing will interpret an explicitly or implicitly returned value None to indicate that all outputs were successful.
The skip_reason argument to the constructor of SensorResult now accepts a string in addition to a SkipReason.
[dagster-k8s] Added a step_k8s_config field to k8s_job_executor that allows you to customize the raw Kubernetes config for each step in a job. See the docs for more information.
[dagster-k8s] Launched run pods now have an additional code location label.
[dagster-ui] The runs table now lets you toggle which tags are always visible.
[dagster-dbt] dagster-dbt project scaffold now creates the scaffold in multiple files:
- constants.py contains a reference to your manifest and dbt project directory
- assets.py contains your initial dbt assets definitions
- definitions.py contains the code to load your asset definitions into the Dagster UI
- schedules.py contains an optional schedule to add for your dbt assets
[dagster-dbt] Added new methods get_auto_materialize_policy and get_freshness_policy to DagsterDbtTranslator.
[dagster-fivertran] Sync options can now be passed to load_assets_from_fivetran_instance.
[dagster-wandb] W&B IO Manager now handles partitions natively. (Thanks @chrishiste!)

Previously, canceling large asset backfills would cause the daemon to time out and display a “not running” error. This has been fixed.
[dagster-ssh] Previously the SSHResource would warn when allow_host_key_change was set. Now known hosts are always loaded from the system hosts file, and the allow_host_key_change parameter is ignored.
Previously, when using AutoMaterializePolicies, partitioned assets downstream of partitioned observable source assets could be materialized before their parent partitions were observed. This has been fixed.

@graph_multi_asset now has an API docs entry.
The GCSComputeLogManager example in the Dagster Instance reference is now correct.
Several outdated K8s documentation links have been removed from the Customizing your Kubernetes deployment guide.
Added callouts to the GitHub and GitLab Branch Deployment guides specifying that some steps are optional for Serverless users.
The “Graphs” page under the “Concepts” section has been renamed to “Op Graphs” and moved inside under the “Ops” heading.
[dagster-dbt] Added API examples for @dbt_assets for the following use-cases:
- Running dbt commands with flags
- Running dbt commands with --vars
- Running multiple dbt commands
- Retrieving dbt artifacts after running a dbt command
- Invoking other Dagster resouces alongside dbt
- Defining and accessing Dagster config alongside dbt