Changelog#

0.9.9#

New

  • [Databricks] solids created with create_databricks_job_solid now log a URL for accessing the job in the Databricks UI.
  • The pipeline execute command now defaults to using your current directory if you don’t specify a working directory.

Bugfixes

  • [Celery-K8s] Surface errors to Dagit that previously were not caught in the Celery workers.
  • Fix issues with calling add_run_tags on tags that already exist.
  • Add “Unknown” step state in Dagit’s pipeline run logs view for when pipeline has completed but step has not emitted a completion event

Experimental

  • Version tags for resources and external inputs.

Documentation

  • Fix rendering of example solid config in “Basics of Solids” tutorial.

0.9.8#

New

  • Support for the Dagster step selection DSL: reexecute_pipeline now takes step_selection, which accepts queries like *solid_a.compute++ (i.e., solid_a.compute, all of its ancestors, its immediate descendants, and their immediate descendants). steps_to_execute is deprecated and will be removed in 0.10.0.

Community contributions

  • [dagster-databricks] Improved setup of Databricks environment (Thanks @sd2k!)
  • Enabled frozenlist pickling (Thanks @kinghuang!)

Bugfixes

  • Fixed a bug that pipeline-level hooks were not correctly applied on a pipeline subset.
  • Improved error messages when execute command can't load a code pointer.
  • Fixed a bug that prevented serializing Spark intermediates with configured intermediate storages.

Dagit

  • Enabled subset reexecution via Dagit when part of the pipeline is still running.
  • Made Schedules clickable and link to View All page in the schedule section.
  • Various Dagit UI improvements.

Experimental

  • [lakehouse] Added CLI command for building and executing a pipeline that updates a given set of assets: house update --module package.module —assets my_asset*

Documentation

  • Fixes and improvements.

0.9.7#

Bugfixes

  • Fixed an issue in the dagstermill library that caused solid config fetch to be non-deterministic.
  • Fixed an issue in the K8sScheduler where multiple pipeline runs were kicked off for each scheduled execution.

0.9.6#

New

  • Added ADLS2 storage plugin for Spark DataFrame (Thanks @sd2k!)
  • Added feature in the Dagit Playground to automatically remove extra configuration that does not conform to a pipeline’s config schema.
  • [Dagster-Celery/Celery-K8s/Celery-Docker] Added Celery worker names and pods to the logs for each step execution

Community contributions

  • Re-enabled dagster-azure integration tests in dagster-databricks tests (Thanks @sd2k!)
  • Moved dict_without_keys from dagster-pandas into dagster.utils (Thanks @DavidKatz-il)
  • Moved Dask DataFrame read/to options under read/to keys (Thanks @kinghuang)

Bugfixes

  • Fixed helper for importing data from GCS paths into Bigquery (Thanks @grabangomb (https://github.com/grabangomb)!)
  • Postgres event storage now waits to open a thread to watch runs until it is needed

Experimental

  • Added version computation function for DagsterTypeLoader. (Actual versioning will be supported in 0.10.0)
  • Added version attribute to solid and SolidDefinition. (Actual versioning will be supported in 0.10.0)

0.9.5#

New

  • UI improvements to the backfill partition selector
  • Enabled sorting of steps by failure in the partition run matrix in Dagit

Bugfixes

  • [dagstermill] fixes an issue with output notebooks and s3 storage
  • [dagster_celery] bug fixed in pythonpath calculation (thanks @enima2648!)
  • [dagster_pandas] marked create_structured_dataframe_type and ConstraintWithMetadata as experimental APIs
  • [dagster_k8s] reduced default job backoff limit to 0

Docs

  • Various docs site improvements