Development Infrastructure encompasses the tools, processes, and systems used to develop, test, and release Dagster itself. This infrastructure ensures code quality, automates testing across multiple Python versions and environments, and manages the release process for Dagster's core packages and integration libraries.
The development infrastructure consists of three main areas:
This page provides an overview of these systems and how they work together.
Development Infrastructure High-Level Architecture
Sources: .buildkite/dagster-buildkite/dagster_buildkite/pipelines/dagster_oss_main.py72-139 Makefile79-90 .buildkite/dagster-buildkite/dagster_buildkite/cli.py16-34
Dagster uses Buildkite as its CI/CD platform. The pipeline system dynamically generates test steps based on changed files and package dependencies.
The CI system has three main pipeline entry points defined in .buildkite/dagster-buildkite/dagster_buildkite/cli.py16-34:
| Entry Point | Pipeline | Purpose |
|---|---|---|
dagster() | dagster_oss_main_steps() | Main PR and commit pipeline |
dagster_nightly() | dagster_oss_nightly_steps() | Nightly integration tests |
prerelease_package() | prerelease_package_steps() | Package release workflow |
CI/CD Pipeline Components
Sources: .buildkite/dagster-buildkite/dagster_buildkite/steps/packages.py82-280 .buildkite/dagster-buildkite/dagster_buildkite/steps/tox.py37-96 .buildkite/buildkite-shared/buildkite_shared/quarantine.py24-122
The pipeline uses PackageSpec objects to define test requirements for each package, including Python version compatibility, tox factors (dependency variations), and test splits for parallelization. A quarantine system integrates with Buildkite Test Analytics to automatically handle flaky tests.
For detailed information about the CI/CD pipeline, including package testing strategies, test splitting, and the quarantine system, see page 9.1.
The build and release process handles versioning, changelog management, and publishing to PyPI.
Dagster uses dual versioning for core and library packages:
| Package Type | Version Format | Example |
|---|---|---|
Core (dagster, dagster-graphql, dagster-webserver) | X.Y.Z | 1.12.1 |
Libraries (dagster-* integration libraries) | 0.Y.Z | 0.28.1 |
The version mapping is calculated by library_version_from_core_version() in .buildkite/dagster-buildkite/dagster_buildkite/utils.py123-128 which maintains library versions at 0.(16 + CORE_MINOR).PATCH.
Release Pipeline Flow
Sources: scripts/build_and_publish.sh1-74 .buildkite/dagster-buildkite/dagster_buildkite/pipelines/prerelease_package.py9-66
The build_and_publish.sh script:
setup.py or accepts manual inputsedpython -m buildtwineThe CHANGES.md file follows a structured format with sections for New features, Bugfixes, Breaking changes, Deprecations, and package-specific annotations like [dagster-dbt] or [ui]. Release branch checks validate that changelog entries exist for the current release using .buildkite/dagster-buildkite/dagster_buildkite/steps/dagster.py115-128
For complete details on the build and release process, including Docker image publishing, Helm chart releases, and versioning strategies, see page 9.2.
The development environment provides standardized tooling for local development, code quality checks, and testing.
The root Makefile provides standard development commands organized by purpose:
| Command Category | Commands | Purpose |
|---|---|---|
| Setup | make dev_install, make dev_install_quiet | Install dependencies and build UI |
| Code Quality | make ruff, make check_ruff | Format and lint Python code |
| Type Checking | make pyright, make quick_pyright | Run static type analysis |
| UI Development | make rebuild_ui, make graphql | Build frontend and generate GraphQL |
| Validation | make check_manifest, make sanity_check | Verify package integrity |
Sources: Makefile9-112
Development Environment Component Integration
Sources: Makefile61-90 python_modules/dagster/tox.ini1-77 .buildkite/buildkite-shared/buildkite_shared/python_version.py7-66
The AvailablePythonVersion enum in .buildkite/buildkite-shared/buildkite_shared/python_version.py7-66 manages Python version compatibility:
The PackageSpec.unsupported_python_versions parameter allows packages to specify version constraints, and the tox configuration maps Python versions to tox factors like py310, py312, py313.
For complete details on development environment setup, dependency management, and testing workflows, see page 9.3.
Dagster's documentation uses Docusaurus with custom plugins and automated deployment. The system is configured in docs/docusaurus.config.ts8-265 and includes:
| Component | Implementation | Purpose |
|---|---|---|
| Docusaurus | Next.js-based static site | Documentation framework |
| Custom Plugins | llms-txt, sidebar-scroll-into-view | AI export, UX improvements |
| GitHub Actions | .github/workflows/build-docs.yml1-154 | Automated build and deploy |
| Vercel | Production and preview deployments | Hosting and CDN |
| Algolia | Search integration (production only) | Site search |
The build workflow triggers on changes to docs/** or example files, builds the site using Docusaurus, and deploys to Vercel with branch-specific preview URLs for pull requests and production deployments for release-* branches.
Sources: docs/docusaurus.config.ts8-265 .github/workflows/build-docs.yml1-154 docs/src/plugins/llms-txt/index.ts15-82
The root Makefile provides standardized development commands:
Key development commands:
make dev_install: Full development environment setupmake ruff: Run code formatting and lintingmake pyright: Type checking across the entire codebasemake rebuild_ui: Build the React frontendmake sanity_check: Verify no production packages are installedSources: Makefile9-112
The development infrastructure integrates several code quality tools:
| Tool | Purpose | Configuration |
|---|---|---|
| Ruff | Python linting and formatting | Makefile32-42 |
| Pyright | Static type checking | Makefile9-30 |
| Prettier | YAML/Markdown formatting | Makefile44-59 |
| check-manifest | Package manifest validation | Makefile94-98 |
These tools are integrated into both the local development workflow and the CI pipeline, ensuring consistent code quality across the entire codebase.
Sources: .buildkite/dagster-buildkite/dagster_buildkite/steps/dagster.py85-202 Makefile32-59
Refresh this wiki