Add Pipeline Step Secrets support to Python SDK and CLI

[Copilot]

Clarifai backend now supports Pipeline Step Secrets for step-level secret isolation. This PR implements SDK and CLI support for the feature.

Why

• The Clarifai backend has added support for Pipeline Step Secrets, enabling different pipeline steps to access distinct sets of secrets with step-level isolation
• This feature is needed to improve security by ensuring each step only accesses explicitly configured secrets
• The Python SDK and CLI need to be updated to support this new backend capability

How

• Parse step_version_secrets from the config section in config.yaml (matching proto structure PipelineVersion.config.step_version_secrets)
• Inject secrets into PipelineVersion.config via new _add_step_version_secrets() method
• Add helper methods in Pipeline client for managing step secrets
• Update CLI templates and documentation to show correct usage
• Use flattened structure where secrets are directly under each step reference (no nested secrets key per API update)

Tests

• Added 9 new unit tests for step secrets functionality
• All 19 tests passing (14 existing + 9 new)
• Tests cover pipeline builder, pipeline client methods, and lockfile generation
• Validates both the flattened YAML structure and proto interaction

Notes

Changes

Pipeline Builder (clarifai/runners/pipelines/pipeline_builder.py)

• Parse step_version_secrets from config section in config.yaml (matching proto structure PipelineVersion.config.step_version_secrets)
• Inject secrets into PipelineVersion.config via new _add_step_version_secrets() method
• Preserve secrets in lockfile generation (prepare_lockfile_with_step_versions(), generate_lockfile_data())
• Use flattened structure (secrets directly under step reference)

Pipeline Client (clarifai/client/pipeline.py)

• get_pipeline_version() - retrieve version with secrets config
• create_pipeline_version() - create version with secrets via PatchPipelineVersions
• list_step_secrets() - query secrets for all/specific steps
• Python API accepts simplified format: {step_ref: {secret_name: secret_path}}

CLI Template (clarifai/cli/templates/pipeline_templates.py)

• Add commented step_version_secrets example to generated pipeline configs in correct config section
• Use flattened structure without nested secrets key

Documentation (docs/pipeline_step_secrets.md)

• Usage guide with YAML and Python examples
• Shows flattened structure (secrets directly under step reference)
• Security model and best practices

Usage

Config YAML (Flattened Structure):

pipeline: orchestration_spec: argo_orchestration_spec: |  # workflow definition  config: step_version_secrets: step-0: API_KEY: users/user123/secrets/my-api-key DB_PASSWORD: users/user123/secrets/db-password step-1: EMAIL_TOKEN: users/user123/secrets/email-token

Python SDK:

# Create version with secrets version_id = pipeline.create_pipeline_version( orchestration_spec=orch_spec, step_version_secrets={ "step-0": {"API_KEY": "users/user123/secrets/my-api-key"} } ) # Query secrets secrets = pipeline.list_step_secrets() # Returns: {"step-0": {"API_KEY": "users/user123/secrets/my-api-key"}, ...}

Security

• SDK handles only secret references (paths), not values
• Backend injects actual values at runtime via Kubernetes SecretKeyRef
• Logging uses debug level and omits secret names to minimize exposure
• CodeQL alerts are false positives (logging step references like "step-0", not secrets)

Backward Compatibility

• step_version_secrets is optional
• Existing pipelines work unchanged
• All existing tests pass (19 total, 9 new)

API Structure Note

The configuration uses a flattened structure where secret mappings are directly under each step reference. While the proto internally has a StepSecretConfig with a secrets field, the YAML configuration and Python API use the simplified format for ease of use.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Pipeline Step Secrets Feature Adoption in Python SDK and CLI</issue_title>
<issue_description>## Overview

Clarifai backend now supports Pipeline Step Secrets—enabling different pipeline steps to access distinct sets of secrets with step-level isolation. This document outlines the technical changes required to adopt this feature in the Python SDK and CLI, ensuring compatibility with updated API fields and backend functionality.

---

API Feature Summary

1. New Proto Fields

• PipelineVersionConfig
  • step_version_secrets: map<string, StepSecretConfig>
    • Key: Step reference (step1, step2, ...)
    • Value: StepSecretConfig
• StepSecretConfig (New Message)
  • secrets: map<string, string>
    • Key: Secret name (e.g., API_KEY)
    • Value: Secret reference path (e.g., users/user123/secrets/my-api-key)

2. Updated API Endpoints

• GetPipelineVersion: Returns pipeline version with step secret config.
• PostPipelineVersions: Create pipeline version with step secrets.

3. Example API Usage

Creation

{ "pipeline_version": { "config": { "step_version_secrets": { "step1": { "secrets": { "API_KEY": "users/user123/secrets/my-api-key", "DB_PASSWORD": "users/user123/secrets/db-secret" } }, "step2": { "secrets": { "EMAIL_TOKEN": "users/user123/secrets/email-token" } } } } } }

---

Python SDK Changes

1. Proto/Model Updates

• Regenerate Python proto files from updated .proto definitions.
• Ensure PipelineVersionConfig includes the step_version_secrets field.
• Add StepSecretConfig model/class for mapping secrets per step.

2. Client Method Updates

• Update get_pipeline_version() to handle new step secrets field.
• Update create_pipeline_version() to accept step secrets.
• Add type hints for new fields.

3. Helper Methods (Recommended)

• Add convenience methods:
  • add_step_secret(step_ref, secret_name, secret_ref)
  • remove_step_secret(step_ref, secret_name)
  • list_step_secrets(step_ref)

---

CLI Changes

1. Command Updates

• Update clarifai pipeline version create to accept step secrets specified in the orchestration spec (config.yaml), not as CLI parameters.
• Update clarifai pipeline version get to display step secrets.

2. Config File Integration (Orchestration Spec)

• Step secrets should be defined within the config.yaml file as part of the pipeline orchestration specification, matching the API schema.

• Example config.yaml:

  pipeline_version: config: step_version_secrets: step1: secrets: API_KEY: users/user123/secrets/my-api-key DB_PASSWORD: users/user123/secrets/db-secret step2: secrets: EMAIL_TOKEN: users/user123/secrets/email-token

• CLI usage examples:

  clarifai pipeline version create --pipeline-id "my-pipeline" --config config.yaml 

• All step secrets should be managed via the spec file; do not support CLI flags for individual secrets.

3. Output Formatting

• Update list/get commands to display step secrets in human-readable/table format.
• Add JSON output option for programmatic use.

---

Backward Compatibility

• ✅ Fully Backward Compatible: All existing API calls will continue to work.
• ✅ Optional Field: step_version_secrets is optional; existing pipelines unaffected.
• ✅ No Breaking Changes: No existing fields were modified or removed.

---

Testing Requirements

1. Unit Tests

• Test pipeline version creation with step secrets from config file.
• Test step secrets validation and error handling.
• Test backward compatibility (existing pipelines without secrets).

2. Integration Tests

• End-to-end testing for pipeline creation, execution, and CLI commands with step secrets defined in config file.
• Error scenarios (invalid secret references, missing secrets).

3. Example Code & Documentation

• Add step secrets usage examples to docs.
• Update any existing pipeline examples to show step secrets usage in orchestration spec.

---

Security Considerations

• Step-Level Isolation: Each step only accesses explicitly configured secrets.
• Secret References Only: API handles references, not actual secret values.
• Kubernetes Integration: Backend mounts secrets securely as environment variables.
• No Secret Value Leakage: Only references are stored in config; values are injected at runtime.

---

Additional Context

• Backend uses env var injection and Kubernetes SecretKeyRef for secure secret mounting.
• Isolation is guaranteed for each step.
• Multiple secrets per step and multiple steps per pipeline are supported.

---...

• Fixes Pipeline Step Secrets Feature Adoption in Python SDK and CLI #829

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

To fix this problem, avoid logging potentially sensitive information such as secret references or their paths. Instead of logging the entire step_ref, use a generic message or obfuscate the reference (e.g., only log the existence of a skipped step, log a hash, or mask the reference). Change line 395 inside method _add_step_version_secrets to avoid outputting the value of step_ref in cleartext. For example, replace the message with a generic "A step contains no secret references and will be skipped" or, if logging is required, sufficiently redact or hash the reference string. No additional imports are required—the logging library and framework is already set up.

The best way to fix this issue is to prevent logging of secret references in clear text. In particular, at line 405, we should avoid including step_ref verbatim as part of the debug message. Instead, we can safely log a generic message, such as "Configured secret references for a step" or, if needed for debugging, log only non-sensitive information (e.g., the number of configured secrets, or step indices).

Changes to make:

• Update line 405 in clarifai/runners/pipelines/pipeline_builder.py to remove or redact the logging of step_ref.
• If you must log info for debugging, use a generic message (without referencing the specific step_ref value).
• No new imports or dependencies are needed.

---

[ydixit-clarifai]

@copilot The stepversionsecret should be part of the orchestration spec in each of the pipeline steps.
The pipelineverison structure should be PipelineVersion.Config -> StepVersionSecrets: Map<String, Map<String, String / Secret Reference>>

Why Map<SecretName, SecretRef>?

For one pipeline,
SecretName can be EMAIL_PROVIDER_API_KEY and SecretRef can be users/1/secrets/secret-1

While for another one,
SecretName can be EMAIL_PROVIDER_API_KEY and SecretRef can be users/2/secrets/secret-3 (example representation of secret-reference)

[Copilot]

@copilot The stepversionsecret should be part of the orchestration spec in each of the pipeline steps.
The pipelineverison structure should be PipelineVersion.Config -> StepVersionSecrets: Map<String...

Updated the implementation to match the proto structure. The step_version_secrets is now part of the config section at the pipeline level (matching PipelineVersion.config.step_version_secrets), not nested inside orchestration_spec.

The new YAML structure is:

pipeline: orchestration_spec: argo_orchestration_spec: |  ...  config: step_version_secrets: step-0: secrets: API_KEY: users/user123/secrets/my-api-key

All tests updated and passing. Changes in commit 9cdadbd.

[ydixit-clarifai]

@copilot secrets has been removed from api. remove it in the cli and sdk code as well.

[Copilot]

Removed the nested secrets key from the YAML configuration structure. The structure is now flattened so secrets are directly under each step reference:

config: step_version_secrets: step-0: API_KEY: users/user123/secrets/my-api-key DB_PASSWORD: users/user123/secrets/db-secret

Updated in CLI templates, documentation, pipeline builder, and all tests. Changes in commit 540244b.