Skip to content

comet-ml/s3proxy-chart

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

19 Commits
 
 
 
 
 
 

Repository files navigation

s3proxy

Version: 0.0.3 Type: application AppVersion: 2.7.0

A Helm chart for deploying S3Proxy - Access other storage backends via the S3 API

Prerequisites

  • Kubernetes 1.19+
  • Helm 3.2.0+
  • PV provisioner support in the underlying infrastructure (if using filesystem backend with persistence)

Installation

Add the repository (if published)

helm repo add s3proxy <REPO_URL> helm repo update

Install the chart

# Install with default values (filesystem backend) helm install my-s3proxy ./s3proxy # Install with custom values helm install my-s3proxy ./s3proxy -f my-values.yaml

Configuration

The following section lists the configurable parameters of the s3proxy chart and their default values.

General Parameters

Values

Key Type Default Description
affinity object {} Affinity for pod assignment
autoscaling.enabled bool false Enable HPA
autoscaling.maxReplicas int 100 Maximum number of replicas
autoscaling.minReplicas int 1 Minimum number of replicas
autoscaling.targetCPUUtilizationPercentage int 80 Target CPU utilization percentage
config.authorization string "aws-v4" Authorization type (none, aws-v2, aws-v4, aws-v2-or-v4)
config.backend.awsS3.accessKeyId string "" AWS Access Key ID for backend
config.backend.awsS3.endpoint string "" S3 endpoint
config.backend.awsS3.region string "" AWS region
config.backend.awsS3.secretAccessKey string "" AWS Secret Access Key for backend
config.backend.azureblob.account string "" Storage account name
config.backend.azureblob.endpoint string "" Azure endpoint
config.backend.azureblob.key string "" Storage account key
config.backend.azureblob.sasToken string "" SAS token
config.backend.b2.account string "" B2 account ID
config.backend.b2.applicationKey string "" B2 application key
config.backend.filesystem.basedir string "/data/s3proxy" Base directory for filesystem backend
config.backend.googleCloudStorage.clientEmail string "" Service account email
config.backend.googleCloudStorage.clientId string "" Client ID
config.backend.googleCloudStorage.privateKey string "" Private key
config.backend.googleCloudStorage.privateKeyId string "" Private key ID
config.backend.googleCloudStorage.projectId string "" GCP project ID
config.backend.provider string "filesystem-nio2" Backend provider type (filesystem, filesystem-nio2, transient, transient-nio2, aws-s3, s3, azureblob, azureblob-sdk, b2, google-cloud-storage, openstack-swift, rackspace-cloudfiles-uk, rackspace-cloudfiles-us)
config.backend.swift.authUrl string "" Authentication URL
config.backend.swift.password string "" Password
config.backend.swift.region string "" Region
config.backend.swift.tenantName string "" Tenant name
config.backend.swift.userName string "" Username
config.buckets.alias object {} Map virtual bucket names to actual backend buckets
config.buckets.locator list [] Assign specific buckets to different backends (glob patterns supported)
config.cors.allowCredential bool false Allow credentials
config.cors.allowHeaders list ["Accept","Content-Type"] Allowed headers
config.cors.allowMethods list ["GET","PUT","POST","HEAD","DELETE"] Allowed methods
config.cors.allowOrigins list [] Allowed origins (e.g., ["https://example.com", "https://.+\.example\.com"])
config.cors.enabled bool false Enable CORS support
config.credential string "" S3 Secret Access Key for client authentication
config.identity string "" S3 Access Key ID for client authentication
config.middlewares.eventualConsistency bool false Enable eventual consistency modeling
config.middlewares.largeObjectMocking bool false Enable large object mocking
config.middlewares.readOnly bool false Make backend read-only
config.middlewares.shardedBackend bool false Enable sharded backend containers
config.virtualHost string "" Virtual Host configuration
extraEnvVars list [] Additional environment variables
extraVolumeMounts list [] Additional volume mounts
extraVolumes list [] Additional volumes
fullnameOverride string "" String to fully override s3proxy.fullname template
image.pullPolicy string "IfNotPresent" Image pull policy
image.repository string "andrewgaul/s3proxy" S3Proxy image repository
image.tag string "" Overrides the image tag whose default is the chart appVersion
imagePullSecrets list [] Image pull secrets
ingress.annotations object {} Ingress annotations
ingress.className string "" Ingress class name
ingress.enabled bool false Enable ingress
ingress.hosts list [] Ingress hosts configuration
ingress.tls list [] TLS configuration
nameOverride string "" String to partially override s3proxy.fullname template
nodeSelector object {} Node selector for pod assignment
persistence.accessMode string "ReadWriteOnce" PVC Access Mode
persistence.annotations object {} PVC annotations
persistence.enabled bool true Enable persistence using PVC
persistence.existingClaim string "" Use existing PVC
persistence.size string "10Gi" PVC Storage Request
persistence.storageClass string "" (uses default StorageClass) Storage Class
podAnnotations object {} Annotations to add to the pod
podSecurityContext object {} Pod security context
replicaCount int 1 Number of S3Proxy replicas
resources object {} Resource limits and requests
securityContext object {} Container security context
service.annotations object {} Service annotations
service.port int 8080 Service port
service.targetPort int 8080 Target port (controls both the container port and S3Proxy bind port)
service.type string "ClusterIP" Kubernetes service type
serviceAccount.annotations object {} Annotations to add to the service account
serviceAccount.create bool false Specifies whether a service account should be created
serviceAccount.name string "" The name of the service account to use. If not set and create is true, a name is generated using the fullname template
tolerations list [] Tolerations for pod assignment

Usage Examples

Example 1: Filesystem Backend with Authentication

# values-filesystem.yaml config: authorization: "aws-v4" identity: "myaccesskey" credential: "mysecretkey" backend: provider: "filesystem-nio2" filesystem: basedir: "/data/s3proxy" persistence: enabled: true size: 20Gi

Install:

helm install s3proxy-fs ./s3proxy -f values-filesystem.yaml

Example 2: AWS S3 Backend

# values-aws-s3.yaml config: authorization: "aws-v4" identity: "proxy-access-key" # For clients connecting to s3proxy credential: "proxy-secret-key" backend: provider: "aws-s3" awsS3: region: "us-west-2" accessKeyId: "aws-access-key-id" # For s3proxy to connect to AWS secretAccessKey: "aws-secret-access-key" persistence: enabled: false # Not needed for S3 backend

Install:

helm install s3proxy-s3 ./s3proxy -f values-aws-s3.yaml

Example 3: Azure Blob Storage Backend

# values-azure.yaml config: authorization: "aws-v4" identity: "myaccesskey" credential: "mysecretkey" backend: provider: "azureblob" azureblob: account: "mystorageaccount" key: "storageaccountkey" persistence: enabled: false # Not needed for Azure backend

Install:

helm install s3proxy-azure ./s3proxy -f values-azure.yaml

Example 4: Google Cloud Storage Backend

# values-gcs.yaml config: authorization: "aws-v4" identity: "myaccesskey" credential: "mysecretkey" backend: provider: "google-cloud-storage" googleCloudStorage: projectId: "my-project" clientEmail: "service-account@my-project.iam.gserviceaccount.com" privateKey: |  -----BEGIN RSA PRIVATE KEY-----  ...  -----END RSA PRIVATE KEY-----  persistence: enabled: false # Not needed for GCS backend

Example 5: Anonymous Access (No Authentication)

# values-anonymous.yaml config: authorization: "none" backend: provider: "transient-nio2" # In-memory storage persistence: enabled: false

Testing the Installation

Once deployed, you can test S3Proxy using the AWS CLI:

# Get the service endpoint kubectl get svc # Port-forward for local testing kubectl port-forward svc/my-s3proxy 8080:8080 # Configure AWS CLI (if authentication is enabled) export AWS_ACCESS_KEY_ID=myaccesskey export AWS_SECRET_ACCESS_KEY=mysecretkey # Test S3 operations aws --endpoint-url http://localhost:8080 s3 ls aws --endpoint-url http://localhost:8080 s3 mb s3://test-bucket aws --endpoint-url http://localhost:8080 s3 cp test.txt s3://test-bucket/ aws --endpoint-url http://localhost:8080 s3 ls s3://test-bucket/

CORS Configuration

To enable CORS support:

config: cors: enabled: true allowOrigins: - "https://example.com" - "https://.+\\.example\\.com" allowMethods: - "GET" - "PUT" - "POST" - "HEAD" - "DELETE" allowHeaders: - "Accept" - "Content-Type" allowCredential: true

Middleware Configuration

S3Proxy supports various middlewares:

config: middlewares: readOnly: false # Make backend read-only eventualConsistency: true # Enable eventual consistency modeling shardedBackend: true # Enable sharded backend containers largeObjectMocking: false # Enable large object mocking

Bucket Configuration

Bucket Aliasing

Map virtual bucket names to actual backend buckets:

config: buckets: alias: virtual-bucket: "real-backend-bucket" another-bucket: "actual-bucket-name"

Bucket Locator

Assign specific buckets to different backends:

config: buckets: locator: - "bucket1" - "bucket2" - "*.test" # Glob patterns supported

Monitoring

Check S3Proxy logs:

kubectl logs deployment/my-s3proxy

Upgrading

helm upgrade my-s3proxy ./s3proxy -f my-values.yaml

Uninstalling

helm uninstall my-s3proxy

This will remove all resources created by the chart. If using persistence, the PVC will be retained by default.

Troubleshooting

Common Issues

  1. Authentication failures: Ensure config.identity and config.credential are set correctly for client authentication.

  2. Backend connection issues: Verify backend credentials are correctly configured in the appropriate section (e.g., config.backend.awsS3.*).

  3. Persistence issues: Check that your cluster has a default StorageClass or specify one explicitly.

  4. Port conflicts: If port 8080 is already in use, change service.port and service.targetPort.

References

License

This Helm chart is provided as-is. S3Proxy itself is licensed under the Apache License 2.0.


Autogenerated from chart metadata using helm-docs v1.14.2

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Packages

 
 
 

Contributors 2

  •  
  •  

Languages