Is there a way in Kubernetes to trigger a job to run on my containers on demand?
The use case is to be able to have the containers sync from S3 to a persistent volume on demand. In the application's current architecture, new content is added to S3 and is synced to a group of servers that are managed with Chef. I need to have a way for a team to be able to trigger this functionality in the containers running on Kubernetes.
Actually, Kubernetes represents a flexible mechanism for automated task workloads such as one time processes Jobs and periodic processes like CronJobs, which are included in batch/v1 Kubernetes API object model, therefore solution from @Tim is quite decent for me.
I assume that it might be possible for you to spin up some pod with a aws-cli on a board in order to trigger sync action between mapped PVC into this container and target S3 storage. For that purpose you can build own image ported with necessary binary, or i.e. use ready solution like docker-kubectl-awscli image maintained by @Expert360.
The following Job will execute aws s3 sync command inside particular container and launch sync action between target parties:
apiVersion: batch/v1 kind: Job metadata: name: backup-s3 spec: template: spec: containers: - name: kubectl-aws image: expert360/kubectl-awscli:v1.11.2 env: - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: aws-s3-key key: aws-access-key-id - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: aws-s3-access-key key: aws-secret-access-key command: [ "/bin/bash", "-c", "aws s3 sync s3://my-bucket/ /data/backup" ] volumeMounts: - name: backup-aws mountPath: /data/backup volumes: - name: backup-aws persistentVolumeClaim: claimName: backup-aws-claim restartPolicy: Never You have to supply aws-cli with corresponding AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables stored in particular Kubernetes Secret objects.
Take a look into the project : https://github.com/nuvo/skbn. It can also use kube2iam functionalities
