Is it possible to rerun kubernetes job?

Question

I have the following Kubernetes Job configuration:

--- apiVersion: batch/v1 kind: Job metadata: name: dbload creationTimestamp: spec: template: metadata: name: dbload spec: containers: - name: dbload image: sdvl3prox001:7001/pbench/tdload command: ["/opt/pbench/loadTpcdsData.sh", "qas0063", "dbc", "dbc", "1"] restartPolicy: Never imagePullSecrets: - name: pbenchregkey status: {} 

When I do kubectl create -f dbload-deployment.yml --record the job and a pod are created, Docker container runs to completion and I get this status:

$ kubectl get job dbload NAME DESIRED SUCCESSFUL AGE dbload 1 1 1h $ kubectl get pods -a NAME READY STATUS RESTARTS AGE dbload-0mk0d 0/1 Completed 0 1h 

This job is one time deal and I need to be able to rerun it. If I attempt to rerun it with kubectl create command I get this error

$ kubectl create -f dbload-deployment.yml --record Error from server: error when creating "dbload-deployment.yml": jobs.batch "dbload" already exists 

Of course I can do kubectl delete job dbload and then run kubectl create but I'm wondering if I can somehow re-awaken the job that already exists?

Answer 1

No. There is definitely no way to rerun a kubernetes job. You need to delete it first.

You can however create a new job based on existing job configuration that will be executed immediately:

kubectl create job --from=job/dbload new-dbload -n <namespace>

Answer 2

Simulate a rerun by replacing the job with itself:

1. Backup your job:

• kubectl get job "your-job" -o json > your-job.json

2. Replace the job in place:

• kubectl get job "your-job" -o json | kubectl replace --force -f -

If you get errors due to auto-generated labels or selectors, you can delete or edit them with jq:

• kubectl get job "your-job" -o json | jq 'del(.spec.selector)' | jq 'del(.spec.template.metadata.labels)' | kubectl replace --force -f -

UPDATED with Jeremy Huiskamp's suggestion

Answer 3

You can also avoid the error you mentioned by specifying

 generateName: dbload 

instead of simply name

In that case, each job you submit with this yaml file will have a unique name that will look something like dbloada1b2c. Then you can decide whether you need to delete the old jobs, but you won't have to do it.

Here is a working yaml example:

apiVersion: batch/v1 kind: Job metadata: generateName: netutils- spec: parallelism: 1 template: spec: containers: - image: amouat/network-utils name: netutil restartPolicy: Never 

This is the output from kubectl get job after two kubectl create -f example.yaml commands:

NAME COMPLETIONS DURATION AGE netutils-5bs2s 0/1 14s 14s netutils-dsvfk 0/1 10s 10s 

Answer 4

As an improvement on @F. Santiago's idea, you can simply use the value stored at the annotation "kubectl.kubernetes.io/last-applied-configuration" that holds the initial applied configuration without any auto generated field:

kubectl get job <jobname> -o json | \ jq -r '.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration"' | \ kubectl replace --save-config --force -f - 

Note: for kubectl replace, remember to pass --save-config so it updates the annotation field with the last config applied.

Answer 5

There is no way to run a job that completed but you can simulat a rerun by doing the following

1. Get yaml file of the existing job:

   kubectl get job <job_name> -o yaml > <job_name>.yaml 

2. Delete the existing job:

   kubectl delete job <job_name> 

3. Run the job again:

   kubectl apply -f <job_name>.yaml 

Answer 6

Based on @Marcelo's idea I made it work with the following, without any processing of the template:

kubectl get job <job-name> -o custom-columns=:metadata.annotations.kubectl\.kubernetes\.io/last-applied-configuration > job.json kubectl delete -f job.json kubectl apply -f job.json 

Please note the escaped dots (\.) in the annotation name: kubectl\.kubernetes\.io/last-applied-configuration. Without it, it returns <none>.

Answer 7

I've implemented F. Santiago's method using a kubectl plugin. I have these two files in my PATH [1] so that kubectl picks it up.

Now issuing kubectl replacejob [job name] looks like this:

[w] eddie@eddie ~ $ kubectl replacejob my_job_name Writing out backup file: my_job_name.bak.json job.batch "my_job_name" deleted job.batch/my_job_name replaced 

[1] Files to make the plugin work:

kubectl-replacejob.cmd: Simple wrapper to call python with the same args

@echo off pushd . cd %~dp0 python kubectl-replacejob.py %* popd 

kubectl-replacejob.py: Does the 'hard' work of replacement.

import sys import subprocess import json import io if len( sys.argv ) < 2: print("Error: please specify the job you wish to replace.") sys.exit(-1) job_name = sys.argv[1] # Fetch the job as json job_as_json_text = subprocess.check_output(f'kubectl get job {job_name} -o json', shell=True).decode() job_as_json = json.loads(job_as_json_text) # Save out a backup backup_file = f'{job_name}.bak.json' print(f"Writing out backup file: {backup_file}") with open(backup_file, 'w') as f: f.write(job_as_json_text) # Remove references to controller uid that borks replace def remove_key_if_present(obj, *keys): for i in range(len(keys)): key = keys[i] if key in obj: if i == len(keys)-1: del obj[key] else: obj = obj[key] else: print(f"WARNING: Failed to remove {'.'.join(keys)}: failed finding key at {key}!") return remove_key_if_present(job_as_json, 'spec', 'selector', 'matchLabels', 'controller-uid') remove_key_if_present(job_as_json, 'spec', 'template', 'metadata', 'labels', 'controller-uid') job_as_json_text = json.dumps(job_as_json) # Pretty print for testing #print(json.dumps(job_as_json, indent=4, sort_keys=True)) # Issue the replace subprocess.run(f'kubectl replace --force -f -', shell=True, input=job_as_json_text.encode()) 

Answer 8

On the subject of deleting a Job before you can run it again, as the accepted answer states, note that you can now also set up a Job to automatically clean itself after completion with .spec.ttlSecondsAfterFinished