So I've recently ran into weird problem, my jenkins build fails every odd build for no reason. Build is simple, it runs ansible playbook which in turns deploys code to remote machines, clears caches, runs maintenance, etc.
Build basically runs this:
ansible-playbook -i ansible/hosts ansible/backend.yml Then, when the build number is odd, it fails on this step(runs with force=yes so local modifications should be no problem):
TASK [backend : Clone backend repo] ******************************************** task path: /opt/provision/ansible/roles/backend/tasks/main.yml:5 Using module file /usr/lib/python2.6/site-packages/ansible/modules/core/source_control/git.py <server_ip_address> ESTABLISH SSH CONNECTION FOR USER: username <server_ip_address> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/var/lib/jenkins/.ssh/jenkins"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=username -o ConnectTimeout=10 -o ConnectionAttempts=20 -o ControlPath=/var/lib/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r server_ip_address '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499 `" && echo ansible-tmp-1485853855.83-181127458563499="` echo $HOME/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499 `" ) && sleep 0'"'"'' <server_ip_address> PUT /tmp/tmpdgsmCK TO /home/username/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499/git.py <server_ip_address> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/var/lib/jenkins/.ssh/jenkins"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=username -o ConnectTimeout=10 -o ConnectionAttempts=20 -o ControlPath=/var/lib/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r '[server_ip_address]' <server_ip_address> ESTABLISH SSH CONNECTION FOR USER: username <server_ip_address> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/var/lib/jenkins/.ssh/jenkins"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=username -o ConnectTimeout=10 -o ConnectionAttempts=20 -o ControlPath=/var/lib/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r server_ip_address '/bin/sh -c '"'"'chmod u+x /home/username/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499/ /home/username/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499/git.py && sleep 0'"'"'' <server_ip_address> ESTABLISH SSH CONNECTION FOR USER: username <server_ip_address> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/var/lib/jenkins/.ssh/jenkins"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=username -o ConnectTimeout=10 -o ConnectionAttempts=20 -o ControlPath=/var/lib/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r -tt server_ip_address '/bin/sh -c '"'"'/usr/bin/python /home/username/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499/git.py; rm -rf "/home/username/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499/" > /dev/null 2>&1 && sleep 0'"'"'' fatal: [backend]: FAILED! => { "changed": false, "failed": true, "invocation": { "module_name": "git" }, "module_stderr": "OpenSSH_6.6.1, OpenSSL 1.0.1k-fips 8 Jan 2015\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 56: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 23456\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 4\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to server_ip_address closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_lVW4UB/ansible_module_git.py\", line 1040, in <module>\r\n main()\r\n File \"/tmp/ansible_lVW4UB/ansible_module_git.py\", line 994, in main\r\n result.update(changed=True, after=remote_head, msg='Local modifications exist')\r\nUnboundLocalError: local variable 'remote_head' referenced before assignment\r\n", "msg": "MODULE FAILURE" } to retry, use: --limit @/opt/provision/ansible/backend.retry PLAY RECAP ********************************************************************* backend : ok=2 changed=1 unreachable=0 failed=1 Traceback formated:
Traceback (most recent call last): File \"/tmp/ansible_lVW4UB/ansible_module_git.py\", line 1040, in <module> main() File \"/tmp/ansible_lVW4UB/ansible_module_git.py\", line 994, in main result.update(changed=True, after=remote_head, msg='Local modifications exist') UnboundLocalError: local variable 'remote_head' referenced before assignment Failing step:
- name: Clone backend git: > repo="{{ backend_repo }}" dest="{{ backend_path }}" accept_hostkey=yes key_file="{{ backend_key }}" clone=yes force=yes update=yes version={{ backend_branch }} If you run build immediately after it fails, it builds nicely. Also it builds every time I run it from command line.
