DEV Community

Purushotam Adhikari
Purushotam Adhikari

Posted on

Automating Linux Updates Across Multiple Servers with Ansible

#ansible #automation #devops #infrastructure

Managing updates across multiple Linux servers can be a time-consuming and error-prone task when done manually. Ansible provides an elegant solution to automate this process, ensuring consistency, reducing human error, and saving valuable time. In this comprehensive guide, we'll explore how to set up automated Linux updates using Ansible, covering everything from basic configurations to advanced strategies.

Why Automate Server Updates?

Before diving into the technical implementation, let's understand why automation is crucial:

  • Consistency: Ensures all servers receive the same updates in the same manner
  • Time Efficiency: Eliminates the need to SSH into each server individually
  • Reduced Human Error: Minimizes mistakes that can occur during manual updates
  • Scheduling: Allows updates during maintenance windows
  • Logging and Reporting: Provides detailed logs of what was updated and when
  • Rollback Capabilities: Enables quick recovery if issues arise

Prerequisites

To follow this guide, you'll need:

  • Ansible installed on your control machine
  • SSH access to target servers
  • Sudo privileges on target servers
  • Basic understanding of YAML syntax
  • Target servers running supported Linux distributions (Ubuntu, CentOS, RHEL, Debian)

Setting Up Your Ansible Environment

1. Install Ansible

On Ubuntu/Debian:

sudo apt update sudo apt install ansible
Enter fullscreen mode Exit fullscreen mode

On CentOS/RHEL:

sudo yum install epel-release sudo yum install ansible
Enter fullscreen mode Exit fullscreen mode

2. Configure SSH Key Authentication

Generate SSH keys and copy them to your target servers:

ssh-keygen -t rsa -b 4096 ssh-copy-id user@server1.example.com ssh-copy-id user@server2.example.com
Enter fullscreen mode Exit fullscreen mode

3. Create Your Inventory File

Create an inventory file (hosts.yml) to define your servers:

all: children: production: hosts: web-server-1: ansible_host: 192.168.1.10 ansible_user: ubuntu web-server-2: ansible_host: 192.168.1.11 ansible_user: ubuntu db-server-1: ansible_host: 192.168.1.20 ansible_user: centos staging: hosts: staging-web: ansible_host: 192.168.1.50 ansible_user: ubuntu development: hosts: dev-server: ansible_host: 192.168.1.60 ansible_user: ubuntu
Enter fullscreen mode Exit fullscreen mode

Basic Update Playbook

Let's start with a simple playbook that updates all packages on Ubuntu/Debian systems:

--- - name: Update Linux servers hosts: all become: yes gather_facts: yes tasks: - name: Update apt cache (Ubuntu/Debian) apt: update_cache: yes cache_valid_time: 3600 when: ansible_os_family == "Debian" - name: Upgrade all packages (Ubuntu/Debian) apt: upgrade: dist autoremove: yes autoclean: yes when: ansible_os_family == "Debian" register: apt_upgrade_result - name: Update yum cache (CentOS/RHEL) yum: update_cache: yes when: ansible_os_family == "RedHat" - name: Upgrade all packages (CentOS/RHEL) yum: name: "*" state: latest when: ansible_os_family == "RedHat" register: yum_upgrade_result - name: Check if reboot is required (Ubuntu/Debian) stat: path: /var/run/reboot-required register: reboot_required_file when: ansible_os_family == "Debian" - name: Display upgrade results debug: msg: "{{ apt_upgrade_result.stdout_lines if ansible_os_family == 'Debian' else yum_upgrade_result.results }}"
Enter fullscreen mode Exit fullscreen mode

Save this as update-servers.yml and run it with:

ansible-playbook -i hosts.yml update-servers.yml
Enter fullscreen mode Exit fullscreen mode

Advanced Update Strategies

1. Rolling Updates with Error Handling

For production environments, you'll want to update servers in batches to maintain service availability:

--- - name: Rolling server updates hosts: production become: yes gather_facts: yes serial: 2 # Update 2 servers at a time max_fail_percentage: 10 # Stop if more than 10% fail pre_tasks: - name: Check server connectivity ping: - name: Verify disk space shell: df -h / | awk 'NR==2 {print $5}' | sed 's/%//' register: disk_usage failed_when: disk_usage.stdout|int > 90 tasks: - name: Create backup directory file: path: /backup/pre-update-{{ ansible_date_time.date }} state: directory mode: '0755' - name: Backup package list (Ubuntu/Debian) shell: dpkg --get-selections > /backup/pre-update-{{ ansible_date_time.date }}/packages.list when: ansible_os_family == "Debian" - name: Backup package list (CentOS/RHEL) shell: rpm -qa > /backup/pre-update-{{ ansible_date_time.date }}/packages.list when: ansible_os_family == "RedHat" - name: Update package cache package: update_cache: yes retries: 3 delay: 5 - name: Upgrade all packages package: name: "*" state: latest register: upgrade_result notify: - Check if reboot required - name: Remove unused packages (Ubuntu/Debian) apt: autoremove: yes purge: yes when: ansible_os_family == "Debian" handlers: - name: Check if reboot required stat: path: /var/run/reboot-required register: reboot_required notify: Conditional reboot - name: Conditional reboot reboot: reboot_timeout: 300 pre_reboot_delay: 5 when: reboot_required.stat.exists | default(false) post_tasks: - name: Verify services are running service: name: "{{ item }}" state: started loop: - ssh - cron ignore_errors: yes - name: Send notification mail: to: admin@example.com subject: "Server {{ inventory_hostname }} updated successfully" body: "Updates completed at {{ ansible_date_time.iso8601 }}" when: upgrade_result.changed delegate_to: localhost
Enter fullscreen mode Exit fullscreen mode

2. Scheduled Updates with Maintenance Windows

Create a playbook that respects maintenance windows:

--- - name: Scheduled maintenance updates hosts: all become: yes gather_facts: yes vars: maintenance_start: "02:00" maintenance_end: "04:00" current_time: "{{ ansible_date_time.hour }}:{{ ansible_date_time.minute }}" tasks: - name: Check if we're in maintenance window set_fact: in_maintenance_window: "{{ (current_time >= maintenance_start) and (current_time <= maintenance_end) }}" - name: Skip updates outside maintenance window debug: msg: "Skipping updates - outside maintenance window ({{ maintenance_start }} - {{ maintenance_end }})" when: not in_maintenance_window - name: Proceed with updates block: - name: Update repositories package: update_cache: yes - name: Install security updates only package: name: "*" state: latest security: yes when: ansible_os_family == "RedHat" - name: Install unattended-upgrades for security updates (Ubuntu/Debian) apt: name: unattended-upgrades state: present when: ansible_os_family == "Debian" when: in_maintenance_window
Enter fullscreen mode Exit fullscreen mode

3. Selective Updates with Package Exclusions

Sometimes you need to exclude certain packages from updates:

--- - name: Selective package updates hosts: all become: yes gather_facts: yes vars: excluded_packages: - kernel* - docker* - mysql* tasks: - name: Update all packages except excluded ones (Ubuntu/Debian) apt: upgrade: safe update_cache: yes when: ansible_os_family == "Debian" - name: Hold excluded packages (Ubuntu/Debian) dpkg_selections: name: "{{ item }}" selection: hold loop: "{{ excluded_packages }}" when: ansible_os_family == "Debian" - name: Update non-excluded packages (CentOS/RHEL) yum: name: "*" state: latest exclude: "{{ excluded_packages | join(',') }}" when: ansible_os_family == "RedHat"
Enter fullscreen mode Exit fullscreen mode

Monitoring and Reporting

Update Status Report Playbook

Create a comprehensive reporting system:

--- - name: Generate update report hosts: all become: yes gather_facts: yes tasks: - name: Check for available updates (Ubuntu/Debian) shell: apt list --upgradable 2>/dev/null | grep -v "WARNING" | wc -l register: available_updates_debian when: ansible_os_family == "Debian" changed_when: false - name: Check for available updates (CentOS/RHEL) shell: yum check-update | grep -E "^[a-zA-Z]" | wc -l register: available_updates_redhat when: ansible_os_family == "RedHat" changed_when: false failed_when: false - name: Check last update time stat: path: /var/log/apt/history.log register: apt_history when: ansible_os_family == "Debian" - name: Check system uptime shell: uptime -s register: system_uptime changed_when: false - name: Generate report template: src: update_report.j2 dest: /tmp/update_report_{{ inventory_hostname }}.txt delegate_to: localhost vars: available_updates: "{{ available_updates_debian.stdout if ansible_os_family == 'Debian' else available_updates_redhat.stdout }}" last_boot: "{{ system_uptime.stdout }}"
Enter fullscreen mode Exit fullscreen mode

Create a Jinja2 template (update_report.j2):

Server Update Report =================== Server: {{ inventory_hostname }} OS: {{ ansible_distribution }} {{ ansible_distribution_version }} Architecture: {{ ansible_architecture }} Last Boot: {{ last_boot }} Available Updates: {{ available_updates }} Generated: {{ ansible_date_time.iso8601 }} {% if available_updates|int > 0 %} WARNING: {{ available_updates }} updates available {% else %} Status: System up to date {% endif %} System Information: - Memory: {{ ansible_memtotal_mb }}MB - CPU Cores: {{ ansible_processor_vcpus }} - Disk Usage: {{ ansible_mounts[0].size_available }} bytes available
Enter fullscreen mode Exit fullscreen mode

Best Practices and Tips

1. Testing Strategy

Always test your playbooks in a development environment first:

# Test syntax ansible-playbook --syntax-check update-servers.yml # Dry run ansible-playbook -i hosts.yml update-servers.yml --check # Run on development servers first ansible-playbook -i hosts.yml update-servers.yml --limit development
Enter fullscreen mode Exit fullscreen mode

2. Backup Strategy

Always create backups before major updates:

- name: Create system snapshot (if using LVM) shell: lvcreate -L1G -s -n snapshot-{{ ansible_date_time.date }} /dev/vg0/root when: ansible_lvm is defined ignore_errors: yes
Enter fullscreen mode Exit fullscreen mode

3. Logging and Auditing

Configure comprehensive logging:

- name: Log update activity lineinfile: path: /var/log/ansible-updates.log line: "{{ ansible_date_time.iso8601 }} - Updates applied by {{ ansible_user_id }}" create: yes
Enter fullscreen mode Exit fullscreen mode

4. Error Handling

Implement robust error handling:

tasks: - name: Update packages package: name: "*" state: latest register: update_result failed_when: false - name: Handle update failures debug: msg: "Update failed on {{ inventory_hostname }}: {{ update_result.msg }}" when: update_result.failed - name: Continue with next server meta: clear_host_errors when: update_result.failed
Enter fullscreen mode Exit fullscreen mode

Automation with Cron

Set up automated execution using cron:

# Add to crontab for weekly updates 0 2 * * 0 /usr/bin/ansible-playbook -i /path/to/hosts.yml /path/to/update-servers.yml >> /var/log/ansible-cron.log 2>&1
Enter fullscreen mode Exit fullscreen mode

Conclusion

Automating Linux updates with Ansible provides numerous benefits including consistency, reliability, and time savings. The examples provided in this guide offer a solid foundation that you can customize based on your specific requirements.

Key takeaways:

  • Start with simple playbooks and gradually add complexity
  • Always test in development environments first
  • Implement proper backup and rollback strategies
  • Use rolling updates for production environments
  • Monitor and log all update activities
  • Respect maintenance windows and business requirements

Remember that automation is not set-and-forget. Regularly review and update your playbooks, monitor their execution, and stay informed about security advisories for your systems.

The investment in setting up automated updates pays dividends in reduced manual work, improved security posture, and more reliable infrastructure management.

Have you implemented automated updates in your environment? Share your experiences and tips in the comments below!

Resources

Top comments (0)