Managing Package Updates Across a Mixed-Distro Fleet with Ansible

If your fleet is more than one machine, it’s probably more than one distro. Mine is: Ubuntu LTS boxes, Debian, Fedora, and — the interesting one — XCP-ng hypervisors. The goal is simple: one control node, a handful of playbooks, and the ability to answer two questions at any time: what updates are pending? and are these hosts fully patched?

This post builds the whole system from zero. Every file and command is copy-pasteable. The hostnames and IPs below are made up — swap in your own. At the end there’s a list of the quirks that actually cost me debugging time, so you can skip my detours.

The fleet we’re building for

Four hosts, four distros, all on a private network:

One machine is the control node (any Linux box — in my case a Fedora laptop). It runs Ansible and needs no root access at all.

Step 1 — Install Ansible in a venv

Keep Ansible out of the system packages. A venv means the toolchain is reproducible and the control node stays clean:

python3 -m venv ~/.venvs/ansible
~/.venvs/ansible/bin/pip install ansible ansible-lint

Every command in this post uses the venv binary: ~/.venvs/ansible/bin/ansible-playbook, .../ansible, .../ansible-lint.

Step 2 — Project structures

The whole system is a small git repo:

mkdir -p ~/update-management/{playbooks,group_vars,inventory,roles/system_updates/{tasks,templates}}
cd ~/update-management
git init

~/update-management/
├── ansible.cfg
├── playbooks/
│   ├── updates-report.yml       # read-only: what's pending
│   ├── updates-security.yml     # security updates only
│   └── updates-full.yml         # everything + marker-gated reboot
├── group_vars/
│   ├── all.yml                  # defaults (mode, reboot, timeout)
│   └── {ubuntu,debian,fedora,xcpng}.yml   # distro_id per group
├── roles/system_updates/
│   ├── tasks/main.yml           # assert distro, dispatch by package manager
│   ├── tasks/apt.yml
│   ├── tasks/dnf.yml
│   ├── tasks/yum.yml
│   ├── tasks/reboot.yml
│   └── templates/unattended-upgrades.j2
└── inventory/hosts.yml

Step 3 — ansible.cfg

[defaults]
inventory = inventory/hosts.yml
collections_path = ~/.venvs/ansible/lib/python3.12/site-packages/ansible_collections
host_key_checking = False

(Adjust collections_path to match your venv’s Python version — ls ~/.venvs/ansible/lib will show it. The venv’s ansible-lint needs this line to find its collections.)

Step 4 — Inventory

inventory/hosts.yml — note the pinned ansible_python_interpreter per host. Without it, Ansible runs a discovery task on every connection and spits warnings (and on XCP-ng it fails, see the quirks section):

all:
  children:
    ubuntu:
      hosts:
        ubuntu-01:
          ansible_host: 10.0.0.11
          ansible_user: admin
          ansible_python_interpreter: /usr/bin/python3
    debian:
      hosts:
        debian-01:
          ansible_host: 10.0.0.12
          ansible_user: admin
          ansible_python_interpreter: /usr/bin/python3
    fedora:
      hosts:
        fedora-01:
          ansible_host: 10.0.0.13
          ansible_user: admin
          ansible_python_interpreter: /usr/bin/python3
    xcpng:
      hosts:
        hypervisor-01:
          ansible_host: 10.0.0.14
          ansible_user: root
          ansible_python_interpreter: /opt/python3.11/bin/python3.11

One gotcha that will bite you on the first run: playbooks only find group_vars when the folder sits next to the inventory file (ad-hoc commands find the CWD copy; playbooks don’t). Fix it once with a symlink:

ln -s ../group_vars inventory/group_vars

Step 5 — group_vars

group_vars/all.yml — the defaults every playbook can override:

system_updates_mode: security     # security | full | report
system_updates_reboot: true
system_updates_reboot_timeout: 600
system_updates_enabled: true

One file per distro group (e.g. group_vars/ubuntu.yml):

distro_id: ubuntu

All role variables carry the system_updates_ prefix — that’s a lint rule, not a style preference. Lint catching a typo before a 4-host run is the whole point.

Step 6 — The role

One role, dispatched by distribution.

roles/system_updates/tasks/main.yml:

- name: Assert known distribution
  ansible.builtin.assert:
    that:
      - ansible_facts.distribution in ['Ubuntu', 'Debian', 'Fedora', 'XCP-ng']
    fail_msg: "Unsupported distribution: {{ ansible_facts.distribution }}"

- name: Include distro-specific update tasks
  ansible.builtin.include_tasks: "{{ task_file }}"
  vars:
    task_file: >-
      {{ 'yum.yml' if ansible_facts.distribution == 'XCP-ng'
         else ('dnf.yml' if ansible_facts.pkg_mgr in ['dnf', 'dnf5']
               else 'apt.yml') }}

- name: Check reboot marker
  ansible.builtin.stat:
    path: "{{ '/run/reboot-required' if system_updates_reboot else '/nonexistent' }}"
  register: reboot_marker

- name: Reboot host
  ansible.builtin.reboot:
    reboot_timeout: "{{ system_updates_reboot_timeout }}"
  when: reboot_marker.stat.exists

Two structural rules worth internalizing:

1. The reboot check is a single stat task. If you split it into two tasks sharing one register variable, the skipped task can clobber the result with a dict that lacks the stat key — and your condition is silently false.
2. Vars are set at play level, not in include_role — include_role doesn’t accept a vars parameter. Discovering that the hard way is fun.

roles/system_updates/tasks/apt.yml — note the task order: template the config before installing the package. A broken config file blocks the install task itself:

- name: List pending packages (report)
  ansible.builtin.shell: >-
    (apt list --upgradable 2>/dev/null || true) | awk -F/ 'NR>1 {print $1 " " $3}'
  register: apt_pending
  changed_when: false
  check_mode: false
  when: system_updates_mode == 'report'

- name: Configure unattended-upgrades
  ansible.builtin.template:
    src: unattended-upgrades.j2
    dest: /etc/apt/apt.conf.d/50unattended-upgrades
  when: system_updates_mode == 'security'

- name: Install unattended-upgrades
  ansible.builtin.apt:
    name: unattended-upgrades
    state: present
  when: system_updates_mode == 'security'

- name: Apply security updates
  ansible.builtin.command: unattended-upgrade
  when: system_updates_mode == 'security'

- name: Apply all updates
  ansible.builtin.apt:
    upgrade: dist
  when: system_updates_mode == 'full'

roles/system_updates/tasks/dnf.yml:

- name: List pending packages (report)
  ansible.builtin.command: dnf list --upgradable
  register: dnf_pending
  changed_when: false
  check_mode: false
  when: system_updates_mode == 'report'

- name: Apply security updates
  ansible.builtin.dnf:
    security: true
  when: system_updates_mode == 'security'

- name: Apply all updates
  ansible.builtin.dnf:
  when: system_updates_mode == 'full'

roles/system_updates/tasks/yum.yml — XCP-ng only. The command module drives the yum CLI because the dnf/yum Ansible modules don’t work there (see quirks):

- name: List pending packages (report)
  ansible.builtin.command: yum check-update
  register: yum_pending
  changed_when: false
  check_mode: false
  when: system_updates_mode == 'report'

- name: Apply security updates
  ansible.builtin.command: yum -y update --security
  when: system_updates_mode == 'security'
  register: yum_sec
  changed_when: "'Updated:' in yum_sec.stdout"

- name: Apply all updates
  ansible.builtin.command: yum -y update
  when: system_updates_mode == 'full'
  register: yum_full
  changed_when: "'Updated:' in yum_full.stdout"

roles/system_updates/templates/unattended-upgrades.j2 — the brace syntax matters (quirk #5), and it runs plain, no flags:

Unattended-Upgrade::Allowed-Origins {
    "{{ ansible_facts.lsb.dist_id }} {{ ansible_facts.lsb.codename }}-security";
};
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Mail "off";

Step 7 — The three playbooks

playbooks/updates-report.yml — the daily driver. Read-only, safe to run against the whole fleet:

- name: Report pending updates (read-only)
  hosts: all
  become: true
  vars:
    system_updates_mode: report
    system_updates_reboot: false
  tasks:
    - ansible.builtin.include_role:
        name: system_updates

playbooks/updates-security.yml:

- name: Apply security updates
  hosts: all
  become: true
  serial: 1
  vars:
    system_updates_mode: security
    system_updates_reboot: false
  tasks:
    - ansible.builtin.include_role:
        name: system_updates

playbooks/updates-full.yml:

- name: Apply all updates + marker-gated reboot
  hosts: all
  become: true
  serial: 1
  vars:
    system_updates_mode: full
  tasks:
    - ansible.builtin.include_role:
        name: system_updates

Step 8 — SSH keys and sudo (one-time, per host)

# 1. Key auth
ssh-copy-id -o StrictHostKeyChecking=accept-new admin@10.0.0.11

# 2. Check for passwordless sudo
ssh admin@10.0.0.11 'sudo -n true'

If step 2 prompts for a password, run this on the target (as admin):

sudo bash -c "echo 'admin ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/admin && chmod 440 /etc/sudoers.d/admin"

On the XCP-ng host, also install the static Python first (quirk #17): grab a python-build-standalone release (cpython-3.11.x+TAG-x86_64-unknown-linux-gnu-install_only.tar.gz), extract the tree to /opt/python3.11, keep the /install symlink the binary expects, and point ansible_python_interpreter at /opt/python3.11/bin/python3.11 (as in the inventory above). No system Python changes, fully reversible.

Step 9 — First run

cd ~/update-management
~/.venvs/ansible/bin/ansible all -m ping
~/.venvs/ansible/bin/ansible-playbook playbooks/updates-report.yml

ping succeeds on all four hosts and the report lists pending packages without changing anything — you’re done building. From here on, the two questions from the intro are one command each.

Step 10 — Updating, and the marker-gated reboot

Rebooting a hypervisor means VM downtime — the management daemon shuts every VM down before the kernel reboots. VMs with guest tools installed shut down gracefully; VMs without them get force-stopped. So a blanket “update and reboot” playbook is a footgun.

The pattern decouples the two steps with a one-shot approval marker:

# 1. Approve the reboot (one-shot)
ssh hypervisor-01 'touch /run/reboot-required'

# 2. Run the update playbook against that host
~/.venvs/ansible/bin/ansible-playbook playbooks/updates-full.yml -l hypervisor-01

# 3. Clean up afterwards, so future runs don't auto-reboot
ssh hypervisor-01 'rm /run/reboot-required'

This gives you a nice property on hypervisors: after updates-full without the marker, the new kernel and hypervisor sit dormant while the VMs keep running on the old one. You can verify the pending state anytime:

ssh hypervisor-01 'uname -r; rpm -q kernel xen-hypervisor'
# running kernel ≠ installed kernel = reboot pending

The quirks that actually bite

These are the things that cost real debugging time, in roughly the order I hit them:

Two of these deserve a paragraph:

XCP-ng is YUM 3 in disguise. Despite ansible_facts.pkg_mgr reporting dnf, the userland is classic YUM with the fastestmirror plugin, and there’s no dnf Python module in any interpreter (stock Python 3.6, platform Python 2.7). The dnf/yum Ansible modules simply don’t work. The working pattern is the command module driving yum -y update [--security] with changed_when: "'Updated:' in stdout", dispatched on ansible_facts.distribution == 'XCP-ng'.

Old Python on hypervisors. Stock Python 3.6 can’t run recent ansible-core modules. The fix is a python-build-standalone build extracted to /opt/python3.11 (with the /install symlink the binary expects) and ansible_python_interpreter pointed at it. No system Python changes, fully reversible.

And one I’d call a feature you have to understand: apt phasing on Ubuntu 24.04+. Freshly published updates are “deferred due to phasing” — the update task correctly reports changed=0 while apt list --upgradable still shows them. They install on a later run once the host’s phase is reached. You can force them with -o APT::Get::Always-Include-Phased-Updates=true, but don’t bake that into the playbook — you’d defeat the canary rollout that protects the fleet.

Verifying “fully patched”

The end state is checkable per distro, no playbook required:

# Fedora
ssh fedora-01 'dnf check-update'            # empty = done

# Ubuntu / Debian
ssh ubuntu-01 'apt list --upgradable | grep -c noble-security'   # 0 = security done

# XCP-ng
ssh hypervisor-01 'yum history list'           # transaction with the package count
ssh hypervisor-01 'yum check-update --security' # empty = security done

The change protocol

Every change to the playbooks follows the same five steps, no exceptions:

1. Edit the relevant file
2. ~/.venvs/ansible/bin/ansible-lint — must pass the production profile
3. ~/.venvs/ansible/bin/ansible-playbook playbooks/*.yml --syntax-check
4. Test on one host with -l <host> (report playbook first)
5. git commit

The whole system is a git repo, which means “what did the playbook do last Tuesday” has an answer. That’s the part of this setup I’d replicate first in any new project.