How to monitor particular kit with Oversight: which sensor to use, where on the device to set up access, which user function and extraction rows read the response, and how to write the rules that turn it into an alarm. Search for a vendor, a product or an error message, or ask a question in your own words.
Package agent: patching and change tracking for Linux hosts
The Oversight package agent, oversight-packages, is a small program on each Linux host that tells Oversight what is installed, what updates are waiting, what has changed and who changed it. The hosts appear on the Packages page, where the whole estate can be searched in one go and any host, group or search result can be told to upgrade, remove or reboot, with each host reporting back what it did.
It opens no port and needs no SSH. Every host calls out to Oversight over HTTPS, and instructions come back in the reply. It can upgrade or remove software from repositories the host already trusts, refresh their metadata and reboot; it cannot install anything new, run a command or script, add a repository or import a key.
What it supports
| Family | Covers | Package manager |
|---|---|---|
| EL | AlmaLinux, Rocky, RHEL, Oracle Linux | dnf |
| Debian | Debian and Ubuntu, old releases and new | apt and dpkg |
| Arch | Arch, Manjaro, EndeavourOS | pacman |
The host needs systemd, an x86_64 or arm64 processor, and outbound HTTPS to Oversight. Nothing needs to reach the host.
How a host joins
The agent is installed with one command, as root. It checks its own build against a published checksum, makes a key for the host that is its identity from then on, and turns off the distribution's own automatic update timers, so nothing updates the host behind Oversight's back. It refreshes the repository metadata itself once a day, at a time picked at random for each host so the mirrors do not see the whole estate at once. Installing it again upgrades the agent and keeps the host as it was.
Until it is enrolled, the host sends only its name, every five minutes. To enrol it, a user with the Patching permission opens Packages > Add host and enters the 16-digit host key the install printed. The host joins the tenant that user is signed in to, and sends its full report within a few minutes.
There is no shared secret in the install command and no tenant to choose on the host. A host that has never been enrolled can do nothing but say its name.
Using it
Two permissions apply: Packages to see a tenant's hosts, and Patching to act on them. From a host's page, a group or a search, a Patching user can:
- upgrade named packages, ticked packages or everything waiting;
- remove packages, after seeing what depends on them;
- reboot, always or only if the host says it needs one. Every upgrade form has a "then reboot if needed" box, and the reboot waits until the upgrade has finished and been reported;
- refresh the repository metadata.
Every instruction comes back with its outcome, the transaction it made and the package manager's output. A reboot counts as done only when the host checks in again on a new boot. While instructions are in flight the host checks in every minute.
For 60 minutes after the agent starts, it takes no reboot. A reboot sent in that time shows as busy, "held", and is sent again every ten minutes until it runs. However many reboots are sent, a host reboots at most once an hour.
Limits the host sets for itself
Each host has a short config file of its own, read by the agent and out of Oversight's reach, so nothing on the Oversight side can change it. In it, upgrade, remove and reboot can each be switched off for that host, and an instruction it rules out is refused, with the reason shown on the host's page.
It also holds a list of packages the agent must never remove on that host, on top of its built-in list (below), and can say that those packages are not to be upgraded either, so the agent never touches them at all. An upgrade of everything then upgrades everything else.
Someone very cautious can leave only upgrades on. With removal and reboot switched off, even a compromised Oversight could do no more than upgrade packages from repositories the host already trusts.
A mistake in the file, such as a misspelt setting, stops the agent rather than being ignored, since an ignored mistake would leave on something you meant to turn off. The host then turns red on the Packages page, so it is noticed.
What it never removes
Whatever its config says, the agent refuses to remove these, or anything whose removal would take one of them with it. The host enforces this itself and does not rely on the page, so a host can never be left without the means to reach it or repair it:
| Family | Never removed |
|---|---|
| EL | dnf, yum, rpm, systemd, systemd-udev, openssh-server, glibc, bash, coreutils, NetworkManager, sudo, setup, and everything in /etc/dnf/protected.d |
| Debian | apt, dpkg, libc6, libc-bin, systemd, systemd-sysv, init, openssh-server, bash, coreutils, sudo, login, passwd, base-files, ifupdown, network-manager, netplan.io, proxmox-ve, pve-manager, every Essential package, and the running kernel |
| Arch | pacman, glibc, systemd, systemd-libs, openssh, bash, coreutils, sudo, filesystem, base, iproute2, networkmanager, the keyrings, pacman.conf's HoldPkg list, and the running kernel |
These are protected from removal only. They are still upgraded, since that is where most security fixes land.
Proxmox VE and Ceph nodes
A Proxmox node carries every guest on it, so its Proxmox packages are best upgraded by hand, one node at a time, with the cluster in mind. Oversight already holds Proxmox packages back from its own bulk upgrades and never reboots a Proxmox node, and the node can enforce the same itself, whatever it is sent: its Proxmox packages never upgraded or removed, nothing removed at all, and no reboot. Ordinary Debian packages (openssl, curl, the SSH server) are still upgraded.
A Ceph upgrade has an order to it (monitors, then managers, then OSDs, then metadata servers), and an OSD node rebooted at the wrong moment can leave placement groups degraded. A Ceph node can keep the agent away from Ceph entirely in the same way, while everything else on it is still upgraded. On a Proxmox node that also runs Ceph, both apply together.
Installing it
As root, or with sudo:
The install script picks the build for the machine, checks it against its published checksum, and:
- installs the agent at
/opt/oversight/packages; - makes the host's own key at
/etc/oversight/packages.key, which is its identity from then on; - writes the config,
/etc/oversight/packages.json, readable by root only; - turns off the distribution's own metadata and automatic update timers (
dnf-makecache,dnf-automatic,apt-daily,apt-daily-upgrade,unattended-upgrades), so nothing updates the host behind Oversight's back. The agent refreshes the metadata itself once a day, at a time picked at random for the host; - starts the
oversight-packagesservice.
It runs as root, because reading the package database and running the package manager needs it. It ends by printing the host key:
host key 82b7 fde8 a3ae d837
Enrol it under Packages > Add host by entering this key.
Running the same line again upgrades the agent. The key and the config are kept, so it stays the same host with the same settings.
Configuration
Everything is in /etc/oversight/packages.json. The install writes it like this:
{
"refreshat": "03:40",
"disabled": [
"apt-daily.timer",
"apt-daily-upgrade.timer"
],
"protect": []
}
| Key | Meaning |
|---|---|
refreshat | When the agent refreshes the repository metadata each day, as HH:MM in UTC. Picked at random at install, so the mirrors do not see every host at once. |
disabled | The timers the install turned off. Leave it alone: uninstalling turns these back on. |
protect | Package names the agent must never remove on this host, on top of its built-in list (below). Plain names, no wildcards. A name that is not installed does no harm, so one list can serve several hosts. |
upgradeprotected | false: the packages in protect are not upgraded either, so the agent never touches them at all. An upgrade of everything still upgrades everything else; an upgrade that names a protected package is refused. Missing means true. |
upgrade | false: every upgrade is refused. Missing means true. |
remove | false: every remove instruction is refused. An upgrade may still remove what it replaces, as the package manager requires. Missing means true. |
reboot | false: every reboot is refused, including "then reboot if needed". Missing means true. |
The agent reads the file when it starts, so restart it after a change:
systemctl restart oversight-packages
systemctl is-active oversight-packages
A mistake stops the agent rather than being ignored. A misspelt key, such as "reeboot": false, or a name in protect that is not a package name, would otherwise leave on something you meant to turn off. The service stops with the reason in its log, and the host turns red on the Packages page once it has missed its check-ins:
journalctl -u oversight-packages -n 20
A refused instruction shows on the host's page with the reason, such as "reboot is turned off on this host, in /etc/oversight/packages.json".
Example: a Proxmox VE node
This config makes a Proxmox node hold to the rules described above under Proxmox VE and Ceph nodes, whatever it is sent.
The services are named after the package that ships them: pvedaemon, pveproxy and pvestatd come from pve-manager, pmxcfs from pve-cluster, and corosync from corosync.
{
"refreshat": "03:40",
"disabled": [
"apt-daily.timer",
"apt-daily-upgrade.timer"
],
"protect": [
"proxmox-ve", "pve-manager", "pve-cluster", "corosync",
"qemu-server", "pve-qemu-kvm", "pve-container", "lxc-pve", "lxcfs",
"pve-firewall", "pve-ha-manager",
"libpve-common-perl", "libpve-storage-perl", "libpve-access-control",
"libpve-cluster-perl", "libpve-guest-common-perl", "libpve-network-perl",
"proxmox-default-kernel", "proxmox-kernel-helper", "zfsutils-linux"
],
"upgradeprotected": false,
"remove": false,
"reboot": false
}
Keep your own refreshat and disabled lines as the install wrote them, and add the rest.
Proxmox ships many more packages of its own, and the set differs between releases. To protect every one installed on a node, this prints a protect line built from what is actually there, ready to paste in place of the one above:
dpkg-query -W -f '${Package}\n' | grep -E '^(proxmox-|pve-|libpve-|libproxmox|qemu-server|lxc-pve|lxcfs|corosync|libknet|spiceterm|vncterm|novnc-pve|zfs|libzfs|libnvpair|libuutil)' | sed 's/.*/"&"/' | paste -sd, - | sed 's/^/ "protect": [/; s/$/],/'
Example: a Ceph node
This config keeps the agent away from Ceph on a Ceph node. Everything else on it is still upgraded.
{
"refreshat": "03:40",
"disabled": [
"apt-daily.timer",
"apt-daily-upgrade.timer"
],
"protect": [
"ceph", "ceph-base", "ceph-common", "ceph-mon", "ceph-mgr", "ceph-mgr-modules-core",
"ceph-osd", "ceph-mds", "ceph-volume", "ceph-fuse", "ceph-selinux", "radosgw", "cephadm",
"librados2", "libradosstriper1", "librbd1", "libcephfs2", "librgw2",
"python3-ceph-common", "python3-ceph-argparse", "python3-rados", "python3-rbd", "python3-cephfs"
],
"upgradeprotected": false,
"remove": false,
"reboot": false
}
The names cover Debian, Ubuntu and EL alike; those not installed on a host do no harm. To build the list from what a node actually has, on Debian or Ubuntu:
dpkg-query -W -f '${Package}\n' | grep -E '^(ceph|cephadm|librados|libradosstriper|librbd|libcephfs|librgw|radosgw|rbd-|python3-(ceph|rados|rbd|cephfs|rgw))' | sed 's/.*/"&"/' | paste -sd, - | sed 's/^/ "protect": [/; s/$/],/'
and on EL:
rpm -qa --qf '%{NAME}\n' | grep -E '^(ceph|cephadm|librados|libradosstriper|librbd|libcephfs|librgw|radosgw|rbd-|python3-(ceph|rados|rbd|cephfs|rgw))' | sort -u | sed 's/.*/"&"/' | paste -sd, - | sed 's/^/ "protect": [/; s/$/],/'
A cluster deployed with cephadm runs its daemons in containers, so the packages above are only part of it; the daemons themselves are upgraded with ceph orch upgrade. Add the container engine (podman, or docker-ce and containerd.io) to protect as well, since restarting it restarts every daemon on the node.
On a Proxmox node that also runs Ceph, use both lists together.
Removing it
/opt/oversight/packages --uninstall
This stops and removes the service and turns back on the timers the install turned off. The key and the config are kept, so installing again later brings back the same host, already enrolled, with the same settings.
When something is wrong
- The host is red on the Packages page. It has stopped reporting. Check the service with
systemctl status oversight-packagesand its log withjournalctl -u oversight-packages. A mistake in the config is the usual cause after an edit. - It never appears under Add host. The host cannot reach
oversight.imon port 443. The log says why. - To see what it would send, without sending anything:
/opt/oversight/packages --print. - To check in now, rather than waiting:
/opt/oversight/packages --once. - Which version is installed:
/opt/oversight/packages --version.