Everything Oversight watches is described as one tree, the estate. It is built under Setup, Configuration, where the Estate panel on the left is the tree itself and the panel on the right is whatever is selected in it. The same tree drives the dashboard, the state of every object and where notifications apply.
The levels
| Level | Sits under | What it is |
|---|---|---|
| Tenant | Nothing, it is the root | Your organisation. There is exactly one, created for you on your first visit to Configuration and named after your company. It has no state of its own, and nothing is set or bound on it. |
| Site | The tenant | A location: a data centre, an office, a cloud region. |
| Group | A site, or another group | Any way of gathering devices that means something to you: a rack, a cluster, a service, a customer. Groups nest to any depth. |
| Device | A group | One thing with an address: a server, a switch, a firewall, a web site. |
| Sensor | A device | One check made against that device, such as a ping, a web request or an SNMP read. |
A device cannot sit directly on a site, so every new site is given a group called Default to hold
its first devices. Rename it, or add groups of your own beside it. Names must be unique among the objects sharing
a parent.
Probes are not part of the tree. They are the machines, run by GEN, that make the checks, and a sensor chooses which probe groups watch it. The tree describes what is monitored; probe groups describe where it is watched from.
Adding to the tree
Press Add on the Estate panel. The dialogue suggests the next level down from whatever is selected, so with a group selected it offers a device under that group. Choose the type, where it goes under, and a name. A device also needs its hostname or address. A sensor needs its type, an interval and at least one probe group, and can start from a template; its name may be left empty, in which case it is named after its type.
Objects cannot yet be moved to a different parent. To move a device, save its sensors as a template, apply the template to a new device in the right place, and delete the old one.
States
| State | Colour | Meaning |
|---|---|---|
| OK | Green | Everything that counts is healthy. |
| WARN | Amber | Something needs attention. |
| CRIT | Red | Something is wrong. |
| UNKNOWN | Grey | Not enough evidence to reach a verdict: too few probe groups or children reporting, or a sensor that cannot run. |
| STALE | Purple | A sensor that no probe group is reporting on. Results have stopped arriving. |
| PAUSED | Blue | Paused, suspended or outside its schedule, together with everything beneath it. |
UNKNOWN and STALE are never treated as down. They are shown plainly as no data, they never drive a parent into alarm, and they are kept apart from downtime in availability figures. Equally, they are never counted as healthy.
Rollup: how state travels up the tree
Every object from a sensor upwards has a Rollup panel. It decides the object's state from what sits beneath it, and the same three questions are asked at every level:
| Level | Counts |
|---|---|
| Sensor | Its probe group views, one per probe group watching it |
| Device | Its sensors |
| Group | Its devices and groups |
| Site | Its groups |
Only those beneath with an opinion are counted, meaning those at OK, WARN or CRIT. Anything UNKNOWN, STALE, paused, suspended or outside its schedule is left out of the count altogether. Of those counted, anything at WARN or CRIT is failing, and the number failing falls into one of three buckets:
| Setting | When it applies |
|---|---|
ANY | Exactly one is failing |
SOME | More than one is failing, but not all of them |
ALL | Every one counted is failing |
Each bucket is mapped to OK, WARN or CRIT, and that is the object's state. If nothing is failing the state is OK. Two refinements keep the verdict honest:
- Minimum reporting. Below this many with an opinion, the state is UNKNOWN rather than a verdict reached from one straggler. A sensor watched from three probe groups with a minimum of two stays UNKNOWN while only one of them is heard from.
- The mapping can soften, never inflate. If a bucket maps to CRIT but nothing beneath is actually CRIT, the result is WARN. A handful of warnings never becomes a critical.
Because the buckets count, the arithmetic has two consequences worth knowing. With a single probe group or a
single child, one failure is ALL, not ANY. With exactly two, one failure is
ANY and two is ALL, so SOME can never apply.
The defaults
| Level | ANY | SOME | ALL | Minimum reporting |
|---|---|---|---|---|
| Sensor, group, site | OK | WARN | CRIT | 1 |
| Device | CRIT | WARN | CRIT | 1 |
A sensor shrugs off one probe group struggling, since that is usually the path and not the target. A device goes CRIT as soon as one of its checks does. A group or site treats one failing member as fine, several as a warning and all of them as critical. Set the mapping to suit the thing: a standalone server should go red on any failure, while twenty web servers behind a load balancer only matter when several fail.
Note that on a device the default SOME is WARN, so a device with one critical sensor reads CRIT but
one with two of its five sensors critical reads WARN. If that is not what you want, set SOME to CRIT
on the device.
Contributes upwards
Clear Counts towards its parent's state to watch something without letting it condemn what it sits on. A certificate expiry warning is worth seeing on its own, and is not the device being down.
Inheritance
Some settings are made once, high in the tree, and reach everything beneath. The nearest one always wins.
| Setting | How it passes down |
|---|---|
| Credentials | A credential held on a site, group, device or sensor is used by every sensor of that type beneath it, unless a nearer one replaces it. See Credentials. |
| Substitution values | Collected from every level above, with a nearer value replacing one of the same name. |
| Schedule | An object is polled only while its schedule is active. The nearest schedule wins, so a device can carry its own schedule inside a group that has another. See Actions for how schedules are made. |
| Address | Every sensor targets its device's address unless the sensor sets a target override. |
| Actions | A rule bound to an object can cover that object alone, or it and everything beneath it. |
Interval, timeouts and probe groups are set on each sensor and are not inherited. Templates are the way to set them once for many sensors.
Pausing, schedules and suspension
Pause is at the top of any site, group, device or sensor. Choose how long, from five minutes to indefinitely, and give a reason, which is shown beside the object and recorded in the audit log. A timed pause lifts itself when the period is up; Resume lifts it at once. Nothing beneath a paused object is polled, and none of it counts towards an alarm. On the dashboard it all shows blue.
An object outside its schedule is marked SCHEDULED OFF. It is not polled, any results that arrive are discarded, and it is left out of its parent's state. When the schedule opens again it is given time to report before it can read as stale.
SUSPENDED is set by Oversight, not by you. A sensor whose settings a probe cannot run is suspended after a single attempt and set to UNKNOWN, rather than failing over and over and waking somebody. A banner on Configuration lists every suspended sensor. Correct the sensor and save it, and it is collected again on the probe's next configuration poll. Lift all clears every suspension at once; any sensor still broken is suspended again on its next attempt.
Deleting
Delete removes the object and everything beneath it, and the button shows how many objects that is. The sensors' settings, and any credentials held on the deleted objects, go with it. It cannot be undone, and the tenant itself cannot be deleted.