Oversight is GEN's managed infrastructure monitoring service. It watches servers, network equipment, hypervisors, storage, web services, databases and mail from several places at once, and it is built around one idea: only raise the alarm when the evidence agrees, and when it does, send one message that explains the whole incident.
This page introduces the ideas the rest of the help relies on, in the order they fit together.
The estate
Everything you monitor is described as one tree, called the estate: sites hold groups, groups hold devices and other groups, and devices hold sensors. The tree is not only a way of filing things. Every object in it has a state, each state is worked out from what sits beneath it, and notifications, schedules and credentials are all attached to places in it.
Sensors, probes and probe groups
A sensor is one check against one device: a ping, a TCP port, an HTTP(S) request to a web site or an API, an SNMP read, a DNS lookup, a SIP OPTIONS request, an RDP negotiation, a MySQL, MariaDB or MongoDB query, a file fetched over FTP or FTPS, SMTP and IMAP4 banners, or a real email sent and followed to its destination.
The checks are made by probes, small programs run by GEN in different locations, and on your own network where it is needed. Probes are gathered into probe groups, and a sensor is attached to groups, never to a single probe. A group shares its sensors among its probes, so one probe failing does not take its sensors with it.
Probes only measure. They report what arrived, how long the connection and the answer took, and, when something failed, what kind of failure it was: a name that would not resolve, a refused connection, a timeout, a certificate problem, a rejected login. Everything that decides what a reading means is held centrally.
From a reading to a state
For each probe group, a sensor's reading passes through three steps:
- Extraction takes the values that matter out of the response, such as a temperature from a JSON document or the state of each guest on a cluster, and gives each a name.
- Conditions test those values: status code not equal to 200 is critical, more than ten queued messages is a warning.
- Rules settle the result. Every condition is tested and the worst state reached wins, so the order they were written in can never hide a critical behind a warning. A poll that fails outright counts against the sensor only once it has failed the chosen number of times in a row.
See Sensors.
States
| State | Meaning |
|---|---|
| OK | Healthy. |
| WARN | Needs attention. |
| CRIT | Something is wrong. |
| UNKNOWN | Not enough evidence to decide. |
| STALE | Results have stopped arriving. |
| PAUSED | Paused, suspended or outside its schedule. |
Silence is never read as health, and unknown is never read as down. UNKNOWN and STALE are shown plainly as no data. They never drive anything above them into alarm, and they are kept out of downtime.
Consensus and rollup
One probe that cannot reach a web site usually means a problem on the way, not with the site. So each probe group forms its own view of a sensor, and the sensor's state is decided by how many of those views are failing:
| Failing | Bucket | By default |
|---|---|---|
| Exactly one | ANY | OK: one vantage point struggling is the path |
| More than one, not all | SOME | WARN: something real is happening |
| Every one reporting | ALL | CRIT: nobody can reach it |
The same counting is used at every level of the tree, with its own settings on each object. Sensors roll up into their device, devices into groups, groups into sites. A standalone server can go red on any failed check, a resilient pair of switches can shrug off one lost path, and twenty web servers behind a load balancer need only matter when several fail. What counts as critical is a property of the thing being watched. A minimum reporting number on each object stops one straggler reaching a verdict for everyone.
Actions
When states change, rules decide who is told, by which channel, about which states, how often reminders are sent and at what times. A rule is written once and bound to any part of the estate, for that object alone or for everything beneath it. Escalation is simply a second rule about a worse state, to a different audience.
One failure can change a sensor, its device, its group and its site at once. Rather than a message for each, Oversight gathers everything a rule has open into one message, drawn as a tree, worst first, with charts and the poll log from before the failure attached where the channel carries them. Recovery is reported only to whoever was told about the failure. Messages go by email through your own mail relay, to Matrix and Rocket.Chat rooms, to webhooks, and as tickets on the GEN HelpDesk.
Schedules set when things happen, in quarter hours, on local time through the clock change. On a rule, a schedule holds messages until the window opens. On an object, it stops the object being polled at all.
See Actions.
Templates and credentials
A sensor proved on one device can be saved as a template, and a whole device's sensors as one, then applied to the next device in a moment. Templates never carry addresses or credentials, so they travel safely. See Templates.
Credentials, such as SNMP communities, database logins and API tokens, are held on objects in the tree and reach every sensor beneath that can use them, with a nearer one overriding a further one. They are sealed before they are stored and are never shown again. See Credentials.
Seeing it
The Dashboard shows the estate as a wheel: your organisation at the centre, then sites, groups and devices outward, each segment coloured by its state. Click a segment to move in, and the centre to come back out. Beside it are the tree and the current alarms. Any object can be examined for its availability over the last day, week and month, its charts over any period, and, for a sensor, the log of every poll with what came back.
Built to be trusted
- A mistake does not wake anyone. Settings are checked on save, and a sensor a probe cannot run is suspended after one attempt and flagged in Configuration, not raised as an alarm.
- Outbound only. Probes connect out to Oversight over HTTPS. Nothing connects in to them, so no inbound port is opened on your network.
- Signed probes. Every probe holds its own key, created when it is enrolled, and signs every request it makes.
- Separate estates. Each organisation sees only its own estate, rules, templates and credentials, and every change is recorded in an audit log.
What it costs
Oversight is charged per read, not per sensor or device, and invoiced monthly in arrears. A read is one result from one probe group, and each sensor costs a fixed number of credits per read, from 1 for a ping to 6 for an email delivery test, at £0.00002 a credit. A ping every minute from one location is about £0.86 a month. Every sensor shows its cost per read and a monthly estimate as it is configured. Rules, schedules, templates and delivery by email, Matrix, Rocket.Chat, webhook and ticket are included. See Sensors for the full table.
Getting started
- Open Setup, Configuration. Your estate is created on the first visit.
- Add a site. It comes with a
Defaultgroup. - Add a device to the group, with its hostname or address.
- Add a sensor to the device: start with a ping or a TCP port, and choose a probe group.
- Add any credentials the sensor needs, on the device or higher up.
- Under Setup, Actions, make a connection if your channel needs one, and a rule, then bind the rule to your site in Configuration.
- Watch it on the Dashboard.