Actions are what Oversight does when something changes state: an email, a Matrix or Rocket.Chat message, a webhook, a ticket. They are made of two separate things:
- A rule says what and who: the channel, the states it is about, the recipients, the template, how often it repeats and when it may send. Rules are made under Setup, Actions.
- A binding says where: which part of the estate the rule covers. Bindings are made on the object itself, in its Actions panel under Setup, Configuration.
One rule can be bound in many places, so a change to the rule reaches all of them at once. A rule that is not bound anywhere sends nothing.
Rules · Channels · Binding · When messages are sent · Message templates · Schedules · Testing and deliveries
Rules
Setup, Actions, on the Rules tab. Press New rule, or pick one from the list.
| Field | What it does |
|---|---|
| Name | Required and unique in your estate, such as NOC critical. |
| Channel | How the message travels. See Channels below. |
| Active | Untick Sending to switch the rule off. Switching it off forgets what it had open, so failures already in progress are not announced when it is switched back on. |
| States | Any of WARN, CRIT, UNKNOWN and STALE. An object entering one of these states is news to the rule, and an object leaving them is reported as cleared, to whoever was told it had failed. |
| List down to | Sites, groups, devices or sensors. Anything deeper is counted into its group or device rather than named, which keeps a message to a first line team short. |
| Rows at most | The longest list a message may carry, 50 by default. Each channel has its own ceiling as well, and the lower limit applies. |
| Send | Once, or up to 2, 3, 5, 10 or 20 times. The count includes the first message, so Once means no repeats. |
| Repeat every | From 5 minutes to 24 hours. While alarms are still open they are sent again at this interval, all together in one message, until the send count is reached. New alarms and clearances are always sent straight away. |
| Schedule | Always active, or one of your schedules. The rule sends only while its schedule is active. |
| Template | The channel's default, or one of your message templates for the same channel. |
| Credential | Which connection to send through. Leave it on This estate's, or the global one unless this rule needs a different mail relay or Matrix homeserver from the rest. |
| Recipients | One per line: email addresses, Matrix room ids, URLs or mobile numbers, depending on the channel. Recipients are addresses, not accounts, so a supplier or subcontractor can be told without being given a login. |
| Sender | Email only, such as Oversight {{site}} <noc@example.co.uk>. Left empty,
mail is sent as the connection's login, where that is an address. |
| Subject | Left empty, the channel's default is used. |
Some channels show settings of their own beneath these. Sender, subject and recipients all accept placeholders. Text should be plain ASCII: accented letters, symbols such as the pound sign, and emoji are removed when a rule or template is saved.
Channels
| Channel | Sends | Recipients | Needs |
|---|---|---|---|
| HTML, with charts inline and the poll log attached as CSV | Email addresses, checked for a mail server when saved | An email connection: your SMTP relay | |
| Matrix | Markdown, with up to five charts posted as images after the message | Room ids, or none to use the connection's default room | A Matrix connection: homeserver and bot access token |
| Rocket.Chat | Markdown | Incoming webhook URLs | Nothing further. Optional settings for the channel override, the name it posts as and its emoji |
| Webhook | A JSON body you control | URLs, which accept placeholders | Optionally a connection holding a token, sent as an Authorization header, plus a method and extra headers on the rule |
| HelpDesk ticket | A ticket on the GEN HelpDesk, with charts and the poll log attached | The requester's email address | Your GEN account number. A ticket is raised once, with no repeats and no clearance |
| SMS | A short text: counts and summary, never a list | UK mobile numbers, without +44 | Nothing further |
Connections
The Connections tab holds how each channel reaches the outside world: the SMTP relay's server, port, encryption and login for email, or the homeserver and access token for Matrix. Secrets are sealed and never shown again; leave a secret empty when saving to keep the one already held. A rule uses the connection it names, failing that your estate's first connection for its channel, failing that a global one provided by GEN. The one a rule naming none will use is marked DEFAULT. A connection named by a rule cannot be deleted until the rule is pointed elsewhere.
Binding a rule
Select a site, group, device or sensor under Setup, Configuration and open its Actions panel. Choose the rule and press Bind.
- With And everything beneath ticked, the default, the rule covers the object and everything under it. Unticked, it covers that object's own state only. A sensor is always itself only.
- Binding the same rule to the same object again only changes that choice.
- Rules bound higher up that already reach the object are listed under Reaching here from above. A rule reaching an object by two routes still sends about it once.
- The tenant cannot be bound. To cover the whole estate, bind the rule to each site with everything beneath ticked.
- Unbind removes the binding; anything the rule had open there is dropped at the next run without a message.
The rule page lists everywhere the rule is bound, under Bound to, and the rules list shows a count of bindings, highlighted when a rule has none.
Escalation
Escalation is a second rule, bound to the same place, about a worse state, to a different audience or by a
different channel. For example, First line on WARN and CRIT to a Matrix room, listing down to groups,
and NOC on CRIT only by email, listing down to sensors. Every rule covering an object sends on its
own, so both are told when something goes critical.
When messages are sent
- On a change of state. A message is sent when an object enters the rule's states or moves between them, such as WARN to CRIT. An object that was already failing when the rule was created, bound or switched on is not announced until its state next changes.
- One incident, one message. Everything a rule has open is gathered into one message, worst first, drawn as a tree of site, group, device and sensor. A failure that takes a sensor, its device and its group into alarm is one message, not three. The gathering happens every minute; a real failure is typically reported within about two minutes, depending on the sensor's interval and failure threshold.
- Clearances. An object leaving the rule's states is listed as cleared in the next message, then dropped. On a rule about CRIT only, a move from CRIT to WARN counts as cleared. If the failure had never been sent, for instance because it cleared within the minute or outside the rule's schedule, nothing is sent about the recovery either.
- Repeats. While anything remains open, the rule repeats at its interval until its send count is reached. Repeats stop as soon as everything has cleared. There is not yet a way to acknowledge an alarm to stop its repeats early.
- What is left out. Paused, suspended and scheduled off objects raise nothing.
The message's kind is ALARM if anything in it is new, otherwise RECOVERY if anything has
cleared, otherwise REPEAT.
Message templates
Every channel has a default message, and most rules need nothing else. To change what is sent, create a template on the Message templates tab for the channel, starting from its default, and choose it on the rule. A template belongs to one channel, since each channel has its own format: HTML for email and tickets, Markdown for Matrix and Rocket.Chat, JSON for webhooks and plain text for SMS. Preview renders it against your own estate's sensors, shown as an alarm, without saving.
Placeholders are written {{name}}. A block written between [[ and ]] is
repeated once for each object listed, worst first. Some placeholders describe the whole message and work only
outside a block; others describe one object and work only inside one.
| Outside a block | Gives |
|---|---|
{{kind}} | ALARM, RECOVERY or REPEAT, or TEST for a test |
{{rule}}, {{tenant}} | The rule's name, your estate's name |
{{time}} | When the message was made, UTC |
{{state}} | The worst state in the message |
{{count}}, {{new}}, {{open}}, {{cleared}} | How many objects in all, and how many are new, still open and cleared |
{{summary}} | The incident in one line, such as 25 sensors in Servers, 5 in Edge, 10 devices in Comms Room |
{{tree}} | The incident drawn as a tree, in the channel's format |
{{charts}}, {{logs}} | Every chart, and every poll log, in the message |
{{url}} | The dashboard |
| Inside a block | Gives |
|---|---|
{{name}}, {{objtype}} | The object's name, and SITE, GROUP, DEVICE or SENSOR |
{{state}}, {{previousstate}} | Its state now, and before |
{{kind}} | NEW, OPEN or CLEARED |
{{error}}, {{detail}} | The failure class, such as RESPONSETIMEOUT, and the detail given with it |
{{since}}, {{duration}} | When it went into this state, UTC, and for how long, such as 1d 2h 5m |
{{site}}, {{group}}, {{device}}, {{sensor}},
{{parentname}} | Where it sits |
{{count}}, {{beneath}} | For a row standing for several objects, how many, and a description such as 3 sensors, 1 cleared |
{{chart}}, {{log}} | Its charts, and its latest poll results |
{{url}} | The object on the dashboard |
A minimal Markdown template for Matrix:
**{{kind}}: {{rule}}**
{{summary}}
[[- {{site}} / {{name}}: `{{state}}` since {{since}} UTC {{error}}
]]
[Open the dashboard]({{url}})
Values are escaped for the channel's format as they are filled in. In a JSON template, put the quotes round a
placeholder yourself, as "name": "{{name}}". A trailing comma left before a closing bracket by the last
repeat is removed. Blocks cannot be nested, there are no conditions, and a misspelt placeholder simply comes out
empty, so preview before saving.
Evidence in the message
For sensors, charts and the poll log run from fifteen minutes before the alarm began to the moment of sending.
Charts are drawn in email, as images after the message in Matrix, and as attachments on a ticket; other channels
carry none. Email and tickets also carry the whole poll log as poll-log.csv. Untick
Charts in notifications on a sensor whose chart says nothing, a list of guests that are each
simply up or down for example.
Schedules
Setup, Schedules. A schedule says when something is active, in quarter hours, in a timezone of your choosing, and follows local time through the clock change.
- Everything starts active. Click or drag across the grid to switch quarter hours off or on, a row per day of the week, with All day and Clear for a whole day.
- Add a day of the month covers, say, the first of every month. Add a date covers one day, such as a bank holiday.
- Where rows apply to the same day, the most specific wins: a date over a day of the month over a day of the week. So a date can switch an ordinary working day off, or a normally quiet Sunday on.
A schedule does different things depending on where it is used:
| On a rule | On an object in the tree |
|---|---|
| Decides when the rule may send. Monitoring carries on as normal. Whatever is still wrong when the schedule next opens is sent then, together with clearances for anything already reported. A failure that starts and clears entirely while the schedule is closed is never sent. | Decides when the object is polled at all. Outside it, the object and everything beneath is not polled, shows as SCHEDULED OFF, counts towards nothing, raises nothing and leaves a gap in its history. The nearest schedule in the tree wins. |
To stop being contacted at night while still recording what happens, put the schedule on the rule. To stop monitoring something during planned maintenance hours, put it on the object.
Testing and deliveries
Send a test on a saved rule sends it now, as last saved, to its real recipients over its real connection, built from your estate's own sensors with real charts, and marked as a test. It ignores the rule's active switch, schedule and bindings. The result for each recipient is shown at the top of the page. Save any changes first. A test of a webhook calls the real endpoint, and a test of a HelpDesk ticket rule raises a real ticket.
Recent deliveries, on each rule, shows the latest messages, one row per recipient, as Sent, Waiting, or the last error with the number of attempts. A failed delivery is retried for that recipient alone, after 1, 2, 4, 8 and 16 minutes, up to six attempts in all. Delivery history is kept for 30 days. Tests are not listed there, but are recorded in the audit log.