A sensor is one check made against a device: a ping, a TCP port, a web request, an SNMP read, a database query, a file fetched by FTP, an email sent and followed. This page covers what every sensor has in common. What each type asks for, and what it reads, is explained by the Help button on that type's own settings panel.

From poll to state · Polling · Extractions · Conditions · Rules · Rollup · Examples · Checking a sensor

From poll to state

Every reading goes through the same stages, and each stage has its own panel on the sensor:

1. Polling. Each probe group attached to the sensor makes the request at the sensor's interval. The probe measures and returns what arrived, with the connect and response times and, if it failed, a class of failure. It decides nothing.
2. Extraction. The values that matter are taken from the response and put in numbered slots.
3. Conditions. Each condition tests one slot, and says whether a match means WARN or CRIT.
4. Rules. All the conditions, and any failure to poll, give that probe group's view of the sensor.
5. Rollup. The views from all the probe groups are counted into the sensor's state, which then counts towards its device, and so on up the tree.

All of the thinking happens centrally, not on the probe. A result is decided within about fifteen seconds of arriving.

Polling

Add a sensor from the Add button in Configuration, under the device it checks. The dialogue asks for the type, or a template, an interval and at least one probe group; everything else is set on the sensor once it exists. The Polling panel holds the settings every type shares:

Setting	What it does
Interval, seconds	How often each probe group polls. At least 10 seconds, 60 by default. Email delivery runs no more often than every 300 seconds.
Connect timeout, Response timeout	How long to wait for the connection, then for the answer, 5 and 10 seconds by default. Together they must be less than the interval. Keeping them separate means a firewall dropping the connection is never mistaken for a service that has hung.
Failures before a probe calls it down	How many polls in a row must fail before that probe group reports the sensor CRIT. 3 by default. A single dropped packet or refused connection then goes no further.
Target override	Leave empty to check the device's own address. For a web request, a whole URL here replaces the request URL entirely.
Chart, response time	Places the response time on one of five numbered charts on the dashboard.
Store raw response	Keeps the whole response with every reading, so it can be read in the poll log. Invaluable while setting a sensor up; adds 2 credits a read, so clear it once the sensor is proved.
Charts in notifications	Draws this sensor's charts into its alarms, on channels that carry them.
Probe groups	Where the sensor is watched from. At least one is required.

Probe groups

A sensor is attached to probe groups, never to single probes. Each group is a vantage point, such as a region or your own network, and shares its sensors among its probes, so a probe can fail without its sensors going with it. Each group forms its own view of the sensor, and those views are what Rollup counts. One group is enough to monitor something; two or more are what let Oversight tell a problem with the target from a problem on the way to it.

Credits

The foot of the Polling panel shows what the sensor costs per read, and an estimate for the month. A read is one result from one probe group, so a sensor polled every minute from three groups makes three reads a minute.

Part	Credits per read
Ping	1
DNS, FTP, IMAP4, MongoDB, MySQL, RDP, SIP, SMTP, SNMP, TCP	2
HTTP(S)	4
Email delivery	6
A user function converts the response	add 4
Store raw response	add 2

One credit is £0.00002. A failed poll costs the same as a good one, since the work was done; a probe group that is down makes no reads and costs nothing.

Sharing requests

Several web sensors making exactly the same request, the same URL, method, headers, body and timings, from the same probe, share one fetch. Five sensors reading five values from one status document put one request on the device, not five. Failures are shared as well, so the five agree. Each sensor is still charged per read.

Extractions

Every reading has thirty slots: SV01 to SV20 hold numbers, which can be charted and compared with thresholds, and SD01 to SD10 hold text of up to 255 characters. The Extraction panel lists all thirty.

Each sensor type fills some slots for you. A web request fills SV01 with the status code, SV02 with the response size, SD01 with the status text and SD02 with the content type; an SNMP sensor fills SV01 with the value. Those rows show the type's name for the slot, and their method reads From the type. Your own extractions go in the free slots, usually from SV03. An extraction set on a slot the type fills replaces the type's value.

An extraction row

Field	What it does
Reading	The name of the value. It is what conditions, charts and alarms call it, so make it plain: Active requests, CPU0 temperature, 106-podman.
Method	How the value is found. A row with no method is not stored.
Expression	What the method looks for, where it needs one.
Unit	Shown on charts, such as W or requests. Numbers only; nothing is converted.
Multiplier	Scales the value before it is stored, so a reading in bytes charts in megabytes at 0.000001. Conditions are then written in the scaled units. Changing it affects new readings only.
Labels	JSON naming numeric values, such as {"1":"up","2":"down"}, so a reading shows as 2 (down). The number is still what is stored and tested.
Chart	Places the value on one of five numbered charts.

Methods

Method	Reads	Expression
JSON	A value from a JSON document	A dotted path, with numbers indexing arrays: active_count, PowerControl.0.PowerConsumedWatts
JSONCOUNT	How many items are in an array or object	A dotted path to it
XPATH	A value from XML, the first node matched	An XPath 1.0 expression, or for a namespaced document such as SOAP, {"xpath":"//t:Result","ns":{"t":"urn:example"}}
XPATHCOUNT	How many nodes match	An XPath 1.0 expression
REGEX	Text matched by a regular expression, the first capture group or else the whole match	A pattern with delimiters: /Uptime: (\d+)/
HEADER	A response header	Its name, in any case: Retry-After
STATUS, STATUSTEXT	The status code, or the status line	None
SIZE	The size of the response in bytes	None
WHOLE	The whole response, as text or, if it is a number, as a number	None
DIRECT	The response time in milliseconds	None

JSON paths are deliberately simple: there are no wildcards or filters. XPath needs well formed XML, so an ordinary HTML page rarely parses; use REGEX for those. A value that is not a number is refused by a numeric slot.

User functions

Some responses need more than picking out a value: counting, summing, or judging a list of things by name. For those, choose a User function at the top of the Extraction panel. The function reshapes the response into simple named values before any row reads it, and its description appears when it is chosen, with the Arguments, JSON it accepts. User functions are written by GEN, never supplied by an estate; if you need one that does not exist, raise a ticket. A user function adds 4 credits a read.

STATUS, STATUSTEXT, SIZE and HEADER still read the original response. Where Store raw response is ticked, what is kept is the converted response, which is exactly what the rows read.

One response, many values

A single response can fill up to twenty numbers and ten pieces of text, each with conditions of its own. Five temperatures from one server management API are five rows reading five paths. For a list whose members matter by name, such as the guests on a Proxmox cluster or the endpoints on a phone system, a user function turns the list into one value per name, and each name is read by its own row. An alarm then names what failed: 106-podman is 0, not equal to (1,2,3).

Conditions

Each row of the Rules panel is a condition: one slot, one test and the state a match gives. The panel always offers five empty rows; save and reopen the sensor for more.

Field	What it does
Slot	The slot to test, shown with its reading name.
Is	The test, below.
Value	What to compare with: 200, offline, or a list in brackets such as (1,2,3).
And	The upper value, for between and outside only.
Then	WARN or CRIT.

Tests

Is	Matches when the reading
equal to	Equals the value. Given a list in brackets, equals any one of them.
not equal to	Differs from the value. Given a list in brackets, equals none of them.
greater than, at least	Is above the value, or at or above it.
less than, at most	Is below the value, or at or below it.
between	Lies from the value to the upper value, both included.
outside	Lies below the value or above the upper value.
containing, not containing	Contains the text, or does not.
matching, not matching	Matches a regular expression with delimiters, such as /^220 /i, or does not.

A few things to bear in mind when writing them:

• Where both sides are numbers they are compared as numbers, so equal to 1 matches a stored 1.000000. Otherwise they are compared as text, exactly and case sensitively. For a test that ignores case, use matching with /i.
• The comparisons greater than, at least, less than, at most, between and outside only ever match numbers.
• Test the number, not its label: equal to 2, not equal to down.
• Thresholds are in the units stored, after the multiplier.
• A condition on a slot that the reading did not fill is skipped. It neither fires nor holds the sensor in its last state. If a web API returns an error page instead of the document you expected, your extraction finds nothing and its conditions do not fire, but the status code is still filled. So give every web sensor that extracts values a condition of SV01 not equal to 200 then CRIT.

Rules

There is no ordering and no AND or OR between conditions. Every condition is tested, and the worst state any of them reaches is the one recorded, CRIT over WARN over OK. The order on the screen is only for reading, so a critical can never be hidden behind a warning written above it. A reading that matches no condition is OK. Where several conditions match at the worst state, all of them are named in the alarm.

Two thresholds on one value are simply two conditions: greater than 10 then WARN and greater than 20 then CRIT. At 25 both match, and CRIT is recorded.

When a poll fails

Conditions only apply to a poll that got an answer. When a poll fails, the probe group counts it. Until the count reaches Failures before a probe calls it down the view stays OK; at the threshold it becomes CRIT with the class of failure, and the first good poll resets the count. With a 60 second interval and the default of 3, a sensor that stops answering is CRIT from that group after about three minutes. A failure never gives WARN; only conditions do.

Class	Meaning
DNS	The name could not be looked up from the probe.
REFUSED	The connection was refused or rejected, or the failure could not be classed more exactly.
CONNECTTIMEOUT	No connection within the connect timeout: usually a firewall, routing, or the device being off.
RESPONSETIMEOUT	Connected, but no complete answer within the response timeout.
TLS	The certificate or the encrypted handshake failed.
AUTH	The login was refused.
PARSE	An answer arrived but nothing could be read from it. The view is UNKNOWN rather than CRIT.

Settings that cannot run

If a probe cannot run a sensor at all, for instance a URL that is not a URL, or an SNMP object the agent does not have, it says so without trying, and the sensor is suspended after that single attempt and shown as UNKNOWN. Nobody is woken by a typing mistake. Correct the sensor and save it to lift the suspension.

Rollup

Each probe group reaches its own view, OK, WARN, CRIT or UNKNOWN. The sensor's Rollup panel turns those views into one state, by counting how many are failing:

1. Only views at OK, WARN or CRIT are counted. A group that is silent, or whose view is UNKNOWN, is left out, never counted as healthy.
2. If no group is reporting at all, the sensor is STALE.
3. If fewer than Minimum reporting groups are reporting, it is UNKNOWN.
4. If none is failing, it is OK.
5. Otherwise, one failing is ANY, more than one but not all is SOME, and every reporting group failing is ALL. Each is mapped to OK, WARN or CRIT.
6. A mapping to CRIT gives WARN if no failing view is actually CRIT.

The defaults for a sensor are ANY OK, SOME WARN and ALL CRIT: one vantage point struggling is the path, several is something real, and every one failing means nobody can reach it. Change the mapping where the thing deserves it; a public web site might reasonably go WARN on ANY.

The counting has consequences worth planning for:

• One probe group: a failure is ALL, so it is CRIT by default.
• Two groups: one failing is ANY, two is ALL, and SOME cannot happen.
• Three or more are needed for all three buckets to mean something. A minimum reporting of two then stops a single remaining group reaching a verdict on its own.

A group's view is expected again within its interval, plus a minute, plus half the interval again: 150 seconds for a sensor polled every minute. After that it is overdue and drops out of the count; once every group is overdue the sensor is STALE. Nothing is wrong with the target, only with our view of it, and STALE is never counted as downtime.

From here the sensor counts towards its device, which counts towards its group, using the same rules with the mapping set on each. Clear Counts towards its parent's state for a sensor that is worth watching but should not condemn the device, such as a certificate expiry warning. See The Tree.

Examples

A value in a JSON status page

A proxy publishes /proxy/stats/clients, returning {"active_count": 7, ...}.

• Type HTTP(S). Request http://, port 8001, path /proxy/stats/clients, method GET.
• Extraction SV03, reading Active requests, unit requests, method JSON, expression active_count, chart 1.
• Conditions SV01 not equal to 200 then CRIT; SV03 greater than 10 then WARN; SV03 greater than 20 then CRIT.

Without the first condition, an error page with no active_count in it would leave SV03 unread and the sensor OK. At 4 credits a read every minute from one group, it costs about £3.46 a month.

An SNMP value with named states

A UPS reports its battery status as a number, which the UPS MIB names.

• Type SNMP. From a MIB: choose UPS-MIB and upsBatteryStatus. The OID, value type and labels {"1":"unknown","2":"batteryNormal","3":"batteryLow","4":"batteryDepleted"} are filled in for you.
• An SNMP credential holding the community, on the device or anywhere above it.
• Conditions SV01 equal to 3 then WARN; SV01 equal to (1,4) then CRIT.

The poll log shows the reading as 2 (batteryNormal), while the conditions test the number. Where an object is scaled, a load average in hundredths for instance, set the multiplier to 0.01 and write the thresholds in whole units.

Every guest on a Proxmox cluster

• Type HTTP(S), port 8006, path /api2/json/cluster/resources?type=vm, header Authorization: PVEAPIToken={{basicuser}}={{apikey}}, with the token held as an HTTP(S) credential on the cluster's device.
• User function uf_pveguests, arguments {"expect":[101,102,103]}. It gives each guest a code: 1 running, 2 in backup, 3 migrating, 0 stopped, 8 not found, and others. Listing the guests you expect means one that vanishes reads 8 rather than simply going missing.
• Extraction one row per guest: SV03 reading 101-web, method JSON, expression 101; SV04 reading 102-mail, expression 102; and so on.
• Conditions SV01 not equal to 200 then CRIT, then for each guest not equal to (1,2,3) then CRIT, so backups and migrations are not failures.

One request covers the whole cluster, and an alarm names the guest that stopped. Untick Charts in notifications, since a chart of guests that are each simply up or down says nothing. At 8 credits a read every minute from one group it is about £6.91 a month.

Text answers

• A DNS sensor: SD02, the answer, not equal to www.example.co.uk. then CRIT.
• A MySQL query that returns ok when replication is healthy: SD01 not equal to ok then CRIT.
• An SMTP banner: SD01 not matching /^220 / then CRIT.

Checking a sensor

Changes reach the probes at their next configuration collection and run at the next interval, so allow a minute or two after saving. To see what the sensor is really getting:

1. Tick Store raw response and save.
2. Press Examine at the top of the sensor. The dashboard opens on its Log, one row per poll per probe group, with what was acquired, the verdict, and the connect and response times.
3. Open a poll. It shows every slot filled, with labels, and what the probe got back. If a slot you expected is missing, compare your expression with the response shown.
4. Clear Store raw response once the sensor is right.

The History tab charts the numbered charts over any period, and By group draws a line per probe group, which shows at once whether a slow response is the target or one vantage point. The Credentials panel on the sensor says where its credential comes from, or that nothing resolves.