False-alert rate
How often the detector alerts when nothing happened, and how much it catches, on a locked benchmark. Every number below is read from the benchmark's reference file; none is typed by hand.
| Dataset | Caught (of detectable) | False alerts per watch-month | Watches over 1 false alert a month | z-score: caught | z-score: false per watch-month |
|---|---|---|---|---|---|
| Syntheticseries drawn from the detector's own noise model | 30 of 39 (77%) | 0.034 | 0 of 20 | 9 of 39 (23%) | 0.051 |
| Off-grid syntheticincidents placed off the 5-minute grid and the hour | 46 of 68 (68%) | 0.097 | 1 of 80 | 4 of 68 (6%) | 0.11 |
| Foreign noisereal residuals (whole weeks, block bootstrap), not our noise model | 36 of 56 (64%) | 0.54 | 20 of 80 | 2 of 56 (4%) | 0.464 |
| Wikipedia page views v1real traffic, labelled incidents | 17 of 24 (71%) | 1.234 | 4 of 10 | 15 of 24 (63%) | 1.234 |
| Wikipedia page views v2real traffic, a second labelling | 8 of 21 (38%) | 1.401 | 5 of 10 | 11 of 21 (52%) | 1.234 |
How it is measured
- Detectable: an incident that stands out from its series' own noise by the detector's noise test. Catch rates count only these. A definition that does not use the detector's own test is still open.
- False alert: an alert that matches no labelled incident. The rate is per watch per month of checked data; the bar is 1 false alert per watch-month.
- z-score: a simple baseline on the same data, senders and check times. It compares each window with the same window on each of the last 14 days and alerts beyond k standard deviations (one k per dataset). It shows what a simple method gets on the same sets.
- Locked: the reference changes only when the measurement method is corrected, and every change is recorded. A change to the detector is judged against it by pre-registered rules.
What these numbers do not say
- On real backgrounds the detector ties the z-score on catches and beats it on false alerts in some sets, not all. We do not claim better detection.
- A sealed holdout, run once, failed: on Wikipedia the detector caught 60% with 6 of 10 watches over the false-alert bar. Nothing was tuned on it.
- On foreign noise, most of the "false" spikes are real surges an owner would recognise: the false rate an owner would see there is lower than the table shows.
- Your data is not these datasets. Import history to see what the detector would have done on yours.