Logstag
How we stopped one finding from firing hundreds of times
A detector that never accepts a new normal ends up reporting itself, not the database. This is what that looked like on our own development PostgreSQL, what we changed, and what comes next.
The problem
Logstag Insights learns how each database normally behaves, hour by hour, and raises a finding when today stops looking like that. It is how Logstag spots trouble before anyone files a ticket.
On the PostgreSQL behind our own development environment, one finding, "Query throughput +93%", kept firing for more than a week, hundreds of times. Nothing was broken. Traffic had roughly tripled in a few days and stayed there.
Insights was comparing every hour with a history that had not caught up with the new level yet, so every hour looked unusual. It was right every time, about the wrong thing: the database had a new normal, and the detector would not accept it.
A second case had the opposite shape. On a very quiet database, "average duration +33%" fired as Critical for a move from 1.21 ms to 1.61 ms. On paper, a large change. In practice, nobody would ever notice it.
Two different mistakes
The first finding was news on its first day and noise every day after. The second was never news at all: a database with almost no variation turns any small change into a big number. Both were correct by the test we had written, and neither helped anyone.
Fix one: is this still new?
Insights now asks two questions instead of one. Is this unusual compared with the database's longer history? And is it still unusual compared with the last few days? A spike stands out on both. A step change stands out on both on its first day, so you hear about it when it happens. Once the new level has settled in, it no longer stands out against recent days: the finding stops firing and resolves on its own.
| Shape | Result |
|---|---|
| A short spike | Fires |
| A step change, on its first day | Fires: this is news |
| The same step change, after that | Quiet, then resolves |
| A slow ramp | Rarely fires: drift is not an incident |
One rule did not move: missing data never silences a finding. When there is not enough recent history to judge, Insights falls back to the longer history and behaves exactly as before. A gap in data must never be the reason a real problem stays hidden.
Fix two: big enough to matter
A change now has to be large enough to make a practical difference before it counts. This is a cut on what matters, not a noise filter, and we chose it on purpose: a slowdown that hurts your users clears the bar, and what it removes is the class of findings nobody ever acted on.
What we did not do
- Cap the number of occurrences. It would turn a real, ongoing problem into silence on a timer.
- Add a cooldown. It would hide repeats without making the detector any smarter.
- Rebuild the statistics from scratch. It would mean retuning everything for the same result on the cases we had actually seen.
What comes next
Fewer findings, each one worth reading, is a moving target. On the headline case we expect far fewer repeats, and we keep tuning Insights on real workloads so that what reaches you is news, not noise. How Insights works.