Skip to content

fix: use slog.DiscardHandler in promslog.NopLogger - #960

Merged
bwplotka merged 2 commits into
prometheus:mainfrom
FUSAKLA:fus-slog-noop
Aug 18, 2026
Merged

bwplotka merged 2 commits into
prometheus:mainfrom
FUSAKLA:fus-slog-noop

Conversation

@FUSAKLA

@FUSAKLA FUSAKLA commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Investigating huge latencies in pushgateway after upgrade from 1.10 to 1.11 I found out that the cause is the promslog.NopLogger (lets put aside that pushgateway logging might be too excessive)

It turned out that NewNopLogger() returns New(&Config{Writer: io.Discard}) — a regular slog.TextHandler at info level.
Its Enabled() returns true, so slog builds the record, the handler formats every attribute, and only then drops the bytes at io.Discard. For larger attributes this leads to meaningless serialization just to be thrown away.

  • checkWriteRequest → processWriteRequest runs on every push.
  • It logs logger.Info(..., "new", mf, "old", existingMF) — both *dto.MetricFamily with all their metrics.
  • With a nop logger both get fully formatted on every push and thrown away.

Fix:

Use the slog.DiscardHandler added in Go 1.24 which is really no-op.

⚠️ What would change
  • slog.DiscardHandler needs Go 1.24; this module is on 1.25.0.
    • Might be an issue if someone is using the common lib in older Go?
  • LogValuer implementations with side effects will no longer be called.
  • Code wrapping NewNopLogger() and expecting Enabled() to be true will stop seeing those calls.
Benchmark

Benchmark details — go test ./promslog/ -run XXX -bench BenchmarkNopLogger -benchmem, go1.26.5 linux/amd64, AMD Ryzen AI 7 445 (12 threads). Old is the current implementation, the other one is the patch.

func BenchmarkNopLoggerOld(b *testing.B) {
	logger := New(&Config{Writer: io.Discard}) // current NewNopLogger()
	value := struct{ Field string }{Field: "value"}
	for b.Loop() {
		logger.Info("test", "key", value)
	}
}

func BenchmarkNopLogger(b *testing.B) {
	logger := NewNopLogger() // slog.New(slog.DiscardHandler)
	value := struct{ Field string }{Field: "value"}
	for b.Loop() {
		logger.Info("test", "key", value)
	}
}
goos: linux
goarch: amd64
pkg: github.com/prometheus/common/promslog
cpu: AMD Ryzen AI 7 445 w/ Radeon 840M
BenchmarkNopLoggerOld-12      352051      2890 ns/op     440 B/op    11 allocs/op
BenchmarkNopLogger-12       22288611      48.5 ns/op      16 B/op     1 allocs/op

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>
Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

@bwplotka bwplotka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is reasonable, thanks! We have 1.26 already, so expecting minimum 1.24 makes sense.

@bwplotka
bwplotka merged commit e3e7492 into prometheus:main Aug 18, 2026
8 checks passed
@SuperQ

SuperQ commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Interestingly, this caused an issue with one of the blackbox_exporter's unit tests.

See https://github.com/prometheus/blackbox_exporter/pull/1681

KR-Ravindra added a commit to KR-Ravindra/pushgateway that referenced this pull request Oct 6, 2026
checkWriteRequest runs the consistency check on a scratch DiskMetricStore
with promslog.NewNopLogger(). Up to prometheus/common v0.70.x that logger
was a TextHandler writing to io.Discard whose Enabled() returned true, so
the Info calls in processWriteRequest and GetMetricFamilies that pass
whole *dto.MetricFamily values as attributes serialized every metric of
the affected families on every push, even with --log.level=warn. Push
latency therefore scaled with the size of the store whenever any stored
metric family had an inconsistent help string.

prometheus/common v0.71.0 (prometheus/common#960) makes NewNopLogger
return a disabled slog.DiscardHandler; master picked that version up in
prometheus#871. Add a regression test that checks a push against a 1000-metric
family with an inconsistent help string does not allocate more than a
push with a consistent one.

Signed-off-by: KR Ravindra <42912207+KR-Ravindra@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants