insights: v23.1.13: nil quantile.Stream in withFingerprintLatencySummary

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5378038262/?referrer=webhooks_plugin

Panic Message:

panic.go:884: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | runtime.panicmem
  | 	GOROOT/src/runtime/panic.go:260
  | runtime.sigpanic
  | 	GOROOT/src/runtime/signal_unix.go:839
  | github.com/cockroachdb/cockroach/pkg/util/quantile.(*Stream).ByteSize
  | 	github.com/cockroachdb/cockroach/pkg/util/quantile/stream.go:230
  | github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights.(*anomalyDetector).withFingerprintLatencySummary
  | 	github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/detector.go:135
  | github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights.(*anomalyDetector).isSlow
  | 	github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/detector.go:82
  | github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights.(*compositeDetector).isSlow
  | 	github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/detector.go:50
  | github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights.(*lockingRegistry).ObserveTransaction
  | 	github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/registry.go:103
  | github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights.(*concurrentBufferIngester).ingest
  | 	github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/ingester.go:108
  | github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights.(*concurrentBufferIngester).Start.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/ingester.go:72
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
  | runtime.goexit
  | 	GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) runtime.errorString
-- report composition:
runtime.errorString
panic.go:884: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/asm_amd64.s#L1593-L1595

cockroach/pkg/util/stop/stopper.go

Lines 469 to 471 in 8d065df

	sp.UpdateGoroutineIDToCurrent()
	f(ctx)
	}()

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/ingester.go#L71-L73
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/ingester.go#L107-L109
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/registry.go#L102-L104
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/detector.go#L49-L51
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/detector.go#L81-L83
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/detector.go#L134-L136

cockroach/pkg/util/quantile/stream.go

Lines 229 to 231 in 8d065df

	const s3 = int64(unsafe.Sizeof(Sample{}))
	return s1 + s2 + s3*int64(len(s.b)+len(s.l))
	}

GOROOT/src/runtime/signal_unix.go#L838-L840
GOROOT/src/runtime/panic.go#L259-L261
GOROOT/src/runtime/panic.go#L883-L885

GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/ingester.go in pkg/sql/sqlstats/insights.(*concurrentBufferIngester).Start.func1 at line 72
pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/ingester.go in pkg/sql/sqlstats/insights.(*concurrentBufferIngester).ingest at line 108
pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/registry.go in pkg/sql/sqlstats/insights.(*lockingRegistry).ObserveTransaction at line 103
pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/detector.go in pkg/sql/sqlstats/insights.(*compositeDetector).isSlow at line 50
pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/detector.go in pkg/sql/sqlstats/insights.(*anomalyDetector).isSlow at line 82
pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/detector.go in pkg/sql/sqlstats/insights.(*anomalyDetector).withFingerprintLatencySummary at line 135
pkg/util/quantile/stream.go in pkg/util/quantile.(*Stream).ByteSize at line 230
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 839
GOROOT/src/runtime/panic.go in runtime.panicmem at line 260
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884

Tags

Tag	Value
Command	start-single-node
Environment	v23.1.13
Go Version	go1.19.13
Platform	linux amd64
Distribution	CCL
Cockroach Release	v23.1.13
Cockroach SHA	8d065df
# of CPUs	6
# of Goroutines	382

Jira issue: CRDB-38874

[yuzefovich]

seems to have the same root cause as #124407

[yuzefovich]

Actually never mind, this seems different from #124407