Skip to content

runtime: programs that allocate in a loop can blow past the heap goal #75848

@mknyszek

Description

@mknyszek

As part of investigating #74052, I did a deep-dive into the root cause of the OOMs. That issue is closed because the test allocates less aggressively, but it is not a complete fix.

There's been this problem with GC pacing (e.g. #37331) since long before the last pacer overhaul (in Go 1.18) that doesn't really ever seem to affect real programs, but does make it really easy to write flaky tests. (I don't think I've ever seen a bug report about this for real programs, in all this time.) I remember investigating the same root cause with #37331, which remains unresolved.

The root of the problem appears to be thus: at the beginning of the mark phase, things are good and we have a tiny scannable heap. From the pacer's perspective, we're confident that the dedicated workers will handle this. GC credit is abundant, because we've got so much runway compared to the size of the heap we expect to scan. GC credit is so abundant in fact, that when a goroutine goes to assist, it can steal some background work credit with a very favorable conversion rate (60:1, heap:scan, or higher), netting it a tremendous amount of allocation credit. Later in the GC cycle, if for some reason the background workers lag behind (even just the OS temporarily descheduling the thread), all the goroutines just allocating in a loop have an extraordinary amount of combined runway before they end up doing any kind of assist.

In this way, any program that just allocates in a loop (without slowing itself down by doing anything with that memory) can end up completely blowing past the heap goal.

#73136 appears to be another symptom, but only on the new darwin/arm64 longtest builders. Probably just something slightly different about scheduling that's causing this to happen. If the mark worker threads can get a foothold with the OS, then everything is usually fine. It's hard for allocating goroutines to straight-up outpace mark workers blasting through a tiny heap.

Metadata

Metadata

Assignees

Labels

BugReportIssues describing a possible bug in the Go implementation.NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.compiler/runtimeIssues related to the Go compiler and/or runtime.

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions