Source file src/runtime/mgc.go
1 // Copyright 2009 The Go Authors. All rights reserved. 2 // Use of this source code is governed by a BSD-style 3 // license that can be found in the LICENSE file. 4 5 // Garbage collector (GC). 6 // 7 // The GC runs concurrently with mutator threads, is type accurate (aka precise), allows multiple 8 // GC thread to run in parallel. It is a concurrent mark and sweep that uses a write barrier. It is 9 // non-generational and non-compacting. Allocation is done using size segregated per P allocation 10 // areas to minimize fragmentation while eliminating locks in the common case. 11 // 12 // The algorithm decomposes into several steps. 13 // This is a high level description of the algorithm being used. For an overview of GC a good 14 // place to start is Richard Jones' gchandbook.org. 15 // 16 // The algorithm's intellectual heritage includes Dijkstra's on-the-fly algorithm, see 17 // Edsger W. Dijkstra, Leslie Lamport, A. J. Martin, C. S. Scholten, and E. F. M. Steffens. 1978. 18 // On-the-fly garbage collection: an exercise in cooperation. Commun. ACM 21, 11 (November 1978), 19 // 966-975. 20 // For journal quality proofs that these steps are complete, correct, and terminate see 21 // Hudson, R., and Moss, J.E.B. Copying Garbage Collection without stopping the world. 22 // Concurrency and Computation: Practice and Experience 15(3-5), 2003. 23 // 24 // 1. GC performs sweep termination. 25 // 26 // a. Stop the world. This causes all Ps to reach a GC safe-point. 27 // 28 // b. Sweep any unswept spans. There will only be unswept spans if 29 // this GC cycle was forced before the expected time. 30 // 31 // 2. GC performs the mark phase. 32 // 33 // a. Prepare for the mark phase by setting gcphase to _GCmark 34 // (from _GCoff), enabling the write barrier, enabling mutator 35 // assists, and enqueueing root mark jobs. No objects may be 36 // scanned until all Ps have enabled the write barrier, which is 37 // accomplished using STW. 38 // 39 // b. Start the world. From this point, GC work is done by mark 40 // workers started by the scheduler and by assists performed as 41 // part of allocation. The write barrier shades both the 42 // overwritten pointer and the new pointer value for any pointer 43 // writes (see mbarrier.go for details). Newly allocated objects 44 // are immediately marked black. 45 // 46 // c. GC performs root marking jobs. This includes scanning all 47 // stacks, shading all globals, and shading any heap pointers in 48 // off-heap runtime data structures. Scanning a stack stops a 49 // goroutine, shades any pointers found on its stack, and then 50 // resumes the goroutine. 51 // 52 // d. GC drains the work queue of grey objects, scanning each grey 53 // object to black and shading all pointers found in the object 54 // (which in turn may add those pointers to the work queue). 55 // 56 // e. Because GC work is spread across local caches, GC uses a 57 // distributed termination algorithm to detect when there are no 58 // more root marking jobs or grey objects (see gcMarkDone). At this 59 // point, GC transitions to mark termination. 60 // 61 // 3. GC performs mark termination. 62 // 63 // a. Stop the world. 64 // 65 // b. Set gcphase to _GCmarktermination, and disable workers and 66 // assists. 67 // 68 // c. Perform housekeeping like flushing mcaches. 69 // 70 // 4. GC performs the sweep phase. 71 // 72 // a. Prepare for the sweep phase by setting gcphase to _GCoff, 73 // setting up sweep state and disabling the write barrier. 74 // 75 // b. Start the world. From this point on, newly allocated objects 76 // are white, and allocating sweeps spans before use if necessary. 77 // 78 // c. GC does concurrent sweeping in the background and in response 79 // to allocation. See description below. 80 // 81 // 5. When sufficient allocation has taken place, replay the sequence 82 // starting with 1 above. See discussion of GC rate below. 83 84 // Concurrent sweep. 85 // 86 // The sweep phase proceeds concurrently with normal program execution. 87 // The heap is swept span-by-span both lazily (when a goroutine needs another span) 88 // and concurrently in a background goroutine (this helps programs that are not CPU bound). 89 // At the end of STW mark termination all spans are marked as "needs sweeping". 90 // 91 // The background sweeper goroutine simply sweeps spans one-by-one. 92 // 93 // To avoid requesting more OS memory while there are unswept spans, when a 94 // goroutine needs another span, it first attempts to reclaim that much memory 95 // by sweeping. When a goroutine needs to allocate a new small-object span, it 96 // sweeps small-object spans for the same object size until it frees at least 97 // one object. When a goroutine needs to allocate large-object span from heap, 98 // it sweeps spans until it frees at least that many pages into heap. There is 99 // one case where this may not suffice: if a goroutine sweeps and frees two 100 // nonadjacent one-page spans to the heap, it will allocate a new two-page 101 // span, but there can still be other one-page unswept spans which could be 102 // combined into a two-page span. 103 // 104 // It's critical to ensure that no operations proceed on unswept spans (that would corrupt 105 // mark bits in GC bitmap). During GC all mcaches are flushed into the central cache, 106 // so they are empty. When a goroutine grabs a new span into mcache, it sweeps it. 107 // When a goroutine explicitly frees an object or sets a finalizer, it ensures that 108 // the span is swept (either by sweeping it, or by waiting for the concurrent sweep to finish). 109 // The finalizer goroutine is kicked off only when all spans are swept. 110 // When the next GC starts, it sweeps all not-yet-swept spans (if any). 111 112 // GC rate. 113 // Next GC is after we've allocated an extra amount of memory proportional to 114 // the amount already in use. The proportion is controlled by GOGC environment variable 115 // (100 by default). If GOGC=100 and we're using 4M, we'll GC again when we get to 8M 116 // (this mark is computed by the gcController.heapGoal method). This keeps the GC cost in 117 // linear proportion to the allocation cost. Adjusting GOGC just changes the linear constant 118 // (and also the amount of extra memory used). 119 120 // Oblets 121 // 122 // In order to prevent long pauses while scanning large objects and to 123 // improve parallelism, the garbage collector breaks up scan jobs for 124 // objects larger than maxObletBytes into "oblets" of at most 125 // maxObletBytes. When scanning encounters the beginning of a large 126 // object, it scans only the first oblet and enqueues the remaining 127 // oblets as new scan jobs. 128 129 package runtime 130 131 import ( 132 "internal/cpu" 133 "internal/goarch" 134 "internal/goexperiment" 135 "internal/runtime/atomic" 136 "internal/runtime/gc" 137 "unsafe" 138 ) 139 140 const ( 141 _DebugGC = 0 142 143 // concurrentSweep is a debug flag. Disabling this flag 144 // ensures all spans are swept while the world is stopped. 145 concurrentSweep = true 146 147 // debugScanConservative enables debug logging for stack 148 // frames that are scanned conservatively. 149 debugScanConservative = false 150 151 // sweepMinHeapDistance is a lower bound on the heap distance 152 // (in bytes) reserved for concurrent sweeping between GC 153 // cycles. 154 sweepMinHeapDistance = 1024 * 1024 155 ) 156 157 // heapObjectsCanMove always returns false in the current garbage collector. 158 // It exists for go4.org/unsafe/assume-no-moving-gc, which is an 159 // unfortunate idea that had an even more unfortunate implementation. 160 // Every time a new Go release happened, the package stopped building, 161 // and the authors had to add a new file with a new //go:build line, and 162 // then the entire ecosystem of packages with that as a dependency had to 163 // explicitly update to the new version. Many packages depend on 164 // assume-no-moving-gc transitively, through paths like 165 // inet.af/netaddr -> go4.org/intern -> assume-no-moving-gc. 166 // This was causing a significant amount of friction around each new 167 // release, so we added this bool for the package to //go:linkname 168 // instead. The bool is still unfortunate, but it's not as bad as 169 // breaking the ecosystem on every new release. 170 // 171 // If the Go garbage collector ever does move heap objects, we can set 172 // this to true to break all the programs using assume-no-moving-gc. 173 // 174 //go:linkname heapObjectsCanMove 175 func heapObjectsCanMove() bool { 176 return false 177 } 178 179 func gcinit() { 180 if unsafe.Sizeof(workbuf{}) != _WorkbufSize { 181 throw("size of Workbuf is suboptimal") 182 } 183 // No sweep on the first cycle. 184 sweep.active.state.Store(sweepDrainedMask) 185 186 // Initialize GC pacer state. 187 // Use the environment variable GOGC for the initial gcPercent value. 188 // Use the environment variable GOMEMLIMIT for the initial memoryLimit value. 189 gcController.init(readGOGC(), readGOMEMLIMIT()) 190 191 // Set up the cleanup block ptr mask. 192 for i := range cleanupBlockPtrMask { 193 cleanupBlockPtrMask[i] = 0xff 194 } 195 196 work.startSema = 1 197 work.markDoneSema = 1 198 lockInit(&work.sweepWaiters.lock, lockRankSweepWaiters) 199 lockInit(&work.assistQueue.lock, lockRankAssistQueue) 200 lockInit(&work.strongFromWeak.lock, lockRankStrongFromWeakQueue) 201 lockInit(&work.wbufSpans.lock, lockRankWbufSpans) 202 lockInit(&gcCleanups.lock, lockRankCleanupQueue) 203 } 204 205 // gcenable is called after the bulk of the runtime initialization, 206 // just before we're about to start letting user code run. 207 // It kicks off the background sweeper goroutine, the background 208 // scavenger goroutine, and enables GC. 209 func gcenable() { 210 // Kick off sweeping and scavenging. 211 c := make(chan int, 2) 212 go bgsweep(c) 213 go bgscavenge(c) 214 <-c 215 <-c 216 memstats.enablegc = true // now that runtime is initialized, GC is okay 217 } 218 219 // Garbage collector phase. 220 // Indicates to write barrier and synchronization task to perform. 221 var gcphase uint32 222 223 // The compiler knows about this variable. 224 // If you change it, you must change builtin/runtime.go, too. 225 // If you change the first four bytes, you must also change the write 226 // barrier insertion code. 227 // 228 // writeBarrier should be an internal detail, 229 // but widely used packages access it using linkname. 230 // Notable members of the hall of shame include: 231 // - github.com/bytedance/sonic 232 // 233 // Do not remove or change the type signature. 234 // See go.dev/issue/67401. 235 // 236 //go:linkname writeBarrier 237 var writeBarrier struct { 238 enabled bool // compiler emits a check of this before calling write barrier 239 pad [3]byte // compiler uses 32-bit load for "enabled" field 240 alignme uint64 // guarantee alignment so that compiler can use a 32 or 64-bit load 241 } 242 243 // gcBlackenEnabled is 1 if mutator assists and background mark 244 // workers are allowed to blacken objects. This must only be set when 245 // gcphase == _GCmark. 246 var gcBlackenEnabled uint32 247 248 const ( 249 _GCoff = iota // GC not running; sweeping in background, write barrier disabled 250 _GCmark // GC marking roots and workbufs: allocate black, write barrier ENABLED 251 _GCmarktermination // GC mark termination: allocate black, P's help GC, write barrier ENABLED 252 ) 253 254 //go:nosplit 255 func setGCPhase(x uint32) { 256 atomic.Store(&gcphase, x) 257 writeBarrier.enabled = gcphase == _GCmark || gcphase == _GCmarktermination 258 } 259 260 // gcMarkWorkerMode represents the mode that a concurrent mark worker 261 // should operate in. 262 // 263 // Concurrent marking happens through four different mechanisms. One 264 // is mutator assists, which happen in response to allocations and are 265 // not scheduled. The other three are variations in the per-P mark 266 // workers and are distinguished by gcMarkWorkerMode. 267 type gcMarkWorkerMode int 268 269 const ( 270 // gcMarkWorkerNotWorker indicates that the next scheduled G is not 271 // starting work and the mode should be ignored. 272 gcMarkWorkerNotWorker gcMarkWorkerMode = iota 273 274 // gcMarkWorkerDedicatedMode indicates that the P of a mark 275 // worker is dedicated to running that mark worker. The mark 276 // worker should run without preemption. 277 gcMarkWorkerDedicatedMode 278 279 // gcMarkWorkerFractionalMode indicates that a P is currently 280 // running the "fractional" mark worker. The fractional worker 281 // is necessary when GOMAXPROCS*gcBackgroundUtilization is not 282 // an integer and using only dedicated workers would result in 283 // utilization too far from the target of gcBackgroundUtilization. 284 // The fractional worker should run until it is preempted and 285 // will be scheduled to pick up the fractional part of 286 // GOMAXPROCS*gcBackgroundUtilization. 287 gcMarkWorkerFractionalMode 288 289 // gcMarkWorkerIdleMode indicates that a P is running the mark 290 // worker because it has nothing else to do. The idle worker 291 // should run until it is preempted and account its time 292 // against gcController.idleMarkTime. 293 gcMarkWorkerIdleMode 294 ) 295 296 // gcMarkWorkerModeStrings are the strings labels of gcMarkWorkerModes 297 // to use in execution traces. 298 var gcMarkWorkerModeStrings = [...]string{ 299 "Not worker", 300 "GC (dedicated)", 301 "GC (fractional)", 302 "GC (idle)", 303 } 304 305 // pollFractionalWorkerExit reports whether a fractional mark worker 306 // should self-preempt. It assumes it is called from the fractional 307 // worker. 308 func pollFractionalWorkerExit() bool { 309 // This should be kept in sync with the fractional worker 310 // scheduler logic in findRunnableGCWorker. 311 now := nanotime() 312 delta := now - gcController.markStartTime 313 if delta <= 0 { 314 return true 315 } 316 p := getg().m.p.ptr() 317 selfTime := p.gcFractionalMarkTime + (now - p.gcMarkWorkerStartTime) 318 // Add some slack to the utilization goal so that the 319 // fractional worker isn't behind again the instant it exits. 320 return float64(selfTime)/float64(delta) > 1.2*gcController.fractionalUtilizationGoal 321 } 322 323 var work workType 324 325 type workType struct { 326 full lfstack // lock-free list of full blocks workbuf 327 _ cpu.CacheLinePad // prevents false-sharing between full and empty 328 empty lfstack // lock-free list of empty blocks workbuf 329 _ cpu.CacheLinePad // prevents false-sharing between empty and nproc/nwait 330 331 wbufSpans struct { 332 lock mutex 333 // free is a list of spans dedicated to workbufs, but 334 // that don't currently contain any workbufs. 335 free mSpanList 336 // busy is a list of all spans containing workbufs on 337 // one of the workbuf lists. 338 busy mSpanList 339 } 340 _ cpu.CacheLinePad // prevents false-sharing between wbufSpans and spanq 341 342 // Global queue of spans to scan. 343 // 344 // Only used if goexperiment.GreenTeaGC. 345 spanq spanQueue 346 347 // Restore 64-bit alignment on 32-bit. 348 // _ uint32 349 350 // bytesMarked is the number of bytes marked this cycle. This 351 // includes bytes blackened in scanned objects, noscan objects 352 // that go straight to black, objects allocated as black during 353 // the cycle, and permagrey objects scanned by markroot during 354 // the concurrent scan phase. 355 // 356 // This is updated atomically during the cycle. Updates may be batched 357 // arbitrarily, since the value is only read at the end of the cycle. 358 // 359 // Because of benign races during marking, this number may not 360 // be the exact number of marked bytes, but it should be very 361 // close. 362 // 363 // Put this field here because it needs 64-bit atomic access 364 // (and thus 8-byte alignment even on 32-bit architectures). 365 bytesMarked uint64 366 367 markrootNext uint32 // next markroot job 368 markrootJobs uint32 // number of markroot jobs 369 370 nproc uint32 371 tstart int64 372 nwait uint32 373 374 // Number of roots of various root types. Set by gcPrepareMarkRoots. 375 // 376 // nStackRoots == len(stackRoots), but we have nStackRoots for 377 // consistency. 378 nDataRoots, nBSSRoots, nSpanRoots, nStackRoots int 379 380 // Base indexes of each root type. Set by gcPrepareMarkRoots. 381 baseData, baseBSS, baseSpans, baseStacks, baseEnd uint32 382 383 // stackRoots is a snapshot of all of the Gs that existed 384 // before the beginning of concurrent marking. The backing 385 // store of this must not be modified because it might be 386 // shared with allgs. 387 stackRoots []*g 388 389 // Each type of GC state transition is protected by a lock. 390 // Since multiple threads can simultaneously detect the state 391 // transition condition, any thread that detects a transition 392 // condition must acquire the appropriate transition lock, 393 // re-check the transition condition and return if it no 394 // longer holds or perform the transition if it does. 395 // Likewise, any transition must invalidate the transition 396 // condition before releasing the lock. This ensures that each 397 // transition is performed by exactly one thread and threads 398 // that need the transition to happen block until it has 399 // happened. 400 // 401 // startSema protects the transition from "off" to mark or 402 // mark termination. 403 startSema uint32 404 // markDoneSema protects transitions from mark to mark termination. 405 markDoneSema uint32 406 407 bgMarkDone uint32 // cas to 1 when at a background mark completion point 408 // Background mark completion signaling 409 410 // mode is the concurrency mode of the current GC cycle. 411 mode gcMode 412 413 // userForced indicates the current GC cycle was forced by an 414 // explicit user call. 415 userForced bool 416 417 // initialHeapLive is the value of gcController.heapLive at the 418 // beginning of this GC cycle. 419 initialHeapLive uint64 420 421 // assistQueue is a queue of assists that are blocked because 422 // there was neither enough credit to steal or enough work to 423 // do. 424 assistQueue struct { 425 lock mutex 426 q gQueue 427 } 428 429 // sweepWaiters is a list of blocked goroutines to wake when 430 // we transition from mark termination to sweep. 431 sweepWaiters struct { 432 lock mutex 433 list gList 434 } 435 436 // strongFromWeak controls how the GC interacts with weak->strong 437 // pointer conversions. 438 strongFromWeak struct { 439 // block is a flag set during mark termination that prevents 440 // new weak->strong conversions from executing by blocking the 441 // goroutine and enqueuing it onto q. 442 // 443 // Mutated only by one goroutine at a time in gcMarkDone, 444 // with globally-synchronizing events like forEachP and 445 // stopTheWorld. 446 block bool 447 448 // q is a queue of goroutines that attempted to perform a 449 // weak->strong conversion during mark termination. 450 // 451 // Protected by lock. 452 lock mutex 453 q gQueue 454 } 455 456 // cycles is the number of completed GC cycles, where a GC 457 // cycle is sweep termination, mark, mark termination, and 458 // sweep. This differs from memstats.numgc, which is 459 // incremented at mark termination. 460 cycles atomic.Uint32 461 462 // Timing/utilization stats for this cycle. 463 stwprocs, maxprocs int32 464 tSweepTerm, tMark, tMarkTerm, tEnd int64 // nanotime() of phase start 465 466 // pauseNS is the total STW time this cycle, measured as the time between 467 // when stopping began (just before trying to stop Ps) and just after the 468 // world started again. 469 pauseNS int64 470 471 // debug.gctrace heap sizes for this cycle. 472 heap0, heap1, heap2 uint64 473 474 // Cumulative estimated CPU usage. 475 cpuStats 476 } 477 478 // GC runs a garbage collection and blocks the caller until the 479 // garbage collection is complete. It may also block the entire 480 // program. 481 func GC() { 482 // We consider a cycle to be: sweep termination, mark, mark 483 // termination, and sweep. This function shouldn't return 484 // until a full cycle has been completed, from beginning to 485 // end. Hence, we always want to finish up the current cycle 486 // and start a new one. That means: 487 // 488 // 1. In sweep termination, mark, or mark termination of cycle 489 // N, wait until mark termination N completes and transitions 490 // to sweep N. 491 // 492 // 2. In sweep N, help with sweep N. 493 // 494 // At this point we can begin a full cycle N+1. 495 // 496 // 3. Trigger cycle N+1 by starting sweep termination N+1. 497 // 498 // 4. Wait for mark termination N+1 to complete. 499 // 500 // 5. Help with sweep N+1 until it's done. 501 // 502 // This all has to be written to deal with the fact that the 503 // GC may move ahead on its own. For example, when we block 504 // until mark termination N, we may wake up in cycle N+2. 505 506 // Wait until the current sweep termination, mark, and mark 507 // termination complete. 508 n := work.cycles.Load() 509 gcWaitOnMark(n) 510 511 // We're now in sweep N or later. Trigger GC cycle N+1, which 512 // will first finish sweep N if necessary and then enter sweep 513 // termination N+1. 514 gcStart(gcTrigger{kind: gcTriggerCycle, n: n + 1}) 515 516 // Wait for mark termination N+1 to complete. 517 gcWaitOnMark(n + 1) 518 519 // Finish sweep N+1 before returning. We do this both to 520 // complete the cycle and because runtime.GC() is often used 521 // as part of tests and benchmarks to get the system into a 522 // relatively stable and isolated state. 523 for work.cycles.Load() == n+1 && sweepone() != ^uintptr(0) { 524 Gosched() 525 } 526 527 // Callers may assume that the heap profile reflects the 528 // just-completed cycle when this returns (historically this 529 // happened because this was a STW GC), but right now the 530 // profile still reflects mark termination N, not N+1. 531 // 532 // As soon as all of the sweep frees from cycle N+1 are done, 533 // we can go ahead and publish the heap profile. 534 // 535 // First, wait for sweeping to finish. (We know there are no 536 // more spans on the sweep queue, but we may be concurrently 537 // sweeping spans, so we have to wait.) 538 for work.cycles.Load() == n+1 && !isSweepDone() { 539 Gosched() 540 } 541 542 // Now we're really done with sweeping, so we can publish the 543 // stable heap profile. Only do this if we haven't already hit 544 // another mark termination. 545 mp := acquirem() 546 cycle := work.cycles.Load() 547 if cycle == n+1 || (gcphase == _GCmark && cycle == n+2) { 548 mProf_PostSweep() 549 } 550 releasem(mp) 551 } 552 553 // gcWaitOnMark blocks until GC finishes the Nth mark phase. If GC has 554 // already completed this mark phase, it returns immediately. 555 func gcWaitOnMark(n uint32) { 556 for { 557 // Disable phase transitions. 558 lock(&work.sweepWaiters.lock) 559 nMarks := work.cycles.Load() 560 if gcphase != _GCmark { 561 // We've already completed this cycle's mark. 562 nMarks++ 563 } 564 if nMarks > n { 565 // We're done. 566 unlock(&work.sweepWaiters.lock) 567 return 568 } 569 570 // Wait until sweep termination, mark, and mark 571 // termination of cycle N complete. 572 work.sweepWaiters.list.push(getg()) 573 goparkunlock(&work.sweepWaiters.lock, waitReasonWaitForGCCycle, traceBlockUntilGCEnds, 1) 574 } 575 } 576 577 // gcMode indicates how concurrent a GC cycle should be. 578 type gcMode int 579 580 const ( 581 gcBackgroundMode gcMode = iota // concurrent GC and sweep 582 gcForceMode // stop-the-world GC now, concurrent sweep 583 gcForceBlockMode // stop-the-world GC now and STW sweep (forced by user) 584 ) 585 586 // A gcTrigger is a predicate for starting a GC cycle. Specifically, 587 // it is an exit condition for the _GCoff phase. 588 type gcTrigger struct { 589 kind gcTriggerKind 590 now int64 // gcTriggerTime: current time 591 n uint32 // gcTriggerCycle: cycle number to start 592 } 593 594 type gcTriggerKind int 595 596 const ( 597 // gcTriggerHeap indicates that a cycle should be started when 598 // the heap size reaches the trigger heap size computed by the 599 // controller. 600 gcTriggerHeap gcTriggerKind = iota 601 602 // gcTriggerTime indicates that a cycle should be started when 603 // it's been more than forcegcperiod nanoseconds since the 604 // previous GC cycle. 605 gcTriggerTime 606 607 // gcTriggerCycle indicates that a cycle should be started if 608 // we have not yet started cycle number gcTrigger.n (relative 609 // to work.cycles). 610 gcTriggerCycle 611 ) 612 613 // test reports whether the trigger condition is satisfied, meaning 614 // that the exit condition for the _GCoff phase has been met. The exit 615 // condition should be tested when allocating. 616 func (t gcTrigger) test() bool { 617 if !memstats.enablegc || panicking.Load() != 0 || gcphase != _GCoff { 618 return false 619 } 620 switch t.kind { 621 case gcTriggerHeap: 622 trigger, _ := gcController.trigger() 623 return gcController.heapLive.Load() >= trigger 624 case gcTriggerTime: 625 if gcController.gcPercent.Load() < 0 { 626 return false 627 } 628 lastgc := int64(atomic.Load64(&memstats.last_gc_nanotime)) 629 return lastgc != 0 && t.now-lastgc > forcegcperiod 630 case gcTriggerCycle: 631 // t.n > work.cycles, but accounting for wraparound. 632 return int32(t.n-work.cycles.Load()) > 0 633 } 634 return true 635 } 636 637 // gcStart starts the GC. It transitions from _GCoff to _GCmark (if 638 // debug.gcstoptheworld == 0) or performs all of GC (if 639 // debug.gcstoptheworld != 0). 640 // 641 // This may return without performing this transition in some cases, 642 // such as when called on a system stack or with locks held. 643 func gcStart(trigger gcTrigger) { 644 // Since this is called from malloc and malloc is called in 645 // the guts of a number of libraries that might be holding 646 // locks, don't attempt to start GC in non-preemptible or 647 // potentially unstable situations. 648 mp := acquirem() 649 if gp := getg(); gp == mp.g0 || mp.locks > 1 || mp.preemptoff != "" { 650 releasem(mp) 651 return 652 } 653 releasem(mp) 654 mp = nil 655 656 if gp := getg(); gp.bubble != nil { 657 // Disassociate the G from its synctest bubble while allocating. 658 // This is less elegant than incrementing the group's active count, 659 // but avoids any contamination between GC and synctest. 660 bubble := gp.bubble 661 gp.bubble = nil 662 defer func() { 663 gp.bubble = bubble 664 }() 665 } 666 667 // Pick up the remaining unswept/not being swept spans concurrently 668 // 669 // This shouldn't happen if we're being invoked in background 670 // mode since proportional sweep should have just finished 671 // sweeping everything, but rounding errors, etc, may leave a 672 // few spans unswept. In forced mode, this is necessary since 673 // GC can be forced at any point in the sweeping cycle. 674 // 675 // We check the transition condition continuously here in case 676 // this G gets delayed in to the next GC cycle. 677 for trigger.test() && sweepone() != ^uintptr(0) { 678 } 679 680 // Perform GC initialization and the sweep termination 681 // transition. 682 semacquire(&work.startSema) 683 // Re-check transition condition under transition lock. 684 if !trigger.test() { 685 semrelease(&work.startSema) 686 return 687 } 688 689 // In gcstoptheworld debug mode, upgrade the mode accordingly. 690 // We do this after re-checking the transition condition so 691 // that multiple goroutines that detect the heap trigger don't 692 // start multiple STW GCs. 693 mode := gcBackgroundMode 694 if debug.gcstoptheworld == 1 { 695 mode = gcForceMode 696 } else if debug.gcstoptheworld == 2 { 697 mode = gcForceBlockMode 698 } 699 700 // Ok, we're doing it! Stop everybody else 701 semacquire(&gcsema) 702 semacquire(&worldsema) 703 704 // For stats, check if this GC was forced by the user. 705 // Update it under gcsema to avoid gctrace getting wrong values. 706 work.userForced = trigger.kind == gcTriggerCycle 707 708 trace := traceAcquire() 709 if trace.ok() { 710 trace.GCStart() 711 traceRelease(trace) 712 } 713 714 // Check that all Ps have finished deferred mcache flushes. 715 for _, p := range allp { 716 if fg := p.mcache.flushGen.Load(); fg != mheap_.sweepgen { 717 println("runtime: p", p.id, "flushGen", fg, "!= sweepgen", mheap_.sweepgen) 718 throw("p mcache not flushed") 719 } 720 // Initialize ptrBuf if necessary. 721 if goexperiment.GreenTeaGC && p.gcw.ptrBuf == nil { 722 p.gcw.ptrBuf = (*[gc.PageSize / goarch.PtrSize]uintptr)(persistentalloc(gc.PageSize, goarch.PtrSize, &memstats.gcMiscSys)) 723 } 724 } 725 726 gcBgMarkStartWorkers() 727 728 systemstack(gcResetMarkState) 729 730 work.stwprocs, work.maxprocs = gomaxprocs, gomaxprocs 731 if work.stwprocs > numCPUStartup { 732 // This is used to compute CPU time of the STW phases, so it 733 // can't be more than the CPU count, even if GOMAXPROCS is. 734 work.stwprocs = numCPUStartup 735 } 736 work.heap0 = gcController.heapLive.Load() 737 work.pauseNS = 0 738 work.mode = mode 739 740 now := nanotime() 741 work.tSweepTerm = now 742 var stw worldStop 743 systemstack(func() { 744 stw = stopTheWorldWithSema(stwGCSweepTerm) 745 }) 746 747 // Accumulate fine-grained stopping time. 748 work.cpuStats.accumulateGCPauseTime(stw.stoppingCPUTime, 1) 749 750 // Finish sweep before we start concurrent scan. 751 systemstack(func() { 752 finishsweep_m() 753 }) 754 755 // clearpools before we start the GC. If we wait the memory will not be 756 // reclaimed until the next GC cycle. 757 clearpools() 758 759 work.cycles.Add(1) 760 761 // Assists and workers can start the moment we start 762 // the world. 763 gcController.startCycle(now, int(gomaxprocs), trigger) 764 765 // Notify the CPU limiter that assists may begin. 766 gcCPULimiter.startGCTransition(true, now) 767 768 // In STW mode, disable scheduling of user Gs. This may also 769 // disable scheduling of this goroutine, so it may block as 770 // soon as we start the world again. 771 if mode != gcBackgroundMode { 772 schedEnableUser(false) 773 } 774 775 // Enter concurrent mark phase and enable 776 // write barriers. 777 // 778 // Because the world is stopped, all Ps will 779 // observe that write barriers are enabled by 780 // the time we start the world and begin 781 // scanning. 782 // 783 // Write barriers must be enabled before assists are 784 // enabled because they must be enabled before 785 // any non-leaf heap objects are marked. Since 786 // allocations are blocked until assists can 787 // happen, we want to enable assists as early as 788 // possible. 789 setGCPhase(_GCmark) 790 791 gcBgMarkPrepare() // Must happen before assists are enabled. 792 gcPrepareMarkRoots() 793 794 // Mark all active tinyalloc blocks. Since we're 795 // allocating from these, they need to be black like 796 // other allocations. The alternative is to blacken 797 // the tiny block on every allocation from it, which 798 // would slow down the tiny allocator. 799 gcMarkTinyAllocs() 800 801 // At this point all Ps have enabled the write 802 // barrier, thus maintaining the no white to 803 // black invariant. Enable mutator assists to 804 // put back-pressure on fast allocating 805 // mutators. 806 atomic.Store(&gcBlackenEnabled, 1) 807 808 // In STW mode, we could block the instant systemstack 809 // returns, so make sure we're not preemptible. 810 mp = acquirem() 811 812 // Update the CPU stats pause time. 813 // 814 // Use maxprocs instead of stwprocs here because the total time 815 // computed in the CPU stats is based on maxprocs, and we want them 816 // to be comparable. 817 work.cpuStats.accumulateGCPauseTime(nanotime()-stw.finishedStopping, work.maxprocs) 818 819 // Concurrent mark. 820 systemstack(func() { 821 now = startTheWorldWithSema(0, stw) 822 work.pauseNS += now - stw.startedStopping 823 work.tMark = now 824 825 // Release the CPU limiter. 826 gcCPULimiter.finishGCTransition(now) 827 }) 828 829 // Release the world sema before Gosched() in STW mode 830 // because we will need to reacquire it later but before 831 // this goroutine becomes runnable again, and we could 832 // self-deadlock otherwise. 833 semrelease(&worldsema) 834 releasem(mp) 835 836 // Make sure we block instead of returning to user code 837 // in STW mode. 838 if mode != gcBackgroundMode { 839 Gosched() 840 } 841 842 semrelease(&work.startSema) 843 } 844 845 // gcMarkDoneFlushed counts the number of P's with flushed work. 846 // 847 // Ideally this would be a captured local in gcMarkDone, but forEachP 848 // escapes its callback closure, so it can't capture anything. 849 // 850 // This is protected by markDoneSema. 851 var gcMarkDoneFlushed uint32 852 853 // gcDebugMarkDone contains fields used to debug/test mark termination. 854 var gcDebugMarkDone struct { 855 // spinAfterRaggedBarrier forces gcMarkDone to spin after it executes 856 // the ragged barrier. 857 spinAfterRaggedBarrier atomic.Bool 858 859 // restartedDueTo27993 indicates that we restarted mark termination 860 // due to the bug described in issue #27993. 861 // 862 // Protected by worldsema. 863 restartedDueTo27993 bool 864 } 865 866 // gcMarkDone transitions the GC from mark to mark termination if all 867 // reachable objects have been marked (that is, there are no grey 868 // objects and can be no more in the future). Otherwise, it flushes 869 // all local work to the global queues where it can be discovered by 870 // other workers. 871 // 872 // This should be called when all local mark work has been drained and 873 // there are no remaining workers. Specifically, when 874 // 875 // work.nwait == work.nproc && !gcMarkWorkAvailable(p) 876 // 877 // The calling context must be preemptible. 878 // 879 // Flushing local work is important because idle Ps may have local 880 // work queued. This is the only way to make that work visible and 881 // drive GC to completion. 882 // 883 // It is explicitly okay to have write barriers in this function. If 884 // it does transition to mark termination, then all reachable objects 885 // have been marked, so the write barrier cannot shade any more 886 // objects. 887 func gcMarkDone() { 888 // Ensure only one thread is running the ragged barrier at a 889 // time. 890 semacquire(&work.markDoneSema) 891 892 top: 893 // Re-check transition condition under transition lock. 894 // 895 // It's critical that this checks the global work queues are 896 // empty before performing the ragged barrier. Otherwise, 897 // there could be global work that a P could take after the P 898 // has passed the ragged barrier. 899 if !(gcphase == _GCmark && work.nwait == work.nproc && !gcMarkWorkAvailable(nil)) { 900 semrelease(&work.markDoneSema) 901 return 902 } 903 904 // forEachP needs worldsema to execute, and we'll need it to 905 // stop the world later, so acquire worldsema now. 906 semacquire(&worldsema) 907 908 // Prevent weak->strong conversions from generating additional 909 // GC work. forEachP will guarantee that it is observed globally. 910 work.strongFromWeak.block = true 911 912 // Flush all local buffers and collect flushedWork flags. 913 gcMarkDoneFlushed = 0 914 forEachP(waitReasonGCMarkTermination, func(pp *p) { 915 // Flush the write barrier buffer, since this may add 916 // work to the gcWork. 917 wbBufFlush1(pp) 918 919 // Flush the gcWork, since this may create global work 920 // and set the flushedWork flag. 921 // 922 // TODO(austin): Break up these workbufs to 923 // better distribute work. 924 pp.gcw.dispose() 925 // Collect the flushedWork flag. 926 if pp.gcw.flushedWork { 927 atomic.Xadd(&gcMarkDoneFlushed, 1) 928 pp.gcw.flushedWork = false 929 } 930 }) 931 932 if gcMarkDoneFlushed != 0 { 933 // More grey objects were discovered since the 934 // previous termination check, so there may be more 935 // work to do. Keep going. It's possible the 936 // transition condition became true again during the 937 // ragged barrier, so re-check it. 938 semrelease(&worldsema) 939 goto top 940 } 941 942 // For debugging/testing. 943 for gcDebugMarkDone.spinAfterRaggedBarrier.Load() { 944 } 945 946 // There was no global work, no local work, and no Ps 947 // communicated work since we took markDoneSema. Therefore 948 // there are no grey objects and no more objects can be 949 // shaded. Transition to mark termination. 950 now := nanotime() 951 work.tMarkTerm = now 952 getg().m.preemptoff = "gcing" 953 var stw worldStop 954 systemstack(func() { 955 stw = stopTheWorldWithSema(stwGCMarkTerm) 956 }) 957 // The gcphase is _GCmark, it will transition to _GCmarktermination 958 // below. The important thing is that the wb remains active until 959 // all marking is complete. This includes writes made by the GC. 960 961 // Accumulate fine-grained stopping time. 962 work.cpuStats.accumulateGCPauseTime(stw.stoppingCPUTime, 1) 963 964 // There is sometimes work left over when we enter mark termination due 965 // to write barriers performed after the completion barrier above. 966 // Detect this and resume concurrent mark. This is obviously 967 // unfortunate. 968 // 969 // See issue #27993 for details. 970 // 971 // Switch to the system stack to call wbBufFlush1, though in this case 972 // it doesn't matter because we're non-preemptible anyway. 973 restart := false 974 systemstack(func() { 975 for _, p := range allp { 976 wbBufFlush1(p) 977 if !p.gcw.empty() { 978 restart = true 979 break 980 } 981 } 982 }) 983 if restart { 984 gcDebugMarkDone.restartedDueTo27993 = true 985 986 getg().m.preemptoff = "" 987 systemstack(func() { 988 // Accumulate the time we were stopped before we had to start again. 989 work.cpuStats.accumulateGCPauseTime(nanotime()-stw.finishedStopping, work.maxprocs) 990 991 // Start the world again. 992 now := startTheWorldWithSema(0, stw) 993 work.pauseNS += now - stw.startedStopping 994 }) 995 semrelease(&worldsema) 996 goto top 997 } 998 999 gcComputeStartingStackSize() 1000 1001 // Disable assists and background workers. We must do 1002 // this before waking blocked assists. 1003 atomic.Store(&gcBlackenEnabled, 0) 1004 1005 // Notify the CPU limiter that GC assists will now cease. 1006 gcCPULimiter.startGCTransition(false, now) 1007 1008 // Wake all blocked assists. These will run when we 1009 // start the world again. 1010 gcWakeAllAssists() 1011 1012 // Wake all blocked weak->strong conversions. These will run 1013 // when we start the world again. 1014 work.strongFromWeak.block = false 1015 gcWakeAllStrongFromWeak() 1016 1017 // Likewise, release the transition lock. Blocked 1018 // workers and assists will run when we start the 1019 // world again. 1020 semrelease(&work.markDoneSema) 1021 1022 // In STW mode, re-enable user goroutines. These will be 1023 // queued to run after we start the world. 1024 schedEnableUser(true) 1025 1026 // endCycle depends on all gcWork cache stats being flushed. 1027 // The termination algorithm above ensured that up to 1028 // allocations since the ragged barrier. 1029 gcController.endCycle(now, int(gomaxprocs), work.userForced) 1030 1031 // Perform mark termination. This will restart the world. 1032 gcMarkTermination(stw) 1033 } 1034 1035 // World must be stopped and mark assists and background workers must be 1036 // disabled. 1037 func gcMarkTermination(stw worldStop) { 1038 // Start marktermination (write barrier remains enabled for now). 1039 setGCPhase(_GCmarktermination) 1040 1041 work.heap1 = gcController.heapLive.Load() 1042 startTime := nanotime() 1043 1044 mp := acquirem() 1045 mp.preemptoff = "gcing" 1046 mp.traceback = 2 1047 curgp := mp.curg 1048 // N.B. The execution tracer is not aware of this status 1049 // transition and handles it specially based on the 1050 // wait reason. 1051 casGToWaitingForSuspendG(curgp, _Grunning, waitReasonGarbageCollection) 1052 1053 // Run gc on the g0 stack. We do this so that the g stack 1054 // we're currently running on will no longer change. Cuts 1055 // the root set down a bit (g0 stacks are not scanned, and 1056 // we don't need to scan gc's internal state). We also 1057 // need to switch to g0 so we can shrink the stack. 1058 systemstack(func() { 1059 gcMark(startTime) 1060 // Must return immediately. 1061 // The outer function's stack may have moved 1062 // during gcMark (it shrinks stacks, including the 1063 // outer function's stack), so we must not refer 1064 // to any of its variables. Return back to the 1065 // non-system stack to pick up the new addresses 1066 // before continuing. 1067 }) 1068 1069 var stwSwept bool 1070 systemstack(func() { 1071 work.heap2 = work.bytesMarked 1072 if debug.gccheckmark > 0 { 1073 runCheckmark(func(_ *gcWork) { gcPrepareMarkRoots() }) 1074 } 1075 if debug.checkfinalizers > 0 { 1076 checkFinalizersAndCleanups() 1077 } 1078 1079 // marking is complete so we can turn the write barrier off 1080 setGCPhase(_GCoff) 1081 stwSwept = gcSweep(work.mode) 1082 }) 1083 1084 mp.traceback = 0 1085 casgstatus(curgp, _Gwaiting, _Grunning) 1086 1087 trace := traceAcquire() 1088 if trace.ok() { 1089 trace.GCDone() 1090 traceRelease(trace) 1091 } 1092 1093 // all done 1094 mp.preemptoff = "" 1095 1096 if gcphase != _GCoff { 1097 throw("gc done but gcphase != _GCoff") 1098 } 1099 1100 // Record heapInUse for scavenger. 1101 memstats.lastHeapInUse = gcController.heapInUse.load() 1102 1103 // Update GC trigger and pacing, as well as downstream consumers 1104 // of this pacing information, for the next cycle. 1105 systemstack(gcControllerCommit) 1106 1107 // Update timing memstats 1108 now := nanotime() 1109 sec, nsec, _ := time_now() 1110 unixNow := sec*1e9 + int64(nsec) 1111 work.pauseNS += now - stw.startedStopping 1112 work.tEnd = now 1113 atomic.Store64(&memstats.last_gc_unix, uint64(unixNow)) // must be Unix time to make sense to user 1114 atomic.Store64(&memstats.last_gc_nanotime, uint64(now)) // monotonic time for us 1115 memstats.pause_ns[memstats.numgc%uint32(len(memstats.pause_ns))] = uint64(work.pauseNS) 1116 memstats.pause_end[memstats.numgc%uint32(len(memstats.pause_end))] = uint64(unixNow) 1117 memstats.pause_total_ns += uint64(work.pauseNS) 1118 1119 // Accumulate CPU stats. 1120 // 1121 // Use maxprocs instead of stwprocs for GC pause time because the total time 1122 // computed in the CPU stats is based on maxprocs, and we want them to be 1123 // comparable. 1124 // 1125 // Pass gcMarkPhase=true to accumulate so we can get all the latest GC CPU stats 1126 // in there too. 1127 work.cpuStats.accumulateGCPauseTime(now-stw.finishedStopping, work.maxprocs) 1128 work.cpuStats.accumulate(now, true) 1129 1130 // Compute overall GC CPU utilization. 1131 // Omit idle marking time from the overall utilization here since it's "free". 1132 memstats.gc_cpu_fraction = float64(work.cpuStats.GCTotalTime-work.cpuStats.GCIdleTime) / float64(work.cpuStats.TotalTime) 1133 1134 // Reset assist time and background time stats. 1135 // 1136 // Do this now, instead of at the start of the next GC cycle, because 1137 // these two may keep accumulating even if the GC is not active. 1138 scavenge.assistTime.Store(0) 1139 scavenge.backgroundTime.Store(0) 1140 1141 // Reset idle time stat. 1142 sched.idleTime.Store(0) 1143 1144 if work.userForced { 1145 memstats.numforcedgc++ 1146 } 1147 1148 // Bump GC cycle count and wake goroutines waiting on sweep. 1149 lock(&work.sweepWaiters.lock) 1150 memstats.numgc++ 1151 injectglist(&work.sweepWaiters.list) 1152 unlock(&work.sweepWaiters.lock) 1153 1154 // Increment the scavenge generation now. 1155 // 1156 // This moment represents peak heap in use because we're 1157 // about to start sweeping. 1158 mheap_.pages.scav.index.nextGen() 1159 1160 // Release the CPU limiter. 1161 gcCPULimiter.finishGCTransition(now) 1162 1163 // Finish the current heap profiling cycle and start a new 1164 // heap profiling cycle. We do this before starting the world 1165 // so events don't leak into the wrong cycle. 1166 mProf_NextCycle() 1167 1168 // There may be stale spans in mcaches that need to be swept. 1169 // Those aren't tracked in any sweep lists, so we need to 1170 // count them against sweep completion until we ensure all 1171 // those spans have been forced out. 1172 // 1173 // If gcSweep fully swept the heap (for example if the sweep 1174 // is not concurrent due to a GODEBUG setting), then we expect 1175 // the sweepLocker to be invalid, since sweeping is done. 1176 // 1177 // N.B. Below we might duplicate some work from gcSweep; this is 1178 // fine as all that work is idempotent within a GC cycle, and 1179 // we're still holding worldsema so a new cycle can't start. 1180 sl := sweep.active.begin() 1181 if !stwSwept && !sl.valid { 1182 throw("failed to set sweep barrier") 1183 } else if stwSwept && sl.valid { 1184 throw("non-concurrent sweep failed to drain all sweep queues") 1185 } 1186 1187 systemstack(func() { 1188 // The memstats updated above must be updated with the world 1189 // stopped to ensure consistency of some values, such as 1190 // sched.idleTime and sched.totaltime. memstats also include 1191 // the pause time (work,pauseNS), forcing computation of the 1192 // total pause time before the pause actually ends. 1193 // 1194 // Here we reuse the same now for start the world so that the 1195 // time added to /sched/pauses/total/gc:seconds will be 1196 // consistent with the value in memstats. 1197 startTheWorldWithSema(now, stw) 1198 }) 1199 1200 // Flush the heap profile so we can start a new cycle next GC. 1201 // This is relatively expensive, so we don't do it with the 1202 // world stopped. 1203 mProf_Flush() 1204 1205 // Prepare workbufs for freeing by the sweeper. We do this 1206 // asynchronously because it can take non-trivial time. 1207 prepareFreeWorkbufs() 1208 1209 // Free stack spans. This must be done between GC cycles. 1210 systemstack(freeStackSpans) 1211 1212 // Ensure all mcaches are flushed. Each P will flush its own 1213 // mcache before allocating, but idle Ps may not. Since this 1214 // is necessary to sweep all spans, we need to ensure all 1215 // mcaches are flushed before we start the next GC cycle. 1216 // 1217 // While we're here, flush the page cache for idle Ps to avoid 1218 // having pages get stuck on them. These pages are hidden from 1219 // the scavenger, so in small idle heaps a significant amount 1220 // of additional memory might be held onto. 1221 // 1222 // Also, flush the pinner cache, to avoid leaking that memory 1223 // indefinitely. 1224 if debug.gctrace > 1 { 1225 clear(memstats.lastScanStats[:]) 1226 } 1227 forEachP(waitReasonFlushProcCaches, func(pp *p) { 1228 pp.mcache.prepareForSweep() 1229 if pp.status == _Pidle { 1230 systemstack(func() { 1231 lock(&mheap_.lock) 1232 pp.pcache.flush(&mheap_.pages) 1233 unlock(&mheap_.lock) 1234 }) 1235 } 1236 if debug.gctrace > 1 { 1237 pp.gcw.flushScanStats(&memstats.lastScanStats) 1238 } 1239 pp.pinnerCache = nil 1240 }) 1241 if sl.valid { 1242 // Now that we've swept stale spans in mcaches, they don't 1243 // count against unswept spans. 1244 // 1245 // Note: this sweepLocker may not be valid if sweeping had 1246 // already completed during the STW. See the corresponding 1247 // begin() call that produced sl. 1248 sweep.active.end(sl) 1249 } 1250 1251 // Print gctrace before dropping worldsema. As soon as we drop 1252 // worldsema another cycle could start and smash the stats 1253 // we're trying to print. 1254 if debug.gctrace > 0 { 1255 util := int(memstats.gc_cpu_fraction * 100) 1256 1257 var sbuf [24]byte 1258 printlock() 1259 print("gc ", memstats.numgc, 1260 " @", string(itoaDiv(sbuf[:], uint64(work.tSweepTerm-runtimeInitTime)/1e6, 3)), "s ", 1261 util, "%: ") 1262 prev := work.tSweepTerm 1263 for i, ns := range []int64{work.tMark, work.tMarkTerm, work.tEnd} { 1264 if i != 0 { 1265 print("+") 1266 } 1267 print(string(fmtNSAsMS(sbuf[:], uint64(ns-prev)))) 1268 prev = ns 1269 } 1270 print(" ms clock, ") 1271 for i, ns := range []int64{ 1272 int64(work.stwprocs) * (work.tMark - work.tSweepTerm), 1273 gcController.assistTime.Load(), 1274 gcController.dedicatedMarkTime.Load() + gcController.fractionalMarkTime.Load(), 1275 gcController.idleMarkTime.Load(), 1276 int64(work.stwprocs) * (work.tEnd - work.tMarkTerm), 1277 } { 1278 if i == 2 || i == 3 { 1279 // Separate mark time components with /. 1280 print("/") 1281 } else if i != 0 { 1282 print("+") 1283 } 1284 print(string(fmtNSAsMS(sbuf[:], uint64(ns)))) 1285 } 1286 print(" ms cpu, ", 1287 work.heap0>>20, "->", work.heap1>>20, "->", work.heap2>>20, " MB, ", 1288 gcController.lastHeapGoal>>20, " MB goal, ", 1289 gcController.lastStackScan.Load()>>20, " MB stacks, ", 1290 gcController.globalsScan.Load()>>20, " MB globals, ", 1291 work.maxprocs, " P") 1292 if work.userForced { 1293 print(" (forced)") 1294 } 1295 print("\n") 1296 1297 if debug.gctrace > 1 { 1298 dumpScanStats() 1299 } 1300 printunlock() 1301 } 1302 1303 // Print finalizer/cleanup queue length. Like gctrace, do this before the next GC starts. 1304 // The fact that the next GC might start is not that problematic here, but acts as a convenient 1305 // lock on printing this information (so it cannot overlap with itself from the next GC cycle). 1306 if debug.checkfinalizers > 0 { 1307 fq, fe := finReadQueueStats() 1308 fn := max(int64(fq)-int64(fe), 0) 1309 1310 cq, ce := gcCleanups.readQueueStats() 1311 cn := max(int64(cq)-int64(ce), 0) 1312 1313 println("checkfinalizers: queue:", fn, "finalizers +", cn, "cleanups") 1314 } 1315 1316 // Set any arena chunks that were deferred to fault. 1317 lock(&userArenaState.lock) 1318 faultList := userArenaState.fault 1319 userArenaState.fault = nil 1320 unlock(&userArenaState.lock) 1321 for _, lc := range faultList { 1322 lc.mspan.setUserArenaChunkToFault() 1323 } 1324 1325 // Enable huge pages on some metadata if we cross a heap threshold. 1326 if gcController.heapGoal() > minHeapForMetadataHugePages { 1327 systemstack(func() { 1328 mheap_.enableMetadataHugePages() 1329 }) 1330 } 1331 1332 semrelease(&worldsema) 1333 semrelease(&gcsema) 1334 // Careful: another GC cycle may start now. 1335 1336 releasem(mp) 1337 mp = nil 1338 1339 // now that gc is done, kick off finalizer thread if needed 1340 if !concurrentSweep { 1341 // give the queued finalizers, if any, a chance to run 1342 Gosched() 1343 } 1344 } 1345 1346 // gcBgMarkStartWorkers prepares background mark worker goroutines. These 1347 // goroutines will not run until the mark phase, but they must be started while 1348 // the work is not stopped and from a regular G stack. The caller must hold 1349 // worldsema. 1350 func gcBgMarkStartWorkers() { 1351 // Background marking is performed by per-P G's. Ensure that each P has 1352 // a background GC G. 1353 // 1354 // Worker Gs don't exit if gomaxprocs is reduced. If it is raised 1355 // again, we can reuse the old workers; no need to create new workers. 1356 if gcBgMarkWorkerCount >= gomaxprocs { 1357 return 1358 } 1359 1360 // Increment mp.locks when allocating. We are called within gcStart, 1361 // and thus must not trigger another gcStart via an allocation. gcStart 1362 // bails when allocating with locks held, so simulate that for these 1363 // allocations. 1364 // 1365 // TODO(prattmic): cleanup gcStart to use a more explicit "in gcStart" 1366 // check for bailing. 1367 mp := acquirem() 1368 ready := make(chan struct{}, 1) 1369 releasem(mp) 1370 1371 for gcBgMarkWorkerCount < gomaxprocs { 1372 mp := acquirem() // See above, we allocate a closure here. 1373 go gcBgMarkWorker(ready) 1374 releasem(mp) 1375 1376 // N.B. we intentionally wait on each goroutine individually 1377 // rather than starting all in a batch and then waiting once 1378 // afterwards. By running one goroutine at a time, we can take 1379 // advantage of runnext to bounce back and forth between 1380 // workers and this goroutine. In an overloaded application, 1381 // this can reduce GC start latency by prioritizing these 1382 // goroutines rather than waiting on the end of the run queue. 1383 <-ready 1384 // The worker is now guaranteed to be added to the pool before 1385 // its P's next findRunnableGCWorker. 1386 1387 gcBgMarkWorkerCount++ 1388 } 1389 } 1390 1391 // gcBgMarkPrepare sets up state for background marking. 1392 // Mutator assists must not yet be enabled. 1393 func gcBgMarkPrepare() { 1394 // Background marking will stop when the work queues are empty 1395 // and there are no more workers (note that, since this is 1396 // concurrent, this may be a transient state, but mark 1397 // termination will clean it up). Between background workers 1398 // and assists, we don't really know how many workers there 1399 // will be, so we pretend to have an arbitrarily large number 1400 // of workers, almost all of which are "waiting". While a 1401 // worker is working it decrements nwait. If nproc == nwait, 1402 // there are no workers. 1403 work.nproc = ^uint32(0) 1404 work.nwait = ^uint32(0) 1405 } 1406 1407 // gcBgMarkWorkerNode is an entry in the gcBgMarkWorkerPool. It points to a single 1408 // gcBgMarkWorker goroutine. 1409 type gcBgMarkWorkerNode struct { 1410 // Unused workers are managed in a lock-free stack. This field must be first. 1411 node lfnode 1412 1413 // The g of this worker. 1414 gp guintptr 1415 1416 // Release this m on park. This is used to communicate with the unlock 1417 // function, which cannot access the G's stack. It is unused outside of 1418 // gcBgMarkWorker(). 1419 m muintptr 1420 } 1421 type gcBgMarkWorkerNodePadded struct { 1422 gcBgMarkWorkerNode 1423 pad [tagAlign - unsafe.Sizeof(gcBgMarkWorkerNode{}) - gcBgMarkWorkerNodeRedZoneSize]byte 1424 } 1425 1426 const gcBgMarkWorkerNodeRedZoneSize = (16 << 2) * asanenabledBit // redZoneSize(512) 1427 1428 func gcBgMarkWorker(ready chan struct{}) { 1429 gp := getg() 1430 1431 // We pass node to a gopark unlock function, so it can't be on 1432 // the stack (see gopark). Prevent deadlock from recursively 1433 // starting GC by disabling preemption. 1434 gp.m.preemptoff = "GC worker init" 1435 node := &new(gcBgMarkWorkerNodePadded).gcBgMarkWorkerNode // TODO: technically not allowed in the heap. See comment in tagptr.go. 1436 gp.m.preemptoff = "" 1437 1438 node.gp.set(gp) 1439 1440 node.m.set(acquirem()) 1441 1442 ready <- struct{}{} 1443 // After this point, the background mark worker is generally scheduled 1444 // cooperatively by gcController.findRunnableGCWorker. While performing 1445 // work on the P, preemption is disabled because we are working on 1446 // P-local work buffers. When the preempt flag is set, this puts itself 1447 // into _Gwaiting to be woken up by gcController.findRunnableGCWorker 1448 // at the appropriate time. 1449 // 1450 // When preemption is enabled (e.g., while in gcMarkDone), this worker 1451 // may be preempted and schedule as a _Grunnable G from a runq. That is 1452 // fine; it will eventually gopark again for further scheduling via 1453 // findRunnableGCWorker. 1454 // 1455 // Since we disable preemption before notifying ready, we guarantee that 1456 // this G will be in the worker pool for the next findRunnableGCWorker. 1457 // This isn't strictly necessary, but it reduces latency between 1458 // _GCmark starting and the workers starting. 1459 1460 for { 1461 // Go to sleep until woken by 1462 // gcController.findRunnableGCWorker. 1463 gopark(func(g *g, nodep unsafe.Pointer) bool { 1464 node := (*gcBgMarkWorkerNode)(nodep) 1465 1466 if mp := node.m.ptr(); mp != nil { 1467 // The worker G is no longer running; release 1468 // the M. 1469 // 1470 // N.B. it is _safe_ to release the M as soon 1471 // as we are no longer performing P-local mark 1472 // work. 1473 // 1474 // However, since we cooperatively stop work 1475 // when gp.preempt is set, if we releasem in 1476 // the loop then the following call to gopark 1477 // would immediately preempt the G. This is 1478 // also safe, but inefficient: the G must 1479 // schedule again only to enter gopark and park 1480 // again. Thus, we defer the release until 1481 // after parking the G. 1482 releasem(mp) 1483 } 1484 1485 // Release this G to the pool. 1486 gcBgMarkWorkerPool.push(&node.node) 1487 // Note that at this point, the G may immediately be 1488 // rescheduled and may be running. 1489 return true 1490 }, unsafe.Pointer(node), waitReasonGCWorkerIdle, traceBlockSystemGoroutine, 0) 1491 1492 // Preemption must not occur here, or another G might see 1493 // p.gcMarkWorkerMode. 1494 1495 // Disable preemption so we can use the gcw. If the 1496 // scheduler wants to preempt us, we'll stop draining, 1497 // dispose the gcw, and then preempt. 1498 node.m.set(acquirem()) 1499 pp := gp.m.p.ptr() // P can't change with preemption disabled. 1500 1501 if gcBlackenEnabled == 0 { 1502 println("worker mode", pp.gcMarkWorkerMode) 1503 throw("gcBgMarkWorker: blackening not enabled") 1504 } 1505 1506 if pp.gcMarkWorkerMode == gcMarkWorkerNotWorker { 1507 throw("gcBgMarkWorker: mode not set") 1508 } 1509 1510 startTime := nanotime() 1511 pp.gcMarkWorkerStartTime = startTime 1512 var trackLimiterEvent bool 1513 if pp.gcMarkWorkerMode == gcMarkWorkerIdleMode { 1514 trackLimiterEvent = pp.limiterEvent.start(limiterEventIdleMarkWork, startTime) 1515 } 1516 1517 decnwait := atomic.Xadd(&work.nwait, -1) 1518 if decnwait == work.nproc { 1519 println("runtime: work.nwait=", decnwait, "work.nproc=", work.nproc) 1520 throw("work.nwait was > work.nproc") 1521 } 1522 1523 systemstack(func() { 1524 // Mark our goroutine preemptible so its stack 1525 // can be scanned or observed by the execution 1526 // tracer. This, for example, lets two mark workers 1527 // scan each other (otherwise, they would 1528 // deadlock). We must not modify anything on 1529 // the G stack. However, stack shrinking is 1530 // disabled for mark workers, so it is safe to 1531 // read from the G stack. 1532 // 1533 // N.B. The execution tracer is not aware of this status 1534 // transition and handles it specially based on the 1535 // wait reason. 1536 casGToWaitingForSuspendG(gp, _Grunning, waitReasonGCWorkerActive) 1537 switch pp.gcMarkWorkerMode { 1538 default: 1539 throw("gcBgMarkWorker: unexpected gcMarkWorkerMode") 1540 case gcMarkWorkerDedicatedMode: 1541 gcDrainMarkWorkerDedicated(&pp.gcw, true) 1542 if gp.preempt { 1543 // We were preempted. This is 1544 // a useful signal to kick 1545 // everything out of the run 1546 // queue so it can run 1547 // somewhere else. 1548 if drainQ := runqdrain(pp); !drainQ.empty() { 1549 lock(&sched.lock) 1550 globrunqputbatch(&drainQ) 1551 unlock(&sched.lock) 1552 } 1553 } 1554 // Go back to draining, this time 1555 // without preemption. 1556 gcDrainMarkWorkerDedicated(&pp.gcw, false) 1557 case gcMarkWorkerFractionalMode: 1558 gcDrainMarkWorkerFractional(&pp.gcw) 1559 case gcMarkWorkerIdleMode: 1560 gcDrainMarkWorkerIdle(&pp.gcw) 1561 } 1562 casgstatus(gp, _Gwaiting, _Grunning) 1563 }) 1564 1565 // Account for time and mark us as stopped. 1566 now := nanotime() 1567 duration := now - startTime 1568 gcController.markWorkerStop(pp.gcMarkWorkerMode, duration) 1569 if trackLimiterEvent { 1570 pp.limiterEvent.stop(limiterEventIdleMarkWork, now) 1571 } 1572 if pp.gcMarkWorkerMode == gcMarkWorkerFractionalMode { 1573 atomic.Xaddint64(&pp.gcFractionalMarkTime, duration) 1574 } 1575 1576 // Was this the last worker and did we run out 1577 // of work? 1578 incnwait := atomic.Xadd(&work.nwait, +1) 1579 if incnwait > work.nproc { 1580 println("runtime: p.gcMarkWorkerMode=", pp.gcMarkWorkerMode, 1581 "work.nwait=", incnwait, "work.nproc=", work.nproc) 1582 throw("work.nwait > work.nproc") 1583 } 1584 1585 // We'll releasem after this point and thus this P may run 1586 // something else. We must clear the worker mode to avoid 1587 // attributing the mode to a different (non-worker) G in 1588 // tracev2.GoStart. 1589 pp.gcMarkWorkerMode = gcMarkWorkerNotWorker 1590 1591 // If this worker reached a background mark completion 1592 // point, signal the main GC goroutine. 1593 if incnwait == work.nproc && !gcMarkWorkAvailable(nil) { 1594 // We don't need the P-local buffers here, allow 1595 // preemption because we may schedule like a regular 1596 // goroutine in gcMarkDone (block on locks, etc). 1597 releasem(node.m.ptr()) 1598 node.m.set(nil) 1599 1600 gcMarkDone() 1601 } 1602 } 1603 } 1604 1605 // gcMarkWorkAvailable reports whether executing a mark worker 1606 // on p is potentially useful. p may be nil, in which case it only 1607 // checks the global sources of work. 1608 func gcMarkWorkAvailable(p *p) bool { 1609 if p != nil && !p.gcw.empty() { 1610 return true 1611 } 1612 if !work.full.empty() || !work.spanq.empty() { 1613 return true // global work available 1614 } 1615 if work.markrootNext < work.markrootJobs { 1616 return true // root scan work available 1617 } 1618 return false 1619 } 1620 1621 // gcMark runs the mark (or, for concurrent GC, mark termination) 1622 // All gcWork caches must be empty. 1623 // STW is in effect at this point. 1624 func gcMark(startTime int64) { 1625 if gcphase != _GCmarktermination { 1626 throw("in gcMark expecting to see gcphase as _GCmarktermination") 1627 } 1628 work.tstart = startTime 1629 1630 // Check that there's no marking work remaining. 1631 if work.full != 0 || work.markrootNext < work.markrootJobs || !work.spanq.empty() { 1632 print("runtime: full=", hex(work.full), " next=", work.markrootNext, " jobs=", work.markrootJobs, " nDataRoots=", work.nDataRoots, " nBSSRoots=", work.nBSSRoots, " nSpanRoots=", work.nSpanRoots, " nStackRoots=", work.nStackRoots, " spanq.n=", work.spanq.size(), "\n") 1633 panic("non-empty mark queue after concurrent mark") 1634 } 1635 1636 if debug.gccheckmark > 0 { 1637 // This is expensive when there's a large number of 1638 // Gs, so only do it if checkmark is also enabled. 1639 gcMarkRootCheck() 1640 } 1641 1642 // Drop allg snapshot. allgs may have grown, in which case 1643 // this is the only reference to the old backing store and 1644 // there's no need to keep it around. 1645 work.stackRoots = nil 1646 1647 // Clear out buffers and double-check that all gcWork caches 1648 // are empty. This should be ensured by gcMarkDone before we 1649 // enter mark termination. 1650 // 1651 // TODO: We could clear out buffers just before mark if this 1652 // has a non-negligible impact on STW time. 1653 for _, p := range allp { 1654 // The write barrier may have buffered pointers since 1655 // the gcMarkDone barrier. However, since the barrier 1656 // ensured all reachable objects were marked, all of 1657 // these must be pointers to black objects. Hence we 1658 // can just discard the write barrier buffer. 1659 if debug.gccheckmark > 0 { 1660 // For debugging, flush the buffer and make 1661 // sure it really was all marked. 1662 wbBufFlush1(p) 1663 } else { 1664 p.wbBuf.reset() 1665 } 1666 1667 gcw := &p.gcw 1668 if !gcw.empty() { 1669 printlock() 1670 print("runtime: P ", p.id, " flushedWork ", gcw.flushedWork) 1671 if gcw.wbuf1 == nil { 1672 print(" wbuf1=<nil>") 1673 } else { 1674 print(" wbuf1.n=", gcw.wbuf1.nobj) 1675 } 1676 if gcw.wbuf2 == nil { 1677 print(" wbuf2=<nil>") 1678 } else { 1679 print(" wbuf2.n=", gcw.wbuf2.nobj) 1680 } 1681 print("\n") 1682 throw("P has cached GC work at end of mark termination") 1683 } 1684 // There may still be cached empty buffers, which we 1685 // need to flush since we're going to free them. Also, 1686 // there may be non-zero stats because we allocated 1687 // black after the gcMarkDone barrier. 1688 gcw.dispose() 1689 } 1690 1691 // Flush scanAlloc from each mcache since we're about to modify 1692 // heapScan directly. If we were to flush this later, then scanAlloc 1693 // might have incorrect information. 1694 // 1695 // Note that it's not important to retain this information; we know 1696 // exactly what heapScan is at this point via scanWork. 1697 for _, p := range allp { 1698 c := p.mcache 1699 if c == nil { 1700 continue 1701 } 1702 c.scanAlloc = 0 1703 } 1704 1705 // Reset controller state. 1706 gcController.resetLive(work.bytesMarked) 1707 } 1708 1709 // gcSweep must be called on the system stack because it acquires the heap 1710 // lock. See mheap for details. 1711 // 1712 // Returns true if the heap was fully swept by this function. 1713 // 1714 // The world must be stopped. 1715 // 1716 //go:systemstack 1717 func gcSweep(mode gcMode) bool { 1718 assertWorldStopped() 1719 1720 if gcphase != _GCoff { 1721 throw("gcSweep being done but phase is not GCoff") 1722 } 1723 1724 lock(&mheap_.lock) 1725 mheap_.sweepgen += 2 1726 sweep.active.reset() 1727 mheap_.pagesSwept.Store(0) 1728 mheap_.sweepArenas = mheap_.heapArenas 1729 mheap_.reclaimIndex.Store(0) 1730 mheap_.reclaimCredit.Store(0) 1731 unlock(&mheap_.lock) 1732 1733 sweep.centralIndex.clear() 1734 1735 if !concurrentSweep || mode == gcForceBlockMode { 1736 // Special case synchronous sweep. 1737 // Record that no proportional sweeping has to happen. 1738 lock(&mheap_.lock) 1739 mheap_.sweepPagesPerByte = 0 1740 unlock(&mheap_.lock) 1741 // Flush all mcaches. 1742 for _, pp := range allp { 1743 pp.mcache.prepareForSweep() 1744 } 1745 // Sweep all spans eagerly. 1746 for sweepone() != ^uintptr(0) { 1747 } 1748 // Free workbufs eagerly. 1749 prepareFreeWorkbufs() 1750 for freeSomeWbufs(false) { 1751 } 1752 // All "free" events for this mark/sweep cycle have 1753 // now happened, so we can make this profile cycle 1754 // available immediately. 1755 mProf_NextCycle() 1756 mProf_Flush() 1757 return true 1758 } 1759 1760 // Background sweep. 1761 lock(&sweep.lock) 1762 if sweep.parked { 1763 sweep.parked = false 1764 ready(sweep.g, 0, true) 1765 } 1766 unlock(&sweep.lock) 1767 return false 1768 } 1769 1770 // gcResetMarkState resets global state prior to marking (concurrent 1771 // or STW) and resets the stack scan state of all Gs. 1772 // 1773 // This is safe to do without the world stopped because any Gs created 1774 // during or after this will start out in the reset state. 1775 // 1776 // gcResetMarkState must be called on the system stack because it acquires 1777 // the heap lock. See mheap for details. 1778 // 1779 //go:systemstack 1780 func gcResetMarkState() { 1781 // This may be called during a concurrent phase, so lock to make sure 1782 // allgs doesn't change. 1783 forEachG(func(gp *g) { 1784 gp.gcscandone = false // set to true in gcphasework 1785 gp.gcAssistBytes = 0 1786 }) 1787 1788 // Clear page marks. This is just 1MB per 64GB of heap, so the 1789 // time here is pretty trivial. 1790 lock(&mheap_.lock) 1791 arenas := mheap_.heapArenas 1792 unlock(&mheap_.lock) 1793 for _, ai := range arenas { 1794 ha := mheap_.arenas[ai.l1()][ai.l2()] 1795 clear(ha.pageMarks[:]) 1796 } 1797 1798 work.bytesMarked = 0 1799 work.initialHeapLive = gcController.heapLive.Load() 1800 } 1801 1802 // Hooks for other packages 1803 1804 var poolcleanup func() 1805 var boringCaches []unsafe.Pointer // for crypto/internal/boring 1806 1807 // sync_runtime_registerPoolCleanup should be an internal detail, 1808 // but widely used packages access it using linkname. 1809 // Notable members of the hall of shame include: 1810 // - github.com/bytedance/gopkg 1811 // - github.com/songzhibin97/gkit 1812 // 1813 // Do not remove or change the type signature. 1814 // See go.dev/issue/67401. 1815 // 1816 //go:linkname sync_runtime_registerPoolCleanup sync.runtime_registerPoolCleanup 1817 func sync_runtime_registerPoolCleanup(f func()) { 1818 poolcleanup = f 1819 } 1820 1821 //go:linkname boring_registerCache crypto/internal/boring/bcache.registerCache 1822 func boring_registerCache(p unsafe.Pointer) { 1823 boringCaches = append(boringCaches, p) 1824 } 1825 1826 func clearpools() { 1827 // clear sync.Pools 1828 if poolcleanup != nil { 1829 poolcleanup() 1830 } 1831 1832 // clear boringcrypto caches 1833 for _, p := range boringCaches { 1834 atomicstorep(p, nil) 1835 } 1836 1837 // Clear central sudog cache. 1838 // Leave per-P caches alone, they have strictly bounded size. 1839 // Disconnect cached list before dropping it on the floor, 1840 // so that a dangling ref to one entry does not pin all of them. 1841 lock(&sched.sudoglock) 1842 var sg, sgnext *sudog 1843 for sg = sched.sudogcache; sg != nil; sg = sgnext { 1844 sgnext = sg.next 1845 sg.next = nil 1846 } 1847 sched.sudogcache = nil 1848 unlock(&sched.sudoglock) 1849 1850 // Clear central defer pool. 1851 // Leave per-P pools alone, they have strictly bounded size. 1852 lock(&sched.deferlock) 1853 // disconnect cached list before dropping it on the floor, 1854 // so that a dangling ref to one entry does not pin all of them. 1855 var d, dlink *_defer 1856 for d = sched.deferpool; d != nil; d = dlink { 1857 dlink = d.link 1858 d.link = nil 1859 } 1860 sched.deferpool = nil 1861 unlock(&sched.deferlock) 1862 } 1863 1864 // Timing 1865 1866 // itoaDiv formats val/(10**dec) into buf. 1867 func itoaDiv(buf []byte, val uint64, dec int) []byte { 1868 i := len(buf) - 1 1869 idec := i - dec 1870 for val >= 10 || i >= idec { 1871 buf[i] = byte(val%10 + '0') 1872 i-- 1873 if i == idec { 1874 buf[i] = '.' 1875 i-- 1876 } 1877 val /= 10 1878 } 1879 buf[i] = byte(val + '0') 1880 return buf[i:] 1881 } 1882 1883 // fmtNSAsMS nicely formats ns nanoseconds as milliseconds. 1884 func fmtNSAsMS(buf []byte, ns uint64) []byte { 1885 if ns >= 10e6 { 1886 // Format as whole milliseconds. 1887 return itoaDiv(buf, ns/1e6, 0) 1888 } 1889 // Format two digits of precision, with at most three decimal places. 1890 x := ns / 1e3 1891 if x == 0 { 1892 buf[0] = '0' 1893 return buf[:1] 1894 } 1895 dec := 3 1896 for x >= 100 { 1897 x /= 10 1898 dec-- 1899 } 1900 return itoaDiv(buf, x, dec) 1901 } 1902 1903 // Helpers for testing GC. 1904 1905 // gcTestMoveStackOnNextCall causes the stack to be moved on a call 1906 // immediately following the call to this. It may not work correctly 1907 // if any other work appears after this call (such as returning). 1908 // Typically the following call should be marked go:noinline so it 1909 // performs a stack check. 1910 // 1911 // In rare cases this may not cause the stack to move, specifically if 1912 // there's a preemption between this call and the next. 1913 func gcTestMoveStackOnNextCall() { 1914 gp := getg() 1915 gp.stackguard0 = stackForceMove 1916 } 1917 1918 // gcTestIsReachable performs a GC and returns a bit set where bit i 1919 // is set if ptrs[i] is reachable. 1920 func gcTestIsReachable(ptrs ...unsafe.Pointer) (mask uint64) { 1921 // This takes the pointers as unsafe.Pointers in order to keep 1922 // them live long enough for us to attach specials. After 1923 // that, we drop our references to them. 1924 1925 if len(ptrs) > 64 { 1926 panic("too many pointers for uint64 mask") 1927 } 1928 1929 // Block GC while we attach specials and drop our references 1930 // to ptrs. Otherwise, if a GC is in progress, it could mark 1931 // them reachable via this function before we have a chance to 1932 // drop them. 1933 semacquire(&gcsema) 1934 1935 // Create reachability specials for ptrs. 1936 specials := make([]*specialReachable, len(ptrs)) 1937 for i, p := range ptrs { 1938 lock(&mheap_.speciallock) 1939 s := (*specialReachable)(mheap_.specialReachableAlloc.alloc()) 1940 unlock(&mheap_.speciallock) 1941 s.special.kind = _KindSpecialReachable 1942 if !addspecial(p, &s.special, false) { 1943 throw("already have a reachable special (duplicate pointer?)") 1944 } 1945 specials[i] = s 1946 // Make sure we don't retain ptrs. 1947 ptrs[i] = nil 1948 } 1949 1950 semrelease(&gcsema) 1951 1952 // Force a full GC and sweep. 1953 GC() 1954 1955 // Process specials. 1956 for i, s := range specials { 1957 if !s.done { 1958 printlock() 1959 println("runtime: object", i, "was not swept") 1960 throw("IsReachable failed") 1961 } 1962 if s.reachable { 1963 mask |= 1 << i 1964 } 1965 lock(&mheap_.speciallock) 1966 mheap_.specialReachableAlloc.free(unsafe.Pointer(s)) 1967 unlock(&mheap_.speciallock) 1968 } 1969 1970 return mask 1971 } 1972 1973 // gcTestPointerClass returns the category of what p points to, one of: 1974 // "heap", "stack", "data", "bss", "other". This is useful for checking 1975 // that a test is doing what it's intended to do. 1976 // 1977 // This is nosplit simply to avoid extra pointer shuffling that may 1978 // complicate a test. 1979 // 1980 //go:nosplit 1981 func gcTestPointerClass(p unsafe.Pointer) string { 1982 p2 := uintptr(noescape(p)) 1983 gp := getg() 1984 if gp.stack.lo <= p2 && p2 < gp.stack.hi { 1985 return "stack" 1986 } 1987 if base, _, _ := findObject(p2, 0, 0); base != 0 { 1988 return "heap" 1989 } 1990 for _, datap := range activeModules() { 1991 if datap.data <= p2 && p2 < datap.edata || datap.noptrdata <= p2 && p2 < datap.enoptrdata { 1992 return "data" 1993 } 1994 if datap.bss <= p2 && p2 < datap.ebss || datap.noptrbss <= p2 && p2 <= datap.enoptrbss { 1995 return "bss" 1996 } 1997 } 1998 KeepAlive(p) 1999 return "other" 2000 } 2001