Commit Graph

52356 Commits

Author SHA1 Message Date
Russ Cox
cdcb4b6ef3 [dev.boringcrypto] cmd/compile: remove the awful boringcrypto kludge
CL 60271 introduced this “AwfulBoringCryptoKludge.”
iant approved that CL saying “As long as it stays out of master...”

Now that the rsa and ecdsa code uses boring.Cache, the
“boring unsafe.Pointer” fields are gone from the key structs, and this
code is no longer needed. So delete it.

With the kludge deleted, we are one step closer to being able to merge
dev.boringcrypto into master.

For #51940.

Change-Id: Ie549db14b0b699c306dded2a2163f18f31d45530
Reviewed-on: https://go-review.googlesource.com/c/go/+/395884
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
2022-04-29 14:24:53 +00:00
Russ Cox
e845f572ec [dev.boringcrypto] crypto/ecdsa, crypto/rsa: use boring.Cache
In the original BoringCrypto port, ecdsa and rsa's public and private
keys added a 'boring unsafe.Pointer' field to cache the BoringCrypto
form of the key. This led to problems with code that “knew” the layout
of those structs and in particular that they had no unexported fields.

In response, as an awful kludge, I changed the compiler to pretend
that field did not exist when laying out reflect data. Because we want
to merge BoringCrypto in the main tree, we need a different solution.
Using boring.Cache is that solution.

For #51940.

Change-Id: Ideb2b40b599a1dc223082eda35a5ea9abcc01e30
Reviewed-on: https://go-review.googlesource.com/c/go/+/395883
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
2022-04-29 14:23:32 +00:00
Russ Cox
a840bf871e [dev.boringcrypto] crypto/internal/boring: add GC-aware cache
In the original BoringCrypto port, ecdsa and rsa's public and private
keys added a 'boring unsafe.Pointer' field to cache the BoringCrypto
form of the key. This led to problems with code that “knew” the layout
of those structs and in particular that they had no unexported fields.

In response, as an awful kludge, I changed the compiler to pretend
that field did not exist when laying out reflect data. Because we want
to merge BoringCrypto in the main tree, we need a different solution.

The different solution is this CL's boring.Cache, which is a
concurrent, GC-aware map from unsafe.Pointer to unsafe.Pointer (if
generics were farther along we could use them nicely here, but I am
afraid of breaking tools that aren't ready to see generics in the
standard library yet).

More complex approaches are possible, but a simple, fixed-size hash
table is easy to make concurrent and should be fine.

For #51940.

Change-Id: I44062a8defbd87b705a787cffc64c6a9d0132785
Reviewed-on: https://go-review.googlesource.com/c/go/+/395882
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
2022-04-29 14:23:31 +00:00
Russ Cox
0184fe5ece [dev.boringcrypto] crypto/x509: remove VerifyOptions.IsBoring
This API was added only for BoringCrypto, never shipped in standard
Go. This API is also not compatible with the expected future evolution
of crypto/x509, as we move closer to host verifiers on macOS and Windows.

If we want to merge BoringCrypto into the main tree, it is best not to
have differing API. So instead of a hook set by crypto/tls, move the
actual check directly into crypto/x509, eliminating the need for
exposed API.

For #51940.

Change-Id: Ia2ae98c745de818d39501777014ea8166cab0b03
Reviewed-on: https://go-review.googlesource.com/c/go/+/395878
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
2022-04-29 14:23:29 +00:00
Russ Cox
9e9c7a0aec [dev.boringcrypto] crypto/..., go/build: align deps test with standard rules
One annoying difference between dev.boringcrypto and master is that
there is not a clear separation between low-level (math/big-free)
crypto and high-level crypto, because crypto/internal/boring imports
both encoding/asn1 and math/big.

This CL removes both those problematic imports and aligns the
dependency rules in the go/build test with the ones in the main
branch.

To remove encoding/asn1, the crypto/internal/boring APIs change to
accepting and returning encoded ASN.1, leaving crypto/ecdsa to do the
marshaling and unmarshaling, which it already contains code to do.

To remove math/big, the crypto/internal/boring package defines
type BigInt []uint, which is the same representation as a big.Int's
internal storage. The new package crypto/internal/boring/bbig provides
conversions between BigInt and *big.Int. The boring package can then
be in the low-level crypto set, and any package needing to use bignum
APIs (necessarily in the high-level crypto set) can import bbig to
convert.

To simplify everything we hide from the test the fact that
crypto/internal/boring imports cgo. Better to pretend it doesn't and
keep the prohibitions that other packages like crypto/aes must not use
cgo (outside of BoringCrypto).

	$ git diff origin/master src/go/build/deps_test.go
	diff --git a/src/go/build/deps_test.go b/src/go/build/deps_test.go
	index 6ce872e297..a63979cc93 100644
	--- a/src/go/build/deps_test.go
	+++ b/src/go/build/deps_test.go
	@@ -402,9 +402,13 @@ var depsRules = `
	 	NET, log
	 	< net/mail;

	+	NONE < crypto/internal/boring/sig;
	+	sync/atomic < crypto/internal/boring/fipstls;
	+	crypto/internal/boring/sig, crypto/internal/boring/fipstls < crypto/tls/fipsonly;
	+
	 	# CRYPTO is core crypto algorithms - no cgo, fmt, net.
	 	# Unfortunately, stuck with reflect via encoding/binary.
	-	encoding/binary, golang.org/x/sys/cpu, hash
	+	crypto/internal/boring/sig, encoding/binary, golang.org/x/sys/cpu, hash
	 	< crypto
	 	< crypto/subtle
	 	< crypto/internal/subtle
	@@ -413,6 +417,8 @@ var depsRules = `
	 	< crypto/ed25519/internal/edwards25519/field, golang.org/x/crypto/curve25519/internal/field
	 	< crypto/ed25519/internal/edwards25519
	 	< crypto/cipher
	+	< crypto/internal/boring
	+	< crypto/boring
	 	< crypto/aes, crypto/des, crypto/hmac, crypto/md5, crypto/rc4,
	 	  crypto/sha1, crypto/sha256, crypto/sha512
	 	< CRYPTO;
	@@ -421,6 +427,7 @@ var depsRules = `

	 	# CRYPTO-MATH is core bignum-based crypto - no cgo, net; fmt now ok.
	 	CRYPTO, FMT, math/big, embed
	+	< crypto/internal/boring/bbig
	 	< crypto/rand
	 	< crypto/internal/randutil
	 	< crypto/ed25519
	@@ -443,7 +450,8 @@ var depsRules = `
	 	< golang.org/x/crypto/hkdf
	 	< crypto/x509/internal/macos
	 	< crypto/x509/pkix
	-	< crypto/x509
	+	< crypto/x509;
	+	crypto/internal/boring/fipstls, crypto/x509
	 	< crypto/tls;

	 	# crypto-aware packages
	@@ -653,6 +661,9 @@ func findImports(pkg string) ([]string, error) {
	 	}
	 	var imports []string
	 	var haveImport = map[string]bool{}
	+	if pkg == "crypto/internal/boring" {
	+		haveImport["C"] = true // kludge: prevent C from appearing in crypto/internal/boring imports
	+	}
	 	fset := token.NewFileSet()
	 	for _, file := range files {
	 		name := file.Name()

For #51940.

Change-Id: I26fc752484310d77d22adb06495120a361568d04
Reviewed-on: https://go-review.googlesource.com/c/go/+/395877
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
2022-04-29 14:23:28 +00:00
Russ Cox
0ec08283c8 [dev.boringcrypto] crypto/internal/boring: make SHA calls allocation-free
The standard Go implementations are allocation-free.
Making the BoringCrypto ones the same helps avoid
surprises, including in some of our own tests.

For #51940.

Change-Id: Ic9c5dc46f5e29ca85f571244be2b380ec2cf89c9
Reviewed-on: https://go-review.googlesource.com/c/go/+/395876
Reviewed-by: Roland Shoemaker <roland@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-29 14:23:27 +00:00
Russ Cox
3cb10d14b7 [dev.boringcrypto] crypto/internal/boring: avoid allocation in big.Int conversion
The conversion via byte slices is inefficient; we can convert via word slices
and avoid the copy entirely.

For #51940.

Change-Id: I06f747e0acffffae427d9706d43bdacf146c027d
Reviewed-on: https://go-review.googlesource.com/c/go/+/395875
Reviewed-by: Roland Shoemaker <roland@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-29 14:23:26 +00:00
Russ Cox
509776be5d [dev.boringcrypto] cmd/dist: default to use of boringcrypto
The dev.boringcrypto branch has historically forced use of boringcrypto
with no additional configuration flags. The previous CL undid that.
This CL redoes it, so that direct uses of dev.boringcrypto don't lapse
unexpectedly into not having boringcrypto enabled.

When dev.boringcrypto is merged into master, we will undo this change
as part of the merge, so that the only final difference between master
and dev.boringcrypto will be this CL.

For #51940.

Change-Id: I816593a0b30b4e71093a7da9451bae7807d7167e
Reviewed-on: https://go-review.googlesource.com/c/go/+/402597
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-29 14:23:25 +00:00
Russ Cox
f4c0f42f99 [dev.boringcrypto] all: add boringcrypto build tags
A plain make.bash in this tree will produce a working,
standard Go toolchain, not a BoringCrypto-enabled one.

The BoringCrypto-enabled one will be created with:

	GOEXPERIMENT=boringcrypto ./make.bash

For #51940.

Change-Id: Ia9102ed993242eb1cb7f9b93eca97e81986a27b3
Reviewed-on: https://go-review.googlesource.com/c/go/+/395881
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2022-04-29 14:23:22 +00:00
Russ Cox
1f0547c4ec [dev.boringcrypto] cmd/go: pass dependency syso to cgo too
Proposal #42477 asked for a way to apply conditional build tags
to syso files (which have no source code to hold //go:build lines).

We ended up suggesting that the standard answer should be to
put the syso in its own package and then import that package from
a source file that is itself conditionally compiled.

A followup comment on that issue pointed out a problem that I did
not understand until I tried to use this approach myself: the cgo
build fails by default, because the link step only uses syso files from
the current package. You have to override this explicitly by arranging
to pass a “ignore unresolved symbols” flag to the host linker.
Many users will not know how to do this.
(I don't know how to do this off the top of my head.)

If we want users to use this approach, we should make it work better.
This CL does that, by including the syso files from dependencies of
the current package in the link step.

For #51940.

Change-Id: I53a0371b2df17e39a000a645b7686daa6a98722d
Reviewed-on: https://go-review.googlesource.com/c/go/+/402596
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-29 14:23:21 +00:00
Russ Cox
e5407501cb [dev.boringcrypto] cmd: use notsha256 instead of md5, sha1, sha256
When we add GOEXPERIMENT=boringcrypto, the bootstrap process
will not converge if the compiler itself depends on the boringcrypto
cgo-based implementations of sha1 and sha256.

Using notsha256 avoids boringcrypto and makes bootstrap converge.
Removing md5 is not strictly necessary but it seemed worthwhile to
be consistent.

For #51940.

Change-Id: Iba649507e0964d1a49a1d16e463dd23c4e348f14
Reviewed-on: https://go-review.googlesource.com/c/go/+/402595
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-29 14:23:19 +00:00
Russ Cox
fe006d6410 [dev.boringcrypto] cmd/internal/notsha256: add new package
Package notsha256 implements the NOTSHA256 hash,
defined as bitwise NOT of SHA-256.

It will be used from the Go compiler toolchain where an
arbitrary hash is needed and the code currently reaches
for MD5, SHA1, or SHA256. The problem with all of those
is that when we add GOEXPERIMENT=boringcrypto, the
bootstrap process will not converge if the compiler itself
depends on the boringcrypto cgo code.
Using notsha256 avoids boringcrypto.

It is possible that I don't fully understand the convergence
problem and that there is a way to make the compiler converge
when using cgo, but keeping cgo out of the compiler seems safest.
It also makes clear that (except for the hack in codesign)
the code using this package doesn't care which hash is used.

For #51940.

Change-Id: Ie7c661183eacf8413a9d2074c96cbb9361e125ef
Reviewed-on: https://go-review.googlesource.com/c/go/+/402594
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-29 14:23:17 +00:00
Chressie Himpel
ec7f5165dd [dev.boringcrypto] all: merge master into dev.boringcrypto
Change-Id: Ic5f71c04f08c03319c043f35be501875adb0a3b0
2022-04-27 20:09:28 +02:00
Filippo Valsorda
f0ee7fda63 crypto/tls: remove tls10default GODEBUG flag
Updates #45428

Change-Id: Ic2ff459e6a3f1e8ded2a770c11d34067c0b39a8a
Reviewed-on: https://go-review.googlesource.com/c/go/+/400974
Reviewed-by: Filippo Valsorda <valsorda@google.com>
Auto-Submit: Filippo Valsorda <valsorda@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Filippo Valsorda <valsorda@google.com>
Reviewed-by: Roland Shoemaker <roland@golang.org>
2022-04-27 17:20:34 +00:00
Filippo Valsorda
0b5218cf4e crypto/elliptic: split up P-256 field and group ops
This makes Gerrit recognize the rename of the field implementation and
facilitates the review. No code changes.

For #52182

Change-Id: I827004e175db1ae2fcdf17d0f586ff21503d27e3
Reviewed-on: https://go-review.googlesource.com/c/go/+/390754
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Russ Cox <rsc@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
Run-TryBot: Filippo Valsorda <filippo@golang.org>
Auto-Submit: Filippo Valsorda <filippo@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-27 15:22:15 +00:00
Filippo Valsorda
f0c0e0f255 crypto/elliptic: inline marshaling into nistec pointFromAffine
Marshal behavior for invalid points is undefined, so don't use it to
check if points are valid.

For #52182

Change-Id: If167893bc4b029f71bb2528564f2bd96bee7221c
Reviewed-on: https://go-review.googlesource.com/c/go/+/382994
Run-TryBot: Filippo Valsorda <filippo@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Roland Shoemaker <roland@golang.org>
2022-04-27 14:23:28 +00:00
Filippo Valsorda
83ff0b0c61 crypto/elliptic: use generics for nistec-based curves
There was no way to use an interface because the methods on the Point
types return concrete Point values, as they should.

A couple somewhat minor annoyances:

    - Allocations went up due to #48849. This is fine here, where
      math/big causes allocations anyway, but would probably not be fine
      in nistec itself.

    - Carrying the newPoint/newGenerator functions around as a field is
      a little weird, even if type-safe. It also means we have to make
      what were functions methods so they can access newPoint to return
      the zero value. This is #35966.

For #52182

Change-Id: I050f3a27f15d3f189818da80da9de0cba0548931
Reviewed-on: https://go-review.googlesource.com/c/go/+/360015
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Filippo Valsorda <filippo@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
2022-04-27 14:22:53 +00:00
Filippo Valsorda
6796a7924c crypto/elliptic: refactor package structure
Not quite golang.org/wiki/TargetSpecific compliant, but almost.

The only substantial code change is in randFieldElement: it used to use
Params().BitSize instead of Params().N.BitLen(), which is semantically
incorrect, even if the two values are the same for all named curves.

For #52182

Change-Id: Ibc47450552afe23ea74fcf55d1d799d5d7e5487c
Reviewed-on: https://go-review.googlesource.com/c/go/+/315273
Run-TryBot: Filippo Valsorda <filippo@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Roland Shoemaker <roland@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
2022-04-27 14:21:49 +00:00
cuiweixie
24b570354c time: document hhmmss formats
Fixes #52516

Change-Id: I173fdb09c245563e09be4e1aacfd374c3a764d74
GitHub-Last-Rev: 14a81e5061
GitHub-Pull-Request: golang/go#52538
Reviewed-on: https://go-review.googlesource.com/c/go/+/402058
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-27 00:11:22 +00:00
Keith Randall
68b655f2b9 runtime: disable windowed Smhasher test on 32-bit systems
This test tends to be flaky on 32-bit systems.
There's not enough bits in the hash output, so we
expect a nontrivial number of collisions, and it is
often quite a bit higher than expected.

Fixes #43130

Change-Id: If35413b7c45eed778a08b834dacf98009ceca840
Reviewed-on: https://go-review.googlesource.com/c/go/+/402456
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Keith Randall <khr@google.com>
2022-04-27 00:09:45 +00:00
Michael Anthony Knyszek
d29f5247b8 runtime: refactor the scavenger and make it testable
This change refactors the scavenger into a type whose methods represent
the actual function and scheduling of the scavenger. It also stubs out
access to global state in order to make it testable.

This change thus also adds a test for the scavenger. In writing this
test, I discovered the lack of a behavior I expected: if the
pageAlloc.scavenge returns < the bytes requested scavenged, that means
the heap is exhausted. This has been true this whole time, but was not
documented or explicitly relied upon. This change rectifies that. In
theory this means the scavenger could spin in run() indefinitely (as
happened in the test) if shouldStop never told it to stop. In practice,
shouldStop fires long before the heap is exhausted, but for future
changes it may be important. At the very least it's good to be
intentional about these things.

While we're here, I also moved the call to stopTimer out of wake and
into sleep. There's no reason to add more operations to a context that's
already precarious (running without a P on sysmon).

Change-Id: Ib31b86379fd9df84f25ae282734437afc540da5c
Reviewed-on: https://go-review.googlesource.com/c/go/+/384734
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-26 22:15:21 +00:00
Michael Anthony Knyszek
d8cf2243e0 runtime: disable idle mark workers with at least one dedicated worker
This change completes the proposal laid out in #44163. With #44313
resolved, we now ensure that stopped Ms are able to wake up and become
dedicated GC workers. As a result, idle GC workers are in theory no
longer required to be a proxy for scheduling dedicated mark workers.

And, with at least one dedicated mark worker running (which is
non-preemptible) we ensure the GC makes progress in all circumstances
when at least one is running. Currently we ensure at least one idle mark
worker is available at all times because it's possible before #44313
that a dedicated worker doesn't ever get scheduled, leading to a
deadlock if user goroutines block on a GC completing. But now that extra
idle mark worker should be unnecessary to ensure GC progress when at
least one dedicated mark worker is going to be scheduled.

Fixes #44163.

Change-Id: I62889ef2db4e69d44da883e8e6eebcfe5398c86d
Reviewed-on: https://go-review.googlesource.com/c/go/+/395634
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-26 22:09:56 +00:00
Michael Anthony Knyszek
13b18ec947 runtime: move scheduling decisions by schedule into findrunnable
This change moves several scheduling decisions made by schedule into
findrunnable. The main motivation behind this change is the fact that
stopped Ms can't become dedicated or fractional GC workers. The main
reason for this is that when a stopped M wakes up, it stays in
findrunnable until it finds work, which means it will never consider GC
work. On that note, it'll also never consider becoming the trace reader,
either.

Another way of looking at it is that this change tries to make
findrunnable aware of more sources of work than it was before. With this
change, any M in findrunnable should be capable of becoming a GC worker,
resolving #44313. While we're here, let's also make more sources of
work, such as the trace reader, visible to handoffp, which should really
be checking all sources of work. With that, we also now correctly handle
the case where StopTrace is called from the last live M that is also
locked (#39004). stoplockedm calls handoffp to start a new M and handle
the work it cannot, and once we include the trace reader in that, we
ensure that the trace reader gets scheduled.

This change attempts to preserve the exact same ordering of work
checking to reduce its impact.

One consequence of this change is that upon entering schedule, some
sources of work won't be checked twice (i.e. the local and global
runqs, and timers) as they do now, which in some sense gives them a
lower priority than they had before.

Fixes #39004.
Fixes #44313.

Change-Id: I5d8b7f63839db8d9a3e47cdda604baac1fe615ce
Reviewed-on: https://go-review.googlesource.com/c/go/+/393880
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-26 22:09:34 +00:00
Michael Anthony Knyszek
e1b5f347e7 runtime: reduce max idle mark workers during periodic GC cycles
This change reduces the maximum number of idle mark workers during
periodic (currently every 2 minutes) GC cycles to 1.

Idle mark workers soak up all available and unused Ps, up to GOMAXPROCS.
While this provides some throughput and latency benefit in general, it
can cause what appear to be massive CPU utilization spikes in otherwise
idle applications. This is mostly an issue for *very* idle applications,
ones idle enough to trigger periodic GC cycles. This spike also tends to
interact poorly with auto-scaling systems, as the system might assume
the load average is very low and suddenly see a massive burst in
activity.

The result of this change is not to bring down this 100% (of GOMAXPROCS)
CPU utilization spike to 0%, but rather

  min(25% + 1/GOMAXPROCS*100%, 100%)

Idle mark workers also do incur a small latency penalty as they must be
descheduled for other work that might pop up. Luckily the runtime is
pretty good about getting idle mark workers off of Ps, so in general
the latency benefit from shorter GC cycles outweighs this cost. But, the
cost is still non-zero and may be more significant in idle applications
that aren't invoking assists and write barriers quite as often.

We can't completely eliminate idle mark workers because they're
currently necessary for GC progress in some circumstances. Namely,
they're critical for progress when all we have is fractional workers. If
a fractional worker meets its quota, and all user goroutines are blocked
directly or indirectly on a GC cycle (via runtime.GOMAXPROCS, or
runtime.GC), the program may deadlock without GC workers, since the
fractional worker will go to sleep with nothing to wake it.

Fixes #37116.
For #44163.

Change-Id: Ib74793bb6b88d1765c52d445831310b0d11ef423
Reviewed-on: https://go-review.googlesource.com/c/go/+/393394
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-26 22:08:42 +00:00
Michael Anthony Knyszek
226346bb76 runtime: yield instead of sleeping in runqgrab on OpenBSD
OpenBSD has a coarse sleep granularity that rounds up to 10 ms
increments. This can cause significant STW delays, among other issues.
As far as I can tell, there's only 1 tightly timed sleep without an
explicit wakeup for which this actually matters.

Fixes #52475.

Change-Id: Ic69fc11096ddbbafd79b2dcdf3f912fde242db24
Reviewed-on: https://go-review.googlesource.com/c/go/+/401638
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-26 22:08:12 +00:00
Michael Anthony Knyszek
79db59ded9 runtime: make alloc count metrics truly monotonic
Right now we export alloc count metrics via the runtime/metrics package
and mark them as monotonic, but that's not actually true. As an
optimization, the runtime assumes a span is always fully allocated
before being uncached, and updates the accounting as such. In the rare
case that it's wrong, the span has enough information to back out what
did not get allocated.

This change uses 16 bits of padding in the mspan to house another field
that represents the amount of mspan slots filled just as the mspan is
cached. This is information is enough to get an exact count, allowing us
to make the metrics truly monotonic.

Change-Id: Iaff3ca43f8745dc1bbb0232372423e014b89b920
Reviewed-on: https://go-review.googlesource.com/c/go/+/377516
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-26 22:08:00 +00:00
Hana
a0f77e56b7 SECURITY.md: replace golang.org with go.dev
Change-Id: Ic0e882fc6666c9adcd5f2dffc96e201f3146fa0f
Reviewed-on: https://go-review.googlesource.com/c/go/+/402180
Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
2022-04-26 19:59:47 +00:00
Michael Pratt
ba7f3d98d4 runtime: use ABIInternal for most calls to sigtrampgo
sigtramp on openbsd-arm64 is teetering on the edge of the nosplit stack
limit. Add more headroom by calling sigtrampgo using ABIInternal, which
eliminates a 48-byte ABI wrapper frame.

openbsd-amd64 has slightly more space, but is also close to the limit,
so convert it as well.

Other operating systems don't have it as bad, but many have nearly
identical implementations of sigtramp, so I have converted them as well.

I've omitted darwin-arm64 and solaris, as those are quite different and
would benefit from not needing ifdef for both cases.

For #51485.

Change-Id: I70512645d4208b346a59d5e5d03836a45833b1d7
Reviewed-on: https://go-review.googlesource.com/c/go/+/390814
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
2022-04-26 18:48:16 +00:00
Heschi Kreinick
06b0a655a1 net: skip TestDialCancel on darwin-arm64
We're turning up Macs in a network environment that clashes with this
test. I don't think it's critical to get it working, so skip it.

For #49149.

Change-Id: I925e3ecc5356c4cefd208bdcff3d98021215d0b4
Reviewed-on: https://go-review.googlesource.com/c/go/+/402181
Reviewed-by: Alex Rakoczy <alex@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-26 17:50:49 +00:00
Hana
e35763469a README.md: update wiki link
Change-Id: I307c3524f2031e2a3e7ada6e86c73e278481de6d
Reviewed-on: https://go-review.googlesource.com/c/go/+/402179
Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
2022-04-26 16:21:18 +00:00
Bryan C. Mills
7b31af0eae os/exec: use a TestMain to avoid hijacking stdout for helper commands
The previous implementation of helperCommand relied on running a
well-known Test function which implemented all known commands.

That not only added Skip noise in the test's output, but also (and
more importantly) meant that the commands could not write directly to
stdout in the usual way, since the testing package hijacks os.Stdout
for its own use.

The new implementation addresses the above issues, and also ensures
that all registered commands are actually used, reducing the risk of
an unused command sticking around after refactoring.

It also sets the subprocess environment variable directly in the test
process, instead of on each individual helper command's Env field,
allowing helper commands to be used without an explicit Env.

Updates #50599.
(Also for #50436.)

Change-Id: I189c7bed9a07cfe47a084b657b88575b1ee370b9
Reviewed-on: https://go-review.googlesource.com/c/go/+/401934
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
2022-04-26 14:49:07 +00:00
Bryan C. Mills
ad5eaa8c4c os/exec: make skipStdinCopyError a function instead of a variable
This makes clearer that skipStdinCopyError is always defined and never
overridden in tests.

Secondarily, it may also help reduce init-time work and allow the
linker and/or inliner to better optimize this package.

(Noticed while prototyping #50436.)

Change-Id: I4f3c1bc146384a98136a4039f82165ed106c14b8
Reviewed-on: https://go-review.googlesource.com/c/go/+/401897
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
2022-04-26 13:47:05 +00:00
Russ Cox
17d7983b29 time: fix quickcheck test to avoid wraparounds
When we call time.Unix(s, ns), the internal representation is
s + 62135596800,  where 62135596800 is the number of
seconds from Jan 1 1 to Jan 1 1970.

If quickcheck generates numbers too close to 2^63,
the addition can wraparound to make a very negative
internal 64-bit value. Wraparounds are not guarded
against, since they would not arise in any reasonable program,
so just avoid testing near them.

Fixes #52409.

Change-Id: Id466c8a34a49055ab26f2687a6b2b657cb64bed6
Reviewed-on: https://go-review.googlesource.com/c/go/+/402177
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Auto-Submit: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-26 02:28:58 +00:00
Robert Griesemer
09ada1af8f cmd/compile/internal/syntax: parser to accept ~x as unary expression
Accept ~x as ordinary unary expression in the parser but recognize
such expressions as invalid in the type checker.

This change opens the door to recognizing complex type constraint
literals such as `*E|~int` in `[P *E|~int]` and parse them correctly
instead of reporting a parse error because `P*E|~int` syntactically
looks like an incorrect array length expression (binary expression
where the RHS of | is an invalid unary expression ~int).

As a result, the parser is more forgiving with expressions but the
type checker will reject invalid uses as before.

We could pass extra information into the binary/unary expression
parse functions to prevent the use of ~ in invalid situations but
it doesn't seem worth the trouble. In fact it may be advantageous
to allow a more liberal expression syntax especially in the presence
of errors (better parser synchronization after an error).

Preparation for fixing #49482.

Change-Id: I119e8bd9445dfa6460fcd7e0658e3554a34b2769
Reviewed-on: https://go-review.googlesource.com/c/go/+/402255
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Robert Griesemer <gri@google.com>
Auto-Submit: Robert Griesemer <gri@google.com>
2022-04-26 02:19:42 +00:00
zhouguangyuan
e845750744 cmd/compile: fix the missing size for FuncInfoSym
Change-Id: I46543e188bf25384e529a9d5a3095033ac618bbd
Reviewed-on: https://go-review.googlesource.com/c/go/+/402057
Run-TryBot: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
2022-04-25 23:45:52 +00:00
Carl Johnson
a5d61be040 net/http: add MaxBytesError
Fixes #30715

Change-Id: Ia3712d248b6dc86abef71ccea6e705a571933d53
GitHub-Last-Rev: 6ae68402a5
GitHub-Pull-Request: golang/go#49359
Reviewed-on: https://go-review.googlesource.com/c/go/+/361397
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-25 23:36:50 +00:00
cuiweixie
892cd0b636 reflect: support Len and Cap on pointer-to-array Value
Fixes #52411

Change-Id: I2fd13a453622992c52d49aade7cd058cfc8a77ca
GitHub-Last-Rev: d5987c2ec8
GitHub-Pull-Request: golang/go#52423
Reviewed-on: https://go-review.googlesource.com/c/go/+/400954
Reviewed-by: Keith Randall <khr@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@google.com>
Auto-Submit: Keith Randall <khr@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
2022-04-25 23:18:00 +00:00
Bryan C. Mills
35f2aba283 os: skip TestRemoveAllRace on dragonfly
This test occasionally fails on the dragonfly-amd64 builder with
"directory not empty". Since that is the only platform on which we
observe these failures, and since the test had a different (and also
invalid-looking) failure mode prior to this one (in #50716), we
suspect that it is due to either a bug in the platform or a
platform-specific Go bug.

For #52301.

Change-Id: Id36c499651b9c48e6b8b0107d01f73d2a7b6bab8
Reviewed-on: https://go-review.googlesource.com/c/go/+/402155
Run-TryBot: Bryan Mills <bcmills@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
2022-04-25 23:10:54 +00:00
Bryan C. Mills
22a00f2b5c os/exec: in TestImplicitPWD, explicitly request the logical path
Fixes #52537

Change-Id: I70959881a31f425e940e7adf86b36be2596aafb7
Reviewed-on: https://go-review.googlesource.com/c/go/+/402158
Run-TryBot: Bryan Mills <bcmills@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
2022-04-25 22:33:21 +00:00
张云浩
415e3fd8a6 slices: use !{{Less}} instead of {{GreaterOrEqual}}
In CL 371574 PatchSet 18, we replaced all !{{Less}} with {{GreaterOrEqual}} to fix a problem(handle NaNs when sorting float64 slice) in exp/slices.

We don't actually need this change, because we don't guarantee that the slice will be sorted eventually if there are NaNs(we could have a[i] < a[j] for some i,j with i>j).

This CL reverts all the replacements in exp/slices and does not affect any codes in the sort package.

Change-Id: Idc225d480de3e2efef2add35c709ed880d1306cb
Reviewed-on: https://go-review.googlesource.com/c/go/+/400534
Reviewed-by: Keith Randall <khr@golang.org>
Run-TryBot: Keith Randall <khr@golang.org>
Auto-Submit: Keith Randall <khr@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Eli Bendersky <eliben@google.com>
Reviewed-by: Keith Randall <khr@google.com>
Auto-Submit: Keith Randall <khr@google.com>
2022-04-25 19:12:14 +00:00
Austin Clements
60178e6240 cmd/dist: add maymorestack tests
These tests run the runtime, reflect, and sync package tests with the
two maymorestack hooks we have.

These tests only run on the longtest builders (or with
GO_TEST_SHORT=false) because we're running the runtime test two
additional times and the mayMoreStackMove hook makes it about twice as
slow (~230 seconds).

To run just these tests by hand, do

  GO_TEST_SHORT=false go tool dist test -run mayMoreStack

Updates #48297.

This detected #49354, which was found as a flake on the dashboard, but
was reliably reproducible with these tests; and #49395.

Change-Id: If785a8b8d6e1b9ad4d2ae67493b54055ab6cbc85
Reviewed-on: https://go-review.googlesource.com/c/go/+/361212
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Austin Clements <austin@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
2022-04-25 18:12:08 +00:00
Austin Clements
8619d3b2ec runtime: fix stack-move sensitivity in some tests
There are a few tests of the scheduler run queue API that allocate a
local []g and test using those G's. However, the run queue API
frequently converts between *g and guintptr, which is safe for "real"
Gs because they're heap-allocated and hence don't move, but if these
tests get a stack movement while holding one of these local *g's as a
guintptr, it won't get updated and the test will fail.

Updates #48297.

Change-Id: Ifd424147ce1a1b53732ff0cf55a81df1a9beeb3b
Reviewed-on: https://go-review.googlesource.com/c/go/+/402157
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
2022-04-25 18:06:52 +00:00
Park Zhou
12763d141d cmd/compile: align table
Signed-off-by: Park Zhou <buildpaas@gmail.com>
Change-Id: Idbbd2779264a7310b839af8291315e5e38b7ced9
Reviewed-on: https://go-review.googlesource.com/c/go/+/402120
Reviewed-by: Robert Griesemer <gri@google.com>
Reviewed-by: Keith Randall <khr@golang.org>
Auto-Submit: Keith Randall <khr@golang.org>
Reviewed-by: Keith Randall <khr@google.com>
2022-04-25 15:49:44 +00:00
Than McIntosh
94f25ec949 debug/pe: fix off by one error in valid symbol index test
Fix an off-by-one error in COFFSymbolReadSectionDefAux, specifically
the code that tests whether a symbol index is valid.

Fixes #52525.

Change-Id: I1b6e5dacfd99249c694bef5ae606e90fdb2ef521
Reviewed-on: https://go-review.googlesource.com/c/go/+/402156
Run-TryBot: Than McIntosh <thanm@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-25 15:42:53 +00:00
Than McIntosh
6b1d9aefa8 crypto/ed25519: test fixup
Fix up TestEd25519Vectors to download files into its own temporary mod
cache, as opposed relying on whatever GOPATH or GOMODCACHE setting is
in effect when the test is run.

Change-Id: I523f1862f5874b0635a6c0fa83d35a6cfac6073b
Reviewed-on: https://go-review.googlesource.com/c/go/+/402154
Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-25 14:52:51 +00:00
Robert Findley
8140a605fe go/types, types2: add loong64 to gcArchSizes
Values are taken from cmd/internal/sys/arch.go. Also fix some incorrect
alphabetical sorting to put arm > amd.

Updates #46229
Fixes #52495

Change-Id: If7d2c675238093692060358003915f1649792cf5
Reviewed-on: https://go-review.googlesource.com/c/go/+/401576
Run-TryBot: Robert Findley <rfindley@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: WANG Xuerui <git@xen0n.name>
Reviewed-by: xiaodong liu <teaofmoli@gmail.com>
2022-04-25 13:37:03 +00:00
Meng Zhuo
96c8cc7fea runtime: add ABIInternal to strhash and memhash on riscv64
This CL fixes regression of strhash and memhash on riscv64

Change-Id: Icc10431a8199c8b1eb7b440cb42be4e53420e171
Reviewed-on: https://go-review.googlesource.com/c/go/+/362134
Run-TryBot: mzh <mzh@golangcn.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Russ Cox <rsc@golang.org>
2022-04-24 01:22:21 +00:00
Joel Sing
e6dfdbd11d os: use a lower file count for TestOpenFileLimit on openbsd
OpenBSD has a default soft limit of 512 and hard limit of 1024 - as such,
attempting to open 1200 files is always going to fail unless the defaults
have been changed. On this platform use 768 instead such that it passes
without requiring customisation.

Fixes #51713

Change-Id: I7679c8fd73d4b263145129e9308afdb29d67bb54
Reviewed-on: https://go-review.googlesource.com/c/go/+/401594
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
Reviewed-by: 谢致邦 <xiezhibang@gmail.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Peter Weinberger <pjw@google.com>
2022-04-23 14:26:25 +00:00
eric fang
9717e8f80f runtime: support for debugger function calls on linux/arm64
This CL adds support for debugger function calls on linux arm64
platform. The protocol is basically the same as in CL 109699, except for
the following differences:
1, The abi difference which affect parameter passing and frame layout.
2, Stores communication information in R20.
3, The closure register is R26.
4, Use BRK 0 instruction to generate a breakpoint. The saved PC in
sigcontext is the PC where the signal occurred, not the next PC.

In addition, this CL refactors the existing code (which is dedicated to
amd64) for easier multi-arch scaling.

Fixes #50614

Change-Id: I06b14e345cc89aab175f4a5f2287b765da85a86b
Reviewed-on: https://go-review.googlesource.com/c/go/+/395754
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Run-TryBot: Eric Fang <eric.fang@arm.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-23 05:38:56 +00:00
Keith Randall
7c680974c6 runtime/race: add s390x .syso file
Built using racebuild.

Note that racebuild fails when trying to test the .syso, because the
Go runtime doesn't think we support s390x race yet. But it builds the
.syso as a side effect which I grabbed. There's something of a
chicken-and-egg bootstrapping problem here, unfortunately.

Change-Id: Ibc6d04fd3a9bfb3224d08e8b78dcf09bb139a59d
Reviewed-on: https://go-review.googlesource.com/c/go/+/401714
Run-TryBot: Keith Randall <khr@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Jonathan Albrecht <jonathan.albrecht@ibm.com>
2022-04-22 22:38:00 +00:00