DevOps 6 min read 1238 words

Eight one-byte keys: how Go responded to the eMudhra incident

ES
Eight one-byte keys: how Go responded to the eMudhra incident

A few days ago, in the post on how to mitigate PHP2Shell, I mentioned the eMudhra certificates in passing. The Go team has now published how it responded to the incident, and it’s a good moment to tell the whole story: what happened, why it affected Go in a particular way and what’s worth checking outside the browser.

What happened

eMudhra is an Indian certificate authority whose root is in most trust stores. Eight of its intermediate certificates (six for TLS and two for code signing) had absurdly weak private keys. According to Scott Helme’s analysis, they were P-384 elliptic curve keys in which the private scalar, which should be a random 384-bit number, was 47 zero bytes followed by a single byte. There are only 255 possible values.

Recovering a key like that means multiplying the curve’s generator point by 1, by 2, by 3… until the result matches the certificate’s public key. I tried it with a key generated with the same flaw:

from cryptography.hazmat.primitives.asymmetric import ec

# A "CA" key with a one-byte private scalar
weak = ec.derive_private_key(155, ec.SECP384R1())
public = weak.public_key().public_numbers()

# Recover it: try d = 1, 2, 3... until d·G matches
for d in range(1, 256):
    candidate = ec.derive_private_key(d, ec.SECP384R1()).public_key().public_numbers()
    if candidate == public:
        print(f"Private key recovered: d = {d}")
        break

On my machine it takes under half a second, including starting Python. With any of those keys you can sign a valid TLS certificate for any domain, without proving you control it, and run a man-in-the-middle attack against anyone who trusts that chain.

According to eMudhra, the cause was a bug in the key ceremony script: a second, weak key pair was created on the HSM and, because of similar names, the wrong one was used for signing. The intermediates were created in April 2025, Sectigo cross-signed them in May 2025, and they’re valid until 2039. For about 17 months, those keys were effectively public to anyone who bothered to look.

The response, within hours

A researcher reported it on Saturday 17 October to Apple, Google, Microsoft, Mozilla and eMudhra, and the reaction was quick:

WhoWhat they did
Google ChromeBlocked the intermediates with a CRLSet update on the night of the 17th
MozillaAdded them to OneCRL in Firefox hours after the report, and on the 20th shipped an NSS update removing the eMudhra root
AppleHas blocked the 12 certificates (six intermediates, each signed by two roots) since the 19th, with no system update needed, and the emSign ECC Root CA - G3 root since the 23rd
GoReleased Go 1.27.3 and 1.26.10 on the 19th, about seven and a half hours after finding out

Why Go had to step in

This is the interesting part of Go’s case. Its standard library verifies TLS certificates for net/http and crypto/tls, and does so differently depending on the system: on Windows and macOS it delegates to the platform APIs, which receive Microsoft’s and Apple’s blocks; but on Linux it uses its own verifier, because there’s no common system mechanism for checking revocation. A Go binary on a Linux server would have kept accepting forged certificates until the system store was updated, and that depends on each distribution.

So the Go security team decided to block those intermediates in its own verifier and used its URGENT release track for the first time. Their timeline for 19 October, Pacific time:

TimeStep
11:45An independent researcher alerts the Go security team
12:30Mozilla confirms it independently, asks them not to disclose yet and encourages releasing a fix as soon as possible
14:00Patch finished, tested against certificates signed with the recovered keys
15:00Release process for 1.27.3 and 1.26.10 begins
19:13Releases published and announced

The announcement was deliberately vague: it spoke of “specific security concerns around their operation” so as not to tip off attackers. By the next morning, several researchers had already published the keys.

What I like most about the post is the lessons learned section. Of those seven and a half hours, the release process itself took almost four and a quarter, because of issues that had never mattered when a release was spread over several days. They’ve already improved it, and acknowledge they could have pre-announced the urgent release when they started it, giving people at least a few hours to prepare. They also say they’ll prioritise revocation support on Linux on a par with other platforms, and are looking at the upki project for that.

What to check

If you have Go binaries that speak TLS, directly or through dependencies like gRPC, they need to be rebuilt with Go 1.27.3 or 1.26.10 or later. The Go team explains how to check:

# Toolchain version
go version

# Go version a binary was built with
go version -m ./my-binary

# Check whether the binary is affected
GOTOOLCHAIN=go1.27.3+auto go install golang.org/x/vuln/cmd/govulncheck@latest
govulncheck -mode=binary ./my-binary

Rebuilding is the key point: a Go binary carries the standard library inside it, so upgrading Go on the machine changes nothing in binaries already deployed. The same goes for Docker images that ship a compiled binary.

If you use macOS from the terminal, Apple’s note explains a detail I didn’t expect: Safari, Mail and apps using Apple’s frameworks already block these certificates, but the curl, git, python3, ruby, perl and openssl included with macOS or the Xcode Command Line Tools verify against /etc/ssl/cert.pem, without checking Apple’s block or revocation. That file will be updated in an upcoming macOS release, and in the meantime Apple gives the commands to remove the roots by hand. To check whether they’re still there:

openssl crl2pkcs7 -nocrl -certfile /etc/ssl/cert.pem \
  | openssl pkcs7 -print_certs -noout \
  | grep -E 'emSign ECC Root CA - G3|AAA Certificate Services'

Note a subtlety the note itself explains: removing only the eMudhra root isn’t enough, because the six intermediates cross-signed by Sectigo still validate through AAA Certificate Services. Removing that root too may break verification for other legitimate sites that depend on it, so it’s worth reading the full note before touching anything.

Outside the browser in general. Scott Helme found that curl, Python and Node, with their default settings, accepted a forged certificate for a real domain on both Linux and macOS. Browsers were protected because they require Certificate Transparency and receive the blocks, but many clients only check whether the chain reaches a root on their list. On servers and containers, that means updating the distribution’s root certificate package as soon as it’s released, and not forgetting applications that ship their own store.

My takeaway

This case seems to me a good reminder that trust in TLS breaks at the silliest link: no sophisticated attack was needed, just a script that generated the wrong key and nobody checking that the key was the size it should be.

It also shows clearly the difference between the browser and everything else. Browsers have mechanisms to react within hours, but most of a server’s TLS traffic comes from clients like curl, HTTP libraries or Go binaries that trust a list of roots and little else. That’s why I value Go acting in its own verifier without waiting for distributions, and explaining it in detail, including what went worse than expected.

Source: Our Response To The eMudhra Key Disclosure Incident, Roland Shoemaker and the Go security team. More context in About the compromised eMudhra intermediate CAs, Apple, and Eight bits of trust, Scott Helme.