Summary
“Jia Tan” kept submitting high-quality patches, earned the project’s trust, and obtained de facto maintainer authority, including creating and signing releases. The backdoor was not written directly into the source tree; it was hidden as a combination of a build script and test files included only in the release archives, and the archives were validly signed with “Jia Tan“‘s PGP key. A valid signature still lets a release from a developer whose real existence cannot be confirmed pass as official, unless who holds the key is checked before authority is handed over.
A pre-execution check means checking, before authority is handed over and before a release is pulled in, that the key-holder is an approved real person, and keeping a record. Detection means noticing from anomalous behavior after distribution. Dates follow the timeline in section 05.
They reached development distributions such as Fedora Rawhide
What got through
- What got through
- Publishing release archives 5.6.0 / 5.6.1 signed with “Jia Tan“‘s PGP key, and each distribution pulling them in as legitimate packages. Before that, after about two years of contributions and direct contact with the existing maintainer, de facto maintainer authority, including creating and signing releases, had been handed over.
- What reached the receiving side
- In general, the side pulling in a release receives the archive and a valid signature; who holds the key cannot be told from the signature. Here, the backdoor was a combination of an Autoconf build script (
build-to-host.m4) and obfuscated test files included only in the.tar.gzarchives, and it did not appear in the source from agit clone(Andres Freund, Binarly). - How far it reached
- Fedora Rawhide and parts of Debian testing / unstable. It was caught before reaching major stable releases. No evidence that the attack was actually used has been confirmed.
- Not disclosed
- Who “Jia Tan” is. Analysis of time zones, commit timestamps, and similar features has led to discussion of nation-state-level involvement, but attribution is not confirmed.
Pre-execution check
Whatever the route, the condition for telling them apart is checking, before release or signing authority is handed over, that the key-holder corresponds to a real person the project approved.
A valid signature still lets a release from a developer whose real existence cannot be confirmed pass as official, unless who holds the key is checked before authority is handed over.
- Who approves
- The maintainers of the project handing over authority, and the distributions that pull in its releases. The volume or quality of contributions, or a valid signature, is not an approval of the key-holder.
- Where it is checked
- Before maintainer or release authority is handed over, and before a signed release is pulled in. Releases signed with a key whose holder cannot be checked are held, not pulled in.
- What is recorded
- Who handed which authority to whom, and which release was pulled in on which signer’s confirmation. Later, it shows which version came in under whose signature.
It was found before reaching stable releases, but by then the signed releases had already entered development distributions as legitimate packages. Other cases where a signature or credential was valid while the origin of the contents or the signer was not checked include a push with a legitimate developer’s stolen credentials (Brief 004) and malicious npm packages published with a legitimate OIDC signature (Brief 014).
What Lemma does
Lemma builds the pre-execution check into the steps that hand over authority and pull in releases. It does not watch for anomalous behavior; before authority is handed over and before a release is pulled in, it checks that the key-holder is an approved real person, and does not let through what cannot be checked.
Industries and operations to watch
- IndustryDevelopers & agent operations
- Developers, organizations, and distributions that pull in OSS releases as dependencies or packages and feed them into builds and distribution.
- OperationSending & delivery
- The flow in which signed release archives are published and each distribution ships them as legitimate packages.
- OperationAccounts & permissions
- The steps by which an OSS project hands commit rights, or authority to create and sign releases, to contributors.
Current controls vs Lemma
The route in this case is distributions pulling in a release signed by someone who had obtained maintainer authority.
| Control | Handing over maintainer authority | Pulling in a signed release |
|---|---|---|
| Signature verification | Not applicable | Not effective |
Review of the source from git clone |
Not applicable | Not effective |
| Detection of anomalous behavior (delay, Valgrind output) | Not applicable | Partial |
| Lemma (pre-execution check) | Effective | Partial |
Signature verification does not catch the backdoor, because the signature was valid. Source review struggles to tell it apart, because the backdoor did not appear in the source from a git clone and was included only in the archives. Detection of anomalous behavior was the only path of discovery here and worked before stable releases were reached, but the releases had already entered development distributions. Lemma checks, before authority is handed over, that the recipient’s key corresponds to an approved real person. Before a release is pulled in, it checks that the signer passed that check, but it does not judge whether the contents are safe.
Note: Lemma’s check works only when it is built into the steps that hand over authority and pull in releases, and the project manages which approved people correspond to which keys. If a person who passed the check signs with malicious intent, or the key of someone who passed the check is stolen, this check can be passed. The table compares what each control checks; it does not show whether this incident would have been stopped. Lemma does not detect or prevent intrusions.
Three things to confirm for development organizations that pull in OSS releases
- Do you confirm that the OSS release archives you pull in have the same contents as the published source?
- Besides verifying the signature, is there a procedure to confirm who holds the signing key?
- Do you record which release was pulled in, on confirmation of whose signature?
Facts and timeline
- Subject
- xz utils (liblzma), a compression library widely used across Linux / Unix-family systems and shipped by default in numerous distributions.
- CVE
- CVE-2024-3094 (CVSS 10.0, NIST NVD).
- Discovery
- Andres Freund (an engineer at Microsoft) noticed Valgrind output and a slight delay in SSH logins (about 500ms), investigated, and reported to the Openwall mailing list on 2024-03-29.
- Attacker
- “Jia Tan” (GitHub: JiaT75), a person whose real existence cannot be confirmed. Nation-state-level involvement remains circumstantial and unconfirmed.
- Technique
- The backdoor was hidden as a combination of an Autoconf build script (
build-to-host.m4) and obfuscated test files included only in the.tar.gzarchives. When a systemd-linkedopenssh-serverloads liblzma, it hooks public-key authentication, putting the holder of the attacker’s specific private key in a position to execute RCE. - Signature
- The release archives were signed with “Jia Tan“‘s PGP key, and the signature was technically valid.
- How far it reached
- Fedora Rawhide and parts of Debian testing / unstable. It was caught before reaching major stable releases.
Glossary
PGP signature — A way to show that a file was signed with a particular key. A valid signature alone does not tell you who holds the key.
Release archive — A compressed bundle of the source for distribution (.tar.gz). Here, files absent from the git source were included only in the archive.
Timeline
- Late 2021 to early 2022
- “Jia Tan” begins contributing to the xz project, earning trust by continuously submitting high-quality patches.
- 2022 to 2023
- Obtains commit rights and, by gradually taking over the work of the existing maintainer (Lasse Collin), becomes a core contributor. During the same period, suspicious accounts pressing the existing maintainer from outside to “release more aggressively” were also observed.
- 2024-02-24
- xz utils 5.6.0 is released. The build script containing the backdoor is included for the first time.
- 2024-03-09
- xz utils 5.6.1 is released. Details of the backdoor are updated.
- 2024-03-29
- Andres Freund finds the liblzma backdoor and reports it to the Openwall mailing list and to security stakeholders.
- 2024-03-29 to 30
- CISA and the Linux distributors issue emergency advisories recommending a downgrade from affected versions.
- After 2024-03-29
- “Jia Tan“‘s account activity stops. Detailed technical analyses follow from Binarly, Openwall, and others.
Names, the CVE, and versions are based on primary sources from Openwall (Andres Freund’s first report), NIST NVD, and CISA. Attribution to a nation-state-level actor remains circumstantial and unconfirmed. Each distributor’s remediation status varies over time, so consult the latest information. What has not been disclosed does not mean there were no controls.
After disclosure
- Andres Freund (Microsoft): Notified the community and CISA on the day of discovery.
- CISA: Issued an emergency alert (2024-03-29) recommending a downgrade from affected versions.
- The Linux distributors: Fedora, Debian, openSUSE, Arch, and others distributed downgrade packages.
- Binarly: Published a detailed technical analysis of the obfuscated build scripts and test files.
Sources
- Andres Freund (primary report): Openwall mailing-list post “backdoor in upstream xz/liblzma leading to ssh server compromise” (2024-03-29) — https://www.openwall.com/lists/oss-security/2024/03/29/4
- CISA emergency alert: “CISA Adds One Known Exploited Vulnerability to Catalog” / XZ Utils Backdoor Advisory (2024-03-29)
- Binarly: “XZ Utils Supply Chain Backdoor: Technical Analysis” (2024-03 to 04)
- NIST NVD: CVE-2024-3094 entry — https://nvd.nist.gov/vuln/detail/CVE-2024-3094
- Ars Technica: “What we know about the xz Utils backdoor that almost infected the world” (2024-03-29)
References: “The last layer left for cyber defense in the age of AI”