The Critical Importance of Supporting Open Source Projects: Lessons from the xz Backdoor Incident
On March 29, 2024, software developer Andres Freund discovered a malicious backdoor in the Linux utility xz within the liblzma library, affecting versions 5.6.0 and 5.6.1. Released by an account under the name “Jia Tan” in February 2024, this backdoor has raised alarms across the tech community. Although the backdoored version had not yet reached widespread production deployment, it was present in development versions of major Linux distributions. This incident serves as a stark reminder of the critical importance of supporting open source projects and their maintainers, as too many companies exploit free software without contributing back, leaving the community vulnerable to such security threats. The backdoor in question grants an attacker with a specific Ed448 private key the ability to execute remote code on an affected Linux system, a vulnerability assigned the highest CVSS score of 10.0. Freund, a Microsoft employee and PostgreSQL developer, reported the backdoor after noticing unusual CPU usage during SSH connections and errors in the memory debugging tool Valgrind on Debian Sid. His findings, shared with the Openwall Project’s security mailing list, alerted various software vendors to the threat. The backdoor manipulates the behavior of OpenSSH’s SSH server daemon via the systemd library, allowing attackers to gain administrator access. The investigation into this sophisticated attack revealed a three-year effort by the attacker, using the pseudonym Jia Tan and several sock puppet accounts, to gain a position of trust within the XZ Utils project. Through persistent pressure and manipulation, Jia Tan became a co-maintainer, enabling the signing off of the compromised versions. The attacker’s high level of operational security and the elaborate nature of the backdoor point to a potentially state-sponsored actor, with speculation ranging from APT29 (associated with the Russian SVR) to other state or well-resourced non-state actors. The backdoor works by altering the OpenSSH function RSA_public_decrypt via the glibc IFUNC mechanism, triggered under specific conditions involving a third-party patch of the SSH server. This intricate mechanism highlights the attack’s sophistication and the severe implications it could have had if left undetected. The malicious code lay dormant in the git repository and was activated under precise build conditions, demonstrating the attacker’s deep understanding of the software’s deployment and usage. In response to this discovery, major Linux vendors, including Red Hat, SUSE, and Debian, rolled back to uncompromised versions of the affected packages. The US Cybersecurity and Infrastructure Security Agency (CISA) issued advisories, and GitHub temporarily disabled the xz repository mirrors. Canonical, in an abundance of caution, postponed the beta release of Ubuntu 24.04 LTS to ensure no further compromises in their package builds. This incident has sparked a broader discussion on the sustainability and security of open source projects, especially those maintained by unpaid volunteers. The critical role of such projects in the global software ecosystem, as humorously but poignantly captured by the XKCD comic no. 2347 “Dependency,” underscores the fragility of relying on volunteer-maintained software for essential infrastructure. Computer scientist Alex Stamos remarked that this could have been the most pervasive and effective backdoor ever, potentially giving attackers a master key to millions of systems worldwide. The xz backdoor incident serves as a clarion call for better support and funding for open source maintainers. Many companies benefit immensely from free software without contributing resources to ensure its security and sustainability. This imbalance needs rectification to prevent similar incidents in the future. Companies should invest in the open source projects they depend on, ensuring these critical systems remain secure and resilient against sophisticated threats. Supporting open source projects is not just a matter of fairness; it is a strategic necessity for the security and stability of the global software ecosystem. By investing in these projects and their maintainers, we can build a more secure and robust foundation for all software that relies on open source components. For a visual take on this issue, check out the relevant XKCD comic.