LiteLLM Compromised - TeamPCP Hackers Inject Backdoor

What Happened

A major security incident has unfolded involving the LiteLLM Python package, which boasts over 95 million monthly downloads. This widely used open-source library was compromised on the Python Package Index (PyPI). Security vendors Endor Labs and JFrog discovered that versions 1.82.7 and 1.82.8 of LiteLLM contained a sophisticated backdoor. The malicious code was cleverly injected directly into the PyPI distribution, bypassing the clean upstream GitHub repository.

The attackers, identified as TeamPCP, are known for targeting developer and security tools. They executed a supply chain attack, embedding malicious code within legitimate library functions. In version 1.82.7, a 12-line base64-encoded payload was inserted into the litellm/proxy/proxy_server.py file, triggering silently upon module import. The subsequent version escalated the threat by adding a litellm_init.pth file, ensuring the payload executed during any Python invocation, even if LiteLLM wasn't explicitly imported.

Who's Being Targeted

Organizations and developers using the LiteLLM package are at risk. The compromised versions were designed to execute an aggressive three-stage attack sequence. Upon execution, the payload initiates a credential harvester targeting sensitive information such as SSH keys, cloud provider tokens, and database credentials. This means that anyone utilizing the affected versions of LiteLLM could unknowingly expose critical data to attackers.

The attack is particularly concerning for environments rich with production secrets, such as those in Kubernetes. The malware attempts lateral movement within these environments, seeking to deploy privileged containers across cluster nodes. As such, the implications of this breach extend beyond individual developers to entire organizations relying on LiteLLM for their operations.

Signs of Infection

There are several indicators of compromise (IOCs) that organizations should be aware of. The primary C2 domain used for exfiltration is models.litellm.cloud, where stolen credentials are sent. Additionally, the persistent backdoor can be found in the form of a systemd unit named sysmon.service, which disguises itself as a telemetry process. This backdoor allows attackers to maintain access and execute further commands remotely.

Organizations should also look for the presence of the archive tpcp.tar.gz, which contains the exfiltrated data. If any of these indicators are detected, it is crucial to treat the environment as fully compromised and initiate a thorough security review.

How to Protect Yourself

Immediate action is essential for any organization using LiteLLM. Security teams must audit their environments for the compromised versions. If detected, they should initiate a comprehensive credential rotation protocol to mitigate the risks associated with this breach.

Moreover, organizations should enhance their security posture by implementing stricter controls on package management and regularly reviewing dependencies for vulnerabilities. This incident serves as a stark reminder of the risks associated with supply chain attacks, highlighting the need for vigilance in software development practices.