How We Overcame The PyTorch Supply Chain Attack

How Our VSM Tool Overcame The PyTorch Supply Chain Attack | LeanIX

SBOM Jens Rieskamp | Director Product Value Stream Management (VSM) | LeanIX s are rapidly becoming an essential part of value stream management. Jens Rieskamp, our Director Product Value Stream Management, explains how we used SBOMs and our LeanIX solution to quickly protect our IT landscape during the PyTorch cyber attack last month.

What is PyTorch?

PyTorch is a Python-based, open-source software library that was developed by Meta's artificial intelligence (AI) research lab. It's widely used in the scientific research community for building and deploying machine- and deep-learning applications.

PyTorch provides high-level interfaces for working with and training deep-learning models. It's also becoming increasingly popular for use in machine learning for business, including within some of our applications here at LeanIX.

The torchtriton supply chain attack

On New Year's Eve, Pytorch learned about a compromised PyTorch-nightly dependency chain between December 25th and December 30th, 2022. A malicious dependency package named "torchtriton" was uploaded to the Python Package Index (PyPI) code repository. It was given the same package name as the one shipped on the PyTorch nightly package index.

Users installing the PyTorch nightly package index on Linux via the Python preferred installer program (PIP) added this dependency and ran a malicious binary. This is known as a 'supply chain attack', and directly affects dependencies for packages that are hosted on public package indices.

In these circumstances, it's crucial for an organization discovering such a vulnerability to rapidly determine which parts of their IT landscape are impacted. The most effective way to do this is to keep a software bill of materials (SBOM) for each part of your landscape.

What is an SBOM?

A software bill of materials (SBOM) is a list of all the modules, libraries, and components that were used to build a piece of software and all the supply chain links between them.

The items are included in the list whether they are open source, owned, or paid for by the company. This means any party can quickly identify whether each piece of software is impacted by a vulnerability or redundancy.

Equally important, since the information doesn't contain proprietary details, it can be freely shared in public spaces, allowing for intelligence regarding vulnerabilities to be shared by all parties involved. This is why SBOMs are now a requirement under the ISO standard for open-source software, and the US government is mandating all software providers it contracts with must have SBOMs prepared.

Using SBOMs against torchtriton vulnerability

Thankfully, LeanIX is well ahead of the curve with software bill of materials (SBOM). We are currently rolling out the process to create and scan detailed SBOM files for our entire microservice landscape.

Microservices view | How Our VSM Tool Overcame The PyTorch Supply Chain Attack | LeanIX

We utilize over 500 microservices, maintained by more than 20 teams within LeanIX. Without our SBOMs, identifying which utilized PyTorch would be an extensive task.

Within our LeanIX, however, we maintain an inventory of each microservice, including owner, links to external source systems, and data gathered automatically from build and runtime environments.

Automatically gathered data | How Our VSM Tool Overcame The PyTorch Supply Chain Attack | LeanIX

As part of the data we collect, we use CycloneDX to gather SBOMs for each service. This is the standard for SBOMs, which are increasingly preferred by the US government National Institute of Standards and Technology Software Security in Supply Chains: Software Bill of Materials (SBOM) standard and provide detailed information on the technical makeup of each service's used licenses, libraries, and transitive dependencies.

Our teams make use of the plugins in the CycloneDX tool center, which you can find out more about in our engineering blog post, A Tool Overview for Generating CycloneDX SBOM. The automatically gathered data can be further enhanced by manual metadata, such as action class assessments.

In addition to the data for each service, our platform also has a library catalog that allows an immediate impact analysis across hundreds of SBOMs. As such, when the torchtriton vulnerability came to light, we were immediately able to see:

which services were impacted
who owned them
which products relied on them

Torch library catalog | How Our VSM Tool Overcame The PyTorch Supply Chain Attack | LeanIX

This meant LeanIX could resolve the impact of the incident rapidly, with no loss of service or data breaches for our customers.

How SBOMs have changed since the Log4J incident

If you're a LeanIX customer, you may remember how we mitigated the Log4J vulnerability "Log4Shell" within 48 hours in December 2021.

Our approach and our success were similar to the torchtriton vulnerability, but the difference is in the switch from a scanner to the SBOM standard. SBOMs allow for more flexibility in source data, thanks to the CycloneDX tool center.

In both cases, our LeanIX proved essential for quickly bringing data together at scale, and analyzing and identifying the impact of key vulnerabilities.