Executive Summary / TL;DR
On Dec. 9, 2021, a remote code execution (RCE) vulnerability in Apache log4j 2 was identified as being exploited in the wild. Investigation revealed that this exploitation was incredibly easy to perform. Due to the broad usage of the popular Java library, many IT Systems and SaaS providers are now at severe risk. By submitting a specially crafted request to a vulnerable system, an attacker is able to instruct that system to download and subsequently execute a malicious payload. LeanIX mitigated the risk on our side by upgrading all crucial systems as of Saturday, Dec 11th. Analysis of our logs did not show any indications of exploitation. This rapid response – 48 hours – was possible thanks to the coordinated efforts of our passionate engineering team and the full visibility across our own (Micro)services provided by LeanIX Value Stream Management (VSM).
Identifying all affected services
A crucial first step for our team was to identify which of our software artifacts were affected by the vulnerability. Since we maintain an automated inventory of all services we use, we could query our internal VSM inventory for the affected log4j versions:
This data is always accurate because we automatically receive the dependencies to underlying libraries from data produced in our CI/CD toolchain when building our services.
Understanding the full impact on teams and products
With 150 employees in our product department organised in 5 tribes and 20+ teams, it was important to understand the full picture and see which services in which teams were affected:
In order to identify any potential customer impact, we also needed to understand exactly, which of our products was impacted by the vulnerability:
Upgrading log4j library and monitoring the progress
To mitigate the vulnerability, our engineers implemented a strategy focused on upgrading the affected software versions. Once the logging library was updated and the service was rebuilt via the CI/CD pipelines, the updated versions automatically showed up in the audit trail.
Once deployment was confirmed (increase of deployment count), the status of the service was set back to "Green". In total, libraries in 7 services had to be upgraded.
