What is Log4Shell?

Log4Shell is an infamous vulnerability (CVE-2021-44228) in a popular logging library for Java called log4j2.

The vulnerability allows one to trigger code injection through the creation of a specially crafted log message. What you can achieve with it depends on the sophistication of the security measures you have in place, which in turn depends on the professionalism and paranoia of your security team. In principle, this vulnerability can allow an attacker to hack into the innermost systems of an organisation.

Why is it a Big Thing?

The Impact

Few would care about the vulnerability had it been in some obscure and rarely used software. However, log4j2 is very popular. Perhaps more than half of all software in Java and more than half of its deployments use it. So the breadth of the vulnerability is significant. On top of that, creating an exploit and trying it out is fairly easy.

The second reason for the scare was that there was no trivial solution to mend this mess. Of course, some attempts were made immediately, but they had to be followed by others as the understanding improved.

Let’s consider version 2.14.1 for example. It lists the following direct vulnerabilities:

• CVE-2021-44228 - the original one
• CVE-2021-44832
• CVE-2021-45046
• CVE-2021-45105

… and more than 20 vulnerabilities in the dependencies of this library. Although to be fair, those would be compile-time / provided / test dependencies, so they do not affect log4j2 when it is used in a normal way.

Eventually, it was discovered that there is more than one vulnerability and not only log4j2 has problems, but also the older generation of the logging library — log4j] — is vulnerable (although differently). For example, its most recent version 1.2.17 lists the following direct vulnerabilities:

• CVE-2022-23307
• CVE-2022-23305
• CVE-2022-23302
• CVE-2021-4104
• CVE-2019-17571

The older library is obsolete for a decade already, but as of this writing, it is still widely used. In particular, Maven‘s “dependency” plugin still depends on it indirectly, and not even on the most recent version. Amazing, really.

Common Wisdom vs Objective Reality

Another big reason for this vulnerability being important is this: log4j2 is a popular Open Source library. The common wisdom says that one of the great benefits of Open Source is that there are many eyes reviewing the same code and so it is more likely that vulnerabilities would be spotted. Hence, people conclude that Open Source is safer.

Well… If log4j2 is such a popular library (few are more popular than it), then why such a massive security hole was not noticed? Correction: “multiple holes”. And it is not like the vulnerability is something that requires special knowledge to understand like the hard-core mathematics of cryptography. It is, in fact, a fairly obvious thing.

I’ll tell you a secret: nobody cares to look at the code. Including me. People see log4j2 as a utility. They add it as a dependency and use it. It works, and people are happy. Until they are no longer happy, and it is time to pay the price.

What is Wrong with Log4j2?

Is there anything wrong with log4j2 per se? Some would say “nothing”, the library is similar to a plethora of other libraries in many ways. It is not unusually broken, in comparison.

I would say there are two fundamental problems in log4j2, and they were significant contributors to the emergence of Log4Shell. However, I would also note that in comparison, log4j2 is not as bad as other software. In fact, it is quite good. Which, if you ask me, is scary.

Food Processor Issue

In my view, the problem with log4j2 is that it suffers from a Food Processor Issue. That is, it is trying to do many things and aims for features rather than simplicity and composability. It provides a logging API, a number of different ways to configure the logging, several different mechanisms for logging and formatting, and a wealth of obscure features that are used by less than 1% of people.

It is one of those obscure features which has resulted in Log4Shell. Specifically, the original CVE was due to log4j2 being able to look up various JNDI resources (such as resources accessible via LDAP), download that data and even execute it.

Yes, that works only if you use log4j in an inappropriate way like this:

log.info("User " + user + " has logged in.");

rather than the correct one

log.info("User {} has logged in.", user);

In other words, the vulnerability is open only when the code allows for an injection in the first place. However, how certain can you be that all of your software is using log4j2 the way it was meant to be used? And only that, how certain can you be that all the libraries used by your software also contain no errors of this sort?

But let’s skip the matter of imperfect usage of API. Can you see a reason why would anyone need a logging library to download data from LDAP? In fact, why would there ever be a need to download anything at all at the point of logging instead of just logging whatever was passed to the logging library directly and is already available? I am genuinely interested to know at least one valid example.

And if there is one, why would anyone want that to be executed? Isn’t it a textbook definition of a backdoor? And why was it enabled by default?

It is important to remember the simple truth: the larger the software is, the larger its surface of attack is. Small software is — all things being equal — less vulnerable than large one. Had log4-core been indeed a “core” and had only the most basic logging functionality, 99% of its users would not have suffered from Log4Shell.

Easy ≠ Better

The second problem with log4j2, as I see it, is that while making it easy for people to use the library, decisions were made which resulted in log4j2 having certain weaknesses.

For example, it is very easy to start using log4j. You do not need much before you being logging something with those log4j.info(...). But then comes the time when you need your software to be robust and predictable, and you begin asking questions like “What configuration is being loaded?”, “Would it load log4j.properties, log4j.xml, or something else?”, “At what point does log4j2 shut itself down?”, “How do I ensure all asynchronous logging is written out before my application stops?”. At that point, the ivory tower begins to crack and tilt.

You can never be sure at which point would your log4j2 initialise and what would be its configuration, especially in larger systems. And shutting down log4j2 cleanly and reliably has been a big pain for a while. But these things are foundations, and they were apparently sacrificed in favour of the mythical “ease of use”.

What does it have to do with Log4Shell? A lot, actually. If it was not for a desire to make it easy for users, why would anyone add those weird features to log4j2 in the first place? What was the benefit of it? Or would you say those features were added because someone had absolutely nothing else to do?

Log4j2 Rocks

Despite everything I’ve mentioned above, log4j2 is relatively good software. Of course, it is not without flaws. However, at least those are its flaws.

The vast majority of other software pulls tens or hundreds of dependencies without which it cannot work. A good example of that is Apache Hadoop libraries. OMG. Just look at that beauty:

   org.apache.hadoop:hadoop-core:jar:1.2.1:compile
   +- commons-cli:commons-cli:jar:1.2:compile
   +- xmlenc:xmlenc:jar:0.52:compile
   +- com.sun.jersey:jersey-core:jar:1.8:compile
   +- com.sun.jersey:jersey-json:jar:1.8:compile
   |  +- org.codehaus.jettison:jettison:jar:1.1:compile
   |  |  \- stax:stax-api:jar:1.0.1:compile
   |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
   |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
   |  |     +- javax.xml.stream:stax-api:jar:1.0-2:compile
   |  |     \- javax.activation:activation:jar:1.1:compile
   |  +- org.codehaus.jackson:jackson-core-asl:jar:1.7.1:compile
   |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.7.1:compile
   |  \- org.codehaus.jackson:jackson-xc:jar:1.7.1:compile
   +- com.sun.jersey:jersey-server:jar:1.8:compile
   |  \- asm:asm:jar:3.1:compile
   +- commons-io:commons-io:jar:2.1:compile
   +- commons-httpclient:commons-httpclient:jar:3.0.1:compile
   |  +- junit:junit:jar:3.8.1:compile
   |  \- commons-logging:commons-logging:jar:1.0.3:compile
   +- commons-codec:commons-codec:jar:1.4:compile
   +- org.apache.commons:commons-math:jar:2.1:compile
   +- commons-configuration:commons-configuration:jar:1.6:compile
   |  +- commons-collections:commons-collections:jar:3.2.1:compile
   |  +- commons-lang:commons-lang:jar:2.4:compile
   |  +- commons-digester:commons-digester:jar:1.8:compile
   |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
   |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
   +- commons-net:commons-net:jar:1.4.1:compile
   +- org.mortbay.jetty:jetty:jar:6.1.26:compile
   |  \- org.mortbay.jetty:servlet-api:jar:2.5-20081211:compile
   +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
   +- tomcat:jasper-runtime:jar:5.5.12:compile
   +- tomcat:jasper-compiler:jar:5.5.12:compile
   +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:compile
   |  \- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:compile
   +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:compile
   |  \- ant:ant:jar:1.6.5:compile
   +- commons-el:commons-el:jar:1.0:compile
   +- net.java.dev.jets3t:jets3t:jar:0.6.1:compile
   +- hsqldb:hsqldb:jar:1.8.0.10:compile
   +- oro:oro:jar:2.0.8:compile
   +- org.eclipse.jdt:core:jar:3.1.1:compile
   \- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile

And this is only the “core”. In order to connect to Hadoop, you need “hadoop-client” which has many more dependencies.

Each dependency is a library which may have its vulnerabilities. If you do not know about them, that is because nobody cared enough to look and notice them. From a security point of view, this is nasty.

It is also nasty from an engineering point of view. In a large software, you are likely to end up with diamond dependencies and other joys, collectively known as “dependency hell”.

So while I am critical of log4j2, I admit that at least it does not pull unnecessary dependencies. Kudos to its authors:

   org.apache.logging.log4j:log4j-core:jar:2.14.1:compile
   \- org.apache.logging.log4j:log4j-api:jar:2.14.1:compile

Practical Implications

Let’s say your organisation really does not want to get hacked.

Authentication and authorisation are well-designed and implemented, firewalls are placed in all sensible places and rigorously control the traffic, access to Internet is done via proxies that meticulously analyze the traffic, patches are promptly applied for both software and firmware, all in-house software is developed rigorously (code reviews, static code analysis, quarantine for known bad artefacts), physical security is top-notch, CCTV cameras are everywhere and they actually work, and all of this is governed according to a set of well-designed policies.

And then comes a day when a popular logging/network/dependency injection library is hacked, and your walled garden is torched from within by a ten-year-old script kiddie.

The consequences would depend on the kind of organisation and the damage caused by the breach. In a good scenario, you will pay a fine, possibly lose some clients, and have some reputation impact. In a bad scenario, some of your people may die.

Was it your fault that you used the software everyone was happily using? Perhaps yes, but you cannot write every single bit of code yourself. Even if you do, it is possible that your alternative is going to be even more buggy.

You cannot perfectly protect yourself from the risk of your software getting poisoned by third-party libraries, and you cannot avoid vulnerabilities in third-party applications.

But what can you do?

There are a few things which can minimise the risk (although never get rid of it entirely). Each of them has its cost. So one needs to find a suitable balance between those immediate costs and the costs of the potential breach.

Here is a list of generic measures:

1. Get into a habit that absolutely every single bit of infrastructure (libraries, applications, operating systems, virtual machines, physical devices, networks) can be breached and subverted by an attacker. This applies even to the innermost systems of your organisation.
2. Assume that air gap is not an absolute protection (although it is valuable). For the details on how bad it can be, feel free to re-read the classic story of Stuxnet having lunch at Iran’s Natanz nuclear fuel enrichment plant.
3. Remembering the above, design your systems in a way which minimises the probability that a compromised system may attack other systems nearby or sniff around for information and leak it. I know, this is easier said than done.
4. Where possible, segregate observation and control. If possible, do it through physically different means. You need to see what’s going on but there is a value in ensuring that observers cannot hack into observed systems if they themselves get hacked.
5. Remember: as far as security is concerned, paranoia is a virtue.

These measures have their cost, of course. For example, if you do not trust your text editor then you need to ensure that it is allowed to do only the things that it really needs. Are you going to prevent it from opening network connections? How? What if it actually needs to do it because it needs to work with documents on a shared drive? If you allow that, would you also allow for other kinds of network access? And so on, and so forth.

Now let’s consider the matter of in-house software. Here, you may have a bit more visibility and control. However, almost all such software uses various third-party libraries, and those libraries can have vulnerabilities. How do you minimise the risk?

Here are a few suggestions:

1. If possible, avoid dependencies altogether. You do not need fancy libraries for trivial things like padding a text string. More on that here.
2. If you have to use a library, prefer those with few or no dependencies and those which aim to do little but do it well.
3. In case you cannot avoid a nasty big library with lots of dependencies, consider that your application may be unsafe due to its nature and segregate it thoroughly from the rest of the systems.
4. Educate your programmers. They almost never need frameworks, least of all for trivialities like wiring components together and starting them. Not only frameworks are largely useless, but they also bring lots of worthless dependencies and reduce the overall quality of code, turning it into an unmanageable mess. That is not good news for security.
5. Religiously avoid complexity. Everything must be made as simple as possible, but not simpler.
6. Find a way to keep track of which libraries your software depends on, have a centralised inventory of your software and match its dependencies against the known vulnerabilities. If there is one - act immediately. Bolster the defence by ensuring that only the software that is built according to vetted procedures and has the correct list of dependencies is deployed, and by running scanners which are on the lookout for vulnerable or unidentified software.

These measures will not be enough to protect you with certainty, but they should reduce the likelihood of your organisation suffering a breach.

Conclusion

Log4Shell has demonstrated that even the most popular and familiar Open Source libraries can be dangerous. For some, it has dispelled the myth that Open Source is safe because “many people are looking at the code”. For those who can see, it has also highlighted the value of simplicity and the dangers of depending on third-party code.

Unfortunately, situations like Log4Shell will be happening more often as the complexity of software is increasing while the overall quality of the code drops.

For most people, this situation is the norm. When the worst happens, the firm pays the fine and carries on. However, organisations, where security does matter, are now in a tough spot.

The way I see it, the problem cannot be solved. However, it can be minimised by striving for keeping the systems as simple as possible, having their capabilities externally constrained, and ensuring a thorough oversight of their behaviour. These measures may, however, have a significant cost and go against the traditions of how your teams tend to operate.