Introduction

In April 2021 Juniper networks reported on a new Linux rootkit designed to steal SSH credentials from Linux servers.  A month later Netlab 360 published a deeper analysis of the same rootkit they named “Facefish”.  Both reports provide enough information to implement a network-based detector with Zeek.  We will be implementing our detection algorithm through Zeek’s Spicy protocol analyzers (https://github.com/zeek/spicy-analyzers), so we recommend becoming familiar with and installing Spicy if you have not done so already.  Some prior Spicy documentation written by this author can be found at:

• https://zeek.org/2021/04/08/a-zeek-openvpn-protocol-analyzer-in-spicy/
• https://zeek.org/2021/04/20/zeeks-ipsec-protocol-analyzer/

The Facefish C2 Protocol

Starting with “0x2: Introduction to C2 commands” in the Netlab 360 report explains the rootkit’s C2 network activity.  All C2 communication occurs over a raw TCP connection with the following message structure:

 struct facefish {
      WORD payload_len;  // payload length
      WORD cmd;          // command
      DWORD payload_crc; // payload crc      
      unsigned char payload[payload_len]; // payload
}

This is a structure we can easily represent in Spicy.  The commands (“cmd”) outlined in the report are:

• 0x200 (512) – Key exchange #1.
• 0x201 (513) – Key exchange #2.
• 0x203 (514) – Key exchange #3.
• 0x300 (768) – Report stolen information.
• 0x301 (769) – Collect information and wait for further instructions.
• 0x302 (770) – Reverse shell.
• 0x305 (773) – Send registration information.
• 0x310 (784) – Execute a system command.
• 0x311 (785) – Return result of bash execution.
• 0x312 (786) – Re-collect and report host information.

They also state that the first message will be from the rootkit to the C2 server with a command of 0x200.  This rootkit uses little endian byte layout, so that makes the first eight bytes:

00 00 00 02 00 00 00 00

We use this message to trigger our protocol analyzer through dynamic protocol detection (DPD) signatures.  In the following sections we sketch how the Spicy code comes together to detect this rootkit.

A Testing PCAP

A public PCAP of the Facefish rootkit did not exist at the time we wrote this Spicy analyzer.  In order to test our Spicy code we needed a PCAP with data that looks like Facefish traffic.  We created a PCAP with some simple command line utilities such as netcat, tcpdump, and echo.  First we set up a netcat listener with this command:

$ nc -l -p 9999

Next we open another terminal and start tcpdump:

# tcpdump -i en0 -s 0 -w facefish.pcap port 9999

Then, we send our generated key exchange message across the network in a third terminal:

echo -n -e \\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00 | nc 127.0.0.1 9999

Now if we open our facefish.pcap, we see our generated key exchange message:

This PCAP can be found at:

https://github.com/zeek/spicy-analyzers/blob/a3a5c11dba6076d454df09dffdf70c39502abfb1/tests/Traces/facefish_rootkit_generated.pcap

The Spicy Source Code

You can find the source code in the following directory:

https://github.com/zeek/spicy-analyzers/tree/a3a5c11dba6076d454df09dffdf70c39502abfb1/analyzer/protocol/facefish_rootkit

We begin by looking at the “dpd.sig” file:

This signature will look for the 0x200 command we described in the prior section and attach the Spicy “Facefish_Rootkit” protocol analyzer to the connection.  This protocol analyzer is defined in lines 3 and 4 from the “facefish_rootkit.evt” file:

Line 9 says any time a “FacefishRecord” is successfully parsed, generate the Zeek event “Facefish_Rootkit::facefish_rootkit_message”.  More on that record in the code listing below.  Line 4 above says that we will parse the TCP connection using “FacefishRecords”, which can be found in the file “facefish_rootkit.spicy” in lines 9-11 below:

Returning to the FacefishRecord, you can find it in lines 13-18 above.  This is the structure discussed in the Netlab 360 blog that begins with a payload_len, followed by a command, then a CRC32 of the payload, finished with the payload.  We place only two parsing constraints: 1) a command value must be between 0x200 and 0x312 for a successful parse, and 2) the payload must be payload_len bytes long.  You will note that the first key exchange packet (command 0x200) has a payload_len of zero, so the payload does not exist in that case.

The file “facefish_rootkit_zeek.spicy” tells Zeek to confirm the protocol on a successful parse andreject it on an error, and the file provides a function named “create_facefishmsg” we used in line 9 of “facefish_rootkit.evt”:

Lastly, we look at the “main.zeek” file’s export section:

Line 4 sets up the new “facefish_rootkit.log” file and line 5 sets up the new Notice type.  Lines 7-16 define the information inside a Facefish C2 message, while lines 18-33 set up the log output format.  Line 40 declares the facefish_rootkit_message event while line 44 declares the log event one can use to handle an output record before it is written.

Lines 47-65 of “main.zeek” create the log stream and log any Facefish messages to “facefish_rootkit.log”:

The file “consts.zeek” provides a lookup table to translate known numerical commands to strings, or to “unknown-command-#” with an unknown command number if it is not in the lookup table.  These mappings match the bullets listed in this blog previously:

Once spicy-analyzers is installed, you can see our new protocol analyzer with the following command:

$ zeek -NN _Zeek::Spicy

_Zeek::Spicy – Support for Spicy parsers (*.spicy, *.evt, *.hlto) (dynamic, version 1.1.1)

    [Analyzer] spicy_DHCP (ANALYZER_SPICY_DHCP, enabled)

    [Analyzer] spicy_DNS (ANALYZER_SPICY_DNS, enabled)

    [Analyzer] spicy_Facefish_Rootkit (ANALYZER_SPICY_FACEFISH_ROOTKIT, enabled)

    [Analyzer] spicy_HTTP (ANALYZER_SPICY_HTTP, enabled)

    [Analyzer] spicy_LDAP_TCP (ANALYZER_SPICY_LDAP_TCP, enabled)

    [Analyzer] spicy_OpenVPN_TCP (ANALYZER_SPICY_OPENVPN_TCP, enabled)

    [Analyzer] spicy_OpenVPN_TCP_HMAC (ANALYZER_SPICY_OPENVPN_TCP_HMAC, enabled)

    [Analyzer] spicy_OpenVPN_UDP (ANALYZER_SPICY_OPENVPN_UDP, enabled)

    [Analyzer] spicy_OpenVPN_UDP_HMAC (ANALYZER_SPICY_OPENVPN_UDP_HMAC, enabled)

    [File Analyzer] spicy_PE (ANALYZER_SPICY_PE)

    [File Analyzer] spicy_PNG (ANALYZER_SPICY_PNG)

    [Analyzer] spicy_TFTP (ANALYZER_SPICY_TFTP, enabled)

    [Analyzer] spicy_Wireguard (ANALYZER_SPICY_WIREGUARD, enabled)

    [File Analyzer] spicy_ZIP (ANALYZER_SPICY_ZIP)

    [Analyzer] spicy_ipsec_ike_udp (ANALYZER_SPICY_IPSEC_IKE_UDP, enabled)

    [Analyzer] spicy_ipsec_tcp (ANALYZER_SPICY_IPSEC_TCP, enabled)

    [Analyzer] spicy_ipsec_udp (ANALYZER_SPICY_IPSEC_UDP, enabled)

…

If we go back to the test PCAP we created earlier, we can run our new logic on it with the following command:

$ zeek -Cr facefish.pcap spicy-analyzers

$ cat conn.log 
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path conn
#open 2021-06-03-14-56-10#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents
#types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string]
1622568704.959537 Cl0TA53874aF7eiRL3 127.0.0.1 56740 127.0.0.1 9999 tcp spicy_facefish_rootkit 0.000183 8 0 SF - - 0 ShAaDFf 5 280 5 272 -
#close 2021-06-03-14-56-10

$ cat facefish_rootkit.log 
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path facefish_rootkit
#open 2021-06-03-14-56-10
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p is_orig payload_len command crc32_payload
#types time string addr port addr port bool count string count
1622568704.959658 Cl0TA53874aF7eiRL3 127.0.0.1 56740 127.0.0.1 9999 T 0 KeyEx1 0
#close 2021-06-03-14-56-10

You can see that the “facefish_rootkit.log” contains the one message in our testing PCAP with the command of 512 “KeyEx1”, as expected.  We generated a Notice as well:

$ cat notice.log 
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path notice
#open 2021-06-03-14-56-10
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p fuid file_mime_type file_desc proto note msg sub src dst p n peer_descr actions suppress_for remote_location.country_code remote_location.region remote_location.city remote_location.latitude remote_location.longitude
#types time string addr port addr port string string string enum enum string string addr addr port count string set[enum] interval string string string double double
1622568704.959658 Cl0TA53874aF7eiRL3 127.0.0.1 56740 127.0.0.1 9999 - - - tcp Facefish_Rootkit::FACEFISH_ROOTKIT_C2 Potential Facefish rootkit C2 detected. More info: https://blog.netlab.360.com/ssh_stealer_facefish_en/ 127.0.0.1 127.0.0.1 9999 - - Notice::ACTION_LOG 60.000000 - - - -
#close 2021-06-03-14-56-10

A Real World PCAP

If you visit JoeSandbox at the following link, you can download a PCAP of Facefish C2:

https://www.joesandbox.com/analysis/355141/0/html#network

After opening this PCAP in Wireshark you can see the first key exchange message, just like our testing PCAP:

When we run this PCAP through Zeek, we see Facefish detected:

$ zeek -Cr dump-38fb322cc6d09a6ab85784ede56bc5a7.pcap spicy-analyzers

$ cat conn.log 
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path conn
#open 2021-06-03-14-53-30
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents
#types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string]
1613702572.079957 CWc0UD3fGLX0c6bj89 192.168.2.20 43448 176.111.174.26 443 tcp - 3.002074 0 0 S0 - - 0 S 3 180 0 0 -
1613702572.190619 CCo5Jp4eSxYMobOp3i 192.168.2.20 43450 176.111.174.26 443 tcp - 3.003396 0 0 S0 - - 0 S 3 180 0 0 -
1613702579.089432 CbPIFh4Olo8i1G0KV2 192.168.2.20 43448 176.111.174.26 443 tcp - - - - S0 - - 0 S 1 60 0 0 -
1613702579.201449 CPyuMa4IZGRzzBEOvh 192.168.2.20 43450 176.111.174.26 443 tcp - - - - S0 - - 0 S 1 60 0 0 -
1613702587.104275 C7nx2B1vfFr2VroO59 192.168.2.20 43448 176.111.174.26 443 tcp - - - - S0 - - 0 S 1 60 0 0 -
1613702603.150049 CUcCUW2x5WZS1yWyci 192.168.2.20 43448 176.111.174.26 443 tcp - - - - S0 - - 0 S 1 60 0 0 -
1613702635.209441 ChmbMq44oJeZIR6tVf 192.168.2.20 43448 176.111.174.26 443 tcp - - - - S0 - - 0 S 1 60 0 0 -
1613702587.216303 Clfyzh4ifzsqqUzvqc 192.168.2.20 43450 176.111.174.26 443 tcp spicy_facefish_rootkit 29.625531 4304 32 S1 - - 0 ShADTad 19 5348 12 688 -
#close 2021-06-03-14-53-30

$ cat facefish_rootkit.log 
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path facefish_rootkit
#open 2021-06-03-14-53-30
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p is_orig payload_len command crc32_payload
#types time string addr port addr port bool count string count
1613702587.313451 Clfyzh4ifzsqqUzvqc 192.168.2.20 43450 176.111.174.26 443 T 0 KeyEx1 0
1613702616.553325 Clfyzh4ifzsqqUzvqc 192.168.2.20 43450 176.111.174.26 443 F 24 KeyEx2 707025536
1613702616.557061 Clfyzh4ifzsqqUzvqc 192.168.2.20 43450 176.111.174.26 443 T 8 KeyEx3 2984358853
1613702616.746288 Clfyzh4ifzsqqUzvqc 192.168.2.20 43450 176.111.174.26 443 T 4272 Registration 2325026424
#close 2021-06-03-14-53-30

$ cat notice.log 
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path notice#open 2021-06-03-14-53-30
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p fuid file_mime_type file_desc proto note msg sub src dst p n peer_descr actions suppress_for remote_location.country_code remote_location.region remote_location.city remote_location.latitude remote_location.longitude
#types time string addr port addr port string string string enum enum string string addr addr port count string set[enum] interval string string string double double
1613702587.313451 Clfyzh4ifzsqqUzvqc 192.168.2.20 43450 176.111.174.26 443 - - - tcp Facefish_Rootkit::FACEFISH_ROOTKIT_C2 Potential Facefish rootkit C2 detected. More info: https://blog.netlab.360.com/ssh_stealer_facefish_en/ 192.168.2.20 176.111.174.26 443 - - Notice::ACTION_LOG 60.000000 - - - -
#close 2021-06-03-14-53-30

This confirms that the logic we developed in this blog works on real world Facefish C2 traffic!

Conclusion

This blog walked you through the creation of a Spicy protocol analyzer that detects the C2 traffic from the Facefish Linux rootkit.  We first created a testing PCAP to code against and followed with a code walkthrough.  When detected, the analyzer logs the Facefish messages to a new log called “facefish_rootkit.log” while generating a new Notice.  We concluded by confirming that the logic works on a real world Facefish sample found at JoeSandbox.  While this blog discussed building a detector for Facefish specifically, a similar approach can be used for other malware communication protocols.  This code can be used as a template for additional Spicy malware C2 protocol analyzers in the future.