API tools faq
paste
Advertisement
BaSs_HaXoR

How To Identify The Bad Processes On A Hacked Linux Box

BaSs_HaXoR
Nov 21st, 2016
252
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 19.32 KB | None | 0 0
raw download clone embed print report
  1. ### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
  2. How To Identify The Bad Processes On A Hacked Linux Box
  3. Mattias Geniar, Friday, August 8, 2008 - last modified: Saturday, August 8, 2015
  4. ### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
  5.  
  6. Let's say you have a hacked Linux server. Let's say you want to find out who did it, as well as what the server is doing now. It's not always easy, but this guide will show you how to use tools like netstat and lsof to find which processes are running on which ports.
  7.  
  8. First of all, if you still have SSH access to the server, log in through a remote session. Otherwise, you'll need physical access to the server.
  9.  
  10. Most hacked servers have only 2 primary functions: try to hack other servers, in order to create a bigger botnet, or to wreck havoc on an unsuspecting server. They do have one thing in common: several outbound connections are made to other servers (portscan, irc-bot, ftp server, ...). We'll identify the ports used for those outbound connections, and the user/process that is executing them. Then we'll locate the files for those processes, and disable them (after analyzing them).
  11.  
  12. Note; this article turned out a bit larger than I had thought, so prepare for a wall of text, with the occasional comic -- to keep you entertained ;-)
  13.  
  14. Holy jokers Batman, how do we do that?!
  15. So at first, a little warning. If you're trying to fix a hacked server, it's very possible that the binaries you're executing (such as ps, ls, lsof, netstat, ...) have been replaced by hacked versions, meaning they won't show you the full information.
  16.  
  17. Be cautious with trusting the information they show you, and try using the BackTrack live CD (if you have physical access to do so) to perform your analysis.
  18.  
  19. First we need to find out what the server is doing. Seeing as a hacked server is pretty useless if it isn't making contact with other servers, we can do this by checking all our incoming/outgoing connections. This can help us find out which ports are being used, and what the server's doing.
  20.  
  21. ### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
  22. To get a list of all current active connections, you can use the netstat command.
  23.  
  24. $ netstat -an
  25. With the -a parameter, we tell the command to show all connections (including the LISTENING ones), instead of only those who are connected.
  26.  
  27. The -n parameter shows the different port numbers used. By default, it would display items with a textual description (such as :pop3, :smtp, :http). It's more useful when we can see the actual ports, shown as :110, :25, :80.
  28.  
  29. This will generate a list, much like this one (if it's a standard webserver, it could differ for your server(s) ). I've shortened it a bit, it's usually a lot longer.
  30.  
  31. $ netstat -an
  32. Active Internet connections (servers and established)
  33. Proto Recv-Q Send-Q Local Address Foreign Address State
  34. tcp 0 0 0.0.0.0:110 0.0.0.0:* LISTEN
  35. tcp 0 0 0.0.0.0:143 0.0.0.0:* LISTEN
  36. tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN
  37. tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN
  38. tcp 0 0 :::80 :::* LISTEN
  39. tcp 0 0 :::22 :::* LISTEN
  40. tcp 0 0 :::443 :::* LISTEN
  41.  
  42. Active UNIX domain sockets (servers and established)
  43. Proto Flags Type State I-Node Path
  44. unix [ ACC ] STREAM LISTENING 548928452 /var/lib/mysql/mysql.sock
  45. unix [ ] DGRAM 548927555 @/org/kernel/udev/udevd
  46. unix [ ACC ] STREAM LISTENING 548928520 /tmp/spamd_full.sock
  47. unix [ ] DGRAM 548927910 /dev/log
  48. unix [ ACC ] STREAM LISTENING 548928786 /var/run/saslauthd/mux
  49. unix [ ] DGRAM 3285180947
  50. unix [ ] DGRAM 550618919
  51. unix [ ] DGRAM 550618862
  52. unix [ ] DGRAM 548928785
  53. unix [ ] DGRAM 548928764
  54. unix [ ] STREAM CONNECTED 548928535
  55. unix [ ] STREAM CONNECTED 548928534
  56. You'll probably notice that these connections all look harmless. There are processes listening on port 80 (HTTP), 110 (POP3), 21 (FTP), 25 (SMTP), 22 (SSH) and 443 (HTTPS). For this example, we'll assume we want to find out which process is listening on port 80.
  57.  
  58. Note; if your server is hacked and being used as an IRC Bot you'll see many connections in the range of 6660-7000. You could follow the same steps, by just replacing port 80 by port 6660. It's just easier to show it this way.
  59.  
  60. ### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
  61. XKCD - Hacked, SQL Injection
  62.  
  63. Now, to find out which user and which process is making connections on this port, we'll use the lsof (List Open Files) command.
  64.  
  65. $ lsof -i tcp:80 -P -R
  66. This command shows us all running processes that are using port 80 for any kind of communication.
  67.  
  68. The -i parameter specifies we want to list the processes, by identifying them with IPv4 or IPv6.
  69. The tcp:80 part means we only want to show TCP connections (and ignore UDP for the time being) using port 80.
  70. Using -P we specify we want to see the port numbers (80, 21, ...) instead of the names (HTTP, FTP) which are shown by default. Much like the -n parameter with netstat.
  71. With -R we also show the Parent Process ID, to see who initiated this process.
  72. It will show output like this.
  73.  
  74. $ lsof -i tcp:80 -P -R
  75. COMMAND PID PPID USER FD TYPE DEVICE SIZE NODE NAME
  76. httpd 13538 16324 apache 4u IPv6 548928212 TCP *:80 (LISTEN)
  77. httpd 14279 16324 apache 4u IPv6 548928212 TCP *:80 (LISTEN)
  78. httpd 16324 1 root 4u IPv6 548928212 TCP *:80 (LISTEN)
  79. httpd 28042 16324 apache 4u IPv6 548928212 TCP *:80 (LISTEN)
  80. httpd 29772 16324 apache 4u IPv6 548928212 TCP *:80 (LISTEN)
  81. httpd 32424 16324 apache 4u IPv6 548928212 TCP *:80 (LISTEN)
  82. Wow! Lots of numbers!
  83. Take a deep breath. It looks more complicated than it is, trust me. What can we learn from this? We see 6 instances of httpd, the HTTP daemon that is acting as the webserver. Five of these processes are run by the apache user, and one by root.
  84.  
  85. All apache httpd processes were started by a main process (PID 16324) which was executed by the root user (PPID column).
  86.  
  87. The column "command" displays the name of the process. Under "PID" we can see the Process ID, this can be used to easily stop, or kill the process. With "PPID" we can see the process ID that started this specific process.
  88.  
  89. Beneath "USER" we see the user that is executing the process. We can now see that all httpd processes that are being run as the apache user, were started by Parent Process ID 16324, the httpd process executed as root.
  90.  
  91. Note; you should be alerted when you see a postgres user running all sorts of programs on all sorts of exotic ports. If that's the case, the server's probably hacked, and used for means it wasn't supposed to.
  92.  
  93. Now you have a list of all the processes that are using a specific port to communicate. Let's find out which files these processes use, where they are located, and then stop the process -- to prevent further harm. We won't delete the files that were used just yet, because we might learn something afterwards when we examine them. We will, however, move them outside the user's directory.
  94.  
  95. $ lsof -p 16324
  96. With -p we can specify a Process ID (PID) to filter our results to. This'll only list open files for the PID 16324.
  97.  
  98. $ lsof -p 16324
  99. COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
  100. httpd 16324 root cwd DIR 0,61 4096 14624004 /
  101. httpd 16324 root rtd DIR 0,61 4096 14624004 /
  102. httpd 16324 root txt REG 0,61 312372 19768930 /usr/sbin/httpd
  103. httpd 16324 root mem REG 0,61 11704 20455174 /usr/lib/libgpg-error.so.0.3.0
  104. httpd 16324 root mem REG 0,61 346148 20455190 /usr/lib/libgcrypt.so.11.2.2
  105. httpd 16324 root mem REG 0,61 213116 20455786 /usr/lib/libxslt.so.1.1.17
  106. httpd 16324 root mem REG 0,61 67932 20455784 /usr/lib/libexslt.so.0.8.13
  107. httpd 16324 root mem REG 0,61 23172 20513414 /usr/lib/php/modules/xsl.so
  108. httpd 16324 root mem REG 0,61 32216 20513412 /usr/lib/php/modules/xmlwriter.so
  109. ....
  110. This list usually goes on for a while, as any given process could have hundreds of files open. However, this list will tell you where the culprit is located. In our case, the program started in /usr/sbin/httpd (see line 3).
  111.  
  112. Let's end the process using kill, remove the execute permissions, and move the files to another location.
  113.  
  114. $ kill 16324
  115. The kill command ends a process with a given PID, in this case 16324.
  116.  
  117. ### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
  118. $ chmod -x /usr/sbin/httpd
  119. The chmod command lets you edit the permissions to this file. We'll use it to remove any execution rights it might have "-x".
  120.  
  121. $ mkdir /home/tmp-sandbox/
  122. mv /usr/sbin/httpd /home/tmp-sandbox/
  123. We'll create the directory "/home/tmp-sandbox/" and copy the files there, so we can view them later without risk.
  124.  
  125. Ok Batman, I stopped the program. Can I go out and play now?
  126. Oh no, not yet.
  127.  
  128. You may have ended the program, but you still have to prevent if from being started again. First of all, check to see if the program was filed under /etc/init.d . Every command located in this directory will be executed upon boot -- and you don't want the process to restart if you have to reboot the server.
  129.  
  130. ### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
  131. Next, make sure it's not created a crontab entry to be executed every X-seconds.
  132.  
  133. $ crontab -l
  134. This will list all your cronjobs, for the current user -- probably root. If you see any command in there that is started from the location where you just found your culprit, remove it from the crontab.
  135.  
  136. $ crontab -e
  137. This lets you edit the cronjobs, just like a normal vi. Remove the line, and save the file (:wq). Also make sure to log in to the user that executed your bad process (su <user>) and check those cronjobs.
  138.  
  139. Next up, update all your software. Either via yum or apt-get, or manually -- just make sure it's all up to date. Also, avoid using tools such as Webmin to manage a server -- it's highly vulnerable to these kinds of attacks.
  140.  
  141. You might want to check your /etc/, /tmp/ and /home/ folders just in case, for any files that were left there by your hacker. Remember: it's not because you stopped the process, that you're safe.
  142.  
  143. Assume there are other files, still hidden, on your server. Find them, and disable them. Then study their contents, and learn from it.
  144.  
  145. If this article was in any way useful to you, please leave a message and state how it helped you. If you see errors, of have other tips -- don't hesitate to share!
  146.  
  147. Hi! My name is Mattias Geniar. I'm a Support Manager at Nucleus Hosting in Belgium, a general web geek, public speaker and podcaster. If you're interested in keeping up with me, have a look at my podcast and weekly newsletter below. For more updates, follow me on Twitter as @mattiasgeniar.
  148.  
  149. Enter your e-mail here
  150. SUBSCRIBE TO CRON.WEEKLY
  151. I respect your privacy and you won't get spam. Ever.
  152. Just a weekly newsletter about Linux and open source.
  153. SysCast podcast
  154. In the SysCast podcast I talk about Linux & open source projects, interview sysadmins or developers and discuss web-related technologies. A show by and for geeks!
  155. cron.weekly newsletter
  156. A weekly newsletter - delivered every Sunday - for Linux sysadmins and open source users. It helps keeps you informed about open source projects, Linux guides & tutorials and the latest news.
  157. Share this post
  158. Did you like this post? Will you help me share it on social media? Thanks!
  159.  
  160.  
  161. ### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
  162. ### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
  163. Comments
  164. Rafal Los Saturday, August 9, 2008 at 06:57
  165. You appear to assume that the box’s processes and files haven’t been replaced with trojaned versions to hide the original malware…
  166.  
  167. Reply ↓
  168. Mattias Geniar Saturday, August 9, 2008 at 10:48
  169. Well, this is indeed the simplified version. Most hacked servers are done using simple techniques, to do simple things, which can then simply be discovered.
  170.  
  171. Perhaps I’ll expand this to identify trojans and other malware too, for now it’ll just be this basic version.
  172.  
  173. Reply ↓
  174. Pete Monday, September 22, 2008 at 15:03
  175. This was a very helpful little tutorial on the practical use of lsof. Thanks for taking the time to write this!
  176.  
  177. –Pete
  178.  
  179. Reply ↓
  180. Tim Tuesday, April 7, 2009 at 16:36
  181. Great article! Very helpful. Much appreciated!
  182.  
  183. Reply ↓
  184. MikeT Thursday, May 6, 2010 at 19:03
  185. This really opened my eyes. Thanks for using your time to write this…
  186.  
  187. MikeT
  188.  
  189. Reply ↓
  190. Emre Sevinç Friday, May 21, 2010 at 12:57
  191. You said: “Be cautious with trusting the information they show you, and try using the BackTrack live CD (if you have physical access to do so) to perform your analysis.”
  192.  
  193. Booting the hacked server with a clean BackTrack Live CD and then analyzing the connections? How can that happen? I mean if you boot it with a clean, live CD then it is a clean, live server and no suspicious connections exist because you just booted it with a live CD. Your hacked server is not booted anymore. I’m a little bit confused.
  194.  
  195. Reply ↓
  196. Matti Sunday, May 23, 2010 at 10:06
  197. @Emre; it only goes back to the “If you’re trying to fix a hacked server, it’s very possible that the binaries you’re executing (such as ‘ps’, ‘ls’, lsof’, ‘netstat’, …) have been replaced by hacked versions”-line. Those binaries shouldn’t be trusted, and can be “replaced” with the live CD ones.
  198.  
  199. The open connections are indeed useless at that point.
  200.  
  201. Reply ↓
  202. zobin Tuesday, October 26, 2010 at 18:48
  203. Thank you for your article,and i found the spam process,but i don’t how to clear it in one time.
  204. The below is the list which i ran lsof -p 2628(the pid of the main spam process)
  205.  
  206. COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
  207. sendmail 2628 root cwd DIR 8,2 1699840 3899771 /var/spool/mqueue
  208. sendmail 2628 root rtd DIR 8,2 4096 2 /
  209. sendmail 2628 root txt REG 8,2 806460 6567022 /usr/sbin/sendmail.sendmail
  210. sendmail 2628 root mem REG 8,2 7733281 /lib/libnsl-2.5.so (path inode=7733286)
  211. sendmail 2628 root mem REG 8,2 7733338 /lib/libwrap.so.0.7.6 (path inode=7733452)
  212. sendmail 2628 root mem REG 8,2 6563225 /usr/lib/libhesiod.so.0.0.0 (path inode=6563806)
  213. sendmail 2628 root mem REG 8,2 7733446 /lib/libcom_err.so.2.1 (path inode=7734509)
  214. sendmail 2628 root mem REG 8,2 14848 6654557 /usr/lib/sasl2/libplain.so.2.0.22
  215. sendmail 2628 root mem REG 8,2 7733275 /lib/libcrypt-2.5.so (path inode=7734506)
  216. sendmail 2628 root DEL REG 8,2 6565112 /usr/lib/liblber-2.3.so.0.2.31.#prelink#.oxts6J
  217. sendmail 2628 root mem REG 8,2 6565055 /usr/lib/libkrb5.so.3.3 (path inode=6573533)
  218. sendmail 2628 root DEL REG 8,2 6565057 /usr/lib/libkrb5support.so.0.1.#prelink#.QePuRn
  219. sendmail 2628 root mem REG 8,2 6565045 /usr/lib/libk5crypto.so.3.1 (path inode=6573532)
  220. sendmail 2628 root mem REG 8,2 6561754 /usr/lib/libz.so.1.2.3 (path inode=6567559)
  221. sendmail 2628 root mem REG 8,2 7733295 /lib/libpthread-2.5.so (path inode=7733278)
  222. sendmail 2628 root mem REG 8,2 7733264 /lib/ld-2.5.so (path inode=7733265)
  223. sendmail 2628 root mem REG 8,2 7733339 /lib/libkeyutils-1.2.so (path inode=7733472)
  224. sendmail 2628 root mem REG 8,2 7733332 /lib/libsepol.so.1 (path inode=7733300)
  225. sendmail 2628 root DEL REG 8,2 7733467 /lib/libcrypto.so.0.9.8e.#prelink#.SBMg0S
  226. sendmail 2628 root mem REG 8,2 7733277 /lib/libdl-2.5.so (path inode=7733296)
  227. sendmail 2628 root DEL REG 8,2 6565114 /usr/lib/libldap-2.3.so.0.2.31.#prelink#.093P7r
  228. sendmail 2628 root mem REG 8,2 7733297 /lib/libresolv-2.5.so (path inode=7734508)
  229. sendmail 2628 root mem REG 8,2 7733469 /lib/libssl.so.0.9.8e (path inode=7734511)
  230. sendmail 2628 root mem REG 8,2 905200 6654584 /usr/lib/sasl2/libsasldb.so.2.0.22
  231. sendmail 2628 root mem REG 8,2 7733462 /lib/libdb-4.3.so (path inode=7733448)
  232. sendmail 2628 root mem REG 8,2 7733271 /lib/libc-2.5.so (path inode=7733272)
  233. sendmail 2628 root mem REG 8,2 50848 7733287 /lib/libnss_files-2.5.so
  234. sendmail 2628 root DEL REG 8,2 6565041 /usr/lib/libgssapi_krb5.so.2.2.#prelink#.GnGQSz
  235. sendmail 2628 root mem REG 8,2 7733442 /lib/libselinux.so.1 (path inode=7733304)
  236. sendmail 2628 root mem REG 8,2 14752 6654553 /usr/lib/sasl2/liblogin.so.2.0.22
  237. sendmail 2628 root mem REG 8,2 6562037 /usr/lib/libsasl2.so.2.0.22 (path inode=6563124)
  238. sendmail 2628 root mem REG 8,2 14372 6652653 /usr/lib/sasl2/libanonymous.so.2.0.22
  239. sendmail 2628 root 0r CHR 1,3 1290 /dev/null
  240. sendmail 2628 root 1w CHR 1,3 1290 /dev/null
  241. sendmail 2628 root 2w CHR 1,3 1290 /dev/null
  242. sendmail 2628 root 3u unix 0xc3616580 7786 socket
  243. sendmail 2628 root 4u IPv4 7787 TCP localhost.localdomain:smtp (LISTEN)
  244. sendmail 2628 root 5wW REG 8,2 33 3900362 /var/run/sendmail.pid
  245. Reply ↓
  246. Mattias Geniar Tuesday, October 26, 2010 at 23:25
  247. For starters, stop sendmail so it no longer sends out messages. Then run some kind of rootkit hunter, and parse your log messages.
  248. Sorry I can’t be more precise, but this differs from each server, and the way the server is being abused to send out spam. If possible, contact your hosting provider/sysadmin for more guidance.
  249.  
  250. Reply ↓
  251. julian Wednesday, February 9, 2011 at 23:43
  252. Hi Matti,
  253.  
  254. you said:
  255. @Emre; it only goes back to the “If you’re trying to fix a hacked server, it’s very possible that the binaries you’re executing (such as ‘ps’, ‘ls’, lsof’, ‘netstat’, …) have been replaced by hacked versions”-line. Those binaries shouldn’t be trusted, and can be “replaced” with the live CD ones.
  256.  
  257. I am ok with this idea if the replaced binaries are static but if they aren’t how could you trust the librairies used by those binaries ? I think you can’t turst a hacked server until you re-install it… you never now what they did with your server and how much this hacked machine could be corrupted.
  258.  
  259. Reply ↓
  260. Caiya Monday, November 14, 2011 at 16:16
  261. Wait, I cannot fathom it being so srtaigthforward.
  262. ### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
  263. ### ----------------- [###############] /---- \_+_/ -----\ [###############] ----------------- ###
  264. //SOURCE: https://ma.ttias.be/how-to-identify-the-bad-processes-on-a-hacked-linux-box/
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!