How to quickly determine if server has been hacked?



  • The scenario: You SSH into a server you maintain. It might or might not have been hacked. You are on a schedule and need to quickly determine the status. What do you do?

    Usually, I start with htop and iftop. If there's nothing eating up CPU or sending out suspicious requests, I'm pretty much done.

    I realize this isn't enough. Thus this thread. What do you do? Check dmesg? History? CRON? What's the most worthy of effort? Any custom scripts you might run?

    Consider the server doesn't have any of the usual hacking targets (apache, PHP, wordpress). I'm just interested in the machine itself.



  • Recent FTP logs to mitigate the unlikely, but largest risk, access point.



  • Hmm, I do have a few FTP enabled servers, but mostly I use just the SSH.



  • I don't know how effective they are, but there's chkrootkit and rkhunter. Both run relatively quickly (a few seconds, and ~2 min on my system, respectively), so it's a fairly low effort.

    I also know of AIDE. From what I understand, it basically stores information (hash etc etc) about selected files when first initialized. Later, you can check if any of those files were changed. It's a bit more effort to use (i.e., need to update the database whenever you run system updates or change things), and you're supposed to store its database elsewhere for obvious reasons.

    How effective any of this is in the end ... I don't know.


  • Discourse touched me in a no-no place

    My first steps would be pre-emptive - getting used to seeing what the servers are like when they're 'normal'. For example the visual fingerprint for each server when manually logging in:

    Slightly obfuscated:

    [pjh@lenovo Dropbox]$ ssh example.net
    Host key fingerprint is 27:bf:7b:9c:9c:ec:76:0c:2e:35:b9:0c:2a:ea:b8:71
    +--[ RSA 2048]----+
    |                 |
    |                 |
    |                 |
    |                 |
    |        S .  .   |
    |         +. =    |
    |  . E    ..O B   |
    |   +  . . ..% o  |
    |  ooo. .  o*..   |
    +-----------------+
    
    Last login: Wed Jul 30 16:41:51 2014 from 78.33.59.98
    [pjh@sofa ~]$ ssh laptop.work
    Host key fingerprint is f0:58:39:95:26:2d:6d:ab:81:de:07:43:75:9d:b5:8c
    +--[ RSA 2048]----+
    |         +o... o.|
    |        +o*.  = .|
    |      .o+= . E o |
    |      .=+..      |
    |     ...S=       |
    |      . o .      |
    |         .       |
    |                 |
    |                 |
    +-----------------+
    
    Last login: Thu Jul 31 01:07:37 2014 from lenovo.pjh.example.com
    Keychain: SSH_AGENT_PID is set, so running keychain to load keys.
    
     * keychain 2.7.1 ~ http://www.funtoo.org
     * Found existing ssh-agent: 4100
     * Found existing gpg-agent: 4125
     * Known ssh key: /home/pjh/.ssh/id_rsa_ccu
     * Known ssh key: /home/pjh/.ssh/4096.pjh_rsa
     * Known ssh key: /home/pjh/.ssh/pjh_rsa_private
    
    [pjh@lenovo ~]$ 
    

    All I've done for this is to put:

    host *
       VisualHostKey yes
    

    in ~/.ssh/config.

    Next is install an IDS. I've got OSSEC on my servers. Most of the mails I get from it are expected (when I've updated software, or restarted the server,) or legitimate warnings about hacking attempts.


  • :belt_onion:

    @cartman82 said:

    I realize this isn't enough. Thus this thread. What do you do? Check dmesg? History? CRON? What's the most worthy of effort? Any custom scripts you might run?

    ps -ef and look for unexpected thingers?



  • Use ossec. Before you get hacked. And maybe rkhunter. I don't have any strong feelings about rkhunter, though.



  • @cvi said:

    I don't know how effective they are, but there's chkrootkit and rkhunter. Both run relatively quickly (a few seconds, and ~2 min on my system, respectively), so it's a fairly low effort.

    Tried chkrootkit. It's quick, easy to use, can be installed by simple yum / apt-get install... Exactly what I need. I'm including this in my install scripts and procedures.

    @cvi said:

    I also know of AIDE. From what I understand, it basically stores information (hash etc etc) about selected files when first initialized. Later, you can check if any of those files were changed. It's a bit more effort to use (i.e., need to update the database whenever you run system updates or change things), and you're supposed to store its database elsewhere for obvious reasons.

    Hmm, seems like a stripped down version of ossec. See below.

    @PJH said:

    For example the visual fingerprint for each server when manually logging in:

    Interesting idea. Unfortunately,

    1. It is difficult to implement when you have dozens of servers, with new ones constantly being added, old ones removed.
    2. As far as I know, this image is based on private SSH key. This can save you from man in the middle stuff, but not from someone installing a botnet on your server.

    But combined with...

    @darkmatter said:

    ps -ef and look for unexpected thingers?

    ... it occurs to me I could create some kind of ps 'fingerprint'. So I can filter out the stuff that's normally there and be able to see just the unusual processes. Instead of changing my current approach it will merely make it easier, which is an attractive proposition, psychologically.

    Not sure how useful that would be, but now I'm itching to give it a try 😄

    @Captain said:

    Use ossec. Before you get hacked. And maybe rkhunter. I don't have any strong feelings about rkhunter, though.

    @PJH said:

    Next is install an IDS. I've got OSSEC on my servers. Most of the mails I get from it are expected (when I've updated software, or restarted the server,) or legitimate warnings about hacking attempts.

    OK, two recommendations for ossec. I guess that must be the least ghetto way of doing this.

    Installation seems simple enough. Unfortunately, it seems to really get the benefit out of it, you must carefully configure it to slurp up all the relevant logs, OS modules, etc... Could be problematic to setup and maintain in my use case.

    This one goes into the longer term infrastructure todo list.



  • Actually, ossec is a little bit ghetto. It doesn't have a Debian installer, for example, because they (ossec) refuse to use the Linux Standard Base hierarchy. I think the only option to install is to build it yourself. But it's high quality and well thought out, despite its minor annoyance.

    The default configuration works fine. Make sure that you put it on a clean, production ready machine, and it will do hashes of all the files in the important directories (including /etc, /usr, and maybe even /var) It will report changes by email or to an ossec server (though I've never set that up).

    It also does other things, like putting up the firewall if you fail to log in to ssh 5 times, or ping the machine too much. I remember that being a big pain in my ass when I had ossec, plex (the movie server), and a dev environment on the same machine. Mistype a password and you can't even watch videos until the lockout ends. 😄



  • @Captain said:

    Actually, ossec is a little bit ghetto. It doesn't have a Debian installer, for example, because they (ossec) refuse to use the Linux Standard Base hierarchy. I think the only option to install is to build it yourself. But it's high quality and well thought out, despite its minor annoyance.

    Most of the crappy VPS-es I'm concerned with getting hacked are Centos, so that's ok :-)

    @Captain said:

    The default configuration works fine. Make sure that you put it on a clean, production ready machine, and it will do hashes of all the files in the important directories (including /etc, /usr, and maybe even /var) It will report changes by email or to an ossec server (though I've never set that up).

    OK, I can try the default config. Can't hurt to set it up.

    @Captain said:

    It also does other things, like putting up the firewall if you fail to log in to ssh 5 times, or ping the machine too much. I remember that being a big pain in my ass when I had ossec, plex (the movie server), and a dev environment on the same machine. Mistype a password and you can't even watch videos until the lockout ends.

    Ugh. Since these servers are remote, sounds like cure that is worse than the disease.


  • Discourse touched me in a no-no place

    @cartman82 said:

    @PJH said:
    For example the visual fingerprint for each server when manually logging in:

    Interesting idea. Unfortunately,

    1. It is difficult to implement when you have dozens of servers, with new ones constantly being added, old ones removed.
    2. As far as I know, this image is based on private SSH key. This can save you from man in the middle stuff, but not from someone installing a botnet on your server.
    1. The generation of the image is a client-side configuration option, it only needs to be done on the computers you're logging in from.
    2. So you're not concerned about MitM? ;)

    @cartman82 said:

    Installation seems simple enough. Unfortunately, it seems to really get the benefit out of it, you must carefully configure it to slurp up all the relevant logs, OS modules, etc... Could be problematic to setup and maintain in my use case.

    Have a central server (use one of your longer lived servers, or use a dedicated one), set up all other servers as clients and have a 'default client install' to use for new servers - it can be made less onerous than you seem to think it is.

    All configuration (after the installation of the client) is then centralised on one server and is effectively 'pushed' out to all current (and future) clients.

    @Captain said:

    Mistype a password and you can't even watch videos until the lockout ends.

    I got bitten by that (well not passwords but it was an automated script that did it) early on. Hence:

    [root@s15474768 ~]# grep white -B1 -A1 /var/ossec/etc/ossec.conf
      <global>
    [...]
        <white_list>(My work IP address)/32</white_list>
        <white_list>(My home IP address when I last remembered to update it)/32</white_list>
      </global>
    [root@s15474768 ~]#
    


  • @PJH said:

    1) The generation of the image is a client-side configuration option, it only needs to be done on the computers you're logging in from.

    The issue is, how does seeing that image helps me? I'm managing dozens of constantly changing servers. It's not like I will remember the image for each one. Also, in my experience, ssh will not let me connect if the host signature changes anyways.

    Actually, I never understood the point of that picture anyway. I always thought it was some inside linuxy joke, like cowsay or something. Since I acknowledge I'm dunce with security, I'm probably missing something...

    @PJH said:

    So you're not concerned about MitM?

    Honestly? No. We are not dealing with money or private data. There's no data they can steal. All we have that's worth taking are network and cpu resources.

    @PJH said:

    Have a central server (use one of your longer lived servers, or use a dedicated one), set up all other servers as clients and have a 'default client install' to use for new servers - it can be made less onerous than you seem to think it is.

    All configuration (after the installation of the client) is then centralised on one server and is effectively 'pushed' out to all current (and future) clients.

    Problem is, not all clients are the same. And I don't control how they are configured. Using default config is more realistic for now.

    Central, non-changing server, I can do. If I did that, however, I will rather try to integrate it into the central thingy we already have. Put it in a separate web interface, and I'll never look at it again.


  • Discourse touched me in a no-no place

    @cartman82 said:

    I'm managing dozens of constantly changing servers.

    .. which, I may remind you, wasn't in the original spec ;)

    It is, as you've realised, more useful for more static servers, and it can be turned on/off selectively in ~/.ssh/config for those instances where it would be useful.

    @cartman82 said:

    Actually, I never understood the point of that picture anyway. I always thought it was some inside linuxy joke, like cowsay or something. Since I acknowledge I'm dunce with security, I'm probably missing something...

    If you're interested, there's a short paper on it: "The Drunken Bishop" 21 pages (18 if you ignore the front page, index and appendix)



  • FWIW, I ended up playing around with AIDE a bit (I've been setting up a VPS for private use for the last few days, so I'd been looking at it anyway). AIDE is included in my distro's package manager, so installing it was not a problem.

    There's a few different ways of checking files (i.e., hashes, size, permissions, owners, but also stuff like inode number to see if the file was moved/deleted+recreated, and it can be told to remember file size and complain if the new size is smaller than a previous one). By default, the config defines a couple of different combinations of these, appropriate for e.g. binaries, config files and log files. It then specifies a couple of default locations to scan (/lib, /bin, /sbin, same in /usr etc etc).

    I had to change some of the config for my setup (e.g., check /lib32 and /lib64 rather than /lib, and disable the log-file checks, because they were not really compatible with metalog's way of doing stuff). But it was quite straight-forward.

    Running aide --check takes around 2.5min on my system. Updating the DB a bit more. Output from the checks is concise enough to be useful. All in all, I might end up using it more regularly.

    (My only potential complaint is that I have to remember that updating the DB creates a aide.db.new which you have to manually replace the old DB with.)



  • @cartman82 said:

    It might or might not have been hacked. You are on a schedule and need to quickly determine the status. What do you do?

    Easy. You give me the server's ip address and root credentials and I determine its status for you.



  • @presidentsdaughter said:

    Easy. You give me the server's ip address and root credentials and I determine its status for you.

    You mean taking it from possibly hacked to definitely hacked? I love that certainty.

    Here's mine: 127.0.0.1, user/hunter2.


  • Discourse touched me in a no-no place

    @presidentsdaughter said:

    Easy. You give me the server's ip address and root credentials and I determine its status for you.

    Kevin Quinn of Zamfoo - is that yooou?



  • Bit coin farmer?



  • @cvi said:

    FWIW, I ended up playing around with AIDE a bit (I've been setting up a VPS for private use for the last few days, so I'd been looking at it anyway). AIDE is included in my distro's package manager, so installing it was not a problem.

    There's a few different ways of checking files (i.e., hashes, size, permissions, owners, but also stuff like inode number to see if the file was moved/deleted+recreated, and it can be told to remember file size and complain if the new size is smaller than a previous one). By default, the config defines a couple of different combinations of these, appropriate for e.g. binaries, config files and log files. It then specifies a couple of default locations to scan (/lib, /bin, /sbin, same in /usr etc etc).

    I had to change some of the config for my setup (e.g., check /lib32 and /lib64 rather than /lib, and disable the log-file checks, because they were not really compatible with metalog's way of doing stuff). But it was quite straight-forward.

    Running aide --check takes around 2.5min on my system. Updating the DB a bit more. Output from the checks is concise enough to be useful. All in all, I might end up using it more regularly.

    OK, that doesn't sound too bad. I'll try it out.

    @presidentsdaughter said:

    Easy. You give me the server's ip address and root credentials and I determine its status for you.

    No way I'm falling for that again. Fool me once, twice or trice, shame on you. Fool me four times, shame on me.

    @Matches said:

    Bit coin farmer?

    Bit coins are so 2013, get with the times granpa.



  • Don't make me take your computing power away.



  • Bit coins are so 2013, get with the times granpa.

    Paranoid thought of the hour:

    Bitcoin is a Google conspiracy to buy up cheap computing power once the bubble bursts.

    Of course, it doesn't have to be a conspiracy for them to benefit from the strategy.



  • @Captain said:

    Paranoid thought of the hour:

    Bitcoin is a Google conspiracy to buy up cheap computing power once the bubble bursts.

    Of course, it doesn't have to be a conspiracy for them to benefit from the strategy.

    Never heard of that one. Me likey.


Log in to reply