Root Compromise

N

Naushad

Member
Hi friends,
Being on KH is the only thing I feel comfortable. Few days back my VPS got hacked. most websites are still under the influence. The server is still root compromised. The suggestion I have got from support is to get another container, a fresh one, and migrate accounts on it. This according to them may solve issue of security.
But what I really don't get is when the accounts are compromised too, how will it be still safe?
The strange thing is that I have blocked SSH login but still successful attempts are made through WHM. Even I have got password changed from support and got IP banned. the password is only known to me and the support.
Any suggestions? I am not a Server Admin or a big network guy, just an ordinary man relying on my small business of selling hosting accounts and happy with the "managed vps" feature offered by KH.
Your supportive comments and suggestions are awaited.
 
How many accounts do you have?
How many have been affected?

I had a client get their WordPress site get hacked due to a bad plugin. I could not for the life of me find out how they kept getting in. They kept screwing with pages. I would change the account password, monitor inbound IPs and just could not find how they were getting in. Then, I found a strange page that was only occasionally being accessed and it was amazing what they did. They had a php script deep in WP that allowed them to execute commands and with it they pretty much had access to the entire account, and luckily only that account. After removing that page and removing the bad plugin, it's never been an issue since.

If your hacker managed to hack your WHM account then maybe switching VPS is a solution, so long as the problem doesn't get migrated with everything else.
 
There are many accounts on wordpress and joomla. I have been communicating with clients to update their plugin and the version of site software but seems they pay less heed. Now I am thinking to suspend the accounts that show outdated plugins and site software. Besides, I also found some websites having wordpress as site software are being used to send spam/huge number of mails. I did get their plugins blocked by changing permissions etc. but strange thing is that they were actually accessed from within the wordpress, which tells me that you are right, something is terribly wrong now.
So you suggest to change the vps but again how to avoid such malicious migration?
 
Any common denominators between the affected accounts?
All WordPress? All joomla? All have the same outdated plugins?
 
no. The KH support initially told me that the first access was made through WHMCS (most probably). There is some kind of symlink thing that was used. since all the server logs were deleted by the hacker it was difficult to see footprints. So no idea! currently server is acting strange. Sometimes it sends spam from one account and some times from some other account. Sometimes the hacker would only log in to see the server load and leave.
No idea what is going on.
 
Once they get into an account they will often put in a back door and hope that you don't find it. Sometimes it will simply be a file or two in the root of the domain and other times a deeply embedded plugin or file like phpAddict suggested. Sometimes they'll even take and modify multiple files. Do you have backups of your sites? If not then KnownHost should although I don't know how far back they go as you'll want to restore from a backup prior to when they got in. After restoring from the backup go through and change all passwords including root and for the accounts.

I strongly suggest using Lastpass and having it generate completely random passwords and, of course, using it to remember them as well. It has plugins for all common browsers and even your Apple or Android device for a small yearly fee. If you don't need it on your portable device it's free.
 
Well let's back up a little...
What is the hacker doing specifically? Just spamming? Modifying pages? How is it you know the hacker logs in to check the server load? If you server is being monitored to that extent, I'd assume there is IP logging does blocking his IP not work? Is the hacker logging in from multiple IPs?
 
Responding to Dan:
As soon as I found an unauthorized log in attempt, I informed it to Support. They looked after it well. They quickly restored the back up and changed root password. But since it was not possible for changing passwords to all accounts at once, what I did is increased the login security and quickly banned few IPs. But somehow since the root was already compromised I think they got in again.

Responding to phpAddict:
hacker is exactly doing the same you said. Monitors the server load and does spamming. Blocking the IP does not work too.

For your clarity, let me paste the support's response here when it was earlier done:

Mail Body:

I have done a fair bit of investigating and have determined your server have been compromised at the root level. There have been many system file changes as seen below:

---
cP/vz02-tx/12025 root@162.211.86.23 [/etc]# rpm -Va | grep "..[5?]"
S.5....T. c /etc/chkserv.d/clamd
S.5....T. c /usr/local/cpanel/3rdparty/etc/cpclamav.conf
S.5....T. /usr/include/linux/posix_types.h
SM5..U.T. c /etc/named.conf
S.5....T. /etc/init/rc.conf
S.5....T. /etc/init/rcS.conf
S.5....T. c /etc/rc.d/rc.local
S.5....T. c /etc/sysconfig/init
S.5....T. c /etc/sysctl.conf
S.5....T. c /etc/sysconfig/saslauthd
SM5....T. c /etc/cron.deny
S.5....T. c /etc/pam.d/sshd
S.5....T. c /etc/ssh/sshd_config
SM5....T. c /etc/pure-ftpd.conf
SM5....T. c /etc/mail/spamassassin/init.pre
SM5....T. c /etc/mail/spamassassin/local.cf
SM5....T. c /etc/mail/spamassassin/v310.pre
SM5....T. c /etc/mail/spamassassin/v320.pre
S.5....T. c /usr/local/cpanel/3rdparty/php/54/etc/pear.conf
S.5....T. c /usr/local/cpanel/3rdparty/php/54/etc/php.ini
S.5....T. c /usr/local/cpanel/3rdparty/php/54/lib/php/.depdb
S.5....T. c /usr/local/cpanel/3rdparty/php/54/lib/php/.filemap
S.5....T. /usr/include/bits/typesizes.h
S.5....T. c /etc/issue
S.5....T. c /etc/issue.net
SM5....T. c /usr/local/cpanel/base/horde/config/conf.php
S.5....T. c /etc/exim.pl
S.5....T. c /etc/eximrejects
S.5....T. c /etc/localaliases
SM5...GT. c /etc/localdomains
S.5....T. c /etc/dovecot/dovecot.conf
SM5....T. c /etc/bashrc
S.5....T. c /etc/hosts.allow
S.5....T. c /etc/hosts.deny
S.5....T. c /etc/profile
S.5....T. c /etc/rsyslog.conf
S.5....T. c /etc/yum/pluginconf.d/fastestmirror.conf
S.5....T. c /etc/yum.conf
S.5....T. c /usr/local/cpanel/3rdparty/etc/pango/pango.modules
---


Another indication would be the body of the /etc/issue.net file seen below:

---
cP/vz02-tx/12025 root@162.211.86.23 [/etc]# cat issue.net
Fuck You MothaFuka SpyDy Here
---


The changes trace back to May 5th, see below:

---
cP/vz02-tx/12025 root@162.211.86.23 [/etc]# stat issue.net
File: `issue.net'
Size: 29 Blocks: 8 IO Block: 4096 regular file
Device: dah/218d Inode: 64249037 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2014-04-27 09:46:30.507786000 -0400
Modify: 2014-05-05 12:53:42.753370374 -0400
Change: 2014-05-05 12:53:42.753370374 -0400
---


Unfortunately, tracing the source of this compromise is now impossible as they have made changes to cover the tracks. This also applies for any changes they did and wanted to hide.

We also do not have a internal backup that dates before May 5th so a server roll back is not a option.

At this point, the only way to guarantee your security would be to re provision the server. This means that a server needs to be bought so that we can migrate your cpanel accounts over.
 
Do it. It's unfortunate but you really have no choice. You need to discuss the fact that accounts are compromised with them too though and see if they have suggestions on that. This and the WHMCS hole both need to be addressed beforehand.
 
Thank you for the support, Dan. But what I really want to know is how? I mean how to make sure that the accounts being migrated have no back doors. I hope you understand, I can't even openly discuss this with clients that the server root is compromised, else I may loose huge business. Most of the accounts are dynamic running different applications like moodle, accounting software, hr software, etc. this will hurt their data if I restore the previous clean backup. There must be some way to ensure the accounts are clean before migration. Or lets say that only clean accounts get migrated and other remain on the old server till they either get cleaned or find some other solution for themselves.
 
Dito to Dan's post. You'll at least succeed in getting your WHM locked down again.
However, depending on what back doors the hacker has placed he may still have access to individual accounts after the migration which you'll either need to find and fix, or rebuild the account from scratch so you start fresh.
Use an extra strong password like Dan suggested. Mine for my WHM is 18 characters long and a mixture of upper, lower, numbers, and special chars. Check your password strength at https://howsecureismypassword.net
For my password: "It would take a desktop PC about 5 quintillion years to crack your password"
 
@phpAddict agreed! but one more clarity please. As I came to know that the possibility was entrance though WHMCS means using hash value. Any suggestions on what measures to take to secure server? i can get ssh block or allowed to only for the IPs of KH for their support tasks. i can also get whm entry a bit difficult by keeping questions at logging in. I can also keep an expiry on password that after every certain number of days the user has to change the cpanel password, etc. any other?
 
KH seems to change the SSH port by default which is a big help in preventing brute-force attacks, but locking it down by IP certainly isn't a bad idea. You can enforce password policies to make sure all your accounts have strong passwords, but in your case since it seems WHM was compromised I'd first recommend that you yourself have the strongest password possible.
 
@phpAddict I did get a strong password rest couple of times but I don't know how it was leaked again and again. Even when it was only reset by support and informed me. but I am sure since I have to keep my WHMCS connected with WHM it must have got somehow accessed through it. Now today I have got it changed again but this time i have not updated it in WHMCS so will monitor it for at lease one day to see if another attempt is made or not.
Besides, can you also tell me how to secure SQL? one of my clients has a medical reporting script. The script connects with sql from desktop using java and updates the database with reports etc. Then any of their patients can see their reports online. But since they add data from different computers they can't afford to have a static IP for every other PC. To solve this, i had to get host access allowed for All for SQL. any suggestions on it?
 
Only allowing certain IPs root access only works if you have a static IP number. I don't know where you live but my ISP (Comcast) charges an arm and a leg for a static so mine is dynamic. Additionally I'm not sure KH can even tell you all of the IP numbers they may access your VPS from.

I'm not sure what you mean by using a hash value for the WHMCS password. Normally a password being stored will be hashed (perhaps you mean that's an option in WHMCS) and if that's simply an option then yes it should always be enabled. But the fact of the matter is there was a hole in WHMCS that gave them access to that password and that hole needs to be plugged or nothing you do will make any difference.

Forcing people to change their passwords is just going to piss them off. If you require a pretty long and complex password from the get go I should think that should be good. Then CSF will block brute force attacks and block their IP numbers automatically.

You don't have to have KH change your password. You can do that yourself in WHM. At this point though I sincerely doubt changing it is doing you any good as they obviously have their own way in, almost seems like they have access to your email...

As far as the question about the doctor's report script and your having to allow connections to all you should never have agreed to that. There are SQL hacks and scripts that will seriously take advantage of that. Their software should be able to be username/password protected (secure passwords of course) and not simply protected by limiting access by IP. Either that or they pick one computer with a static IP and use it for uploads.
 
@Naushad
Really depends on the programming. At the very least I'd recommend changing your SQL user password and update your program with the new one.
I'd recommend you have 2 SQL user accounts. 1 with edit privileges for the java app that the medical company uses (your java app), and the other just for patients to only view (online reports), unless of course patients need to edit the data. That way if one becomes compromised you'll have a better idea where to look. The java app should be connecting securely (https, not http). But, again all depends on the programming. Each computer doesn't need it's own dedicated IP, just the site does. If there is only one site that does these updates then they should easily be able to afford a static IP to greatly strengthen their data.
 
You guys are right about it. So I am going to get a new VPS set up (right now using vps 2 so would go for the same one). Then gradually shift accounts making sure they are clean one by one. I should rather keep strict security policy this time.
For doctor's script I am not sure how to go about it. Since its not my purview to examine their programming I can only offer them suggestions on what they should do and what don't. Looking at the two SQL users, i think, this should work for them. Additionally I would try to convince them to get a dedicated IP for their domain/site and use it for connectivity. That way I can only allow one IP to access host that too monitored.
Hope this secures the server in future.
 
@Dave G thanks . I am not a tech guy and would probably wont' be :( I read the link but not sure if they manage it on my behalf or not. What is your experience of this thing? It is 'a bit' costly to have. I will have to raise little price of my products.
 
You must log in or register to reply here.
Top