Is anyone else getting memory errors all of a sudden?

lalaland

New Member
Recently my VPS has been hitting max memory and crashing CPanel, HTTPD and pretty much everything else running on the server.

I've searched and searched and can't find anything that's changed, even the number of users hasn't increased (if anything in the recent day's it's decreased!). I genuinely cannot see why over the last few weeks my memory used % is rocketing through the ceiling.

On some occasions the CPU is maxed out with the memory, others it's just the memory. The strange thing is that there's only a handful of people connected to my forum and nothing else. It's certainly not demanding stuff, I can't work out why my VPS is dying.

It's got to the stage where if I don't check my VPS every hour or two I find it's down and I'm losing members of my website because of this :(

I need help if anyone's able to suggest anything? Where can I start looking? I've checked Cpanel and WHM, I've checked VZPP but all they say is privvmpages is maxed out.

While typing this I've just restarted my entire VPS via the VZPP control panel (using reboot option) and the second it's restarted, the memory's at 100% again rendering my entire website unavailable.

What's going on? Nobody's able to connect and this is getting more than annoying.

I genuinely can't see why the demand has changed, there are less users and it's the same vBulletin processes but since a few weeks back the memory and occasionally CPU is just maxing out and dying taking me plenty of time restarting and trying to find faults that don't seem to exist.

Help?
 
Right now this is my process list from VZPP with my memory maxed out and nothing able to run
1 0.0 0.0 init [3] 0 23 600 S 00:00:00 0
1664 0.0 0.0 syslogd -m 0 0 24 544 S 00:00:00 0
1683 0.0 0.0 /usr/sbin/named -u named 0 19 2836 S 00:00:00 25
1710 0.0 0.0 /usr/sbin/courierlogger -pid=/var/spool/authdaemon/pid -facility=mail -start /usr/libexec/courier-authlib/authdaemond 0 22 376 S 00:00:00 0
1712 0.0 0.0 /usr/libexec/courier-authlib/authdaemond 0 20 616 S 00:00:00 0
1725 0.0 0.0 /usr/sbin/sshd 0 18 1120 S 00:00:00 0
1726 0.0 0.0 /usr/libexec/courier-authlib/authdaemond 0 21 236 S 00:00:00 0
1728 0.0 0.0 /usr/libexec/courier-authlib/authdaemond 0 20 236 S 00:00:00 0
1734 0.0 0.0 /usr/libexec/courier-authlib/authdaemond 0 20 236 S 00:00:00 0
1735 0.0 0.0 /usr/libexec/courier-authlib/authdaemond 0 20 236 S 00:00:00 0
1736 0.0 0.0 /usr/libexec/courier-authlib/authdaemond 0 20 236 S 00:00:00 0
1751 0.0 0.0 xinetd -stayalive -pidfile /var/run/xinetd.pid 0 21 788 S 00:00:00 0
1770 0.0 0.0 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/ns1.simalert.com.pid 0 21 1124 S 00:00:00 0
1795 27.8 1.1 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/ns1.simalert.com.pid --skip-external-locking 0 24 92568 S 00:01:08 100
1838 0.0 0.0 chkservd 0 23 8032 S 00:00:00 0
1859 0.0 0.0 /usr/sbin/courierlogger -pid=/var/run/imapd.pid -start -name=imapd /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=40 -maxperip=30 -nodnslookup -noidentlookup 143 /usr/lib/courier-imap/sbin/imaplogin /usr/lib/courier-imap/bin/imapd Maildir 0 18 300 S 00:00:00 0
1860 0.0 0.0 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=40 -maxperip=30 -nodnslookup -noidentlookup 143 /usr/lib/courier-imap/sbin/imaplogin /usr/lib/courier-imap/bin/imapd Maildir 0 21 492 S 00:00:00 0
1873 0.0 0.0 /usr/sbin/courierlogger -pid=/var/run/imapd-ssl.pid -start -name=imapd-ssl /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=40 -maxperip=30 -nodnslookup -noidentlookup 993 /usr/lib/courier-imap/bin/couriertls -server -tcpd /usr/lib/courier-i 0 19 300 S 00:00:00 0
1875 0.0 0.0 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=40 -maxperip=30 -nodnslookup -noidentlookup 993 /usr/lib/courier-imap/bin/couriertls -server -tcpd /usr/lib/courier-imap/sbin/imaplogin /usr/lib/courier-imap/bin/imapd Maildir 0 21 492 S 00:00:00 0
1882 0.0 0.0 /usr/sbin/courierlogger -pid=/var/run/pop3d.pid -start -name=pop3d /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=40 -maxperip=30 -nodnslookup -noidentlookup 110 /usr/lib/courier-imap/sbin/pop3login /usr/lib/courier-imap/bin/pop3d Maildir 0 16 300 S 00:00:00 0
1883 0.0 0.0 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=40 -maxperip=30 -nodnslookup -noidentlookup 110 /usr/lib/courier-imap/sbin/pop3login /usr/lib/courier-imap/bin/pop3d Maildir 0 21 492 S 00:00:00 0
1893 0.0 0.0 /usr/sbin/courierlogger -pid=/var/run/pop3d-ssl.pid -start -name=pop3d-ssl /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=40 -maxperip=30 -nodnslookup -noidentlookup 995 /usr/lib/courier-imap/bin/couriertls -server -tcpd /usr/lib/courier-i 0 19 300 S 00:00:00 0
1894 0.0 0.0 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=40 -maxperip=30 -nodnslookup -noidentlookup 995 /usr/lib/courier-imap/bin/couriertls -server -tcpd /usr/lib/courier-imap/sbin/pop3login /usr/lib/courier-imap/bin/pop3d Maildir 0 21 492 S 00:00:00 0
1980 0.0 0.0 /usr/sbin/exim -bd -q60m 0 24 1180 S 00:00:00 47
1988 0.0 0.0 /usr/sbin/exim -tls-on-connect -bd -oX 465 0 14 1128 S 00:00:00 47
1995 0.0 0.0 antirelayd 0 23 1172 S 00:00:00 0
3117 0.0 0.0 pure-ftpd (SERVER) 0 21 1308 S 00:00:00 0
3120 0.0 0.0 /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/sbin/pureauth 0 21 928 S 00:00:00 0
3130 0.0 0.0 crond 0 23 924 S 00:00:00 0
3148 0.0 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 9056 S 00:00:00 0
3257 0.0 0.1 cpsrvd - waiting for connections 0 23 8268 S 00:00:00 0
3265 0.0 0.0 cpbandwd 0 23 4812 S 00:00:00 0
3266 0.0 0.0 cpanellogd - sleeping for logs 19 5 6912 SN 00:00:00 0
3365 0.0 0.0 /usr/sbin/saslauthd -m /var/run/saslauthd -a shadow -n 2 0 21 804 S 00:00:00 0
3366 0.0 0.0 /usr/sbin/saslauthd -m /var/run/saslauthd -a shadow -n 2 0 21 464 S 00:00:00 0
3409 0.0 0.0 /usr/sbin/portsentry -tcp 0 21 392 S 00:00:00 0
3545 0.4 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14900 S 00:00:01 99
3549 0.6 0.2 /usr/local/apache/bin/httpd -DSSL 0 24 16816 S 00:00:01 99
3555 0.3 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14900 S 00:00:00 99
3568 0.7 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14772 S 00:00:01 99
3577 0.3 0.1 /usr/local/apache/bin/httpd -DSSL 0 24 14772 S 00:00:00 99
3599 0.1 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14828 S 00:00:00 99
3600 0.1 0.1 /usr/local/apache/bin/httpd -DSSL 0 24 14724 S 00:00:00 99
3601 0.1 0.1 /usr/local/apache/bin/httpd -DSSL 0 22 14828 S 00:00:00 99
3602 0.2 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14904 S 00:00:00 99
3610 0.2 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14908 S 00:00:00 99
3611 0.1 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14828 S 00:00:00 99
3612 0.1 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14852 S 00:00:00 99
3613 0.1 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14852 S 00:00:00 99
3614 0.1 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14700 S 00:00:00 99
3625 0.4 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14900 S 00:00:00 99
3627 0.1 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14832 S 00:00:00 99
3628 0.1 0.1 /usr/local/apache/bin/httpd -DSSL 0 24 14828 S 00:00:00 99
3635 0.3 0.1 /usr/local/apache/bin/httpd -DSSL 0 22 14900 S 00:00:00 99
3857 0.2 0.1 /usr/local/apache/bin/httpd -DSSL 0 22 14900 S 00:00:00 99
3870 0.2 0.1 /usr/local/apache/bin/httpd -DSSL 0 22 14840 S 00:00:00 99
4062 0.3 0.1 /usr/local/apache/bin/httpd -DSSL 0 22 14844 S 00:00:00 99
4069 0.2 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14764 S 00:00:00 99
5243 0.2 0.1 /usr/local/apache/bin/httpd -DSSL 0 22 14764 S 00:00:00 99
5256 0.2 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14764 S 00:00:00 99
9918 0.6 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14764 S 00:00:00 99
10051 1.2 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14904 S 00:00:00 99
11343 0.5 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14828 S 00:00:00 99
13315 3.0 0.0 [httpd <defunct>] 0 21 0 Z 00:00:00 99
13316 12.0 0.1 /usr/local/apache/bin/httpd -DSSL 0 23 14828 S 00:00:00 99
13317 0.0 0.0 [httpd <defunct>] 0 18 0 Z 00:00:00 99
13318 1.0 0.0 [httpd <defunct>] 0 18 0 Z 00:00:00 99
 
I've tried restarting again and once again the memory is straight in to the red zone.

So even if I restart my entire server it's immedialty dying!
 
lalaland,

What kind of VPS plan do you have and what sort of traffic do you have on your sites?

Regards,
Paul
 
I'm on VPS Mtx and I only run one site from it (there are a couple of domains on there, one for email and the rest are holding).

I have about 30 people on the forum at normal times about up to about 50 at busy times. Tonight there are about 10 people but the server keeps dying, this is about the 6th time I've rebooted in the last few minutes!
 
lalaland,

VPS-M isn't the plan which suppose to handle 50 people online. You should seriously consider upgrading your VPS to the higher plan to get access to more resources. Based on the process list you've posted above MySQL alone is using 92568 KB of memory on your system, plus you have good number of Apache processes running and this brings you to the limit.

If you have to reboot the VPS with 10 connected people and it goes into the black zone (I suppose we're talking about privvmpages resource, right?), then there is a good chance that somebody is attacking your site by opening countless number of connections to your system. You should be able to see sorted list of connections by running this command:

Code:
netstat -anp | grep "^tcp\|^udp" | sort -nk 6

If you'll see, say, 20+ connections from the same IP on the same port, then there is a chance that this guy is attacking your system. You can block the IP in runtime configuration using this command:

iptables -A INPUT -s IPADDR -j DROP

where IPADDR needs to be replaced with attacker's IP address.

Regards,
Paul
 
Top