KNOWNHOST WIKI

User Tools

Site Tools


control-panels:misc:many-crond-processes-causing-out-of-memory-errors

Many Crond Processes Causing Out of Memory Errors

Symptoms & Cause

If you are seeing service failures or find that your server is essentially inaccessible, it may be due to hanging and subsequently stacking crond processes resulting from running old systemd processes after the systemd binaries have been updated. The old systemd binaries are still in use if the server hasn't been rebooted following the update.

You can view the crond processes as the root user via SSH using the ps, top, or pgrep commands, for example. The following shows the last 20 lines output from using the ps command and listing on crond processes:

root@host [~]# ps auxf | grep crond | tail -20
root      9901  0.0  0.1  74680  1732 ?        S    07:19   0:00  \_ /usr/sbin/crond -n
root      9902  0.0  0.1  74680  1732 ?        S    07:19   0:00  \_ /usr/sbin/crond -n
root     10166  0.0  0.1  74680  1732 ?        S    07:24   0:00  \_ /usr/sbin/crond -n
root     10373  0.0  0.1  74680  1732 ?        S    07:27   0:00  \_ /usr/sbin/crond -n
root     10416  0.0  0.1  74680  1732 ?        S    07:28   0:00  \_ /usr/sbin/crond -n
root     10467  0.0  0.1  74680  1732 ?        S    07:29   0:00  \_ /usr/sbin/crond -n
root     10468  0.0  0.1  74680  1732 ?        S    07:29   0:00  \_ /usr/sbin/crond -n
root     10713  0.0  0.1  74680  2248 ?        S    07:34   0:00  \_ /usr/sbin/crond -n
root     10714  0.0  0.1  74680  2248 ?        S    07:34   0:00  \_ /usr/sbin/crond -n
root     10715  0.0  0.1  74680  2248 ?        S    07:34   0:00  \_ /usr/sbin/crond -n
root     10947  0.0  0.1  74680  2248 ?        S    07:38   0:00  \_ /usr/sbin/crond -n
root     10982  0.0  0.1  74680  2248 ?        S    07:39   0:00  \_ /usr/sbin/crond -n
root     10983  0.0  0.1  74680  2248 ?        S    07:39   0:00  \_ /usr/sbin/crond -n
root     11184  0.0  0.1  74680  2248 ?        S    07:42   0:00  \_ /usr/sbin/crond -n
root     11239  0.0  0.1  74680  2248 ?        S    07:44   0:00  \_ /usr/sbin/crond -n
root     11467  0.0  0.1  74680  2248 ?        S    07:49   0:00  \_ /usr/sbin/crond -n
root     11468  0.0  0.1  74680  2248 ?        S    07:49   0:00  \_ /usr/sbin/crond -n
root     11469  0.0  0.1  74680  2248 ?        S    07:49   0:00  \_ /usr/sbin/crond -n
root     11705  0.0  0.1  74680  2248 ?        S    07:54   0:00  \_ /usr/sbin/crond -n
root     11933  0.0  0.0 112664   976 pts/0    S+   07:57   0:00          \_ grep --color=auto crond
root@host [~]#

In this particular server, there were approximately 3700 crond processes:

root@host [~]# ps auxf | grep crond | wc  -l
3704

To confirm that the many crond processes are due to a recent systemd update, you can run the following command and see where systemd had been updated but not since restarted:

needs-restarting -r

Output like the following can be expected:

root@host [~]# needs-restarting -r
Core libraries or services have been updated:
  systemd -> 219-67.el7_7.4

Reboot is required to ensure that your system benefits from these updates.

More information:
https://access.redhat.com/solutions/27943
root@host [~]# 


The Fix

This seems to be a repeating issue with systemd as our many of our staff recall this happening in early 2019 also. A temporary fix would involve killing the crond processes and restarting the crond daemon, however, a reboot of the server is required so that the updated binaries can be used.

The following one-liner can be used to kill all crond processes:

for each in $(ps auxf | grep crond | awk {'print $2'}); do kill -9 $each ; done

Or this command:

pkill -9 crond

Then you can confirm that the crond processes have been killed like so:

ps auxf | grep crond | wc  -l

Or with this command:

 pgrep -c crond 

You can expect output like the following:

root@host [~]# for each in $(ps auxf | grep crond | awk {'print $2'}); do kill -9 $each ; done
-bash: kill: (12133) - No such process
root@host [~]# ps auxf | grep crond | wc  -l
1
root@host [~]# 

To fix this problem, issue a reboot to the server. This can be done as the root user via SSH like so:

reboot now

Once the reboot is complete, log back into the server via SSH as the root user and run the 'needs-restarting -r' command again to confirm systemd's updated binaries have taken effect:

root@host [~]# needs-restarting -r
No core libraries or services have been updated.
Reboot is probably not necessary.
root@host [~]# 

If you don't feel comfortable using the command line interface, please feel free to open a support request and we'll be happy to help!

control-panels/misc/many-crond-processes-causing-out-of-memory-errors.txt · Last modified: 2020/05/13 11:43 by Karson N.