kmemsize limit

[jimdahl]

I've recently noticed infrequent problems with MySQL not being able to start a new thread. I was curious because I'm usually nowhere near my numproc or memory limits.

Looking at the QoS alerts in the Virtuozzo CP, I apparently often trip over the kmemsize warning threshold. All the documentation I can find indicates that this kernel memory is just for the kernel's process/thread table.

I wouldn't have guessed that anything in my configuration launched an inordinately large number of processes or threads, nor does the QoS alerts log indicate any numproc or memory problems. Pretty standard software -- Apache 2.2 (prefork MPM), MySQL 4.1.

It's really only been a problem in the past week or so, not particularly coincidental with any big changes to any of my software versions. Anyone have any thoughts?

 

[KH-Paul]

Jim,

Kmemsize controls amount of kernel memory used by internal data structures related to your specific VPS. Each process running inside the VPS typically use around 30-60 KB. If you're seeing high kmemsize utilization, then this probably means that you're running some pretty large processes that create massive data structures in the low memory. We usually recommend to take the following steps to reduce low memory usage:
- If your sites are based on PHP engine and perform a lot of database queries it might make sense to disable persistent MySQL connections to get db connections closed automatically as soon as it become unused;
- Reduce value of the Timeout variable in your Apache configuration to something like 10-20;
- Set MaxRequestsPerChild parameter to 150-500 (depending on volume of web traffic you receive).

Above things should help to reduce low memory usage. If you see no difference in kmemsize usage, then something else is taking your low memory. Might be a good idea to check other (less common) services and stop / start them to see how kmemsize resource usage will be affected.

Regards,
Paul

 

[jimdahl]

- If your sites are based on PHP engine and perform a lot of database queries it might make sense to disable persistent MySQL connections to get db connections closed automatically as soon as it become unused;
- Reduce value of the Timeout variable in your Apache configuration to something like 10-20;
- Set MaxRequestsPerChild parameter to 150-500 (depending on volume of web traffic you receive).

I did just try reducing the number of live MySQL processes by setting them to timeout after 15 secs (and creating a small thread cache pool to offset the construction/destruction CPU penalty). A similar attack on the number of the life of HTTP processes makes sense.

My real problem is that I've never been able to catch it in near the limit. Whenever I look, I'm usually around 5-6M, nowhere near the limit of 15M. But, once or so a day, it generates a "yellow zone" alert.

I think I'll try to set up a cron job to check the kmemsize (in /proc/user_beancounters) and record a snapshot of active processes when its over some threshold. I'm not sure how much that will reveal, since it will just show a ton of HTTP processes running and I still won't know precisely what they're doing, but it's a place to start.

 

[rafaelsorto]

Hello jimdahl , im having problems with kmemsize from time to time, and ive noticed some things that may be useful to you.

- When i have problems with kmemsize is when apache is starting a lot of child processes , so even if im good with my guaranteed/burst RAM , the kmemsize value is limiting this.

- When apache is starting a lot of child processes is because mysql is not giving the results as quick as apache needs it, so when other connections are made to the website apache tries to keep the new connections waiting until the first ones made are solved. Im telling you this because whenever im having kmemsize problems there are like 100 connections locked in mysql.

- The kmemsize problems are the exact same days as googlebot crawls my busiest site. so there you go all points to the fact that i need to optimize apache/mysql so they can both handle my usual visitors load and also the googlebot.

One thing to keep in mind is that you should do some benchmarks to the apache and see if it can take the imaginary load, maybe you can do it late at midningth when you dont have too many visitors, as i am hosting some clients sites this is not something i can do.

So i've limited the max_user_connections to 30 , so that apache/mod_php can die with the "i was not able to connect" error.

I think i have optimized mysql so it can take the load also altering some values like table_cache, key_buffer, thread_cache_size, and stuff like that so im just waiting for the next googlebot crawl to see if the server can take it XD