KNOWNHOST WIKI

User Tools

Site Tools


general:disk-space-management

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Last revision Both sides next revision
general:disk-space-management [2017/02/26 17:28]
mscherf [Backups] added note about site plugins for backups
general:disk-space-management [2020/05/13 11:39]
Karson N. [Trash Folders - IMAP]
Line 3: Line 3:
 It is important to make sure that the server does not fully run out of disk space. If it does, the services running in the server will quit working, and some of the database tables could even become corrupted. Here are some things to consider, to help ensure that your server has enough disk space to work correctly. It is important to make sure that the server does not fully run out of disk space. If it does, the services running in the server will quit working, and some of the database tables could even become corrupted. Here are some things to consider, to help ensure that your server has enough disk space to work correctly.
  
 +\\
 ===== Backups ===== ===== Backups =====
  
 Although backups do themselves take disk space, it is still very important to have them. To protect against data loss, it is always strongly recommended not only to keep multiple sets of backups, but also to keep copies of at least some of them in an offsite location. Although we do create our own [[vps-hosting:information-about-vps-backups-and-restorations|backups of the VPSs]], there are multiple reasons it is important not only for you to create your own backups as well, but also to keep at least one of the copies offsite. Although backups do themselves take disk space, it is still very important to have them. To protect against data loss, it is always strongly recommended not only to keep multiple sets of backups, but also to keep copies of at least some of them in an offsite location. Although we do create our own [[vps-hosting:information-about-vps-backups-and-restorations|backups of the VPSs]], there are multiple reasons it is important not only for you to create your own backups as well, but also to keep at least one of the copies offsite.
  
-For cPanel/WHM servers, you can read more details about [[https://documentation.cpanel.net/display/ALD/Backup+Configuration|configuring automatic backups]] or [[https://documentation.cpanel.net/display/ALD/Backup+Wizard|manually triggering backups at the time you want them]]. For DirectAdmin servers, you can configure backups from [[https://help.directadmin.com/item.php?id=269|any of the three access levels]] and it is possible to [[https://help.directadmin.com/item.php?id=511|store the generated backup both locally and remotely]]. For servers still using Plesk,((Plesk is no longer available for new servers)) you can learn more about the backup system [[http://docs.plesk.com/en-US/12.5/advanced-administration-guide-linux/backing-up-restoring-and-migrating-data/backing-up-data.68837/|here]].+For cPanel/WHM servers, you can read more about how to configure automatic backups and some of our recommendations for them [[control-panels:cpanel-whm:backup-configuration|here]]. For manually triggering backups at the time you want them via cPanel, you can read more [[https://documentation.cpanel.net/display/ALD/Backup+Wizard|here]]. For DirectAdmin servers, you can configure backups from [[https://help.directadmin.com/item.php?id=269|any of the three access levels]] and it is possible to [[https://help.directadmin.com/item.php?id=511|store the generated backup both locally and remotely]]. For servers still using Plesk,((Plesk is no longer available for new servers)) you can learn more about the backup system [[http://docs.plesk.com/en-US/12.5/advanced-administration-guide-linux/backing-up-restoring-and-migrating-data/backing-up-data.68837/|here]].
  
-If you have a Dedicated Server, it is likely that automatic backups are already enabled in WHM, since for Dedicated Servers we enable that by defaultBut even on the Dedicated Serverswe cannot automatically configure remote Additional Destinations, since you would need to provide some of the [[https://documentation.cpanel.net/display/ALD/Backup+Configuration#BackupConfiguration-AdditionalDestinations|relevant information]]. Additionally, when accounts are migrated into a server, they keep their backup settings (that is, whether backups are enabled/disabled) from the old server.+<WRAP important> 
 +It is particularly strongly recommended **not** to use site plugins or similar to generate backups. Very frequently this type of site plugin runs at an unnecessarily high i/o priority, which in many cases can use up all the disk time and prevent other processes (such as mysql or apache) from having quick access to the diskFor this reasonthis type of plugin is extremely likely to slow down all of the sites on your server, and they are not recommended. 
 +</WRAP>
  
 On the Dedicated Servers, the backups are by default configured to be stored on the secondary disk, so even in the event that the partition for the backups becomes completely full, it will not interfere with the functionality of the daemons (though it would cause those backup runs to fail, and those backup files to become corrupted). But on a VPS, there is only a single partition, so it is very important to make sure the backups being generated and stored there will not cause the disk to become completely full. On the Dedicated Servers, the backups are by default configured to be stored on the secondary disk, so even in the event that the partition for the backups becomes completely full, it will not interfere with the functionality of the daemons (though it would cause those backup runs to fail, and those backup files to become corrupted). But on a VPS, there is only a single partition, so it is very important to make sure the backups being generated and stored there will not cause the disk to become completely full.
  
-It is not recommended to include backups inside other backups. For example, if there is a site plugin making backups, and then Softaculous makes backups of the site which contain the backups made by the plugins, then on top of that WHM is making backups of the cPanel account which contain the Softaculous backups (which, as mentioned, also in turn contain the backups made by the plugin), then each backup set will cause multiple times the amount of diskspace to be taken as otherwise. If redundancy is desired (which is recommended), then rather than having nested backups, choose one backup type, and then run that type more often, retain more sets, and/or copy the sets to more locations. If you are sure you would like to use multiple backup systems simultaneously, it is strongly recommended to exclude each type of backup from the backups generated by the other systems. You can learn more about excluding files from cPanel backups [[https://documentation.cpanel.net/display/CKB/How+to+Exclude+Files+From+Backups|here]]. +Overall, when choosing a backup configuration, it is important to consider all of the following factors:
- +
-<WRAP important>It is particularly strongly recommended **not** to use site plugins or similar to generate backups. Very frequently this type of site plugin runs at an unnecessarily high i/o priority, which in many cases can use up all the disk time and prevent other processes (such as mysql or apache) from having quick access to the disk. For this reason, this type of plugin is extremely likely to slow down all of the sites on your server, and they are not recommended.</WRAP> +
- +
-Because sometimes the days chosen for weekly and monthly backups coincide, it is recommended to retain at least two of at least one of the three frequencies.((ie, retain at least two daily backups, or at least two weekly backups, or at least two monthly backups)) For example, if you only retain one each of daily, weekly, and monthly backups, then whenever the chosen day of the week for the weekly backups happens on the first or fifteenth of the month, you will only be retaining one backup that time, resulting in a single point of failure. If for some reason on that day backups fail to complete, you will not have any backups at all until they run again. +
- +
-So overall, when choosing a backup configuration, it is important to consider all of the following factors:+
  
   * How much disk space you have available in the server   * How much disk space you have available in the server
Line 26: Line 23:
   * How long ago you want to be able to revert accounts to   * How long ago you want to be able to revert accounts to
  
-If you're not sureyou may want to start with a basic configuration as follows: +For more specific recommendations,((for cPanel/WHM servers)) please read more [[control-panels:cpanel-whm:backup-configuration|here]].
- +
-  * Global Settings: +
-    * Backup Status: Enable +
-    * Backup Type: Compressed +
-  * Scheduling and Retention: +
-    * Backup Daily: Enable +
-      * Select days +
-      * Retain (1 or 2) Daily Backups +
-    * Backup Weekly: Enable +
-      * Select day +
-      * Retain (1 or 2) Weekly Backups +
-    * Backup Monthly: Enable +
-      * Select day(s) +
-      * Retain (1 or 2) Monthly Backups +
-  * Files: +
-    * Backup Accounts: Enable +
-      * Select Users: make sure all users on this page are set to "On" in the "Backups" column +
-    * Backup System Files: Enable +
-  * Databases: +
-    * Per Account and Entire MySQL Directory +
-  * Configure Backup Directory: +
-    * Retain backups in the default backup directory: Enable +
-    * Mount Backup Drive as Needed: Disable (if VPS) +
-  * Additional Destination: Add information as appropriate for each additional destination desired +
- +
-Notes: +
- +
-  * These settings are written for a cPanel server. For Plesk or DirectAdmin, similar equivalent settings are recommended where possible. +
-  * Retain at least two backups of at least one of the three types (daily/weekly/monthly) +
-  * It is **strongly recommended** to have //at least// one "Additional Destination" configured pointing to a remote destination. +
-  * It is recommended to only use Additional Destinations that can use secure, encrpyted connections.((eg, Amazon S3, SFTP, or WebDAV with SSL Enabled)+
-  * If there is not enough disk space within the server to store the backups, then once you have confirmed that the remote Additional Destination is working, you can deselect (disable) the option "Retain backups in the default backup directory".((in section "Configure Backup Directory")) +
-  * There needs to be enough disk space in the server to //generate// the backups, even if they are not being stored in the server. For this purpose, you should ensure you have at least twice the size of the largest account free, plus an extra gigabyte. If you do not have this much disk space in your server, it is strongly recommended to add more disk space to your server, so that you have adequate room to generate backups. +
-  * For convenience, it is recommended if possible to store the backups in the server as well as at the remote destination. For this, it is necessary to have at least twice the amount of free space as is used by one set of backups, since old backups are not deleted until the new backups have finished being generated. +
-  * This example would require approximately((the exact amount of space needed can vary, depending on several factors, but this is an approximation)) space to store four sets of backups, plus space for two sets of backups during backup generation (the one being generated, plus the one that is soon going to be deleted), plus 2 * (largest account size) + 1GB for the account backup currently in progress. (This is not including the disk space required for the accounts themselves or the system files for the sites and services to run.) +
- +
-The backup settings can be changed later if needed.+
  
 +\\
 ===== Account Quotas ===== ===== Account Quotas =====
  
Line 75: Line 36:
 It is also important to note that (in a cPanel server) files owned by "nobody" will not count against the cPanel user's disk space quota. This is yet another reason to use mod_ruid2 (or, if this is not possible, suphp) so that the site files will be correctly owned by the cPanel user. It is also important to note that (in a cPanel server) files owned by "nobody" will not count against the cPanel user's disk space quota. This is yet another reason to use mod_ruid2 (or, if this is not possible, suphp) so that the site files will be correctly owned by the cPanel user.
  
 +\\
 ===== Monitoring and Troubleshooting ===== ===== Monitoring and Troubleshooting =====
  
Line 100: Line 62:
 If more disk space is being used than expected, it is recommended to check what is using the disk space. Where to check first will depend on how you find out about it. If more disk space is being used than expected, it is recommended to check what is using the disk space. Where to check first will depend on how you find out about it.
  
-If the server has used up enough of its disk space overall that it is causing services (daemons) to be interrupted, you might not be able to use the cPanel((or DirectAdmin or Plesk, as applicable)) interface to investigate. Sometimes even if the panel is not working, you will be able to connect using [[software:how-can-i-use-putty-software-to-ssh-to-my-cpanel-server|ssh]]. If ssh is not working either, you may be able to connect via [[my-knownhost:manage-services:vps-control:java-console|the console]].((if it is a VPS)) If that is not working either, or if you are not comfortable with ssh and/or the console, you can [[my-knownhost:howto-login|open a Support Ticket]] for assistance.+If the server has used up enough of its disk space overall that it is causing services (daemons) to be interrupted, you might not be able to use the cPanel((or DirectAdmin or Plesk, as applicable)) interface to investigate. Sometimes even if the panel is not working, you will be able to connect using [[software:how-can-i-use-putty-software-to-ssh-to-my-cpanel-server|SSH]]. If SSH is not working either, you can [[my-knownhost:howto-login|open a Support Ticket]] for assistance 24x7x365.
  
 +\\
 ==== Troubleshooting via SSH ==== ==== Troubleshooting via SSH ====
  
Line 108: Line 71:
 <WRAP important> If you are not //able// to log into cPanel/WHM,((or DirectAdmin or Plesk, as applicable)) it is **strongly** recommended to open a Support Ticket for assistance, unless you are very comfortable with ssh, enough to attempt to resolve the issue on your own. It is important to be **extremely** careful if attempting to resolve disk space issues yourself, since it is possible to remove wrong files, thereby breaking server functionality. </WRAP> <WRAP important> If you are not //able// to log into cPanel/WHM,((or DirectAdmin or Plesk, as applicable)) it is **strongly** recommended to open a Support Ticket for assistance, unless you are very comfortable with ssh, enough to attempt to resolve the issue on your own. It is important to be **extremely** careful if attempting to resolve disk space issues yourself, since it is possible to remove wrong files, thereby breaking server functionality. </WRAP>
  
-If you are logged into ssh as root,((or if you have gotten a connection via the console)) you can check the server's overall disk space usage with the ''%%df%%'' utility. For example, you can check the disk space usage in the root ''%%/%%'' partition as follows:+If you are logged into ssh as root, you can check the server's overall disk space usage with the ''%%df%%'' utility. For example, you can check the disk space usage in the root ''%%/%%'' partition as follows:
  
 <code> <code>
Line 125: Line 88:
  
 The output will appear similar as in the above example, but with data for ''%%/backup%%'' instead of ''%%/%%''. The output will appear similar as in the above example, but with data for ''%%/backup%%'' instead of ''%%/%%''.
 +
 +If you want to check how many backups are present and how big they are, you can check with this command:
 +
 +<code>
 +root@host [~/support]# find /backup -type d -name accounts -print0 | xargs -0 du -sch | sort -h
 +4.0K /backup/2017-07-02/accounts
 +4.0K /backup/2017-07-03/accounts
 +4.0K /backup/2017-07-04/accounts
 +4.0K /backup/2017-07-05/accounts
 +4.0K /backup/2017-07-06/accounts
 +4.0K /backup/2017-07-07/accounts
 +4.0K /backup/monthly/2017-07-01/accounts
 +4.0K /backup/weekly/2017-06-25/accounts
 +4.0K /backup/weekly/2017-07-02/accounts
 +5.0G /backup/monthly/2017-06-01/accounts
 +5.0G /backup/monthly/2017-06-15/accounts
 +10G total
 +</code>
 +
 +In this case, a Remote Destination was configured, and later the server was configured not to retain the backups locally. The backups that were already present before that were not removed, but all subsequent ones were deleted from the server after successful transfer to the remote destination.
 +
 +Although this only looks for the account backups, these are usually larger than the system backups.
 +
 +If you decide you want to remove one or more of the backups to save space, you can use the `cd` command to go to the desired folder, and then use the `rm -v` command to remove the specific files you would like to remove.
  
 After checking the total disk space usage of the server, it is useful to find how much is being used by each account. You can get a general overview using ''%%repquota%%'': After checking the total disk space usage of the server, it is useful to find how much is being used by each account. You can get a general overview using ''%%repquota%%'':
Line 156: Line 143:
 <code> <code>
 root@host [~]# cd /path/to/folder/ root@host [~]# cd /path/to/folder/
-root@host [/path/to/folder]# du -h --max-depth=1 .+root@host [/path/to/folder]# du -hd 1 .
 932K ./.cphorde 932K ./.cphorde
 84K ./etc 84K ./etc
Line 184: Line 171:
  
 <code> <code>
-root@host [/path/to/folder]# du -h --max-depth=1 . | sort -h | tail+root@host [/path/to/folder]# du -hd 1 . | sort -h | tail
 100K ./.razor 100K ./.razor
 580K ./.cpanel 580K ./.cpanel
Line 200: Line 187:
  
 <code> <code>
-root@host [/path/to/folder]# du -ha --max-depth=1 .+root@host [/path/to/folder]# du -had 1 .
 932K ./.cphorde 932K ./.cphorde
 4.0K ./.contactemail 4.0K ./.contactemail
Line 243: Line 230:
  
 <code> <code>
-root@host [/path/to/folder]# du -ha --max-depth=1 . | sort -h | tail+root@host [/path/to/folder]# du -had 1 . | sort -h | tail
 100K ./.razor 100K ./.razor
 580K ./.cpanel 580K ./.cpanel
Line 268: Line 255:
  
 <code> <code>
-root@host [/path/to/folder]# du -h --max-depth=1 . | sort -h | tail+root@host [/path/to/folder]# du -hd 1 . | sort -h | tail
 100K ./.razor 100K ./.razor
 580K ./.cpanel 580K ./.cpanel
Line 284: Line 271:
  
 <code> <code>
-root@host [/path/to/folder]# du -ha --max-depth=1 . | sort -h | tail+root@host [/path/to/folder]# du -had 1 . | sort -h | tail
 580K ./.cpanel 580K ./.cpanel
 932K ./.cphorde 932K ./.cphorde
Line 297: Line 284:
 </code> </code>
  
 +\\
 ==== Troubleshooting via cPanel/WHM ==== ==== Troubleshooting via cPanel/WHM ====
  
Line 307: Line 295:
 If there is an account in particular you would like to investigate, you can click on the "cP" icon of the row in the "cPanel" column, to open the cPanel interface for the account. From there, you can go to the "[[https://documentation.cpanel.net/display/ALD/Disk+Usage|Disk Usage]]" page in the "Files" section, to see a breakdown of approximately how that account is using its disk space. How to proceed from there depends heavily on what category is using most of the disk space. If there is an account in particular you would like to investigate, you can click on the "cP" icon of the row in the "cPanel" column, to open the cPanel interface for the account. From there, you can go to the "[[https://documentation.cpanel.net/display/ALD/Disk+Usage|Disk Usage]]" page in the "Files" section, to see a breakdown of approximately how that account is using its disk space. How to proceed from there depends heavily on what category is using most of the disk space.
  
 +\\
 ==== Temporary Files ==== ==== Temporary Files ====
  
-Usually temporary files will not be the main cause of the server running out of disk space, but if it is so full you need to use the console instead of regular ssh, it is possible that removing files from ''%%/tmp%%'' might, depending on the severity of the issue, clear enough space for ssh to start working. It is very important //**not**// to remove the folder itself, or any socket files or subdirectories that might be present. In a cPanel serverthis command is likely to be safe:+//Usually// temporary files will not be the main cause of the server running out of disk space. However, it is very important //**not**// to remove the ''%%/tmp/%%'' directory itself, or any socket files or subdirectories that might be present. Before removing the filescheck how much space they are taking up:
  
 {{howhard>5}} {{howhard>5}}
 +
 +<code>
 +# find /tmp -type f -print0 | xargs -0 du -sch | sort -h | tail
 +</code>
 +
 +If the files in ''%%/tmp%%'' are large enough to make a difference, you may want to consider removing them. To do so, in a cPanel server, this command is likely to be safe:
  
 <code> <code>
Line 317: Line 312:
 </code> </code>
  
 +\\
 ==== Trash Folders - cPanel File Manager ==== ==== Trash Folders - cPanel File Manager ====
  
 cPanel File Manager has a trash folder for each account, so that if something is deleted by accident from the File Manager, it can be brought back relatively easily. But, this trash folder is not automatically emptied. If any of your users make extensive use of the File Manager in cPanel, especially if they do not often use the [[https://documentation.cpanel.net/display/ALD/File+Manager#FileManager-Deleteandrestorefilesandfolders|Empty Trash]] button within File Manager, it is possible that these trash folders can get quite large. cPanel File Manager has a trash folder for each account, so that if something is deleted by accident from the File Manager, it can be brought back relatively easily. But, this trash folder is not automatically emptied. If any of your users make extensive use of the File Manager in cPanel, especially if they do not often use the [[https://documentation.cpanel.net/display/ALD/File+Manager#FileManager-Deleteandrestorefilesandfolders|Empty Trash]] button within File Manager, it is possible that these trash folders can get quite large.
 +
 +\\
  
 === via cPanel === === via cPanel ===
Line 327: Line 325:
 These trash folders can be emptied within cPanel File Manager by clicking the "Empty Trash" button. These trash folders can be emptied within cPanel File Manager by clicking the "Empty Trash" button.
  
-=== via SSH or Console ===+\\ 
 + 
 +=== via SSH ===
  
 {{howhard>5}} {{howhard>5}}
  
-These trash folders are named ''%%.trash%%'' and are direct subdirectories of each user's home directory. If logged into ssh as root((or if logged into the console)) you can check which accounts are taking the most space with their File Manager trash folders as follows:+These trash folders are named ''%%.trash%%'' and are direct subdirectories of each user's home directory. If logged into SSH as rootyou can check which accounts are taking the most space with their File Manager trash folders as follows:
  
 <code> <code>
Line 344: Line 344:
  
 <code> <code>
-# find /home*/*/.trash -depth -mindepth 1 -exec rm -rv ++# find /home*/*/.trash -depth -mindepth 1 \( \( ! -type d -exec rm -v {} + \) -o \( -exec rmdir -v {} \) \)
 </code> </code>
  
 +\\
 ==== Trash Folders - IMAP ==== ==== Trash Folders - IMAP ====
  
 If your users use IMAP to connect to their mail folders, then when they delete messages via their mail client they will usually at first be moved to a trash folder. In cPanel servers, it is possible to configure the IMAP trash folders to "Autoexpunge", which means that messages which have been in these trash folders a certain number of days will be automatically removed. Within the filesystem, these folders are named ''%%.Trash%%'', capitalized, rather than lowercase like the File Manager trash folders. If your users use IMAP to connect to their mail folders, then when they delete messages via their mail client they will usually at first be moved to a trash folder. In cPanel servers, it is possible to configure the IMAP trash folders to "Autoexpunge", which means that messages which have been in these trash folders a certain number of days will be automatically removed. Within the filesystem, these folders are named ''%%.Trash%%'', capitalized, rather than lowercase like the File Manager trash folders.
 +
 +\\
  
 === Autoexpunge === === Autoexpunge ===
Line 356: Line 359:
  
 In a cPanel server, The "Autoexpunge" option can be found in WHM at ''%%Home >> Service Configuration >> Mailserver Configuration%%''. Also on this page is the option "Trash Expire Time" which controls how many days old a message is kept in the trash folder before it is automatically removed. In a cPanel server, The "Autoexpunge" option can be found in WHM at ''%%Home >> Service Configuration >> Mailserver Configuration%%''. Also on this page is the option "Trash Expire Time" which controls how many days old a message is kept in the trash folder before it is automatically removed.
 +
 +\\
  
 === via cPanel === === via cPanel ===
Line 367: Line 372:
  
 If you started File Manager from the site's docroot, ((aka, the www or public_html folder)) you will need to go up one level in order to see the mail folder. If you started File Manager from the site's docroot, ((aka, the www or public_html folder)) you will need to go up one level in order to see the mail folder.
 +
 +\\
  
 === via Webmail === === via Webmail ===
Line 374: Line 381:
 //If// you have **not** yet enabled the new [[https://documentation.cpanel.net/display/ALD/Mailbox+Conversion|mdbox]] mailbox format, you can see the contents of the email accounts from the webmail interface of the overarching email account of the cPanel account [[https://documentation.cpanel.net/display/ALD/Email+Accounts#EmailAccounts-Defaultemailaccount|itself]]. You may need to "subscribe" to the relevant folders in the settings within the preferred webmail application. But, if you have already converted from maildir format to the new mdbox format, the symlinks necessary for this method of access will not work. //If// you have **not** yet enabled the new [[https://documentation.cpanel.net/display/ALD/Mailbox+Conversion|mdbox]] mailbox format, you can see the contents of the email accounts from the webmail interface of the overarching email account of the cPanel account [[https://documentation.cpanel.net/display/ALD/Email+Accounts#EmailAccounts-Defaultemailaccount|itself]]. You may need to "subscribe" to the relevant folders in the settings within the preferred webmail application. But, if you have already converted from maildir format to the new mdbox format, the symlinks necessary for this method of access will not work.
  
-=== via SSH or Console ===+\\ 
 + 
 +=== via SSH ===
  
 {{howhard>5}} {{howhard>5}}
Line 413: Line 422:
  
 In this case, the largest file manager trash folder was smaller than the smallest imap trash folder. In this case, the largest file manager trash folder was smaller than the smallest imap trash folder.
 +
 +Similarly, if you want to empty all of the trash folders at once, you can do so as follows:
 +
 +<code>
 +# find /home*/*/{.trash,mail/{*/*/,}.Trash} -depth -mindepth 1 \( \( ! -type d -exec rm -v {} + \) -o \( -exec rmdir -v {} + \) \)
 +</code>
  
 And remember, if you need any help, you can always [[my-knownhost:howto-login|open a Support Ticket]] for assistance. And remember, if you need any help, you can always [[my-knownhost:howto-login|open a Support Ticket]] for assistance.
 +
general/disk-space-management.txt · Last modified: 2020/06/01 15:28 by Karson N.