ModSecurity (aka mod_security, security2_module, or modsec) is an apache module designed to work similarly to a Web Application Firewall, to help protect websites from certain types of attacks. Which types of attacks the sites are then protected from would depend on which ruleset is in use. In terms of the 7-Layer OSI Model, it is only Level 7 (the Application Layer) which is affected by ModSecurity.
In general, ModSecurity looks at the incoming apache requests, compares it to patterns described in the rules in the ruleset, and takes actions on the requests based on the results of the tests.
In case you are interested, the full Reference Manual for ModSecurity can be found here.
But, it is very important to note that in recent cPanel versions, it is usually not necessary to do any of the ModSecurity configuration manually.
It is also useful to know that which ruleset is appropriate for your server depends heavily on the specific types of sites you will be hosting, and more importantly, the specific types of apache requests that will be needed in order for the site applications to work. If one of the types of requests being blocked by the ModSecurity ruleset installed is needed in order for one of the sites to work, then parts of that site will not work while that ruleset is installed.
The tools for configuring ModSecurity do not do anything unless the module has been installed in Apache. If ModSecurity is not yet installed, this can be done via EasyApache. Make sure that ModSecurity has been selected before beginning the build process.
Once the build process has completed, the next step is to add a ruleset. The recommended way to do this is in WHM at
Home >> Security Center >> ModSecurity™ Vendors as described here. By default, there should be a cPanel-curated OWASP ruleset available to choose. If there is a different published ruleset you prefer to use instead, check with the developer of the ruleset if they make their ruleset available as a Vendor that can be added via WHM. If the ruleset is available in this format, it is the strongly prefered method of installing the ruleset.
If there are modifications that need to be made to the installed ruleset, this can be done via WHM at
Home >> Security Center >> ModSecurity™ Tools as described here.
For example, if there are rules in the ruleset that are not suitable for your specific site, you may want to disable those specific rules, or report the false positives to the developer of the ruleset. If you suspect that the ModSecurity ruleset is interfering with site functionality, check the Hits List section of the ModSecurity Tools page of WHM. In that list, find examples of requests that you are sure are expected and should not be blocked. In one of those examples, click "Report this Hit" and enter in the needed information as described here. By reporting the false positives, you can allow the developer of the ruleset to improve the ruleset. When a new version of the ruleset is released, cPanel/WHM will automatically update it when the daily update/maintenance tasks are run.
Also on the ModSecurity Tools page is the Rules List. From here, you can enable or disable individual rules, or add or delete or edit rules. You cannot edit or delete rules that are part of a Vendor ruleset, but you can edit or delete rules that have been manually added. Vendor rules can be individually enabled or disabled if needed.
Some types of rules are not compatible with mod_ruid2. In particular, any ModSecurity rules that need to store values in files will not work when mod_ruid2 due to the interaction between the file permissions and ownerships, and the users the apache processes would run as. Rules that do not work with mod_ruid2 for this reason are also not likely to work with mpm-itk for the same reasons. If you are not sure whether a specific ruleset has rules in it that need to store values in files, you may want to check with the developer of the ruleset before trying to use the ruleset in a server using mod_ruid2 or mpm-itk.
If you also have ConfigServer Security&Firewall installed in the server, you may want be aware that LF_MODSEC is enabled by default. If this feature is enabled, then if the same IP address triggers modsec rules a certain number of times within a certain period of time, that IP address will be blocked in the firewall (either temporarily or permanently, depending on how related settings are configured). But, since it is possible for some rules to conflict with some web applications, it is strongly recommended to make sure the installed modsec ruleset is appropriate for the specific server before enabling this firewall feature. Otherwise, typical visitors doing typical site activities might become blocked in the firewall.