KNOWNHOST WIKI

User Tools

Site Tools


developmental:memcrashed-what-is-it-memcache

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
developmental:memcrashed-what-is-it-memcache [2019/12/20 08:49]
Karson N. [Memcrashed]
developmental:memcrashed-what-is-it-memcache [2020/05/29 12:01] (current)
Karson N.
Line 1: Line 1:
 ====== Memcrashed: Installing Memcached and Securing Against the Memcached Vulnerability ====== ====== Memcrashed: Installing Memcached and Securing Against the Memcached Vulnerability ======
  
 +\\
 ===== Memcached ===== ===== Memcached =====
  
 Memcached is a memory object caching system for speeding up dynamic web applications... It's open source and has been designed to work with a large number of open connections. Database calls, API calls, or page rendering are a few examples of the arbitrary data results that are stored in memory to alleviate database load. One can see why it has been so popular for caching. Its popularity also means that is potential for use as an attack vector is greater simply because it is more available to weaponize.  Memcached is a memory object caching system for speeding up dynamic web applications... It's open source and has been designed to work with a large number of open connections. Database calls, API calls, or page rendering are a few examples of the arbitrary data results that are stored in memory to alleviate database load. One can see why it has been so popular for caching. Its popularity also means that is potential for use as an attack vector is greater simply because it is more available to weaponize. 
  
 +\\
 ===== Memcrashed ===== ===== Memcrashed =====
  
-A critical vulnerability in Memcached was discovered in February of  2018 that allows attackers to launch and execute powerful DDoS amplification attacks. (( https://nvd.nist.gov/vuln/detail/CVE-2018-1000115 )) This vulnerability has been described as a "new chapter in DDoS attack executions” by Ashley Stephenson, CEO of Corero Network Security. Corero Network Security is the security company that found a 'kill-switch' for these attacks. Here is her complete statement:+A critical vulnerability in Memcached was discovered in February of  2018 that allows attackers to launch and execute powerful DDoS amplification attacks. ((https://nvd.nist.gov/vuln/detail/CVE-2018-1000115))[[https://nvd.nist.gov/vuln/detail/CVE-2018-1000115|Learn more about other vulnerabilities]] This vulnerability has been described as a "new chapter in DDoS attack executions” by Ashley Stephenson, CEO of Corero Network Security. Corero Network Security is the security company that found a 'kill-switch' for these attacks. Here is her complete statement:
  
- “Memcached represents a new chapter in DDoS attack executions. Previously, the most recent record-breaking attacks were being orchestrated from relatively low bandwidth Internet of Things (IoT) devices. In contrast, these Memcached servers are typically connected to higher bandwidth networks and, as a result of high amplification factors, are delivering data avalanches to crippling effect. Unless operators of Memcached servers take action, these attacks will continue.” <sub>%% https://www.corero.com/company/newsroom/press-releases/corero-network-security-discovers-memcached-ddos-attack-kill-switch-and-also-reveals-memcached-exploit-can-be-used-to-steal-or-corrupt-data/ %%</sub>+ “Memcached represents a new chapter in DDoS attack executions. Previously, the most recent record-breaking attacks were being orchestrated from relatively low bandwidth Internet of Things (IoT) devices. In contrast, these Memcached servers are typically connected to higher bandwidth networks and, as a result of high amplification factors, are delivering data avalanches to crippling effect. Unless operators of Memcached servers take action, these attacks will continue.”  
 +((https://www.corero.com/company/newsroom/press-releases/corero-network-security-discovers-memcached-ddos-attack-kill-switch-and-also-reveals-memcached-exploit-can-be-used-to-steal-or-corrupt-data/)) 
 +[[https://www.corero.com/company/newsroom/press-releases/corero-network-security-discovers-memcached-ddos-attack-kill-switch-and-also-reveals-memcached-exploit-can-be-used-to-steal-or-corrupt-data/|]]
  
  
-This vulnerability was dubbed "Memcrashed" by Cloudflare and is responsible for a 1.35 Tbps DDoS that hit Github  February 28, 2018, (( https://githubengineering.com/ddos-incident-report/ ))+This vulnerability was dubbed "Memcrashed" by Cloudflare and is responsible for a 1.35 Tbps DDoS that hit Github  February 28, 2018,  
 +((https://githubengineering.com/ddos-incident-report/)) 
 +[[https://githubengineering.com/ddos-incident-report/|February 28th DDoS Incident Report]]
  
 +A few different PoC exploit codes have been released, one of which utilizes the Shodan search engine API to obtain a fresh list of vulnerable Memcached servers each time. The Shodan engines show 29,411 servers still unpatched as of June 16th, 2018.  
  
-A few different PoC exploit codes have been released, one of which utilizes the Shodan search engine API to obtain a fresh list of vulnerable Memcached servers each time. The Shodan engines show 29,411 servers still unpatched as of June 16th, 2018.  (( https://thehackernews.com/2018/03/memcached-ddos-exploit-code.html ))  (( https://www.shodan.io/search?query=11211 ))  (( https://github.com/649/Memcrashed-DDoS-Exploit ))+(( https://thehackernews.com/2018/03/memcached-ddos-exploit-code.html )) 
 +[[https://thehackernews.com/2018/03/memcached-ddos-exploit-code.html|Memcached DDoS Exploit Code and List of 17,000 Vulnerable Servers Released]] 
 + 
 +(( https://www.shodan.io/search?query=11211 )) 
 +[[https://www.shodan.io/search?query=11211|Shodan Search]] 
 + 
 +(( https://github.com/649/Memcrashed-DDoS-Exploit )) 
 +[[https://github.com/649/Memcrashed-DDoS-Exploit|MEMCRASHED DDOS EXPLOIT TOOL(GitHub)]]
  
 We've done a great job taking care of this threat overall as a community, but there is still work to be done. Many reports were previously reporting approximately 95,000 to 100,000 vulnerable servers.  We must remain proactive in identifying these servers and securing them!  We've done a great job taking care of this threat overall as a community, but there is still work to be done. Many reports were previously reporting approximately 95,000 to 100,000 vulnerable servers.  We must remain proactive in identifying these servers and securing them! 
  
 +\\
 ===== Getting Amp'd ===== ===== Getting Amp'd =====
  
 Why is this type of attack such a big deal? Because of the immense bandwidth amplification factor, or BAF, which is the potential effect of an amplification attack. A BAF can be calculated as the number of UDP payload bytes that an amplifier sends to answer a request, compared to the number of UDP payload bytes of the request. Why is this type of attack such a big deal? Because of the immense bandwidth amplification factor, or BAF, which is the potential effect of an amplification attack. A BAF can be calculated as the number of UDP payload bytes that an amplifier sends to answer a request, compared to the number of UDP payload bytes of the request.
  
-"15 bytes of request triggered 134KB of response. This is an amplification factor of 10,000x! In practice we've seen a 15-byte request result in a 750kB response (that's a 51,200x amplification)," Cloudflare says. (( https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port-11211/ ))+"15 bytes of request triggered 134KB of response. This is an amplification factor of 10,000x! In practice we've seen a 15-byte request result in a 750kB response (that's a 51,200x amplification)," Cloudflare says.  
 +(( https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port-11211/ )) 
 +[[https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port-11211/|Memcrashed - Major amplification attacks from UDP port 11211]]
  
 Some other popularly known DDoS amplification attack vectors include poorly secured DNS resolution servers and network time protocol, or NTP. Looking at these and a few others below, you will see each listed with its corresponding BAF: Some other popularly known DDoS amplification attack vectors include poorly secured DNS resolution servers and network time protocol, or NTP. Looking at these and a few others below, you will see each listed with its corresponding BAF:
Line 35: Line 52:
 |  Memcached  |  51,000  | |  Memcached  |  51,000  |
  
-These numbers make it incredibly easy to understand why Memcrashed is worrisome and potentially devastating for those that have not secured against it. ((https://christian-rossow.de/articles/Amplification_DDoS.php )) (( https://www.us-cert.gov/ncas/alerts/TA14-017A ))+These numbers make it incredibly easy to understand why Memcrashed is worrisome and potentially devastating for those that have not secured against it. 
  
 +((https://christian-rossow.de/articles/Amplification_DDoS.php))
 +[[https://christian-rossow.de/articles/Amplification_DDoS.php|Article]]
  
 +((https://www.us-cert.gov/ncas/alerts/TA14-017A))
 +[[https://www.us-cert.gov/ncas/alerts/TA14-017A|Government Alert (TA14-017A)]]
  
 +\\
 ===== Memcrashed "Kill-Switch" and Data Theft and/or Corruption ===== ===== Memcrashed "Kill-Switch" and Data Theft and/or Corruption =====
  
-While Corero Network Security, a DDoS & Network Security Solutions provider, was investigating this issue, they discovered a 'kill-switch' that they claim has been tested on live attack servers and has been 100% effective.  The memcached "flush_all" command will invalidate the cache. If memcached is reloaded before the vulnerability patch is applied, then the "kill-switch' will be necessary again. The command should be sent to the originating attack servers,  with a single 'flush_all' command issued to each attacker. (( https://www.corero.com/resources/ddos-attack-types/memcached-attack.html +While Corero Network Security, a DDoS & Network Security Solutions provider, was investigating this issue, they discovered a 'kill-switch' that they claim has been tested on live attack servers and has been 100% effective.  The memcached "flush_all" command will invalidate the cache. If memcached is reloaded before the vulnerability patch is applied, then the "kill-switch' will be necessary again. The command should be sent to the originating attack servers,  with a single 'flush_all' command issued to each attacker. 
- ))+ 
 +((https://www.corero.com/resources/ddos-attack-types/memcached-attack.html)) 
 +[[https://www.corero.com/resources/ddos-attack-types/memcached-attack.html|What is a Memcached Attack?]]
  
 Unfortunately, while discovering the 'kill-switch', the team at Corero also found that this vulnerability is even worse than previously thought. This is mainly due to the way UDP and memcached functions.  Unfortunately, while discovering the 'kill-switch', the team at Corero also found that this vulnerability is even worse than previously thought. This is mainly due to the way UDP and memcached functions. 
Line 53: Line 77:
  
 There are several ways to flush Memcached. I prefer Telnet, so here is the sequence of commands you would use to accomplish this: There are several ways to flush Memcached. I prefer Telnet, so here is the sequence of commands you would use to accomplish this:
 +<code>
   telnet memcached-server-ip PORT   telnet memcached-server-ip PORT
   flush_all   flush_all
   quit   quit
 +</code>
  
 Where: Where:
Line 63: Line 88:
  
 **NOTE**: Make sure you apply the patch and then restart Memcached: **NOTE**: Make sure you apply the patch and then restart Memcached:
 +<code>
   service memcached restart   service memcached restart
 +</code>
  
 +\\
 ===== What Can I Do?  ===== ===== What Can I Do?  =====
  
-Memcached developers have patched this (( https://github.com/memcached/memcached/wiki/ReleaseNotes156 +Memcached developers have patched this ((https://github.com/memcached/memcached/wiki/ReleaseNotes156)) [[https://github.com/memcached/memcached/wiki/ReleaseNotes156|memcached patch notes]], but if you are running a version of Memcached older than version 1.5.6, you //must// either update or apply the required configuration changes yourself.
- )), but if you are running a version of Memcached older than version 1.5.6, you //must// either update or apply the required configuration changes yourself.+
  
 If you are concerned about your Memcached installation, please don't hesitate to open a ticket with Knownhost support so we can check the version and patch/upgrade if necessary.  If you are concerned about your Memcached installation, please don't hesitate to open a ticket with Knownhost support so we can check the version and patch/upgrade if necessary. 
  
 +\\
 ===== How Can I Secure My Memcached Installation By Editing the Existing Configuration?  ===== ===== How Can I Secure My Memcached Installation By Editing the Existing Configuration?  =====
  
Line 95: Line 121:
  
 Restart Memcached: Restart Memcached:
 +<code>
   service memcached restart   service memcached restart
 +</code>
  
 Check the process now: Check the process now:
Line 126: Line 153:
 {{:developmental:current-running-process-shows-as-secured.png?nolink&1200|}} {{:developmental:current-running-process-shows-as-secured.png?nolink&1200|}}
  
-TCP is not currently considered a high-risk Memcached amplification vector because TCP queries are far less vulnerable to spoofing than UDP queries due the fact that you would also need to spoof TCP header fields relating to the state of the connection. Sending a spoofed TCP packet requires the attacker to guess the sequence number, which cannot be done reliably. (( http://seclists.org/basics/2007/Jul/59 )) +TCP is not currently considered a high-risk Memcached amplification vector because TCP queries are far less vulnerable to spoofing than UDP queries due the fact that you would also need to spoof TCP header fields relating to the state of the connection. Sending a spoofed TCP packet requires the attacker to guess the sequence number, which cannot be done reliably. 
  
 +(( http://seclists.org/basics/2007/Jul/59 ))
 +[[http://seclists.org/basics/2007/Jul/59|Why TCP is more secure than UDP?]]
  
 +\\
 ===== Installing and Securing Memcached for EasyApache 4 CentOS 7/CPanel Servers: Everything You Need to Know ===== ===== Installing and Securing Memcached for EasyApache 4 CentOS 7/CPanel Servers: Everything You Need to Know =====
  
 At the time of writing this article, installing memcached via yum installs an outdated version. I'll show you how to install it, check the version, and if necessary, secure the installation here.  We will also add it to cPanel's Service Manager so that the service will be restarted upon reboot and we will also configure the firewall to ignore the false positive alerts that it will send about this foreign process that it does not recognize as legitimate (we know its legitimate because we installed it and configured it to be safe ;-) ).  At the time of writing this article, installing memcached via yum installs an outdated version. I'll show you how to install it, check the version, and if necessary, secure the installation here.  We will also add it to cPanel's Service Manager so that the service will be restarted upon reboot and we will also configure the firewall to ignore the false positive alerts that it will send about this foreign process that it does not recognize as legitimate (we know its legitimate because we installed it and configured it to be safe ;-) ). 
- 
  
 First, log into your server via SSH as root (reminder: the default Knownhost SSH port is set to 2200): First, log into your server via SSH as root (reminder: the default Knownhost SSH port is set to 2200):
 +<code>
   ssh root@<IP> -p2200   ssh root@<IP> -p2200
 +</code>
  
 Now, install memcached: Now, install memcached:
 +<code>
   yum install memcached   yum install memcached
 +</code>
  
 Now, to check to see what version is installed by using telnet as shown in the screenshot below using the command "telnet localhost 11211". Once connected, type 'version' and press //Enter//, and the Memcached version will be returned. If the version is less than 1.5.6, then you will need to secure your memcached installation following the steps below.  Now, to check to see what version is installed by using telnet as shown in the screenshot below using the command "telnet localhost 11211". Once connected, type 'version' and press //Enter//, and the Memcached version will be returned. If the version is less than 1.5.6, then you will need to secure your memcached installation following the steps below. 
Line 148: Line 178:
  
 Now, to secure the installation, use nano to edit the file /etc/sysconfig/memcached: Now, to secure the installation, use nano to edit the file /etc/sysconfig/memcached:
 +<code>
   nano /etc/sysconfig/memcached   nano /etc/sysconfig/memcached
 +</code>
  
 You will change the following line: You will change the following line:
 +<code>
   OPTIONS=""   OPTIONS=""
 +</code>
  
 To this: To this:
 +<code>
   OPTIONS="-l 127.0.0.1 -U 0"   OPTIONS="-l 127.0.0.1 -U 0"
 +</code>
  
 And then Ctrl X + y +Enter to exit and save the file.  And then Ctrl X + y +Enter to exit and save the file. 
  
 Now, start memcached.: Now, start memcached.:
 +<code>
   service memcached start   service memcached start
 +</code>
  
 Add memcached to the Service Manager so that it will be monitored and restarted as needed (following reboots, etc): Add memcached to the Service Manager so that it will be monitored and restarted as needed (following reboots, etc):
 +<code>
   chkconfig memcached on   chkconfig memcached on
 +</code>
  
 Lastly, we need to make sure that the CSF/LFD firewall that is installed by default with Knownhost servers is configured appropriately. Port 11211 should not be opened in the firewall by default, but it never hurts to check: Lastly, we need to make sure that the CSF/LFD firewall that is installed by default with Knownhost servers is configured appropriately. Port 11211 should not be opened in the firewall by default, but it never hurts to check:
 +<code>
   grep 11211 /etc/csf/csf.conf   grep 11211 /etc/csf/csf.conf
 +</code>
  
 If you get no output, then the port is not open. It only needs to be accessible locally, so the port should be closed.  If you get no output, then the port is not open. It only needs to be accessible locally, so the port should be closed. 
  
 Next, we need to edit the firewall's process ignore file so that the firewall won't send us alerts about this newly added process. Open the process ignore file with nano: Next, we need to edit the firewall's process ignore file so that the firewall won't send us alerts about this newly added process. Open the process ignore file with nano:
 +<code>
   nano /etc/csf/csf.pignore   nano /etc/csf/csf.pignore
 +</code>
  
 Scroll to the bottom of the file and add either of these (use the executable option if you are likely to adjust the configuration of your installation, or be prepared to edit the cmd option in the firewall's process ignore file when you do): Scroll to the bottom of the file and add either of these (use the executable option if you are likely to adjust the configuration of your installation, or be prepared to edit the cmd option in the firewall's process ignore file when you do):
 +<code>
   exe:/usr/bin/memcached   exe:/usr/bin/memcached
   cmd:/usr/bin/memcached -u memcached -p 11211 -m 64 -c 1024 -l 127.0.0.1 -U 0   cmd:/usr/bin/memcached -u memcached -p 11211 -m 64 -c 1024 -l 127.0.0.1 -U 0
 +</code>
  
 Exit with Ctrl X + y + Enter and restart the firewall so that these changes take effect: Exit with Ctrl X + y + Enter and restart the firewall so that these changes take effect:
 +<code>
   csf -ra   csf -ra
 +</code>
  
 That's it! Enjoy your secured Memcached installation! :-)  That's it! Enjoy your secured Memcached installation! :-) 
Line 192: Line 231:
  
 **Note**: If you also require the Memcache PHP extension, you can use the following command for EasyApache 4, however, you will need to replace ## with the PHP version that you intend to install it for: **Note**: If you also require the Memcache PHP extension, you can use the following command for EasyApache 4, however, you will need to replace ## with the PHP version that you intend to install it for:
 +<code>
   /opt/cpanel/ea-php##/root/usr/bin/pecl install memcache   /opt/cpanel/ea-php##/root/usr/bin/pecl install memcache
 +</code>
  
 Use the following command to confirm the installation (again, replacing ## with the PHP version required, e.g., 54, 55, 56, 70, 71, etc): Use the following command to confirm the installation (again, replacing ## with the PHP version required, e.g., 54, 55, 56, 70, 71, etc):
 +<code>
   /opt/cpanel/ea-php##/root/usr/bin/php -m | grep memcache   /opt/cpanel/ea-php##/root/usr/bin/php -m | grep memcache
 +</code>
  
 As always, if you have any questions or any trouble with this installation on your Knownhost server, please open a support request and we'll be glad to help! As always, if you have any questions or any trouble with this installation on your Knownhost server, please open a support request and we'll be glad to help!
  
 +\\
 ===== cPanel and Memcached ===== ===== cPanel and Memcached =====
  
-Note that cPanel has memcached and memcache RPMs in their experimental EasyApache 4 repos at this time.  (( https://features.cpanel.net/topic/memcached-in-easyapache4 )) These are not yet secured via SASL authentication, but the statement in their Feature Request seems to indicate that they plan to use SASL to secure Memcached. You may want to watch for these to be moved into the stable repo so that you can replace your Memcached installation with a cPanel-supported installation when that time comes. Until then, make sure to either use a later version of Memcached, or to open a ticket with Knownhost so that we can secure it for you. +Note that cPanel has memcached and memcache RPMs in their experimental EasyApache 4 repos at this time.  ((https://features.cpanel.net/topic/memcached-in-easyapache4))[[https://features.cpanel.net/topic/memcached-in-easyapache4|cPanel preview]]. These are not yet secured via SASL authentication, but the statement in their Feature Request seems to indicate that they plan to use SASL to secure Memcached. You may want to watch for these to be moved into the stable repo so that you can replace your Memcached installation with a cPanel-supported installation when that time comes. Until then, make sure to either use a later version of Memcached, or to open a ticket with Knownhost so that we can secure it for you. 
  
developmental/memcrashed-what-is-it-memcache.1576853344.txt.gz · Last modified: 2019/12/20 08:49 by Karson N.