CloudFlare, a content delivery network and web security provider used by millions of websites, has admitted that a severe security vulnerability has caused sensitive data to be exposed across a number of different websites. It has been dubbed Cloudbleed and is said to be worse than Heartbleed, a similar bug from 2014. What exactly is Cloudbleed and how could you be affected by it? Let’s find out.
What Is Cloudbleed?
If you don’t care much about the technical details here’s the TL;DR version: A bug that causes the exposure of sensitive information on websites hosted behind Cloudflare. Affected services have been turned off. Skip to the next section to see how this bug might affect you.
Cloudflare is a content delivery network (CDN) and web security service provider that is used by over 5.5 million websites around the world. It acts as a proxy; a user that makes a HTTP request for content on website that sits behind Cloudflare, which then fetches the content from the web server. HTML code in the web content is parsed through Cloudflare on its edge servers for optimisation and security, and then sent to the user as a HTTP response.
Earlier today, Google security researcher Tavis Ormandy revealed that he found a buffer overflow bug on Cloudflare services that made some HTML pages hosted behind the company’s CDN respond to page requests with random bits of data. (Just to clarify, the data comes Cloudflare’s server memory.) Pages with a specific combination of unbalanced html tags triggered the issue.
“I’m finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We’re talking full HTTPS requests, client IP addresses, full responses, cookies, passwords, [encryption] keys, data, everything.”
Authentication tokens, HTTP POST bodies, requests made over HTTPS and other sensitive information were also leaked. According to Cloudflare, customer SSL private keys were not leaked: “Cloudflare has always terminated SSL connections through an isolated instance of NGINX that was not affected by this bug.”
The bits of data were interspersed through the HTTP response and typical users probably wouldn’t have a clue about it. But that doesn’t make up for the fact that getting hold of the exposed data is as simple as making a HTTP request in a browser. Compounding the issue is search engines like Google and Bing caching the content; that means people could get to the information by crafting queries on search engines.
Cloudflare has revealed that the problem was related to its HTML parser and affected three security features: email obfuscation, Server-side Excludes and Automatic HTTPS Rewrites. The affected features have been turned off or patched and Cloudflare is working with Google to clean up the cached data.
According to Cloudflare: “The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that’s about 0.00003% of requests).”
But Ormandy argued that Cloudflare was being disingenuous and claimed, based on Google’s cached data, the bug had existed for months.
How Does This Affect Me?
A lot of Cloudflare’s services rely on parsing HTML pages and modifying them through the company’s edge servers.
Of course, if you’re using Cloudflare services in front of your website, this has the potential to impact you as the bug could have exposed sensitive data that flowed between your back-end servers and end-users via CloudFlare’s proxies.
But even if you don’t use Cloudflare directly, there is a chance that websites you visit and web services you use may have been affected.
Cloudflare has identified 770 unique URIs (covering 161 unique domains) that contained leaked memory and were cached by search engines like Google, Bing and Yahoo. The company said: “The leaked memory has been purged with the help of the search engines”.
Cloudflare hasn’t released a list of websites that have been impacted by this bug but a user by the name of ‘pirate’ has assembled a list of around four million domains that use the company’s services (and could potentially be affected) on GitHub. Bear in mind that this isn’t a comprehensive list.
Critically, major sites and even password managers may have had sensitive data exposed even if it was sent through HTTPS, which is meant to encrypt the traffic. According to Ormandy:
I don’t know if this issue was noticed and exploited, but I’m sure other crawlers have collected data and that users have saved or cached content and don’t realize what they have, etc. We’ve discovered (and purged) cached pages that contain private messages from well-known services, PII from major sites that use Cloudflare, and even plaintext API requests from a popular password manager that were sent over https (!!).
There were claims that data from password manager 1Password was leaked but the company has since come out to refute this. In a blog post addressing customers, 1Password said:
Your actual data is encrypted with three layers (including SSL/TLS), and the other two layers remain secure even if the secrecy of an SSL/TLS channel is compromised.
The three layers are:
- SSL/TLS. This is what puts the “S” in HTTPS. And this is what data may have been exposed due to the Cloudflare bug during the vulnerable period.
- Our own transport layer authenticated encryption using a session key that is generated using SRP during sign in. The secret session keys are never transmitted.
- The core encryption of your data. Except for when you are viewing your data on your system, it is encrypted with keys that are derived from your Master Password and your secret Account Code. This is the most important layer, as it would protect you even if our servers were to be breached. (Our servers were not breached.)
It’s a good reminder that you shouldn’t just rely on one security method to protect your web assets.
What Should I Do Now?
Security researcher Ryan Lackey made the following recommendations on a Medium post.
For site operators using Cloudflare: while it may be disruptive, you should seriously consider forcing a password change for all of your users. Be careful not to “scare” them, but it may be prudent to push such a change.
There are very real costs to forcing a user password change - users might lose trust in your service, and it’s an inconvenience. It doesn’t appear large numbers of credentials have been compromised, so for a consumer service with limited risk to compromised accounts, it may not be worth the effort.
For administrator credentials, or for any sites processing highly sensitive information through Cloudflare, the lack of a quantifiable maximum exposure probably means it is worth forcing a password update.
Additionally, any sites not on Cloudflare but with a lot of users who use Cloudflare-hosted sites (essentially any large or consumer sites on the Internet) should consider forcing user password changes in case their users had re-used the same passwords on each site.
If you’re a user of online services (that’s almost everyone), you should consider changing your passwords.
Full disclosure: Lifehacker Australia uses Cloudflare but we’ve checked with our IT team and confirmed that we are not affected by Cloudbleed.