CloudFlare, a content delivery network and web security provider used by millions of websites, has admitted that a severe security vulnerability has caused sensitive data to be exposed across a number of different websites. It has been dubbed Cloudbleed and is said to be worse than Heartbleed, a similar bug from 2014. What exactly is Cloudbleed and how could you be affected by it? Let’s find out.
What Is Cloudbleed?
If you don’t care much about the technical details here’s the TL;DR version: A bug that causes the exposure of sensitive information on websites hosted behind Cloudflare. Affected services have been turned off. Skip to the next section to see how this bug might affect you.
Cloudflare is a content delivery network (CDN) and web security service provider that is used by over 5.5 million websites around the world. It acts as a proxy; a user that makes a HTTP request for content on website that sits behind Cloudflare, which then fetches the content from the web server. HTML code in the web content is parsed through Cloudflare on its edge servers for optimisation and security, and then sent to the user as a HTTP response.
Earlier today, Google security researcher Tavis Ormandy revealed that he found a buffer overflow bug on Cloudflare services that made some HTML pages hosted behind the company’s CDN respond to page requests with random bits of data. (Just to clarify, the data comes Cloudflare’s server memory.) Pages with a specific combination of unbalanced html tags triggered the issue.
“I’m finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We’re talking full HTTPS requests, client IP addresses, full responses, cookies, passwords, [encryption] keys, data, everything.”
Authentication tokens, HTTP POST bodies, requests made over HTTPS and other sensitive information were also leaked. According to Cloudflare, customer SSL private keys were not leaked: “Cloudflare has always terminated SSL connections through an isolated instance of NGINX that was not affected by this bug.”
The bits of data were interspersed through the HTTP response and typical users probably wouldn’t have a clue about it. But that doesn’t make up for the fact that getting hold of the exposed data is as simple as making a HTTP request in a browser. Compounding the issue is search engines like Google and Bing caching the content; that means people could get to the information by crafting queries on search engines.
Cloudflare has revealed that the problem was related to its HTML parser and affected three security features: email obfuscation, Server-side Excludes and Automatic HTTPS Rewrites. The affected features have been turned off or patched and Cloudflare is working with Google to clean up the cached data.
According to Cloudflare: “The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that’s about 0.00003% of requests).”
But Ormandy argued that Cloudflare was being disingenuous and claimed, based on Google’s cached data, the bug had existed for months.
How Does This Affect Me?
A lot of Cloudflare’s services rely on parsing HTML pages and modifying them through the company’s edge servers.
Of course, if you’re using Cloudflare services in front of your website, this has the potential to impact you as the bug could have exposed sensitive data that flowed between your back-end servers and end-users via CloudFlare’s proxies.
But even if you don’t use Cloudflare directly, there is a chance that websites you visit and web services you use may have been affected.
Cloudflare has identified 770 unique URIs (covering 161 unique domains) that contained leaked memory and were cached by search engines like Google, Bing and Yahoo. The company said: “The leaked memory has been purged with the help of the search engines”.
Cloudflare hasn’t released a list of websites that have been impacted by this bug but a user by the name of ‘pirate’ has assembled a list of around four million domains that use the company’s services (and could potentially be affected) on GitHub. Bear in mind that this isn’t a comprehensive list.
Critically, major sites and even password managers may have had sensitive data exposed even if it was sent through HTTPS, which is meant to encrypt the traffic. According to Ormandy:
I don’t know if this issue was noticed and exploited, but I’m sure other crawlers have collected data and that users have saved or cached content and don’t realize what they have, etc. We’ve discovered (and purged) cached pages that contain private messages from well-known services, PII from major sites that use Cloudflare, and even plaintext API requests from a popular password manager that were sent over https (!!).
There were claims that data from password manager 1Password was leaked but the company has since come out to refute this. In a blog post addressing customers, 1Password said:
Your actual data is encrypted with three layers (including SSL/TLS), and the other two layers remain secure even if the secrecy of an SSL/TLS channel is compromised.
The three layers are:
- SSL/TLS. This is what puts the “S” in HTTPS. And this is what data may have been exposed due to the Cloudflare bug during the vulnerable period.
- Our own transport layer authenticated encryption using a session key that is generated using SRP during sign in. The secret session keys are never transmitted.
- The core encryption of your data. Except for when you are viewing your data on your system, it is encrypted with keys that are derived from your Master Password and your secret Account Code. This is the most important layer, as it would protect you even if our servers were to be breached. (Our servers were not breached.)
It’s a good reminder that you shouldn’t just rely on one security method to protect your web assets.
What Should I Do Now?
Security researcher Ryan Lackey made the following recommendations on a Medium post.
For site operators using Cloudflare: while it may be disruptive, you should seriously consider forcing a password change for all of your users. Be careful not to “scare” them, but it may be prudent to push such a change.
There are very real costs to forcing a user password change - users might lose trust in your service, and it’s an inconvenience. It doesn’t appear large numbers of credentials have been compromised, so for a consumer service with limited risk to compromised accounts, it may not be worth the effort.
For administrator credentials, or for any sites processing highly sensitive information through Cloudflare, the lack of a quantifiable maximum exposure probably means it is worth forcing a password update.
Additionally, any sites not on Cloudflare but with a lot of users who use Cloudflare-hosted sites (essentially any large or consumer sites on the Internet) should consider forcing user password changes in case their users had re-used the same passwords on each site.
If you’re a user of online services (that’s almost everyone), you should consider changing your passwords.
Full disclosure: Lifehacker Australia uses Cloudflare but we’ve checked with our IT team and confirmed that we are not affected by Cloudbleed.
Comments
10 responses to “Cloudflare’s ‘Cloudbleed’ Bug Exposes Sensitive Data: Who Is Affected?”
I don’t like the amount of results that “.gov.au” brings up in that list of 4.3 million sites :/
How can you confirm that you were not affected? Every site behind CloudFlare is potentially vulnerable.
Hi there,
Sorry for the late response. We received an email from Cloudflare to confirm that they looked into the issue and found that we were not affected.
Cheers,
Spandas
I have been passed a similar message from other vendors using Cloudflare. The wording of the message appears sneaky, and underplays the potential impact IMHO. It’s more a case that they haven’t found any data leaked from your organisation to date, but that doesn’t mean it’s not out there – they’re just not aware of it.
Hi askvictor,
Sorry for the late response!
I’m not at liberty to talk about how our back-end works in detail. All I can say is that I’ve talked with the IT team at length about this issue and we’re in the clear!
I’m sorry I can’t go into more detail.
Hope that helps!
Cheers,
Spandas
Three features were dangerous. If your site used any of them then Cloudflare should have emailed you already.
1. Email obfuscation : If you were using the cloudflare obfuscation tool
2. Serverside Excludes : If you were using <–SSE–>protected text<–/SSE–> in your HTML
3. Automatic HTTPS rewrites : If your server is HTTP and you had cloudflare HTTPS it.
The github list is a dump of cloudflares DNS. It is stupid and misleading, if not even dangerous that he has irresposibly done this. 99% of websites on it, or more, are NOT affected by this bug.
Unless something has been restated, my understanding was that it ultimately didn’t matter whether a site contained the errors that caused the memory leak, as much as it was using Cloudflare. The memory leak wasnt just affecting the data of one site, it was affecting all data in memory.
As in sites A, B, and C use Cloudflare, sites A and C are properly programmed but B isn’t. Therefore, site B’s requests may return data for sites A, B, and C, because the error affects anything in physical memory, which isn’t dedicated to any one site.
Hi there,
Thanks for the comment. Yes, that’s correct. In this article I noted that unbalanced HTML tags triggered the issue.
Edit: To avoid further confusion, I’ve added a clarification in the article.
Cheers!
Spandas
@spandas fyi I’m “pirate” the author of the repo, and we’ve shortened it down to ~4,000,000 domains after removing duplicates. Can you correct the article to reflect that?
Hi Nick,
Thanks for getting in touch.
I’ve updated this information in the article 🙂
Thanks again!
Spandas