The first step is to execute the beforeunload event handler if a document is already loaded. It allows the page to prompt the user whether they want to leave the current one. It is useful in cases such as forms, where the result has not been submitted, so the form data is not lost when moving to a new document. The user can cancel the navigation and no more work will be performed.

If there is no beforeunload handler registered or the user agreed to proceed, the next step is the browser making a network request to the specified URL to retrieve the contents of the document to be rendered. Chromium’s implementation uses the term “provisional load” to describe the state it is in at the start of the network request. Assuming no network level error is encountered (e.g. DNS resolution error, socket connection timeout, etc.), server responds with data and the response headers come first. Once the headers are parsed, they give enough information to determine what needs to be done next.

The HTTP response code allows the browser to know whether one of these conditions has occured:

• A successful response follows (2xx)
• A redirect has been encountered (response 3xx)
• An HTTP level error has occurred (response 4xx, 5xx)

There are two cases where a navigation can complete without resulting in a new document being rendered. The first one is HTTP response code 204 and 205, which tell the browser that the response was successful, but there is no content that follows, therefore the current document must remain active. The other case is when the server responds with a header indicating that the response must be treated as a download. All the data read by the browser is then saved to the local filesystem based on the browser configuration.

The server can also sent a redirect, upon which the browser makes another request based on the HTTP response code and the additional headers. It continues following redirects until either an error or success is encountered.

Once there are no more redirects, if the response is not a 204/205 or a download, the browser reads a small chunk of the actual response data that the server has sent. By default this is used to perform MIME type sniffing, to determine what type of response the server has sent. This behavior can be suppressed by sending a “X-Content-Type-Options: nosniff” header as part of the response headers. At this point the browser is ready to switch to rendering the new document. In Chromium’s implementation, this term used for this point in time is “commit”. Basically the browser has committed to rendering the new document and remove the old one.

However, before the commit is performed, the old document needs to be notified that it is going away, so the browser executes the unload event handler of the old document, if one is registered. Once that is complete, the old document is no longer active, the new document is committed, and in strict terms, the navigation is complete.

The astute reader will realize that even though I said navigation is complete, the user actually doesn’t see anything at this point. Even though most people use the word navigation to describe the act of moving from one page to another, I think of that process as consisting of two phases. So far I have described the navigation phase and once the navigation has been committed, the browser moves into the loading phase. It consists of reading the remaining response data from the server, parsing it, rendering the document so it is visible to the user, executing any script accompanying it, as well as loading any subresources specified by the document. The main reason for splitting it into those two phases is how errors are handled.

This brings us back to the case where the server responds with an error code. When this happens, the browser still commits a new document, but that document is an error page it either generates based on the HTTP response code or reads as the response data from the server. On the other hand, if a successful navigation has committed a real document from the server and has moved to the loading phase it is still possible to encounter an error, for example a network connection can be terminated or times out. In that case the browser is displaying as much of the new document as it has parsed.

Chromium exposes the various stages of navigation and document loading through methods on the WebContentsObserver interfce.

Navigation

• DidStartNavigation - invoked at the point after executing the beforeunload event handler and before making the initial network request.
• DidRedirectNavigation - invoked every time a server redirect is encountered.
• ReadyToCommitNavigation - invoked at the time the browser has determined that it will commit the navigation.
• DidFinishNavigation - invoked once the navigation has committed. It can be either an error page if the server responded with an error code or the browser has switched to the loading phase for the new document on successful response.

Document loading

• DidStartLoading - invoked when a navigation is about to start, after executing the beforeunload handler.
• DocumentLoadedInFrame - invoked when the document itself has completed loading, however it does not mean that all subresources have completed loading.
• DidFinishLoad - invoked when the document and all of its subresources have been loaded.
• DidStopLoading - invoked when the document, all of its subresources, all subframes and their subresources have completed loading.
• DidFailLoad - invoken when the document load failed, for example due to network connection termination before reading all of the response data.

Hopefully this post gives a good introduction to navigations in the browser and should be a good base to build on for future posts.

Chromium uses the operating system process as a unit of isolation. It uses Blink to render web documents, which it runs in restricted renderer processes. The sandbox does not allow any renderer processes to communicate between each other and the only way to achieve that is to use the browser process as an intermediary. This design allows us to isolate web pages from each other and potentially have a different level of privileges for each process.

Before delving into the actual models the browser supports, there are couple of more bits of detail to cover - cross-process navigation and SiteInstance caveats.

Cross-process navigation

A tab in the browser UI gets a visual representation of the web page from the renderer process and draws it as its content. When navigating from one page to another, the browser makes a network request and gives the response to Blink for rendering. Often, the same instance of the rendering engine, running in the same process, is used. However, in many cases, navigations can result in a new renderer process being created and a brand new instance of Blink being instantiated. The response is handed off to the new renderer process and the tab is then associated with the new process.

The ability to perform cross-process navigations is a core part of Chromium’s design. It incurs the cost of starting a new process, however it also improves performance, as using a new process has a clear memory space, free of fragmentation. Abandoning the old process can be quick (process kill) and also helps mitigate memory leaks, as that process exits and its memory is released back to the operating system. Most importantly, though, changing processes under the hood is a key building block of the security model.

Chromium’s security model also allows for different privilege levels for content being rendered. In general any content coming from the web is considered lowest privilege level. Chromium internal pages, such as chrome://settings, require more privileges, as they need to read or modify settings or data available only in the browser process. The security model does not allow pages from different privilege levels to use the same process, so a cross-process navigation is enforced when crossing privilege level.

SiteInstance caveats

Previous posts in this series described SiteInstance and said that in the example setup, all of a.com, b.com, c.com, d.com were SiteInstances. This is the ideal model to use and is the goal of the “Site Isolation” project, but currently Chromium does not reflect this in reality. Here is a list of caveats that apply to the default Chromium configuration at the time of this post:

• SiteInstance is assigned only to the top-level frame by default. Subframes share the same one as the main frame.

• SiteInstance does not always reflect the URL of the current document. Once the SiteInstance URL is set, it doesn’t change, even though the frame can navigate across many different Sites. However, SiteInstance can change when navigating cross-site.[1]

Chromium avoids process swaps on cross-site renderer-initiated navigations (e.g. link clicks) because those would be likely to break script calls on windows that expect to communicate with each other (and thus break compatibility with the web). In contrast, it tends to use process swaps on cross-site browser-initiated navigations (e.g. typing URL in the omnibox) because the user is making an effort to leave the site, so it’s not as bad to break the script calls.

Process models

To help illustrate the difference between the different process models, I have included screenshots of the Chromium Task Manager showing the processes and what URLs they are rendering. The setup I have used is the following:

• Tab navigated to http://tests.netsekure.org/main-b-c.html. The main document includes two iframes - one to https://google.com and one to https://github.com. It also contains a button that opens a new tab through window.open() call. It causes the new tab to be in the same BrowsingInstance as the tab that opened it.

• Newly opened tab from the initial tab, which is navigated to http://tests.netsekure.com/main-sub.c.html. Notice that it is different top level domain from the initial tab (.org vs .com). It includes an iframe to https://pages.github.com.

• User opened tab navigated to http://tests.netsekure.com/empty.html

• User opened tab navigated to http://tests.netsekure.com/to_slow.html

At the time of this post, the current set of process models is:

• Single process

• Process per tab

• Process per Site

• Process per Site instance (default)

• Site per process (experimental)

Single process

This is a mode in which Chromium does not use multiple processes. Rather, it combines all of its parts into a single process. It is also a mode in which there is no sandboxing, as the browser needs access to both the network and the filesystem. It exists mainly for testing and it should never be used!

Process per tab

This process model is the simplest one to understand and is what most people intuitively think is the mode of operation of the browser. Each tab gets a dedicated sandboxed process that runs the Blink rendering engine. Navigations do not usually change processes. Note however that since the security model does not allow for content with different privileges to live in the same process, it does actually change processes on privilege change. An example would be navigation from https://dev.chromium.org to chrome://settings.

Process per Site

In this process model, each Site gets mapped to a single process. When multiple tabs are navigated to the same Site, they will all share the same process. Navigations can change processes.

It is not the default model, since running multiple tabs with heavy web pages, such as Google Docs, leads to low performance - too much contention on the main thread, memory fragmentation, etc.

Process per Site instance

This is the default process model for Chromium. Each SiteInstance is mapped to a process by default. Multiple tabs navigated to the same Site end up in separate SiteInstances, therefore they reside in separate processes. Navigations also can change processes. All the SiteInstance caveats apply and not the idealized version of SiteInstance.

Site per process

This is an experimental process model for developing the “Site Isolation” project. It comes closer to the desired design for Chromium, where there is a SiteInstance for each frame. Additionally, it is using the idealized definition of SiteInstance, where only URLs from the same SiteInstance can be loaded in the same process. Navigations can change processes in any frame on a page, whereas all other process models support changing processes only on the top frame.

I hope these posts have helped demystify a bit how the Chromium makes decisions on which process to use for specific tab and URL. If there are other clarifications I can make, feel free to ping me over on Twitter and I would be happy to.

When a browser is navigated to an URL, it makes a network request to the server specified for the document identified in the URL. The response is a document *, which is then parsed and rendered in a window. Those should be familiar, since they correspond to the identically named objects in JavaScript. This holds true for iframes as well, which have their own window objects, which host the respective documents. The HTML spec uses different naming for window - “browsing context”, while it keeps document as the same concept. There are a few types defined by the standard:

• top-level browsing context - the main window for a page
• nested browsing context - window embedded in a different window, for example through <iframe> tag
• auxiliary browsing context - a top-level browsing context “related” to another browser context, or put in simpler speak - any window created through window.open() API, or a link with target attribute.

I will use frame to refer generically to any browsing context - be it a page or an iframe, as they are basically the same concept with two different names based on the role they play.

There are two concepts the HTML spec defines that are important to understand. The first one is “reachable browsing context”. This is somewhat intuitive, as all frames that are part of a web page are reachable to each other. In JavaScript this is exposed through the window.parent and window.frames properties. In addition, related browsing contexts are reachable too, by using the return value of window.open() and the window.opener property. For example, if we have a page with two iframes, which opens a new window with an iframe, then all of the frames are reachable.

The set of reachable frames - all of them in the above case - form the other concept the standard defines - “unit of related browsing contexts”. It is important because documents that want to communicate with other documents are allowed to do so only if they are part of the same unit of related browsing contexts. Internally, the Chromium source code uses the BrowsingInstance class to represent this concept. For the sake of brevity, I’ll use this name from here on.

When two documents want to communicate with each other, they need to have a reference to the window object of the target document. Any frame in a BrowsingInstance can get a reference to any other frame in the same BrowsingInstance since they are all reachable by definition.

How documents can interact with each other is governed by the same origin policy. When documents are from the same origin or can relax their origin to a common one, they are allowed to access each other directly. Cross-origin documents on the other hand are not allowed such access. So a BrowsingInstance can be split in sets of frames and grouped by the origin they are from. But recall that we can’t easily use the origin as a security principle in Chromium. This is why we use the concept of SiteInstance - the set of frames in a BrowsingInstance which host documents from the same Site. It is vital to remember that the Chromium browser process makes all of its process model and isolation decisions based on SiteInstance and not based on origins.

The HTML spec requires all same origin documents, which are part of the same unit of related browsing contexts, to run on the same event loop - or in other words the same thread of execution within a process. This means that all frames which are part of the same SiteInstance must execute on the same thread, however different SiteInstances can run on different ones. In the example above, the two pages are in the same BrowsingInstance because they are related through the window.open() call. The different SiteInstances should be for a.com, b.com, c.com, d.com.

Overall it all boils down to the following rules that Chromium needs to abide by:

• All frames within a BrowsingInstance can reference each other.
• All frames within a SiteInstance can access each other directly and must run on the same event loop.
• Frames from different SiteInstances can run on separate event loops.

Phew! Now there is enough background to start delving into the details of Chromium’s implementation of these concepts from the HTML spec and its process allocation model.

* Unless the result is a file to be downloaded or handled by external application.

I decided to write up an explanation of the default process model and how it actually works. The goal is for it to be comprehensible to as many people as possible, not requiring a degree in Computer Science. However, basic familiarity with the web platform (HTML/JS) is expected. In order to get to it, there are some concepts that need to be defined, so this is the first post in a series, which will explain some of Chromium’s internals and demystify some parts of the HTML spec.

I have found the easiest mental model of the Chromium architecture to be that of an operating system - a kernel running in high privilege level and a number of less privileged usermode application processes. In addition, the usermode processes are isolated from each other in terms of address space and execution context.

The equivalent of the kernel is the main process, which we call the “browser process”. It runs with the privileges of the underlying OS user account and handles all operations that require regular user permissions - communication over the network, displaying UI, rendering, processing user input, writing files to disk, etc. The equivalent of the usermode processes are the various types of processes that Chromium’s security model supports. The most common ones are:

• Renderer process - used for parsing and rendering web content using the Blink rendering engine
• GPU process - used for communicating with the GPU driver of the underlying operating system
• Utility process - used for performing untrusted operations, such as parsing untrusted data
• Plugin process - used for running plugins

They all run in a sandboxed environment and are as locked down as possible for the functionality they perform.

In the modern operating systems design, the principle of least privilege is key and separation between different user accounts is fundamental. User account is a basic unit of separation and I would refer to from here on to this concept as “security principal”. Each operating system has a different way of representing security principals, for example UIDs in Unix and SIDs in Windows, etc. On the web, the security principal is the origin - the combination of the scheme, host, and port of the URL the document has originated from. Access control on the web is governed by the Same Origin Policy (SOP), which allows documents that belong to the same origin to communicate directly with each other and access each other synchronously. Two documents that do not belong to the same origin cannot access each other directly and can only communicate asynchronously, usually through the postMessage API. Overall, the same origin policy has worked very well for the web, but it also has some quirks, which make it unsuitable to treat origins as the security principal for Chromium.

The first reason comes from the HTML specification itself. It allows documents to “relax” its origins for the purpose of evaluating same origin policy. Since the origin contains the full domain of the host serving the document, it can be a subdomain, for example “foo.bar.example.com”. In most cases, however, the example.com domain has full control over all of its subdomains and when documents that belong in separate subdomains want to communicate directly, they are not allowed due to the restrictions of same origin policy. To allow this scenario to work, though, documents are allowed to change their domain for the purposes of evaluating SOP. In the case above, “foo.bar.example.com” can relax its domain up to example.com, which would allow any document on example.com itself to communicate with it. This is achieved through the “domain” property of the document object. It does come with restrictions though.

In order to understand the restrictions of what document.domain can be set to, one needs to know about the Public Suffix List and how it fits in the security model of the web. Top-level domains like “com”, “net”, “uk”, etc., are treated specially and no content can (should) be hosted on those. Each subdomain of a top-level domain can be registered by different entity and therefore must be treated as completely separate. There are cases, however, where they aren’t a top-level domain, but still act as such. An example would be “co.uk”, which serves as a parent domain for commercial entities in the UK to register their domains. Because those cases are effectively in the role of a top-level domain, but are not one, the public suffix list exists as a comprehensive source for browsers and other software to use.

Now that we know about PSL, let’s get back to document.domain. A document cannot change its domain to be anything completely generic or very encompassing, such as “.”. Browsers allow documents to relax their domain up the DNS hierarchy. To use the example from above, “foo.bar.example.com” can set its domain to “bar.example.com” or “example.com”. However, since “.com” is a top-level domain, allowing the document to set its domain to “.com” will lead to security problems. It will allow the document to potentially access documents from any other “.com” domain. Therefore browsers disallow setting the domain to any value in the Public Suffix List and enforce that it must be a valid format of a domain under one of the entries in the PSL. This concept is often referred to as “eTLD+1” - effective top-level domain (a.k.a. entry in the PSL) + one level of domains under it. I will use this naming for brevity from here on.

It is this behavior defined by the HTML spec allowing documents to change their origins that gives us one of the reasons we cannot use the origin as a security principal in our model. It can change in runtime and security decisions made in earlier point in time might no longer be valid. The consistent part that can be taken from the origin is only the eTLD+1 part.

The next oddity of the web is the concept of cookies. It is quite possibly the single most used feature of the web today, but it has its fair share of strange behaviors and brings numerous security problems with it. The problems stem from the fact that cookies don’t really play very well with origins. Recall that origin is the tuple (scheme, host, port), right? The spec however is pretty clear that “Cookies do not provide isolation by port”. But that isn’t all, the spec goes to the next paragraph and says “Cookies do not provide isolation by scheme”. This part has been patched up as the web has evolved though and the notion of “Secure” attribute on cookies was introduced. It marks cookies as available only to hosts running over HTTPS and since HTTP is the other most used protocol on the web, the scheme of an origin is somewhat better isolated and port numbers are completely ignored when cookies are concerned. So basically it is impossible to use origin as a security principal to use and perform access controls against cookie storage.

Finally there is enough background to understand the security principal used by Chromium - site. It is defined as the combination of scheme and the eTLD+1 part of the host. Subdomains and port numbers are ignored. In the case of https://foo.bar.example.com:2341 the effective site for it will be https://example.com. This allows us to perform access control in a web compatible way while still providing a granular level of isolation.

Taking Chromium and forking it is fairly easy process, just clone the repository. Make all the changes you would like to do - add missing features, include enhancements, create a totally new UI - it is only limited by one’s imagination. Building the binary from the source code is a little bit laborious, though not too hard. It does take beefy hardware and some time. Once it is built, publishing it is deceptively easy. However, what comes next?

Software in today’s world is not static. As a colleague of mine likes to say - it is almost like a living organism and continuously evolves. There is no shipping it as it was the norm in the ‘90s. The web is in a constant release mode and its model of development has trickled to client side software - be it desktop or mobile apps. Chromium has adopted this model from its initial release and is updating on a very short cycle - currently averaging six weeks between stable releases and two weeks between intermediate stable updates. It is this constant change that makes forking it a bit more challenging. However, there are few steps that one can take to ensure a smoother ride.

Infrastructure

With constantly changing codebase, having a continuous build system is a must for project as big as Chromium and is very useful even for much smaller projects. Setting one up from the get go will be tremendously useful if there is more than one developer working on the code. Its value is even higher if the project needs to build on more than one platform.

What is more important and I would argue it is a must - using a continuous integration system. Running tests on each commit (or thereabout) to ensure there are no breaking changes. It is a requirement for any software project that needs to be in a position to release a new version at any point in time.

The system used in the Chromium project - buildbot - is actually open source and can be adapted to most projects.

Making changes

The most important action one can take when forking Chromium is to study the design of the browser before diving in and making any changes. There are multiple components and layers involved, which interact through well defined interfaces. Understanding the architecture and the patterns used will pay off tremendously in the long run.

Chromium has two main component layers - content and chrome. The former is what implements the barebones of a browser engine - networking stack, rendering engine, browser kernel, multiprocess support, navigation and session history, etc. The chrome layer is built on top of content to implement the browser UI, extensions system, and everything else visible to the user that is not web content.

Each layer communicates with the upper ones through two main patterns - observer and delegate interfaces. Using those interfaces should be the preferred way of extending the browser and building on top of it. Whenever this is not possible, changes to the core are needed. I would strongly suggest preferring to upstream those, if possible of course. It will make maintaining the fork much easier by reduing the burden of keeping up with changes and also shares the improvements with the whole community!

Finally, do yourself a favor to keep you sane in the long run - write tests for all the features you are adding or changes made. It is the only way to ensure that long term the regressions and bug rate is manageable. It will save your sanity!

Keep it moving

The Chromium codebase changes constantly and gets around 100 commits each day. The sane way to keep up with the rate of change is to rebase (or merge) your code on tip-of-tree (ToT) daily or at most weekly. Letting more time lapse makes resolving conflicts a lot harder.

Updating the install base is key to long term success. The update cient used in Chrome on Windows, called Omaha, is also open source. The server side code is not available, though, since it depends heavily on how Google’s internal infrastructure is setup. However the protocol used to communicate between the client and the server is publicly documented.

Development for Chromium relies quite a bit on mailing lists. Subscribing to the two main ones - chromium-dev@chromium.org and blink-dev@chromium.org - is very helpful. It is place where major changes are announced, discussion on development happens, and questions about Chromium development are answered. The security team has a dedicated list for discussions - security-dev@chromium.org.

Keep it secure

Security is one of the core tenets of Chromium. Keeping up with security fixes can be a challenging task, which is best solved by keeping your code always rebased on tip-of-tree. If this is not possible, it is best to subscribe to the security-notify@chromium.org list. It is the communication mechanism the security team uses to keep external projects based on Chromium up-to-date with all the security bugfixes happening in the project.

Plugins

The web is moving more and more to a world without plugins. For me, this is a very exciting time, as plugins usually tend to weaken the browser security. There are two plugins bundled with Chromium to produce Chrome - Adobe Flash Player and a PDF viewer. The latter is now an open source project of its own - PDFium. It can be built and packaged with Chromium, though the same care should be taken as with the browser itself - keep it up-to-date.

–

Overall, maintaining a fork of Chromium isn’t trivial, but it isn’t impossible either. There are a bunch of examples, including the successful migration of the Opera browser from their own rendering engine to building on top of the Chromium content module.

Last, but not least - feel free to reach out and ask questions or advice.

Software security is a very complex field and a asymmetric problem space. Arguments whether defense or offense is harder have been fought for a long time and they will likely never stop. I think most of us can agree on those two statements:

• offense needs to find only a handful of problems and work hard to turn them into a compromise
• defense needs to architect software to be resilient and work hard to ideally (though not practically) not introduce any problems

Each side requires unique skills and is extremely rare for people to be really good at both. What really irks me is that lots of people in the security industry tend to bash the other side. It is easy, one understands their problem space very very well and knows how hard it is to be an expert. Also, it feels that the other side is not that hard. I mean, how hard can it be, right? Wrong!

In this post I will pick the side of the defender, since this is where I spend most of my time. The example I will use is the recent events with the Aviator browser, because it is near and dear to my heart. One thing I want to make clear from the get go - I totally respect their efforts and applaud them for trying. Forking Chromium is not a small feat and not for the faint of heart. The goals for Aviator are admirable and we definitely need people to experiment with bold and breaking changes. It is through trial and error that we learn, even in proper engineering disciplines :). What can we use more of the security industry?

Humbleness!

It is no surprise people on the offensive side bash software developers for “stupid” mistakes, since the grass is always greener on the other side. The problem is that many trivialize the work required to fix those mistakes. Some are indeed easy. Fixing a stack-based buffer overflow is not too hard. In other cases, it is harder due to code complexity or just fundamental architecture of the code.

What humbles me personally is having tried the attack side. It is not too bad if you want to exploit a simple example problem. Once you try to exploit a modern browser, it is a completely different game. I admire all the exploit writers for it and am constantly amazed by their work. Same goes for a lot of the offensive research going on.

I have secretly wished in the past for some of the offensive folks to try and develop and ship a product. When WhiteHat Security released the Aviator browser, I was very much intrigued how it will develop. It is not a secret that Jeremiah Grossman and Robert Hansen have given lots of talks on how the web is broken and how browser vendors do not want to fix certain classes of issues. They have never been kind in their remarks to browser vendors, but now they have become one. I watched with interest to see how they have mitigated the issues they have been discussing. Heck, I wanted to see clickjacking protection implemented in Chromium, since it is the authors of Aviator that found this attack vector and I have personally thought about that problem space in the past.

Chris Palmer and I have played around with the idea of “Paranoid Mode” in Chromium and as a proof of concept we have written Stannum (source) to see how far we can push it through the extensions APIs. It is much safer to add features to Chromium using extensions than writing C++ code in the browser itself¹. So when Aviator was announced and released initially, I reached out to WhiteHat Security to discuss whether the features they have implemented in C++ could be implemented through the extensions API. My interest was primarily motivated by learning what they have done and what are the limitations of the extensions subsystem. Unfortunately, the discussion did not go far :(.

Where do I believe they could have done better? You might have guessed it right - being humble. The marketing for Aviator is very bold - “the most secure and private Web browser available”. This is a very daring claim to make, hard promise to uphold and anyone who has been in security should know better. Securing a complex piece of software, such as a browser, is a fairly hard task and requires lots of diligence. It takes quite a bit of effort just to stay on top of all the bugs being discovered and features committed, let alone develop defenses and mitigations.

Releasing the source for Aviator was a great step by WhiteHat. It gives us a great example to learn from. Looking at the changes made, it is clear that most the code was written by developers who are new to C++. When making such bold statments, I would have expected more mature code. Skilled C++ developers that understand browsers are rare, but it is a problem that can be solved. It takes a lot of time, effort and desire for someone to learn to use the language and most importantly understand the architecture of the browser. Unfortunately, I did not see any evidence that whoever wrote the Aviator specific code did any studying of the source code or attempted to understand how Chromium is written and integrate the changes well.

What really matters at the end of the day, though, is not the current state of a codebase. After all, every piece of software has bugs. I believe there is one key factor which can determine long term success or failure:

Attitude!

Security vulnerabilities are a fact of life in every large enough codebase. Even in the project I work on we have introduced code that allowed geohot to pull off his total ChromeOS pwnage! We owned up to it, the bug was fixed and we looked around to ensure we did not miss other similar instances.

However, what I was most disappointed by was the reaction from WhiteHat when a critical vulnerability was found in the Aviator specific code:

“Yup, Patches welcome, it’s open source.” ²

Our industry would go further if we follow a few simple steps:

• Do not trivialize the work of the opposite side, it is more complex than it appears on the surface.
• When working on a complex software or problem, study it first
• Share ideas and collaborate
• Own up to your mistakes
• Be humble
  

<sup>1. Even Blink is starting to implement rendering engine features in JavaScript.
2. Nevermind that there is no explanation on how to build Aviator, so one can actually verify that the fix works.</sup>

I have been using separate browsers for a while now to isolate generic web browsing from “high value” browsing, such as banking or administration of this blog. The reason I’ve been doing this is that a compromise during generic web browsing is going to be isolated to the browser being used and the “high value” browser will remain secure (barring compromise of the underlying OS).

Recently I’ve decided to give the experimental Chrome feature - "isolated apps" - a try, especially since I’ve recently started working on Chrome and will likely contribute to taking this feature to completion. Chrome already does have a multi-process model in which it uses different renderer processes, which if compromised, should limit the damage that can be done to the overall browser. One of the limitations that exists is that renderer processes have access to all of the cookies and other storage mechanisms in the browser (from here on I will only use cookies, though I mean to include other storage types as well). If an attacker can use a bug in WebKit to get code execution in the renderer process, then this limitation allows requesting your highly sensitive cookies and compromising those accounts. What “isolated apps” helps to solve is isolating storage for web applications from the generic web browsing storage, which helps solve the problem of a compromised renderer stealing all your cookies. In essence, it simulates running the web application in its own browser, without the need to branch out of your current browser. The aim of this blog post is not to describe how this will work, but how to take advantage of this feature. For the full details, read the paper by Charlie Reis and Adam Barth (among others), which is underlying the “isolated apps” work.

In the spirit of my “30 days with …” experiments, I created manifests for the financial sites I use and for my blog. I wanted to see if I will hit any obvious breaking cases or degraded user experience with those “high value” sites. A sample manifest file looks like this:

{
  "name": "netsekure blog",
  "version": "1",
  "app": {
    "urls": [ "*://netsekure.org/" ],
    "launch": {
      "web_url": "https://netsekure.org/wp-admin/"
    },
    "isolation": [ "storage" ]
  },
  "permissions": [ "experimental" ]
}

The way to read the file is as follows:

• The “urls” directive is an expression defining the extent encompassed by the web application.
• The “web_url” is the launch page for the web app, which provides a good known way to get to the application.
• The “isolation” directive is instructing Chrome to isolate the storage for this web app from the generic browser storage.

Once the manifest is authored, you can place it in any directory on your local machine, but ensure the directory has no other files. To actually take advantage of this, you need to do a couple of things:

• Enable experimental APIs either through chrome://flags or through the command line with –enable-experimental-extension-apis.
• Load the manifest file as an extension. Go to the Chrome Settings page for Extensions, enable “Developer Mode”, and click on “Load unpacked extension”, then navigate to the directory where the manifest file resides and load it.

Once you have gone through the above steps, when you open a new tab, it will have an icon of the isolated web application you have authored. You can use the icon to launch the web app, which will use the URL from the manifest and will run in a separate process with isolated storage.

Now that there is an isolated app installed in Chrome, how can one be assured that this indeed works? There are a couple of things I did to confirm. First, when a Chrome web app is opened, the Chrome Task Manager shows it with a different prefix. Generic web pages start with “Tab: ” followed by the title of the currently displayed page. The prefix for the apps is “App: ”, which indicates that the browser treats this tab as a web application.

In addition to seeing my blog being treated differently, I wanted to be sure that cookies are not shared with the generic browser storage, so I made sure to delete all cookies for my own domain in the “Cookies and Other Data” settings panel. As expected, but still to my surprise, the site continued functioning, since deleting the cookies only affected the general browser storage and my isolated app cookies were not cleared. This intrigued me as to where those cookies are being stored. It turns out, since this is still just an experimental feature, there is no UI to show the storage for the isolated app yet. If you want to prove this to yourself, just like I wanted to, you have to use a tool to let you peek into a SQLite database, which stores those cookies in a file very cleverly named - Cookies. The Cookies db and the cache are located in the directory for your current profile in a subdirectory “Isolated Apps” followed by the unique ID of the app, as generated by Chrome. You can find the ID on the Extensions page, if you expand to see the details for the web app you’ve “installed”. In my case on Windows, the full directory is “%localappdata%\Google\Chrome\User Data\Default\Isolated Apps\dgipdfobpcceghbjkflhepelgjkkflae”. Here is an example of the cookies I had when I went and logged into my blog:

As you can see, there are only two cookies, which were set by WordPress and no other cookies are present.

Now, after using isolated apps for 30 days, I haven’t found anything that was broken by this type of isolation. The sites I’ve included in my testing, besides my blog, are bankofamerica.com, americanexpress.com, and fidelity.com*. The goal now is get this to more usable state, where you don’t need to be a Chrome expert to use it ;).

* Can’t wait for all the phishing emails now to start arriving ;)

“pass-the-hash attacks are functionally equivalent to cookie stealing attacks”

If you think about the pass-the-hash attack, it requires administrator privileges, which means you can get LocalSystem level privileges, at which point you own the operating system. Then you extract the user’s hash out of memory or from the SAM database and you inject them into the attacker’s machine. Then you rely on single-sign on built on top of NTLM/Kerberos to authenticate to remote resources.

What if we assume the following mapping: OS -> Browser, LocalSystem code execution -> Browser code execution, User’s hash -> User’s cookie, Single Sign On -> HTTP session with cookies?

It can be easily observed that the pass-the-hash attack is equivalent to attacker having code execution in the context of the browser, stealing the user’s cookies, injecting them into the attacker’s browser, and accessing remote resources. Actually, in the web world, one doesn’t even need code execution in the browser to steal the user’s cookies, it can be done through purely web based attacks.

Is it possible to defend against attacker using your cookies? It is extremely hard, because to the remote server, your cookie is *you*. From that perspective, a Windows domain is no different than web HTTP domain, so remote resources have no way of telling apart the real you and someone having your token, be it a password hash or a cookie. I haven’t gone through the thought experiment of mapping best practices for securing against cookie stealing attacks to see if those will nicely map into best practices for defending against pass-the-hash attacks, so I’d leave that as an exercise for the reader.

• Any CA can issue a certificate for any web site on the Internet, which I call “certificate binding” problem
• Revocation checking as deployed is ineffective

To counteract these deficiencies, multiple proposals have either existed or emerged. The most notable two are using DNSSEC for certificate binding and Convergence, which is based on independent system of notaries. While both have merits, I believe both have sufficient problems that will prevent them from being deployed widely, which is required for any successful change.

Convergence
Moxie Marlinspike has the right idea about trust agility and his proposal, which he calls Convergence, has a very good foundation. Where I believe it falls short is the fact that many corporate networks block outgoing traffic to a big portion of the Internet. Unless all notaries are white-listed for communication, traffic to those will be blocked, which will prevent Convergence from working properly. Also, the local caching creates a problem with timely revocation - if a certificate is found to be compromised, then until the cache expires, it will still be treated as a valid one.

      struct {
          CertificateStatusType status_type;
          select (status_type) {
              case ocsp: OCSPStatusRequest;
          } request;
      } CertificateStatusRequest;

      enum { ocsp(1), (255) } CertificateStatusType;

The structure is extensible allowing for any other type of certificate status to be requested, as long as it is defined. I can easily see this message defining DNSSEC and Convergence as values of CertificateStatusType, then define the appropriate format of the request sent by the client. Conveniently, the response from the server is also very much extensible:

      struct {
          CertificateStatusType status_type;
          select (status_type) {
              case ocsp: OCSPResponse;
          } response;
      } CertificateStatus;

      opaque OCSPResponse<1..2^24-1>;

Currently, the only value defined is for OCSP response, which is treated as opaque value as far as TLS is concerned. Nothing prevents whatever information the above proposals return from being transmitted as opaque data to the client.

Just like Moxie explored in his presentation, using the server to do the work of retrieving trust verification data preserves the privacy of the client. It does put some extra burden on the server to have proper connectivity, but that is much more manageable and totally under the control of the administrator.

It is true that there will have to be change implemented by both clients and servers which that will take time. I fully acknowledge that fact. I do believe though, that using the Certificate Status Request is the most logical piece of infrastructure to use to avoid all possible network related problems and provide an inline, fresh, binding trust verification data from the server to the client.

One thing I have not yet answered to myself is - how do we make any new model to fail safe. Having hard failure and denying the user access has been problem forever, but if we keep on failing unsafe, we will continue chasing the same problem into the future.

When TLS server is configured to ask for client authentication, it sends as part of the handshake the TLS CertificateRequest message. The TLS 1.2 RFC defines the message as follows:

      struct {
          ClientCertificateType certificate_types<1..2^8-1>;
          SignatureAndHashAlgorithm
            supported_signature_algorithms<2^16-1>;
          DistinguishedName certificate_authorities<0..2^16-1>;
      } CertificateRequest;

where the supported_signature_algorithms is addition in the 1.2 version of the TLS protocol. The certificate_authorities part of the message is further defined:

opaque DistinguishedName<1..2^16-1>;

When the server sends this message, it optionally fills in the certificate_authorities part of the message with a list of distinguished names of acceptable CAs on the server. The main reason for this list is for the server to help the client in narrowing down the set of acceptable certificates to choose from. For example, if the server only accepts certificates issued by the company private CA, there is no need for the client to send a certificate issued by a public CA, as the server won’t trust it. Nothing in the RFC prevents the client from sending any certificate, but it is in the best interest of the client to send appropriate certificate.

On Windows, the way the TLS is implemented, the server picks all the certificates that are present in the “Local Computer” “Trusted Root Certification Authorities” store (or in short the local machine root store). With Windows Server 2008 and later, the default list of trusted authorities is very small as I’ve described in a previous post, so including those distinguished names in the message does not pose a problem. However, if the server has most of the trusted roots installed or has additional root certificates, it is possible for the combined length of the distinguished names to exceed the limit of the TLS record size, which is 2¹⁴ (16384) bytes. The TLS standard supports this, as it breaks messages up into records and a single message can span multiple records - this is called record fragmentation. Windows does not implement this part of the RFC though, so it cannot send messages that are bigger than what a TLS record can hold. In most cases this works just fine, but in this particular instance it is a problem.

Now, what can be done to solve this problem. The are two solutions - either decrease the list of root certificates in the message or do not send that list at all (allowed by the RFC). The former approach is possible, but is more error prone and I wouldn’t recommend it for most people. I would argue that the latter approach is the preferred one, but I always get backlash when I propose this. If one were to think about the original purpose of this message and the way Windows has implemented this, it will be easier to understand why this is better. Remember:

• the server wants to “help” the client do an informed decision on which certificate to send as its identity
• Windows sends the contents of the local machine root store as the list

If we take those two factors together, the end result is that the server is not helping *at all* the client to make a good decision.  A few sources - my own research, ssllabs.com’s SSL Survey, and the EFF SSL Observatory - point out that 10-15 root CAs issue the majority of the certificates seen on the web, therefore if we send 16k worth of these, the probability that any certificate the client has is *not* issued by someone in the list is close to zero. Therefore, in this configuration, the list the server presents to the client doesn’t effectively filter down the set of certificates on the client. It is almost equivalent to sending an empty list to the client and ask it to chose randomly.

In short, if you are hitting this problem, you are better of using Method 3 described in KB 933430 and setting SendTrustedIssuerList to 0, which disables sending of the list of CAs than any other method.

Let’s start with some preliminary background information:

• Certificates are validated through a process of building a chain up to a trust anchor, the underlying of which is constructing a graph of nodes (certificates) and edges (issuance relationships) and then the graph is traversed to find all possible complete chains (in most cases just one). The smaller the graph is, the quicker it is to find a complete chain. Once the chain is complete, the certificate at the root of the chain is checked for trust. If the chain ends in a certificate present in the list of trusted root certificates and all other verifications pass, the certificate validation is successful.
• Microsoft has a specific program called “Microsoft Root Certificate Program”, which is how certificate authorities (CAs) submit their root certificates for inclusion in Windows. The end result of this program is a *fixed* list of root certificates that Windows considers trusted. The entire list is available on TechNet and is updated whenever there are any changes. This list (or the equivalent of it at the time) was present in full in the root certificate store in Windows XP and earlier, but starting with Vista, this default list in the root certificate store is much smaller in order to increase performance while validating certificates. If a chain ends up in root certificate which is part of the Root Program but is not present in the list of trusted roots currently on the machine, Windows downloads the appropriate root certificate directly from Windows Update. The full process for all versions of the OS is described in a KB article. I’m not going to rehash the explanation of how it works, but the key point is that only those certificates accepted through the root program will be downloaded from Windows Update.

The IE low integrity processes are not instructing the broker to do anything, it all happens under the hood in the crypto APIs. Now, here is why the IE behavior is observed. You go to a web site, which certificate is issued by a CA not yet in your trusted root list. Because IE uses the standard cryptography API that Windows provides, when certificate validation is performed, Windows itself (not IE, nor its broker process) goes and fetches the root certificate from Windows Update *if* that certificate is part of the Root Program. The same behavior will be seen for any program that is using the same API to do certificate validation. Chrome as far as I know uses the Windows crypto APIs to do certificate validation and relies on the trusted roots list from Windows, so if you browse with Chrome, you will see the exact same thing happen (though I haven’t verified it personally).

Some people have asked me how they can prevent Windows from going to Windows Update in such cases. This can be achieved by disabling automatic root update through policy as described on TechNet. It should be noted that one should exercise caution in doing so, because disabling root update means that Windows will no longer manage certificate trust for you. You will have to manage the set of trusted root certificates on your own. To that end, import the full list of CAs part of the Root Program such that those are available in the list of trusted roots. Microsoft provides a small program – rootsupd.exe, as part of KB931125 which accomplishes this task. With this approach, you have control over trust management, but you need to keep the list updated whenever the set of roots in the root program changes. Microsoft has a wiki page which includes the information about new certificates that are part of an update. This page has RSS feed available which you can subscribe to so that you are notified of new updates, which makes keeping track of the updates an easy job.

I hope this helps explain how this all works and clarify that there is no real violation of the No-Write-Up policy, even though it might seem like it from high level.

All major browsers are updating to blacklist those certificates and I’d suggest you install updates as soon as you can to prevent possible attacks. Since none of the certificates have been seen in the wild, the chance is very very slim, but it doesn’t hurt to do an update.

It was very interesting to see Jacob Appelbaum correlate multiple sources of information to discover this independently from the actual announcement. I’ve been advocating that bad guys are already doing this, but very few people believe it. Now I hope this demonstrates that automated correlation can reveal lots of data. Furthermore Adam Langley has a good discussion why revocation has problems and we should be looking into how to improve the state of it.

Advisories:

• Comodo
• Microsoft
• Mozilla, follow up
• US-CERT
• Google

A bit of background first. TLS extensions are a method of extending the TLS protocol without having to change the specification of the core protocol and are described in RFC 4366. It is defined as arbitrary extra data that can be appended to the ClientHello and/or ServerHello messages (which are the first messages sent by each side). Servers are supposed to ignore data following the ClientHello if they don’t understand it.

Since the TLS extensions were not a formal RFC in the past, some server implementations were written to fail requests which have more data following the ClientHello message, which makes them non-interoperable with clients that send TLS extensions. This is the precise reason why RFC 5746 has adopted the idea of Signaling Cipher Suite Value (SCSV) to avoid breaking interoperability with servers not accepting TLS extensions. The recommended approach, though, is to use the TLS extension defined by the RFC.

Now here are the important details. By default, any version of Windows prior to Vista did not send TLS extensions when using the TLSv1.0 protocol. With the new update, this has changed and if TLSv1.0 is enabled, then the renegotiation indication extension will be sent as part of the TLS handshake, as recommended. So the small set of servers (no one that I know of knows the actual percentage of such servers) which do not tolerate this behavior will cause interoperability problems. This is where the UseScsvForTls setting described in the Microsoft KB comes in. Setting the registry key to non-zero value will cause the SSL/TLS stack to generate TLS ClientHello messages containing SCSV and without extensions, so interoperability with such servers can be restored. As far as the other two keys, AllowInsecureRenegoClients and AllowInsecureRenegoServers, they control the compatible vs strict mode and will not make any difference on the structure of the messages on the wire. The only effect is whether communication is allowed to continue with unpatched party or not.

If you are like me and rely heavily on SSL for the admin area, you might already know and be frustrated by the busted defaults for most plugins. In their defense, it is a WordPress issue, since their built in functions return http based URLs instead of https when hosted over SSL. I believe this is being addressed in 3.0, but so far I’ve had to patch each new release of WP to get safety, but I digress.

The only issue with the plugin itself is that it reported the username I picked as used, even though it was not. You can safely ignore this or if you want to do it right, apply the following patch:

---<
@@ -171,6 +171,7 @@ pf.testNumberCompleted = function(data) {
                                                        CURLOPT_POST => true,
                                                        CURLOPT_POSTFIELDS => $post,                                                                                                                       CURLOPT_RETURNTRANSFER => TRUE,
+                                                       CURLOPT_SSL_VERIFYPEER => false                                                                                                            );                                                                                                                                                 $curl = curl_init();
                                                foreach ($curl_options as $option => $value) curl_setopt($curl, $option, $value);
---<

With that taken care of, one can sign up for an account right from the plugin. Once it is all said and done this plugin is AWESOME. Now I can actually authorize each login to my blog and ensure that even if someone were to get their hands on my password, they won’t be able to login to the blog. And the best part is that the call feature is actually free!

Thanks PhoneFactor for a great plugin!

I don’t know the reason to be honest, as this is one of the basic rules of security. Few months ago I did it for my blog, since I don’t see a valid reason why anyone coming across the Internet needs to have “drop table” privilege or similar on my blog’s database. I looked around for a convenient mailing list to post my idea to and unfortunately I couldn’t find one, so I gave up on trying to message the idea directly to WordPress developers, but now I’ve decided to at least post it for people to take advantage of it.

Here is what I have done that should give you basic level of privilege separation with just a simple WordPress tweak.
First, you need to have the two separate database users - admin and public. We also need to assign the proper privileges/permissions (after all this is the main idea):

public -> db: [SELECT,INSERT]
admin -> db: [SELECT,INSERT,UPDATE,CREATE,DELETE,ALTER,DROP,INDEX,CREATE TEMPORARY TABLES,LOCK TABLES,CREATE VIEW,SHOW VIEW]

Since I have comments enabled, I had to give the public user the INSERT privilege, but at least I have taken away the 10 extra privs away from it. If you don’t allow comments on the blog, you can even remove the INSERT functionality, though I haven’t tested this one and don’t know if anything else would break.

The only other thing left to do is configure WordPress to pick the right database user. This is accomplished easily through wp-config.php:

if ( defined('WP_ADMIN') || defined('DOING_CRON') )  {
  define('DB_USER', 'admin');
  define('DB_PASSWORD', 'admin_password');
} else {
  define('DB_USER', 'public');
  define('DB_PASSWORD', 'public_password');
}

I’ve only had this customization interfere once in my almost half year since I’ve implemented it. It was during WordPress upgrade, which required a change to the database schema. In such cases, it is trivial to restore back to the default behavior for the duration of the upgrade and then go back to separate users.

I hope this helps people reduce some of the attack surface on their blogs and ideally WordPress will do this natively in the future.

CN=Equifax Secure Global eBusiness CA-1, O=Equifax Secure Inc., C=US
7e784a101c8265cc2de1f16d47b440cad90a1945

OU = VeriSign Trust Network, OU = “© 1998 VeriSign, Inc. - For authorized use only”, OU = Class 3 Public Primary Certification Authority - G2, O = “VeriSign, Inc.”, C = US
85371ca6e550143dce2803471bde3a09e8f8770f

OU=Class 3 Public Primary Certification Authority, O=VeriSign, Inc., C=US
742c3192e607e424eb4549542be1bbc53e6174e2

OU=Equifax Secure Certificate Authority, O=Equifax, C=US
d23209ad23d314232174e40d7f9d62139786633a

CN=GTE CyberTrust Global Root, OU=“GTE CyberTrust Solutions, Inc.”, O=GTE Corporation, C=US
97817950d81c9670cc34d809cf794431367ef474

CN=Entrust.net Secure Server Certification Authority, OU=© 1999 Entrust.net Limited, OU=www.entrust.net/CPS incorp. by ref. (limits liab.), O=Entrust.net, C=US
99a69be61afe886b4d2b82007cb854fc317e1539

CN=AddTrust External CA Root, OU=AddTrust External TTP Network, O=AddTrust AB, C=SE
02faf3e291435468607857694df5e45b68851868

E=premium-server@thawte.com, CN=Thawte Premium Server CA, OU=Certification Services Division, O=Thawte Consulting cc, L=Cape Town, S=Western Cape, C=ZA
627f8d7827656399d27d7f9044c9feb3f33efa9a

OU=Go Daddy Class 2 Certification Authority, O=“The Go Daddy Group, Inc.”, C=US
2796bae63f1801e277261ba0d77770028f20eee4

CN = GlobalSign Root CA, OU = Root CA, O = GlobalSign nv-sa, C = BE
b1bc968bd4f49d622aa89a81f2150152a41d829c

The last one I’ve included for completeness, since I don’t really need it, but I had to enable it to access openssl.org over https. It is currently not trusted.

While this is a good list of certs to enable for security geeks like myself, I’m not quite sure how feasible this is today for the average user, so I wouldn’t recommend doing this to your parents' computer. Even for me it was hard to realize that application failures (such as twhirl completely stopping to work) are due to a root certificate no longer being trusted and SSL connections failing. I also had to look at the wire traffic on a few occasions where the UI would never expose the “I want to see which certificate is failing” option.

One needs to be very careful which certs are disabled. Since it is hard to troubleshoot failures that result from disabling trusted roots, reading up and getting familiar with how certificates work is a great idea. Firefox has its own certificate storage, completely separate from the OS, so messing with it is not as big of an issue, as any errors are isolated to Mozilla applications. Here are some resources for Windows (which affects IE and Chrome):

• There is a list of mandatory certificates that Windows needs to operate, which is listed here.
• There is a great overview of how the trusted roots certificates work on Windows and explains why people see things “change” under the hood.
• Also, in newer versions there seem to be a lot more control on how certificates are validated and what roots are trusted.
• The list of CAs that Windows trusts.

I hope this information is helpful to people. Feel free to ping me with questions you might have related to this small project.

The different browsers store certificates differently. IE, Chrome, and I believe Safari as well (haven’t tested it) on Windows use the OS built-in certificate infrastructure, while Firefox uses its own certificate storage. As such, here are the steps you need to take for the two different cases:

IE, Chrome (Safari?)

You need to run the certmgr.msc utility (either through Start->Run/Search or from a command prompt). This will launch the UI used to manage the certificate stores in Windows for the current user.

CertMgr Certificate Stores

The “Third-Party Root Certification Authorities” stores all the trusted 3rd party CAs. You will find either a fairly small set of those if Windows hasn’t downloaded the full list, or quite a bit of them after the full list has arrived. To disable the root certificates, select the ones you want and drag them to the “Untrusted Certificates” store and drop them under the “Certificates” subfolder. This instructs the certificate infrastructure in Windows to not trust these certificates. The result is that even though you have the certificates in other stores, the operations will fail. The “Untrusted Certificates” store trumps all others, so you don’t have to worry about forgetting a certificate somewhere else.

Keep in mind that doing this in Windows will affect all programs that use SSL/TLS and certificates. I’ve broken my twitter client for example by removing all CAs from the trusted list : ).

Firefox

You will need to click on Tools->Options, select the Advanced category, select the Encryption, click View Certificates, and click on the Authorities tab. This will open up a window with all the trusted certificate authorities. For each of those, once you select it, you can click on the “Edit” button and you will see a window that looks like this:

Firefox Trusted CA

This CA is trusted for all 3 types of identification. To disable the certificate, just uncheck all the check boxes and click Ok:

Firefox Disabled CA

The result is that this certificate is no longer trusted to vouch for the identity of anything. You need to repeat the process for all the certificates you want to disable and I don’t know of an easy way to automate this. For the certificates listed as “Builtin Object Token”, Marsh Ray has tried deleting them and claims that this results in disabling them (since they are built-in and cannot be deleted) after restarting Firefox.

After you have disabled the CA certificates, you can expect SSL/TLS connections to fail if the certificate is issued by a disabled CA.

Have fun browsing with minimized attack surface : )

I will run in this mode for 30 days, at the end of which I will report how many root certificates I had to enable to allow me to go through life. In the meantime, I am tweeting every time I need to enable a CA along with the site that needed it.

It is a fun ride so far, so let’s see where it is going to take me.

How many trusted roots does the average person need in their browser?

To answer the question, I wrote a small tool to collect the root cert for a list of sites and ran it on a sets of data - a mix of known SSL/TLS sites and hosts on the Alexa Top 1 Million list. So out of 350k hosts queried, I was able to collect 50812 entries. The stats are fairly interesting and somewhat expected. The number of certificate authorities that have issued more than 50 certificates for that set of data is 37.

While I was gathering the data, it became known that even Mozilla includes some root certificates that don’t have complete clarity of ownership.

My plan now is to remove most of the CA root certificates that ship in browsers. It will be informative to see what breaks and how many issues I run into. After a month or so of usage, I will post the details and hopefully it will be an easy guide as to the smallest set of trusted CAs to have and not be impacted in daily business. Granted this will be a US centric list, most international users can probably add one or two trusted roots that are for CAs issuing country specific certificates.

What follows is the list of root certificates (also available as text file) sorted with decreasing popularity. The format is “Number of issued certs | Friendly name | Subject”.

7519 | GeoTrust | OU=Equifax Secure Certificate Authority, O=Equifax, C=US
4277 | USERTrust | CN=AddTrust External CA Root, OU=AddTrust External TTP Network, O=AddTrust AB, C=SE
4007 | Go Daddy Class 2 Certification Authority | OU=Go Daddy Class 2 Certification Authority, O="The Go Daddy Group, Inc.", C=US
3701 | VeriSign Class 3 Public Primary CA | OU=Class 3 Public Primary Certification Authority, O="VeriSign, Inc.", C=US
2948 | USERTrust | CN=UTN-USERFirst-Hardware, OU=http://www.usertrust.com, O=The USERTRUST Network, L=Salt Lake City, S=UT, C=US
2649 | thawte | E=premium-server@thawte.com, CN=Thawte Premium Server CA, OU=Certification Services Division, O=Thawte Consulting cc, L=Cape Town, S=Western Cape, C=ZA
2077 |  | E=info@plesk.com, CN=plesk, OU=Plesk, O=Parallels, L=Herndon, S=Virginia, C=US
1898 | Equifax Secure Global eBusiness CA-1 | CN=Equifax Secure Global eBusiness CA-1, O=Equifax Secure Inc., C=US
1806 | VeriSign | OU=VeriSign Trust Network, OU="(c) 1998 VeriSign, Inc. - For authorized use only", OU=Class 3 Public Primary Certification Authority - G2, O="VeriSign, Inc.", C=US
1580 |  | E=webaster@localhost, CN=localhost, OU=none, O=none, L=Sometown, S=Someprovince, C=US
1461 |  | E=root@localhost.localdomain, CN=localhost.localdomain, OU=SomeOrganizationalUnit, O=SomeOrganization, L=SomeCity, S=SomeState, C=--
1378 | thawte | E=server-certs@thawte.com, CN=Thawte Server CA, OU=Certification Services Division, O=Thawte Consulting cc, L=Cape Town, S=Western Cape, C=ZA
1366 | VeriSign | CN=VeriSign Class 3 Public Primary Certification Authority - G5, OU="(c) 2006 VeriSign, Inc. - For authorized use only", OU=VeriSign Trust Network, O="VeriSign, Inc.", C=US
1189 |  | E=info@plesk.com, CN=plesk, OU=Plesk, O="SWsoft, Inc.", L=Herndon, S=Virginia, C=US
922 | Entrust | CN=Entrust.net Secure Server Certification Authority, OU=(c) 1999 Entrust.net Limited, OU=www.entrust.net/CPS incorp. by ref. (limits liab.), O=Entrust.net, C=US
902 | GlobalSign | CN=GlobalSign Root CA, OU=Root CA, O=GlobalSign nv-sa, C=BE
783 | GTE CyberTrust Global Root | CN=GTE CyberTrust Global Root, OU="GTE CyberTrust Solutions, Inc.", O=GTE Corporation, C=US
737 | CúOúMúOúDúO | CN=COMODO Certification Authority, O=COMODO CA Limited, L=Salford, S=Greater Manchester, C=GB
692 |  | E=ca@snakeoil.dom, CN=Snake Oil CA, OU=Certificate Authority, O="Snake Oil, Ltd", L=Snake Town, S=Snake Desert, C=XY
461 | DigiCert | CN=DigiCert High Assurance EV Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
394 | Starfield Class 2 Certification Authority | OU=Starfield Class 2 Certification Authority, O="Starfield Technologies, Inc.", C=US
264 |  | E=hostmaster@ispgateway.de, CN=webserver.ispgateway.de, O=ispgateway, L=Kempten, S=Bayern, C=DE
257 |  | E=info@parallels.com, CN=plesk, OU=Plesk, O="Parallels, Inc.", L=Herndon, S=Virginia, C=US
208 | Entrust (2048) | CN=Entrust.net Certification Authority (2048), OU=(c) 1999 Entrust.net Limited, OU=www.entrust.net/CPS_2048 incorp. by ref. (limits liab.), O=Entrust.net
198 | Trustwave | CN=SecureTrust CA, O=SecureTrust Corporation, C=US
168 |  | E=sslsign@lxlabs.com, CN=*.lxlabs.com, OU=web, O=lxlabs, L=WA, S=WA, C=IN
143 | thawte | CN=thawte Primary Root CA, OU="(c) 2006 thawte, Inc. - For authorized use only", OU=Certification Services Division, O="thawte, Inc.", C=US
101 |  | E=info@confixx.com, CN=confixx, OU=Confixx, O="SWsoft, Inc.", L=Herndon, S=Virginia, C=US
98 | StartCom Certification Authority | CN=StartCom Certification Authority, OU=Secure Digital Certificate Signing, O=StartCom Ltd., C=IL
92 |  | E=support@cacert.org, CN=CA Cert Signing Authority, OU=http://www.cacert.org, O=Root CA
87 | USERTrust | CN=UTN - DATACorp SGC, OU=http://www.usertrust.com, O=The USERTRUST Network, L=Salt Lake City, S=UT, C=US
71 | VeriSign | OU=Secure Server Certification Authority, O="RSA Data Security, Inc.", C=US
70 | Starfield Technologies | E=info@valicert.com, CN=http://www.valicert.com/, OU=ValiCert Class 2 Policy Validation Authority, O="ValiCert, Inc.", L=ValiCert Validation Network
64 |  | CN=localhost, OU=For testing purposes only, O=Apache HTTP Server
63 |  | E=admin@suresupport.com, CN=suresupport.com, OU=suresupport.com, O=suresupport.com, L=US, S=US, C=US
62 | SECOM Trust Systems CO LTD | OU=Security Communication RootCA1, O=SECOM Trust.net, C=JP
61 | Network Solutions | CN=Network Solutions Certificate Authority, O=Network Solutions L.L.C., C=US

      Client                                               Server

      ClientHello                  -------->
                                                      ServerHello
                                                      Certificate
                                   <--------      ServerHelloDone
      ClientKeyExchange
      [ChangeCipherSpec]
      Finished                     -------->
                                               [ChangeCipherSpec]
                                   <--------             Finished
      Application Data             <------->     Application Data

One thing to keep in mind that will influence the calculation is the variable size of most of the messages. The variable nature will not allow to calculate a precise value, but taking some reasonable average values for the variable fields, one can get a good approximation of the overhead. Now, let’s go through each of the messages and consider their sizes.

• ClientHello – the average size of initial client hello is about 160 to 170 bytes. It will vary based on the number of ciphersuites sent by the client as well as how many TLS ClientHello extensions are present. If session resumption is used, another 32 bytes need to be added for the Session ID field.
• ServerHello – this message is a bit more static than the ClientHello, but still variable size due to TLS extensions. The average size is 70 to 75 bytes.
• Certificate – this message is the one that varies the most in size between different servers. The message carries the certificate of the server, as well as all intermediate issuer certificates in the certificate chain (minus the root cert). Since certificate sizes vary quite a bit based on the parameters and keys used, I would use an average of 1500 bytes per certificate (self-signed certificates can be as small as 800 bytes). The other varying factor is the length of the certificate chain up to the root certificate. To be on the more conservative side of what is on the web, let’s assume 4 certificates in the chain. Overall this gives us about 6k for this message.
• ClientKeyExchange – let’s assume again the most widely used case – RSA server certificate. This corresponds to size of 130 bytes for this message.
• ChangeCipherSpec – fixed size of 1 (technically not a handshake message)
• Finished – depending whether SSLv3 is used or TLS, the size varies quite a bit – 36 and 12 bytes respectively. Most implementations these days support TLSv1.0 at least, so let’s assume TLS will be used and therefore the size will be 12 bytes.

Now that we have an average size of each message exchanged, we can calculate the average handshake size. One has to keep in mind that messages exchanged have TLS Record header for each record sent (5 bytes), as well as TLS Handshake header (4 bytes). The most common case can be simplified such that each arrow in the handshake diagram is a TLS Record, so we have 4 Records exchanged for total of 20 bytes. Each message has the handshake header (except the ChangeCipherSpec one), so we have 7 times the Handshake header for total of 28 bytes.

The total overhead to establish a new TLS session comes to about 6.5k bytes on average (20 + 28 + 170 + 75 + 6000 + 130 + 2*1 + 2*12 = 6449).

TLS sessions once established can also be resumed. In the session resumption, some of the messages are omitted and the handshake looks as follows:

      Client                                               Server

      ClientHello                  -------->
                                                      ServerHello
                                               [ChangeCipherSpec]
                                   <--------             Finished
      [ChangeCipherSpec]
      Finished                     -------->
      Application Data             <------->     Application Data

The main difference here is that the ClientHello message will contain extra 32 bytes for the session ID it wants to resume.

The total overhead to resume an existing TLS session comes to about 330 bytes on average (15 + 16 + 202 + 75 + 2*1 + 2*12 =332 ).

Now let’s look at the overhead on the wire for the encrypted application data. The data is carried in TLS Records over the wire, so there are 5 bytes of header. Since data is encrypted and integrity protected, there is additional overhead that is incurred. Let’s assume that the ciphersuite negotiated between the client and the server is TLS_RSA_WITH_AES_128_CBC_SHA, which is mandatory for TLS1.2 and hopefully will be commonly negotiated going forward. Since AES is a block cipher, it requires the data to be sized in multiple of the block size. TLS 1.0 defines the encrypted data with block cipher as:

    block-ciphered struct {
        opaque content[TLSCompressed.length];
        opaque MAC[CipherSpec.hash_size];
        uint8 padding[GenericBlockCipher.padding_length];
        uint8 padding_length;
    } GenericBlockCipher;

Since most implementations don’t use compression, we can assume the data is the same size. The MAC in this case is computed using SHA1, so the size will be 20 bytes. AES128 has a block size of 16 bytes, so the maximum padding we can add to the data will be 15 bytes.

The total overhead of the encrypted data is about 40 bytes (20 + 15 + 5).

It is easy to modify the above calculations to reflect more precisely the specifics of an environment, so this should be considered a basis for TLS overhead and not the authoritative answer to the question posed.