Inside the Qt HTTP stack

The Qt HTTP stack is the base for any HTTP communication with Qt; it is for instance used by the Qt Webkit port. In Qt 5, quite a bit of the HTTP internals have been rewritten, mostly by Markus from woboq. This blog post is trying to shed some light on the HTTP internals, even though some classes are omitted here for simplicity. To start with, the public classes can be used as follows.

Simple example code

QUrl url("http://qt.gitorious.org");

QNetworkRequest request(url);

QNetworkAccessManager manager;

QNetworkReply *reply = manager.get(request);

QObject::connect(reply, SIGNAL(finished()),

myClass, SLOT(replyFinished()));

1. The public classes and their friends

The example shows the main classes: A QUrl is used to create a QNetworkRequest, which resembles an HTTP request. The request is then passed to the QNetworkAccessManager, which sends the request over the network and returns a QNetworkReply, which resembles an HTTP reply. Depending on the scheme of the URL, the QNetworkAccessManager instantiates an internal subclass of QNetworkReply; in the case of a URL scheme of 'http://' or 'https://', the class instantiated is called QNetworkReplyHttpImpl. This class is the main class for setting up the request and "enriching" it with additional data like caching or cookie information before sending the request. In case the request is supposed to upload data, (i.e. when using HTTP POST or PUT), the HTTP impl class uses a QNonContiguousByteDevice. Such a non contiguous byte device can be used to read from a file, byte array or others without expensive memcpy'ing. This looks like the following in UML notation:

2. The worker thread

One feature that was introduced in Qt 4.8 was a threaded HTTP backend: This new backend executes the sending and receiving of data over the sockets and the HTTP message parsing in its own thread (see also the blog post about the change). The QNetworkReplyHttpImpl creates a class that lives in the new thread, the so-called HTTP thread. This class is called QHttpThreadDelegate, and serves as Facade class for everything that is going on in the HTTP thread. The cross-thread communication between QNetworkReplyHttpImpl and QHttpThreadDelegate happens through signals and slots communication, and data travels both ways. This means that both the Delegate has slots which are invoked by signals emitted in the HttpImpl and vice versa. Whenever a QNetworkReplyHttpImpl is created, it creates a corresponding QHttpThreadDelegate, connects signals and slots and moves the Delegate to the HTTP thread. The Delegate contains classes that resemble the HTTP request and reply, namely QHttpNetworkRequest and QHttpNetworkReply. This might be a bit confusing, since there are already the public QNetworkRequest and QNetworkReply; the latter classes are public interfaces for many of the HTTP specific attributes of the request and reply, e.g. set HTTP pipelining setting, status code or other other HTTP headers. The internal classes QHttpNetworkRequest and QHttpNetworkReply parse the HTTP message into header fields and body from the socket data stream.

With those new classes added to the diagram, it looks as follows.

3. The lower layers

The HTTP requests and replies are sent and received over so-called channels; a channel is basically a socket with a bit more logic for state handling and HTTP features. The socket of a channel can be a QTcpSocket for a normal HTTP request, or it can be a QSslSocket in case a 'https://' scheme is used. A set of channels connected to the same server form a connection. There is always only one connection to a server, but up to 6 channels used concurrently. In addition, there can be even more requests on the fly if HTTP pipelining is enabled. When a reply has been received over a socket, the latter is not automatically closed, but by default reused for later requests, to save socket setup time and to already have a socket with a larger TCP window size.

Now the UML diagram of the HTTP stack is almost complete:

4. The rest

Two classes have not been mentioned yet, even though they are of importance:

QNetworkSession: This class is used mostly for mobile use cases, when there is no permanent connection to the Internet. In case there is no connection to the Internet, the QNetworkSession and related classes will try to establish a connection (e.g. an application would connect to a signal by the QNetworkSession and let the user choose between 3G and Wifi). The session is instantiated by the QNetworkAccessManager.
QNetworkAccessAuthenticationManager: This is a global class for storing authentication credentials that need to be reused. If a server requires authentication, the network access manager emits a signal (QNetworkAccessManager::authenticationRequired()) which should trigger the user to enter his user name and password. The authentication manager class caches that data and sends it automatically to the server with subsequent requests. Interestingly, the authentication manager also uses a QNetworkAccessCache, just like the QHttpNetworkConnection does to cache connections.

So the (for the scope of this article) complete diagram of the Qt HTTP stack looks like the following:

What has not been mentioned

Even though the diagram above seems to be complicated enough, there are several classes and areas that have been omitted, namely:

Bearer classes: In order to tweak connection setup etc. more closely, there are several classes beyond the aforementioned QNetworkSession.
Cookies: Of course cookie parsing and sending is by default supported by the Qt HTTP stack; see the classes QNetworkCookie and QNetworkCookieJar.
HTTP caching: Qt also supports HTTP caching; however, it is not enabled by default.
Proxies: Qt supports network transfer over HTTP and SOCKS5 proxies.
uploading data: Some classes related to uploading data have been omitted, among others classes for handling HTTP multipart messages.