HPACK - Header Compression for HTTP/2Google, Incfenix@google.comCanon CRFherve.ruellan@crf.canon.fr
Applications
HTTPbisHTTPHeader
This specification defines HPACK, a compression format for
efficiently representing HTTP header fields in the context
of HTTP/2.
Discussion of this draft takes place on the HTTPBIS working group
mailing list (ietf-http-wg@w3.org), which is archived at .
Working Group information and related documents can be found at
(Wiki) and (source code and
issues tracker).
The changes in this draft are summarized in .
This specification defines HPACK, a compression format for
efficiently representing HTTP header fields in the context
of HTTP/2 (see ).
In HTTP/1.1 (see ), header fields are
encoded without any form of compression. As web pages have grown
to include dozens to hundreds of requests, the redundant
header fields in these requests now measurably increase
latency and unnecessarily consume bandwidth (see and ).
SPDY initially addressed this
redundancy by compressing header fields using the DEFLATE
format , which proved
very effective at efficiently representing the redundant header
fields. However, that approach exposed a security risk as
demonstrated by the CRIME attack (see ).
This document describes HPACK, a new compressor for header
fields which eliminates redundant header fields, is not
vulnerable to known security attacks, and which also has a
bounded memory requirement for use in constrained environments.
The HTTP header field encoding defined in this document
is based on a header table that maps name-value pairs to
index values. The header table is incrementally updated
during the HTTP/2 connection.
A set of header fields is treated as an unordered collection
of name-value pairs. Names and values are considered to be
opaque sequences of octets. The order of header fields is
not guaranteed to be preserved after being compressed and
decompressed.
As two consecutive sets of header fields often have header
fields in common, each set is coded as a difference from
the previous set. The goal is to only encode the changes
(header fields present in one of the sets that are absent
from the other) between the two sets of header fields.
A header field is represented either literally or as a
reference to a name-value pair in the header table. A set
of header fields is stored as a set of references to
entries in the header table (possibly keeping only a
subset of it, as some header fields may be missing a
corresponding entry in the header table). Differences
between consecutive sets of header fields are encoded as
changes to the set of references.
The encoder is responsible for deciding which header
fields to insert as new entries in the header table. The
decoder executes the modifications to the header table and
reference set prescribed by the encoder, reconstructing
the set of header fields in the process. This enables
decoders to remain simple and understand a wide variety of
encoders.
Examples illustrating the use of these different
mechanisms to represent header fields are available in
.
The encoding and decoding of header fields relies on some
components and concepts:
A name-value pair. Both the name and value are
treated as opaque sequences of octets.
The header table (see ) is a component used
to associate stored header fields to index values.
The static table (see ) is a component used
to associate static header fields to index values.
This data is ordered, read-only, always
accessible, and may be shared amongst all encoding
contexts.
The reference set (see ) is a component
containing an unordered set of references to
entries in the header table. This is used for the
differential encoding of a new header set.
A header set is an unordered group of
header fields that are encoded jointly. A complete
set of key-value pairs contained in a HTTP
request or response is a header set.
A header field can be represented in encoded form
either as a literal or as an index (see ).
The entire set of encoded header field
representations which, when decoded, yield a
complete header set.
When decoding a set of header field
representations, some operations emit a header
field (see ).
Emitted header fields are added to the current
header set and cannot be removed.
The set of mutable structures used within an encoding
context include a header table and a reference set.
Everything else is either immutable or conceptual.
HTTP messages are exchanged between a client and a
server in both directions. The encoding of header
fields in each direction is independent from the
other direction. There is a single encoding context
for each direction used to encode all header fields
sent in that direction.
A header table consists of a list of header fields
maintained in first-in, first-out order.
The first and newest entry in a header table is always
at index 1, and the oldest entry of a header table is
at the index len(header table).
The header table is initially empty.
There is typically no need for the header table to
contain duplicate entries. However, duplicate entries
MUST NOT be treated as an error by a decoder.
The encoder decides how to update the header table and
as such can control how much memory is used by the
header table. To limit the memory requirements of
the decoder, the header table size is strictly
bounded (see ).
The header table is updated during the processing of
a set of header field representations (see ).
A reference set is an unordered set of references to
entries of the header table.
The reference set is initially empty.
The reference set is updated during the processing of
a set of header field representations (see ).
The reference set enables differential encoding,
whereby only differences between the previous header
set and the current header set need to be encoded. The
use of differential encoding is optional for any
header set.
When an entry is evicted from the header table, if it
was referenced from the reference set, its reference
is removed from the reference set.
To limit the memory requirements on the decoder side
for handling the reference set, only entries within
the header table can be contained in the reference
set. To still allow entries from the static table to
take advantage of the differential encoding, when a
header field is represented as a reference to an entry
of the static table, this entry is inserted into the
header table (see ).
An encoded header field can be represented either as a
literal or as an index.
A literal representation defines a new header
field. The header field name is represented
either literally or as a reference to an entry
of the header table. The header field value is
represented literally.
Two different literal representations are
provided:
A literal representation that does not
add the header field to the header
table (see ).
A literal representation that adds the
header field as a new entry at the
beginning of the header table (see
).
The indexed representation defines a header
field as a reference to an entry in either the
header table or the static table (see ).
Indices between 1 and len(header
table), inclusive, refer to elements
in the header table, with index 1
referring to the beginning of the
table.
Indices between len(header table)+1 and
len(header table)+len(static table),
inclusive, refer to elements in the
static table, where the index
len(header table)+1 refers to the first
entry in the static table.
Index 0 signals a modification of the
encoding context: either the reference
set is emptied, or the maximum size of
the header table is updated (see ).
Any other indices MUST be treated as
erroneous, and the compression context
considered corrupt and unusable.
The emission of a header field is the process of
marking a header field as belonging to the current
header set. Once a header has been emitted, it cannot
be removed from the current header set.
On the decoding side, an emitted header field can be
safely passed to the upper processing layer as part of
the current header set. The decoder MAY pass the
emitted header fields to the upper processing layer in
any order.
By emitting header fields instead of emitting header
sets, the decoder can be implemented in a streaming
way, and as such has only to keep in memory the header
table and the reference set. This bounds the amount of
memory used by the decoder, even in presence of a very
large set of header fields. The management of memory
for handling very large sets of header fields can
therefore be deferred to the upper processing layers.
The processing of a header block to obtain a header set is
defined in this section. To ensure that the decoding will
successfully produce a header set, a decoder MUST obey the
following rules.
All the header field representations contained in a
header block are processed in the order in which they
are presented, as specified below.
An indexed representation with an index
value of 0 entails one of the following actions,
depending on what is encoded next:
The reference set is emptied.The maximum size of the header table is
updated.
An indexed representation corresponding
to an entry present in the
reference set entails the following actions:
The entry is removed from the reference
set.
An indexed representation corresponding
to an entry not present in the
reference set entails the following actions:
If referencing an element of the static table:
The header field corresponding to the
referenced entry is emitted.The referenced static entry is inserted
at the beginning of the header
table.A reference to this new header table
entry is added to the reference set
(except if this new entry didn't fit
in the header table).If referencing an element of the header table:
The header field corresponding to the
referenced entry is emitted.The referenced header table entry is
added to the reference set.
A literal representation that is
not added to the header table entails
the following action:
The header field is emitted.
A literal representation that is
added to the header table entails
the following actions:
The header field is emitted.The header field is inserted at the beginning
of the header table.A reference to the new entry is added to the
reference set (except if this new entry didn't
fit in the header table).
Once all the representations contained in a header
block have been processed, the header fields
referenced in the reference set which have not
previously been emitted during this processing are
emitted.
Once all of the header field representations have been
processed, and the remaining items in the reference
set have been emitted, the header set is complete.
To limit the memory requirements on the decoder side,
the size of the header table is bounded. The size
of the header table MUST stay lower than or equal to its
maximum size.
By default, the maximum size of the header table is
equal to the value of the HTTP/2 setting
SETTINGS_HEADER_TABLE_SIZE defined by the decoder (see
). The encoder can change this
maximum size (see ), but it must
stay lower than or equal to the value of
SETTINGS_HEADER_TABLE_SIZE.
The size of the header table is the sum of the
size of its entries.
The size of an entry is the sum of its name's length
in octets (as defined in ), of its
value's length in octets () and of
32 octets.
The lengths are measured on the non-encoded entry
name and entry value (for the case when a Huffman
encoding is used to transmit string values).
The 32 octets are an accounting for the entry
structure overhead. For example, an entry structure
using two 64-bits pointers to reference the name and
the value and the entry, and two 64-bits integer for
counting the number of references to these name and
value would use 32 octets.
Whenever an entry is evicted from the header table,
any reference to that entry contained by the reference
set is removed.
Whenever the maximum size for the header table is made
smaller, entries are evicted from the end of the
header table until the size of the header table is
less than or equal to the maximum size.
The eviction of an entry from the header
table causes the index of the entries in the static
table to be reduced by one.
Whenever a new entry is to be added to the table, any
name referenced by the representation of this new
entry is cached, and then entries are evicted from the
end of the header table until the size of the header
table is less than or equal to (maximum size - new
entry size), or until the table is empty.
If the size of the new entry is less than or equal to
the maximum size, that entry is added to the table. It
is not an error to attempt to add an entry
that is larger than the maximum size.
Integers are used to represent name indexes, pair
indexes or string lengths. To allow for optimized
processing, an integer representation always finishes
at the end of an octet.
An integer is represented in two parts: a prefix that
fills the current octet and an optional list of octets
that are used if the integer value does not fit within
the prefix. The number of bits of the prefix (called
N) is a parameter of the integer representation.
The N-bit prefix allows filling the current octet. If
the value is small enough (strictly less than
2N-1), it is encoded within the N-bit
prefix. Otherwise all the bits of the prefix are set
to 1 and the value is encoded using an
unsigned variable length integer
representation. N is always between 1 and 8 bits. An
integer starting at an octet-boundary will have an 8-bit
prefix.
The algorithm to represent an integer I is as follows:
For informational purpose, the algorithm to decode an
integer I is as follows:
This integer representation allows for values of
indefinite size. It is also possible for an encoder to
send a large number of zero values, which can waste
octets and could be used to overflow integer values.
Excessively large integer encodings - in value or octet
length - MUST be treated as a decoding error. Different
limits can be set for each of the different uses of
integers, based on implementation constraints.
The value 10 is to be encoded with a 5-bit prefix.
10 is less than 31 (= 25 -
1) and is represented using the 5-bit
prefix.
The value I=1337 is to be encoded with a 5-bit
prefix.
1337 is greater than 31 (=
25 - 1).
The 5-bit prefix is filled with its
max value (31).I = 1337 - (25 - 1) =
1306.I (1306) is greater than or equal
to 128, the while loop body
executes:I % 128 == 2626 + 128 == 154154 is encoded in 8 bits as:
10011010I is set to 10 (1306 / 128 ==
10)I is no longer greater than
or equal to 128, the while
loop terminates.
I, now 10, is encoded on 8 bits
as: 00001010
The process ends.
The value 42 is to be encoded starting at an
octet-boundary. This implies that a 8-bit prefix is
used.
42 is less than 255 (= 28 -
1) and is represented using the 8-bit
prefix.
Header field names and header field values are
encoded as sequences of octets. A header field name or
a header field value is encoded in three parts:
One bit, H, indicating whether or not
the octets are Huffman encoded.
The number of octets required to hold the
result of the next step, represented as an
integer with a 7-bit prefix (see ),
immediately following the first bit.
The encoded data of the string:
If H is '1', then the encoded
string data is the bitwise
concatenation of the canonical Huffman code
corresponding to
each octet of the data, followed by
between 0-7 bits of padding.
If H is '0', then the encoded string
is the octets of the field value
without modification.
Padding is necessary when doing Huffman encoding to
ensure that the remaining bits between the actual end of
the data and the next octet boundary are not
misinterpreted as part of the input data.
When padding for Huffman encoding, the bits from the
EOS (end-of-string) entry in the Huffman table are
used, starting with the MSB (most significant bit).
This entry is guaranteed to be at least 8 bits long.
String literals which use Huffman encoding
are encoded with the Huffman Codes (see examples in Request
Examples with Huffman and in
Response Examples with Huffman ).
The EOS symbol is represented with value 256, and is
used solely to signal the end of the Huffman-encoded
key data or the end of the Huffman-encoded value data.
Given that only between 0-7 bits of the EOS symbol is
included in any Huffman-encoded string, and given that
the EOS symbol is at least 8 bits long, it is expected
that it should never be successfully decoded.
An indexed header field representation either identifies an
entry in the header table or static table. The processing
of an indexed header field representation is described in
.
This representation starts with the '1' 1-bit pattern,
followed by the index of the matching pair, represented as
an integer with a 7-bit prefix.
The index value of 0 is reserved for signalling changes in
the encoding context (see ).
Literal header field representations contain a literal
header field value. Header field names are either
provided as a literal or by reference to an existing
header table or static table entry.
Literal representations all result in the emission of a
header field when decoded.
A literal header field without indexing causes the
emission of a header field without altering the header
table.
This representation starts with the '01' 2-bit pattern.
If the header field name matches the header field name
of a (name, value) pair stored in the Header Table or
Static Table, the header field name can be represented
using the index of that entry. In this case, the index
of the entry, index (which is strictly greater than 0),
is represented as an integer with a 6-bit prefix (see
).
Otherwise, the header field name is represented as a
literal. The value 0 is represented on 6 bits followed
by the header field name (see ).
The header field name representation is followed by
the header field value represented as a literal string
as described in .
A literal header field with incremental indexing adds
a new entry to the header table.
This representation starts with the '00' 2-bit
pattern.
If the header field name matches the header field name
of a (name, value) pair stored in the Header Table or
Static Table, the header field name can be represented
using the index of that entry. In this case, the index
of the entry, index (which is strictly greater than 0),
is represented as an integer with a 6-bit prefix (see
).
Otherwise, the header field name is represented as a
literal. The value 0 is represented on 6 bits followed
by the header field name (see ).
The header field name representation is followed by
the header field value represented as a literal string
as described in .
An indexed value of 0 is reserved for signalling changes
in the encoding context. The type of the change is
encoded on the following octet(s). Any change in the
encoding context is applied immediately.
An octet with its high bit set to '1' signals that the
reference set is emptied. The remaining bits are set
to '0'.
An octet with its high bit set to '0' signals the new
maximum size of the header table. This new maximum size
MUST be lower than or equal to the value of the setting
SETTINGS_HEADER_TABLE_SIZE (see ).
The new maximum size is encoded as an integer with a 7-bit
prefix.
Change in the maximum size of the header table can trigger
entry evictions (see ).
This compressor exists to solve security issues present in
stream compressors such as DEFLATE whereby the compression
context can be efficiently probed to reveal secrets.
A conformant implementation of this specification should be
fairly safe against that kind of attack, as the reaping of any
information from the compression context requires more work than
guessing and verifying the plain text data directly with the
server. As with any secret, however, the longer the length
of the secret, the more difficult the secret is to guess. It
is inadvisable to have short cookies that are relied upon to
remain secret for any duration of time.
A proper security-conscious implementation will also need to
prevent timing attacks by ensuring that the amount of time it
takes to do string comparisons is always a function of the
total length of the strings, and not a function of the number
of matched characters.
A decoder needs to ensure that larger values or encodings of
integers do not permit exploitation. Decoders MUST limit the
size of integers, both in value and encoded length, that it
accepts (see ).
Another common security problem is when the remote endpoint
successfully causes the local endpoint to exhaust its memory.
This compressor attempts to deal with the most obvious ways
that this could occur by limiting both the peak and the
steady-state amount of memory consumed in the compressor
state, by providing ways for the application to consume/flush
the emitted header fields in small chunks, and by considering
overhead in the state size calculation. Implementors must
still be careful in the creation of APIs to an implementation
of this compressor by ensuring that header field keys and
values are either emitted as a stream, or that the compression
implementation have a limit on the maximum size of a key or
value. Failure to implement these kinds of safeguards may
still result in a scenario where the local endpoint exhausts
its memory.
A particular care should be used for the maximum size of the
header table. While an endpoint can fully control the maximum
size of its header table for the decoding size, by using
SETTINGS_HEADER_TABLE_SIZE, the maximum size of the encoding
size is controlled by the remote peer. The endpoint should
check the SETTINGS_HEADER_TABLE_SIZE defined by the remote
peer, and decrease the maximum size for the encoding size if
needed.
This document includes substantial editorial contributions
from the following individuals: Mike Bishop, Jeff Pinner,
Julian Reschke, Martin Thomson.
Hypertext Transfer Protocol version 2TwistGoogleMozilla
Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and RoutingAdobe Systems Incorporatedfielding@gbiv.comgreenbytes GmbHjulian.reschke@greenbytes.deSPDY ProtocolTwistGoogleDEFLATE Compressed Data Format Specification version 1.3Aladdin EnterprisesThe CRIME AttackIETF83: SPDY and What to Consider for HTTP/2.0SPDY: What I Like About YouA Method for the Construction of Minimum Redundancy CodesGenerating a canonical prefix encoding
Regenerated examples.
Only one Huffman table for requests and responses.
Added maximum size for header table, independent of
SETTINGS_HEADER_TABLE_SIZE.
Added pseudo-code for integer decoding.
Improved examples (removing unnecessary removals).
Updated examples: take into account changes in the
spec, and show more features.
Use 'octet' everywhere instead of having both 'byte'
and 'octet'.
Added reference set emptying.
Editorial changes and clarifications.
Added "host" header to the static table.
Ordering for list of values (either NULL- or
comma-separated).
A large number of editorial changes; changed the
description of evicting/adding new entries.
Removed substitution indexing
Changed 'initial headers' to 'static headers', as per
issue #258
Merged 'request' and 'response' static headers, as per
issue #259
Changed text to indicate that new headers are added at
index 0 and expire from the largest index, as per
issue #233
Corrected error in integer encoding pseudocode.
Refactored of Header Encoding Section: split
definitions and processing rule.
Backward incompatible change: Updated
reference set management as per issue #214. This
changes how the interaction between the reference
set and eviction works. This also changes the
working of the reference set in some specific
cases.
Backward incompatible change: modified initial
header list, as per issue #188.
Added example of 32 octets entry structure (issue
#191).
Added Header Set Completion section. Reflowed
some text. Clarified some writing which was
akward. Added text about duplicate header entry
encoding. Clarified some language w.r.t Header
Set. Changed x-my-header to mynewheader. Added
text in the HeaderEmission section indicating that
the application may also be able to free up memory
more quickly. Added information in Security
Considerations section.
Fixed bug/omission in integer representation algorithm.Changed the document title.Header matching text rewritten.Changed the definition of header emission.Changed the name of the setting which dictates how
much memory the compression context should
use.Removed "specific use cases" sectionCorrected erroneous statement about what index can
be contained in one octetAdded descriptions of opcodesRemoved security claims from introduction.
The static table consists of an unchangeable ordered list of
(name, value) pairs. The first entry in the table is always
represented by the index len(header table)+1, and the last entry
in the table is represented by the index len(header
table)+len(static table).
The ordering of these tables is currently arbitrary.
The tables in this section should be updated and ordered
such that the table entries with the smallest indices are
those which, based on a statistical analysis of the
frequency of use weighted by size, achieve the largest
decrease in octets transmitted subject to HTTP 2 header
field rules (like removal of some header fields). This set
of header fields is currently very likely incomplete, and
should be made complete.
The following table lists the pre-defined header fields that
make-up the static header table.
IndexHeader NameHeader Value1:authority2:methodGET3:methodPOST4:path/5:path/index.html6:schemehttp7:schemehttps8:status2009:status50010:status40411:status40312:status40013:status40114accept-charset15accept-encoding16accept-language17accept-ranges18accept19access-control-allow-origin20age21allow22authorization23cache-control24content-disposition25content-encoding26content-language27content-length28content-location29content-range30content-type31cookie32date33etag34expect35expires36from37host38if-match39if-modified-since40if-none-match41if-range42if-unmodified-since43last-modified44link45location46max-forwards47proxy-authenticate48proxy-authorization49range50referer51refresh52retry-after53server54set-cookie55strict-transport-security56transfer-encoding57user-agent58vary59via60www-authenticate
The table give the index of each entry in the static table.
The full index of each entry, to be used for encoding a
reference to this entry, is computed by adding the number of
entries in the header table to this index.
The following Huffman codes are used when encoding string
literals.
This table will be regenerated.
A number of examples are worked through here, for both
requests and responses, and with and without Huffman coding.
This section show several independent representation examples.
The header field representation uses a literal name and a literal value.
Reference set: empty.
The header field representation uses an indexed name and a literal value.
Reference set: empty.
Header table (after decoding): empty.
The header field representation uses an indexed header field, from
the static table. Upon using it, the static table entry is copied into
the header table.
Reference set: empty.
The header field representation uses an indexed header field, from
the static table. In this example, the SETTINGS_HEADER_TABLE_SIZE is set to 0,
therefore, the entry is not copied into the header table.
Reference set: empty.
Header table (after decoding): empty.
This section shows several consecutive header sets, corresponding
to HTTP requests, on the same connection.
Reference set: empty.
This request takes advantage of the differential encoding of header sets.
This request has not enough headers in common with the previous request
to take advantage of the differential encoding. Therefore, the reference set
is emptied before encoding the header fields.
This section shows the same examples as the previous section, but
using Huffman encoding for the literal values.
Reference set: empty.
This request takes advantage of the differential encoding of header sets.
This request has not enough headers in common with the previous request
to take advantage of the differential encoding. Therefore, the reference set
is emptied before encoding the header fields.
This section shows several consecutive header sets, corresponding
to HTTP responses, on the same connection. SETTINGS_HEADER_TABLE_SIZE is set
to the value of 256 octets, causing some evictions to occur.
Reference set: empty.
The (":status", "302") header field is evicted from the header
table to free space to allow adding the (":status", "200") header
field, copied from the static table into the header table.
The (":status", "302") header field doesn't need to be
removed from the reference set as it is evicted from the header
table.
Several header fields are evicted from the header table during the
processing of this header set. Before evicting a header belonging to the
reference set, it is emitted, by coding it twice as an Indexed Representation.
The first representation removes the header field from the reference set, the
second one adds it again to the reference set, also emitting it.
This section shows the same examples as the previous section, but
using Huffman encoding for the literal values. The eviction mechanism uses the
length of the decoded literal values, so the same evictions occurs as in the
previous section.
Reference set: empty.
The (":status", "302") header field is evicted from the header
table to free space to allow adding the (":status", "200") header
field, copied from the static table into the header table.
The (":status", "302") header field doesn't need to be
removed from the reference set as it is evicted from the header
table.
Several header fields are evicted from the header table during the
processing of this header set. Before evicting a header belonging to the
reference set, it is emitted, by coding it twice as an Indexed Representation.
The first representation removes the header field from the reference set, the
second one adds it again to the reference set, also emitting it.