Network Working Group J. Jena
Internet-Draft
Intended status: Standards Track S. Dalal
Expires: September 28, 2024 March 27, 2024
The Idempotency-Key HTTP Header Field
draft-ietf-httpapi-idempotency-key-header-05
Abstract
The HTTP Idempotency-Key request header field can be used to make
non-idempotent HTTP methods such as "POST" or "PATCH" fault-tolerant.
About This Document
This note is to be removed before publishing as an RFC.
Status information for this document may be found at
.
Discussion of this document takes place on the HTTPAPI Working Group
mailing list (), which is archived at
. Subscribe at
. Working Group
information can be found at .
Source for this draft and an issue tracker can be found at
.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 28, 2024.
Jena & Dalal Expires September 28, 2024 [Page 1]
Internet-Draft The Idempotency-Key HTTP Header Field March 2024
Copyright Notice
Copyright (c) 2024 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1. Notational Conventions . . . . . . . . . . . . . . . . . 3
2. The Idempotency-Key HTTP Request Header Field . . . . . . . . 3
2.1. Syntax . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2. Uniqueness of Idempotency Key . . . . . . . . . . . . . . 3
2.3. Idempotency Key Validity and Expiry . . . . . . . . . . . 4
2.4. Idempotency Fingerprint . . . . . . . . . . . . . . . . . 4
2.5. Responsibilities . . . . . . . . . . . . . . . . . . . . 4
2.5.1. Client . . . . . . . . . . . . . . . . . . . . . . . 4
2.5.2. Resource . . . . . . . . . . . . . . . . . . . . . . 5
2.6. Idempotency Enforcement . . . . . . . . . . . . . . . . . 5
2.7. Error Handling . . . . . . . . . . . . . . . . . . . . . 5
3. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7
3.1. The Idempotency-Key HTTP Request Header Field . . . . . . 7
4. Implementation Status . . . . . . . . . . . . . . . . . . . . 7
4.1. Implementing the Concept . . . . . . . . . . . . . . . . 9
5. Security Considerations . . . . . . . . . . . . . . . . . . . 11
6. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 12
7. References . . . . . . . . . . . . . . . . . . . . . . . . . 12
7.1. Normative References . . . . . . . . . . . . . . . . . . 12
7.2. Informative References . . . . . . . . . . . . . . . . . 13
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 13
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 13
1. Introduction
In mathematics and computer science, an idempotent operation is one
that can be applied multiple times without changing the result beyond
the initial application. It does not matter if the operation is
called only once or tens of times over.
Jena & Dalal Expires September 28, 2024 [Page 2]
Internet-Draft The Idempotency-Key HTTP Header Field March 2024
Idempotency is important in building fault-tolerant HTTP APIs. An
HTTP request method is considered "idempotent" if the intended effect
on the server of multiple identical requests with that method is the
same as the effect for a single such request. Per [RFC9110], the
methods "OPTIONS", "HEAD", "GET", "PUT" and "DELETE" are idempotent
while methods "POST" and "PATCH" are not.
Let's say a client of an HTTP API wants to create (or update) a
resource using a "POST" method. Repeating the request multiple times
can result in duplication or incorrect updates. Consider a scenario
where the client sent a "POST" request to the server, but the request
timed out. The client does not know if the resource was created (or
updated), because it does not know when the timeout occured from the
server's perspective. Furthermore, the client does not know if it
can safely retry the request.
1.1. Notational Conventions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
This specification uses the Augmented Backus-Naur Form (ABNF)
notation of [RFC5234] and includes, by reference, the IMF-fixdate
rule as defined in Section 7.1.1.1 of [RFC7231].
The term "resource" is to be interpreted as defined in Section 2 of
[RFC7231], that is identified by an URI.
2. The Idempotency-Key HTTP Request Header Field
An idempotency key is a unique value generated by the client which
the resource uses to recognize subsequent retries of the same
request. The "Idempotency-Key" HTTP request header field carries
this value.
2.1. Syntax
"Idempotency-Key" is an Item Structured Header [RFC8941]. Its value
MUST be a String (Section 3.3.3 of [RFC8941]).
2.2. Uniqueness of Idempotency Key
The idempotency key MUST be unique and MUST NOT be reused with
another request with a different request payload.
Jena & Dalal Expires September 28, 2024 [Page 3]
Internet-Draft The Idempotency-Key HTTP Header Field March 2024
Uniqueness of the key MUST be defined by the resource owner and MUST
be implemented by the clients of the resource. It is RECOMMENDED
that a UUID [RFC4122] or a similar random identifier be used as an
idempotency key.
The following example shows an idempotency key whose value is a UUID
[RFC4122]:
Idempotency-Key: "8e03978e-40d5-43e8-bc93-6894a57f9324"
2.3. Idempotency Key Validity and Expiry
The resource MAY require time based idempotency keys to be able to
purge or delete a key upon its expiry. The resource SHOULD define
such expiration policy and publish it in the documentation.
2.4. Idempotency Fingerprint
An idempotency fingerprint MAY be used in conjunction with an
idempotency key to determine the uniqueness of a request. Such a
fingerprint is generated from request payload data by the resource.
An idempotency fingerprint generation algorithm MAY use one of the
following or similar approaches to create a fingerprint.
o Checksum of the entire request payload.
o Checksum of selected element(s) in the request payload.
o Field value match for each field in the request payload.
o Field value match for selected element(s) in the request payload.
o Request digest/signature.
2.5. Responsibilities
2.5.1. Client
Clients of HTTP API requiring idempotency SHOULD understand the
idempotency related requirements as published by the server and use
appropriate algorithm to generate idempotency keys.
Clients MAY choose to send an Idempotency-Key field with any valid
value to indicate the user's intent is to only perform this action
once. Without a priori knowledge, a general client cannot assume the
server will respect this request.
For each request, a client SHOULD:
Jena & Dalal Expires September 28, 2024 [Page 4]
Internet-Draft The Idempotency-Key HTTP Header Field March 2024
o Send a unique idempotency key in the HTTP "Idempotency-Key"
request header field.
2.5.2. Resource
Resources MUST publish a idempotency related specification. This
specification MUST include expiration related policy if applicable.
A resource is responsible for managing the lifecycle of the
idempotency key.
For each request, a server SHOULD:
o Identify idempotency key from the HTTP "Idempotency-Key" request
header field.
o Generate idempotency fingerprint if required.
o Enforce idempotency (see below)
2.6. Idempotency Enforcement
o First time request (idempotency key and fingerprint has not been
seen)
The resource SHOULD process the request normally and respond with
an appropriate response and status code.
o Duplicate request (idempotency key and fingerprint has been seen)
Retry
The request was retried after the original request completed. The
resource SHOULD respond with the result of the previously
completed operation, success or an error. See Error Scenarios for
details on errors.
Concurrent Request
The request was retried before the original request completed.
The resource SHOULD respond with a resource conflict error. See
Error Scenarios for details.
2.7. Error Handling
If the "Idempotency-Key" request header is missing for a documented
idempotent operation requiring this header, the resource SHOULD reply
with an HTTP "400" status code with body containing a link pointing
Jena & Dalal Expires September 28, 2024 [Page 5]
Internet-Draft The Idempotency-Key HTTP Header Field March 2024
to relevant documentation. Following examples shows an error
response describing the problem using [RFC7807].
HTTP/1.1 400 Bad Request
Content-Type: application/problem+json
Content-Language: en
{
"type": "https://developer.example.com/idempotency",
"title": "Idempotency-Key is missing",
"detail": "This operation is idempotent and it requires correct
usage of Idempotency Key.",
}
Alternately, using the HTTP header "Link", the client can be informed
about the error as shown below.
HTTP/1.1 400 Bad Request
Link: ;
rel="describedby"; type="text/html"
If there is an attempt to reuse an idempotency key with a different
request payload, the resource SHOULD reply with a HTTP "422" status
code with body containing a link pointing to relevant documentation.
The status code "422" is defined in Section 15.5.21 of [RFC9110].
HTTP/1.1 422 Unprocessable Content
Content-Type: application/problem+json
Content-Language: en
{
"type": "https://developer.example.com/idempotency",
"title": "Idempotency-Key is already used",
"detail": "This operation is idempotent and it requires
correct usage of Idempotency Key. Idempotency Key MUST not be
reused across different payloads of this operation.",
}
The server can also inform the client by using the HTTP header "Link"
as shown below.
HTTP/1.1 422 Unprocessable Content
Link: ;
rel="describedby"; type="text/html"
If the request is retried, while the original request is still being
processed, the resource SHOULD reply with an HTTP "409" status code
with body containing problem description.
Jena & Dalal Expires September 28, 2024 [Page 6]
Internet-Draft The Idempotency-Key HTTP Header Field March 2024
HTTP/1.1 409 Conflict
Content-Type: application/problem+json
Content-Language: en
{
"type": "https://developer.example.com/idempotency",
"title": "A request is outstanding for this Idempotency-Key",
"detail": "A request with the same Idempotency-Key for the
same operation is being processed or is outstanding.",
}
Or, alternately using the HTTP header "Link" pointing to the relevant
documentation
HTTP/1.1 409 Conflict
Link: ;
rel="describedby"; type="text/html"
Error scenarios above describe the status of failed idempotent
requests after the resource prcocesses them. Clients MUST correct
the requests (with the exception of 409 where no correction is
required) before performing a retry operation, or the resource MUST
fail the request and return one of the above errors.
For other 4xx/5xx errors, such as 401, 403, 500, 502, 503, 504, 429,
or any other HTTP error code that is not listed here, the client
SHOULD act appropriately by following the resource's documentation.
3. IANA Considerations
3.1. The Idempotency-Key HTTP Request Header Field
The Idempotency-Key field name should be added to the "Hypertext
Transfer Protocol (HTTP) Field Name Registry".
Field Name: Idempotency-Key
Status: permanent
Specification document: This specification, Section 2
4. Implementation Status
Note to RFC Editor: Please remove this section before publication.
This section records the status of known implementations of the
protocol defined by this specification at the time of posting of this
Internet-Draft, and is based on a proposal described in [RFC7942].
The description of implementations in this section is intended to
Jena & Dalal Expires September 28, 2024 [Page 7]
Internet-Draft The Idempotency-Key HTTP Header Field March 2024
assist the IETF in its decision processes in progressing drafts to
RFCs. Please note that the listing of any individual implementation
here does not imply endorsement by the IETF. Furthermore, no effort
has been spent to verify the information presented here that was
supplied by IETF contributors. This is not intended as, and must not
be construed to be, a catalog of available implementations or their
features. Readers are advised to note that other implementations may
exist.
According to RFC 7942, "this will allow reviewers and working groups
to assign due consideration to documents that have the benefit of
running code, which may serve as evidence of valuable experimentation
and feedback that have made the implemented protocols more mature.
It is up to the individual working groups to use this information as
they see fit".
Organization: Stripe
o Description: Stripe uses custom HTTP header named "Idempotency-
Key"
o Reference: https://stripe.com/docs/idempotency
Organization: Adyen
o Description: Adyen uses custom HTTP header named "Idempotency-Key"
o Reference: https://docs.adyen.com/development-resources/api-
idempotency/
Organization: Dwolla
o Description: Dwolla uses custom HTTP header named "Idempotency-
Key"
o Reference: https://docs.dwolla.com/
Organization: Interledger
o Description: Interledger uses custom HTTP header named
"Idempotency-Key"
o Reference: https://github.com/interledger/
Organization: WorldPay
o Description: WorldPay uses custom HTTP header named "Idempotency-
Key"
Jena & Dalal Expires September 28, 2024 [Page 8]
Internet-Draft The Idempotency-Key HTTP Header Field March 2024
o Reference: https://developer.worldpay.com/docs/wpg/idempotency
Organization: Yandex
o Description: Yandex uses custom HTTP header named "Idempotency-
Key"
o Reference: https://cloud.yandex.com/docs/api-design-
guide/concepts/idempotency
Organization: http4s.org
o Description: Http4s is a minimal, idiomatic Scala interface for
HTTP services.
o Reference: https://github.com/http4s/http4s
Organization: Finastra
o Description: Finastra uses custom HTTP header named "Idempotency-
Key"
o Reference: https://developer.fusionfabric.cloud/
Organization: Datatrans
o Description: Datatrans focuses on the technical processing of
payments, including hosting smart payment forms and correctly
routing payment information.
o Reference: https://docs.datatrans.ch/docs/api-endpoints
4.1. Implementing the Concept
This is a list of implementations that implement the general concept,
but do so using different mechanisms:
Organization: Django
o Description: Django uses custom HTTP header named
"HTTP_IDEMPOTENCY_KEY"
o Reference: https://pypi.org/project/django-idempotency-key
Organization: Twilio
o Description: Twilio uses custom HTTP header named "I-Twilio-
Idempotency-Token" in webhooks
Jena & Dalal Expires September 28, 2024 [Page 9]
Internet-Draft The Idempotency-Key HTTP Header Field March 2024
o Reference: https://www.twilio.com/docs/usage/webhooks/webhooks-
connection-overrides
Organization: PayPal
o Description: PayPal uses custom HTTP header named "PayPal-Request-
Id"
o Reference: https://developer.paypal.com/docs/business/develop/
idempotency
Organization: RazorPay
o Description: RazorPay uses custom HTTP header named "X-Payout-
Idempotency"
o Reference: https://razorpay.com/docs/razorpayx/api/idempotency/
Organization: OpenBanking
o Description: OpenBanking uses custom HTTP header called "x-
idempotency-key"
o Reference: https://openbankinguk.github.io/read-write-api-
site3/v3.1.6/profiles/read-write-data-api-profile.html#request-
headers
Organization: Square
o Description: To make an idempotent API call, Square recommends
adding a property named "idempotency_key" with a unique value in
the request body.
o Reference: https://developer.squareup.com/docs/build-basics/using-
rest-api
Organization: Google Standard Payments
o Description: Google Standard Payments API uses a property named
"requestId" in request body in order to provider idempotency in
various use cases.
o Reference: https://developers.google.com/standard-payments/
payment-processor-service-api/rest/v1/TopLevel/capture
Organization: BBVA
Jena & Dalal Expires September 28, 2024 [Page 10]
Internet-Draft The Idempotency-Key HTTP Header Field March 2024
o Description: BBVA Open Platform uses custom HTTP header called "X-
Unique-Transaction-ID"
o Reference:
https://bbvaopenplatform.com/apiReference/APIbasics/content/x-
unique-transaction-id
Organization: WebEngage
o Description: WebEngage uses custom HTTP header called "x-request-
id" to identify webhook POST requests uniquely to achieve events
idempotency.
o Reference: https://docs.webengage.com/docs/webhooks
5. Security Considerations
This section is meant to inform developers, information providers,
and users of known security concerns specific to the idempotency
keys.
Resources that do not implement strong idempotency keys, such as
UUIDs, or have appropriate controls to validate the idempotency keys,
could be victim to various forms of security attacks from malicious
clients:
o Injection attacks - When the resources does not validate the
idempotency key in the client request and performs a idempotent
cache lookup, there can be security attacks (primarily in the form
of injection), compromising the server.
o Data leaks-When an idempotency implementation allows low entropy
keys, attackers MAY determine other keys and use them to fetch
existing idempotent cache entries, belonging to other clients.
To prevent such situations, the specification recommends the
following best practices for idempotency key implementation in the
resource.
o Establish a fixed format for the idempotency key and publish the
key's specification.
o Always validate the key as per its published specification before
processing any request.
o On the resource, implement a unique composite key as the
idempotent cache lookup key. For example, a composite key MAY be
Jena & Dalal Expires September 28, 2024 [Page 11]
Internet-Draft The Idempotency-Key HTTP Header Field March 2024
implemented by combining the idempotency key sent by the client
with other client specific attributes known only to the resource.
6. Examples
The first example shows an idempotency-key header field with key
value using UUID version 4 scheme:
Idempotency-Key: "8e03978e-40d5-43e8-bc93-6894a57f9324"
Second example shows an idempotency-key header field with key value
using a random string generator:
Idempotency-Key: "clkyoesmbgybucifusbbtdsbohtyuuwz"
7. References
7.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
.
[RFC4122] Leach, P., Mealling, M., and R. Salz, "A Universally
Unique IDentifier (UUID) URN Namespace", RFC 4122,
DOI 10.17487/RFC4122, July 2005,
.
[RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", STD 68, RFC 5234,
DOI 10.17487/RFC5234, January 2008,
.
[RFC7231] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer
Protocol (HTTP/1.1): Semantics and Content", RFC 7231,
DOI 10.17487/RFC7231, June 2014,
.
[RFC7807] Nottingham, M. and E. Wilde, "Problem Details for HTTP
APIs", RFC 7807, DOI 10.17487/RFC7807, March 2016,
.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, .
Jena & Dalal Expires September 28, 2024 [Page 12]
Internet-Draft The Idempotency-Key HTTP Header Field March 2024
[RFC8941] Nottingham, M. and P. Kamp, "Structured Field Values for
HTTP", RFC 8941, DOI 10.17487/RFC8941, February 2021,
.
[RFC9110] Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke,
Ed., "HTTP Semantics", STD 97, RFC 9110,
DOI 10.17487/RFC9110, June 2022,
.
7.2. Informative References
[RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running
Code: The Implementation Status Section", BCP 205,
RFC 7942, DOI 10.17487/RFC7942, July 2016,
.
7.3. URIs
[1] https://github.com/paypal/api-standards/blob/master/
patterns.md#idempotency
[2] https://stripe.com/docs/idempotency
[3] https://tools.ietf.org/html/draft-nottingham-http-poe-00
Acknowledgments
The authors would like to thank Mark Nottingham for his support for
this Internet Draft. We would like to acknowledge that this draft is
inspired by Idempotency related patterns described in API
documentation of PayPal [1] and Stripe [2] as well as Internet Draft
on POST Once Exactly [3] authored by Mark Nottingham.
The authors take all responsibility for errors and omissions.
Authors' Addresses
Jayadeba Jena
Email: jayadebaj@gmail.com
Sanjay Dalal
Email: sanjay.dalal@cal.berkeley.edu
URI: https://github.com/sdatspun2
Jena & Dalal Expires September 28, 2024 [Page 13]