Usage
The entrypoint to pyqwest is Client for async applications and
SyncClient for synchronous applications.
Clients are lightweight - while we generally expect you'll initialize them once for your application,
it should generally be fine to even create them per-operation. There is no close type of method on
a client because they share an application-scoped default transport, which is the actual connection
pool. This should feel familiar to those coming from Go's net/http.
With a client, you can use methods corresponding to the HTTP methods, or execute, to issue a request
and get back a full response.
Multipart forms
To send a multipart/form-data request, for example to upload files, pass a
Multipart object as the request content with Client, or a
SyncMultipart with SyncClient. Parts can be provided
as bytes or str for simple form fields, or as a Part /
SyncPart to set a filename or part headers such as a
content type. A part's content can also be an iterator of bytes to stream it. The
multipart boundary is generated automatically when constructing the request and the
content-type header is set to match it.
from pyqwest import Multipart, Part
async def file_chunks():
yield b"file "
yield b"content"
response = await client.post(
"https://httpbingo.org/post",
content=Multipart(
{
"field": "value",
"file": Part(
file_chunks(),
filename="hello.txt",
headers={"content-type": "text/plain"},
),
}
),
)
print(response.text())
from pyqwest import SyncMultipart, SyncPart
def file_chunks():
yield b"file "
yield b"content"
response = client.post(
"https://httpbingo.org/post",
content=SyncMultipart(
{
"field": "value",
"file": SyncPart(
file_chunks(),
filename="hello.txt",
headers={"content-type": "text/plain"},
),
}
),
)
print(response.text())
Transport
The default transport is setup to behave closely to a web browser, using standard root certificates
and having timeouts, TCP keepalive, etc configured in a reasonable way, borrowing from the defaults
of the Go net/http package. You may need a custom transport though to configure TLS settings or
timeouts, in which case you create an HTTPTransport or SyncHTTPTransport. Unlike clients, transports
are heavy, with connection pools. Generally you should only create one per application and ensure it
is closed.
TLS
The default transport will use standard root certificates that can access sites served via
https in the same way as a browser would. For internal use cases, you may use certificates
issued by a custom certificate authority. You can initialize an HTTPTransport
or SyncHTTPTransport with a CA certificate for this case.
If using mTLS with client certificates, just add tls_cert and tls_key similarly.
Middleware
HTTP middleware are themselves just Transport implementations that accept another
Transport to wrap it. By having the same signature, they can operate on any part of the
request/response lifecycle.
pyqwest includes the following middleware.
Retry
Retry middleware automatically reissues requests on errors. The default behavior is to
retry known-safe errors, which include connection errors and transient error responses for
GET, HEAD, PUT, and DELETE. Whether a request or response is retryable can be customized by
subclassing the middleware class and implementing should_retry_request or
should_retry_response, for example to match against request.url.
should_retry_request may return False to disable retries, True to retry if
should_retry_response also returns True, or a RetryMode explicitly.
RetryMode.BUFFERED retains streamed request content in memory as it is sent so it can be
replayed. This can increase peak memory usage by the full size of the body even when the
first attempt succeeds. A streamed request is not retried after reading its content has
raised, because the rest of the content is lost. When a failure or response arrives while
an attempt is waiting on the content for its next chunk, a retry replays what was already
read and then waits for that chunk. RetryMode.UNBUFFERED retains no streamed content and
makes streamed requests only retry connection errors. Content provided as bytes is
already replayable, so it follows the normal response retry policy in either mode.
from pyqwest import Client, HTTPTransport, Request
from pyqwest.middleware.retry import RetryMode, RetryTransport
class MyRetryTransport(RetryTransport):
def should_retry_request(self, request: Request) -> bool | RetryMode:
if request.url.endswith("/unsafe-method"):
return False
return RetryMode.UNBUFFERED
client = Client(transport=MyRetryTransport(HTTPTransport()))
await client.get("http://localhost/safe-method") # will retry on transient errors
await client.get("http://localhost/unsafe-method") # will not retry
from pyqwest import SyncClient, SyncHTTPTransport, SyncRequest
from pyqwest.middleware.retry import RetryMode, SyncRetryTransport
class MyRetryTransport(SyncRetryTransport):
def should_retry_request(self, request: SyncRequest) -> bool | RetryMode:
if request.url.endswith("/unsafe-method"):
return False
return RetryMode.UNBUFFERED
client = SyncClient(transport=MyRetryTransport(SyncHTTPTransport()))
client.get("http://localhost/safe-method") # will retry on transient errors
client.get("http://localhost/unsafe-method") # will not retry
Proxies
The transport can be configured to send all requests through a proxy by passing its URL.
The URL scheme may be http, https, socks5, or socks5h. Credentials in the URL
will be used for proxy authentication.
For more control, pass a Proxy object instead of a URL. It supports basic
authentication without embedding credentials in the URL, extra headers to send
to the proxy, restricting the proxy to http or https requests, and a
no_proxy exclusion list of hosts that should connect directly.
from pyqwest import Proxy
proxy = Proxy(
"http://localhost:8030",
auth=("user", "pass"),
headers={"x-tenant": "my-tenant"},
no_proxy="localhost, internal.example.com",
scheme="https",
)
A sequence of proxies can also be passed to apply multiple routing rules. The
first proxy matching a request is used. An empty sequence, like None,
configures no explicit proxy, in which case proxy environment variables such
as HTTP_PROXY still apply.
Unix domain sockets
On Unix, the transport can be configured to connect to a Unix domain socket for all
requests instead of the host and port of the URL. The URL still provides the scheme,
Host header, and path, and https URLs still use TLS over the socket.
Timeouts
The transport can be configured with timeouts for overall operations, connect, and reads.
Sync clients can override the transport's overall timeout per call by passing timeout.
Async calls can set a shorter deadline with asyncio.wait_for, or with asyncio.timeout on
Python 3.11 and later, while the transport's timeout still applies. With Trio, use
trio.fail_after: its deadline raises trio.TooSlowError, and the transport's timeout
still raises TimeoutError. Connect and read timeouts cannot be configured per call.
Redirects
By default, transports follow redirects automatically up to a maximum of 10, which can be
configured with max_redirects.
To instead receive 301 or 302 responses directly, set follow_redirects=False.
A request that exceeds max_redirects fails with pyqwest.TooManyRedirects.
When using pyqwest as an httpx transport, you may prefer to disable follow_redirects
so HTTPX handles them as usual, notably filling response.history.
Logging
pyqwest integrates with Python's standard logging
module using two loggers, both emitting at DEBUG level:
pyqwest.access— one summary line per HTTP request, in the same format as httpx, emitted once response headers are received.pyqwest— granular request lifecycle records, e.g. when a request is sent or fails without a response.
Unlike httpx, no records are emitted at INFO level, so enabling INFO for your whole
application will not also log every HTTP request. Opt in explicitly instead:
import logging
logging.basicConfig(
format="%(levelname)s [%(asctime)s] %(name)s - %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
logging.getLogger("pyqwest.access").setLevel(logging.DEBUG)
response = client.get("https://pyqwest.dev")
Will send log output such as:
Setting the pyqwest logger to DEBUG instead enables the full request lifecycle,
including the access records through standard logger inheritance. The two never
duplicate each other, and an explicit level on pyqwest.access always takes
precedence, so the access log can be enabled on its own or silenced while keeping
the granular records.
The log records are emitted directly from the Rust HTTP transports with no overhead
when neither logger is enabled for DEBUG. Requests that fail without a response,
such as connection errors, appear only on the pyqwest logger, keeping the access
log to requests with responses like httpx.
Worker threads
Requests are driven by a process-wide tokio runtime, built on first use with one worker thread per available core. You may want to cap the number of request worker threads, for example when running in a container sharing a node without setting CPU limits. Notably, container CPU requests don't enforce elastic limits and a worker will be spawned for each CPU on the shared node. This may cause excessive memory usage.
Set the TOKIO_WORKER_THREADS environment variable to use a custom value of
workers. The value must be numeric.
Load Balancing
pyqwest supports DNS load balancing, distributing requests across all the addresses returned for a host lookup rather than just one. This is commonly used with HTTP/2 where otherwise only a single backend server would ever be used.
HTTP/2 stream limit
Even when enable_dns_load_balancing is not enabled, pyqwest will distribute requests
across multiple connections to the same backend when a connection reaches the stream
limit set by the server. If there is no connection with available streams, pyqwest will
initiate a new connection and distribute streams to it, while also using the previous
connection when it becomes available. To cap the number of connections to any given
server, set max_connections_per_address.