Introduction to httpx: Python’s Modern HTTP Client

By /

Introduction to httpx: Python’s Modern HTTP Client

In the vast ecosystem of Python libraries, tools for interacting with the web hold a special place. For years, the requests library reigned supreme, offering a simple, elegant, and “human-friendly” way to make HTTP requests. It became the de facto standard, beloved by developers for its ease of use. However, as the landscape of web development and Python itself evolved, particularly with the rise of asynchronous programming, the need for a more modern HTTP client became apparent. Enter httpx.

httpx is a next-generation HTTP client for Python, designed to be fully featured, intuitive, and, crucially, offering both synchronous and asynchronous APIs. It aims to provide the same level of usability that made requests popular while incorporating support for modern web features like HTTP/2, robust connection pooling, type hinting, and first-class async support via asyncio and trio.

This article serves as a comprehensive introduction to httpx. We will explore its core concepts, delve into its synchronous and asynchronous capabilities, examine its advanced features, compare it with its predecessor requests, and discuss best practices for leveraging its power in your Python applications. By the end, you’ll understand why httpx is rapidly becoming the go-to choice for developers needing a capable and forward-looking HTTP client.

Why httpx? The Motivation Behind a New Client

Before diving into the specifics of httpx, it’s essential to understand the context and motivations that led to its creation. While requests is an excellent library, its design predates several significant shifts in the Python and web ecosystems:

  1. The Rise of Asynchronous Programming: Python’s asyncio module, introduced in Python 3.4 and significantly enhanced in subsequent versions, brought native asynchronous programming capabilities to the language. Async I/O is particularly beneficial for network-bound tasks like making HTTP requests, allowing applications to handle many concurrent connections efficiently without relying on threads. requests was built on a synchronous foundation, and adding native async support proved challenging without significant architectural changes. While workarounds like using thread pools exist, they don’t offer the same performance benefits or clean integration as a natively async library. httpx was designed from the ground up with async support as a primary goal.
  2. The Need for HTTP/2: HTTP/2 is a major revision of the HTTP network protocol, offering significant performance improvements over HTTP/1.1, such as multiplexing (sending multiple requests/responses over a single connection), header compression, and server push. Modern web applications increasingly leverage HTTP/2. requests, by default, only supports HTTP/1.1. While adapters exist, native, seamless HTTP/2 support is a core feature of httpx.
  3. Modern Python Features: Newer versions of Python introduced features like type hints (typing module), which improve code clarity, maintainability, and enable static analysis tools. httpx embraces these features, providing a fully type-annotated codebase.
  4. Desire for API Parity with Refinements: The requests API is widely praised. httpx intentionally maintains a high degree of API compatibility with requests, making the transition easier for developers. However, it also takes the opportunity to refine certain aspects and provide a more consistent experience between its sync and async interfaces.

httpx doesn’t aim to simply replace requests but rather to provide a modern alternative that addresses these evolving needs, offering a robust, performant, and future-proof solution for HTTP communication in Python.

Getting Started with httpx

Installation is straightforward using pip:

bash
pip install httpx

This installs the core httpx library with support for HTTP/1.1 and basic features. However, httpx leverages optional dependencies for certain functionalities, notably HTTP/2 support and advanced content decoding.

To install httpx with HTTP/2 support (using the h2 library):

bash
pip install httpx[http2]

You might also want support for Brotli encoding (a common compression algorithm often used with HTTP/2) or IDNA 2008 support for international domain names:

bash
pip install httpx[brotli]
pip install httpx[idna]

You can combine these extras:

bash
pip install httpx[http2,brotli,idna]

Or install all optional dependencies:

bash
pip install httpx[all]

It’s generally recommended to install the http2 extra if you anticipate interacting with modern web servers.

Core Concepts: The Synchronous API

One of the key strengths of httpx is its familiar API, closely mirroring requests. If you’ve used requests, the synchronous part of httpx will feel immediately comfortable.

Making Basic Requests

httpx provides top-level functions for common HTTP methods: GET, POST, PUT, DELETE, HEAD, OPTIONS, PATCH.

“`python
import httpx
import json

Making a GET request

try:
response = httpx.get(‘https://httpbin.org/get’)
response.raise_for_status() # Raise an exception for bad status codes (4xx or 5xx)
print(f”GET Status Code: {response.status_code}”)
# Accessing response content as text
# print(f”GET Response Text:\n{response.text}”)
# Accessing response content as JSON
data = response.json()
print(f”GET Response JSON Origin IP: {data.get(‘origin’)}”)

except httpx.RequestError as exc:
print(f”An error occurred while requesting {exc.request.url!r}: {exc}”)
except httpx.HTTPStatusError as exc:
print(f”Error response {exc.response.status_code} while requesting {exc.request.url!r}.”)
print(f”Response content: {exc.response.text}”)

Making a POST request with JSON data

post_data = {‘key’: ‘value’, ‘framework’: ‘httpx’}
try:
response = httpx.post(‘https://httpbin.org/post’, json=post_data)
response.raise_for_status()
print(f”\nPOST Status Code: {response.status_code}”)
post_response_data = response.json()
print(f”POST Response JSON (echoed data): {post_response_data.get(‘json’)}”)

except httpx.RequestError as exc:
print(f”An error occurred while requesting {exc.request.url!r}: {exc}”)
except httpx.HTTPStatusError as exc:
print(f”Error response {exc.response.status_code} while requesting {exc.request.url!r}.”)

Other methods work similarly:

response = httpx.put(‘https://httpbin.org/put’, data={‘key’: ‘new_value’})

response = httpx.delete(‘https://httpbin.org/delete’)

response = httpx.head(‘https://httpbin.org/get’)

response = httpx.options(‘https://httpbin.org/get’)

“`

Handling Responses

The Response object returned by httpx methods contains all the information about the server’s response.

  • response.status_code: The integer HTTP status code (e.g., 200, 404).
  • response.reason_phrase: The text representation of the status code (e.g., ‘OK’, ‘Not Found’). Note: This might be an empty string in HTTP/2.
  • response.headers: A case-insensitive dictionary-like object containing the response headers.
    python
    print(f"Content-Type Header: {response.headers.get('Content-Type')}")
    # Accessing headers case-insensitively
    print(f"content-type Header: {response.headers.get('content-type')}")
  • response.text: The response body decoded as a string. httpx attempts to guess the encoding based on headers (or defaults to UTF-8). You can explicitly set the encoding:
    python
    # response.encoding = 'iso-8859-1'
    # print(response.text)
  • response.content: The response body as raw bytes. Useful for non-textual content like images or files.
    python
    # image_response = httpx.get('https://httpbin.org/image/png')
    # with open('image.png', 'wb') as f:
    # f.write(image_response.content)
  • response.json(): Decodes the response body from JSON into a Python dictionary or list. Raises a json.JSONDecodeError if the body is not valid JSON.
  • response.url: The final URL after any redirects.
  • response.request: The Request object associated with this response.
  • response.history: A list of Response objects from any redirects that were followed. Empty if no redirects occurred.
  • response.elapsed: A timedelta object representing the time taken from sending the request to receiving the full response.
  • response.http_version: The HTTP protocol version used (e.g., ‘HTTP/1.1’, ‘HTTP/2’).

Passing Parameters in URLs (Query Parameters)

To add URL query parameters (e.g., ?key=value&foo=bar), use the params argument:

“`python
params = {‘key1’: ‘value1’, ‘key2’: [‘value2a’, ‘value2b’]}

Results in URL: https://httpbin.org/get?key1=value1&key2=value2a&key2=value2b

response = httpx.get(‘https://httpbin.org/get’, params=params)
print(f”\nGET with Params URL: {response.url}”)
print(f”GET with Params Response Args: {response.json().get(‘args’)}”)
“`

Sending Request Body Data

httpx offers flexible ways to send data in the request body, typically used with POST, PUT, and PATCH requests.

  • Form Encoded Data (data): Sends data as application/x-www-form-urlencoded (like an HTML form submission). Pass a dictionary to the data argument.
    python
    form_data = {'username': 'user', 'password': 'password123'}
    response = httpx.post('https://httpbin.org/post', data=form_data)
    print(f"\nPOST Form Data Response (form): {response.json().get('form')}")
  • JSON Encoded Data (json): Automatically encodes a Python object (usually a dict or list) as JSON and sets the Content-Type header to application/json.
    python
    json_payload = {'id': 123, 'name': 'example', 'active': True}
    response = httpx.post('https://httpbin.org/post', json=json_payload)
    print(f"\nPOST JSON Data Response (json): {response.json().get('json')}")
  • Raw Bytes (content): Sends raw bytes. You should usually set the Content-Type header manually.
    python
    binary_data = b'\x00\x10\x20\x30'
    headers = {'Content-Type': 'application/octet-stream'}
    response = httpx.post('https://httpbin.org/post', content=binary_data, headers=headers)
    print(f"\nPOST Binary Data Response (data): {response.json().get('data')}")

  • Multipart File Uploads (files): Sends data as multipart/form-data, typically used for file uploads.
    “`python
    files_to_upload = {‘file’: (‘report.txt’, b’This is the content of the report.’, ‘text/plain’)}
    # You can also send additional form data alongside files
    other_data = {‘description’: ‘Monthly Report’}
    response = httpx.post(‘https://httpbin.org/post’, files=files_to_upload, data=other_data)
    post_files_response = response.json()
    print(f”\nPOST Files Response (files): {post_files_response.get(‘files’)}”)
    print(f”POST Files Response (form data): {post_files_response.get(‘form’)}”)

    Uploading an actual file from disk

    with open(‘my_image.jpg’, ‘rb’) as f:

    files = {‘image_file’: (‘photo.jpg’, f, ‘image/jpeg’)}

    response = httpx.post(‘https://httpbin.org/post’, files=files)

    “`

Custom Headers

Pass custom HTTP headers using the headers argument, which accepts a dictionary:

python
custom_headers = {
'User-Agent': 'MyAwesomeApp/1.0',
'X-Custom-Header': 'custom-value'
}
response = httpx.get('https://httpbin.org/headers', headers=custom_headers)
print(f"\nGET with Custom Headers Response: {response.json().get('headers')}")

Timeouts

Network operations can sometimes hang indefinitely. httpx allows setting timeouts to prevent this. Timeouts can be specified as a single float (total timeout) or as an httpx.Timeout object for finer control.

  • Default Timeout: 5 seconds for all operations (connect, read, write, pool).
  • Single Value: Applies to connect, read, and write timeouts.
    python
    try:
    # Wait max 1 second for the entire operation
    response = httpx.get('https://httpbin.org/delay/3', timeout=1.0)
    except httpx.TimeoutException as exc:
    print(f"\nTimeout occurred: {exc}")

  • httpx.Timeout Object: Allows specifying individual timeouts.
    “`python
    # 1s connect, 5s read, None (infinite) write, None (infinite) pool
    timeout_config = httpx.Timeout(1.0, read=5.0, write=None, pool=None)
    try:
    response = httpx.get(‘https://httpbin.org/delay/6’, timeout=timeout_config)
    except httpx.ReadTimeout as exc: # More specific exception
    print(f”\nRead Timeout occurred: {exc}”)

    Disable timeouts completely

    response = httpx.get(‘…’, timeout=None)

    ``
    The four types of timeouts are:
    *    connect: Time allowed for establishing the connection.
    *    read: Time allowed between consecutive read operations (data chunks arriving).
    *    write: Time allowed between consecutive write operations (sending data chunks).
    *    pool`: Time allowed to wait for a connection from the connection pool.

Error Handling

Network requests can fail in various ways. httpx has a hierarchy of exceptions:

  • httpx.RequestError: Base class for exceptions occurring during the request process (e.g., connection issues, timeouts).
    • httpx.ConnectError: Could not establish a connection.
    • httpx.ReadError: Error reading data from the server.
    • httpx.WriteError: Error sending data to the server.
    • httpx.PoolTimeout: Timed out waiting for a connection from the pool.
    • httpx.TimeoutException: Base class for specific timeouts.
      • httpx.ConnectTimeout
      • httpx.ReadTimeout
      • httpx.WriteTimeout
    • httpx.NetworkError: A general network error occurred.
    • httpx.ProtocolError: Low-level protocol error (e.g., invalid HTTP).
    • httpx.ProxyError: Error related to proxy connection.
    • httpx.TooManyRedirects
    • httpx.InvalidURL
  • httpx.HTTPStatusError: Raised by response.raise_for_status() when the response has a 4xx (client error) or 5xx (server error) status code. It subclasses httpx.RequestError and holds the request and response objects.

Good practice involves wrapping requests in try...except blocks to handle potential httpx.RequestError and httpx.HTTPStatusError exceptions.

“`python
url = ‘https://httpbin.org/status/404’
try:
response = httpx.get(url, timeout=5.0)
response.raise_for_status() # Checks for 4xx/5xx status codes
# Process successful response
print(f”\nSuccessfully fetched {url}”)

except httpx.ConnectError as exc:
print(f”\nConnection error to {exc.request.url!r}: {exc}”)
except httpx.ReadTimeout as exc:
print(f”\nRead timeout requesting {exc.request.url!r}: {exc}”)
except httpx.HTTPStatusError as exc:
print(f”\nHTTP error {exc.response.status_code} for {exc.request.url!r}. ”
f”Reason: {exc.response.reason_phrase}”)
# You can inspect the response that caused the error
# print(f”Response content: {exc.response.text}”)
except httpx.RequestError as exc:
# Catch any other httpx request-related errors
print(f”\nAn unexpected httpx error occurred: {exc}”)
except Exception as exc:
# Catch non-httpx errors
print(f”\nA non-httpx error occurred: {exc}”)
“`

The Power of Async: The Asynchronous API

This is where httpx truly shines and differentiates itself from requests. The asynchronous API allows you to make concurrent HTTP requests efficiently without blocking the execution thread, ideal for I/O-bound applications like web servers, scrapers, or API clients handling many simultaneous operations.

httpx supports both asyncio (Python’s built-in async library) and trio (an alternative async library). We’ll focus on asyncio here.

Introducing AsyncClient

Instead of using top-level functions like httpx.get(), the async API primarily uses an httpx.AsyncClient instance. This client manages connection pools and other resources efficiently across multiple asynchronous requests.

“`python
import httpx
import asyncio

async def fetch_url(client, url):
“””Asynchronous function to fetch a URL.”””
try:
print(f”Starting request to {url}”)
response = await client.get(url, timeout=10.0)
response.raise_for_status()
print(f”Finished request to {url}, Status: {response.status_code}, HTTP Version: {response.http_version}”)
return response.json() # Or response.text, response.content, etc.
except httpx.HTTPStatusError as exc:
print(f”HTTP error {exc.response.status_code} for {exc.request.url!r}”)
return None
except httpx.RequestError as exc:
print(f”Request error for {exc.request.url!r}: {exc}”)
return None

async def main():
“””Main async function to demonstrate AsyncClient.”””
urls = [
‘https://httpbin.org/get?id=1’,
‘https://httpbin.org/delay/1’, # Simulate a 1-second delay
‘https://httpbin.org/get?id=2’,
‘https://httpbin.org/status/404’, # Will cause an HTTPStatusError
‘https://httpbin.org/delay/2’, # Simulate a 2-second delay
‘https://invalid-domain-that-does-not-exist.xyz’, # Will cause a ConnectError
‘https://httpbin.org/get?id=3’
]

# Create an AsyncClient instance. It's often used with 'async with'.
# 'async with' ensures the client and its connections are properly closed.
async with httpx.AsyncClient(http2=True) as client: # Enable HTTP/2
    # Create a list of tasks to run concurrently
    tasks = [fetch_url(client, url) for url in urls]

    # Run tasks concurrently and gather results
    # asyncio.gather runs the awaitables (tasks) concurrently.
    # return_exceptions=True prevents gather from stopping on the first exception.
    results = await asyncio.gather(*tasks, return_exceptions=True)

    print("\n--- Results ---")
    for url, result in zip(urls, results):
        if isinstance(result, Exception):
            print(f"URL: {url}, Result: Error ({type(result).__name__}: {result})")
        elif result is None:
            print(f"URL: {url}, Result: Failed (handled error)")
        else:
            # Process successful results (here just printing origin IP)
            origin = result.get('origin', 'N/A')
            args = result.get('args', {})
            print(f"URL: {url}, Result: Success (Origin: {origin}, Args: {args})")

To run the async code

if name == “main“:
print(“Running async main function…”)
# In Python 3.7+, asyncio.run is the standard way to run the top-level async function
asyncio.run(main())
print(“Async main function finished.”)

“`

Key points about the async example:

  1. async def: Functions using await must be defined with async def.
  2. await: The await keyword pauses the execution of the current coroutine (fetch_url), allowing the event loop to run other tasks (like starting other requests) until the awaited operation (e.g., client.get) completes.
  3. httpx.AsyncClient(): Creates the client instance. Using async with is crucial for resource management (closing connections). It supports arguments like http2=True.
  4. await client.get(...): All request methods on AsyncClient are coroutines and must be awaited.
  5. asyncio.gather: A standard way to run multiple awaitables concurrently. The requests to /delay/1 and /delay/2 will run largely in parallel with the other requests, significantly reducing the total execution time compared to making them sequentially.
  6. asyncio.run(main()): The entry point to start the asyncio event loop and run the main asynchronous function.

The benefits of the async approach become clear when dealing with multiple I/O-bound operations. Instead of waiting idly for one request to finish before starting the next (like the synchronous approach), asyncio allows the program to switch between tasks whenever one is waiting for I/O, leading to much higher throughput.

Advanced Features

Beyond basic requests, httpx offers a rich set of features for more complex scenarios.

Client Instances (httpx.Client / httpx.AsyncClient)

While top-level functions like httpx.get() are convenient for single requests, using client instances (httpx.Client for sync, httpx.AsyncClient for async) is recommended for multiple requests to the same host or for configuring settings across requests.

Benefits:

  1. Connection Pooling: Clients maintain underlying TCP connections, reusing them for subsequent requests to the same origin (host/port/scheme). This significantly reduces the latency associated with establishing new connections, especially for HTTPS where TLS handshakes are involved.
  2. Configuration Persistence: You can set base URLs, default headers, cookies, query parameters, authentication, timeouts, and other settings on the client instance, which will apply to all requests made through that client.
  3. Resource Management: Using the client as a context manager (with httpx.Client() as client: or async with httpx.AsyncClient() as client:) ensures that connections and resources are properly cleaned up.

Synchronous Example:

“`python
import httpx

Configure a client with base URL, default headers, and timeout

base_url = “https://httpbin.org”
headers = {“Client-Type”: “Sync-Example”}
timeout = httpx.Timeout(10.0)

Use ‘with’ for automatic cleanup

with httpx.Client(base_url=base_url, headers=headers, timeout=timeout, http2=True) as client:
try:
# Relative URLs are resolved against the base_url
response_get = client.get(“/get”)
response_get.raise_for_status()
print(f”\nClient GET Status: {response_get.status_code}, HTTP: {response_get.http_version}”)
print(f”Client GET Headers Sent: {response_get.json()[‘headers’]}”)

    response_post = client.post("/post", json={"data": 123})
    response_post.raise_for_status()
    print(f"\nClient POST Status: {response_post.status_code}, HTTP: {response_post.http_version}")
    print(f"Client POST Headers Sent: {response_post.json()['headers']}")

except httpx.RequestError as exc:
    print(f"An error occurred: {exc}")

Client connections are closed automatically upon exiting the ‘with’ block

“`

Asynchronous Example (similar structure):

“`python
import httpx
import asyncio

async def async_client_example():
base_url = “https://httpbin.org”
headers = {“Client-Type”: “Async-Example”}
timeout = httpx.Timeout(10.0)

# Use 'async with' for automatic cleanup in async context
async with httpx.AsyncClient(base_url=base_url, headers=headers, timeout=timeout, http2=True) as client:
    try:
        # Make requests concurrently
        task1 = client.get("/get")
        task2 = client.post("/post", json={"data": 456})

        response_get, response_post = await asyncio.gather(task1, task2)

        response_get.raise_for_status()
        print(f"\nAsync Client GET Status: {response_get.status_code}, HTTP: {response_get.http_version}")
        print(f"Async Client GET Headers Sent: {response_get.json()['headers']}")

        response_post.raise_for_status()
        print(f"\nAsync Client POST Status: {response_post.status_code}, HTTP: {response_post.http_version}")
        print(f"Async Client POST Headers Sent: {response_post.json()['headers']}")

    except httpx.RequestError as exc:
        print(f"An error occurred: {exc}")

if name == “main“:

asyncio.run(async_client_example())

“`

HTTP/2 Support

As mentioned, httpx supports HTTP/2 out of the box if the required dependency (h2) is installed (pip install httpx[http2]). When interacting with a server that supports HTTP/2, httpx will automatically negotiate and use it via ALPN (Application-Layer Protocol Negotiation) during the TLS handshake.

You can enable/prefer HTTP/2 explicitly when creating a client:

“`python

Prefer HTTP/2, but allow fallback to HTTP/1.1

client = httpx.Client(http2=True)
async_client = httpx.AsyncClient(http2=True)

Require HTTP/1.1 only

client = httpx.Client(http1=True, http2=False) # Default behavior if h2 not installed

async_client = httpx.AsyncClient(http1=True, http2=False)

Require HTTP/2 only (will raise error if server doesn’t support it)

client = httpx.Client(http1=False, http2=True)

async_client = httpx.AsyncClient(http1=False, http2=True)

“`

The response.http_version attribute will confirm which protocol was actually used (e.g., ‘HTTP/1.1’ or ‘HTTP/2’).

Authentication

httpx provides built-in support for common authentication schemes and a flexible system for custom authentication.

  • Basic & Digest Auth: Pass a tuple (username, password) to the auth argument. httpx automatically determines whether to use Basic or Digest based on the server’s WWW-Authenticate response (if any) for the initial 401 Unauthorized status.
    “`python
    auth_creds = (‘testuser’, ‘testpass’)
    # Sync
    response = httpx.get(‘https://httpbin.org/basic-auth/testuser/testpass’, auth=auth_creds)
    print(f”\nBasic Auth Status: {response.status_code}”) # Should be 200 if creds are correct
    # print(response.json())

    Async

    async with httpx.AsyncClient() as client:

    response = await client.get(‘https://httpbin.org/basic-auth/testuser/testpass’, auth=auth_creds)

    print(f”Async Basic Auth Status: {response.status_code}”)

    * **Custom Authentication (`httpx.Auth`):** For more complex schemes (like custom token auth, OAuth signing, Hawk), you can create a custom authentication class by inheriting from `httpx.Auth` and implementing the `auth_flow` method (a generator) or `sync_auth_flow` / `async_auth_flow` methods.python
    import httpx

    class TokenAuth(httpx.Auth):
    def init(self, token):
    self._token = token

    def auth_flow(self, request):
    # Modify the request to add the Authorization header
    request.headers['Authorization'] = f"Bearer {self._token}"
    yield request # Send the modified request

    Usage (Sync)

    my_token = “your_secret_api_token”

    response = httpx.get(“https://api.example.com/data”, auth=TokenAuth(my_token))

    Usage (Async) – requires implementing async_auth_flow or using auth_flow if it’s compatible

    async with httpx.AsyncClient(auth=TokenAuth(my_token)) as client:

    response = await client.get(“https://api.example.com/data”)

    Note: For complex OAuth flows, dedicated libraries like ‘authlib’ often integrate with httpx.

    “`

Proxies

httpx supports routing requests through HTTP, HTTPS, SOCKS4, and SOCKS5 proxies.

  • Configuring Proxies: Pass a dictionary mapping URL schemes (or specific domains) to proxy URLs via the proxies argument.
    “`python
    # Route all HTTP and HTTPS traffic through ‘http://localhost:8080’
    proxy_config = {
    “http://”: “http://localhost:8080”,
    “https://”: “http://localhost:8080”,
    }

    Route only traffic to ‘example.com’ through a SOCKS5 proxy

    Requires ‘pip install httpx[socks]’

    proxy_config_socks = {

    “all://example.com”: “socks5://localhost:9050”, # Route HTTP/HTTPS for example.com via SOCKS

    “all://”: None # Direct connection for other domains

    }

    Usage (Sync)

    with httpx.Client(proxies=proxy_config) as client:

    response = client.get(“http://example.com”) # Goes via proxy

    response_secure = client.get(“https://example.com”) # Goes via proxy

    Usage (Async)

    async with httpx.AsyncClient(proxies=proxy_config) as client:

    response = await client.get(“http://google.com”) # Goes via proxy

    ``
    * **Environment Variables:**    httpxalso respects standard proxy environment variables likeHTTP_PROXY,HTTPS_PROXY, andALL_PROXY.
    * **Proxy Authentication:** If the proxy requires authentication, include credentials in the proxy URL:    http://user:password@localhost:8080`.

Cookies

httpx handles cookies automatically when using a Client or AsyncClient.

  • Receiving Cookies: Cookies set by the server (Set-Cookie header) are stored in the client’s cookie jar.
  • Sending Cookies: Cookies stored in the client’s jar are automatically sent back to the appropriate domains on subsequent requests (Cookie header).

“`python

Sync Example

with httpx.Client() as client:
# Request a page that sets a cookie
response1 = client.get(‘https://httpbin.org/cookies/set?mycookie=12345’)
print(f”\nCookies Response 1 Headers: {response1.headers.get(‘Set-Cookie’)}”)

# Check the client's cookie jar
print(f"Client Cookies after request 1: {client.cookies}")

# Make another request to a path where cookies are expected
response2 = client.get('https://httpbin.org/cookies')
# The 'mycookie=12345' should be sent back automatically
print(f"Cookies Response 2 (sent cookies): {response2.json()}")

You can also manually set cookies on the client or per-request

client = httpx.Client(cookies={“sessionid”: “abc”})
response = client.get(“https://example.com”)

Per-request cookies

response = httpx.get(“https://httpbin.org/cookies”, cookies={“persistent”: “yes”})
print(f”\nManual Cookie Request: {response.json()}”)

AsyncClient works similarly with cookies.

“`

Redirects and History

By default, httpx automatically follows redirects (like 301, 302, 307, 308).

  • Disabling Redirects: Set follow_redirects=False.
    “`python
    # Sync
    response = httpx.get(‘https://httpbin.org/redirect/2’, follow_redirects=False)
    print(f”\nNo Redirect Status: {response.status_code}”) # Will be 302
    print(f”No Redirect Location Header: {response.headers.get(‘Location’)}”)
    print(f”Is Redirect: {response.is_redirect}”) # True

    Async

    async with httpx.AsyncClient() as client:

    response = await client.get(‘…’, follow_redirects=False)

    * **Inspecting History:** When redirects are followed, the `response.history` attribute contains a list of the `Response` objects from the intermediate redirects, in order. The final `response` object represents the destination after following all redirects.python
    response = httpx.get(‘https://httpbin.org/redirect/3’) # Follows 3 redirects
    print(f”\nRedirect Final Status: {response.status_code}”) # Should be 200 (final destination)
    print(f”Redirect Final URL: {response.url}”)
    print(f”Redirect History Count: {len(response.history)}”)
    for i, resp in enumerate(response.history):
    print(f” Redirect {i+1}: Status={resp.status_code}, URL={resp.url}, Location={resp.headers.get(‘Location’)}”)
    ``
    * **Limiting Redirects:** Use the    max_redirects` parameter on the client or request.

Streaming Requests and Responses

For handling large amounts of data without loading everything into memory at once, httpx supports streaming.

  • Streaming Request Bodies: Pass an iterator (sync) or an async iterator (async) to the content parameter. This is useful for uploading large files generated on-the-fly.
    “`python
    # Sync Example: Stream generated data
    def data_generator():
    yield b”Part 1, ”
    yield b”Part 2, ”
    yield b”Part 3.”

    response = httpx.post(‘https://httpbin.org/post’, content=data_generator())

    Async Example: Stream generated data

    async def async_data_generator():
    yield b”Async Part 1, ”
    await asyncio.sleep(0.1) # Simulate work
    yield b”Async Part 2, ”
    await asyncio.sleep(0.1)
    yield b”Async Part 3.”

    async with httpx.AsyncClient() as client:

    response = await client.post(‘https://httpbin.org/post’, content=async_data_generator())

    print(f”\nAsync Stream Upload Response Data: {response.json()[‘data’]}”)

    * **Streaming Response Bodies:** Use the `stream=True` parameter in the request function. This prevents the response body from being downloaded immediately. You then need to use the response object as a context manager (`with response:` or `async with response:`) and iterate over the content using methods like `iter_bytes()`, `iter_text()`, `iter_lines()`, or their async counterparts (`aiter_bytes()`, `aiter_text()`, `aiter_lines()`).python

    Sync Example: Download and save a large file

    url = “https://speed.hetzner.de/100MB.bin” # Example large file URL

    try:

    with httpx.stream(“GET”, url, timeout=None) as response: # Use top-level stream

    response.raise_for_status()

    print(f”\nStreaming download from {url}…”)

    with open(“downloaded_file.bin”, “wb”) as f:

    for chunk in response.iter_bytes(chunk_size=8192): # Process in chunks

    f.write(chunk)

    print(“Download complete.”)

    except httpx.RequestError as exc:

    print(f”Error during streaming download: {exc}”)

    Async Example: Process streamed text data

    async def stream_lines_example():
    url = “https://httpbin.org/stream/5” # Endpoint that streams 5 JSON lines
    async with httpx.AsyncClient() as client:
    try:
    async with client.stream(“GET”, url) as response:
    response.raise_for_status()
    print(f”\nStreaming lines from {url}:”)
    async for line in response.aiter_lines():
    print(f” Received line: {line}”)
    except httpx.RequestError as exc:
    print(f”Error during async streaming: {exc}”)

    if name == “main“:

    asyncio.run(stream_lines_example())

    ``
    **Important:** When using    stream=True, you *must* ensure the response content is consumed or the response context manager is closed (response.close()). Usingwith response:orasync with response:` handles this automatically. Failing to do so can leave connections open and lead to resource leaks.

Event Hooks

httpx allows you to register callback functions (hooks) that are executed during the request/response cycle. This is useful for logging, monitoring, modifying requests/responses on the fly, or implementing custom caching logic.

Available hooks:
* request: Called just before sending the request.
* response: Called just after receiving the response headers, before reading the body (unless stream=True).

“`python
def log_request(request):
print(f”>>> Sending request: {request.method} {request.url}”)
# print(f” Headers: {request.headers}”)

def log_response(response):
# Note: Accessing response.text or response.content here will read the
# entire body, potentially multiple times if other hooks also access it.
# Use response.request to link back to the original request.
request = response.request
print(f”<<< Received response: {response.status_code} for {request.method} {request.url}”)
# For streaming responses, read() must be called to trigger this hook fully.
# await response.aread() # If async

Register hooks on a Client

hooks = {“request”: [log_request], “response”: [log_response]}

with httpx.Client(event_hooks=hooks) as client:

client.get(“https://httpbin.org/get”)

client.post(“https://httpbin.org/post”, json={‘hook’: ‘test’})

Register hooks per-request

response = httpx.get(“https://httpbin.org/get”, event_hooks=hooks)

Async hooks must be async functions

async def async_log_request(request):
print(f”>>> Sending async request: {request.method} {request.url}”)

async def async_log_response(response):
request = response.request
print(f”<<< Received async response: {response.status_code} for {request.method} {request.url}”)
# If streaming, ensure body is read or response closed for hook completion
# await response.aread()

async_hooks = {“request”: [async_log_request], “response”: [async_log_response]}

async with httpx.AsyncClient(event_hooks=async_hooks) as client:

await client.get(“https://httpbin.org/get”)

“`

Transport API and Mounts

For advanced customization, httpx exposes a Transport API. Transports handle the low-level details of sending requests and receiving responses (connection pooling, HTTP protocol implementation, proxy handling, etc.).

The mounts argument of a client allows you to map URL prefixes to specific transport instances. This is powerful for:

  • Testing WSGI/ASGI Applications: Mount a transport that directly calls your local web application without going through the network stack.
  • Custom Connection Logic: Use different proxy settings or TLS configurations for specific domains.
  • Mocking: Mount a mock transport for testing purposes.

“`python
import httpx
from httpx import WSGITransport # Or ASGITransport for async apps

Example: Testing a simple WSGI app

def my_wsgi_app(environ, start_response):

status = ‘200 OK’

output = b”Hello from WSGI App!”

response_headers = [(‘Content-type’, ‘text/plain’),

(‘Content-Length’, str(len(output)))]

start_response(status, response_headers)

return [output]

wsgi_transport = WSGITransport(app=my_wsgi_app)

Mount the transport to handle requests to ‘http://testserver’

with httpx.Client(mounts={‘http://testserver’: wsgi_transport}) as client:

response = client.get(“http://testserver/some/path”)

print(f”\nWSGI Mount Response Status: {response.status_code}”)

print(f”WSGI Mount Response Text: {response.text}”)

Similarly for ASGI apps using ASGITransport and AsyncClient

from httpx import ASGITransport

async def my_asgi_app(scope, receive, send): …

asgi_transport = ASGITransport(app=my_asgi_app)

async with httpx.AsyncClient(mounts={‘http://testserver’: asgi_transport}) as client:

response = await client.get(“http://testserver/”)

“`

Testing with httpx

httpx‘s design facilitates testing:

  1. Testing Local ASGI/WSGI Apps: As shown above, mounting ASGITransport or WSGITransport is the ideal way to test web frameworks like FastAPI, Starlette, Flask, or Django directly.
  2. Mocking External Services: For testing code that interacts with external APIs, you need to mock the HTTP requests.
    • unittest.mock: Python’s built-in mocking library can be used to patch httpx.Client.request, httpx.AsyncClient.request, or the top-level functions. This requires careful setup to return mock Response objects.
    • pytest-httpx: A popular pytest plugin that provides a convenient fixture (httpx_mock) for mocking httpx requests declaratively in tests.
      python
      # Example using pytest-httpx (requires installation)
      # import pytest
      # import httpx
      #
      # def my_api_call(url):
      # with httpx.Client() as client:
      # response = client.get(url)
      # response.raise_for_status()
      # return response.json()
      #
      # @pytest.mark.usefixtures("httpx_mock")
      # def test_api_call_success(httpx_mock):
      # test_url = "https://api.example.com/data"
      # mock_data = {"id": 1, "value": "mocked"}
      # httpx_mock.add_response(url=test_url, method="GET", json=mock_data, status_code=200)
      #
      # result = my_api_call(test_url)
      # assert result == mock_data
      #
      # @pytest.mark.usefixtures("httpx_mock")
      # def test_api_call_failure(httpx_mock):
      # test_url = "https://api.example.com/data"
      # httpx_mock.add_response(url=test_url, method="GET", status_code=500)
      #
      # with pytest.raises(httpx.HTTPStatusError):
      # my_api_call(test_url)

httpx vs. requests – A Quick Comparison

Feature httpx requests
API Style Sync & Async (Client, AsyncClient) Sync only (Session)
Async Support Native (asyncio, trio) No native support (requires threads/workarounds)
HTTP Version HTTP/1.1 & HTTP/2 (optional dep h2) HTTP/1.1 only (HTTP/2 via third-party adapters)
Client Usage Client/AsyncClient recommended Session recommended
API Compatibility High degree of compatibility with requests The established standard API
Type Hinting Fully type-hinted codebase Limited/gradual type hinting
Dependencies Minimal core; optional extras for HTTP/2 etc. Bundles dependencies (e.g., urllib3)
Streaming Robust sync/async streaming (stream=True) Sync streaming (stream=True)
Timeouts Granular control (httpx.Timeout) Tuple (connect, read) or single float
Testing (WSGI/ASGI) Built-in WSGITransport/ASGITransport Requires separate libraries (e.g., werkzeug)
Project Status Actively developed, modern features Mature, stable, widely adopted

When to Choose httpx:

  • You need asynchronous HTTP requests (asyncio or trio).
  • You want native HTTP/2 support.
  • You are starting a new project and want a modern, type-hinted library.
  • You need to test ASGI/WSGI applications directly.
  • You appreciate the refined API details and granular timeout control.

When requests Might Still Be Suitable:

  • You only need synchronous requests and have no immediate plans for async.
  • You are working on a legacy codebase heavily reliant on requests.
  • You prefer the stability and vast community resources of a more mature library.
  • You cannot easily add the h2 dependency for HTTP/2.

For most new Python projects involving HTTP requests, especially those leveraging modern Python features or requiring high concurrency, httpx is arguably the better choice moving forward.

Best Practices for Using httpx

  1. Use Client Instances: For multiple requests, always prefer httpx.Client or httpx.AsyncClient over top-level functions to benefit from connection pooling and persistent settings.
  2. Use Context Managers: Always use clients within with (sync) or async with (async) blocks to ensure proper resource cleanup.
  3. Handle Exceptions: Wrap requests in try...except blocks, catching httpx.HTTPStatusError for bad responses and httpx.RequestError (or more specific subclasses like ConnectError, ReadTimeout) for network/protocol issues.
  4. Set Sensible Timeouts: Avoid infinite waits by setting appropriate timeouts, either globally on the client or per-request. Use httpx.Timeout for fine-grained control if needed.
  5. Use Streaming for Large Data: Employ stream=True and iter_*/aiter_* methods when dealing with large request or response bodies to avoid high memory consumption. Remember to consume or close streamed responses.
  6. Leverage Async for Concurrency: If your application involves many I/O-bound HTTP calls, use the httpx.AsyncClient and asyncio/trio features (asyncio.gather, etc.) to achieve concurrency and improve performance.
  7. Install Optional Extras: Install extras like [http2] if you expect to interact with modern servers supporting it.
  8. Check response.raise_for_status(): Call this method if you want your program to treat 4xx/5xx responses as errors rather than successes.
  9. Consult the Documentation: httpx has excellent official documentation covering all features in detail.

Conclusion

httpx represents a significant step forward for HTTP clients in the Python ecosystem. By providing a familiar requests-like API while seamlessly integrating first-class asynchronous support and modern features like HTTP/2, it addresses the evolving needs of contemporary Python development. Its design emphasizes correctness, usability, and performance, offering features like robust connection pooling, granular timeouts, streaming capabilities, and a flexible authentication system.

Whether you are building high-performance async web services, scraping websites concurrently, interacting with numerous third-party APIs, or simply need a reliable HTTP client for a synchronous application, httpx provides the tools you need. Its high compatibility with the requests API makes migration relatively smooth for many projects, while its new capabilities unlock possibilities previously difficult to achieve.

As Python’s asynchronous ecosystem continues to mature, httpx is well-positioned to be the standard library for network communication, empowering developers to build faster, more efficient, and more resilient applications. If you’re making HTTP requests in Python today, httpx deserves your serious consideration.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top