What a realistic header set contains
Servers compare the headers you send against what a real browser would send for the same navigation. A lone User-Agent is not enough: Accept, Accept-Language, Accept-Encoding and the Sec-Fetch-* family must agree with each other.
Python dicts do not guarantee the wire order headers are sent in, so for strict targets prefer a client that preserves order, such as curl_cffi.
Headers browsers send
| Header | Typical value | Why it matters |
|---|---|---|
| User-Agent | Mozilla/5.0 … Chrome/131 | The default python-requests value is blocked by many sites. |
| Accept | text/html,application/xhtml+xml,… | Missing or */* is a common bot signal. |
| Accept-Language | en-US,en;q=0.9 | Real browsers always send it. |
| Sec-Fetch-Mode | navigate | Must be consistent with Sec-Fetch-Dest and Site. |
| Upgrade-Insecure-Requests | 1 | Present on top-level document navigations only. |
Related tools
Python · cURL converters cURL to Python requests
Paste a cURL command, get a runnable requests.request() call. Parsed locally, nothing is uploaded. Python · Scraper bypass Fix 403 Forbidden in python requests
Paste the failing request. Get a version with realistic headers, a Session and a curl_cffi fallback. FAQ
Why does python-requests get blocked with its default headers?
The default User-Agent is python-requests/x.y.z, which many sites reject outright. Sending a full, consistent browser header set avoids the cheapest bot checks.
Is setting headers enough to avoid bot detection?
No. Headers fix the easy checks. TLS fingerprinting and JavaScript challenges need a different client, such as curl_cffi, or a real browser.