Urllib3 can automatically retry idempotent requests. This same mechanism also handles redirects. You can control the retries using the retries parameter to request. By default, urllib3 will retry requests 3 times and follow up to 3 redirects.
- PyPI page Home page Author: Andrey Petrov License: MIT Summary: HTTP library with thread-safe connection pooling, file post, and more. Latest version: 1.26.4 Required dependencies: ipaddress.
- Urllib3 ¶ urllib3 is a powerful, user-friendly HTTP client for Python. Much of the Python ecosystem already uses urllib3 and you should too. Urllib3 brings many critical features that are missing from the Python standard libraries.
- The urllib3 is an easy to use library which can be used in python program just importing the urllib3 package. What are the modules of urllib3 library? The urllib3 library comes with many modules which makes the HTTP programming easy for Python developers.
last modified July 6, 2020
Python urllib3 tutorial introduces the Python urllib3 module. We show how tograb data, post data, stream data, work with JSON, and use redirects.
ZetCode has also a concise Python tutorial.
The Hypertext Transfer Protocol (HTTP) is an application protocol for distributed, collaborative,hypermedia information systems. HTTP is the foundation of data communication for the World Wide Web.
urllib3 module is a powerful, sanity-friendly HTTP client for Python. It supports thread safety, connection pooling, client-side SSL/TLS verification, file uploads with multipart encoding, helpers for retrying requests and dealing with HTTP redirects, gzip and deflate encoding, and proxy for HTTP and SOCKS.
We install the
urllib3 module with
Python urllib3 version
The first program prints the version of the urllib3 module.
The program prints the version or
This is a sample output of the example.
Python urllib3 status
HTTP response status codes indicate whether a specific HTTP request has beensuccessfully completed. Responses are grouped in five classes:
- Informational responses (100–199)
- Successful responses (200–299)
- Redirects (300–399)
- Client errors (400–499)
- Server errors (500–599)
The example creates a GET request to the
webcode.me. It prints the status code of the response.
We create a
PoolManager to generate a request. It handles all of the details of connection pooling and thread safety.
This is the URL to which we send the request.
request() method, we make a GET request to the specifiedURL.
We print the status code of the response.
The 200 status code means that the request has succeeded.
Python urllib3 GET request
The HTTP GET method requests a representation of the specified resource.
The example sends a GET request to the
webcode.me webpage. It returns the HTML code of the home page.
A GET request is generated.
We get the data or the response and decode it into text.
This is the output.
Python urllib3 HEAD request
A HEAD request is a GET request without a message body.
In the example, we create a HEAD request to the
The response object contains the
headers dictionary, which has the various header fields, such as server and date.
From the output we can see that the web server of the website is nginx and the content type is HTML code.
Python urllib3 HTTPS request
urllib3 provides client-side TLS/SSL verification. For this, weneed to download the
certifi module. It is a carefully curatedcollection of Root Certificates for validating the trustworthiness of SSLcertificates while verifying the identity of TLS hosts. It has been extractedfrom the Requests project.
To reference the installed certificate authority (CA) bundle, we use the built-in
We create a GET request to the
We pass the root CA bundle to the
PoolManager. Without this CA bundle, the request would issue the following warning:
InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised..
Python urllib3 query parameters
Query parameters are the part of a uniform resource locator (URL) which assignsvalues to specified parameters. This is one way of sending data to thedestination server.
The query parameters are specified after the ? character. Multiple fields are separated with the &. Special characters, such as spaces, are encoded. In the above string, the space is encoded with the
In the example, we send a GET request with some query parameters to the
https://httpbin.org/get. The link simply returns some data back to the client, including the query parameters. The site is used for testing HTTP requests.
This is the payload to be sent.
The query parameters are specified with the
httpbin.org responded with a JSON string, which includes our payload as well.
Python urllib3 POST request
The HTTP POST method sends data to the server. It is often used when uploadinga file or when submitting a completed web form.
The example sends a POST request. The data is specified with the
This is the output.
Python urllib3 send JSON
In requests, such as POST or PUT, the client tells the server what type of data is actually sent with the
The example sends JSON data.
We encode the JSON data into binary format.
We specify the
Content-Type header in the request.
We decode the returned data back to text and print it to the console.
Python urllib3 binary data
In the following example, we download binary data.
The example downloads a small icon.
req.data is in a binary format, which we can directly write to the disk.
Python urllib3 stream data
Chunked transfer encoding is a streaming data transfer mechanism available sinceHTTP 1.1. In chunked transfer encoding, the data stream is divided into a series of non-overlapping chunks.
The chunks are sent out and received independently of one another. Each chunk ispreceded by its size in bytes.
False means that urllib3will stream the response content. The
stream() method iteratesover chunks of the response content. When streaming, we should call
release_conn() to release the http connection backto the connection pool so that it can be re-used.
In the example, we download a PDF file.
preload_content=False, we enable streaming.
We iterate over the chunks of data and save them to a file.
In the end, we release the connection.
Python urllib3 redirect
A redirect sends users and search engines to a different URL from the onethey originally requested. To follow redirects, we set the
The example follows a redirect.
This is the output.
Python urllib3 Flask example
In the following example, we send a request to a small Flask web application.Learn more about Flask web framework in Python Flask tutorial.
We need to install the
The application has one route. It sends the user agent and connection header fields of a request to the client.
In this program, we send a request to our Flask application.
Flask runs on port 5000 by default.
make_headers() helper method, we create a headers dictionary.
We send a GET request to the URL; we specify the headers dictionary.
We print the response to the terminal.
We run the Flask application.
From a different terminal, we launch the
In this tutorial, we have worked with the Python urllib3 module.
List all Python tutorials.
Websites can be accessed using the urllib module. You can use the urllib module to interact with any website in the world, no matter if you want to get data, post data or parse data.
If you want to do web scraping or data mining, you can use urllib but it’s not the only option. Urllib will just fetch the data, but if you want to emulate a complete web browser, there’s also a module for that.
Web Scraping in Python with BeautifulSoup & Scrapy Framework
We can download a webpages HTML using 3 lines of code:
The variable html will contain the webpage data in html formatting. Traditionally a web-browser like Google Chrome visualizes this data.
A web-browsers sends their name and version along with a request, this is known as the user-agent. Python can mimic this using the code below. The User-Agent string contains the name of the web browser and version number:
Given a web-page data, we want to extract interesting information. You could use the BeautifulSoup module to parse the returned HTML data.
You can use the BeautifulSoup module to:
There are several modules that try to achieve the same as BeautifulSoup: PyQuery and HTMLParser, you can read more about them here.
The code below posts data to a server: