Urllib3

• Urllib3 Open
• Urllib3 Download
• Urllib3 Retry
• Urllib3 Install

Urllib3 can automatically retry idempotent requests. This same mechanism also handles redirects. You can control the retries using the retries parameter to request. By default, urllib3 will retry requests 3 times and follow up to 3 redirects.

• PyPI page Home page Author: Andrey Petrov License: MIT Summary: HTTP library with thread-safe connection pooling, file post, and more. Latest version: 1.26.4 Required dependencies: ipaddress.
• Urllib3 ¶ urllib3 is a powerful, user-friendly HTTP client for Python. Much of the Python ecosystem already uses urllib3 and you should too. Urllib3 brings many critical features that are missing from the Python standard libraries.
• The urllib3 is an easy to use library which can be used in python program just importing the urllib3 package. What are the modules of urllib3 library? The urllib3 library comes with many modules which makes the HTTP programming easy for Python developers.

last modified July 6, 2020

Python urllib3 tutorial introduces the Python urllib3 module. We show how tograb data, post data, stream data, work with JSON, and use redirects.

ZetCode has also a concise Python tutorial.

The Hypertext Transfer Protocol (HTTP) is an application protocol for distributed, collaborative,hypermedia information systems. HTTP is the foundation of data communication for the World Wide Web.

Python urllib3

The urllib3 module is a powerful, sanity-friendly HTTP client for Python. It supports thread safety, connection pooling, client-side SSL/TLS verification, file uploads with multipart encoding, helpers for retrying requests and dealing with HTTP redirects, gzip and deflate encoding, and proxy for HTTP and SOCKS.

We install the urllib3 module with pip.

Python urllib3 version

The first program prints the version of the urllib3 module.

The program prints the version or urllib3.

This is a sample output of the example.

Python urllib3 status

HTTP response status codes indicate whether a specific HTTP request has beensuccessfully completed. Responses are grouped in five classes:

• Informational responses (100–199)
• Successful responses (200–299)
• Redirects (300–399)
• Client errors (400–499)
• Server errors (500–599)

The example creates a GET request to the webcode.me. It prints the status code of the response.

We create a PoolManager to generate a request. It handles all of the details of connection pooling and thread safety.

This is the URL to which we send the request.

With the request() method, we make a GET request to the specifiedURL.

We print the status code of the response.

The 200 status code means that the request has succeeded.

Python urllib3 GET request

The HTTP GET method requests a representation of the specified resource.

Urllib3 Open

The example sends a GET request to the webcode.me webpage. It returns the HTML code of the home page.

A GET request is generated.

We get the data or the response and decode it into text.

This is the output.

Python urllib3 HEAD request

A HEAD request is a GET request without a message body.

In the example, we create a HEAD request to the webcode.me website.

The response object contains the headers dictionary, which has the various header fields, such as server and date.

From the output we can see that the web server of the website is nginx and the content type is HTML code.

Python urllib3 HTTPS request

The urllib3 provides client-side TLS/SSL verification. For this, weneed to download the certifi module. It is a carefully curatedcollection of Root Certificates for validating the trustworthiness of SSLcertificates while verifying the identity of TLS hosts. It has been extractedfrom the Requests project.

We install certifi.

To reference the installed certificate authority (CA) bundle, we use the built-in where() function.

We create a GET request to the https://httpbin.org/anything page.

We pass the root CA bundle to the PoolManager. Without this CA bundle, the request would issue the following warning: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised..

Python urllib3 query parameters

Query parameters are the part of a uniform resource locator (URL) which assignsvalues to specified parameters. This is one way of sending data to thedestination server.

The query parameters are specified after the ? character. Multiple fields are separated with the &. Special characters, such as spaces, are encoded. In the above string, the space is encoded with the %20value.

Urllib3 Download

In the example, we send a GET request with some query parameters to the https://httpbin.org/get. The link simply returns some data back to the client, including the query parameters. The site is used for testing HTTP requests.

This is the payload to be sent.

The query parameters are specified with the fields option.

The httpbin.org responded with a JSON string, which includes our payload as well.

Python urllib3 POST request

The HTTP POST method sends data to the server. It is often used when uploadinga file or when submitting a completed web form.

The example sends a POST request. The data is specified with the fieldsoption.

This is the output.

Python urllib3 send JSON

In requests, such as POST or PUT, the client tells the server what type of data is actually sent with the Content-Type header.

The example sends JSON data.

We encode the JSON data into binary format.

We specify the Content-Type header in the request.

We decode the returned data back to text and print it to the console.

Python urllib3 binary data

In the following example, we download binary data.

The example downloads a small icon.

The req.data is in a binary format, which we can directly write to the disk.

Python urllib3 stream data

Chunked transfer encoding is a streaming data transfer mechanism available sinceHTTP 1.1. In chunked transfer encoding, the data stream is divided into a series of non-overlapping chunks.

Urllib3 Retry

The chunks are sent out and received independently of one another. Each chunk ispreceded by its size in bytes.

Setting preload_content to False means that urllib3will stream the response content. The stream() method iteratesover chunks of the response content. When streaming, we should call release_conn() to release the http connection backto the connection pool so that it can be re-used.

In the example, we download a PDF file.

Urllib3 Install

With preload_content=False, we enable streaming.

We iterate over the chunks of data and save them to a file.

In the end, we release the connection.

Python urllib3 redirect

A redirect sends users and search engines to a different URL from the onethey originally requested. To follow redirects, we set the redirectoption to True.

The example follows a redirect.

This is the output.

Python urllib3 Flask example

In the following example, we send a request to a small Flask web application.Learn more about Flask web framework in Python Flask tutorial.

We need to install the flask module.

The application has one route. It sends the user agent and connection header fields of a request to the client.

In this program, we send a request to our Flask application.

Flask runs on port 5000 by default.

With the make_headers() helper method, we create a headers dictionary.

We send a GET request to the URL; we specify the headers dictionary.

We print the response to the terminal.

We run the Flask application.

From a different terminal, we launch the send_req.py program.

In this tutorial, we have worked with the Python urllib3 module.

List all Python tutorials.

Websites can be accessed using the urllib module. You can use the urllib module to interact with any website in the world, no matter if you want to get data, post data or parse data.

If you want to do web scraping or data mining, you can use urllib but it’s not the only option. Urllib will just fetch the data, but if you want to emulate a complete web browser, there’s also a module for that.

Related course:
Web Scraping in Python with BeautifulSoup & Scrapy Framework

python urllib

Download website
We can download a webpages HTML using 3 lines of code:

The variable html will contain the webpage data in html formatting. Traditionally a web-browser like Google Chrome visualizes this data.

Web browser
A web-browsers sends their name and version along with a request, this is known as the user-agent. Python can mimic this using the code below. The User-Agent string contains the name of the web browser and version number:

Parsing data
Given a web-page data, we want to extract interesting information. You could use the BeautifulSoup module to parse the returned HTML data.

You can use the BeautifulSoup module to:

There are several modules that try to achieve the same as BeautifulSoup: PyQuery and HTMLParser, you can read more about them here.

Posting data
The code below posts data to a server: