Python3 Requests

Posted : admin On 1/25/2022
  1. Python3 Requests
  2. Python3 Requests Headers

Some common request methods are GET, POST, and PUT. GET requests are normally for reading data only without making a change to something, while POST and PUT requests generally are for modifying data on the server. So for example, the Stripe API allows you to use POST requests to create a new charge so a user can purchase something from your app. Sends a DELETE request to the specified url. Get ( url, params, args) Sends a GET request to the specified url. Head ( url, args) Sends a HEAD request to the specified url. Patch ( url, data, args) Sends a PATCH request to the specified url. Post ( url, data, json, args) Sends a POST request to the specified url. Python requests authentication with an X.509 certificate and private key can be performed by specifying the path to the cert and key in your request. An example using python requests client certificate: requests.get('cert=('/path/client.cert', '/path/client.key')) The certificate and key may also be combined into the same file. If you receive an SSL error on your python requests call, you have likely given an invalid path, key, or certificate in your request.

  • Selected Reading

The http or Hyper Text Transfer Protocol works on client server model. Usually the web browser is the client and the computer hosting the website is the server. IN python we use the requests module for creating the http requests. It is a very powerful module which can handle many aspects of http communication beyond the simple request and response data. It can handle authentication, compression/decompression, chunked requests etc.

An HTTP client sends an HTTP request to a server in the form of a request message which includes following format:

  • A Request-line
  • Zero or more header (General Request Entity) fields followed by CRLF
  • An empty line (i.e., a line with nothing preceding the CRLF) indicating the end of the header fields
  • Optionally a message-body

The following sections explain each of the entities used in an HTTP request message.

Request-Line

The Request-Line begins with a method token, followed by the Request-URI and the protocol version, and ending with CRLF. The elements are separated by space SP characters.

Let's discuss each of the parts mentioned in the Request-Line.

Python3 requests post json

Request Method

The request method indicates the method to be performed on the resource identified by the given Request-URI. The method is case-sensitive and should always be mentioned in uppercase. The following table lists all the supported methods in HTTP/1.1.

S.N.Method and Description
1GET

The GET method is used to retrieve information from the given server using a given URI. Requests using GET should only retrieve data and should have no other effect on the data.

2HEAD

Same as GET, but it transfers the status line and the header section only.

3POST

A POST request is used to send data to the server, for example, customer information, file upload, etc. using HTML forms.

4PUT

Replaces all the current representations of the target resource with the uploaded content.

5DELETE

Removes all the current representations of the target resource given by URI.

6CONNECT

Establishes a tunnel to the server identified by a given URI.

7OPTIONS

Describe the communication options for the target resource.

8TRACE

Performs a message loop back test along with the path to the target resource.

Request-URI

The Request-URI is a Uniform Resource Identifier and identifies the resource upon which to apply the request. Following are the most commonly used forms to specify an URI:

S.N.Method and Description
1The asterisk * is used when an HTTP request does not apply to a particular resource, but to the server itself, and is only allowed when the method used does not necessarily apply to a resource. For example:

OPTIONS * HTTP/1.1

2The absoluteURI is used when an HTTP request is being made to a proxy. The proxy is requested to forward the request or service from a valid cache, and return the response. For example:

GET http://www.w3.org/pub/WWW/TheProject.html HTTP/1.1

3The most common form of Request-URI is that used to identify a resource on an origin server or gateway. For example, a client wishing to retrieve a resource directly from the origin server would create a TCP connection to port 80 of the host 'www.w3.org' and send the following lines:

GET /pub/WWW/TheProject.html HTTP/1.1

Host: www.w3.org

Note that the absolute path cannot be empty; if none is present in the original URI, it MUST be given as '/' (the server root).

Python

Using Python requests

We will use the module requests for learning about http request.

In the below example we see a case of simple GET request annd print out the result of the response. We choose to print only the first 300 characters.

When we run the above program, we get the following output −

Eager to get started? This page gives a good introduction in how to get startedwith Requests.

First, make sure that:

  • Requests is installed
  • Requests is up-to-date

Let’s get started with some simple examples.

Make a Request¶

Making a request with Requests is very simple.

Begin by importing the Requests module:

Now, let’s try to get a webpage. For this example, let’s get GitHub’s publictimeline:

Now, we have a Response object called r. We canget all the information we need from this object.

Requests’ simple API means that all forms of HTTP request are as obvious. Forexample, this is how you make an HTTP POST request:

Nice, right? What about the other HTTP request types: PUT, DELETE, HEAD andOPTIONS? These are all just as simple:

That’s all well and good, but it’s also only the start of what Requests cando.

Passing Parameters In URLs¶

You often want to send some sort of data in the URL’s query string. Ifyou were constructing the URL by hand, this data would be given as key/valuepairs in the URL after a question mark, e.g. httpbin.org/get?key=val.Requests allows you to provide these arguments as a dictionary of strings,using the params keyword argument. As an example, if you wanted to passkey1=value1 and key2=value2 to httpbin.org/get, you would use thefollowing code:

You can see that the URL has been correctly encoded by printing the URL:

Note that any dictionary key whose value is None will not be added to theURL’s query string.

You can also pass a list of items as a value:

Response Content¶

We can read the content of the server’s response. Consider the GitHub timelineagain:

Requests will automatically decode content from the server. Most unicodecharsets are seamlessly decoded.

When you make a request, Requests makes educated guesses about the encoding ofthe response based on the HTTP headers. The text encoding guessed by Requestsis used when you access r.text. You can find out what encoding Requests isusing, and change it, using the r.encoding property:

Requests

If you change the encoding, Requests will use the new value of r.encodingwhenever you call r.text. You might want to do this in any situation whereyou can apply special logic to work out what the encoding of the content willbe. For example, HTML and XML have the ability to specify their encoding intheir body. In situations like this, you should use r.content to find theencoding, and then set r.encoding. This will let you use r.text withthe correct encoding.

Requests will also use custom encodings in the event that you need them. Ifyou have created your own encoding and registered it with the codecsmodule, you can simply use the codec name as the value of r.encoding andRequests will handle the decoding for you.

Binary Response Content¶

You can also access the response body as bytes, for non-text requests:

The gzip and deflate transfer-encodings are automatically decoded for you.

For example, to create an image from binary data returned by a request, you canuse the following code:

JSON Response Content¶

There’s also a builtin JSON decoder, in case you’re dealing with JSON data:

In case the JSON decoding fails, r.json() raises an exception. For example, ifthe response gets a 204 (No Content), or if the response contains invalid JSON,attempting r.json() raises ValueError:NoJSONobjectcouldbedecoded.

It should be noted that the success of the call to r.json() does notindicate the success of the response. Some servers may return a JSON object in afailed response (e.g. error details with HTTP 500). Such JSON will be decodedand returned. To check that a request is successful, user.raise_for_status() or check r.status_code is what you expect.

Raw Response Content¶

In the rare case that you’d like to get the raw socket response from theserver, you can access r.raw. If you want to do this, make sure you setstream=True in your initial request. Once you do, you can do this:

Python3 Requests

In general, however, you should use a pattern like this to save what is beingstreamed to a file:

Using Response.iter_content will handle a lot of what you would otherwisehave to handle when using Response.raw directly. When streaming adownload, the above is the preferred and recommended way to retrieve thecontent. Note that chunk_size can be freely adjusted to a number thatmay better fit your use cases.

Note

An important note about using Response.iter_content versus Response.raw.Response.iter_content will automatically decode the gzip and deflatetransfer-encodings. Response.raw is a raw stream of bytes – it does nottransform the response content. If you really need access to the bytes as theywere returned, use Response.raw.

Custom Headers¶

If you’d like to add HTTP headers to a request, simply pass in a dict to theheaders parameter.

For example, we didn’t specify our user-agent in the previous example:

Note: Custom headers are given less precedence than more specific sources of information. For instance:

  • Authorization headers set with headers= will be overridden if credentialsare specified in .netrc, which in turn will be overridden by the auth=parameter.
  • Authorization headers will be removed if you get redirected off-host.
  • Proxy-Authorization headers will be overridden by proxy credentials provided in the URL.
  • Content-Length headers will be overridden when we can determine the length of the content.

Furthermore, Requests does not change its behavior at all based on which custom headers are specified. The headers are simply passed on into the final request.

Note: All header values must be a string, bytestring, or unicode. While permitted, it’s advised to avoid passing unicode header values.

More complicated POST requests¶

Typically, you want to send some form-encoded data — much like an HTML form.To do this, simply pass a dictionary to the data argument. Yourdictionary of data will automatically be form-encoded when the request is made:

The data argument can also have multiple values for each key. This can bedone by making data either a list of tuples or a dictionary with listsas values. This is particularly useful when the form has multiple elements thatuse the same key:

There are times that you may want to send data that is not form-encoded. Ifyou pass in a string instead of a dict, that data will be posted directly.

For example, the GitHub API v3 accepts JSON-Encoded POST/PATCH data:

Instead of encoding the dict yourself, you can also pass it directly usingthe json parameter (added in version 2.4.2) and it will be encoded automatically:

Note, the json parameter is ignored if either data or files is passed.

Using the json parameter in the request will change the Content-Type in the header to application/json.

POST a Multipart-Encoded File¶

Requests makes it simple to upload Multipart-encoded files:

You can set the filename, content_type and headers explicitly:

If you want, you can send strings to be received as files:

In the event you are posting a very large file as a multipart/form-datarequest, you may want to stream the request. By default, requests does notsupport this, but there is a separate package which does -requests-toolbelt. You should read the toolbelt’s documentation for more details about how to use it.

For sending multiple files in one request refer to the advancedsection.

Warning

It is strongly recommended that you open files in binarymode. This is because Requests may attempt to providethe Content-Length header for you, and if it does this valuewill be set to the number of bytes in the file. Errors may occurif you open the file in text mode.

Response Status Codes¶

We can check the response status code:

Requests also comes with a built-in status code lookup object for easyreference:

If we made a bad request (a 4XX client error or 5XX server error response), wecan raise it withResponse.raise_for_status():

But, since our status_code for r was 200, when we callraise_for_status() we get:

All is well.

Note

raise_for_status returns the response object for a successful response. This eases chaining in trivial cases, where we want bad codes to raise an exception, but use the response otherwise:

Response Headers¶

We can view the server’s response headers using a Python dictionary:

The dictionary is special, though: it’s made just for HTTP headers. According toRFC 7230, HTTP Header namesare case-insensitive.

So, we can access the headers using any capitalization we want:

It is also special in that the server could have sent the same header multipletimes with different values, but requests combines them so they can berepresented in the dictionary within a single mapping, as perRFC 7230:

A recipient MAY combine multiple header fields with the same field nameinto one “field-name: field-value” pair, without changing the semanticsof the message, by appending each subsequent field value to the combinedfield value in order, separated by a comma.

If you do need to access each individual value sent with the same header, thenyou can use the getlist method to get a sequence of all the values returnedfor a particular header.

Cookies¶

If a response contains some Cookies, you can quickly access them:

To send your own cookies to the server, you can use the cookiesparameter:

Cookies are returned in a RequestsCookieJar,which acts like a dict but also offers a more complete interface,suitable for use over multiple domains or paths. Cookie jars canalso be passed in to requests:

Redirection and History¶

By default Requests will perform location redirection for all verbs exceptHEAD.

We can use the history property of the Response object to track redirection.

The Response.history list contains theResponse objects that were created in order tocomplete the request. The list is sorted from the oldest to the most recentresponse.

For example, GitHub redirects all HTTP requests to HTTPS:

If you’re using GET, OPTIONS, POST, PUT, PATCH or DELETE, you can disableredirection handling with the allow_redirects parameter:

If you’re using HEAD, you can enable redirection as well:

Timeouts¶

You can tell Requests to stop waiting for a response after a given number ofseconds with the timeout parameter. Nearly all production code should usethis parameter in nearly all requests. Failure to do so can cause your programto hang indefinitely:

Note

timeout is not a time limit on the entire response download;rather, an exception is raised if the server has not issued aresponse for timeout seconds (more precisely, if no bytes have beenreceived on the underlying socket for timeout seconds). If no timeout is specified explicitly, requests donot time out.

Errors and Exceptions¶

In the event of a network problem (e.g. DNS failure, refused connection, etc),Requests will raise a ConnectionError exception.

Response.raise_for_status() willraise an HTTPError if the HTTP requestreturned an unsuccessful status code.

If a request times out, a Timeout exception israised.

If a request exceeds the configured number of maximum redirections, aTooManyRedirects exception is raised.

All exceptions that Requests explicitly raises inherit fromrequests.exceptions.RequestException.

Ready for more? Check out the advanced section.

Python3 Requests Headers

If you’re on the job market, consider taking this programming quiz. A substantial donation will be made to this project, if you find a job through this platform.