13.5. Retrieving web pages with
While we can manually send and receive data over HTTP using the socket
library, there is a much simpler way to perform this common task in
Python by using the
urllib, you can treat a web page much like a file.
You simply indicate which web page you would like to retrieve and
urllib handles all of the HTTP protocol and header details.
The equivalent code to read the romeo.txt file from the web
urllib is as follows:
Once the web page has been opened with
can treat it like a file and read through it using a
When the program runs, we only see the output of the contents of the
file. The headers are still sent, but the
consumes the headers and only returns the data to us.
As an example, we can write a program to retrieve the data for
romeo.txt and compute the frequency of each word in the
file as follows:
Again, once we have opened the web page, we can read it like a local file.