Plan 3: Get a soup from multiple URLs

Subsection 14.4.1 Plan 3: Example

Sometimes we want to get information from multiple web pages that have the same layout. For example, all of the UMSI faculty pages have the same general design.

We are interested in getting information about mutliple UMSI professors: Dr. Barb Ericson, Dr. Steve Oney, and Dr. Paul Resnick.

Their webpages are:

https://web.archive.org/web/20230128074139/https://www.si.umich.edu/people/

In this code, we get a soup from multiple UMSI faculty pages.

# Load libraries for web scraping
from bs4 import BeautifulSoup
import requests
# Get a soup from multiple URLs
base_url = 'https://web.archive.org/web/20230128074139/https://www.si.umich.edu/people/'
endings = ['barbara-ericson', 'steve-oney', 'paul-resnick']
for ending in endings:
    url = base_url + ending
    r = requests.get(url)
    soup = BeautifulSoup(r.content, 'html.parser')

Subsection 14.4.2 Plan 3: When to use this plan

Use this plan when you want to scrape the same thing from multiple webpages.

Subsection 14.4.3 Plan3: How to use this plan

Look at the webpages you want to scrape and determine which parts they have in common, and which parts are different. The parts that they have in common are the base_url. The parts that are different are the endings.

Subsection 14.4.4 Plan 3: Exercises

If you want to also get the link to the most recent news item from Dr Robin Brewer’s page, how would you change the code below? Her web page is https://web.archive.org/web/20230110174202/https://www.si.umich.edu/people/robin-brewer.

Change the code and run it to see if you’re right!

Activity 14.4.1.

Note 14.4.3.

Click here to go back to the Faculty Pages example