Learn Web Scraping!

What is web scraping?

The internet is full of information people would like to collect. Copying and pasting it would take forever! In this ebook, we’ll explore an easier way.

Web scraping is the process of getting information from web pages with code.

How this ebook works

During this lesson, you will learn code plans used by real professionals.

These plans represent the most common patterns in web scraping with the BeautifulSoup library. The plans in this ebook were created after an anaylsis of 100 web scraping files from Github, as well as two interviews with people who use web scraping in their jobs.

Two scraping examples

• Scrape all the Cottage Inn Pizza locations
• Get news links from faculty webpages

Plans 2-3: Get the webpage

• Plan 1: Get a soup from a URL
• Plan 2: Get a soup from multiple URLs

Plans 4-5: Extract info from the webpage

• Plan 4: Get info from a single tag
• Plan 3: Get info from all tags of a certain type

Plans 9-10: Do something with the info

• Plan 5: Print info
• Plan 10: Store info in a json file

Putting it together

• Code writing activity part 1
• Code writing activity part 2
• Code writing activity part 3
• Code debugging activity
• Code explaining activity

Download the plans

• You can download all the plans here

You have attempted of activities on this page