What is BeautifulSoup?

BeautifulSoup is a Python library for parsing HTML and XML documents. It creates a parse tree that makes it easy to extract data from web pages.

Installation

Install BeautifulSoup:

pip install beautifulsoup4 lxml

Basic Usage

Parse HTML:

from bs4 import BeautifulSoup

html = "<html><body><p>Hello World</p></body></html>"
soup = BeautifulSoup(html, 'html.parser')

print(soup.p.text)  # "Hello World"

Finding Elements

Find elements by tag, class, or ID:

# Find by tag
soup.find('div')
soup.find_all('p')

# Find by class
soup.find(class_='content')
soup.find_all(class_='item')

# Find by ID
soup.find(id='header')

# Find by attribute
soup.find('a', href='https://example.com')

Extracting Data

Get text and attributes:

# Get text
element = soup.find('p')
text = element.text
text = element.get_text()

# Get attributes
link = soup.find('a')
url = link['href']
link_text = link.text

CSS Selectors

Use CSS selectors:

# Select by CSS
soup.select('div.content')
soup.select('#header')
soup.select('.item')
soup.select('div > p')  # Direct child

Best Practices

  • Use appropriate parser: html.parser, lxml, or html5lib
  • Handle missing elements: Check if element exists
  • Use selectors: More powerful than find methods
  • Clean data: Strip whitespace and normalize text

Next Steps

Combine with Requests to scrape web pages.