Python | Extract URL from HTML using lxml

Last Updated : 17 Jul, 2026

Extracting URLs from HTML is a common task in web scraping and web crawling. The lxml library is a Python library for parsing HTML and XML documents. It allows to locate HTML elements using XPath expressions and easily extract links from web pages.

Installation lxml

Before using lxml, install it using pip:

pip install lxml

If you are using Linux and prefer installing it through your package manager, you can also use:

sudo apt install python3-lxml

lxml Library

lxml library provides the html module for parsing HTML documents. It converts HTML into an Element Tree, making it easy to navigate the document and extract information such as links, images, headings, and tables.

html.fromstring() function parses an HTML string into an Element Tree, while the iterlinks() method iterates through all the links present in the document.

iterlinks() returns a tuple containing:

Each item returned by iterlinks() is a tuple containing four values:

  • element: HTML element containing the link.
  • attribute: where the link is found (usually href or src).
  • link: extracted URL.
  • position: position of the link within the attribute value.

The following example parses a simple HTML string and counts the number of links present in it.

Python 
from lxml import html

html_doc = '<h2>Welcome</h2><a href="/about">About</a>'
document = html.fromstring(html_doc)
links = list(document.iterlinks())
print("Number of links:", len(links))

Output

Number of links: 1

Explanation:

  • Imports the html module from lxml.
  • Parses the HTML string using fromstring().
  • Uses iterlinks() to find all links in the document.
  • Converts the result into a list and prints the total number of links.

Each value returned by iterlinks() contains information about the extracted link.

Python 
from lxml import html

html_doc = '<a href="/about">About</a>'
document = html.fromstring(html_doc)
element, attribute, link, position = list(document.iterlinks())[0]

print("Attribute:", attribute)
print("Link:", link)
print("Position:", position)

Output

Attribute: href
Link: /about
Position: 0

Explanation:

  • Extracts the first tuple returned by iterlinks().
  • Displays the attribute containing the link (href).
  • Prints the extracted URL.
  • Displays the position of the link within the attribute.

Understanding Element Tree

  • When lxml parses an HTML document, it creates an Element Tree.
  • Each HTML tag becomes a node in the tree, making it easy to navigate the document and extract information such as titles, headings, paragraphs, links, and images.
  • The lxml.html module provides HTML-specific parsing functions, while lxml.etree contains the core functionality for working with XML and HTML trees.

Parsing HTML from a Web Page

Instead of parsing an HTML string, you can fetch a web page using the requests library and then parse its HTML using lxml.

Python 
import requests
from lxml import html

response = requests.get("https://example.com")
document = html.fromstring(response.text)
title = document.xpath("//title")[0]

print("Tag:", title.tag)
print("Title:", title.text_content())
print("Parent:", title.getparent().tag)

Output

Tag: title
Title: Example Domain
Parent: head

Explanation:

  • Downloads the HTML source of a web page using requests.get().
  • Parses the HTML into an Element Tree.
  • Uses an XPath expression to locate the <title> tag.
  • Prints the tag name, page title, and its parent element.

Parsing HTML from a Local File

You can also parse an HTML file stored on your computer.

Python 
from lxml import html

document = html.parse("sample.html")
root = document.getroot()
print(root.tag)

Explanation:

  • Loads an HTML file using html.parse().
  • Retrieves the root element of the document.
  • Prints the root HTML tag.

Fetching HTML Using Requests Library

The requests library is commonly used to download web pages before parsing them with lxml.

Python 
import requests

response = requests.get("https://example.com")
print(response.status_code)
print(response.text[:300])

Output

200
<!doctype html><html lang="en"><head><title>Example Domain</title><link rel="icon" href="data:,"><meta name="viewport" content="width=device-width, initial-scale=1"><style>body{background:#eee;width:60vw;margin:15vh auto;font-family:system-ui,sans-serif}h1{font-size:1.5em}div{opacity:0.8}a:link,a:vi

Explanation:

  • Sends an HTTP GET request to the specified URL.
  • Prints the HTTP status code returned by the server.
  • Displays the first few characters of the downloaded HTML content.
  • The HTML can then be parsed using lxml for further processing.
Comment

Explore