#65MediumFiles & Modules· Standard Library

Web Scraper: Extract Article Links

The heart of a scraper is the parsing step: turning a page's HTML into data. In a real project you would download the page with a library such as `requests` and parse it with a library such as BeautifulSoup. Here the page arrives as a string, and you parse it with the standard library's `html.parser`, so nothing needs to be installed. Write `extract_articles(html, base_url)` that returns `[title, url]` pairs for every link (`<a href=...>`) that sits inside an `<h2>` heading, in page order: - `title` is the link's text with surrounding whitespace removed and inner runs of whitespace collapsed to one space. - `url` is the `href`. If it starts with `/`, prefix it with `base_url` (without a trailing slash). Links whose `href` starts with `#`, or with no `href` at all, are skipped. - Skip links whose title is empty, and skip a URL that was already returned.

Input format

`html`: an HTML document as a string. `base_url`: the site's address, such as `"https://blog.example.com"`.

Output format

A list of `[title, url]` pairs.

Examples

Input: html = "<h2><a href=\"/posts/intro\">Getting started</a></h2><p><a href=\"/about\">About</a></p><h2><a href=\"https://other.example.org/x\">Elsewhere</a></h2>", base_url = "https://blog.example.com"

Output: [["Getting started","https://blog.example.com/posts/intro"],["Elsewhere","https://other.example.org/x"]]

The `About` link is not inside an `<h2>`. The relative link gets the base URL, and the double space in the title is collapsed.

Input: html = "<h2><a href=\"#top\">Top</a> <a>Draft</a></h2>", base_url = "https://blog.example.com/"

Output: []

A fragment link and a link without `href` are both skipped.

Constraints

len(html) <= 10^5
Headings are not nested inside each other

Study first

Files and Modules in Python

#python
#web-scraping
#html
#standard-library