link_scraper.py in Python

Scrape a page's links, splitting internal from external.

"""Scrape a page for links and report internal vs external counts."""
import sys
from urllib.parse import urljoin, urlparse

import requests
from bs4 import BeautifulSoup


def scrape_links(url):
    response = requests.get(url, timeout=10,
                            headers={"User-Agent": "typestar-bot/1.0"})
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "html.parser")
    host = urlparse(url).netloc
    internal, external = [], []
    for anchor in soup.select("a[href]"):
        full = urljoin(url, anchor["href"])
        target = internal if urlparse(full).netloc == host else external
        target.append(full)
    return internal, external


def main():
    url = sys.argv[1] if len(sys.argv) > 1 else "https://example.com"
    try:
        internal, external = scrape_links(url)
    except requests.RequestException as exc:
        print(f"failed to fetch {url}: {exc}")
        raise SystemExit(1)
    print(f"{url}")
    print(f"  internal links: {len(internal)}")
    print(f"  external links: {len(external)}")
    for link in external[:5]:
        print(f"    -> {link}")


if __name__ == "__main__":
    main()

How it works

1. requests fetches the page with a user agent.
2. BeautifulSoup selects every anchor with an href.
3. urljoin and urlparse classify each link.

Keywords and builtins used here

• as
• def
• else
• except
• for
• if
• len
• main
• print
• raise
• return
• scrape_links
• try

The run, in numbers

Lines

38

Characters to type

1014

Tokens

256

Three-star pace

115 tpm

At the three-star pace of 115 tokens a minute, this run takes about 134 seconds.

Type this snippet

Step 1 of 1 in Encore, step 9 of 9 in Web scraping.

← Previous