Web scraping
9 steps in 3 sets of Python.
Nine steps, two libraries, one honest warning. requests for getting the page, with sessions, headers, timeouts and the error handling people skip. BeautifulSoup for pulling things out of it: selectors, walking the tree, and the attribute access that gets you a link's href.
The Encore scrapes something end to end. Short tour, but the two libraries in it turn up constantly in scripts that were only ever supposed to run once.
requests
- requests GETThe friendly way to fetch data over HTTP.
- requests POSTSending JSON and headers in a POST request.
- requests sessionsReusing connections, headers, and cookies across calls.
- Handling request errorsCatching the specific ways an HTTP call can fail.
- Streaming downloadsDownloading a large file without loading it all in memory.
BeautifulSoup
- Parsing HTMLTurning markup into a searchable tree with BeautifulSoup.
- CSS selectorsFinding elements with familiar CSS selector syntax.
- Traversing the treeWalking nested tags to extract a table's rows.
Encore
- link_scraper.pyScrape a page's links, splitting internal from external.