webpage-dl

Fetch a webpage’s contents via CLI after it has finished loading.

Scraping HTML from webpages can’t always be done via curl as it is often useful to let the page’s JavaScript play out first. webpage-dl launches a WKWebView and only reads the source of the page once loading has completed.

webpage-dl even allows you to provide your own JavaScript expression, so scraping specific HTML nodes has never been easier.

Installation

Clone this repo
Run make install (requires Xcode 13+)

Usage

USAGE: webpage-dl [--width <width>] [--height <height>] [--timeout <timeout>] [--delay <delay>] [--expression <expression>] <url>

ARGUMENTS:
  <url>                   The URL of the webpage to download.

OPTIONS:
  -w, --width <width>     The width of the simulated web page. (default: 500)
  -h, --height <height>   The height of the simulated web page. (default: 500)
  --timeout <timeout>     The maximum number of seconds the program should run
                          for before exiting (in the event of slow loading).
                          (default: 20.0)
  -d, --delay <delay>     An additional time to wait before executing
                          'expression'.
  --expression <expression>
                          The JavaScript expression to run. The output from
                          this expression with be printed to stdout before the
                          program exits. (default:
                          document.documentElement.outerHTML.toString())
  -h, --help              Show help information.