WGET

June 4, 2020

WGET, an indispensable tool for working with the web. Below are a few examples extracted from my CLI cheat-sheet, with explanation on syntax.

WGET & CURL: equivalent examples

wget -O index.html www.exampledomain.com

# The -O (upper case) is optional, and if omitted it usually saves as the index.html for the website.

curl -L www.exampledomain.com > index.html

# The -L follows all redirects before returning data.
# Curl normally spits out to the CLI, but the '>' redirects the named file.  Beware '>' overwrites, sometimes useful is '>>' to append.

Debugging

Sometimes it's really useful to be able to grab a website in its entirety.

wget -pHk www.exampledomain.com

-p  fetches all accompanying assets (images, css, js) to view the site
-H  enables recursive run, to fetch assets from other hosts
-k  after downloading, this will change all asset links to local/relative

Extra

du -sh $pwd

After a wget -pHk (in an empty directory), use du -sh $pwd to see the size of your website. I've found this to be a good statistic to keep track of for UX/mobile purposes. Though there's a lot to consider whether it's CSS, JS, or other and a large website doesn't necessarily mean it's slow.