Force Archive Websites To Pick Up Webpages With This Handy Tool

Force Archive Websites To Pick Up Webpages With This Handy Tool

Website archive services such as the Internet Archive: Wayback Machine are incredibly useful for when you need to see old versions of sites – either for nostalgia or because you’re looking for a specific bit of information that has since been overwritten or deleted (such as a story you wrote for a former employer, for instance).

Photo: Pixabay

However, these services aren’t perfect. There are times when an archive site might not make a snapshot of a site – typically, exactly when you need that snapshot most. Or perhaps someone has configured their site’s robots.txt file to block archive services from performing their automatic crawls. No fun.

Thanks to a new tool from Motherboard, you can now attempt to archive the current version of a site across three different archive services at once: The Wayback Machine, and (if you’ve set up a free account with them).

Installing Motherboard‘s archiving utility requires a little bit of legwork, but it isn’t too tricky. You’ll first need to install Python’s requests, json and archiveis modules, which are all required for Motherboard‘s mass_archive tool to work. (Alas, this isn’t just some simple executable or utility you can run.) The best way to install requests and json is to install pip first, and then use that to download the modules. You’ll find archiveis here, and you can install it using pip as well.

You’ll also need to grab the script from the aforementioned GitHub project. Once you’re ready, pull up a terminal in macOS or Linux and type this in (obviously, replacing with the website you’re looking to archive):


If you’re using Python via an elevated command prompt in Windows, you can omit the initial “python” from that code.

The Cheapest NBN 50 Plans

Here are the cheapest plans available for Australia’s most popular NBN speed tier.

At Lifehacker, we independently select and write about stuff we love and think you'll like too. We have affiliate and advertising partnerships, which means we may collect a share of sales or other compensation from the links on this page. BTW – prices are accurate and items in stock at the time of posting.