The internet, as a “place” is constantly growing. We build more and more webpages every day — so many, in fact, that it can feel as if certain corners of it are lost to time.
As it turns out, they may only be lost to Google. Earlier this year, web developer-bloggers Tim Bray and Marco Fioretti noted that Google seems to have stopped indexing the entirety of the internet for Google Search.
As a result, certain old websites — those more than 10 years old — did not show up through Google search. Both writers lamented that limiting Google’s effective memory to the last decade, while logical when faced with the daunting task of playing information concierge to our every whimsical question, forces us to reckon with the fact that, when you use Google for historical searches, there are probably more answers out there.
As a BoingBoing post based on Bray’s points out, DuckDuckGo and Bing both still seem to offer more complete records of the internet, specifically showing web pages that Google stopped indexing for search.
If you’re looking for a specific website from before 2009 and can’t find it, either one is a solid first step. If that doesn’t work, it’s always possible someone else who needed the same page you were looking for saved it as an archive on the Wayback Machine.
But what about broad questions? Questions where you don’t already know the answer? Historical research from the early web? There are other, more specialised options for that. A Hacker News forum post suggests a couple of search engines.
Million Short, which allows you to run a search and automatically skip the most popular answers to probe deeper into the web. Wiby.me is a “search engine for classic websites,” made to help people find hobbyist pages and other archaic features of the internet.
The Hacker news thread also brings up Pinboard, a minimalist bookmarking service similar to Pocket, which has a key feature for archivists: If you sign up for its premium service — $15 per year — Pinboard will make a web archive of every page you save.
If you’re looking at older, unindexed material, such a tool can make it easier to go back to specific parts of the older internet that you may want or need to recall again.
Comments