It was actually just a simple act of crawling over source files! Amazon was looking to remove their customer support phone numbers from all web pages (to save money, they wanted people to stop calling them; back in the early 2010s)
I share the wonder that it took several days (perhaps that is the anecdote taking a life of its own, perhaps it was actually several hours. Or maybe it was was several days interspersed with on-call support, meetings, and other work running in parallel).
Though.. the file parsing libraries back in the early 2010s in Java are somewhat basic, probably using Java 5. Which does make me wonder, how long would that exercise take in Java 8 or later compared to Java 5?
Here are some facts for you to ponder:
1. Our Contact Reduction team really did have exactly this problem in 2003. This isn't a made-up example.
2. Someone on our team produced the list within an hour, and the list supported more than just the 2 formats above.
3. About 25% to 35% of all software development engineer candidates, independent of experience level, cannot solve this problem, even given the entire interview hour and lots of hints.
[0] https://sites.google.com/site/steveyegge2/five-essential-pho...
There is a very simple way to write a breadth-first webcrawler that works in stages and saves the frontier in an ordinary text file, this kind of webcrawler can be implemented in almost any programming language, particularly shell. That is, you write a script that loops over the urls in the frontier, fetches them, then writes new urls it find in the next frontier, you can use sort, uniq and such to manage the frontier.
That simple breadth first webcrawler has the big advantage that it does not get stuck in web traps (say a calendar that has a button to go the next year which could easily scan up to the year 9595, if man is still alive). Or rather, after a certain number of passes (maybe N=10-15] all the real URLs have been crawled and the remaining URLs are web traps.
In a language like Java it is tempting to write a much more complex crawl manager that has the potential to crawl faster with threads but takes more care to not crash target sites, get caught in web traps, etc. The first web crawler I wrote (1999) had a highly complex crawl control system, future web crawlers I wrote got simpler and simpler because the simple breadth first crawler works very well for small projects.