Show HN: Scanning the Web for Security.txt Files
github.com
github.com
'All scan results without successful exploitation will be summarily ignored.'
https://beta.shodan.io/host/172.217.31.43#securitytxt
If you have a Shodan account you can also search the contents of the security.txt files using the "http.securitytxt" search filter. For example:
https://beta.shodan.io/search?query=http.securitytxt%3Aconta...
The blog post has more details and a short overview of the results. It's linked at the bottom of the repo. I was hoping to get feedback on the code.
From the blog post-
> Of the 666,771 most popular websites on the Alexa list, I found 2,884 security.txt files that were content-type “text/plain” and returned a HTTP 200 status code. Not all of these were valid security.txt files, but most were.
Seems I was right?
It's been at least 35 years since I first saw a non-three-character file name extension (Amiga 1000), there are probably older examples. Computers are supposed to work for people, not the other way around.
There are only a few important extensions. just remember them.
> Computers are supposed to work for people, not the other way around.
Extensions are not for average computer users. They have icons and filenames. Windows also hides extensions by default.
.xpi is a rather poor example: no one talks about XPInstall (and the majority of its surface area is now even obsolete), so .xpi is to most people completely meaningless unless they have encountered it before and know what it is. .firefox-extension would be a vast improvement over .xpi, because it says what it actually is.
That's an excellent suggestion. WTF does ".xpi" even mean to someone who just wants to install a browser extension?
As far as I recall, extensions became a thing with DOS. They actually had meaning to the OS, e.g. naming a file .exe would make it executable, as there was no other concept of file ownership or permissions.
In Unix/Linux systems, filename extensions have always been for the user. Before GUIs and icons, they were a convention that let the user know something about the file contents. As far as the operating system is concerned, "." is just another character in the name and extensions are meaningless. Note that most binary executables don't have an extension at all, and other files have more than one (e.g. .tar.gz)
CP/M had extensions in 1974, five years before DOS. There may be older examples, but that's the first operating system I ever used.
There was also ITS, which was one of the famous operating systems on the PDP-10. On ITS, filenames consisted of two parts, rach being 6 characters separated with a space. So you could have a file named "abc def". The second half was often used in the same way we use extensions today, but executable files were named "ts name", where name was the name of the command.
I'm not sure what format filenames had on TENEX and TOPS-10.
Extension are only made for backward/universal compatibility file compatibility on different platform. However I do agree with the parent post, computer ought to work for us, not the other way around. Then again changing convention is harder than just type 3 letter extension.
But the common extensions that we all know by heart, why change, what's the gain?
.text is not a common extension. Some things know that it’s text/plain (my Arch Linux /etc/mime.types and /etc/nginx/mime.types both do), but I expect some common server software won’t handle it properly out of the box (haven’t checked beyond nginx’s mime.types), and common OSes won’t have a handler for .text files set up (Windows, for example, comes with .text set to PerceivedType text like .txt has, so that it’ll suggest the right sort of apps to open it, but it’s still not hooked up to any app by default, unlike .txt which is “Text Document”).
Dump .txt for .text.
Dump .jsn for .json.
Keep .html, .jpeg, etc because they are abbreviations for standards.
Sucks for Windows that it can't handle ".text" like other operating systems have since the 1980's.
Dumping three-letter extensions will also help avoid all the extension namespace collisions that happen all the time.
If my computer expects me to type an extra letter every time I name a file it isn't working for me. The short extensions make for more efficient typing.
And, I've never seen ".jsn". I've seen ".json" hundreds of thousands of times, but never ".jsn".
I don't know why you're getting hung up on TXT in particular, that extension has been around for so long that virtually everyone who has even a passing knowledge of computers knows what it stands for.
Also for CLI purposes, I'd sure as hell rather type "ls .bmp" than "ls .bitmap".