Intelligence – A good collection of great OSINT Resources | Hacker News Reader

Intelligence – A good collection of great OSINT Resources

github.com

223 pointsg14740 comments

Something I've always wondered: when you sign up for a proprietary "intelligence management" platform like Palantir Gotham, do they give you access to a bunch of OSINT datasets they scrape and manage themselves, to combine with your proprietary intelligence dataset for enhanced understanding? Sort of like how BigQuery has "BigQuery public datasets"?

If not, seems like a missed opportunity for value-add / lock-in.

And if they do provide such data, it'd be nice if one such platform would make either the scrapers, or the scraped data, open-source. It's all just cleaned+normalized forms of already-public data, after all. (Like how the Yellow Pages could always be seen as a cleaned+normalized form of phone numbers listed in public view on shop windows.)

This is on my mind recently after noticing that there's actually no public dataset of "all WHOIS data for every domain on the Internet", despite WHOIS data being something explicitly designed for public querying. Almost certainly, intelligence agencies compile such a dataset themselves in order to fill in some gaps in cybercrime network-analysis graphs. Almost certainly multiple of them do, such that the number of crawlers doing it probably hurts the WHOIS services. Almost certainly the WHOIS services would be better off partnering with one of them to act as an "official crawler" to build such an index for everyone's use.

A public (historical) WHOIS dataset is a nightmare due to GDPR et al., though, isn't it?

You couldn't opt-out of WHOIS (some TLDs just didn't allow for WHOIS privacy / proxy registrations) in the past, and it's IMO quite a hard sell that the public interest in the historic WHOIS data of some personal website should have any weight at all in this case.

That's an interesting way to put it. There is some interest, no doubt, and there is some potential abusing that information.

Since the information was committed knowingly and willingly to the public domain, though I do not know much about it to begin with, I don't see how GDPR applies at all.

The GDPR allows you to force companies to delete data that was knowingly and willingly commited, though, via article 21 and 17c GDPR [1], except if there are "overriding legitimate grounds for the processing" of the data. I doubt there are, at least for low-profile, personal websites and historical WHOIS data.

You could IMO also make an argument that the collection and scraping of WHOIS data without consent was unlawful processing as a whole, as the data was not provided to the domain NIC with explicit consent for third-parties to collect and save them forever, which would even make any overriding legitimate grounds for processing moot.

17a may also apply: The reason for WHOIS is to find out who owns a domain now, right? There is no version history. Historical WHOIS data does not serve this purpose anymore.

It's quite the can of worms.

[1] https://gdpr-text.com/de/read/article-17/

Some do, some don't. The big thing that Palantir provides you is a structured analysis platform more than anything. The idea is that your intel analysts are doing the same thing and structuring their artifacts in a consistent way. It isn't so much about pulling in feeds (though obviously most pull these in), but as structured process.

On something like whois, there are commercial vendors that provide history/change reporting, but its a pain in the butt to parse/normalize whois records and that is before you even get to the fact that many providers have shut down their whois feeds "for gdpr compliance".

> when you sign up for a proprietary "intelligence management" platform like Palantir Gotham, do they give you access to a bunch of OSINT datasets they scrape and manage themselves, to combine with your proprietary intelligence dataset for enhanced understanding?

I don’t know about OSINT, but in the consumer data landscape (Acxiom, Datalogix, etc) this is the standard MO

Since other people are asking about related tools...

Do you know about a tool for managing personas (vulgo sockpuppets)? E.g. a little database where you store facts about fake identities: Name, age, location, hobbies, favorite topics, and so on. Maybe if you click on an identity, it will connect you with a preconfigured proxy or VPN and allow you to post as them on websites.

I don't really have a use case for myself. I'm just curious because I read a couple of years ago that the US Army was developing something like that for propaganda, and wonder if the OSINT or sousveilance people have come up with a version. A less nefarious use of such technology would be to keep your (real) online personalities separate.

A password manager?

Firefox Multi-Account Containers with Mozilla VPN would be a start.

> Maybe if you click on an identity, it will connect you with a preconfigured proxy or VPN and allow you to post as them on websites.

You can still be fingerprinted across "personas" when your useragent/screen resolution/IP address/proxy provider/etc. are all identical, since you're presumably doing all of this from the same device. This does not mean spin up a bunch of EC2 instances.

Buy a bunch of different burner phones/old laptops from Goodwill or whatever-- use discrete hardware. Get SIM cards from different providers. Pay with Bitcoin or cash where possible. Etc.

These are the sorts of things scammer sweatshops do to avoid accountability. They'll have literal walls full of phones wired up, each preconfigured for a particular identity. Even teenaged fraudsters like Eleanor Williams think to do this sort of stuff to maintain opsec.

The list is cool and full of interesting things. But as someone who has done OSINT a lot for journalism and legal investigations and built tools for the space I can say that people rely most on custom internal tools or the simplest/ most popular tooling among the community!

I feel like most of OSINT is more manual than these lists lead to believe. Or if not, most of it relies on already-built massive archives or tools that scrape those archives.

absolutely. it's just useful for a quick overview. real research happens when you keep digging where others haven't.

Agreed. This list is obviously tilted toward computer security; OSINT also targets nation states and political actors, and more often than not that's through non-technical means (ie looking for information carelessly left public rather than exploiting vulnerabilities to gain access to private information).

Would you mind reaching out to me? I'm trying to learn more about how to do OSINT (with that particular slant!).

at the start of Russian invasion I helped geolocate one video, claiming that Russians entered Kherson. I found it after about one hour and posted on twitter tagging everyone I know. I got no feedback so got discouraged. But few months later, I noticed that France24 used my tweet to prove Russians entered Kherson and a lot of newspapers followed their lead. Pretty cool I was able to help in a minimally viable way to understand what was happening

that's awesome man. oh and feel free to be encouraged but don't ever get discouraged based on other people's behavior or actions. experts like you are definitely needed in today's world. people will eventually appreciate your research.

Bravo. That is definitely probably the only true 'hacker badge' that I would really want. I would wear that little patch on my jacket at the little meeting for sure.

This is a great story. Don't get discouraged!

OSINT is the one of the last ways we humans have, to get some Data that is at least a bit honest. More importaint is that we know how to use and gather it. THX for your link i will take a look at it.

For better or worse, I'm surprised this hasn't happened yet:

Track organizational structures and responsibility over time (especially public ones like government and public corporation executives).

Then enable searches of events / transgressions and claimed fallguys so more responsibility can be traced/blame for these important figures who usually get off scot free.

It would be useful to know "hey who likely was in charge of the divisions that suppressed early global warming research in oil companies" with a relatively simple search.

Earlier discussion about a similar list:

https://news.ycombinator.com/item?id=32951406

Relatedly: is there software / database designs recommended for managing osint facts?

you mean like maltego or osintframework?

I use rolling snapshots. It helps protect against members dropping suddenly, as a lot of OSINT sources aren't intended to be and will only contain a sample of your target population. Opinions, priorities, or biases may change, and thus so will the data.

There’s https://atlos.org for visual investigations

I mean... Including Kaspersky in any shape or form is a bit of a moot point but...

BGP Ranking site https://bgpranking.circl.lu/ serve a 503 error Service unavailable.

I do not know any more details.

Don't forget the Noncredibledefense subreddit.

i did forget. thanks!

Hold up. It was a joke.

good one

Reply on news.ycombinator.com