AOL Moloch: open-source, large scale, packet-capturing, indexing database system
molo.ch
molo.ch
Now that we're to that point, please stop screwing it up and coming up wih your own locations for application binaries, data, etc.
To be clear, the "/data" directory -- under which Moloch's pre-built packages apparently install to, according to the README [1] -- is not part of the FHS.
---
[0]: https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
[1]: https://raw.githubusercontent.com/aol/moloch/master/release/...
So really it doesn't matter where a random dev decides to install their software to by default, since either A) it's proprietary and it's non-standard anyway, or B) it's up to us to package it the way we want it.
Further to this, the moment you want to install somewhere else or have a dependency elsewhere (e.g. /opt/anything) builds break because of hard coded paths and poor assumptions. To top it all, I've no idea why a package manager like Homebrew then contributes to all of this by acting like no of this matters because a Mac is just like Linux, apparently.
If they don't like Apple's way then why not XDG[2]? Reinventing the wheel badly is more than just hubris, it breaks things and it's annoying.
[1] https://developer.apple.com/library/archive/documentation/Ma... [2] https://specifications.freedesktop.org/basedir-spec/latest/
"Where ARE my programs / settings that I installed, I always have to google it!"
is interrelated with someone else asking
"Where should I PUT this program / setting / data? I guess I'll chose one of those three or so locations where it may fit, or several"
For example: Why isn't there a data directory to be found? Where does the data go? Why not replace nondescript and unhelpful names like "opt" and "usr" (which is btw. not where the user data is!)?
First when / ran out of space, new programs were put on the other disk /usr in /usr/bin so everyone could assume programs were in /bin except ones that were newly installed so they would be in /usr/bin
When /usr ran out of space, a new disk was added as /home and things that were easier to move (user directories) were moved first leaving things that everyone's script was depending on (#!/usr/bin...) where they were.
This was done out of necessity, not out of good taste.
https://www.bell-labs.com/usr/dmr/www/notes.html
See also, from:
https://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/usr...
In the original Unix implementations, /usr was where the home directories of the users were placed (that is to say, /usr/someone was then the directory now known as /home/someone).
I'm off to saddle up my piggie squadron for a flypast.
Personally, I think the FHS is stupid marketing wankery that barely made sense in 1995 when the "preciousness" of the root partition was relevant, and has in the annals of time, caused more problems than it ever could have solved. Hey, what's the path to python? Is it:
/usr/bin/python
/usr/bin/python2
/usr/bin/python3
/usr/local/bin/python
/usr/local/bin/python2
/usr/local/bin/python3
/usr/local/bin/python2.7
/usr/local/bin/python3.7
I saw this great quote recently -- I know it was about something else, but I still wish Dan and the other FSSTND/FHS people could have seen and understood it all those years ago:
I suppose people blindly following suggestions on the Internet will eventually learn a hard lesson -- don't.
Because that is what the FHS was: A bunch of suggestions on a mailing list, with a smattering of armchair philosophy lacking any real analysis. But I mean, that's just my opinion. What if I'm wrong? ¯\_(ツ)_/¯
[†]: The answer is whatever /usr/bin/env python says.
Compare also to https://enterprise.verizon.com/products/security/advanced-th... (formerly known as Protectwise), which is not open source.
Is it fully about online publications now?
If you are here, you probably care about internet security. Verizon Media runs a pretty significant Bug Bounty program https://hackerone.com/verizonmedia which you can read about here https://www.protocol.com/hackerone-bug-bounty-virtual-verizo... and their blog here https://www.verizonmedia.com/technology/security#/bug-bounty
You might even be looking for a job as an information security professional. You can join "The Paranoids" team (now that's a good name, don't you think!) by checking out some of their jobs. https://www.verizonmedia.com/careers/search.html?q=paranoids
Verizon Media (Engadget, Huffpost, TechCrunch, Yahoo Finance, News Sports etc) are the other big component of the company.
They're an ad network.
Well, there is also https://github.com/aol/ExtJS4-Fart
Of the two, I'm more familiar with Metron (I actually did a small amount of work on it back before it was an Apache project). The core "thing" of Metron was always a large-scale, high-speed packet capture mechanism that would allow you to apply real-time streaming analytics / ML to packet streams, as well as supporting indexing the packets with ElasticSearch for post-hoc retrieval / analysis.
Spot seems to employ some similar ideas, but I haven't dug into it as deeply.
Just chickens, I hope
There is some previous discussion of Moloch when it was released in this older thread: https://news.ycombinator.com/item?id=20586005
I DID find documents about Moloch floating around my Google Drive from ~2013-ish. I believe I invited your co-author Eion to present at a conference I was running, THREADS, in 2014 and that he was not able to make it. The focus the _year prior_ was exclusively on DARPA CFT. I combined those two events in my head and thought your project got some seed funding from DARPA too. I'm sorry!
Here is the conference:
THREADS 2014 when you were invited: https://github.com/trailofbits/threads/tree/master/2014
THREADS 2013 was a retrospective on DARPA CFT: https://github.com/trailofbits/threads/tree/master/2013
May DoD turns off the forward secrecy stuff and escrows keys?
The longstanding existence of tools like these --- and there are "better" ones that aren't open source, and have been for decades --- is one reason that "vulnerability equities processes" don't make sense; if the DoD uses an exploit against a foreign target, it can't just reveal it a few months later without compromising sources and methods.
(That doesn't mean you should care about that problem; I'm just reporting).
If you had an application http server running, is traffic sent to Moloch first, and forwarded to the http server like a proxy?
Your hypothetical application server would not even be aware that this was taking place.
Names change. The common theme: names are not easy, sometimes they are beloved brands, sometimes they fall out of favor. Sometimes they were just bad ideas from the start, but happened anyway.
This gives rise to an interesting challenge for open source projects when you have an open source project in a github org, and the name of your company changes (or your company gets acquired), should you move the project? The problem is real since you don't want to lose your community, but you don't want to be stuck in the past.
Moloch whose mind is pure machinery! Moloch whose blood is running money! Moloch whose fingers are ten armies! Moloch whose breast is a cannibal dynamo! Moloch whose ear is a smoking tomb!
What else are we supposed to think of?
“The best minds of my generation are thinking about how to make people click ads.” –Jeff Hammerbacher
Allen Ginsberg's poem "Howl" starts with the line:
"I saw the best minds of my generation destroyed by madness"
Profoundly and deeply unsavory.
Your life must be easy.
I won't speculate on whatever led you to such fatuous conclusions, because ad hominem is a bad reasoning.
Moloch isn't the good guy in Howl.
Moloch! Solitude! Filth! Ugliness! Ashcans and unobtainable dollars! Children screaming under the stairways! Boys sobbing in armies! Old men weeping in the parks!
Moloch! Moloch! Nightmare of Moloch! Moloch the loveless! Mental Moloch! Moloch the heavy judger of men!
Moloch the incomprehensible prison! Moloch the crossbone soulless jailhouse and Congress of sorrows! Moloch whose buildings are judgment! Moloch the vast stone of war! Moloch the stunned governments!
Moloch the meddling micromanager from Maine!
Moloch the Junior Architect!
Moloch the 3-space indenter!
At least Moloch isn't real.
> Moloch has been used figuratively in English literature from John Milton's Paradise Lost (1667) to Allen Ginsberg's "Howl" (1955), to refer to a person or thing demanding or requiring a very costly sacrifice.
I mean a project called Zeus would hardly founder on it's namesakes fetish for turning into a bull and assaulting human women.
E.g big cities might get referred to as molochs the way they eat into nature around it or suck people into its anonymity
For me at least, because of the city-context, Moloch matches: The moloch could stand for the amount of activity in a system that is humanly insurmountable to get a better-than-superficial understanding of.
I didn't have that strong of an association of Moloch with big cities as it felt like when I looked it up for the above answer, so I googled a bit of its usage on news sites and it looks like in Germany it is almost exclusively used for big cities, whereas in Switzerland, cities is the most common usage but it is applied way more liberally.
Within a minute or so I saw it used to refer to: The EU, a government bicycle program, a 2km long tunnel in Zürich, iTunes, the Tour de France, Goldman Sachs and fashion.
Software should be cold, calculated and cannibalistic.
Leave cuteness to the experts: puppies and such.
That said, I totally respect the project's decision to name it so. I'm sorry if it comes off as an outrage or an unnecessary rant. I understand it takes away the energy which is otherwise better spent on a constructive discussion of this excellent project, which is also surprisingly well-maintained. But, I hope you realise why the name is controversial, especially since everyone that has to use the project has to get used to seeing/reading/using/typing Moloch everywhere from the docs, to the CLI, to the FAQ pages, and what-not.
May be the project can explain their rationale in a separate webpage (if they want to) and that might help?
Makes more sense.
Why get bent out of shape over the name of a software project? It's virtually a meaningless factor in day-to-day life.
What about hearing phrases "Sacrificing a Chicken to Moloch," the "spirit cooking" culture and related symbolism rampant in elite political circles? Shouldn't we be more interested in that?
Lots of low-hanging fruit to pick, I guess. Someday I might unlock the "downvote" ability on this platform. Until then my opinions don't carry weight here. Also, uhh who decides the "threshold" for downvoting? Hint: nobody knows. [0]
This platform has become more of an echo chamber than a host of rational discussion based on merit. I suppose that's a problem with growth.
There's a lot of conversation around "Moloch" as a name and the subsequent emotional responses... but not a lot of discussion about the tech at-hand. And it's a repost.
Where's the value?
[0]: https://news.ycombinator.com/newsfaq.html
Edit: while this is high-ranking comment, I'd point out that if I had the ability I would have just downvoted the comments I didn't like. Take that for whatever it's worth.
Moloch is a serious open source project, run by serious people who care about network security. They published their code under an open source license, showing off just how confident they are that it is solid. You can use this project to inspect packets on your network, you can learn how they built it and become a valuable network security engineer with a job somewhere finding people trying to hack in to your site. You can propose modifications to make this even better (and if your code is good enough, it will get accepted and used by security teams around the world). Or you could focus on the name they chose and the name of the company at the time this was published.
I'd seize the opportunity to talk tech and focus on network security. Infosec jobs pay better than brand marketing jobs.
Then I see projects like this, or Jaeger/OpenZipkin, or Chaos Monkey, and I'm simply inspired. Much like listening to a killer record.
I don't spend time feeling sorry for myself because an ancient, somewhat esoteric proper noun was used to describe the project.
A lot of people do care about what things (living or not) are named and even how, sometimes and attach a lot of emotional value to it for a good reason.
Quoting Wikipedia: https://en.wikipedia.org/wiki/Product_naming#Principles
- They strategically distinguish the product from its competitors by conveying its unique positioning.
- They hold appeal for the product’s target audience.
- They imply or evoke a salient brand attribute, quality or benefit.
- They allow companies to bond with their customers to create loyalty.
- They have a symbolic association that fortifies the image of a company or a product to the consumers.
- They help motivate customers to buy the product.
I'm sorry but claiming that an uncomfortable discussion on a project's name is not the response one wants to see is exactly what turns this into an echo chamber? To be noted how discussion on a product's technical merits isn't mutually exclusive to discussion on its naming. Both could easily be had, and both have their places, because I don't particularly think either is off-topic. In fact, a previous news.yc thread on this product brought up a similar line of discussion. I guess one could argue the discussion is counter-productive in a technical forum, but I don't think one should avoid it just because... unless the community rules explicitly state so. Technical people aren't robots, at least not yet.
I'm arguing that the naming of a thing matters less than the efficacy of the thing itself. Doubly so in a hacker-oriented forum.
As for the echo-chamber, that's a classic human fallacy which I think is magnified by the centralization of content aggregation into heavily personalized "newsfeed" style design choices perpetuated by major players.
The epidemic of "outrage culture," or "offense culture" is something comedians and free thinkers have been discussing for awhile, and I think is underpinning of my personal frustrations.
-----
"This idea of "I'm offended". I've got news for you - I'm offended by a lot of things too. Where do I send my list? Life is offensive. Get in touch with your outer adult, and grow up, and move on."
-----
Post, comment, discuss as y'all like, right? That's the beauty of the format. I'm simply one of ∞.
The folks in the community here are brilliant, and the reasonable ones are off doing something meaningful, whereas I'm here arguing for practice, basically.
None of the critical comments were particularly strident ('bent out of shape').
Your comment might as well be about itself.
Oh but now I’ve contributed to it as well. Uh oh...
"Look and despair, yo!"
To be anti-grandfathered ?
I’m a rational guy to the best of my ability, but FWIW I would avoid using this project solely because of its name (just the thought of that name evokes a sick feeling to me - that’s how strong the negative association is... I won’t explain why as this isn’t an appropriate venue for that).
We are human, and sometimes visceral responses can’t be ignored regardless of their irrationality.
I could invoke Godwin’s law to give a more universal example of words with negative associations, but I’ll refrain.
Like it or not, people will have internal reactions to things—and I for one would prefer if others let me know if they found a project name I chose to be difficult in some way.
excuse me...w-what?