Show HN: Quine – Streaming Graph and Origin Story
quine.io
quine.io
I’d been bouncing back and forth between software engineer and manager at a few mature enterprise software startups, and so I had first-hand experience of how customer configuration had spiraled out of control. It took an army to deploy the product after closing each sale. Complicated configuration for many product components led to a graph data model. Configuration changes occurred slowly enough that it was feasible to put in a graph database. That was a cool project.
Moving to real-time event streams at another company focused on mobile push notifications, there were challenges similar in complexity, except they occurred at much higher volumes. Graph databases were definitely too slow! —but the problem was still graph shaped. So I worked alongside other engineers trying to "turn the database inside out." We created complicated microservices which ended up covering the same challenges as a database. What if all the parts could be configured together automatically?
Quine was created as an experiment to try to unify a graph data model with a streaming-focused graph computational model. The Actor Model is an old idea (Carl Hewitt, 1973) but a powerful and fundamental abstraction which appears in many surprising ways. It is perfect for this problem.
Every day, all of my spare time was working on Quine. We had a second child and there was even less time, so I'd stay up late coding until I couldn't keep my eyes open. I'd take Saturday daytime to be with my family, and then sneak away in the evening and again all Sunday to keep working. During the sleep-deprived and crying-baby-interruption years, when felt like I wasn't making enough progress, I would get up early to spend 2 hours at a coffee shop before work in order to get some focused time for coding.
By this point, my day job was leading DARPA research programs. One of these was Transparent Computing [1], a program focused on finding Advanced Persistent Threats ("APTs") in enterprise networks. The problem required assembling instrumentation data into a graph, analyzing it on the fly, and finding unusual patterns indicating the attacker's activity. Trying to be good stewards of our research funds, we started with existing graph databases. The program goals quickly exceeded the capabilities of every graph database out there, and I trudged through all the legal paperwork to properly allow the use of my side project on this research program.
Quine was the only way to store and analyze a graph that could keep up with the high volume of data on this project; we had tried everything else. So our team focused on developing Quine in the direction needed by the research. By the end of the project, it was exceeding the research goals (40,000 events per second, ingest+analysis) and it was clear that we had a tiger by the tail.
After the research program concluded, I raised early seed funding and hired a team to help bring Quine to market. In 2022, we realized that goal and Quine was released as open source software. Our team is now fully focused on developing the open source community for Quine and supporting the enterprise version. Our team recently showed linear scaling and tested it well-past 1,000,000 events per second (ingest+analysis). We’re excited about where this headed next! The future of Quine is rooted in the open source project and with many of the most interesting applications coming from the high-volume users we're engaged with now.