Network Processing Powerhouse Kentik (CloudHelix) Raises $12.1M Series A
techcrunch.com
techcrunch.com
Re: data backend and analytics - happy to discuss... A poster we did earlier this year is @ http://avi.net/flocon-hydrasql.pdf or the marketed-up version is a white paper on the site - https://www.kentik.com/wp-content/uploads/2015/06/Kentik-KDE...
We've considered OSSing the backend (what sits behind postgres via FDWs) but it'd take require expanding the team working on it beyond the current ~ 1.5 FTEs. Maybe post-B...
Another note of potential interest: We applied to YC Winter 2014 and didn't get an interview. Same team, last minute sub, not everyone was yet full time, silly idea (though we did mention as an alt what we started Kentik to do). Lessons as are noted in many places: 1) Read the reqs, 2) Work on proposal, 3) Don't stop because you don't get {in,an interview}.
A couple of their founders were early people from CloudFlare, too.
I've met them, and it's a very solid product and solid team. If you want good visibility into complex, large-scale networks, this is a tool to watch.
Funding announcements are boring, but I hope they'll post more about how they're doing high-volume analytics to HN in the future.
My perception is it's getting harder to monitor large systems at the same time as it is getting easier to just push raw packets (in a distributed, CDN-type sense). The complexity of running Enterprise networks in the 1990s used to be lots of non-IP legacy protocols, router bugs, etc.; and the limitations of silicon pushing a lot of things into Layer 2 vs. Layer 3 processing. Weird discontinuities around Cisco slacking off around 2000, too.
The big issue we kept hearing from folks from the notes community is that the current 'cheat' is totake flow (now sFlow, IPFIX as well) and just instantly aggregate it to turn it into tsdb-like summaries.
The challenge is that you don't actually know in advance what you want to look at details for - whether you're investigating perf, availability, security, or just efficient ops.
So that's the current big problem - how to get and store and analyze thousands to tens of thousands to millions (for big networks) of flow records/second.
The 'old' problem from 1996 to 2006 of devices not being able to export has mostly gone away. There are bugs that happen every so often (like Juniper / sampled that ras called out and got resolved last year), but newer Juniper and Ciscos can do line-rate NetFlow/IPFIX to ~ 100-200k flows/sec per devices or sometimes per card. Which feeds back into point #1. Not that useful to have extra resolution if your systems are just doing top N on 10 fixed aggregation dimensions...
What I'd really like to see the silicon do is tcpstats, app decode and latency data. Both have applications for performance and security (even perf data can hint at botnets vs humans, for example, and of course URLs let you look for sql injection and app vs network problems).
The load balancer folks and some of the "Network Packet Brokers" have roadmap or the beginnings of this but it'd be better to have on Broadcom...
Or, an even more basic thing would be to have not per-packet sampling+sFlow like Broadcom and other chips do, but also a primitive for "flow sampling" to get all packets for 1/Nth of flows (say, 1/1000th) to get sampled such decodes up on a daemon running on say Cumulus or Arista.