Hacker News Reader

> - AI based on algorithmic induction tech i.e. not the usual DNN and ML tech everyone calls AI. This is way more interesting if you have a novel approach.

I know it's fashionable to hate on deep learning, but algorithm induction is literally what deep learning does.

I would love to read an elaboration on this. There's so much I don't understand about AI.

I believe he's talking about https://en.wikipedia.org/wiki/Solomonoff's_theory_of_inducti...

> AI based on algorithmic induction tech i.e. not the usual DNN and ML tech everyone calls AI. This is way more interesting if you have a novel approach.

Any chance you can elaborate on what you're talking about here?

I think the OP is referring to algorithmic probability. See

[1] R. J. Solomonoff. A formal theory of inductive inference: Parts 1 and 2. Information and Control, 7:1--22 and 224--254, 1964.

[2] R. J. Solomonoff. Complexity-based induction systems: Comparisons and convergence theorems. IEEE Transactions on Information Theory, IT-24:422-432, 1978.

[3] http://www.scholarpedia.org/article/Algorithmic_probability

I believe he is referring to AI that can generate new ideas, similar to 'thinking' versus pattern recognition, which is 'I've seen something like that before (during training!).

So AI that can develop algorithms on its own? Basically AI that can write code?

Perhaps he means old-fashioned AI techniques like production-rule systems, knowledge representation, and truth maintenance. If so, and if he's right, that would interest me, since I know how to do those things.

I understood the question as, what makes you think: the current system could be streamlined so easily that I can hardly believe someone hasn't already disrupted the complacent firms extracting rent.

Such a question usually gets answers mostly comprised of work that is extremely boring at first glance. For example, create a system than halves the work to complete documentation for some sort of compliance with regulations.

Every one of these ideas just sounds incredibly interesting to work on. The type that gets a lot of really bright people together on a team (that I'd really want to be on too), yet they might not deliver a product, and possibly even less likely figure out how to monetize it.

Anyway, I'm not developing a friction-reducing product either, but worry about choosing something because it sounds like a rewarding, intellectual challenge.

A lot of that is really really specific for what you do. I think people are generally trying to solve ONE problem rather than creating a solution for the general problem, because usually, a general solution breaks down very very quickly and thus low retention rate.

> Remote sensing analytics. Drones and satellites are generating spectacular volumes of this data and no one can usefully analyze data of this type at scale. Today, companies wait weeks for a single analytic output on less than a terabyte of data.

I am co-founder of tensorflight.com. We do computer vision analytics of drone imagery. Interesting that you mention it, as I thought it's a somewhat obscure market. When we talk to investors in the valley 75%+ have to be educated about why what we do is a viable business.

Please get in touch at kozikow@tensorflight.com if you have any ideas!

Very insightful. In your expert opinion, what are some of the core features / functionality that an IoT database platform should have (that existing solutions or combinations of them on AWS / Google Cloud / Azure don't provide)?

Could you be more specific about the IoT DB issue? What would a credible solution do?

Thanks in advance.

An IoT database requires the intersection of three software capabilities in a single platform, which have varying degrees of accessibility (the technology exists in principle) and none are available in open source currently.

- An exabyte-scale storage engine. Nothing too exotic here technically and a few companies have built them, but the design needs to address continuous data corruption, continuous hardware failure, geo-federation, etc.

- A real-time database kernel that supports very high throughput for mixed workloads. A production kernel of this type doesn't currently exist though several people working in closed source databases understand the necessary computer science in principle; the academic literature is far behind the state-of-the-art. The ability to gracefully shift load and transients between servers under full load with many millions of writes per second is not trivial.

- Native discrete topology operators. Necessary for geospatial analytics, sensor coverages, etc. If you can do it natively in the database kernel, it makes the second requirement easier to achieve since you don't need secondary indexing generally.

Any solution even halfway toward the general solution would be viable. The value possible if you have such a system is hard to overestimate. Companies have paid half a million dollars for the output of a single analytic query on tens of trillions of IoT records; the differentiator was that it was possible to execute such a query at all.

It is extremely high-end and polymathic computer science, but serious valuable if you can make a credible dent in it. And unlike some advanced topics in computer science, there are no epic unsolved theoretical problems you have to solve, though some relevant computer science may be unpublished.

A friend is working on an IoT problem with two of your three capabilities: storage and analytics. He has voiced similar concerns. (To be fair, he gripes constantly that nothing exists which would work for him, within the budgetary and timing constraints.)

His company provides sensor systems for civil engineering projects. A single large bridge can have sensor packs every 50m or so - per beam. The amount of sensor data coming in for a single municipality or region is already staggering. Vibration and stress analytics are required on a daily basis.

The final requirement, one which you didn't mention, is that this setup should be fairly low maintenance. If you need a team of rocket scientists to operate and just keep it from falling over, the cost structure will be unsustainable.

A service that provided all this in a platform with sane APIs and good BI integration should be making tons of money.

> budgetary and timing constraints

ain't that the truth

Instead of storing the data and then analyzing it, make it into a stream! eg: collect | analyze | report

During development or if you want to store the raw data you can split the stream. eg: collect | tee >(store) >(anotherAnalyzer) | analyze | report

What are one or two concrete things you'd actually use this for?

Maybe a general solution just isn't cost effective and may never be cost effective, regardless of how much we bring down actual costs for the resources involved in solving data-intensive problems. The (possible) reason? The performance advantage you gain from tailoring specifics to structural relationships within exceptionally large datasets may just be too large to trade off for some "generalized" solution.

I'm just speculating, of course. I was sort of entertaining your idea and going back and forth between "a sounds-good fantasy" and "No, that would be awesome, why does this not exist?"

Hi Andrew - What's the status with SpaceCurve?

http://spacecurve.com

The first two challenges are quite familiar in the financial industry, and kx, with its kdb+ time-series database, is arguably the industry leader. Most people in the open source sector don't pay it much attention, since its proprietary nature and terse APL-inspired syntax [0] lend themselves towards "I'll make my career in this, become a wizard, get Wall Street money, and not be very incentivized to compete with myself by sharing knowledge." To the third question, I'm not sure to what extent it treats topology natively, but kx is certainly positioning themselves as an IoT solution provider [1]. Of course, the cost is likely prohibitive for smaller IoT startups, and more innovation would certainly be good from a supply-demand perspective, but from a technical perspective I'm not sure this is an unsolved problem.

[0] https://a.kx.com/q/d/q.htm

[1] https://kx.com/solutions/utilities/

KX is a leader in what it does but it doesn't really fit the canonical IoT use case, either from a workload or analytics perspective. The challenge is that IoT isn't just temporal, most of the interesting relationships are spatial as well since you are analyzing relationships across sensor and telemetry streams, and the analytics are real-time mixed workloads with ridiculous working set sizes. It isn't their market. They rock the time-series market though.

A good example of an IoT data analytics problem is analyzing a petabyte of drone sensor data, which on a large drone amounts to a few flights worth. Typical raw sources tend to be some combination of hyper-spectral imaging/video and LIDAR. Or RF probability functions e.g. mobile. Or a combination of all of the above because you are fusing multiple sources to reduce the uncertainty for your analysis.

FWIW, the "tens of trillions" of IoT records I mentioned was a real-world example from one of the most famous financial companies. It was a spatial analytic on a polygon model, and a classic IoT data model. If KX solved that particular analysis problem, they would have used it.

kdb is awesome, but very tailored to one-dimensional (time-ordered) data. I've built a couple multi-dimensional array database engines in my time and -- trust me -- it's a really difficult problem to solve. That said, I have a design kicking around for one that meets most of the OP's demands, so if anybody wants to join me...

There's been an explosion in raster data in just the last 5 years. Cheap satellites, cheap drones, and cheap platforms have really turned the firehose to "high". And the resolution on scientific data -- climate and weather model output, astronomy data, particle physics data, seismic data, etc. -- just keeps going up.

This is an area of massive growth for which no existing solution is quite adequate. If I had the time and/or the cash, I'd be diving in head-first.

Flurry (now part of Yahoo) has experience with IoT storage at that scale (or near it). The "T" just happens to be your phone. Data processing (and staying in business) at that scale is a matter of great teams and solid architecture. This won't be a "service" easily sold to a general audience.

The data processing could become a commodity but the architecture won't be -- its highly tied to your specific industry.

I'd also add, there's a component of streaming analytics that isn't solved either.

One of the points I've tried to make at various companies (we've worked at the same one before) is that streaming solutions and batch solutions need to be fused into a single execution engine.

A streaming system on its own (operating on temporal windows) is not nearly as useful as on that can be joined to a storage engine with data at rest. It also needs to be disk based, so windows can be large, which most people do not want to take on. It also needs to be extremely parallel, and efficient.

Thousands of requests a second per server is not even in the right ball-park (which is lots of current execution engines now). Operating at line rate is generally table stakes IMO. The operations on the stream should be parallelized automatically, up to petabytes a day of input. Humans don't have the necessary context to do the partitioning up front, especially with streams that change.

The issue is (and I've tried to come up with designs to address this, though not in practice), is that co-locating the data at rest, with data that is moving through the system is a tricky problem, especially with complicated joins.

They can be the same engine (and should), but traditional database engines tend to have a problem with streaming queries, since they are just repeatedly executing a query against every new record. They are expressible, just not efficient. There is room to innovate in this space, but most people building these engines either solve the parallelism problem naively, or not at all.

There's also the problem of driving this computation to the edge, which is also something I have a solution for in a way that no one is doing, but have not yet met a company willing to take this level of effort on.

All the points you make about the kernel are apt, as are the points about the distribution algorithms. Also, the protocols used aren't nash safe, so at scale most of these systems become an operational juggling act under pressure.

All streaming systems that I know of do not know enough about the underlying data to gracefully rebalance and co-locate, since they all tend to embody the map/reduce paradigm, which is oblivious to underlying data distribution, at least in current practice.

There is available computer science to solve all these issues, I think some of the spatial algorithms out there can also be applied to the streaming space, especially in join evaluation.

You shared a lot of deep insight here. Thanks.

> something I have a solution for in a way that no one is doing

Is the general direction for this something you can share? 30 years of database literature accumulated a lot of knowledge. It's be a bold claim to say there's something powerful yet non-obvious.

Unfortunately, my employer and team are directly involved in this space. We may not go this way (due to the effort), but its something we may tackle.

It's not necessarily new computer science, just a clever (if I can be so bold) way to tackle edge computing in the context of a streaming engine.

As someone that has "repurposed cartographic tools" to analyze spatiotemporal datasets, what specifically do you feel is missing from current tools?

I spent many years repurposing cartographic tools myself. :-)

Three big missing features off the top of my head:

- Insufficiently correct and high precision computational geometry, which compounds with the iterative/recursive nature of many complex sensor analytics. Many people don't notice unless they ground truth their analytic process; I learned this the hard way. For many industries, 1% cumulative computational error is a catastrophic bug for analytics and the reality can be much worse in many common systems.

- Lack of first class tessellation types and operators. Once your data scientists have them, they'll wonder how they lived without them. Such things are completely useless for cartography and therefore don't exist in those platforms.

- High-performance computational geometry. This is particularly noticeable if you work with sensor coverages (like drone data). Your typical cartographic system has serious difficulty joining a few terabytes of complex polygons, but these are tiny data sets for many remote sensing sources. It literally takes weeks or months to run these types of queries. You can optimize this to be much faster but there was no pressing market need in cartography and cartographic systems aren't designed for scale-out generally.

Interesting! I'll ponder over these; thanks for the insight!

Aren't these addressed by CGAL? Or you mean, you'd want database primitives using CGAL?

Unfortunately CGAL is not suitable, and yes you do need to support it in the underlying representation and database. The litmus test for a suitable computational geometry engine is the ability to do arbitrary ellipsoid computational geometry at something like one part per trillion error, and the ability to compute intersections of very complex polygons on that surface quickly.

On that last point, you can execute an effective denial of service attack on most GIS databases with a well-crafted set of polygons and polygon intersection queries. It doesn't even require malicious intent. (Yet another thing I learned the hard way.)

I am not sure if all those goals can be combined.

For example, the intersection of n polygons could have exponentially many segments. To prevent denial of service, accuracy has to be sacrificed, but "one part per trillion error" sounds hard.

As another example, representing the intersection between two line segments without error requires a data type that has three times as many bits as the x/y components of their endpoints, so rounding has to happen here, too.

The only solution I can come up with that would not have those problems would be to subdivide a plane into grid cells of size one trillionth of whichever unit, rasterize all polygons into it do the Boolean operations at pixel-level, but that would require huge amounts of memory and processing power. And additionally it would be vulnerable to a DOS attack where the attacker sends many very large overlapping rectangles.

Lastly, huge amounts of data would require many computers to be able to work on this problem in parallel, but if an attacker sends only overlapping polygons, parallelization would be very tricky if not impossible.

Any chance you could give an example use-case or three? Sounds fascinating -- I use PostGIS and find it to be slower than I'd like day to day, but it sounds like your uses demand a lot more precision than I have call for.

I am late in this conversation, I would like to dig more into the IoT and the needs for tooling for spatial analysis. Could we talk more about the problems you are facing? My email is on the profile page. Thank you!

> Population-scale behavioral analytics. Many startups claim to do this but none of them can actually work with relevant data at a scale that would deliver on it despite increasing availability of the necessary data.

Making Asimov's Psychohistory from Foundation a reality!

Can you clarify what you mean by behavior analytics and why companies eould find that useful?

Think Hari Seldon/psychohistory.

Predicting mid to long-term trends and demographic shifts far enough ahead of time to make investments.

If the data is there, and if we insist that machines need to predict trends for us based on that data, then why do we need to hire analysts for this job?

Sometimes because they are intentionally gatekeeping, as in Half Past Human's conversational delta analysis reports.

Can you expand on the last point?

> no one offers a credible solution

> There are many VCs currently hunting for this product

Implies a difficult problem.

> algorithmic induction tech

Do you mean inference?

Could you describe the applications for these tools?