Hacker News Reader

Comment by jandrewrogers | Hacker News Reader

I've worked in "climate intelligence" for many years. The list overlooks one of the largest and most immediate opportunities around that market: the data infrastructure and analysis tools we have today are profoundly unfit for purpose. Just about everyone is essentially using cartography tools to do large-scale spatiotemporal analysis of sensor and telemetry data. The gaps for both features and practical scalability are massive.

It has made most of the climate intelligence analysis we'd like to do, and for which data is available, intractable. And what we can do is so computationally inefficient that we figuratively burn down a small forest every time we run an analysis on a non-trivial model, which isn't very green either.

(This is definitely something I'd work on if I had the bandwidth, it is a pretty pure deep tech software problem.)

View on HN

Agree 100%. This is big part of the motivation behind our new startup Earthmover: https://earthmover.io/

Our mission is to make it easier to work with scientific data at scale in the cloud, focusing mainly on the climate, weather, and geospatial vertical.

My cofounder Joe Hamman and I are climate scientists who helped create the Pangeo project. We are also core devs on the Python packages Xarray and Zarr. We think that a layer of managed services (think a "modern data stack" oriented around the multidimensional array data model) is exactly what this ecosystem needs to make it easier for teams to build data-intensive products in the climate-tech space.

And we're hiring! https://earthmover.io/posts/earthmover-is-hiring/

I'm curious, have you considered using PyTorch or JAX for tensor processing? ML libraries seem to be much further along when it comes to performing compute-intensive, hardware-accelerated operations on Tensor's. And you get gradients basically for free (in terms of developer time). Also, the kernel compiler being added PyTorch 2 looks very promising.

PyTorch and JAX are used heavily in climate science on the ML side. For more general analytics, not so much. Many of our users like to use Xarray as a high-level API. There has been some work to integrate Xarray with PyTorch (https://github.com/pydata/xarray/issues/3232) but we're not there yet.

The Python Array API standard should help align these different back-ends: https://data-apis.org/array-api/latest/

The primary issue in this domain is not compute - it's I/O, especially when you need to perform complex computations with intermediate data that doesn't fit into memory.

Elixir?

Companies with good climate intelligence tech tend to evolve in to marketplaces because that gets them closer to the money. Climate projects can't afford SaaS, but offset buyers are willing to pay a premium for offsets re-verified by high-tech climate intelligence SaaS.

Example: Pachama https://pachama.com/

Out of curiosity, is accessing & working with large datasets a problem in your areas of work? I run a weather/climate site that makes some of this less painful, taking datasets such as GFS or ERA5/ERA5-Land much faster to access. We have some enterprise clients who really value the time-saving aspect of this but I also feel like everyone has their own data-processing set up and problems are different for everyone.

There are a couple issues I see with basic access and working with large datasets. Ease of access for typical users is also a valid issue.

First, we still mostly move the data to the computation when we should be moving the computation to the data. Moving the data works fine when data is small but if the data volumes are large (as sensor/geo data tends to be) then it can take an incredibly long time to move the data. In many cases, more time is spent shoveling data over the network than actually doing the computation. This has become worse as storage density has increased, hundreds of TB/server is ordinary.

Second, the data is rarely organized in a way that makes it efficient to extract arbitrary subsets. There is still a lot of what is essentially "grep at scale" going on. Again, not a problem if the data is small but if I need a specific 50TB subset of a 10PB source, this becomes prohibitively slow. The data needs to be organized such that we can slice and dice it with high selectivity in place, much more like a proper database and less of a distributed filesystem. Because spatiotemporal analysis tends to involve iterative join-like operations, you want this to be efficient as possible.

The other big problem is many of these data sources are too large for everyone to have their own copy. Or if they did have their own copy, it would be extraordinarily wasteful. This is adjacent to the first issue. EDIT: And herein is the likely business model.

Want to make sure you're familiar with the Pangeo community: www.pangeo.io

I don't think any of these challenges are "solved", but there's a groundswell of technology that is well-situated to make a big impact in these domains. The largest barriers that still remain are the ownership of engineering processes/workflow to transform larger gridded datasets to ARCO (analysis-ready, cloud-optimized) formats, as well as tooling to mediate between heterogeneous datasets (e.g. combining regular vs irregular or arbitrarily gridded data, such as land surveys or ZIP codes).

There are definitely players in the space working on these, but much is left to be done here.

+1 for Pangeo. We use these toolsets heavily (Xarray, Zarr, Dask) to run our service, which is essentially what you described as taking the larger gridded datasest to ARCO format. I think this is still a bit too heavy for casual Excel/GIS analysts so we try to make it as simple as possible for them to get climate data in CSV or NetCDF format for their work.

This sounds really interesting. Would be really interested to work around these things. Thinking and working a lot with similar-ish systems. But not sure how to enter the related green-tech space when living in Europe. Would love to try to build a product myself but then I need a customer to try ideas with.

We are building data storage and processing infrastructure at a company that connects DERs. This involves storing large amounts of structured, time-series data. This data is then further processed and used for all kinds use cases, many of which were mentioned in the first half of the post's article. (e.g. energy management systems, load management etc.)

I am planning to apply to YC, and similar VCs, probably as a non-profit.

I will build better risk models of extreme weather events.

If anyone is interested in joining as a co-founder, get in touch (see email in my profile).

Have you come across Kart? https://kartproject.org/

Any chance you guys provide a free api for the little guy? I would love to have access to climate data via a json rest api. Specifically historical temperature and precipitation data at minimum.

I poked around a while back and wasn't able to find much of anything on the web. Maybe I missed it?

Certainly - take a look (https://oikolab.com) and let me know your use case. There is a free tier but we've also given free access to a quite a few number of researchers, non-profits and university students for their projects when they reached out to us.

I own and operate a high intensity organic farm.

My use case is to attempt to predict how much water I can expect for the growing season and other growing conditions like date of last frost.

I'm trying to apply my software engineering skills to my farm. I am also researching automation strategies.

Your offering is most intriguing.

Sounds good - can you sign up for the free tier and send me an email to support@oikolab.com? This comes to me so I'll know which account belongs to you and we can take it from there.

Many lessons to be learned from Climate Corporation. They used satellite data and drones to make robust crop insurance and ended up being bought by Monsanto and changing their direction.

Would you mind elaborating on say a few specific asks for tools climate people would want to have, that are low-hanging fruit that people might be able to write in their spare time?

I'm very interested in doing something for climate change but I'd like to know what people want.

> Just about everyone is essentially using cartography tools to do large-scale spatiotemporal analysis of sensor and telemetry data. The gaps for both features and practical scalability are massive.

Could you point to any readings or resources that would explain these gaps? I'd be quite curious why our current spatiotemporal analysis techniques would be insufficient. Is it the analysis tools that just need new techniques or is the problem at the source (i.e. the sensors)? Or?

There aren’t really any problems. Some people see that because all of this data can be spatially and temporarily referenced, it can be accessed better, as if there could be one format and one application that could let you do anything you want across the space time, regardless of the source.

The reality is that a lots of models using this data are developed against a specific sensor or dataset, and just don’t work or scale.

I don’t think this will be solved in this domain by pangeo or any startup in particular.

There is an awesome new STACspec that every geo company should be adopting, and that’s the direction to move towards.

But each step towards standardizing the GIS process will require something that truly everyone can adopt, sort of like JSON.

I would also love any references to existing companies, research groups, or the problems in this space if you have the time to share. I found the posted list underwhelming and more of a marketing shotgun approach to try and take advantage of the push for "climate tech" but not solve any real problems.

Not OP, but in my experience:

Jupiter Intelligence (https://jupiterintel.com/)

Descartes Labs (https://descarteslabs.com/)

Microsoft Planetary Computer (https://planetarycomputer.microsoft.com/)

Coiled (managed Dask - python HPC) (https://www.coiled.io/)

CarbonPlan (https://carbonplan.org/)

Salo Sciences (acquired by Planet, satellite imagery company) (https://salo.ai/)

lots of others

Thank you!

I guess it would help to be more specific about how this differs from some of the measurement related startups they list. Taxonomies are difficult, so maybe they do need an entirely separate category, or enlarge the one related to measurement.

We need sensors for carbon presence over distance and time in the ocean. At huge scale, to test the viability of various carbon sequestration schemes. That's pretty expensive, with a large non-software component.

I know this from a peripheral involvement in one the XPRIZE projects.

This is an excellent point. I think the problem is that because it's such a pure software problem it doesn't have an immediate "climate tech" alignment, so it stands to "dilute" these kinds of calls for funding.

Where could I read more about this problem and how it's being tackled today?

I hesitate to link to Twitter here, but Joe Morrison has his finger on the pulse of this and offers a tongue-in-cheek perspective that I appreciate: https://twitter.com/mouthofmorrison

Thanks for posting. His substack piece on the 3 m’s is very appropriate.

What potential customers are there for climate intelligence systems?

Don't think of it as a "climate intelligence" system. It is really an "analyze complex dynamics in the physical world at global scale" system, one application of which is climate intelligence. Everything is fundamentally anchored to space and time, and connected by it.

The applicability cuts across industries. It would enable managing many kinds of risks that have nothing to do with climate.

Makes sense. Although to have such a big scope and understand how to satisfy such cross cutting concerns where tools aren't built yet it sounds like you you kind of need to understand a lot about everything.

I do see that when you really understand the problem you can build such a tool in a non-domain specific way as you describe without inconceivable effort but understanding up to that point is kind of the whole problem?

Edit: Btw, really sounds like a space where I would like to build stuff in. I kind of understand what you're aiming for but I feel like it's very easy to jump to conclusions and build air castles.

Reply on news.ycombinator.com