Comment by mastax | Hacker News Reader

Comment by mastax | Hacker News Reader

Parent Full thread

I've been using QGIS a bit for simple things. Viewing OSM road data in a PostGIS database. Viewing geojson files. Everything has been more difficult than I expected, but I think that might just come with the territory. I'm working through PostGIS in Action to try to get the requisite background knowledge.

One thing that's frustrated me, and I want to make sure it isn't a misconfiguration on my end, is that QGIS feels really slow. For example, I have a 150MB geojson file which has 300,000 points with associated metadata. Even when I'm zoomed in such that I can only see a few thousand points at a time, if I pan the map over by 50% it takes at least 10 seconds before it loads in the new points. Many long operations seem to take place synchronously on the UI thread, so the whole app is unresponsive while they take place. Clicking the drop down arrow on a PostGIS Schema to view the tables spins for several minutes. No other Postgres tool I have takes that long, so it's not the database. The PostGIS import/export tools were also extremely slow and didn't have progress bars. I'm using 3.24 currently. I don't want to rag on it too hard, but it's really hampered my enthusiasm for working with maps and GIS.

QGIS is always a little more difficult than expected. Even as someone with a masters in GIS, its still hard for me to remember where they put things or how to do it. Its extremely powerful but can be extremely frustrating too.

GeoJSON is unfortunately one of the worst formats to keep the data you are using in. Its great for transport and interoperability, but there is no way to index the data in the format. It doesn't matter if you are only drawing a few thousand points, it is still looking through all of them to see which ones to draw.

GIS is awesome, and don't let one tool get you down. If its something interesting to you, there is a lot more to it than just QGIS. Part of my pain with QGIS is because I rarely ever use it so I forget what I learned last time. I spend most of my time purely in python, and don't really need to.

You'd probably be well-served by exporting to some format with a spatial index. GeoJson is just a textfile, not much better than reading in a CSV. The current recommended format is Geopackage (gpkg), which is based on SQLite.

Edit: agreed on the slowness of DB connections, I've found that too, including for relatively small locally-hosted DBs.

Yeah, converting it to gpkg sped it up a lot, thanks!

I assumed that since geojson is completely unsuitable for querying directly, it would load the whole thing into a native in-memory format, but perhaps not.

I don't think geojson is a great format for anything with more than a few MB of data.

I wanted to see exactly how bad it is with a largeish datasets, so I exported the New Zealand address dataset[1] with ~2.5M points as a geopackage (750MB) QGIS loads this fine its a little slow when viewing the entire country but when zoomed into city level it is almost instant to pan around.

Using ogr2ogr I converted it to ndgeojson (2.5GB), It crashed my QGIS while trying to load it. Using shuf I created a random 100,000 points geojson (~110MB) it was unbearably slow in QGIS while panning around 5+ seconds.

I currently use and recommend flatgeobuf[2] for most of my working datasets as it is super quick and doesn't need sqlite to read (eg in a browser).

It is also super easy to convert to/from with ogr2ogr

ogr2ogr -f flatgeobuf output.fgb input.geojson

[1] https://data.linz.govt.nz/layer/105689-nz-addresses/data/ [2] https://github.com/flatgeobuf/flatgeobuf

Can’t you use clustering techniques?

No, you're not victim of misconfiguration. QGIS is very slow with large datasets, no matter what source you use (even if you have Oracle with costly Spatial extensions). It is due to some unfortunate design decisions - the app always loads the full dataset from the source, and draws/shaws every single feature. It is possible (through some recent, obscure check in the prefs) to tell it to skip smaller objects when putting everything on the screen, but generally is not possible to tell it "do not load features from PostGIS that are going to render to only few pixels".

Another reason is that the underlying C++ engine is tied in such way with the fairly-large Python codebase, that it seldom uses all CPU cores. so you have a single process app. Add to this the fact that the GPU is not used extensively and you get one very slow APP.

So, yeah, for datasets larger than 50k points its slow on arbitrary hardware. Some analysis are impossible to run unless you go to PostGIS

To be honest, ESRI is not a company that everyone loves, but they really invest into ArcGIS Pro, even though they also are not there yet. Both QGIS and ESRI shy away from spatial SQL which is like times more effective for spatial analysis.

To be honest, QGIS is quite old-school in design, and the core devs know that it would take enormous effort to implement it from scratch... at least the slow parts. The fact that it works and is open source, does the fundamental things you need does not make it the _top_ software. this "opensource ftw" is really stupid when you have to work with hundreds of layers with hundred Ks of points.

Make sure you have estimated table metadata turned on, otherwise QGIS will run a bunch of queries to understand your tables I believe.

We typically use QGIS as a viewing engine only, if you let postgis do the heavy lifting it's a beautiful setup, especially with a tuned dB and a indexed clustered postgres table

> Make sure you have estimated table metadata turned on, otherwise QGIS will run a bunch of queries to understand your tables I believe.

That fixed it, thanks! Looking at the manual, there is a tip that tells you to turn that on, otherwise it will read the entire table to characterize the geometries... It's a bit mad that the default behavior has it querying potentially gigabytes of information across the network every time you open the app and click a dropdown, with no progress bar. But it's definitely the type of app where you need to read the manual, and it says so right there.

Glad it worked!

It should only do it once per table but if you are dropping and creating tables often it will be making the DB have a heavy workload

I think theirs an assumption within the geo community of read heavy workloads but I could be wrong

Could you provide more info about your hardware setup and data size? Why do you need hundreds of layers? Which domain you are working on?

Reply on news.ycombinator.com