UI considerations for designing large data tables | Hacker News Reader

UI considerations for designing large data tables

coyleandrew.medium.com

39 pointscoyleandrew29 comments

ag-grid is pretty great at all of this. it will get you 90% of the way there, then you just need to ensure you are ordering columns correctly and using the right types, and potentially using a few custom cell renderers to display novel data.

we're using it to infinite scroll / sort / filter a table with 3.2M rows of data and it 'just works'

I'm struggling to think how I'd meaningfully do any of that with 3.2M rows. :(

I'm reminded of old books that were various number sequences for checking calculations. Clearly somewhat useful, but incredibly niche and seems to lose to other tools rather quickly.

Hope I do not misunderstand your comment, but I think the point is not that you use millions of rows but that you should be able to use all of your rows/data without having to use pagination as a workaround to hide the problem that the GUI is incapable to render too many rows at once.

Imagine having such a terrible UX while editing a large source code files, the editor loads only 20 lines then you need to click a button or scroll down to load the next 20 lines, search would also be slow since it would do a request to the backed.

Sorry again if I missunderstood your comment, its not super clear to me what was your point.

Apologies, that was not what I meant. I meant it more directly in assuming the technical side works and you are able to scroll just fine, how do you meaningfully do so with that much data?

And I don't mean this as an attack. I presume there are some techniques that I just don't know. Or data sets I just don't typically interact with.

For the ones I am used to, aggregates are key to working with them. That and graphical visualizations. (Though, it is frustrating how many visualizations can be reduced quickly to "top N" or similar.)

It's something that's both a common B2B requirement because users keep asking for it ("I want to see all of my data at once"), and something functionally ~useless.

Say in my web app I can save "projects" and I have 200 projects in total. Because of DOM performance the devs implemented pagination say 40 items at max so I have 5 pages of projects. We all know the issues with pagination.

If all the projects are loaded I can :

1 scroll all of them , exactly like I scroll a document or how I scroll in my File Manager or Image Viewer

2 I can do Instant Filter and Search, no backend requests, don't you hate when say you open a YouTube account video list and you just can't use the browser Find function because the content is not in the page

One example is YouTube in a specific account Videos section , you are forced to scroll and wait, scroll and wait , scroll and wait , they can try and hack this to be smoother but this could be instant since the JSON for all the videos could be returned at once, it is just text and is more efficient then returning it in chunks , then like in competent GUI toolkits you have a big grid with all the results and the toolkit/framework Grid does the work for you in making sure that everything is rendered efficiently and smooth.

I would prefer at least an option in this apps to offer the customer a setting to decide the Page size , maybe I want a slower initial load but have everything loaded in one screen , imagine paginated File Manager

Agreed on most points. But 200 is a very different number from 3.2m. I would expect I could load all 200 in the client and get fast local operations. For 3.2m, I'd expect sending the filter off client would be faster.

Essentially, for performance there is a tradeoff between sending the data to where you will perform operations, and sending operations to the data. I don't know where the cutoff is, but I'd expect 3.2m to be on the "send operations to the data" side.

if there are 3 million simple objects like title, url, thumbnail url, why would be faster to send the request, to the filtering in the backed db and then send the response back? I suspect that an Array.filter would be as fast with a db search + the request overhead.

I just wish that the DOM would have some built in List,Table, Grid components, like you have in Qt,.Net WPF or Flex4 . Today we just have divs in divs in divs.

I mean, fair, you could have tiny objects? My assertion was to not send all 3m rows to the frontend, but only the results of any aggregations and such. Want to know the top N? That should require sending N rows, period. Same for filtering and such. This is basically just a restatement of the "send the operations to the data" approach. Is a big part of why doing joins in the database is so much faster than doing them in the application layer.

And, as you allude, the shuffling of all of the DOM overhead to manage what is visible is non-trivial. Yes, you can basically flyweight it to save memory, but it is only a matter of time before the user wants Ctrl-F to work and then you try to find a way to put the whole object in a place for the user to directly work against. (Yeah, you would probably try to capture Ctrl-F in the application and fake the native search. But then case folding and other concerns now have to be reimplemented by you.)

I don't really know how to interpret this comment. 3.2M rows are not loading into the browser at once, only about 10k. The page size is configurable. The frontend and backend have a contract to agree on how this works, so as the user scrolls (and frontend needs another page) it asks the backend for more. The frontend will keep up to N pages (also configurable) cached in the client. Works shockingly well without any fuss at all. The AG-Grid team really did a great job here (it is not free for this particular feature, but well worth the cost)

Apologies, it was not the technical side of it that had me musing on it. I was genuinely curious why and how I would "scroll" through that much data in a meaningful way.

And I don't mean this as a heavy criticism of the idea. I'm assuming it is useful to you. Always fun to hear about how this sort of thing is used.

You don't scroll through such a list - you use filters, pivoting etc. For end users, it's often quite comfortable to unify this in an Excel-like interface, as they're used to it. In most CRUD projects I had, users had an aversion against paged/pre-filtered displays and rather would have everything in one list where they can dynamically filter it if necessary.

Right. But that just brings us back to "how do people work with a scrolling list of 3.2m records?"

I get the point of wanting it locally to use power tools. And I get that the browser is probably capable of implementing a lot of power tools. Seems silly to insist on doing it all "in memory" on the browser, though?

That is, if the idea is you are doing pivots and filters, I don't know why a server side hit wouldn't be better for that. Similarly, when I look at something like a stock ticker for the day, I don't expect every single transaction was sent to my browser to create the graph. It /could/ be done that way, but why?

More directly to the question I had here, why and how would someone need a scroll list of every market transaction? For fine audits, I would get it, but even then I'd expect some sort of search or anomaly detection?

Still, I think if the answer is to "get it in the users hand and let them do what they will with the data," I can accept that. Goal isn't necessarily to let the users scroll the data endlessly, but for them to use any bespoke tooling they are already using.

I've tried ag-grid but trying to make it look consistent across light and dark mode was a pain. The available light and dark themes have a lot of discrepancies. To make it look right you have to dive deep into its styling system which is complex.

Have you looked at the AG Grid release from two weeks ago? We released a new theme called Quartz, which I understand is much better. Eg it has automatic switching between dark and bright mode based on users desktop preferance.

Wish HTML tables supported basic grid features like column pinning. Sadly you have to resort to bypassing `table` and instead use divs.

I feel the tension when just looking at those screenshots, thinking of trying to utilize a large table in a website UI - JavaScript, scrolling, dynamic updates, the pauses and hangups, Internet latency, etc., plus the very limited capabilities.

Just give me my local spreadsheet app. Your web UI is never going to come close to that speed and functionality.

I understand that might not fit every use case, but I guess that any user unwilling to use a local spreadsheet will not be interested in the the 'big data' of the complete, large table. In that case, cut the large table to a small one, appropriate to your platform and users, and provide a link for the data scientists to download the full thing.

Think of multi user databases where you want the latest version of the data. Local spreadsheets would be a synchronization nightmare. Also, web applications are seldom one single sheet, but an intertwined view on lots of tables combined with custom functionality you can't replicate easily in a spreadsheet. And that custom functionality needs to be updated once in a while. The web has become the easiest software delivery platform: you only have to update the server side. Even if a web interface is a bit janky it's a better alternative than maintaining lots of native apps (only Windows, Mac, how many Linux distributions?) or one of the cross platform toolkits that don't ever work 100% right and still require you to test on every platform.

Those are good points. I was thinking only of one use case, but obviously there are many more. It makes me think someone might try to create a high-performance spreadsheet-like UI for web applications. I've seen some spreadsheet-like UIs, but none I looked forward to using - though Google Sheets isn't horrible. (Probably, if it could be done easily, everyone would be using it.)

This is exactly what AG Grid Server Side Row Model is designed for. I would recommend anyone considering putting a grid on a large dataset to at least understand what AG Grid has to offer here. The trick is only load what the user needs to see, AG Grid achieves this with Grouping and lazy loading children as the user expands, and also infinite scrolling to present more rows as the user scrolls down.

Data tables is the one ui component that I wish there was a native version of in the HTML spec.

this article is so pedestrian I'm puzzled why it's featured so highly here.

I was equally scrolling for the big reveal only to leave very unsatisfied

Yes, articles like it (and some much more pedestrian) have made the front page in the last year or so, and I'm not sure why. Possibly I just didn't notice them before. Or maybe I'm getting smarter! :D

Maybe it’s people like me who fled Reddit. I’m sorry!

If you're serious, read the Guidelines (link at bottom of the page) for submissions. HN is for intellectual curiousity - table formatting basics don't qualify, IMHO, but I speak only for me.

I found the link interesting, that is why I clicked on it and upvoted it. Perhaps some of us here are interested in good UX design, particularly for hard-to-get-right areas like dense data tables. What satisfies your intellectual curiosity may not be the same for someone else.

Ironically, everyone in this thread complaining about the submission not being appropriate for HN are violating the guidelines.

It's not that UX design is uninteresting, but that the OP was very basic and thus not creative, expert or insightful (IMHO).

> What satisfies your intellectual curiosity may not be the same for someone else.

I pretty clearly said the same in GP.

Well you would be surprised. I had to explain to several people why it is important to align numbers to the right, and why ordering of the columns can't be done unless you understand how customer works.

I shared his original article with bunch of UX people and most of them had said they learned a lot from it.

Reply on news.ycombinator.com