Perspective: Open Source WebAssembly-Powered BI
perspective.finos.org
perspective.finos.org
Sometimes that data is stored in a variety of places (excel files, a smattering of disconnected databases, sometimes enriching the data with more data pulled from an API...).
Extracting that data to a common location (historically called a data warehouse) tends to be the work of a data pipeline, and the tool used to join, display, filter, dynamically aggregate, and visualize the data has been historically categorized as a "Business Intelligence" tool. Normally these BI tools provide data caching and allow temporary integration of multiple data sources.
The intent is to make it easy for business users to explore, analyze, visualize, share, and present datasets or results.
The biggest examples off the top of my head would be PowerBI, Tableau, and Apache Superset, but "data reporting tool" is a competitive market with many entrants.
(1) optimized for bulk inserts and bulk read operations, not updates or deletes, nor for single row updates
(2) stores data from multiple sources
(3) often a "columnstore" columnar table format is used. This format usually compresses columns in a format that is trivial to decompress (so if a column only contains a handful of similar flags, the database will run-length-encode them to optimize a full column scan), and additionally if you limit the number of columns accessed you can substantially reduce the cost of the query
(4) the data warehouse can easily shard or partition data
If you want to technical file format examples, I suggest looking up how the Parquet file format compresses data, or how Microsoft SQL Server columnstore tables work under-the-hood.
> Sounds like essentially consolidating data in a database and visualizing it
Yes, and the database in which you consolidate the data is called a "data warehouse" In many cases the sources of data are ... not optimal for querying. (And I mean like "Spread across a dozen excel spreadsheets that people e-mail out new versions of when they make a change" by "not optimal")
Obviously jargon is a thing, we all deal with it, but when you put something up like this to a more general audience, you’ve got to give some thought to presentation, otherwise you end up with apparent word salad.
We shouldn't have to guess.
If there is ever a project that needed an electron app it’s this one. I’d love to have a Perspective.app that efficiently loads local data style files.
Electron is discouraged nowadays in favor of lightweight solutions like Sciter[0], Tauri[1] or even WebView2[2].
--
[0]: https://sciter.com/
[1]: https://tauri.app/
[2]: https://learn.microsoft.com/en-us/microsoft-edge/webview2/
WebView2 has further optimizations as claimed by Microsoft in their recent blog post about Microsoft Teams[0]: "The key benefits observed from the transition from Electron to WebView2 include reduced memory usage and a lowered disk footprint as resources are shared with Edge. Additionally, we have been able to take greater advantage of the native capabilities provided by WebView2 and ensure support for more up-to-date versions of Chromium (latest performance and security updates)."
So, I'd assume WebView2 is better than Electron.
> What you'd get is significantly smaller binary size but most people don't care much about that.
In the case of Sciter (quoted in my parent comment), Sciter is a bit different: The frontend is HTML, CSS and JS, but the app's local backend is whatever you want it to be: C#, C++, Python, Assembly, etc. That's why it's popular among performance-critical software such as malware scanners, where you need native performance but as well as a you know customers that like a nice UI.
Any kind of software could benefit from Sciter's approach, if just companies liked to drop a bit more of budget at their shipped products.
--
[0]: https://techcommunity.microsoft.com/t5/microsoft-teams-blog/...
The performance gains in rendering of Perspective are due to using canvas + Webassembly. This would be exactly the same in Tauri or Webview2 because it would run in the same browser rendering engines which execute the Canvas renderer and Webassembly like in a browser or in Electron.
> The key benefits observed from the transition from Electron to WebView2 include reduced memory usage and a lowered disk footprint as resources are shared with Edge.
That's exactly pretty much where the benefits end. The memory usage reduction is even debatable if you are not running Edge as your browser. See WebView2 as an installed Electron core that's pre-installed on Windows. Doesn't change rendering performance.
> In the case of Sciter (quoted in my parent comment), Sciter is a bit different: The frontend is HTML, CSS and JS, but the app's local backend is whatever you want it to be: C#, C++, Python, Assembly, etc. That's why it's popular among performance-critical software such as malware scanners, where you need native performance but as well as a you know customers that like a nice UI.
Yea that's why I didn't list Sciter in my reply. With Sciter you don't get the system webview - you get a custom minimalistic web html/css rendering engine that supports a limited subset of the HTML5/CSS3 spec. There's no WebGL and there's no Webassembly. So you'd have to bring your own libraries for that. Easy enough with Webassembly and they could even skip that part and go native but with Canvas and trying to run on all major platforms... it's just a lot of added work for minor gains.
I'm not against libraries like Tauri. Perspective could totally use that to make a desktop app. It would not have any advantage when it comes to rendering performance though.
The main area with the sparkgrid has good horizontal and vertical scrollbars, but they are nearly invisible, being dark grey on slightly lighter dark grey, and only visible if you hover in the area they scroll. Also the vertical scroll bar doesn't take you to the very bottom.
Then there's the top right area, which has a normal looking scrollbar (good), but once you scroll it changes the size of the entire right panel (weird), and if you drag it past the point where it has reached the bottom (really easy to do accidentally) it starts spasming uncontrollably. If you instead use the scroll wheel, you can scroll to the bottom, but if you scroll a bit further you are back on the top? None of the other scrollbars has either of those issues.
Then there's the bottom right, where the scrollbar is visible, takes you all the way to the bottom, and neither spasms nor resets, but just to be special in its own way it starts out as a big bar as if there was little content below the fold, but the further you scroll the smaller the bar gets. As you scroll up, it grows again. It's also stuttering a bit while making these size changes.
Another example besides scroll bars is the lack of tool tips on the UI (they do exist on data though!), which makes for a weird toolbar that moves its content all over the place on hover. And I guess you're supposed to guess that check boxes are delete buttons.
This doesn't take away from the app, it's still cool and useful, but little annoyances like that add up to make it feel weird and un-native (I guess foreign is the word).
I have a recent MacBook Pro and current generation iPhone and they're often indistinguishable from native apps for me - once they've loaded at least.
(my previous comment regarding this issue: https://news.ycombinator.com/item?id=35441337#35454074)
Edit: I see the other commenter and yes, the "treemap" is broken for me as well. Not something a noticed but you are correct.
Firefox 111.0.1 on macOS 12.5.1
As a Firefox user I definitely want to see better FF support here!
The first step is to get a good test suite. Currently our tests only run on Chrome. So after we get tests running on Firefox, we can move to better support it.
They look amazing but I'm a bit puzzled to why in 2023, a11y would not be a priority.
Even this landing page itself isn't keyboard friendly.
Some patterns to help you get started with ARIA: https://www.w3.org/WAI/ARIA/apg/patterns/
Designing for a11y: https://www.w3.org/WAI/tips/designing/
To the downvoters: Do you really think I should open issues for every critic I place to open source projects here and otherwise not mention anything? I also love and contribute to open-source but this feels like too demanding.
No doubt I have poor a11y on my own projects - but if you are going to launch into critique of the landing page, at least do it from a position of authority :)
TBH, I am not sure how a charting lib could effectively be accessible to any great extent. Its not covered in the resources you linked, and in my experience with other charting libs this seems to be commonplace. I would be interested if you have any insights or suggestions on that, as I use charts extensively on one of my projects.
I do it from a position of support, which I had the chance of developing by observing the users of the apps I created, and I made a giant software suit (more than 1000 views) pass an accessibility audit in my current company.
This is not just a charting library, see the examples down the page. Also, see: https://www.highcharts.com/blog/tutorials/best-chart-accessi...
It was in a thread specifically discussing browser support for the guys project where he stated he only has the ability to test on chrome/firefox on fedora: https://news.ycombinator.com/item?id=35448674
> a criteria that was set way later than I built that site
That criterion has been there since 1999.
> Also, see: https://www.highcharts.com/blog/tutorials/best-chart-accessi...
It is highcharts that I use extensively, and I completely disagree that a11y is a solved problem there. Ironically on the referenced page they make the same mistake as you regarding color shades.
I can do this all day. Stop being so hostile.
Apparently 2001 (and I stand corrected). But well, I'm learning and improving am I not? I was a worse web developer a decade ago than I am now, and I am proud of that.
> Stop being so hostile.
You are the one who analyzed my personal website and reported a single color contrast issue to dismiss my complaint, why am I hostile? Regardless of that discussion, isn't my point coming across: Do we need to be authority on issues we report?
I tried to help, linked some resources, why is that wrong? I feel rather you being hostile.
> Ironically on the referenced page they make the same mistake as you regarding color shades.
Okay now you get to criticize open source libraries?
Anyway, few years ago I created an SVG based charting library for internal use, and we did (and still have) users with sight impairments, and yes it is hard to make it right. NVDA is a pain to bring under control, at least :) And a colleague of mine said something I really like: You cannot make software completely accessible, even for a perfectly capable person with above-human IQ, no, but you can make it always more accessible. It's great that they are trying!
> I can do this all day
What you are doing is something I sincerely don't understand. Teaching me a lesson? Proving I was over the line? Reading my first comment, I did see it's a bit whiny, and in another branch I tried to apologize. Whatever it is, maybe we are misunderstanding each other and perhaps everything is coming across more and more hostile. I do apologize from you if it came across to you like that too.
> Apparently 2001
Nope, May 1999: https://www.w3.org/standards/history/WAI-WEBCONTENT
> I do apologize
No problem.
So apparently I didn't have to second-guess myself :)
> Nope, May 1999
That's not what you think it is: https://www.w3.org/TR/1999/WAI-WEBCONTENT-19990324/wai-pagea...
The one which the previous version of my site failed to validate is apparently published in 2008: https://www.w3.org/standards/history/WCAG20 and when I wrote those styles back in who knows when [edit: found it, 2014 https://web.archive.org/web/20141219164444/https://egeozcan.... ], I wasn't aware of it. I first said 2001 because I looked at the first draft.
edit: So my 2014 version of the site gets 65, and the current one gets 95 from lighthouse accessibility check with current standards.
Its exactly what I think it is: "Ensure that foreground and background color combinations provide sufficient contrast" (with reference to lighthouse.org for specifics about contrast).
They didn't specifically define that ratio in the doc until WCAG 2, but thats irrelevant really.
It is relevant. I didn't have any means to test it, and that's what I saw as sufficient back then. You never admit you were wrong, I assume?
edit: you edited your comment, so here is the lighthouse.org page: https://web.archive.org/web/20080211110529/http://www.lighth... There was no ratio, so it's just my gut feeling which WCAG 2 had apparently decided was not enough and I didn't know it back then - not that there were enough tooling.
You were right from the start, I had created a site 9 years ago that gets only 65 points in an accessibility check today! So I can't suggest anyone to focus on accessibility?!
You too have a nice day!
A specific report on the GitHub for issues you find would be much appreciated!
[1](https://www.w3.org/WAI/ARIA/apg/patterns/table/examples/tabl...)
In this example: https://perspective.finos.org/block/?example=superstore
I can't use the tree with the keyboard. From the patterns page I linked, you can see an implementation example for an accessible tree element.
I mean, I can continue. I'm not saying this to devalue your work, and I wish I had the time to go in depth and analyze all the components and give you more feedback, but I don't. So I humbly suggest you do a bit research about the topic and improve the product some.
Perhaps my original comment sounded too dismissive and therefore got many negative feedback, but I'm sincerely concerned. I wanted to hint at this as this is a tool that may be used by businesses and some people cannot do their job because of inaccessible tooling.
I do like what you are doing and the components do look cool - nothing to take away from that. Please take my comment as constructive as you can take it to be because I was trying to go for that, and perhaps my frustration in the last few years with this topic made me also whine a little bit :)
* Some of these (keyboard nav, focus) we test for and sounds like a specific issue with your browser/OS setup, which we'd very much like to fix. GitHub Issues allow us to ask these followup context questions.
* Some of these are vague, I'm not aware of any label+input in the component, e.g., nor how column selection is related to accessibility. The venue for this discovery is GitHub Issues - just because they are obvious to you does not make them obvious to us such that RTFM is a suitable correction, humbly or otherwise.
The project has relatively wide financial industry usage (and some of the code was originally developed at JPMC), and we're very interested in fixing real issues!
> Some of these (keyboard nav, focus) we test for and sounds like a specific issue with your browser/OS setup
It's probably not (tried on another computer and browser). Please open this page: https://perspective.finos.org/blocks/superstore/index.html Try to navigate with your keyboard. There are no outlines and the tree view is not possible to interact with.
> I'm not aware of any label+input in the component
The top and right toolbars here: https://perspective.finos.org/blocks/editable/index.html Are the not the part of the component? Perhaps I'm mistaken to think so. My apologies if they aren't.
> how column selection is related to accessibility
https://developer.mozilla.org/en-US/docs/Web/Accessibility/A...
The column headers aren't possible to interact with a keyboard anyway, and the screen reader doesn't announce anything to indicate that the table can be sorted.
In my current job, we paid a lot of money for an army of auditors to hunt these down for us, and I don't expect that you do something like that for an open-source product, but if you have corporate use (I mean if people are paying for this directly or indirectly, and I hope they do!), maybe it makes sense to go for one?