Kchat: Persistent group chat in <50 lines using kdb+/k/q web sockets, JavaScript | Hacker News Reader

q web sockets, JavaScript

github.com

155 pointstelotortium93 comments

N.B. https://code.kx.com/q/ref/dotz/ is doing the heavy lifting here.

This is such a helpful comment - what looks, to the uninitiated, like line noise actually starts to make some sense now!

Lol, yes! That language looks like it was optimized for code golfing or something.

It was optimized for interactive use by people that already know it.

Which is a nice target to have. Makes me want to learn the language.

Modern tech is so obsessed with high-turnover, sale-by-first-impression user growth toy software mindset, that it completely forgot about the idea of creating and optimizing tools for people who use those tools day in, day out.

No it didn't come on. Languages and libraries are still written by developers, who make tools based on their experiences and preferences using tools. A dense tersely named standard library is simply not pleasant to use for most people, and so most people won't develop one for themselves or others to use.

Something that's easy to understand on a first impression is easy to understand on every subsequent impression as well. At least in terms of languages and libs no one is optimizing for that first impression, it's just a byproduct of making it easy to use in all cases.

As a daily user of kdb+/q, I can say that at least for me, it's one of the "best feeling" and most ergonomic tools i've ever used.

I can actually query and understand tables in a very fast REPL in a way that would require 20-30 lines of SQL, nested subqueries, etc.

Even things like the way "left joins" work in KDB feels more natural than left joins in other SQL databases. https://code.kx.com/q/ref/lj/

Interesting. This syntax reminds me a bit of the R package data.table's terseness for joins:

  B[A, on="x"] # A left join B
  B[A] # same, assuming x is the key column on both, like K's lj [x;y]
  A[B, on=.(x, y<=foo)] # right non-equi join

That's because kdb tables are much more similar to R dataframes (ordered maps of vectors), rather than to relational algebra used in RDBMS/SQL (logically sets of rows). That's for in-memory only (RDB), ignoring historical data (HDB) and persistence since kdb is a real database.

  // kdb+/q
  t: ([] sym:`AAPL`IBM`GOOG; price: 139.09 124.23 948.82; vol: 123456 98765 54321)

  # R
  df <- data.frame(sym=c("AAPL", "IBM", "GOOG"), price=c(139.09, 124.23, 948.82), vol=c(123456, 98765, 54321))

I second that. It feels like an abacus after a while

have you ever wanted to hand write minified javascript? boy have i got the tool for you.

want an ip address as an integer?

    .z.a
    -1062731737i

what about TLS connection status?

    0N!h".z.e";
    `CIPHER`PROTOCOL!`AES128-GCM-SHA256`TLSV1.2

absolutely beautiful language

Pay no attention to that man behind the curtain!

Lol. Seriously though, that's how all of these kinds of $DO_SOPHISTICATED_THING in $ABSURDLY_FEW lines goes.

Your comment makes it sound like they’re using some task-specific library, but this looks like pretty generic standard library functions to me.

This reminds me that XMPP needs to make a comeback. Its push-oriented nature is efficient, was largely built in a time when we didn’t expect users to have unlimited data, and the eXtensible part has allowed it to modernize with multi-device encryption, loading chat history, etc.

Matrix is arguably the modern successor to XMPP.

It is a bit weird to frame one of two competing protocols as the successor of the other, when the only data point for this assertion is that one is older.

Matrix basically solves problems of XMPP:

- resilience against servers going down

- resilience against malicious federation

- Json instead of XML

- unified specification

- handling of E2EE with multiple devices and message history

- not need to rely on other servers for media access

Probably more, this was just off the top of my head

Matrix has its de facto centralized server in Matrix.org where almost all rooms will have its data sent through because of accounts joining rooms. The servers have been too heavy/expensive for self-hosting that many end up relying on the big servers to use (maybe the Rust server will help, Dendrite?), but the “resilience” mentioned is a big cause for this expense (similar issues with Mastodon). JSON isn’t superior to XML--it can be in some cases but it’s not without drawbacks; experimental EXI could help with the size. XEPs are an ad hoc unified spec, and the buy-into-or-not nature has some advantages where it’s easier to experiment and fallbacks are common even with fragmentation. OMEMO isn’t perfect, but it does cover multi-device encryption and carbons work to get message history.

I like Matrix. But I don’t think it’s the only or best solution despite all of the current hype; the XMPP solution should be looked at more often. XMPP offers additional flexibility as well not being limited to just a chat/communications platform where folks use it for presence, UnifiedPush, etc.

Its not to big to self host. It uses maybe a bit to much memory but anybody that has a real server (more then a Pi) can host it mostly fine. And many do.

Most federated system have one large server that has many users.

I hosted an ejabbard server on a 2014 smart phone for a while earlier this year. ejabbard & Prosody are reasonable to host using DDNS on a home network on just about any hardware which is much more accessible and decentralized than needing a “real server”.

There's many servers on matrix and even synapse isn't that heavy.

What bothers me the most about xmpp is that only the barebones stuff is in the main spec and everything else is in extensions which creates a complex maze of support by servers and clients alike.

I've used both but I like matrix the most. Its bridges also seem the most mature.

Being barebones is a part of the selling point for an eXtensible platform though. ejabberd came with really good defaults, and Prosody only a little painful (though most of that could be chalked up to DNS being a pain + Nginx behaving unexpectedly). If you need something more turnkey, Snikkit can be useful if you believe Docker is a good idea.

> Being barebones is a part of the selling point for an eXtensible platform though.

Understood but as a user I don't want core functionality like history to be hidden behind extensions which may or may not be supported or need additional components to client or server.

As a user (and home server operator) I found that Matrix+Element gives me a much better user experience with much less fuss (though I admit it was more than 5 years ago when I last ran ejabberd). And the bridges are amazing, just what I needed. Because Matrix is not very useful until everyone uses it, for the transition there are bridges.

I don't really care about the underlying architecture, I just want it to easily provide me a modern chat experience.

> core functionality like history

I won’t disagree with this. It could be a part of the main spec, but there’s some novelty knowing you could do a ‘complete’ server or client in a day. There are certainly different modern expectations than compared to IRC. Compared to the OP, barebones XMPP is much closer to a ‘simple chat’.

> bridges

XMPP + libpurple were trying to do similar things. For a hot minute everyone, even the like of Facebook Chat and whatever Google’s chat was named at the time, was using XMPP til they all decided to go for a proprietary option and users had the option to chat from anywhere with anyone. That doesn’t discount Matrix for doing something needed now. Proprietary chat lock-in runs in cycles. Here’s to hoping they don’t adopt & abandon Matrix.

> I won’t disagree with this. It could be a part of the main spec, but there’s some novelty knowing you could do a ‘complete’ server or client in a day. There are certainly different modern expectations than compared to IRC. Compared to the OP, barebones XMPP is much closer to a ‘simple chat’.

Novelty sure but it's not really complete in terms of what a user expects. I still use IRC a lot too but with a client that provides all the mod cons I need. Also, nobody writes a protocol handler from scratch anymore. People writing a matrix client will simply use libmatrix.

There's just the reality that "simple chat" akin to ICQ in its early days, is just not what people want anymore. It's not a benefit anymore IMO.

I think the XMPP community is really too focused on architectural cleanliness than on offering the user what they want. Which is their prerogative, sure. But it's mine to choose the chat network that suits me best. This is how I moved to Matrix.

> XMPP + libpurple were trying to do similar things. For a hot minute everyone, even the like of Facebook Chat and whatever Google’s chat was named at the time, was using XMPP til they all decided to go for a proprietary option and users had the option to chat from anywhere with anyone. That doesn’t discount Matrix for doing something needed now. Proprietary chat lock-in runs in cycles. Here’s to hoping they don’t adopt & abandon Matrix.

Yeah I know, they killed federation, I guess mainly because of marketing. If google chat users can reach facebook chat users there's no need to sign up for facebook chat, thus less data mining for facebook, and lower numbers in their growth hacking charts. Keeping users in the walled garden is a goal, not a side-effect.

This is also why I dislike Signal so much, they even ban other versions of their supposedly open client. I'll never adopt that as a whatsapp replacement because it's simply no better in terms of lock-in. I don't want yet another walled garden. Signal solves one or 2 problems of whatsapp but for me to adopt something as the be-all-end-all chat app it will have to solve all or at least most of them.

It doesn't really matter if the third party apps adopt Matrix and stop federating. It won't be any worse than it is now. I'll just bridge where needed.

I'm going to reply to some of your points because they're misguided:

> resilience against servers going down

The matrix model and the XMPP model are different. They solve orthogonal problems. Even the matrix people say it. That's like saying emails are better than github for collaboration because they are resilient against servers going down: it's true, but it's a different model. Not suitable for everyone.

> resilience against malicious federation

Wait for matrix to be taken up by big tech companies, you'll see spam flooding. I don't welcome it, but it's all about economics

> Json instead of XML

This is both subjective and shows the lack of understanding that extensibility is, and how json only does 1/10 of what proper XML can do.

> unified specification

Not all XMPP usages are the same. If you want messaging, you use the compliance suites, which are specs: https://xmpp.org/extensions/inbox/cs-2022.html#im

> handling of E2EE with multiple devices and message history

Has been working for years already: https://xmpp.org/extensions/xep-0384.html

> not need to rely on other servers for media access

Part of the standard installation of prosody and ejabberd, arguably two of the most known servers, for years. A few lines in the conf.

Matrix solves problems and creates others. Your information is outdated by almost a decade, which probably explains why you think matrix solves these.

> > Json instead of XML > > This is both subjective and shows the lack of understanding that extensibility is, and how json only does 1/10 of what proper XML can do.

Can you elaborate on this a bit more? Specifically what can XML do that JSON can't?

I know the feature set is technically larger (eg namespaces, schemas, etc), but AFAIK all those features aren't necessary for transporting messages.

I even consider some things in XML (or at least in common implementations) to be nuisances, like collapsing elements into properties, or some accepting either or both without a schema, and I'm never sure what the precedence rules are. Also, due to being a simpler spec, JSON is more widely available, and generally faster to parse too.

> Specifically what can XML do that JSON can't?

Put XML inside XML, which is most useful for wrapping. Extensibility is built-in, so any parser knows it has to deal with namespaces; it doesn't depend on the flavor of the library.

> I know the feature set is technically larger (eg namespaces, schemas, etc), but AFAIK all those features aren't necessary for transporting messages.

It's not necessary for transporting messages, but both protocols go way beyond messaging. How do you send an invite for a party and a fallback if it isn't understood by the other side ? In XMPP you use namespaces; in JSON you have to define yet another schema that will only be used by your app and everyone else must copy it.

> Also, due to being a simpler spec, JSON is more widely available, and generally faster to parse too.

I'd like to see actual numbers of real software before believing the parsing speed even matters. A matrix server has to sync with dozens, sometimes hundreds of servers, some of them being extremely slow. I don't think the speed of parsing changes anything.

Plus, the spec is simpler indeed, but it is too simple. Namespaces solve a real problem, it's a mistake to not have them.

None of that were problems that stopped XMPP from being popular. The ones you mentioned are problems nerds had with it, not normal users. Except "unified specification", that's a big one

It was simple: UI/UX experience:

* each server supported different set of features * each client supported different set of features * some of them (like gateways to other chats) were "generic" enough that power users were needed to set them up, even if they worked extremely well * Most chats were "text-mostly" with shitty html subset, which was just worse for power and normal users alone compared to now-near-standard "markdown + reactions"

Will you get your chat history on given combination of client and server ? Who the fuck knows

Will you have direct data transfer work ? Who the fuck knows

Can you share file to a group chat ? Who the fuck knows

"Oh look markdown is popular, lets IMPLEMENT SOMETHING ELSE THAT'S SIMILAR BUT INCOMPATIBLE" https://xmpp.org/extensions/xep-0393.html and other great XEPs... I swear whoever is running that is smoking crack

And the XEPs. Millions of them. There is no version of protocol to target by client or server, there is not even "level of features" (lets say "text only", "text + voice", "text + video conferencing"), no, pick and choose, and on both server and client site.

XMPP is absolute mess and mountain of design by committee

> There is no version of protocol to target by client or server, there is not even "level of features" (lets say "text only", "text + voice", "text + video conferencing"), no, pick and choose, and on both server and client site.

Yes, there is. But your entire comment is based on years-old view of what XMPP was

> And the XEPs. Millions of them.

482 of them. A XEP is how someone proposes a feature, and how the community collaborates around the protocol's development. Many XEPs never gain enough traction to be marked as "stable", and they get automatically deferred after a year.

This means out of the 482 XEPs, a much smaller number are actually required to implement working stuff in XMPP.

Hopefully this solves the common misconception, which you seem to have, that you have to implement every XEP. Some aren't even dealing with protocol changes, some are only informational, and some only pertain to client or server features.

False. We publish annual "compliance suites" which contains an array of features and two compliance levels for each ("core" and "advanced").

You can browse XMPP software by compliance levels here: https://xmpp.org/software/

You can find the 2022 compliance suite here: https://xmpp.org/extensions/xep-0459.html and the 2023 edition is due to be finalized soon.

> XMPP is absolute mess and mountain of design by committee

This is your subjective opinion. You're welcome to it, of course. But accept that I (and many others) do not necessarily share this opinion.

Matrix is not even close to the lightweight nature of xmpp.

But you can use your XMPP server to deliver notifications from Element via UnifiedPush because push is XMPP’s nature.

XMPP hasn't gone away, it's incredibly popular: WhatsApp is built upon it, with over a billion users.

can you federate with whatsapp ? No? Then it doesn't matter

Correct me if I'm wrong, but that's a business-guided choice made by whatsapp/meta and not a technical limitation of XMPP in any way.

That's still irrelevant if you can't use it as XMPP, it also doesn't work exactly as XMPP either.

XMPP would be viable if it had a better iOS client. Some of my contacts switched to iOS and every app we tried wasn't as good as "Conversations" on Android.

Is Monal not good? I don’t know since I don’t use or recommend Apple devices, but I saw they were putting in GSoC bids for features.

They did try using monal, the main issue with many of these clients was reliable push notifications. The other was it just not being as polished.

Perhaps the state of things are different. This was about a year and a half back.

XMPP needs to die and stay dead. Its extensible design ended up being flexible in all the wrong ways, at once horrendously complex to implement and yet impossible to make do the right thing in several key areas.

You could make similar arguments against just about any mature network protocol (see the recent HN post about regrets in TCP/IP's design). Luckily we have plenty of choice, and there are bridges between the open solutions.

The fact is that XMPP has been, and still is, happily deployed at scale in various scenarios. You're welcome to your opinions, but they are just that.

Disclaimer: I'm a full-time XMPP developer and executive director of the XMPP Standards Foundation.

> The fact is that XMPP has been, and still is, happily deployed at scale in various scenarios.

And this argument, also, can be applied to almost any mature technology. Remember, there are nuclear reactors in use today that are running happily and safely on PDP-11s programmed in PDP-11 assembly.

Granted, the comment you replied to was not particularly constructive, but it does mirror my own experiences and difficulties with XMPP in the past. This kind of handwavy dismissal does not exactly inspire confidence the situation has improved.

I get your point about PDP-11's, but I'm not sure how it applies, practically. My family communicate with each other via XMPP, but I doubt they would be happy to use a PDP-11 for their daily computing :)

My comment was not intended as an empty dismissal of all XMPP criticism. I'm happy to discuss and debate concrete criticisms of XMPP (contact info in my profile). But the parent post did not provide any such, only what sounds like a negative subjective experience whenever they last encountered XMPP. I don't think one person's opinion ought to decide if something should "die and stay dead", when there are so many projects and products built around it, and people using it daily to solve actual problems.

XMPP has evolved massively over the past 20 years. Is it perfect? No (show me something that is!). Do we continue to improve it? Constantly.

I've never seen a product/system that used XMPP as XMPP (i.e. in extensible form) beat one that didn't. Even systems that should have been a slam-dunk success, like Jabber (there was huge demand for an open-source messenger at the time). And every single big-name successful "built on XMPP" system has ended up switching to a closed model where they control both ends.

As an alternative to web sockets and JavaScript, can I offer my alternative (ok, it's not persistent chat) implemented with character devices as a Linux kernel module? I don't know if it still works, but it has the same name :)

https://github.com/brenns10/kchat

Previous thread from 2015: https://news.ycombinator.com/item?id=9120246

Strange set of comments in this thread.

Personally, I'd not heard of k or kdb before so this was an interesting read.

The code is from 9 years ago, and looks to me like a proof of concept. It's a kinda stepping stone between the 90s approaches * mentioned in sibling comments and the bloat of the modern web.

* A truly amazing and dreadful time. The commercial (and artistic/indie) web pushed developers to twist immature technologies and create perverse superpowers. Never-closing responses in hidden iframes. Constantly exploiting JS engine bugs as features. Disgusting fun.

What is kdb+/k/q? The link to kx.com didn't tell me much.

k and q are array languages in the APL tradition. APL is one the "ur programming languges". https://madhadron.com/programming/seven_ur_languages.html

Here's an open source implementation of k: https://codeberg.org/ngn/k/

kdb is a time-series database with deep integration to k. It is famously used in the financial industry.

It's a somewhat esoteric, and extremely terse, language used by some financial modeling firms. Influenced heavily by APL, but (thankfully!) without the non-ASCII characters.

https://en.wikipedia.org/wiki/K_(programming_language)

It's also used by every investment bank who's active in the algo/e-trading business.

Came here to ask the same. kx.com is just some marketing page. I still have no idea what kdb+/k/q is.

k: super terse language inspired by APL and lisp

q: slightly less-terse language written in k, with similar semantics. The vendor (kx) supports q, not k

kdb+: column-oriented database bundled with q

Skip all the marketing materials & go to the dev pages - https://code.kx.com/q/learn/install/

All good except the 50K a year license to run 'q'.

Oh, yikes. I was interested in following up reading about kdb and now I am not.

cost is for commercial license only

free for personal use for 12 months, then you click some buttons to renew another 12 months, etc https://kx.com/kdb-personal-edition-download/

Per core

Azure has prices that vary:

https://azuremarketplace.microsoft.com/en-us/marketplace/app...

That's for commercial though.

One can get a personal use license for hobbies as other commenters have pointed out.

People paying $MM/year for SaaS that does less, but have prettier websites

No doubt, kdb/q is offers a of bang for buck in a commercial scenario. We use it and for some stuff it's good value, aside from staffing costs.

Fun! Combine it with ios notifications for web apps and rename the title of this post to “<60 lines”.

https://developer.apple.com/documentation/usernotifications/...

It's sadly more than 10 lines, server and client side.

https://peter.hotgarba.ge

Something about the simplicity of the UI is so refreshing. Thanks for sharing.

This is a fun code golf challenge. 42 lines of JS with DriftDB[1]: https://codesandbox.io/s/distracted-wildflower-3jhtjq?file=/...

Each time it opens generates a new room, but you can copy/paste the URL of that room to access the same room (for up to 24 hours)

[1] https://github.com/drifting-in-space/driftdb

I forget about k/q every time until it shows up here. It seems like something fun to learn but that I'd have no practical use for personally.

https://spacetraders.io/ is API based. Considering that q is used for trading and analysis in the real world seems like it's a good way to go on an adventure.

This is pretty cool, thanks for sharing.

Is there a difference between .htm and .html? I never thought so, and noticing they chose .htm reenforces my belief.

For the most part the same, but .htm hails from win 3.x days because of 3 character file extension limitations, while *nix did not.

https://stackoverflow.com/a/138957

Such an incredible stack.

a more scalable and readable version of this today would prolly use something like a partykit.io

wow throwback chrome ui on windows 7 bringing a nice wave of nostalgia this morning

Why use Javascript?

It's fast.

Not really

Reminds me of an interview I conducted years ago. I asked the candidate what they would do to make their code run faster. They suggested using the V8 JavaScript engine so that it would be JIT compiled. We were talking about code written in C++, intended to be run on an embedded microcontroller...

Also reminds me of many years ago in college, when somebody gave a presentation claiming that java was unquestionably more efficient than C++. I had a lot of questions...

Only tangentially related, but this reminds me of a MCSE training I once attended in the 90s, where the trainer claimed Microsoft Passwords are so secure that they can not be cracked in a reasonable amount of time. I couldn't hide my amusment. Since this got him fired up, he went as far as saying "If you believe otherwise, find my password!" to which I replied "No need for that, already done." Thanks l0phtCrack, you made my day.

I met many Microsoft lawyers over the years who behaved the same way. The enthusiasm is effective perhaps, even when it's grounded in ignorance. MS is like a cult.

> more efficient

I was going to make a quip about efficiency being more than execution, with development efficiency also being important. But then with it being a comment about C++ vs Java I decide not to...

Why ask a question that will spawn a comment thread about language superiority without testing or acknowledge what the author provided?

Setup stress testing and worked flawlessly. Impressive work author.

Critique or honest question?

For the former, because the developer wanted to use Javascript.

Why use anything?

On the server? As an example of equivalent (to the k and q versions) code.

On the client? Because endlessly refreshing the page to refresh chat messages is an awful design that we had enough of already in the 90's.

Could be done with long polling or a never-closing HTTP response.

POST to an iframe to keep the page from reloading on send.

That's true, and those options are also awful options we used in the 90's and promptly abandoned once better options became available, though they wouldn't make that much difference here. The alternatives are all brittle and limited in different ways.

Well sort of. Server Sent Events is basically "official long polling" and it works very well. Way simpler than websockets in many cases. Websockets is obviously better here though.

The point isn't that the way the data is sent from the server matters much, but that using javascript to handle the retrieval is beneficial. The first Javascript enabled chats still relied on the same methods of retrieval for a long time, but still significantly improved usability.

Because its the most convenient language on the client there is.

This submission would not have made the front page unless I jump started some discussion. Mine was the first comment. It worked. Front page.

Years ago I had used k-based chat, with GUI, on Windows. Without Javascript. (Many years ago, k used to have a built-in GUI.)

"Why use Javascript" was a genuine question.

Pfft, chat.k is still just way too verbose compared to the good old

   # cat > a.out

Each chat participant opens 2 bash terminals:

Terminal #1

  $ touch chat.txt

  $ tail -f chat.txt
  # or
  $ watch chat.txt

Terminal #2

  $ echo "${USER}: hello" >> chat.txt
  $ echo "${USER}: hello, again" >> chat.txt
  $ echo "${USER}: bye!" >> chat.txt

Also there are builtin UNIX/Linux CLI commands to send messages to users.

Reply on news.ycombinator.com