gsjbjt

That's awesome - can you say anything about what datasets this was trained on? I assume something specifically conversational?

Site: https://dynalang.github.io/

Tweet: https://twitter.com/realJessyLin/status/1687515845281624064

The point is that LLMs can’t backtrack after deciding on a token. So the probability at least one token along a long generation will lead you down the wrong path does indeed increase as the sequence gets longer (especially since we typically sample from these things), whereas humans can plan their outputs in advance, revise/refine, etc.

What are the usual ways to discover discord servers (also through search, or some other means)?

Nice post! I work on NLP and I think a lot of ideas in this post resonate with what I find exciting about working on the intersection of language + the real world: large text datasets as sources of abundant prior knowledge about the world, structure of language ~ structure of concepts that matter to humans, etc.

I feel like the bottleneck is getting access to paired (language, other modality) data though (if your other modality isn't images). i.e. "bolt on generalization" is an intuitively appealing concept, but then it reduces to the hard problem of "how do I learn to ground language to e.g. my robot action space?" I haven't seen a robotics + language paper that actually grapples with the grounding problem / tries to think about how to scale the data collection process for language-conditioned robotics beyond annotating your own dataset as a proof-of-concept. Unlike language modeling / CLIP-type pretraining, it seems (fundamentally?) more difficult to find natural sources of supervision of (language, action). I'd be curious about your thoughts on this!

> When it comes to combining natural language with robots, the obvious take is to use it as an input-output modality for human-robot interaction. The robot would understand human language inputs and potentially converse with the human. But if you accept that “generalization is language”, then language models have a far bigger role to play than just being the “UX layer for robots”.

You should check out Jacob Andreas's work, if you haven't seen it already - esp. his stuff on learning from latent language (https://arxiv.org/abs/1711.00482).

> My theory is that taste is one quality that separates the academics from the business people. Academia doesn't necessitate a lot of taste. If you have it, great. If you don't have it, no big deal.

This might be true for academics in ancient Greek literature, but certainly isn't true for academics in CS nowadays. If you don't have good taste in research problems that are {important for downstream industry applications, scientifically interesting, tractable}, you won't get anything done, and you won't get published. If anything, the pressure for academics to develop good taste is stronger than for people designing product. You can have a product that provides just one utility that users desperately need and have terrible taste for all the other axes that make a product "good," and do just fine. Academic papers get judged (in peer review / traction after publication) purely against the taste and aesthetics of other people in your community.

This (made by a friend) is exactly that idea! https://www.anishathalye.com/2019/12/12/constraint-based-gra...

I’m working at Lilt (lilt.com) right now, and this is exactly why I’m excited about the company ;)

Yeah, it's an excellent course. That is typically the unfortunate downside of self-studying courses, but AFAIK the tests aren't that sophisticated - you can mostly test it yourself just by running the test suite over and over again, making sure that it _really_ works 100% of the time and not just 98% of the time (given the nondeterministic nature of distributed systems failures). This is basically what students would do before submitting their labs, and most people got a full score if they ran it enough times :P

David Spivak and Brendan Fong have been doing lots of work at MIT to evangelize applied category theory. See their recent books/classes:

Seven Sketches in Compositionality: An Invitation to Applied Category Theory https://arxiv.org/abs/1803.05316

Applied Category Theory mini-course: https://ocw.mit.edu/courses/mathematics/18-s097-applied-cate...

Programming with Categories mini-course (with Bartosz Milewski): http://brendanfong.com/programmingcats.html

Implementing it is the fun part! You might like going through the Raft lab in MIT's distributed systems course: https://pdos.csail.mit.edu/6.824/labs/lab-raft.html

+ Jon Gjengset's writeup on some of the "gotcha"s: https://thesquareplanet.com/blog/students-guide-to-raft/

Have you seen this? Not sure if it's helpful for your regulatory problem, perhaps you could reach out this guy:

https://onezero.medium.com/900-000-americans-tell-google-the...

Why don’t data versioning tools like git lfs do the job? Is it lack of awareness or is the problem more complex than that?

The difference between this and most other incubators (at Stanford, Harvard iLab, etc as mentioned below) is that it's very specifically geared towards hard-tech startups that other investors typically shy away from because of the long time to market.