Cybiote

Though humans do often confabulate, they can also say, "I don't know", "what is that", "wait a minute", "there's something fishy about this", "huh?". Sometimes they can invent a new label or phrase on the spot that captures component attributes.

It seems you've written c² in units of Wh/kg? But the fact that per arbitrary mass, you can theoretically get more energetic reactions than by fission or even fusion doesn't mean comparing the productivity of fission to oil is silly. The issue is realistic ways of efficiently liberating and harnessing energy from matter.

One better than fusion not impossible option is carefully feeding small black holes and extracting their huge output of energy. The feedstock for the black hole might even be lifted from the sun. In the meantime, fission is a productive and practical option to add to the energy mix that will more than serve until we've worked out fusion.

One place I think I agree with you is that phrasing it as energy density, rather than focusing on efficiency of a given unit of mass or fuel at yielding energy can be a bit obscuring. That is, it's better to focus on sustainability of fuel source. Energy density (J/m^3 and J/kg, specific energy) are units more appropriate for talking about energy storage and bombs.

> But you have to try and explain the magic.

You don't always have to. Vernor Vinge explains it as, you are allowed one central conceit which frees you to deeply explore some core theme. That single wild assumption has the interesting property of being both inconsequential to the themes you wish to explore and the single thing that makes it all work. A lot of good hard science fiction has this property: there is internal consistency and broad agreement with real world physics but with at most a single unexamined wild assumption from which the consistently handled deviations flow from. The Zones in his Zones of thought or the Bobbles in Across Real-time are examples of this principle applied for excellent effect. Orthogonal by Egan is an interesting one when viewed in this lens, what with it being physically unrealistic but mathematically driven.

A lot of would be hard scifi authors, in the hopes of increased realism, mistakenly add too much unnecessary detail to the detriment of their story or core big idea. On the other hand, if you manage your central conceit well, you might sometimes be able to enlist the imagination of your readers (some of whom will have more technical expertise in the relevant area) to fill in the blanks.

This is one of the themes brilliantly explored (in my opinion), in the (largely hard) scifi book "Lady of Mazes" by Karl Schroeder.

Oh okay, then in that case we don't disagree and are talking about different things. My issue is the few shot learning done by a model which is performing gradient updates should be distinguished from whatever GPT-3 is doing when it's continuing context patterns.

Let me preface this by saying I generally agree with what you're saying. My focus below is on the delta.

> It is true that this model requires a large amount of unlabeled data, in addition to a small amount of labeled data, but gathering unlabeled data is often easy.

Gathering unlabeled data is easier than labeled but can still be challenging. You'll often require careful thought in assembling a distribution of examples. Being able to skip that step yields a significant savings even if not as much as that gained from going from labeled to largely unlabeled data.

> I'd say this approach actually has a serious advantage compared to GPT3, which is locked inside the walls of OpenAI, can only be used with their permission, and is too big for most people to use (let alone train) anyway.

I fully agree and said as much too.

> The cost and effort to use this approach on large real world problems is probably less than using GPT3.

I'd say it depends. Most of the effort with GPT3 will involve edge cases. Having a system in front to handle these might eat into labor savings but you could still end up net positive. It's difficult to say without real world data, you might be correct.

> That is, it may be a more efficient way to train new models from scratch for tasks for which few labeled examples are available.

You're right in general, I think. But it's still worth pointing out GPT3's advantage. It combines a lot of general capabilities, which together with its generative ability and flexibility to input means the level of expertise required to get something useful will be much lower compared to this semi-supervised learning approach. And there are some capabilities it's displayed, one example of many being discussing, querying, pattern matching on computer code that seem hard to replicate with this method.

> What is this sentence supposed to convey?

There are examples where it is able to recognize and continue patterns in strings which if manually generated, would have required a FSM. In fact, some of the more impressive examples would require a stack of some sort so I thought I was rather underselling its capabilities in that arena.

> as GPT isn't a "state machine" as the latent space is continuous and not finite.

Technically speaking, that is impossible since these models leverage floating point numbers and are limited in memory to whatever hidden and self-attention layers.

Practically speaking, in order to generate strings based on patterns as mentioned prior, there must be abstract states which correspond to states and state changes such that thinking in terms of at least state machines is useful.

Studying LMs in terms of automata is not strange, there have been papers which do this for specific trained RNNs (such as https://arxiv.org/abs/1711.09576). I contend GPT-3 is capable, to a certain extent, of generating these dynamically at inference time.

As far as I know this way of extracting what LMs are doing hasn't been done for Transformers but you can also frame Transformers in term of RNNs so there's no reason why such methods wouldn't readily apply to them too.

> Moreover, there is nothing that makes GPT-3 "not capable of learning."

I specifically addressed that: to count as learning, without diluting the utility of the term, it has to be capable of remembering. Without permanent changes to its weights, the use of the term learning stretches the word beyond utility.

Unfortunately, your correction is still misleading because it fails to capture what really sets GPT-3 apart. Ironically, GPT-3's limitation also highlights its strength. As it is not capable of learning (few shot or otherwise) in the strict sense of permanently changing its parameters based on examples, all its demonstrated capabilities are completely at inference time. It somehow configures itself at inference time so that state machines which produce plausible continuations of whatever pattern it was fed, are most probably generated. This means that whenever it succeeds, it is much more flexible in how it produces its responses. It generalizes on and continues those implicit patterns in the provided input.

This paper however, is not replicating that flexibility. Their proposed model is much closer to expectation maximization than it is an instance of what GPT3 does. The novelty of their work, what makes it genuinely useful, is they provide a practical and fairly general way to leverage pre-trained language models to produce classifiers for specific tasks using a very small amount of labeled data. Requiring less effort compared to what would go into fine-tuning. This approach to distillation is an instance of https://en.wikipedia.org/wiki/Semi-supervised_learning.

Compared to GPT-3, this approach remains at a severe disadvantage when amount of effort and time required to gather data and train something useful is accounted for. On the other hand, if you can fit your problem into the format proposed by the paper, you will likely have more control on the final model's behavior using a small amount of labeled examples (for your specific task), at a significantly lower cost of computation at inference time. Focusing on parameters however, is not even wrong.

You raise good points and I suppose valid differences in focus will alter one's notion of nearness. Here's why I made the claim I did:

> Rust that came from ML and that aren’t features of Haskell

> As I understand the history of algebraic data types and pattern matching, they first appeared in Hope, and from there they were added to ML, and later adopted by Hope’s successors

They appeared first in a practical form in Hope but I'm taken to understand that these ideas could already be found in Programs and their Proofs: An Algebraic Approach and the core ideas are clearly present in this 1968 paper by Burstall: http://www.cse.chalmers.se/edu/year/2010/course/DAT140_Types...

Creators of HOPE, VAX ML (which also had limited patterns and case analysis) and LCF/ML all influenced each other and played prominent roles in the design Standard ML, which would in turn strongly influence future functional languages, including Haskell and Ocaml. It's tricky to untangle the history but I don't think it's controversial to say SML popularized functional programming as prototypically understood today (before Haskell evangelized it further).

> Rust’s two most prominent type system features are lifetimes and traits

Other than the initial prototype being written in Ocaml, I'd say Rust's pragmatic feel: eager evaluation, immutability by default but a hatch to mutability and the implementation of the features listed in the posted article all make it feel more like an ML. I agree, though, that this is a subjective take.

I enjoyed reading the perspective given in this essay and agree Haskell's been (and continues to be) an influential language. So take this criticism as maybe my being overly particular about the meaning of descendant.

Idris as a descendant of Haskell I have no argument with.

Rust however, is more correctly viewed as (primarily) a descendant of ML languages, which all possess the properties the articles lists as derived from Haskell. Perhaps it is more correct to call Haskell and Rust cousins, both deriving from the same lineage of languages.

I would also not call Julia a child of Haskell. As it is more correctly viewed as an interpretation of Matlab as a Lisp, I'd therefore call Julia a descendant of Lisp. That Haskell single handedly popularized the utility of category theory in computing and that a category theory library is written in Julia does not make Julia a child of Haskell any more than it would have made Ocaml a child of Haskell (there used to be a category theory library written in Ocaml).

The popularization of functional programming, Domain specific languages as algebras, reactive (not FRP), property based testing, parser combinators, model view update for UIs, design principles secretly based on monads are all but a few of the ways Haskell has influenced programming. Which is the core point of the essay and with which I am in full agreement.

This thread somewhat has the air of an eulogy but it goes without saying that just because something is not the most popular in fashion does not mean it is dying.

You're completely correct but unfortunately, this doesn't matter in practice. It's true that thanks to formats like PDF/UA, PDFs can have decent support for accessibility features. Problem is, no one uses them. Even the barest minimum for accessibility provided by older formats like PDF/A, PDF/A-1a are rarely used. Heck, just something basic like correct meta-data is already asking for too much.

This means getting text out of PDFs requires rather sophisticated computational geometry and machine learning algorithms and is an active area of research. And yet, even after all that, it will always be the case that a fair few words end up mangled because trying to infer words, word order, sentences and paragraphs from glyph locations and size is currently not feasible in general.

Even if better authoring tools were to be released, it would still take a long time for these tools to percolate and then for the bulk of encountered material to have good accessibility.

This recent hn post is relevant: https://umij.wordpress.com/2016/08/11/the-sad-state-of-pdf-a...

A monoid is a mathematical structure which captures associativity (a+(b+c) = (a+b)+c) and has an identity notion (a + 0 = 0 + a = a).

They're useful because knowing them allows for identifying and mechanically applying transformations to better leverage parallelism and easily implement incremental computations†.

I highly recommend this article and the section on monoid homomorphisms in particular, which covers why you should care about monoids and gives a practical example: https://fsharpforfunandprofit.com/posts/monoids-part2/#monoi...

If you've been programming for a while, I can guarantee you already intuitively understand the concept. Giving it a name allows you to automatically recognize and notice that you've already solved this problem before in another guise.

Alternatively, having named the concept, it's now chunked into your toolkit for use whenever applicable. Structuring your solution so you know it's a monoid allows leveraging all those benefits almost for free.

†With divide and conquer, add memoization and pay attention to the order of subproblems such that you can reuse earlier work, and if you can, we're nearly at dynamic programming. We can specify a bit more structure, to get semirings and collapse a whole heap of disparate seeming machine learning algorithms. But now I have gone too far afield.

There is the Curry Howard Lambek Correspondence (or should I say: the Types, Logic, Cartesian Closed Category Correspondence). Curry Howard in particular, says the act of providing a term for a type is the same as providing the proof to a theorem (modulo a few details). Note that this isn't the same as saying writing a concrete computer program and proving a theorem in a type theory are the same type of activity.

Numerical methods and algorithms are fields of math as old as geometry, especially if we focus on the Babylonian or Chinese styles.

Hermann Grassmann sought to formalize arithmetic, not wishing to assume them as granted. In doing this, he also connects recursion, induction and the natural numbers (he would have known of recursion from its early application in the theory of combinatorics). Peano, Dedekind, Frege, Zermelo and many others would also work on the foundations and axiomatization of mathematics and deduction. Computing began as a side-effect of attempts to formalize just how far such an approach could be taken. The Turing Machine arose to tackle Hilbert's Entscheidungsproblem. The lambda calculus as an approach to the foundation of mathematics. Functional programming languages were originally part of tools meant to study formal mathematical objects while Logic programming sought to apply ideas from formal logic and the axiomatization of mathematics to automatically search for programs.

Dedekind said: "In speaking of arithmetic (algebra, analysis) as a part of logic I mean to imply that I consider the number-concept entirely independent of notions or intuitions of space and time, that I consider it an immediate result from the laws of thought."

What we find is computing reaches right to the foundations of mathematics. Whenever we try to systemize thought, we end up with ideas which seeming inevitably also lead to the foundation of computation.

My experience is that, compared to most functional languages, F# has a great IDE experience in Rider, Visual Studio and VS Code. But when compared to big industry focused languages, like C# and Java, tooling is middling at best.

Ironically, despite .NET core being an excellent runtime, it suffers stigma from being assumed to be too tied to Microsoft platforms. This means, depending on your angle, F# lacking specialized ties to the Windows specific tooling you mention need not be considered a disadvantage.

> Cutting out DeepSpeech seems sensible to me, it’s out of place in the general portfolio of products.

I disagree precisely because of the point you make later: "I’m somewhat concerned that Firefox will be irrelevant in fives years".

Functionality provided by deep learning is going to be an important component of many types of software interactions going forward. The logistics of this will be quite different from what we are used to in open source, with the need to fund and coordinate compute, collect and handle data being a more vital aspect compared to the past.

There are STT software, some mentioned in this thread, that match or are even better than DeepSpeech but none of them are as ergonomic. Accounting for the value of time, this means it will be more cost effective to outsource such capabilities to the cloud. Which comes with trade-offs that are difficult to appreciate in the short term: https://news.ycombinator.com/item?id=24236489

I'd say DeepSpeech fits in the mold of Mozilla as a company providing solutions to complicated software problems that are better at respecting the user and their privacy.

In the old days, the most accurate TTS and STT models were built into the OS. These days, you need to call into the cloud to get the best stuff. In [1], Internet Archive complains about the quality of their OCR software. It's not that OCR is so bad, it's that the best OCR is found on Google's and Microsoft's computers. It's possible to cobble something together using open source solutions like EasyOCR, Tesseract+OpenCV but that will only get you part of the way there. What makes the cloud offerings so good is they have enough resources to devote to pre-processing pipelines and architecture tweaks and settings better able to handle edge cases. Most of the mass resides in edge cases.

From my vantage, the future looks to be one of software as thin layers built atop APIs which call into programs running on the servers of a handful of companies. You might not think this a big deal but these software will be the ones scanning the environment, writing the emails, completing the thoughts and planning the calendars for the majority of humans.

[1] https://blog.archive.org/2020/08/21/can-you-help-us-make-the...

It still remains a risk well worth taking. With Mozilla, it is merely uncertain but for just about every other company, it is all but guaranteed. You can plainly see this philosophy in the industry's naming sense, where super-computers of yester-year are relegated to "edge" roles.

As of today, the open source and free software equivalents to machine learning and AI products are sorely lacking when compared to commercial offerings. Whether it is open-ended speech to text with good ergonomics, text to speech, intent recognition, speaker recognition, OCR for text, OCR in the wild, translation, object recognition, image segmentation, image to text or natural language processing, commercial offerings are leagues ahead of what free software can do.

If we look at one of the most impressive AI demonstrations in history, GPT-3, it is not apparent whether open source can even replicate it because with AI, unlike in the past, time and skill is no longer directly fungible with money. I would argue the concentration of such capabilities to Microsoft and Google servers is a threat to the ideals of free software as great as any it has seen before. Yet, relatively little attention is spent there because people are too focused on yesterday's problems.

This concentration is difficult to avoid because current algorithms require large amounts of data and computing ability, which only large corporations can marshal. Mozilla is far from perfect but despite their many stumbles, they're the only large organization seriously attempting to address this imbalance. As much as these algorithms are marketed as AI to users, ML is better thought of as libraries, in the line of ffmpeg, to programmers. Mozilla still do seem to care about creating a local-first offering. If everyone stops using them then what is gained exactly?

That's a great point. I also observed that and the fact ratings seem fairly volatile outside the top 10. I was only noting positional changes of the languages mentioned in the article over time.

Its main point of other reasons to to choose a language than popularity is well taken but I do not thinking making using of the TIOBE index as it does argues this well.

It was interesting to observe that since this was written, both F# (from 67 to 32) and Lisp (63 to 29) have risen dramatically on the TIOBE index. Haskell has also risen from 42 to 39, though not as big a gain as the other two. Haskell now ranks above Apex while PL/I has fallen below the top 50, where Erlang has remained. I must live in a bubble since I was surprised to see TypeScript and Julia rank so low.

It's amusing from a trivia point of view but I don't think such a ranking should figure in any serious decision on what language to use.

The case fatality rate is estimated at 2% - 3% but that is with medical care. Without ventilators and ICU care, CFR might rise to as high as 10%-15%. With its exponential growth rate, it'd be highly surprising for the virus to go unnoticed so long.

The article addresses that issue, its core thesis is that AI supports declarative programming. One of the author's main points is that a sufficiently intelligent compiler would rewrite code to be better optimized and have better computational complexity, eliminating low level C++ programming type jobs.

Category theory is to support the creation of specifications that are both easy to understand for a human who knows category theory and easy to optimize for the AI, compared to the original quick sort example.

The author also thinks many HTML and JS type jobs will also disappear. What I am skeptical of is that while Go, Chess and Jeopardy are challenging, they are closed domains. I think people underestimate just how much complexity building CRUD apps involves. Just like we underestimated how difficult walking to a cupboard to retrieve a mug would be for AI.

Yeah, the subtle difference is that discovering a geometric concept and being able to express it in a compact and flexible notation are not quite the same.

It's very easy to take for granted the sheer amount of innovation and ideas that went into being able to express that formula just because they are grade school topics today. The concept and acceptance of irrational numbers, the relationship between geometry and algebra, the use of symbols in algebra and even assigning symbols for special real numbers took centuries and a steady stream of creative leaps of genius.

An excellent and detailed explanation I highly recommend can be found here: https://hsm.stackexchange.com/a/1872.

These are known as https://en.wikipedia.org/wiki/Parallel_coordinates and are a pretty old technique. Although the wiki lists process control as one of the major users, their history has them as dormant until revived in their modern form in the 1960s, which doesn't quite match your timeline. Perhaps the discrepancy is resolved by whether parallel axes were used as a equation solving aid versus a visualization aid?

They're also not outdated, you can find them in many libraries, like highcharts here: https://www.highcharts.com/docs/chart-and-series-types/paral...

Excellent technical coverage is: Parallel Coordinates for Visualizing Multi-Dimensional Geometry, from 1987 at https://link.springer.com/chapter/10.1007/978-4-431-68057-4_...

Wikipedia links to this 1880 chart: https://www.davidrumsey.com/luna/servlet/detail/RUMSEY~8~1~3...

It's worth reading Vannevar Bush on the value of associative trails: https://en.wikipedia.org/wiki/As_We_May_Think#Concept_realiz...

Hi, can you point to the source for this? I failed to find any mention of it in the article. Or did you mean the two games annotated by Kramnik? Thanks.

Your arithmetic is not relevant to my point because as I said, when normalized to the same domain, it is valid.

> Which, once again, must imply that you have confidence in that evidence.

Nope, think of it as a bayesian update which induces a non-zero but low relative entropy, with respect to the new posterior distribution.

I do not believe this generalization is accurate. It wasn't me who said that but the person I quoted. F# is plenty functional but what it does lack are organizational constructs which reduce code duplication. In practice, these are not deal breakers for what you get in return (especially that you can work around them by leveraging more OOP features, which isn't necessarily a bad thing) but that as always, depends on preferences and priorities. Among functional languages, F# is pretty streamlined, believe it or not. F#'s pragmatism means the subset of Haskell that gets used in practice to keep things simple shares a fair amount of overlap with F#. Even more-so for Ocaml.

But, the person might have meant something like, if you already know Ocaml, you might complain about the lack of functors or polymorphic variants. Similarly, if you already know Haskell you might complain about the lack of higher-kinded types or GADTs. On the other hand, F# offers its own features (clean implementations of active patterns, computation expressions, type providers, async, multicore) and advantages, mostly from being able to leverage the .NET ecosystem (which also explains why those specific features are missing).

It does not. Because, since both systems in comparison are in violation, when normalized to just this setting, the comparisons are meaningful. That is, while microbenchmarks are not something that should replace task relevant testing, they do have utility as a coarse indicator. The information should be taken with a large helping of uncertainty but it also points in the general correct direction in terms of relative ordering of the compared.

The reality is often that other things will dominate. Things such as computational complexity, appropriateness and optimizations of data structures in use, I/O bounds, cache locality and specific details of the problem that will tend to reduce and not magnify the differences between languages near each other in a relative ordering, when things are done properly. Or slow things majorly when things are not done properly. This holds especially if idiomatic code is not anymore expensive to write in any of the compared languages, as is the case here.

Certainly, not every language will fit everyone's preferred style of thinking and approach to breaking down problems. This is why I think a diversity in available options is so useful.

> stop the world GC (my car ran you over because of a random unpredictable pause!)

Given a certain baseline of available resources and for the places where GC issues can be problematic (regimes I do not believe common) unpredictability of GC is the main issue, which can be readily addressed. You can for example, allocate and then manually manage or opt for a specialized JVM. If startup-time is an issue there are native compilation options. Or just not use the JVM, that's an option too.

I'll also note that Automatic reference counting is no panacea either and Python is no better suited for the scenario you've given. Personally, I've not found this to be an issue given modern low pause concurrent GC but your mileage may vary.

> its bytecode is object-oriented

> impossible at the moment to do CUDA style GPU programming without horrible JNI calls

I don't think these are deal breakers. More easily writing kernels in a high level language is very much an open problem. Even Tensorflow faces this issue, with most workflows optimized for a handful of prewritten kernels. On the JVM there are options such as https://index.scala-lang.org/thoughtworksinc/compute.scala/b... or http://aparapi.com/ for GPU backed ND-arrays or JVM translation.

> F# has two big crippling factors: (1) the .NET at the end of its name and (2) functional programming, which is a huge barrier for 99% of developers (and it's not functional enough for the other 1%). F# is dead in the water--I mean that respectfully. To replace Python for deep learning, you need a language that other people will actually use, not should.

I don't think (1) is true or even if so, see its technical relevance. If you're doing numeric/array-based differentiable programming then you shouldn't have any problem with functional programming. I'd even argue functional programming makes things easier as you get many things free from existing combinators. Many concepts are naturally expressed with the features such languages tend to have.

I don't think Python needs replacement, I'd much more rather see interoperability and language agnosticism. I can tell you that for each of Julia, Haskell, Scala, Kotlin, Ocaml, Nim, F#, Rust and of course Swift, at least one fascinating machine learning library is being built. I think that's a great thing.

By bundling. I believe there are couple a options to reduce the resulting package size some. As with all things, it's a matter of priorities when weighing language and deployment options.