spi

The image in the main page doesn't inspire all that confidence... as it gets bits of Italian wrong:

- The chatbot ("Bianca") starts off with "grazie per chiedere!" which is a literal translation of "thanks for asking!" but not how you would say that in Italian. E.g. Google translating "thanks for asking!" to Italian gets it right at "grazie per avermelo chiesto!", which literally would be "thanks for having asked that!". Though really most often in Italian you'd just say "grazie!", for "thanks!".

- "voi siete" as plural or formal is not wrong but... weird. Using second person plural for formal form (which is how you'd do that in French), in Italian is quite rare. Currently it's only used in some regions or in very formal bureaucratic lingo. In most circumstances, you would say "lei è" (i.e. same as third person singular, feminine) - this has been the standard for a long time, Mussolini fought it back to introduce the "voi siete" again (which I guess it's why it's still there in formal lingo, which hasn't changed much since Fascism), but well, for most people that experiment ended 80 years ago?

- When the user asks "E "voi siete", "noi siamo", corretto?" (which is not great Italian, but that's fine, the user isn't supposed to be fluent; though it's weird to ask literally something which the bot stated exactly the line above), somehow the bot gets overly sheepish and apologises for an error where there was none, just to repeat the very same thing it said (because, well, indeed it was already correct).

I agree with your conclusions, but not necessarily with the reasons you present. I don't think it's _that_ easy for a current transformer to pass the information unaltered (i.e. to effectively replace softmax with 0).

In particular, I think the feedforward point you list in your "Second" is actually wrong. Replacing a softmax with 0, as the OP wants to do, is tantamount to passing the information unchanged, because the attention block is within a residual (skip) connection. If it's set to zero, the next output is identical to the previous layer output. There is no way to recover this effect with the feedforward layer.

The part that you can set V to zero is true, but somehow a different idea: the Q and K should be able to set to 0 if no token wants to be "close" to some other token, in some sense. But the V layer shouldn't "know" about this, because it can't look at other tokens. This is of course only how we think of transformers, which might or might not (more likely, the latter) be how it actually works. But nevertheless, having a 0 value coming out of the K.Q^T part only would be very meaningful.

Your "first" point is technically true (albeit logically false): if you have a sequence of length 32k, like GPT4-32k, and your softmax logits all predict the same value, the result will be an average of the V layer, divided by 32k, which is effectively close to zero. However, calibrating "exactly the same value" is extremely hard for a neural network, and there is no "default value" it can predict to make sure that's the case - even if you push all the values to one side, the result doesn't change, because softmax is translation invariant. Plus, if you have a short sentence, that's not true anymore. If you only have two tokens, one of them must be activated, or both with only a 0.5 factor. Surely if you have very few tokens there's much more contamination between Q, K, and V, so in that case V can indeed take a 0 value, but it's non-trivial and requires more layers.

All in all, adding that "+1" isn't quite meaningless, I think. Nevertheless, I believe it won't change much: these very big models have ways to get around any kind of smart small modification you do. If the intuition is very right, it might be that you can squeeze 1% out more accuracy in a handful of tests, after you carefully optimize all other parameters, which would be enough to get you a paper in a top conference. And it might also be implemented as a standard from them on (because, in this case, it basically doesn't cost any more computations, so it's "free"). But I would bet it won't be a major revolution.

That said, as you say, the only way to know would be to train a few models with this option and check the actual quality of them (certainly not GPT-style, nor GPT4-size, models, to begin with, but something quicker to train and easier to test in a fully automated way; old "boring" models like those in the BERT family would be a good point to start testing). But to do that effectively, you'd need somebody skilled in training this kind of models, with the cleaned data ready at hand, etc. (and a small compute budget, of course, but nothing revolutionary, a few thousand $ in GPU credits could be enough)

Yeah that is a big red flag - as the OP mentions, there is basically no way of making k=2 statistically different from k=1, that's why nobody uses it.

I suppose the authors just tried many different k and selected k=2 because it performed surprisingly well (likely due to the bug the OP found out). But if the results were significantly better than k=1 or k=3, it's a bit weird the authors never double checked why that was the case. I guess it can be one of those things you settle on early in the overall process with a few experiments, and just take for granted afterwards, never checking it again, but still, it sounds like something that should pop out at some point while writing the paper...?

Having a similar background (PhD in math, working in deep learning), so knowing nothing of the specifics of medicine, I think that, if anything, there the situation is probably _worse_. Results are less transparent, and you can't just read a paper and know if the results are made "look nicer", if you don't have the raw numbers of the experiments (which you almost never have). At least, in ML you typically can get an idea by reading the paper attentively, and if you're determinate enough, often you even have the source code to check.

I guess that's why in medicine (and some other sciences), meta-reviews are so important: they read dozens or hundreds of papers on the same topic, compare methodologies, and try to deduce more realistic confidence intervals to limit statistical noise. That's how we now know that smoking definitely leads to higher probability of cancer, and most likely that's the case with red meat (and even more likely with smoked or otherwise preserved meat). The risk increases are small enough that a single study can never have the statistical power to prove it. Otherwise, when the famously wrong paper linking vaccines and autism came out, all families of poor autistic children could sue a lot (it was later proved that that one paper was fabricated with malice, but that doesn't change anything, if it were a random error the effect would be the same)

This is why I share your concern for the billions of $ in sues due to this one paper. I have no idea who is abstractly "right" - it might be there was asbestos, or maybe not, and it might be that these talc products increased cancer likelihood, or maybe not. But health, and particularly cancer, is tricky: we all develop multiple cancers, sooner or later, unless we die before. Trying to pinpoint a specific cancer to a source is like fining the proverbial last straw that broke the camel's back - except that in this case it's not the sum of the straws that kill the camel, but rather one random straw across all its load.

That said, I don't want to use this argument to wave company responsibilities away - of course companies who knowingly neglected safety measures, or endangered patient safety, should be punished with large fines (and possibly forced to shut down, in particularly extreme cases). As a European, I'm just uncomfortable with the idea of settling this in spectacular "patient vs. company" trials, where everything is dictated by the ability of lawyers and the will of judges, with results rather randomly ranging from nothing to literally making you rich (and again, your particular cancer might or might not be caused by that... and you can't really know, in any way). With the small additional caveat that you must have survived that in the first place, in order for the company to actually be significantly punished.

Having every model re-trained in each language is a certain path towards having any non-English (or at most a couple of other languages from countries with big pockets, like Chinese) language model be always massively behind - the resources required to train a model are huge, you can't expect e.g. the Polish community (plus anyone else) to replicate every good English model that comes out. GPT4 is less capable in Polish than in English, but probably much more than any Polish-specific model ever trained - and I suspect the gap is bigger than that with the best non-GPT4 English model.

Furthermore, I think you are exaggerating the memory issue of multilingual models significantly. Especially for languages using the same (Latin) script, the additional characters to care about are very few. Also a significant part of the vocabulary and language fall into a few buckets, so training a joint model makes all the sense in the world - much like an Italian native speaker could likely study a scientific text in Spanish and understand its content, even without speaking the language.

The memory impact comes mostly from having bigger embedding layers that have to account for vocabulary in many languages (the most problematic case being Chinese and Japanese, with their huge set of tokens). But even there, the largest vocabularies in use are maybe of size 100k (vs. about 30k for English-only), with a hidden dimension of 4k that makes for a total of 400M parameters. It's a lot, but a drop in the ocean of 100B+ parameters (or 1T+ for GPT4) we're seeing today.

P.S. Answering to GP, I think the Pile is English only, though - or at least, models on HuggingFace trained on the Pile, like the various Pythia models, are tagged as English only.

Isn't that almost always the case, and in truth the beauty of automation? Replacing simple, repetitive tasks that, if you have to do that, you say "I wish I had a machine doing this for me"?

Of course, that might impact the number of available jobs and it's something that need to be taken under control.

But, if talking about writers, while AI won't realistically replace imaginative authors and investigative journalists (at least, certainly not in the short term), it will likely replace (some) copy editors and dollar-per-thousand-words "journalists" that have to fill tabloids and free subway "newspapers". While it might be a financial loss for the people involved, the society will hardly sink due to that. In fact, if I feel optimist, in theory it might reduce costs for publishing houses and newspapers, to invest more into quality (human-made) content.

I'm not sure we're looking at the same photos... for some of them, sure, you're right. But many are bad photos like those you mention, except more egregious (hence they deserve the place on that site). The "garden" in one picture is rather awful, but why put that plastic chair facing against the wall to make it creepy on top of that? Why take a picture half naked (or not half - thankfully we'll never know) in front of a mirror? Why the cheap Christmas tree? Why those two sad soft toys in the corner of an empty room? Why a mower in the living room? Why include an old man watching TV in your photo?

As for the points you say, I'm not into real estate, I think often it comes down to limitations from reality. You know, actual _people_ live in those apartments you're trying to have pictures of. Maybe they'll just deny you entry (it's their place, after all, they might be renting and thus not give a damn about you willing to sell the property), in which case all you have to show is pictures from the outside. Or they might only agree to send you pictures themselves, in which case blurry pictures is all you get.

Of course, in general it's mostly incompetence, but hey, if everybody were perfect at their job the world would look totally different, in more important sectors than real estate...

I am seeing a rather clear (if incomplete) parallelism with 3D television (which, oddly, I never see mentioned, even by detractors). What was it, 10 years ago? All TV producers pushed it hard (it was impossible to buy a high end TV without it). AAA movie producers pushed it hard (a lot of blockbuster movies were _designed_ to be 3D first).

Just like with VR, the technology wasn't new: I had watched a 3D movie some 30 years ago at a theme park, liked it, threw up and passed out afterwards, for the first and only time in my life - doing it after a rollercoaster was not a great idea. Just like with VR, it required you to put something on your eyes that people found awkward - and ultimately, this was possibly what made them unpopular, if you want to relax, you don't want to focus on your hardware.

A bit like VR, the early glasses were cheap and disposable (just pieces of red/blue plastic); the "pro" versions with active filtering were significantly more expensive and technological. But for all that, they didn't solve the main issue, i.e. they weren't less intrusive, if nothing else more so (heavier, requiring batteries, etc.). In this case, I'm not sure if the Apple vision is a step forward or backward - if I understand correctly it makes the experience "better" than other visors, but it is also bulkier.

A lot of optimism on this technology comes from saying "it is not yet mature, give it time, the first iPhone was very limited, etc.". But the main issues I see seem just too hard to surpass in a few technological iterations: - The experience is limited to the eyes. No matter how magic Apple is, I'm not ready to believe that in 5 years wearing something as simple as a goggle will give me tactile feelings or anything else. Though I'll admit, 3D vision might be enough. - Battery power won't evolve tenfolds. The first iPhone didn't have a shorter battery life than the current ones, and that's because research for better batteries and more efficient CPUs is already cutting-edge - a 2x improvement would be huge already. Everybody just assumes it will get better, but not everything does, or at least not always fast enough. - The main issue everybody notices is lack of apps and, in general, something to do with it. Here, people cite the iPhone as an example of thing that developed a lot of apps, which made all the difference between the first and later versions. This might happen thanks to the Apple Vision Pro, but I'm not ready to believe it without proof. Just a few years ago, Apple pushed hard, very hard, the touch bar on basically all of its users. Certainly more than half the developers (not just mac developers; all developers) had it integrated in their working machine, whether they wanted it or not. Yet, I only saw apps starting integrating functionalities there years later, and it was often half-baked things, and even when they were not I found myself never using them - the old hotkeys, or the F-keys, were easier to memorize, and faster. And Apple did roll back on that, now I have again physical buttons which I very happily use to change screen light or volume.

All in all, maybe this technology will fly, maybe it won't, but if we look at the past we shouldn't forget the large fraction of technologies that didn't make it, or which at most thrived as a niche sector unlikely to ever explode. Not everything that Apple makes needs to be the new iPhone - and why should that be a problem, really?

I'm pretty sure in GPT3.5+ models, this concept of attention holds no longer true. In [1] the author suggests they are using "Intra-tensor sparsity" from a 2021 paper from Google [2].

Details aside, math suggests they _must_ be using some sparse attention method: the memory used by attention is O(l^2 * d) (here O notation is a bit of overkill: it's exactly that number in 8 bit quantization or twice that in 16 bit floats). With l=32k, and d probably in the order of a few k (even "old" BERT model had it at 768, it only went up since), that would be of the order of a few TB. For (a piece of) _a single_ layer, they probably have dozens, if not hundreds, of it. The largest GPUs in commerce have 80GB of memory; there's no way they are really using that many GPUs for each single layer (even leaving aside the fact that runtime would probably be absolutely horrible).

Implicitly or explicitly, the transformer must learn to have both a "summarized" version of attention (that tells it where to focus on) and more "detailed" ones. It doesn't need to be specifically coded, if using sparse attention, the model might for example get different levels of attentions at different layers, somehow, but they probably have coded something a bit smarter than that.

[1] https://kir-gadjello.github.io/posts/gpt4-some-technical-hyp... [2] https://arxiv.org/abs/2111.12763

Looking here: https://huggingface.co/docs/transformers/perf_train_gpu_one#... It looks like the most standard optimizer (AdamW) uses a whopping 18 bits per parameter during training. Using bf16 should reduce that somehow, but it wasn't really considered in that section, I'm not sure if that part of the guide is a bit outdated (before A10 / A100 this wasn't an option) or if it still has some instability issues ("normal" float16 can't be used for training because multiplying gradients through the hundreds of layers you'd get 0 or infinity values that would kill your learning). You can switch to different optimizers (Adafactor) and modify a few other things, but that typically comes at the cost of either lower accuracy or slower training, or both.

For multiple GPUs there are quite a few ways to improve memory footprint and speed: https://huggingface.co/docs/transformers/perf_train_gpu_many Although I'm not sure if the implementations in HuggingFace are really on par with the SOTA methods (they shouldn't be far away in any case). I guess they should be at least on par, if not better, with whatever OpenAI used for GPT-3 back then, things evolving so quickly in this realm...

On the last point, I can only assume there are some hard thresholds which are difficult to overcome in order to add more memory, otherwise they would. Just an 80GB memory GPU was something unthinkable a dozen years ago, before the deep learning explosion around 2GB was the norm. A couple of years ago, when 16GB or 32GB was the best you'd get from Nvidia, AMD did come out with consumer grade GPUs having significant larger memory (maybe 48GB back then? I can't remember), which could have stirred the market a bit I guess, but it didn't pick up for deep learning (I suspect mostly due to a lack of the equivalent to cudnn / cuda, that makes it possible to "easily" build deep learning frameworks on top of the GPUs).

My take on this is, if there's a competitor who fights hard to regain market share, and bets big on offering more memory, and still the best it comes up with is just a couple of times more than what the others have, it must be not as easy as "let's stick another bank of memory here and sell it", or they would have...?

The "standard" machine for these things has 8x80GB = 640GB memory (p4de instances here: https://aws.amazon.com/ec2/instance-types/p4/), with _very_ fast connections between GPUs. This fits even a large model comfortably. Nowadays probably most training use half precision ("bf16", not exactly float16, but still 2 bytes per parameter). However during training you easily get a 10-20x factor between the number of parameters and the bytes of memory needed, due to additional things you have to store in memory (activations, gradients, etc.). So in practice the largest models (70-175B parameters) can't be trained even on one of these beefy machines. And even if you could, it would be awfully slow.

In practice, they typically use servers with clusters of these machines, up to about 1000 GPUs in total (so around 80TB of memory, give or take a few?). This allows even the biggest models to be trained on large batches of several hundreds, or even thousands, of elements (the total memory usage is _not_ proportional to the product of number of parameters and the batch size, but it does increase as a function of both of them, a term of which being indeed the product of the two). It makes for some very tricky engineering choices to make just the right data travel across connections, trying to avoid as much as possible that you have to sync large amount of data between different machines (so "chunking" things to stay on the 640GB range) with strategies such as ZeRO being published every now and then. Plus of course the practical effort to make physical connections as fast as possible...

To get an idea of how hard these things are, take a look at how long the list of names in the published paper about BLOOM language model is :-)

I'm pretty sure Google is already using plenty of "AI" behind these functionalities. Heck, image classification was the very beginning of the current "AI summer" a dozen years ago!

Not nearly everything in AI is ChatGPT, not quite all is NLP for that matter, there's still plenty of interesting things which are improving. Dropbox might well be pivoting to improve their e.g. photo/video offering with AI, but they might as well have started that 5 years ago.

The current generation of GPT-3, which started with text-davinci-003, was actually released on November 2022, not quite 3 years ago. I'm not even sure the model that was released 3 years ago is still available to test, but it was much less impressive than more recent models - I wouldn't be surprised if LLaMa were actually better.

Yeah but the Python code is so bad that it's easy to get a 10x speedup using only numpy, as well. The current code essentially does:

    import numpy as np

    n_sides = 30
    n_polygons = 10000

    class Polygon:
        def __init__(self, x, y):
            self.x = x
            self.y = y
            self.center = np.array([self.x, self.y]).mean(axis=1)


    def find_close_polygons(
        polygon_subset: List[Polygon], point: np.array, max_dist: float
    ) -> List[Polygon]:
        close_polygons = []
        for poly in polygon_subset:
            if np.linalg.norm(poly.center - point) < max_dist:
                close_polygons.append(poly)

        return close_polygons

    polygons = [Polygon(*np.random.rand(2, n_sides)) for _ in range(n_polygons)]
    point = np.array([0, 0])
    max_dist = 0.5

    %timeit find_close_polygons(polygons, point, max_dist)

(I've made up number of sides and number of polygons to get to the same order of magnitude of runtime; also I've pre-computed centers, as they are cached anyway in their code), which on my machine takes about 40ms to run. If we just change the function to:

    def find_close_polygons(
        polygon_subset: List[Polygon], point: np.array, max_dist: float
    ) -> List[Polygon]:
        centers = np.array([polygon.center for polygon in polygon_subset])
        mask = np.linalg.norm(centers - point[None], axis=1) < max_dist
        return [
            polygon
            for polygon, is_pass in zip(polygon_subset, mask)
            if is_pass
        ]

then the same computation takes 4ms on my machine.

Doing a Python loop of numpy operations is a _bad_ idea... The new code hardly even takes more space than the original one.

(as someone else mentioned in the comments, you can also directly use the sum of the squares rather than `np.linalg.norm` to avoid taking square roots and save a few microseconds more, but well, we're not in that level of optimization here)

I have not tried, but 96GB of GPU memory is plenty, for inference there should certainly be no issue. Their biggest model has 13B parameters, you should be able to run inference (float16) already with 32GB of memory.

With 96GB of memory you should also be able to fine-tune it (possibly some tricks like gradient accumulation and/or checkpointing might be needed), but you have to be ready for many days of computation...

opt-175B doesn't exist; the largest one is opt-66B. And, at least in the tests I've run (not with the biggest one, but only up to a dozen billion parameters), all the opt models severely underperform with respect to even much smaller models. To the point that the launch of OPT (before BLOOM) was literally advertised as "the biggest OpenSource language model released to date", because they couldn't push on much else.

BLOOM goes indeed up to 175B parameters, and is certainly better than OPT. However, at least in my specific tests, it's still significantly inferior to OpenAI models, and actually on par with a few smaller models. There's also a "newer" fine-tuned model, called BLOOMZ, but at least in my tests it's even worse. Of course, that depends a lot on what you ask the model to do...

If LLAMA can indeed match OpenAI products, and do so with much fewer parameters, then it would be really great, and I'd really like to test it. However, even if the weights are now in the wild, using them would be clearly against the user agreement, and there's no way I'm going to do that in my work time :-) so let's hope Meta will come to sense and release them with a more friendly set of terms...

> Later decoder-only Transformer was shown to achieve great performance in language modeling tasks, like in GPT and BERT.

Actually, BERT is an encoder-only architecture, not decoder-only. Aside from trying to solve the same problem, GPT and BERT are quite different. This kind of confusion on now "classic" transformer models makes me kind of dubitative that the more recent and exotic ones are described very accurately...

(Clicking on the link with more details on BERT actually doesn't dispel much of the confusion; it stresses the fact that unlike GPT it's bidirectional, and indeed bidirectional is the "B" in BERT, but that's quite a disingenuous choice of terms itself - it's not "bidirectional" as in Bi-LSTM, that go left-to-right and right-to-left separately, it does the whole sequence at once; that was the real innovation of BERT).

Scrolling down to Transformer-XL starts talking about segments, from the context I _think_ it means that the input text is split into segments that are dealt with separately to cut down on the O(N^2) dependency of the transformer, but I would have assumed this kind of information to be written in a survey article.

IMHO, review articles are really great and useful, because they allow to cut through the BS that every paper has to add to get published, unify notations, and summarize the main points clearly. This article does a commendable job on the second point and, partly, on the first, but sadly lacks the third. Given the enormous task that it certainly was to compile this list, it would probably have profited from treating fewer models but putting things a bit more into perspective...

There's not much of an alternative: even if compute power gets extraordinarily cheap / models are very optimized, either you have a very, very large hard drive, or you have to use the internet for that. You just can't hope to have a model that is trained to know everything in a smallish file, the weights need to be at least as heavy as an ideally compressed version of all the things it knows (which is of course much less than the huge amount of data it is trained on, mainly due to redundancy, but still a lot)

Of course, right now you also need at least 8 super-expensive A100 GPUs and not just your laptop CPU, but maybe that's going to change eventually.

What do you mean by "small players have no chance"? OpenAI was founded in 2015, it used to be a "small player" which just got things right and grew with it - we're not talking of Google or Facebook investing a chunk of their billions cash. In Germany, AlephAlpha has built their own supercomputer and are training similar sized models. It's expensive for sure, but well in the possibilities of startups. In France researchers trained the similarly sized BLOOM model https://huggingface.co/bigscience/bloom. They claim it cost between $2 and $4 millions.

Sure, a single researcher can't replicate this at their university, but even though OpenAI likes to publish it this way, we're not really talking about research here. Research was inventing the transformer architecture, this is just making it bigger by (very smart) engineering choices. It's something companies should do (and are doing), not researchers.

I don't think that figure is correct, you need a "good heap" of GPUs, not just anything... in particular, even just to run inference, you need at least 400 GB of GPU memory, not just RAM. You can't just plug a dozen "cheap" GPUs and call it a day, because if I remember correctly consumer GPUs have at most 32GB of RAM each. Hence you'd need at least 12 of those top-tier GPUs (which certainly don't come at $500 a piece). Probably more, because you can't trivially split weights across GPUs so perfectly (you probably have to put an integer number of layers on each GPU).

In practice these models are typically run using top-tier A100 GPUs, which apparently is the cheapest thing you can do at scale: https://forum.effectivealtruism.org/posts/foptmf8C25TzJuit6/.... It looks like you can get away with just $10/hour, but I'm not sure I believe it. In one hour you can roughly generate 6 million English words this way, that's quite cheap.

But if you want to own the full hardware, then it's quite more expensive. You need 8 of those A100 GPUs, which come at $32k a piece, so you're in the ballpark of > $300k to build the server you need. Then there's of course running costs, these GPUs burn 250W a piece, plus the rest of the server we're at about 3kW power. That's not much, maybe $0.50/hr, plus maybe another $1/hr to cool the room it's in, depending on where it is (and the season, I guess in winter a fan might suffice, it's about as powerful as a couple small electric heaters). So with an upfront expense of > $300k, you're maybe down from $10/hr to $1.5/hr, saving something like $8.5/hr, which is $6k / month (minus the rent of whatever place you put the server in).

All in all, it's definitely feasible for a small start up as well, but not very much for an individual.

As far as I could tell, their OCR capabilities are pretty much the best you can get easily: https://cloud.google.com/vision/docs/ocr?hl=en. If you want free, Google Lens is a pretty neat piece of software you can get on your phone (not sure if Android only or also iPhone). They all use AI, although it's a different kind of AI.

For machine translation, they used to be clearly leading, then deepl arrived and I think they are roughly on par (or maybe deepl is better, but also has fewer languages). Do you have other companies in mind?

As mentioned above, in research they have some great results, translating those to products is not always straightforward of course - or even possible.

It is true, however, that TensorFlow is now considered outdated, with pretty much all the community switching to PyTorch (originally developed by Facebook, now handed over to the Linux foundation). But indeed these are not really products (i.e. they don't bring in the money they cost, by far), more like open source projects.

The closest open source contender is BLOOM: https://huggingface.co/bigscience/bloom. It has an almost identical architecture to GPT-3 (hence, to ChatGPT), and in particular the same number of parameters (175B). It was also trained on a similar amount of data, as far as we can know. Still, it's not like you can just "download it and run it", even just to _load_ the model into memory you need ~400GB of memory, to run it at any decent speed you need a lot of GPUs, so it's not really like consumer hardware. And the process to train it cost about 2 to 4 million $, so replicating it is definitely not for everybody. But also not just for "big corporations"...

Sure, our company deals with business documents and typically sells products higher in the stack. Our OCR offering is available to customers, but only if they buy a significantly larger pack of products that does information extraction. As a matter of fact, OCR results are included in there, so customers could (and very rarely do) buy the whole package for OCR purposes only. It's just not advertised/sold independently, so it doesn't make much sense for most customers to buy it for that purpose (unless they have really tiny volumes) because price-wise is much more expensive than alternative products only selling OCR.

Sorry for the late answer.

Short answer is: we can't and we don't. Most EULAs explicitly prevent users to benchmark results, and we don't want to incur into any such risk. Plus, since we develop a competing product, any "deep look" into the competition might be seen as reverse engineering it, and our company is very careful to avoid such problems.

Our company has dedicated teams to evaluate competition products, so we once asked them (a couple of years ago), and could only look at aggregated, anonymized results. But the patterns were very clear. Anecdotical experience (mostly coming from customers of ours who, themselves, compare our internal engine with alternatives) seemed to point to the fact that most of the competition have rather stable service, so quality likely didn't evolve much in the last two years, but we can't be sure of course.

We constantly track our own accuracy on internally developed benchmarks, because frankly the ones available online (also for research purposes) are very bad. But as said, we can only continuously test our own engine and open source ones (like Tesseract), for legal reasons.

Google OCR is definitely not the same as Tesseract, although it's true that Tesseract is maintained by Google. Google OCR has definitely much higher accuracy and is significantly faster (basically always taking 1s for inference, while Tesseract can easily take 10s or more for dense pages).

Source: I work in developing a competing OCR service and we keep an eye on competition (e.g. aside from Google, solutions by Azure, Amazon, Abbyy, Nuance, Cloudmersive, etc., as well as our internal product of course, which is not available externally), and they are (almost) all significantly better on Tesseract.

The only domain where Tesseract is competitive is for perfect "black text on white paper", it gives pretty poor performance when dealing with colored, distorted text, or even strong page structure effects (tables, etc.).

When I say "pretty poor" I mean: "with respect to the state-of-the-art", of course it's still enormously better than what was the state-of-the-art before deep learning came into the picture, roughly a decade ago. And for things like "search contents of a book" it's basically perfect already.

> towards non-technical folks trying to get their feet wet in software. Eg: Data scientists or business.

A bit tangential to the original post, but where does this belief that data scientists are non-technical folks? I am a data scientist myself, and in my view it's way more technical than most software development. Albeit I wouldn't still call neither data scientists nor software engineers "smarter" than the average.

Sure, if you want to train your bread and butter text classifier it just takes 10 lines of boilerplate code. But you don't need an AI-assisted tool for that - you just go to hugging face, copy paste those 10 lines, done, it's certainly faster than getting some AI-assisted code editor work for you.

For everything that is a bit more complicated, you need endless adjustments to your code, and it's quite unlikely more than a handful of people before you ever wrote the same code. It is, indeed, a somewhat painful and slow process (because just "testing" your code often takes minutes, if not hours, so finding out bugs becomes annoying). And a somewhat simple, AI-based, error highlight tool might be useful to weed out the most stupid ones and save some time.

But I will never trust something like copilot (or Kite, I guess, which I never tried) to write my code for me, as the challenging parts of the work involve long-term connection between different pieces of code (data loader, loss function, model function) that are written independently but must "cooperate" in a very non-trivial way. It is not at all uncommon that I make hours-long screen sharing calls with a colleague, discussing non-trivial mathematical computations, only to end up changing one or two lines of code that don't have an immediate link with the problem we are trying to solve.

This kind of things are notoriously hard for AI to grasp, so they can't do any decent job in writing that for me. Add on top that a lot of the code you find freely online is just ridiculously bad or broken, and you might only get unusable models generated by AI engines trained on those.

So, what kind of work are you referring to when talking about "data scientists or business" in your comment?

Not (at all) an expert, but I think what you are referring to is a harder problem: given an SHA1-hash, to find something that maps to it. Getting a collision is quite simpler: just create two inputs that map to the same thing, or (more realistically in attacks) you know the original content (and therefore its SHA1 hash) and you want to create another content that has the same SHA1. If, on top, you can also decide (to some margin) what to put in, you can add some "invisible" malevolent pieces. As far as I understand, this is much easier than reversing an arbitrary SHA1-key (with or without "chosen" additional content) without knowing one decrypted key, because the collisions that were found so far were always of rather big files sharing a vast majority of their contents. An ideal hashing would give totally different results changing just any byte in the contents, but apparently this is not the case for SHA1 (but again, I'm just a casual reader about these things!).

Deep learning libraries are free; GPT-3 is not a library, it's a pretrained model. It's offered (for a fee) through an API. The company offering it (as much as I dislike their naming OpenAI, while being all but open) spent considerable amount of money training it, and is paying more in hardware costs for each instance you send through their API.

Actually, even though training models is very expensive, most models are even available online for free! A large collection is on HuggingFace (including GPT-2, which is essentially a smaller GPT-3), and there are studies proving that the quality is essentially the same. You can literally just download and run them, pretty much like... a free library.

I saw this mention of self-driving cars a couple of times in these comments. I'm pretty sure you don't want your self driving technology to rely on 5G connection, otherwise a sudden unexpected drop in connection quality (an antenna down for maintenance, an unmapped tunnel, an exceptionally crowded area due to a demonstration with lots of people streaming video, etc.) would also reduce the quality of transmission and maybe make the connection just impossible. What would happen in that case?

I think a "local" network-free source of computing is unavoidable, and once it is available, I just don't see the point of setting up a complicated (and error-prone) infrastructure to make computations remotely when it is available and fall back on-the-fly when not. Given the amount of regulation this kind of things have to comply to, and how error-proof they must be, I wouldn't think anybody would be happy with this cascade approach...

Except that then anybody could literally just download it and start a competing service saving 2 years of development and hundreds of thousands of $ in compute costs over that time?