YaLM-100B: Pretrained language model with 100B parameters
github.com
github.com
You look at OpenAI and how they don't release their models mainly because they fear "bad people" will use them for "bad stuff." This is the trend in the west. Technology is too powerful, we must control it! Russia is like... Hey, we are the bad guys you're talking about so who are we keeping this technology from? The west has bigger language models than we do, so who cares. Also their attitude to copyright and patents, etc. They don't care because that's not how their economy makes money. Cory Doctorow's end of general purpose computing[1] and locked down everything is very fast approaching. I'm glad the Russians are around and aren't very interested in that project.
[1]https://csclub.uwaterloo.ca/resources/tech-talks/cory-doctor...
This is an interesting take given the political censorship in Russia (for some ineffable reason much harsher now than it used to be 4 months ago) and cases like https://twitter.com/kevinrothrock/status/1510944781492531208.
[1]https://www.breitbart.com/tech/2021/08/19/google-whistleblow...
What I'm trying to say is that even if you believe that "was the 2020 US election stolen?" is worth debating, which it isn't, the yandex results are shit.
[1]https://www.breitbart.com/tech/2019/05/12/study-the-cnn-sear...
https://www.cjr.org/tow_center/google-news-algorithm.php
Btw Wikipedia’s first few sentences on Breitbart are not inspiring
> Its journalists are widely considered to be ideologically driven, and much of its content has been called misogynistic, xenophobic, and racist by liberals and traditional conservatives alike.[10] The site has published a number of conspiracy theories[11][12] and intentionally misleading stories.[13][14]
An absurd example of this fallacy would be, Wikipedia, which you cite, has articles that indicate tobacco smoking may cause disease. The nazis were also anti-smoking[1]. Therefore Wikipedia is Nazi propaganda and you should not trust anything on there.
[1]https://www.amazon.com/Nazi-War-Cancer-Robert-Proctor/dp/069...
If Breitbart pulled a Fox News and argued in court that their goal was to entertain and not inform, then you have a point! But until then, you have a terrible misunderstanding of journalistic integrity and what it means for a publisher to attach their name to a journalist's work.
Yandex: 9M results. The top two links are pretty suspect. Top link promotes Dinesh D'Souza's 2000 Mules documentary in the banner which at best is a one-sided take on election fraud. At worst, very misleading.
Secondly, I've yet to see any criticisms of 2000 mules data that aren't addressed by the stringency in the analysis they claim to have done.
I thought the information they presented was extremely valuable. Are we going to overturn an election at this point? No. But the vulnerabilities to the mail-in ballots were obvious, then lied about, then ignored, then clearly taken advantage of. I want to live in a democracy because voting matters. I especially don't want NPO's destroying this by taking advantage of flawed voting infrastructure.
If there are legitimate criticisms of the methods, the data, or anything else coming out of this film I expect a legitimate presentation that can break it down using the actual data in question. All I've seen so far is shilling and gas-lighting.
If you have any evidence, any evidence at all, of significant mail-in ballots fraud, then you should write it up and publish it; and even present it to the USDOJ, because you would have succeeded where Trump's highly-paid teams of lawyers failed.
If you don't have proof, then please STFU.
I personally believe (with obviously no proof) there was definitely fraud going on, on both sides. With such an archaic system and such a great economic and power incentive, you would be stupid not to do it. For sure mail in ballots made it even easier than in the past.
I've heard about Russian hacking the elections after Trump won for a good 2 years.
I didn't even hear about 2000 Mules until I heard some right wing commentator talk about it months after it was released.
Instead I'm shoved the latest Greta Thunberg song (You can shove your climate crisis up you **) 5 milliseconds after she sung it.
As an avid newspaper and news reader, the media bias shifted tremendously in the last 30 years.
https://www.fbi.gov/contact-us/field-offices/newyork/news/pr...
Or maybe that means Dinesh D’Souza is really the most qualified on election fraud since he has done it himself???
GTFO!
I sincerely have deep respect for Yandex for releasing this, and Baidu for some of the amazing research they've released over the years, but both are deeply deeply beholden to their local governments in a way that is incomparable to the relationship between Google and the US government.
Remember that the NSA was literally digging up and tapping fiber around Google data centers in a secret program called MUSCULAR because they didn't think Google was being cooperative enough when handing over data that they were requesting.
https://en.wikipedia.org/wiki/MUSCULAR_(surveillance_program...
FYI, they are Russian subject that follows ALL their censorship laws (and oh boy do they have a lot of it).
>> probably to perhaps to prevent western competitors from using them The irony here. All yandex products are exact copies of western, adjusted to local market.
I don't think you've lived in Russia if you need to ask that question. Breaking the law and getting away with it is a way of life in Russia, that goes for all institutions and social strata
By being on the internet. Russia has always been good at literally hitting you with a physical club if you're crazy enough to take a sign to the streets, but the Russian state doesn't understand the internet, or really anything that's sort of underground or intangible.
There's a reason the country is probably the world's largest place for all things piracy related, scihub and so on. It's not just laxer IP laws, it's also that tech in particular has always skirted all kinds of regulation freely, it's why the country has a relatively healthy tech industry despite at times suffocating regulation. The prevalence of cybercrime in the country is another example of it. Being censorious doesn't make you competent.
Even Telegram which was at some point supposedly blocked was still used by everyone, including funnily enough the foreign ministry itself. These things never really work in Russia.(https://www.reuters.com/article/us-russia-telegram-ban-idUSK...)
Yeah, the same Yandex Maps that stopped showing state borders recently, as they are now "more focused on natural objects", in their words.
Edit: in fact, I just checked, yandex maps still shows state borders.
I didn't see it as I was looking at France. It's weird, because at large scale there are no borders, at medium scale there is a weird mix of national and local borders (Western EU countries have state-level borders, RU/BY/UA/US/CN have local borders, ...).
And to take your example, I have to zoom quite close for BG/TR to switch from state-wide to local borders.
I take it that you're either too young or too untraveled to be aware of the level of state control of technology in "the east". Xerographic machines, mimeographs, and other similar reprographic devices used to be highly controlled machinery behind the Iron Curtain. This is absolutely not something exclusive or even peculiar to "the west".
Laughed out loud!
Anyway I hope somebody creates a playground with this so I can make a computer write a fan fiction about Kirby and Solid Snake trying to raise a human baby on a yacht in the Caspian Sea or whatever other thing people will actually use this for.
THESE are the important things to talk about when it comes to this topic.
I don't know if that's the right way to think about the open sourcing of large language models. I just think we really can't read too much into such releases regarding their motivation.
Who would have thought that one man's joke would become a reality?
I can't say I'm a big fan but my teams is great and I don't have time to look for a job right now.
- useless ai generated intro text
- ten products that actually are the best reviewed per category by users
- brief ai blurb on product
- 3 actual user reviews of the product
So even with the ai text there's still some benefit to the page.
I personally squinted hard when they said removing dropout improves training speed (which is in iterations per second), but said nothing about how it affects the performance (rate of mistakes in inference) of the trained model.
I know from practice that it takes a really really long time to train even a small nn (thousands of params) , so you'll need a lot more hardware to train one with billions... But, it's expensive to buy the hardware, not necessarily to use it. If you, for some reason, have a few hundred GPU lying around, it might be "cheap" to do the necessary training.
Now, that's not your point - cost != price. But, still...
- They were into Ethereum mining and quit.
- They've already built a cluster with them (e.g. in an academic setting).
- They live in a datacenter.
- They are a total psychopath.
But even assuming one magically has all those GPUs available and ready to train, I don't want to calculate the power cost of it anyway. Unless one has access to free or extremely cheap electricity it would still be very expensive.
Not to nitpick, but that is like saying that if you have a Lamborghini lying around, a Sunday trip in one is not so expensive.
- Generate vocabulary - e.g. for biking: handlebars, pedals, shifters, etc
- Generate translation exercises for given topic a learner wants to learn about - e.g. I raised the seat on my bike
- Generate questions for the user - e.g. What are the different types of biking?
- Provide more fluent ways to say things - I went on my bike to the store -> I rode my bike to the store
- Provide explanations of the difference in meaning between two words
And we have fine tuned smaller models to do other thing like grammar correction, exercise grading, and embedded search.
These models are going to completely change the field of education in my opinion.
1) https://squidgies.app - be kind it's still a bit alpha
"a 176 billion parameter transformer model that will be trained on roughly 300 billion words in 46 languages"
So anything smaller than that will become worthless. May be a factor, companies have a last chance to make a PR splash before it happens.
Read more about it: https://bigscience.huggingface.co/blog/model-training-launch...
https://bigscience.huggingface.co/blog/building-a-tb-scale-m...
But maybe your sentence was more about "after BigScience model, open-sourcing anything smaller than that will be useless" which isn't necessarily true either, because there is still room to improve parameter efficiency, i.e. smaller models with comparabale performances
Maybe the dead internet theory will really come true; at least, in some sense of it. https://www.theatlantic.com/technology/archive/2021/08/dead-...
Isn’t that already the case? Sure, it costs $60K, but that is accessible to a surprisingly large minority, considering the potency of this software.
Moore's law didn't stop, just Dennard scaling. Expect graphics and AI to continue to improve radically in performance/price, while more ordinary workloads see only modest improvements.
Not sure about most of the people in here, but I would get really nervous at the thought of running something that eats up 3x300 watts per hour, for 24/7, just as part of a personal/hobby project. The incoming power bills would be too high, you have to be in the wage-percentile for which dropping 60k on a machine just to carry out some hobby project is ok, i.e. you’d have to be “high-ish” middle-class at least.
The recent increases in consumer power prices are a heavy blow for most of the middle-class around Europe (not sure about how things are in the States), so a project like this one is just a no-go for most of middle-class European programmers/computer people.
Things are getting more expensive here but nothing like the situation in Europe (essentially none of our energy was imported from Russia, historically ~10% of oil imports but that was mostly to refine and re-export, we have all the natural gas locally that we need) The US crossed the line into being a net hyrdocarbon energy exporter a while ago (unsure what the case is recently but it is at worst about at parity)
If by "A bit" you mean about 30-40k
> If by "A bit" you mean about 30-40k
30k more expensive: Than your very-low-end-"average" car.
40k more expensive: Than your average used car.
AFAICS. All in what one sees as an "average" car, I suppose.
> The model [...] is supposed to run on multiple GPUs with tensor parallelism.
> It was tested on 4 (A100 80g) and 8 (V100 32g) GPUs, [but should work] with ≈200GB of GPU memory.
I don't know what the price of a V100 is, but given $10k a piece for A100s we would be closer to the $60k estimate.
Also, if you want to have a machine with eight of these cards, it will need to be a pretty high-spec rack-mounted or large tower. To feed these GPUs, you will want to have a decent amount of PCIe-4 lanes, meaning EPYC are the logical choice. So that's $20k for an AMD EPYC server with at least 1.6kw PSUs etc etc.
Note that A100 like other datacenter GPUs are passively cooled. You need a strong airflow and duct in any case that would house them.
It also sounds like they haven't optimized their model, or done any split on it, but if they did, I suspect they could load it up and have it infer slower on fewer GPUs, by using main memory.
Inference latency is a lot higher in relative terms, but even for things like image processing running a CNN on a CPU isn't particularly bad if you're experimenting, or even for low load production work.
But for really transient loads you're better off just renting seconds-minutes on a VM.
Can you spec it out roughly?
You can already see this with Chinchilla:
https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b...
And whether it really matters. That's the bigger question.
I think, for most of us, it does matter. But we're not sure why and what a loss of human reality would really mean.
For a few who wholeheartedly embrace it there's some resonance with the psychedelic/60s creed that sees this as some kind of "liberation".
"Black mirror" was good but it's not nearly enough.
I do not look forward to the day when that story becomes an optimistic view of the future.
We're a long, long way from this. Stringing words/images together into a coherent sequence is arguably the easy bit of creating novels/films, and computers still lag a long way behind humans in this regard.
Structuring a narrative is a harder, subtler step. Our most advanced ML solutions are improving rapidly, but often struggle with coherence over a single paragraph; they're not going to be doing satisfying foreshadowing and emotional beats for a while.
Google's got you!
I'm pretty sure the Marvel franchise is shat out by an algorithm.
We’re probably 18 months away from this. We’re probably less than 5 years away from being able to do this on local hardware. AI/ML is advancing faster than most people realise.
We've come ludicrously far since then. That progress doesn't guarantee that innovation in the space will continue at its current pace, but it sure does feel like it's possible.
You can say that about many movies/series made entirely by humans today. :)
A big reason all the major studios are moving to big franchises is that the real money is in licensing the merch. The movies and TV shows are really just there to sell more merch. Maybe this will work when we all have high quality 3d printers at our desks and we can just print the merch they sell us.
The other big barrier is social. A lot of what people watch, they watch because it was recommended to them by friends or colleagues, and they want to talk about what other people are talking about. I'm sure that there will be many people who will get really into watching custom movies and discussing those movies with chatbots, but I bet most people will still want to socialize and discuss the movies they watch with other humans. FOMO is an underestimated driver of media consumption.
Combine that with the fact that PyTorch recently added support for Apple silicon GPUs.
apart from the fact that you can't use any of the many nvidia-specific things; if you're dependent on cuda, nvcuvid, AMP or other things that's a hard no.
(applies to computing and other technologies like power production and agriculture)
>Gua, treated as a human child, behaved like a human child except when the structure of her body and brain prevented her. This being shown, the experiment was discontinued
There have been a lot of speculation as to other reasons of ending the experiment so prematurely. Maybe exhaustion. One thing which seemed to dawn on the parents - if one reads carefully - is that a human baby is far superior at imitating than the chimpanzee baby, frighteningly so, that they decided to abort the experiment early on in order to prevent any irreversible damage in the development to their human child which at that point had become far more similar to the chimpanzee than the chimpanzee to the human.
So, I would rephrase "the internet is dead" into "the internet becomes increasingly undead" because humans condition themselves in a far more accelerated way to behave like bots than bots are potentially able to do. From the wrong side this could be seen as progress when in fact it's opposite progress. It sure feels like that way for a lot of of people and is a crucial reciprocal element often overlooked/underplayed (mostly in a benign effort to reduce unnecessary complexities) when analyzing human behaviour in interactions with the environment.
[0]https://en.m.wikipedia.org/wiki/Winthrop_Kellogg#The_Ape_and...
I can't put a dumb person under.
I need someone with an active imagination who wants to work with me (for best results)
In other words, people are already behaving like bots; and we're building more and more software to encourage such behavior.
Now primarily employed in a marketing capacity.
Over my career I've worked with: - Doctors - Lawyers - Engineers - Fund managers - Academics (hard and soft sciences) - Mentalists/Hypnotists
All of them believed that they're specific training and temperament made them immune from simple persuasion techniques and that they were purely rational actors.
None of them struck me as any more rational/more independent thinkers than anyone else off the street
Even when someone rates oneself down like when saying of themself that they're dumb, ugly or whatever, they generally mean it in a lesser fashion than for any other peer they'd attribute as such.
these guys are similar, except it's common belief.
The problem is that these problems are less profitable. And that the companies with enough compute to train these types of models are concerned about getting more eyeballs, not making the world a better place.
Sure, using AI to treat people without a human in the loop would clearly do harm. But using AI as an assistant, to help a doctor make the right diagnosis, seems like it'd do the opposite. It'd help doctors serve a larger patient population, make less mistakes, and probably equate to less harm in the long run.
Anyway, I think we can all agree that using AI for anything other than ad targeting is a net win.
Actors attempt to imitate humans. “Good acting” is convincing; the audience believes the actor is giving a reasonable response to the portrayed situation.
But the audience is also trying to imitate the actors to some degree. Like you point out, humans imitate. For some subset of the population, I’d imagine the majority of social situations they are exposed to, and the responses to situations they observe, are portrayed by actors.
At what point are actors defining the social responses that they then try to imitate? In other words, at what point does acting beget acting and how much of our daily social interactions actually are driven by actors? And is this world of actors creating artificial social responses substantially different than bots doing the same?
Famously the bald eagle sounds nothing like it does in tv and the movies and explosions are rarely massive fireballs. For human interaction it’s much harder to pin down cause and effect but if it happens in other cases it would be very surprising to not happen there.
"monkey see, monkey do"
> ...humans condition themselves in a far more accelerated way to behave like bots than bots are potentially able to do.
Than bots can condition themselves to behave like humans, I presume. They can already behave exactly like bots. :-)
Whatever we think will happen will not happen. A less-inspired known-good state will take its place, creating another status quo. Which will funnel us into dystopian futures. I'm just going off my own observations and life experience of the last 20 years, and the way that people in leadership positions keep letting the rest of us down after they make it.
For me, the issue is that use cases and power usage are secondary to the fundamental science of computation. So it's fine to have matrix-processing stuff like OpenGL and TensorFlow, but those should be built on general-purpose hardware or else we end up with the cookie cutter solutions we have today. Want to run a giant artificial life simulation with genetic algorithms? Sorry, you can't do that on a GPU. And it turns out that most of the next-gen stuff I'm interested in just can't be done on a GPU.
There was a lot of progress on transputers and clusters (the old Beowulf cluster jokes) in the 80s and 90s. But researchers came up against memory latency issues (Amdahl's law) and began to abandon those approaches after video cards like the 3dfx Voodoo arrived around 1997.
But there are countless other ways to implement concurrency and parallelism. If you think of all the techniques as a galaxy, then GPUs are way out at the very end of one spiral arm. We've been out on that arm for 25 years. And while video games have gotten faster (at enormous personal effort by millions of people), we've missed out on the low hanging fruit that's possible on the other arms.
For example, code can be auto-parallelized without intrinsics. It can be statically analyzed to detect contexts which don't affect others, and the instructions in those local contexts could be internally spread over many cores. Like what happens in shaders.
But IMHO the greatest travesty of the modern era is that those innovations happened (poorly) in GPUs instead of CPUs. We should be able to go to the system menu and get info on our computer and see something like 1024+ cores running at 3 GHz. We should be able to use languages like Clojure and Erlang and Go and MATLAB and even C++ that auto-parallelize to that many cores. So embarrassingly parallel stuff like affine rasterization and blitters would run in a few cycles with ordinary for-loops instead of needing loops that are unrolled by hand or whatever other tedium that distracts developers from getting real work done. Like, why do we need a completely different paradigm for shaders outside of our usual C/C++/C# workflow, where we can't access system APIs or even the memory in our main code directly? That's nonsense.
And I don't say that lightly. My words are imperfect, but I do have a computer engineering degree. I know what I'm talking about, down to a very low level. Wherever I look, I just see so much unnecessary effort where humans tailor themselves to match the whims of the hardware, which is an anti-pattern at least as bad as repeating yourself. Unfortunately, the more I talk about this, the more I come off as some kind of crackpot as the world keeps rushing headlong out on the GPU spiral arm without knowing there's no there there at the end of it.
My point is that for all the progress in AI and rendering and simulation, we could have had that 20 years ago for a tiny fraction of the effort with more inspired architecture choices. The complexity and gatekeeping we see today are artifacts of those unfortunate decisions.
I dream of a day when we can devote a paltry few billion transistors on a small $100 CPU to 1000+ cores. Instead we have stuff like the Cerebras CS-2 with a trillion transistors for many thousands of dollars, which is cool and everything, but is ultimately gatekeeping that will keep today's Anakin from building C-3PO.
https://en.wikipedia.org/wiki/Multi-core_(computing)#Hardwar...
Before any of the things you describe happen, most states will mandate the equivalent of a carry permit to be able to freely use compute for undeclared and/or unapproved purposes.
I think it's far more likely that in 10 years we'll all become more used to rolling blackouts, and fondly remember we all used to be able to afford to eat out, and laugh over a glass of cheap gin about how wild things were back in the old days before things got really bad.
10 years ago was a much more exciting and hopeful time than today. I remember watching Hinton show off what deep learning was just starting to do. It was frankly more interesting that high parameter language models. Startups were all working on some cool problems rather than just trying to screw over customers.
That's just technology. Economically, socially and ecologically things looks far brighter in 2012 than they do now, and in 2032 I suspect we'll feel the same about today, but far more dramatically.
We've already pass the peak of "things are getting better all the time!" but people are just in denial about this.
It also seems to me that most people would not be ready to give up more than 10% of their luxuries / way of living up-front in order to protect those structures and would continue to watch funny TikTok videos and post IG photos until the very moment their internet access goes out and doesn't come back.
Do you mean train or run? My assumption was all these models could be run on most computers, probably with a simple docker container, as long as there is sufficient RAM to hold the network, which should be most laptops > 16gb ram.
Speaking of which, anyone have recommendations on pre-trained docker containers with weights included?
The Mac Studio maxes out at 128GB currently for around $5K, so 256GB isn't that far out and might work with the ~200GB Yandex says is required.
In theory you could probably assign a powerful enough iGPU a few hundred gigabytes of memory already, but just like Apple Silicon the integrated GPU isn't exactly very powerful. The difference between the M1 iGPU and the AMD 5700G is less than 10% and a loaded out system should theoretically be tweakable to dedicate hundreds of gigabytes of VRAM to it.
It's just a waste of space. An RTX3090 is 6 to 7 times faster than even the M1, and the promised performance increase of about 35% for the M2 will means nothing when the 4090 will be released this year.
I think there are better solutions for this. Leveraging the high throughput of PCIe 5 and resizable BAR support might be used to quickly swap out banks of GPU memory, for example, at a performance decrease.
One big problem with this is that GPU manufacturers have incentive to not implement ways for consumers GPUs to compete with their datacenter products. If a 3080 with some memory tricks can approach an A800 well enough, Nvidia might let a lot of profit slip through their hands and they can't have that.
Maybe Apple's tensor chip will be able to provide a performance boost here, but it's stuck on working with macOS and the implementations all seem proprietary so I don't think cross platform researchers will really care about using it. You're restricted by Apple's memory limitations anyway, it's not like you can upgrade their hardware.
You'll need your own wiring to run your PC soon :-)
I am not a physicist or biologist or anything like that so my intuition is probably completely wrong but it seems to me that for more basic inference operations (lets say add two numbers) power consumption from a processor and a brain is not that different. It’s like seeing how expensive it is for computers to infer for any NLP model, humans should be continuously eating carbs just to talk.
Sure it is - it is too hard to figure it out based on 2 numbers number, but lets multiply that by a billion - how much energy does it take a computer to add two billion numbers? Far less than the energy it would take a human brain to add them.
[It's been quite a while since I studied this stuff, so I can't recall whether 60 mV/decade is a constant for silicon specifically or all semiconductors.]
175 GB aggregate on both RAM and nvme is in the realm of home deep learning workstation.
As long as you aren’t too fussy about inference speed of course.
>but is able to work with different configurations with ≈200GB of GPU memory in total which divide weight dimensions correctly (e.g. 16, 64, 128).
Would you have to scan through all 200GB of data once per character generated? That doesn't actually sound too painful - 1 minute per character seems kinda okay.
And I guess you can easily do lots of data parallelism, so you can get 1 minute per character on lots of inputs and outputs at the same time.
$ dd if=/dev/zero of=/swapfile bs=1G count=250 status=progress
$ chmod 600 /swapfile
$ mkswap -U clear /swapfile
$ swapon /swapfileYou're also likely to get less write amplification if your swap file is continuous.
Of course with all the layers of indirection it's a numbers game, you don't know if your file system allocates adjacent inodes, and you don't know how your SSD will remap the blocks. But all else being equal, trying to make the file as sequential as possible seems preferable.
It's simply because it's an easy way to create a file of a certain size that most Linux users would be familiar with.
The quicker way (and possibly more "proper" way) is to use fallocate, but who has even heard of that vs dd ?
$ ( umask 077; dd if=/dev/zero of=/swapfile bs=1G count=250 status=progress )
$ mkswap /swapfile
$ swapon /swapfileBut this does make me wonder if there's any way to allow a graphics card to use regular RAM in a fast way? AFAIK built-in GPU's inside CPU's can but those GPU's are not powerful enough
Integrated GPUs that use a portion of system memory are an exception to this and do not require memcpys when using unified memory. However, I'm not aware of any powerful iGPUs from Nvidia these days.
There does seem to be a zero copy concept as well and I've certainly used direct memory access over pcie before on other proprietary devices.
https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/ind...
I wonder how bad it comes out with something like Optane?
You can also infer a few tokens at once, so it will be more than 1 char a minute. Probably more like sentence a minute.
It’s difficult to estimate how slow it would be, but I’m guessing unusably slow.
During inference you only need to keep layer outputs until the next layer's outputs are computed.
If we talk about memory bandwidth, it is space requirements that are important, not so much time complexity.
https://aws.amazon.com/ec2/instance-types/p4/
p4d.24xlarge
An alternative is the p3.16xlarge for 8 V100s with 256GB of GPU RAM but you might as well get the A100s since it's only $0.50/hr cheaper
I assume some of the services that offer GPT-J APIs will pick this up, but it doesn't look cheap or easy to get this running.
You can't even search for images "before:date" in Google anymore.
End of the day I’ll use what actually gets the job done.
Same goes for OpenAI and Google AI. If you don’t actually ever release and let others use your stuff and end paralyzed in fear at what your models may do then someone else is gonna release the same tech, and at this rate it seems like that’ll be Chinese or Russian companies who don’t share your sensibilities at all, and their models will be the ones that end up productized.
Their "moral" reasoning behind not publishing models is simply laughtable because they do sell API access to them to anyone who can pay. And "bad guys" generally have money.
Here's an alternative: progressively release better and better models (like 3B params, 10B, 50B, 100B) and let people figure out the best way to fight against bad actors using them.
This is the sort of argument that proves guns would be less dangerous if everyone had access to them.
Or sometimes both, because it's only part of the truth. Maybe the complete version should be: "An armed society is a 'polite' society, but with very frequent killings and not-infrequent massacres." I think I prefer living in a less "polite" society.
I would be totally on their side if their reasoning was that they dont publish models to compete with FAANG more efficiently and get more income for their research, but this moral reasoning just sounds completely fake because bad actors do have funding to train their own models.
Examples of "real cases of misuse encountered in the wild" include "spam promotions for dubious medical products and roleplaying of racist fantasies".
Yes, some bad actors can train their own models, but OpenAI can't do much about that either way. It is doubtful whether spam promoters of dubious medical products can, at least for a while.
Personally, I think using AI to manufacture advertisements on demand is misuse... but will Google agree with me?
why is morality into this? is this the same discussion of car manufacturers not selling cars to certain people because they are worried about misuse?
> MIT " IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
don't commercial licenses have same/similar wording so what liability are you talking about?
They used to be a non-profit with a mission, now they are a for-profit with the only mission of money.
But yes, the name is not what it should be, given their current ideas.
- NVIDIA sells GPUs and interconnect needed for training large models. Releasing a pretrained LM would hurt sales, while only publishing a teaser paper boosts them.
- Google, Microsoft, and Amazon offer ML-as-a-service and TPU/GPU hardware as a part of their cloud computing platforms. Russian and Chinese companies also have their clouds, but they have low global market share and aren't cost-efficient, so nobody would use them to train large LMs anyway.
- OpenAI are selling their models as an API with a huge markup over inference costs; they are also largely sponsored by the aforementioned companies, further aligning their interests with them.
Companies that release large models are simply those who have nothing to lose by doing so. Unfortunately, you need a lot of idle hardware to train them, and companies that have it tend to also launch a public cloud with it, so there is a perpetual conflict of interests here.
What kind of lawyers/regulation do you have in mind? If anything, I'd find the opposite: lawyers and copyright holders should be grateful for such a tool that - when it was still working - allowed you to trace websites using your images illegally.
Now they all use Yandex for this purpose, with relatively good results.
https://www.theverge.com/2018/2/15/17017864/google-removes-v...
https://www.washingtonpost.com/news/the-intersect/wp/2016/08...
https://www.theguardian.com/technology/2016/apr/08/does-goog...
https://www.bloomberg.com/news/articles/2021-10-19/google-qu...
https://theconversation.com/googles-algorithms-discriminate-...
Tineye is definitely better than Google with crops, etc. Google reverse image search seems to have more data, but it seems much less able to recognize even basic modifications to the input.
Who's suing them and on what grounds? If they made changes, it's probably for PR reasons, not legal ones.
Also not all of these seem "fixed" e.g.:
> https://www.theguardian.com/technology/2016/apr/08/does-goog...
Article from 2016, but results look very similar today: https://www.google.com/search?q=unprofessional+hair&source=l...
If they let everyone use the latest models, critics could uncover ugly biases in 10 minutes. Then Google would have to do damage control. These models are very suggestible. You can induce them to make fools of themselves.
Naval Ravikant put it best here: https://twitter.com/naval/status/1002106977273565184
The day will come when we will be able to.
Won’t incremental advancement cover this eventually? (i.e. no major breakthrough required, just patience).
Can anyone recommend any open source machine learning project that would be a good starting point? I want one that does something interesting (whether using text, images, whatever), but simple/efficient enough to run on a gaming PC and see some kind of results in hours, not months. I'm not sure what I want to do with ML yet, I just know I'm interested, and getting something up and running is likely to enthuse me to start playing and researching further.
My spec is: GeForce RTX 2080 Ti (11GB), a 24-core AMD Ryzen Threadripper, and 128GB RAM. I'd be willing to spend on a new graphics card if it would make all the difference. I am a competent coder and familiar with Python but my experience with ML is limited to fawning over things on HN. Any recommendations gratefully received!
1. CS231n Machine Vision https://www.youtube.com/playlist?list=PLkt2uSq6rBVctENoVBg1T...
2. CS234 Reinforcement Learning https://www.youtube.com/watch?v=FgzM3zpZ55o&list=PLoROMvodv4...
3. CS330 Meta Learning https://www.youtube.com/watch?v=0rZtSwNOTQo&list=PLoROMvodv4...
Those will get you on track with general concepts about reasoning, AI engineering and concepts of learning itself
Language models for me a bit of headache because there're in different domain on intersection with linguistics and humanities but here's a good course
https://www.youtube.com/watch?v=rha64cQRLs8&list=PLoROMvodv4...
Those are all free and high-quality but require a lot of brain power
so we looking at crazy prices just for inference. RIP to the first guy's cloud billing account who makes this public
Suddenly that 3090 i wanted to get, does not seem so expensive....
https://yalm-100b.s3.yandex.net/vocab/voc_100b.sp
EDIT: It seems fine if you download with a browser useragent not CURL... I guess I just got hit by some anti-bot thing they have accidentally have turned on.
Having skimmed the GitHub readme and medium article, this announcement seems to be very focused on the number of parameters and engineering challenges scaling the model, but it does not contain any details about the model, training (learning rate schedules, etc.), or data composition.
It is great that more models are getting released publicly, but I would not get excited about it before some evaluations have been published. Having a lot of parameters should not be a goal in and of itself. For all we know this model is not well trained and worse than Eleuther AI's 20B parameter model, while also being inconveniently large.
And yet your own project headline is "Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrough Performance"[0].
0-https://ai.googleblog.com/2022/04/pathways-language-model-pa...
2. 540 billion parameters is notable for its size, which is likely why they lead with that particular headline.
* Good. This is great researchers helping community by sharing great work. (which is what I'd like to assume before I have any proof of the contrary)
* Bad. This very expensive training has been approved by Ya leadership (which is under Western personal sanctions) because they've secretly built in RU's propaganda talking points into the model. Such as "war in Ukraine is not a war but special operation" etc.
But we should have better tools to test for biases/toxicity. Perspective API is great tool for toxicity detection. But I'm not aware of any "propoganda" detection tool.
Start here: https://www.vennify.ai/gpt-neo-made-easy/
Looks to be approximately 50/50 from my random scrolling through the list.
Therefore, the size of the vocab gives a good guide to the size of the data, since if there was 10x more english language data then the optimal distribution would be to dedicate more token space to english than russian.
https://huggingface.co/EleutherAI/gpt-j-6B
isnt that so ?
2. Yandex News service ignores the genocide currently happening in Ukraine.
3. Yandex Search engine hides the pictures of Bucha and Irpin massacre as well as Kharkiv and Mariupol destruction.
Yandex using whitewashing tactics via open source.
obeying to Kremlin is just an aspect of running business in Russia
the only option would be not to operate in Russia at all. Yandex can't do this, because their audience is primarily in Russia
If you're profiteering from our suffering and choose Kremlin's needs over ours, don't be suprised then when we tell you to shove your AI models and your search.
Now they scramble to present a whitewashed image to Western public. They will probably put themselves forward as great contributors to open source.
From context, pretty obviously "we Ukrainians". Didn't do too well in elementary reading, did you?
> I am not in this together. So change it to "I". I don't care about you lot... lmao
Thank you for so effectively demonstrating what a despicable excuse for a human being you are. I'll do my best to remember this when next I come across anything from you.
Of course Yandex likes to pose as the victim of censorship, but the truth is that they are the censors themselves. They've been steamrolled by a runaway process they helped to create.
"Putin's agents reportedly threatened a top Google executive in Moscow with a 24-hour ultimatum – Take down Russia protest vote app or go to prison" -- https://www.businessinsider.com/russia-agents-threatened-goo...
Not yet at least, the political climate may deteriorate to that point, especially when it's about elections, given recent revelations.
Still, at least right now it looks to me - and I have visited Russia and Ukraine several times in the past and still have indirect connections (to people heavily involved in business there) - that there still is considerable more freedom from the government and its wishes for people and companies in the West.
If you publicly criticize a US politician you may get some hate messages, but at least they are from private citizens and you don't have FBI agents knocking on your door threatening you with prison. In Germany some rogue police were found to send threatening messages, but as soon as it was discovered the government acted against it. Also in Germany there even were public rallies from pro-Russian folks, now try that in Moscow with pro-Ukraine banners... Russia even bans the colors yellow and blue, even when they have nothing whatsoever to do with Ukraine and are just decorative: "Russians Strip Yellow and Blue From the Nation’s Streets Over Ukraine War" -- https://www.themoscowtimes.com/2022/04/27/in-photos-russians...
Correct, it's DHS.
https://twitter.com/_secondthought/status/133274617257067725...
The same DHS who bans immigrants that are or have been members of a communist party [1].
[1] https://www.uscis.gov/sites/default/files/document/policy-ma...
It seems plausible; we don't know what gets done under the FISA court but it would presumably involve companies like Google. Some suited agent of the US government turning up at Google HQ and threatening jail time under some FISA warrant if some pro-Trump something doesn't disappear off Google.
That'd be a scandal but not the worst abuse of the secret court system. It hasn't exactly covered itself with glory since inception. They already spy on basically everyone and that is a lot worse than some light censorship.
What about Canadian truckers? Didn't Trudeau call them terrorists, took their trucks, donations, bank accounts and driver licenses... There is no right to protest anywhere, don't kid yourself.
They gave protest a bad name.
Your conclusion that "There is no right to protest anywhere" is simply ridiculous.
BLM rioters did this, and more. Violence + Property damage + Corporate Backing + gov backing. They didn't had their donation money seized,and almost no resistance to establish order.
Julian Assange detailed this in a newsweek article before his name and body were smeared into the ground:
https://www.newsweek.com/assange-google-not-what-it-seems-27...
Oh, but they say he's not trustworthy, or that it's a conspiracy theory that he was intentionally smeared. Well, the CIA and their contractors have been doing it for over a decade, even before he was unfairly accused of helping trump:
https://arstechnica.com/tech-policy/2011/02/the-ridiculous-p...
Google is an arm of the state department, no doubt.
Zero.
Get it?
None.
Zero.
But... :
(persecuted journalist)
>https://www.bbc.com/news/uk-61839256
>https://www.amnesty.org/en/latest/news/2013/07/usa-must-not-...
(jails its opposition)
>https://eu.usatoday.com/storytelling/capitol-riot-mob-arrest...
>"hatespeech" (newspeak)
>fraudulent elections: funny how the concerns that 'half' of the US had with elections were dismissed. Especially when conditions were different, by using a method usually agreed (until now, cuz narrative) prone to tampering. So much for free and fair elections.
So _Zero_ huh?
Zero comparison.
Sure, there's probably a non-zero amount of germs in the pasteurised homogenised and sterilly-packaged milk I buy at the supermarket. So is drinking that equivalent to sucking the pus out of a punctured boil on the arse of a diseased cow?
No. There's zero comparison. None. Zero.
Learn to read, man. If nothing else, it'll make you a better propaganda troll for your Kremlin master.
https://www.reuters.com/technology/yandex-sells-news-content...
They do business in other countries and for that it is best for the business to appear as neutral as possible. We don't know how much they fiddle with the search results and ranking but this still looks quite neutral to me: https://yandex.com/search/?text=russo+ukrainian+war
Yandex is in fact share responsibility for Russian government actions. While it impossible to fight censorship they could certainly shut down their News service completely.
Yandex could also certainly move more of their company and staff out of country. It was their deliberate choice stay in Russia and getting advantages on local market by using their political weight.
https://reclaimthenet.org/youtube-responds-to-cpac-censorshi...
https://reclaimthenet.org/google-expanded-its-censorship-of-...
https://reclaimthenet.org/russia-continues-to-order-google-t...
In US they pretend to "decide" to censor things "on their own" because 1st amendment prevents the government from officially demanding censorship.
Can you please edit name-calling / swipes like that out of your HN comments? It breaks the site guidelines and weakens your point.
Highly emotional tone changes how the data affects the reader. If he is right, I would surely better remember next time that Google is in the same ballpark due to the insult hitting hard. If he is wrong, I will know better to ignore such claims in the future without a direct quote or something else that consumes less time than reading an entire linked article.
A quote from the second one: "cumulative 45 percent decrease in traffic from Google searches"
When a company puts Jon Lennon's Merry Xmas (War is Over) behind age restriction banner[1], the question stops being "Is there censorship?" and becomes about the logic of such censorship.
>The third one was temporary until Google stopped operating in Russia altogether.
They've censored other things on behest of the Russian government for years[2]. Again, I cannot fathom how people on a tech website like HN can be unaware of such things. This is common knowledge broadly covered on mainstream websites.
---
[1] https://reclaimthenet.org/youtube-john-lennon-war-is-over-wa...
[2] https://www.rferl.org/a/google-censors-search-results-after-...
Even in the translation case (which I assume you mean by your "excuse" remark) the original source is still available as is. I am not even sure from the description what translation team it was talking about and what does it have to do with Google exactly. "translate company text for the Russian market" this passage sounds like it talks about translating Google's own interfaces, help pages, press releases, or support articles to Russian. E.g. no external voice is being censored.
In any case as unfortunate owner of Russian passport with friends and collegues in Ukraine I am more affected by Putins war than by anything US does.
So I want Yandex to be seen as part of Kremlin propoganda machine and threated accordingly. This company grew monopoly in many markets in Russia and they directly benefited from Putin regime. Since Ilya Segalovich died company started to be "out of politics" and this complete lack of any political activity lead country to these terrible events.
The difference between holding values, and holding values when convenient rather sums up the entirety of human history in one phrase.
Can't say the same about HN, it's one of the few places that seems to have kept its sanity (for now?)
But yes, there is, sadly, some discrimination. It's got nothing to do with ethnicity though; it's the same thing that happened to ordinary Germans in 1939-1945, and for the same reasons.
What medicine?
If you are so conscious about consuming tainted fruits the only way to escape is to be living on some deserted island catching your own food.
Their government does. They empower it. Just to be clear, it is their army, and their government doing the deed. They elected, they pay salaries to.
And you can back this up how?
What is your problem?
Operation Paperclip...?
No, I think that's very naive.
Yes. HTH!
To elaborate: At some stage, that becomes the only acceptable alternative; not doing so is morally culpable.
The world learned that in 1945, two ways:
1) "I was only following orders" was deemed not a valid excuse at the Nürnberg trials; not refusing orders like that is complicity; and
2) Germany as a whole was de-Nazified. Just like Russia needs to be now. (But more thoroughly: in Germany's case, it was an aberration of a dozen years; in Russia's, it's a millennium of unbroken history of totalitarianism.)
Too bad the world seems to have forgotten those lessons since then.
Russian police is notorious for selling logs, private data or access and also selling databases on a black market.
In my opinion it is not punishment to stop a relationship if the basis of that relationship was destroyed deliberately by one side.
> punishment: the infliction or imposition of a penalty as retribution for an offence.
Then you can play all you want with language to make it say what you'd like but it's pointless.
If you still mantain that the purpose is deterrence, then you must be a fool or worse, since it never works! Can't you learn from the past?
Edit:
Before you even dig out some article like this: https://www.washingtonpost.com/news/worldviews/wp/2014/04/28...
I will make an important point: sanctions CAN work only to prevent something from happening. Once that happened, e.g. the war started, there's no sanction that can stop it and that's exactly when sanctions stop being a deterrent and start being a punishment.
They are pretty effective at preventing future wars. This war happened only because Russia wasn't crippled enough after 2014.
you can't really blame the companies for not wanting to be associated in any way with a nazist regime genociding a neighboring country.
And Starbucks or Mercedes pulling out of Russia isn't punishment. It is freedom of association and economic activity. Russians are whining about "punishment" because they have no idea about freedom, and that is them getting a bit of taste of it. They think they can plaster whole country with their swastika - "Z" - in enthusiastic support of the bloody genocide while the whole word shouldn't be able to express its disgust at those happenings.
Especially funny how Russians are cry-baby style whining about supposed violation of their property rights by the West sanctions while Russians have been violating property rights of more than 40 million of Ukrainians (even if we don't consider all the mass killing and raping of civilians that Russians have been doing there). The deep and profound disintegration of any morals in my old country is stunning.
>or issuing sanctions on private individuals.
due to the size of their wealth and the de-facto rules of economic activity in Russia those aren't private wealth of private individuals - they are integral part of that nazist regime, and thus they are guilty too.
And for the sanctioned Russian government officials - that is for example the Roskosmos CEO Rogozin, who is one of the main founders of the Russian Nazi movement "Motherland" (people from which has since taken prominent roles across the Russian government and the ruling Party) and who is one of the most prominent voices around Putin and the Putin's favorite, giving a Nazi salute and the end of his Nazi speech at the Russian Nazi march in Moscow. The specific phrase they all give Nazi salute to is "Glory to Russia!".
Not even seeing the irony. What did they expect in response to bombing, pillaging, mass rape? Friendly hug?
What did they expect? Probably indifference, same reaction to any war crime committed by the west so far.
If it were not for NATO, Russian rapists would already be in Tallinn, Vilnius, Helsinki. Claiming they were offended by historical injustices therefore women should be raped and men shot dead ("denazified").
Not the people in Donbass and Crimea apparently.
> Russian rapists
Oh please, quit the blatant Lyudmila Denisova propaganda. She was fired by your own government for all the falsehood she spread.
Russian soldiers have been shooting civilians at will (it is called "safari" in Russian army, and those Bucha streets - Yablunska and Vokzalnaya - with the bodies lying along the street is an example of such a "safari"). That would be an unique miracle in history if the soldiers wouldn't rape given such power in such conditions (for example those mass rapes by Soviet soldiers - 2 million of women were raped - back then in Germany in 1945 illustrate a result of such a power by the occupying force). And there is no miracle - just google Ukrainian rapes.
https://greekcitytimes.com/2022/06/01/ukrainian-official-fir...
[1] https://meduza.io/en/feature/2022/04/18/i-can-do-whatever-i-...
I'm not suggesting there have been no rapes or that Russians are the good guys. Just that the reports of mass rapes were fabricated (Denisova admitted she thought it would help Ukraine obtain more sympathy and weapons from the west).
Because most of those "everyone's" are not facing the choices themselves and are basically keyboard warriors. Let's see what they say when they'll be asked to sacrifice their own well being to be on a "high moral ground".
As a relatively neutral party in all of this whose country hasn't tainted itself (but who nevertheless spent all my live under a similar autocracy), I can't help but shake my head at keyboard revolutionaries who definitely would have overthrown the regime, if only they lived in Russia. You just have no freaking idea what you're talking about. Guard your democracy as best you can so you don't have to find out.
If you look at Snowden and Assange, that ship has sailed.
Russian IT and media were showered with relatively high wages to stay quiet while last semblance of elections was finally destroyed, independent media was took over by pro-Putin oligarchs and activists were crushed or murdered.
As was the sarcastic saying coined recently, "if you are apolitical then bullets don't hit you".
Probably move if that's anywhere near a possibility, and if not, cowardly stay as unnoticeable as possible. I know that at least in Finland Russian refugees are mostly welcome (although the border might be closed right now), even if they probably will face a lot of scrutiny from various authorities for obvious reasons. Most certainly it's nothing like the attention such people would face in Russia.
We fondly remember even the smallest acts of defiance that ordinary Germans acted out against their regime during 1933-1945. We all would like to be those people in times of crisis, but obviously most of us are not. They were probably ultimately pretty futile acts during that time, though, but put together with all the other actions that happened against the Nazis played a significant grander role. And we know that more than a few significant Jewish scientists and engineers fled from Nazi Germany and made significant contributions to the war effort. For instance, one guy called Einstein.
Yandex News is IIRC the biggest news media in Russia.
It filtered all results on protests and opposition resources leaving only government propaganda. Same with war. Filtering not meaning downranking. Just straight up not showing.
Editors were fired for not staying in line until it was completely sterilized and filled with pro-war propaganda.
Working in Yandex is being complicit with it.
Why are you so surprised? This is exactly how most of the population behaves everywhere. People go about their business and "support" criminal actions of their governments all the time. This includes the West. Our governments have no problems exterminating, starving and displacing people (as long as they're the "right" people to mess with) while the majority of the population is going on merrily about their business. And Europe keeps buying Russian stuff even now while "Ukrainians are being killed". Where are the mass protests and fights with the police?
Things might change when "messing" with people will ALWAYS have the consequences for ANY country. But this is not what is happening and is unlikely to change. We have no extra terrestrial entity to police us in impartial way.
It's high time to bring "but US bombed Iraq". Classic playbook.
No one likes hypocrisy, especially when it is coming from the same westerners that at most protested for a few weeks back in 2003 when their own countries bombed us for 2 decades, that are now calling for other people to get arrested and possibly tortured/executed by putin's regime because that's just the right thing(tm) to do to stop the war. It would be laughable if it wasn't despicable.
"Yes hundreds of thousands of Muslims died and are still dying, but bringing it up or asking me to do anything about is fallacious! Checkmate"
As you said, who cares about debate tricks when people in the middle east are still dying from the war on terror as we spead? Why are you holding other people to standards that you don't even pretend to hold yourself to? You are expecting people to get arrested to prevent deaths and talk about the situation in ukraine, but I guess making you uncomfortable with "whataboutism" is the limit?
What should russians do you ask? Fight. I did it in Ukraine in 2004. Then in 2014. I didn't run from cops, I didn't let them take my friends. But regardless, now we pay with our lives, being subjected to genocide because of russian cowardliness.
because so far they are all just paying for the genocide.
apache license
>The model is published under the Apache 2.0 license that permits both research and commercial use
you should be fine
Kolmogorov complexity is (I hope) untainted.
Also, Hilbert’s problems are not untainted (and he never flew Nazi Germany!).
Lots of things have to be cleared for backdoors, Intel and AMD are scary with their built-in ME and whatever AMD had, I can't exactly remember, proprietary hardware in general is very scary outside the US due to surveillance and possible backdoors, it's kind of weird. Same goes for China, though they don't surveil foreigners exactly as hard, at least here they don't. It's not exactly an ideal situation and I think there should be agreements done internationally on stuff like this to keep the US and China out of our devices or allowing them to kindly fuck off entirely.
Not exactly the same kind of taint though, your products aren't as much morally tainted as they are simply dangerous to use, like little telescreens you have to carry around.
Edit: I would like to see more details in addition to size and languages (en, ru) about training data. For example, did they use their own Yandex.news (a cesspool of propoganda)?