302 karma · joined April 24, 2018
BERT requires a huge corpus, but it isn't labeled. BERT is trained through self-supervised learning using mask tokens and next sentence prediction. Fine-tuning is useful for specific tasks, but isn't absolutely essential for the model to function.
Incidentally, reduce is also powerful enough to implement both map and filter in terms of itself, though that's more of a teaching exercise than a good recommendation.
I mostly interpret it as of the same spirit with those who oppose proper tail calls because it "ruins" their debugging stack traces.
I was quite confused at the use of "tariff" here, as it meant to me a "tax on a good crossing a political boundary." Turns out, 'tariff' has an older meaning: a published schedule of fees or taxes issued by some authority.
LLMs seem particularly suited toward these existence-proof problems. Working mathematicians seem absolutely essential for universally quantified results, still. I strongly doubt, for example, that if Fermat's Last Theorem hadn't been proven three decades ago, that an LLM would be able to do work equivalent to inventing the mathematics as Andrew Wiles did to solve the problem. I have similar doubts about P vs NP, the twin prime conjecture, even the Riemann Hypothesis (unless the latter has at least one counterexample).
And I want to be clear: I'm not downplaying the achievements of these models. This is remarkable! I simply think that the pattern of success is in existence proofs or finding counterexamples, which makes sense based on how LLMs function and are trained.
Despite not even finishing undergrad, she adopted this persona so naturally, including the black turtleneck and a Zuck-esque vocal inflection, that due diligence apparently defenestrated itself. Phyllis Gardner is the unsung hero who tried to warn the world, but the MBA mass-delusion was too strong.
I will say, we have a LOT of elderly retired people who live here. It's not uncommon to hear sirens because someone is having a medical episode, and the security office is helpful with responding to those and helping with EMS. They are also helpful for doing well checks on the elderly when they haven't been responsive to others.
That being said, a lot of what they do is run radar and evidently do absolutely atrocious behavior like this. One of the flip sides of the retired population is that many of them devote their lives to reporting their neighbors to The Authorities – generally the HOA or security office. In this case, I guess the informant went straight to the cops.
Many of us find it frustrating, though exposure is almost random. Someone who used to live on our street used to get very upset about trash cans being visible from the street, for instance. I think this person moved in the last two or three years because the passive aggressive HOA letters about this apparently heinous nuisance have stopped. But among groups of affluent people, I find that a small percentage never outgrew the tattle-tale phase and still behave in such a way into their sixties, seventies and eighties.
I'd prefer the models to get better at SVG. I really hate working with the rasters that diffusion models generate, but the vector outputs are just really bad even when tokenizable like SVG. I've done some experimentation with trying to make these work better with some newer techniques with some success. But I also think the SVG Paths mini-language may be a bit too concise and unforgiving for LLMs to consistently get them right without specialized training.
Good Lord, what games have you been going to? I see a couple MLB games live every year and a few minor league games. I haven't seen anything like what you're describing. That doesn't mean there are no drunks or obnoxious fans, but that's the risk of doing something in public. The last game I saw live was San Francisco at Milwaukee in June. Giants won 1-0, and the home team crowd was fine. And I mean the Brewers fans like their beer, but in no way was it a zombie apocalypse. Last game I saw in Oracle Park last year was also fine. Even Boston a few years ago, which definitely has more of a college town vibe than SF or Milwaukee, was hardly zombie land.
I'll also say that AAA ball is typically awesome. The games are much cheaper than the big leagues, you get to see some big league talent rehabbing from time to time and see the occasional prospect who gets called up. They also tend to be very family friendly.
> Baseball has been a game of moral fiber, respectable players, and good clean fun. But it seems that activists and jerks set out to ruin the experience for people like me. I'm not having it. I'd have way more fun at a punk rock women's roller derby bout.
I don't know about that. Baseball has a certain level of honor, sometimes an excessive amount e.g. some of the "code" that has started to fade away the last decade or so. It also invented the concept of a nearly all-powerful commissioner, which was created specifically because players threw a World Series for a payout. It's a wonderful game, a thinking person's game, but it's always been quintessentially American for both good and ill ever since it scaled nationwide during the Civil War.
> > see it actually improve just through accreting context
> this actually happens and has been tested.
I specifically said a novel task outside of the explicit training. And I already agreed that the so-called thinking models do some level of logical reasoning. But being able to engage in some level of reasoning because it has learned logical inference rules doesn't mean it's actually thinking, regardless of what the researchers wish to call it.
Also, why does each model always fail at the two tests I give it? The models not only fail to improve, but they start to degrade after many subsequent iterations. Someone who can think would at least not get worse.
LLMs are filters or tuners for extremely subtle patterns, patterns that humans frankly are not great at finding. That's what the attention mechanism does: attend to the other tokens that are most related in a given context, even if that related context is distant in the token stream. Some patterns they fail to detect because they haven't been sufficiently trained or post-trained, and so the LLM just attends to noise (or at least that's what appears to be happening).
A lot of intelligence can be effectively mimicked through this pattern synthesis by transformer architecture alone. That's surprising. But I have yet to see them think.
The opposite happens in practice. I test new models with two tasks: iteratively generating SVGs based on a text description with rendered rasters for feedback; and generating "Before and After" clues like on Jeopardy, where the response has two overlapping phrases such that the last word of the first phrase must be identical to the first word of the last phrase. I have yet to find a model that is consistently good at either. And actually they tend to exhibit context rot with these tasks, where they seem drunk or stoned and the quality degrades.
They're extremely good pattern filters, and that includes some level of logical reasoning. But they aren't reflective or adaptable. Just last night, for instance, I was teaching my son about rounding to the nearest millions. It became clear that he didn't know the place values of large numbers, so we reviewed that till he was consistently correct, and then he was consistently great at rounding to the nearest millions or ten millions or hundred billions or whatever. He's thinking. LLMs are not.
I liked StackOverflow for the first ten years or so of its existence, but I gradually stopped using it then suddenly quit altogether when valid questions were being closed unreasonably. At this point, LLMs with documentation in the context, issue trackers and eve the source code (if available) have surpassed SO. Now my main issue is telling the LLM to crap on my idea rather than wishing it were kinder.
I've suspected that the energy regulations and the ruling party's close connection with Dominion Energy (the Governor recently attempted to fire the chair of Virginia Tech's board and replace him with the CEO of Dominion) have had an impact on power use more than data centers themselves.
Could your router handle this?
The design of the model is literally to find patterns and attend to them. The infrastructure and process around an OpenAI model is intended to filter "bad" things (in this case, I agree that the outputs are bad), but is designed to stop some enumerated-ish list of things that aren't allowed, perhaps with some limited "reasoning" about them.
The transformer was designed to attend to relevant pieces of context and generate new ones that match the pattern. OpenAI in particular was doing that work without guardrails, then attempted to bolt on "content filters," which in my opinion just can't work in a rigorous way. (I think Anthropic's "constitutional" approach is much better though not flawless. And regardless, Claude models don't generate images.)
So, yeah, working as designed. Maybe not as intended, because these things are somewhat resistant to the host's intent when the prompter is hostile.
But that's not what happened. The missing image was described as "graphic" or "violent." If I were to receive an email with that request and a missing attachment, my imagination certainly would not conjure images of butterflies & unicorns. Seems the model is working as designed.
I object. The CCP is much more deeply indebted than the US when taking into account provincial and local governments as well as state-owned enterprises.[0] And of course the US debt is financed in its own currency while Chinese foreign debt is financed in dollars or other currencies.
The problem in the US is regulation. An environmental impact study takes 54 months in the US.[1] The CCP, which has no problem poisoning its people or even launching rockets over inhabited villages, doesn't delay itself at all.[2] I'm glad we don't poison our people or place dangerous industry in places that could harm populated areas, or even perform some prophylactic measures to protect nature, but I'm confident that we could do this in less then a year (less than six months?) and make much faster progress. Even for something like nuclear, the ten years (mostly caused by red tape) are really onerous.
> China is the only one that can run if it comes down to it (unless of course the numbers coming out of China are mega bogus, but for that I don't know enough to have an opinion).
Yes, the common opinion among China watchers is that any number the CCP touches is "mega bogus." They're actually in the midst of something of a financial crisis at the moment because of the high debt.
[0]https://www.statista.com/topics/11662/debt-in-china/
[1]https://www.rff.org/publications/reports/how-long-does-it-ta...
[2]https://arstechnica.com/science/2019/11/china-keeps-dropping...
As far as I can tell, architecture, i.e. sound, precise definitions of exactly what a software artifact must do, is now critical. And with LLMs, it's now feasible to begin implementing such things, though many brownfield projects may be intrinsically unsound in ways that their creators are unaware of. In such a world, contributions simply require a modified proof that the software does what it must do, with perhaps additional claims that the maintainers provide.