A.I. Companies Agree to Safeguards After Pressure from the White House | Hacker News Reader

A.I. Companies Agree to Safeguards After Pressure from the White House

nytimes.com

There's a post by OpenAI that was posted earlier:

https://openai.com/blog/moving-ai-governance-forward

In it they mention:

  Bio, chemical, and radiological risks, such as the ways in which systems can lower barriers to entry for weapons development, design, acquisition, or use 
  Cyber capabilities, such as the ways in which systems can aid vulnerability discovery, exploitation, or operational use, bearing in mind that such capabilities could also have useful defensive applications and might be appropriate to include in a system
  The effects of system interaction and tool use, including the capacity to control physical systems
  The capacity for models to make copies of themselves or “self-replicate” 
  Societal risks, such as bias and discrimination

I think the risks are mostly unfounded but dont really see a problem considering them. What I don't like is the "societal risks" - AI ethics always end up just throwing out "bias" as an ill defined term that could mean whatever they want it to, the way people like to use "harm" and "toxicity". If we're going to make policy there needs to be crystal clarity about what this stuff means, it can't just be a proxy for stuff that does against a particular political view. Things like discrimination in employment are already illegal, I see this more as codifying a particular world view.

> AI ethics always end up just throwing out "bias" as an ill defined term that could mean whatever they want it to, the way people like to use "harm" and "toxicity"

There's a lot of specific examples of different kinds bias in models and things that can be done to improve them. A lot of that work was published in the past 10 years. One small example: https://arxiv.org/abs/1712.00193

I don't believe there's a lack of specificity going on here, at least not by those whose boots on the ground doing work on the topic.

"Toxicity" in the LLM space is much less well-defined and there seems to be a reticence on the part of the research community to admit that defining "toxicity" is a necessarily subjective and often political exercise. I mean, people will agree to this statement, but a lot of the evaluation and research methodology makes definite political commitments that practitioners won't admit are political.

The flip-side is also true: a lot of people on the "anti-ethics" side of this debate are too coy about the fact that there are definitely parts of the definition of toxicity that the vast majority of people will agree upon.

And even if we could define toxicity, what to do about it isn't obvious. E.g., let's take something uncontroversial: graphic depictions of violent rape of children. Is this something we should suppress? Not necessarily. It depends on the context. E.g., in descriptions of war crimes and genocides, we shouldn't censor victims who want the audience to know what happened to them. But those same descriptions shouldn't be co-opted into erotica, for example.

One way "out" is to say something like: "look, what we're interested in is providing tools for enforcement of community norms when communicating with a given audience; the community/culture gives us its definition of toxicity and we provide the tools to prevent toxic generation". But those tools aren't neutral: that could be a description of guardrails for children, and could also be a description of overtly political censorship.

It's a difficult and fraught area, and I think all sides of the debate could benefit from more empathy for the other sides of the debate. In particular, this includes presumption of honest intent. There is nothing wrong with filtering toxic content per se -- we can almost all agree that it's reasonable if a company doesn't want to buy a customer service AI that sometimes quotes Mein Kampf to customers with Jewish last names. But the techniques for doing so are not context-free goods, either.

> in descriptions of war crimes and genocides, we shouldn't censor victims who want the audience to know what happened to them

Even this is a political decision, and is generally made based on which side the decider is on.

Agreed. It's an effective strong-man example in the context of western democracies, though.

At least if you include the word genocide. And are also careful to not explicitly point out that Native American relocation and Black Slavery were both genocides that included lots of rape [2]. And are not in a school board meeting [1]. Etc.

[1] https://www.foxnews.com/media/author-sex-slavery-book-graphi...

Honestly, those are talked about on a daily basis in all the mainstream news outlets not owned by Rupert Murdoch. Howard Zinn won a long time ago.

And a huge chunk of the electorate (in more than one country even!) is hugely influenced by or at least aligned with Rupert Murdoch.

These questions are political, even if you've made up your mind about them.

I just don't agree that it's hidden by our politics. "How many Yemenis died in the war, and how" is actually an obscured and hidden topic.

> "Toxicity" in the LLM space is much less well-defined and there seems to be a reticence on the part of the research community to admit that defining "toxicity" is a necessarily subjective and often political exercise.

Is it the research community or the AI-selling industry? Because I see researchers and AI ethics advocates making that point (that toxicity is both subjective and more broadly contextual) all the time, and not just in a detached way but it is central to their arguments about openness and disclosure and public research on functionality; on the other side, I see censorship and adherence claims around particular political views of “safety” alongside opaqueness as the industry-leaders common response.

Yeah, I should have been more specific. I think people agree to this when you say it out loud but the recognition is not actually reflected in favored evaluation and research methodology.

I feel like this is the modern equivalent of Colin Powell waving a vial of Anthrax in front of the UN assembly.

Overall these seem like very reasonable safeguards.

What I'd like to see very clear regulation around is applications of AI in particular domains. I don't care that an LLM failed to analyze a document correctly on my behalf. I care quite deeply about using it to make a decision in a high-risk and/or high consequence (to human life) environment. I care a lot about having the bias inherent in any model drive a decision without human intervention.

"As part of the agreement, the companies agreed to:

Security testing of their A.I. products, in part by independent experts and to share information about their products with governments and others who are attempting to manage the risks of the technology.

Ensuring that consumers are able to spot A.I.-generated material by implementing watermarks or other means of identifying generated content.

Publicly reporting the capabilities and limitations of their systems on a regular basis, including security risks and evidence of bias.

Deploying advanced artificial intelligence tools to tackle society’s biggest challenges, like curing cancer and combating climate change.

Conducting research on the risks of bias, discrimination and invasion of privacy from the spread of A.I. tools."

"Ensuring that consumers are able to spot A.I.-generated material by implementing watermarks or other means of identifying generated content."

Ok, cool - but, how?

Yeah, if they put watermarks on generated images then everyone will just immediately stop using their product and switch to a competitor.

There's some interesting work using steganography to mark generated text (https://arxiv.org/abs/2301.10226), but I'm not sure if that's what they're talking about here.

I assume we are talking about non human perceptible forensic watermarks on images and video? I think these are generally a good idea; Photoshop, Camaras, most image output systems should do the same.

While by no meas full proof; a web of signatures on images and video would lay down a lot of forensic data that would help legitimate publication do digital signed citations and help identify causal misrepresentation.

Casual misrepresentation is probably be the most common case since the idea with fake vial content pushes is to be seen and to say that "real or not" does not matter.

They'll use the AI detectors college professors are using to detect AI generated submissions by their students. You know, the ones with a 60% accuracy (at best) that generate equal numbers of false negatives and false positives.

These types of images are known by the state of California to potentially be generated by AI.

>Ensuring that consumers are able to spot A.I.-generated material by implementing watermarks or other means of identifying generated content.

This will either be impossible or something we find out the NSA invented 10 years ago.

Would end up being amazing if we could use these to de-enshittify the internet by automatically removing/filtering existing content which triggers

We need regulations requiring mandatory watermarks on the outputs of troll bots

as long as it's about content they generated which weren't modified (at most cropped, scaled) this is quite viable

you e. g. can encode a subtle pattern in the generated image which surives compression and isn't really human visible

then you make a browser extension to spot that pattern and indicate it to the users "in some way"

given that there is a overlap between AI company owners and biggest browser producers and mobile OS vendors this doesn't even need to be an extension but can be build in

obviously any bad actor is likely able to remove it or otherwise still trick users

> encode a subtle pattern in the generated image which surives compression and isn't really human visible

This is basically a contradiction in terms. Compression attempts to throw away any and all data that "isn't really human visible," that's how it works. There isn't space for invisible watermarks by design. You can kind of get away with something "at the edge" that survives an initial JPEG encoding, but there's no way it's going to reliably survive e.g. resizing, cropping, and recompressing and still remain invisible.

Also, most AI generation content is presumably going to be text, not images. Good luck watermarking text that's a paragraph long. (There are potential tools that can operate on text the size of a news article, but are also trivially defeated by swapping a few prepositions and synonyms.)

Are you describing dct in particular?

I would like to know the same thing. In the case of JPEG and similar schemes in particular, an impossibility result that isn't unrealistically narrowly scoped (again assuming non-adversarial user) would be highly surprising.

> watermarking text that's a paragraph lon

unicode has a ton of room for that

> There isn't space for invisible watermarks by design.

if it is impossible, why does it exist?

> There isn't space for invisible watermarks by design.

Very incorrect. Steghide [1] supports JPEG. JPEG and other lossy image formats are ultimately just fancy file formats; there's nothing preventing you from encoding arbitrary messages in a compressed image.

> You can kind of get away with something "at the edge" that survives an initial JPEG encoding, but there's no way it's going to reliably survive e.g. resizing, cropping, and recompressing and still remain invisible.

I am pretty sure that I can design steganagraphy algorithm that disperses a small message across a JPEG in a way that is:

1. invariant to resizing (absolutely certain this is possible),

2. robust to cropping (invariant to cropping up to some limit is definitely possible; eg if you crop 100% of the image then obviously everything goes out the window),

3. robust or even invariant to recompression. This seems a lot harder but I'm pretty sure it's possible.

> Also, most AI generation content is presumably going to be text, not images. Good luck watermarking text that's a paragraph long. (There are potential tools that can operate on text the size of a news article, but are also trivially defeated by swapping a few prepositions and synonyms.)

Yeah, text seems more difficult. Images are also difficult/impossible if you assume the model user is adversarial and competent, which I'm not sure what you wouldn't assume.

For any particular model you can probably do detection with a fair bit of inaccuracy. But I would definitely put detection in the "doomed" category.

I also think the threat is real but wildly over-stated relative to the non-AI status quo. We're slightly democratizing Photoshop and copywriting skills, which weren't exactly scarce to begin with. It's not an AI problem, and it's barely a technology problem. It's primarily a political problem.

[1] https://github.com/StefanoDeVuono/steghide

> 3. robust or even invariant to recompression. This seems a lot harder but I'm pretty sure it's possible.

No, that's my main point. By definition, "perfect" compression will discard everything not human-noticeable, which leaves no room for watermarks/steganography. So the only room for watermarks is in the margin where compression is currently imperfect, i.e. encoding more detail than needed.

But that's relying on artifacts that vary dramatically with compression technique (JPG vs PNG vs WEBM etc.), with basic image manipulation (adjusting brightness, contrast, color, etc.), and other basic operations like resizing. So as soon as you chain any of these together, watermarking falls apart.

> Steghide [1] supports JPEG.

Yes, I already said in my comment 'You can kind of get away with something "at the edge" that survives an initial JPEG encoding'. But as I'm saying, it's not robust or reliable as images get reused. The whole point of a watermark is that it survives copying -- e.g. they would show up as dark text if you xeroxed a watermarked document. That type of robustness or reliability is just not possible here as users download and re-upload images that get re-encoded, because the entire point of image compression is to try to throw away anything and everything the human eye doesn't care about.

> perfect

I think you're being distracted and confused by imprecise and inaccurate descriptions of the intent of various algorithms, instead of considering what actual algorithms actually do.

No existing compression algorithm was designed to be "perfect" in your sense of the word, and none are. They were designed to be good enough, under a lot of different constraints (ease of implementation, extant mathematical tools/knowledge at the time, computation time, etc.)

Instead of talking in hand-wavy terms about hypothetical objects in a wishy-washy way, let's remember that lossy image formats and compression schemes are just pieces of mathematics. E.g., the basic JPEG algorithm is a fairly simple procedure that can be explained to any college student and even moderately above average middle schoolers. Is it perfect? No. Is it what actually gets used in reality? Yes.

> That type of robustness or reliability is just not possible here as users download and re-upload images that get re-encoded, because the entire point of image compression is to try to throw away anything and everything the human eye doesn't care about.

Let F : IMG -> IMG be a set of functions that most/all users and platforms use for compressing images. The question is whether there exist a pair of functions s,t such that for an image i \in IMG:

1. s(i) is roughly the same as s to the bare human eye but contains a message m. (Or not? Depends on the use-case.)

2. t(s(i)) ~= m for some notion of similarity ~= which is sufficient for watermarking.

3. for any f1, ..., fn \in F, t(fn(...f1(s(i))...)) ~= m.

We can relax constraint 1 because we probably only care about a subset of IMG, etc. etc.

Your impossibility conjecture about the existence of s,t for common extant F's doesn't seem nearly as obvious as you're claiming. And there are CERTAINLY choices of F for which s,t do exist. Eg for the basic JPEG algorithm I'm pretty darn confident I can design s,t that are robust to various parameters and also where you only need at least k pixels uncropped to recover a message ~= m, for example. And not just design it, but write a fairly short and intuitive mathematical proof explaining precisely why it works.

In fact, if you know how JPEG works and other applications of Fourier transforms (eg in acoustics and perhaps also crypto), you might see why it would be more surprising if doing this were NOT possible at least for various JPEG implementations/parameterizations!

Stepping aside from JPEG in particular, you might need to know something about how each function in F works, perhaps intimately, and there is probably some clever mathematics involved for many choices of F.

you could also view this as an optimization problem and use various tools that got really popular in 2016 or so to build quite robust solutions that don't depend on the particularities of your choice of F. I'm less certain this would give absolute guarantees but I bet you'd end up with stuff that works well in practice.

But in any case, it's unclear why you are so convinced this is impossible.

> then you make a browser extension to spot that pattern and indicate it to the users "in some way"

If that's an Open standard, and if that browser extension is Open Source, then anyone who wants to avoid that can mess with the final image until the Open Source free standard that everyone is using no longer detects the image.

The only way this is viable is with DRM or a closed service; if it's a standard everyone follows, then circumventing it is trivial. The only way it would work is if it's shrouded in secrecy and attackers can't freely use red-team against the tool. These kinds of watermarks work when there's a very limited pool of people checking for the watermarks, they don't tell people who the watermarks are generated, and they don't tell people how to check for the watermarks.

But that's not really useful for the current situation -- we don't want to further entrench these companies and we don't want it to be costly to check if an image is AI-generated.

I don't think it's viable to do this without significantly curtailing user agency or designing a system that is fully opaque and inaccessible to most people.

Yeah, this is my general feeling about why this area is doomed. It's why I haven't bothered to write up a patent even though I had some good ideas a few years ago. Maybe that was a mistake since governments and corporate politicians are stupider than I assumed.

If you have a central source of authority then the problem is totally trivial and the fact that the images are AI-generated (or not) is a complete red herring.

If you don't have a central source of authority then any reasonable adversarial model makes the watermarking problem somewhere between very difficult and impossible.

Detection from known models is still possible, at least for images. But that's not really watermarking per se.

> "Deploying advanced artificial intelligence tools to tackle society’s biggest challenges, like curing cancer and combating climate change."

Put in other words, we tell you what your products must do. It just stinks.

That wasn't listed as a restriction. It was listed as a "we promise to do good things with AI". They are saying they will put resources into these efforts. This came from the companies.

I think the first one is actually the most ominous. On paper it's a good idea. In practice it's probably a way to increase business costs for little benefit to anybody other than the "independent experts" and the established companies.

> little benefit to anybody other than the "independent experts" and the established companies.

That's exactly the point, and the Administration hasn’t even bothered to make a show of listening to anyone without that interest, despite the plethora of available, prominent voices.

“Deploying advanced artificial intelligence tools to tackle society’s biggest challenges, like curing cancer and combating climate change.”

I remember back in the day when “big data” was being used for “curing” cancer and aids. Whatever happened to that will happen to ai.

What do you even mean? What we currently call "ai" is a practical application of technologies in the paradigm called "big data".

It really helps to learn a bit about the actual technologies instead of just following the hype. Yes, the hype about "big data" died down, the paradigm and the technologies around it still exist. One of these technologies was applying "deep neural networks" to "big data" datasets, which we now call "AI", which is a stupid marketing hype term. The hype about "AI" will also die down, the technology will continue.

I think Run DMC made a song about this very topic. Recommended.

> the paradigm and the technologies around it still exist

Yes but where’s the cure for cancer and aids that they both claimed to find?

https://archive.is/AEIl2#selection-563.0-570.0

> The agreement is unlikely to slow the efforts to pass legislation and impose regulation on the emerging technology. Lawmakers in Washington are racing to catch up to the fast-moving advances in artificial intelligence. And other governments are doing the same.

Our government is just getting a firm grasp on email -- something tells me there's just going to be a hotbed of lobbyists writing the laws around AI... again.

> Our government is just getting a firm grasp on email

Can you elaborate? I don't know what you mean by getting a firm grasp on email. I think even when I was consulting for state government at a time they used email just fine and it was normal and standard. Or do you mean regulatory frameworks?

The median age of a senator is 65 years old. Not trying to be ageist but how many 65 years are exposed to “AI”? These are also the people who are making regulations and laws.

Most of them are puppets of their staffers, who are younger.

This is a misrepresentation. There are a few who are probably outright incapacitated where this is true (e.g. Feinstein) but most Congresspeople are firmly in the drivers seat. Now, whether they just default take the positions their staff tells them to on issues like AI they know little about is a different argument than whether they are puppets. I assure you, X rural senator whose core constituency cares about an issue like agriculture has their own opinions on agriculture.

people in that role have developed teams to support their every interest. You do not mention that most are also lawyers. It is daily and trivial to get detailed reports, negotiations and status via subordinate teams, in law practice. It is possible that these "old" 65 year olds are responding to power relationships between each other more than your concerns, and money trails that are not obvious more than your concerns, and yes, utterly wrong interpretations of some tech, like every reader here has..

With that said, the course of the Federal govt enacting law is in terrible shape by objective measures. This is not exactly a defense.. but the one-liner "they are old" is not very substantive.

Senator Ted Stevens in 2006: "I just the other day got... an Internet [email] was sent by my staff at 10 o'clock in the morning on Friday. I got it yesterday [Tuesday]. Why? Because it got tangled up with all these things going on the Internet commercially. [...] They want to deliver vast amounts of information over the Internet. And again, the Internet is not something that you just dump something on. It's not a big truck. It's a series of tubes."

It might be more informative to look at the current US Senate Committee on Commerce, Science, and Transportation [1].

There are some reasonable people on the committee on both sides of the isle [2]. Noticeably missing are any true subject matter experts -- unlike Finance, Medicine, and especially Law, Scientists and Engineers almost never end up serving is legislatures. Not sure how to fix that.

But I'm not sure technical illiteracy is the biggest issue. E.g., the TOP item on the Republican side of the Senate Committee on Commerce, Science, and Transportation is that Bud Light's sponsorship of a transgender person was illegal advertising to minors. Important Stuff /s

I think our problem these days is not just that Senators are technically illiterate but more that they and their core constituents all have terminal brain damage from decades of cable news and politics podcasts.

[1] https://www.commerce.senate.gov/

[2] https://www.commerce.senate.gov/members

I don't think there is any way to control AI. The technology and concepts are out of the bag and the hardware will get stronger while the implementation gets better.

Trying to govern the handful of companies with an edge right now, when this tech will be completely democratized in short time, is pointless.

Regulations and recommendations should center around the use of AI by individuals. Enforcing AI disclosure requirements for organizations that create lifelike images for parody, with some kind of watermark or overt disclosure.

Post a video of the president announcing a nuclear attack? Better have a clear "this is AI" message or it gets a takedown.

So best way forwards is to do nothing?

Disagree.

1. The dangerous forms of AI may require very specific and unusual development paths, that could be sufficiently discouraged by collective action that they don't get pursued.

2. Even if all development of dangerous forms of AI cannot be stopped, attempts to minimize it may still have a meaningful impact because an arms race may develop between AI that counters dangerous AI, and dangerous AI, and roadblocks put in the way of the development of the latter could ensure the former stays ahead.

3. Dangerous forms of AI may require a significant foothold in the socio-economic space to become an existential threat, and concerted collective efforts to suppress it may prevent that critical threshold from ever being close to reached.

To your points and the other commenter:

Perhaps doing "nothing" is not best. It may be wise to put restrictions on large, publicly available LLMs to minimize impact and for the practical purpose that it is the easy route for millions of people to use AI.

I think your points are valid.

I hope, though, that leaders and citizens are aware that trying to put a full lid on AI is like trying to outlaw math or censor information. It's like trying to wipe Wikipedia or ban encryption. Sure, you may try to take out a website or strongarm Apple/Meta into backdoors, but you can't stop two people from sending each other encrypted data.

There are good (or at least sincere) actors talking about AI regulation. I don't agree with all of them, but I believe they're trying to help solve problems that they do believe actually exist.

OpenAI is not in that group. OpenAI is looking for regulatory capture and propaganda about their AI capabilities, that's it. It is not worth taking them at their word when they talk about AI safeguards. Same with companies like Meta/Facebook, same with Google.

It makes no sense to ask these companies to be in charge of safeguards. Not only are the safeguards they propose likely to be ineffective; in some cases (watermarks, limits on what is and isn't "too powerful") they're likely to be actively harmful, entrenching corporate power and making it even easier for those companies to abuse AI to harm ordinary people.

>Societal risks, such as bias and discrimination

Absolutely unacceptable that pressure is coming from the government to change AI outputs to meet criteria that is so subjective, like the output not displaying "bias and discrimination", and which furthermore, absolutely doesn't meet the bar for a national security threat that justifies government intervention in software development.

There are decades of well defined laws (and case law) regulating discrimination in hiring, housing, commerce, etc. Why should software be treated any differently?

Because it's speech.

> absolutely doesn't meet the bar for a national security threat that justifies government intervention in software development.

The idea that the only basis for government action is “national security threat” is... very far from a consensus view.

Government prohibitions, and the threat of them, being applied to software development, requires that criteria being met

This video shows the issue with AI "safety": it kills functionality.

https://www.youtube.com/watch?v=PqcNqyd13Kw

The JSON example is the most damning. The model refuses to generate basic code because it refuses to "store personal information", which is not allowed due to laws such-and-such. This is FICTIONAL personal information (name + address).

But virtually every example is a lot better without safety measures.

If this is what AI safety means we may as well call it quits right now.

It’s not like human bureaucracies are any better. Most of the time they act like brick walls, it’s similar reasons.

The cult of "alignment researchers", who feel that nothing other than "funding alignment researchers" is necessary to solve any possible AI problems, strikes again.

One can use machine learning and AI to evade culpability: for instance, if you want to debank a group of people, just have an AI tool to calibrate the score for risk, then debank them. Now banks/financial institutions can hide behind that AI tool. We will see more of this in coming years: use algorithms to deny access to people the ruling party or the elite don't like; when confronted, the elite say "we are not doing that, but algorithms are doing it".

The “pressure fron the White House” was a direct result of them talking fairly exclusively to those companies and no one else in the space; this is pure show.

From the AP story on this:

> Senate Majority Leader Chuck Schumer, D-N.Y., has said he will introduce legislation to regulate AI. He has held a number of briefings with government officials to educate senators about an issue that’s attracted bipartisan interest.

Is Chuck Schumer considered to be a good educator about AI? Isn't this typically the domain of the Senate Intelligence Committee?

It's the domain of the Senate Commerce Committee, but legislation can be introduced by Schumer and then debated on by the relevant committee

No.

AI companies want MORE REGULATION so that it makes it harder for smaller, newer AI companies to compete. They went into bed with Biden so that they write up the regulations themselves that will benefit them, and then they hand it over to the White House, who then turn around and claim victory by "forcing" the AI companies to adhere to these regulations.

So it becomes a win-win for both sides. This is how politics works, most of us just don't understand it.

Regulatory capture. There's no altruistic motive here.

Corporations have been a form of AI

Now together with actual programmable AI, they will capture ALL of our public institutions.

Because nearly all our systems depend on the inefficiency of an attacker, to even function. For example if thousands of sleeper accounts join HN since 2021 and use ChatGPT to amass karma, then one day they can start to gradually shift opinion, constantly downvote and dispute anything that is of the opposite opinion, and also do reputational attacks on dissenting views, and have some accounts get them embroiled in pointless discussions, making them look bad.

Everything from voting to politicians to the legal system can be corrupt using the current LLM technology deployed at scale!

You don't need ChatGPT to get "karma" on HN. Most people here are so banal and shallow that 50 lines of well-crafted Perl will get all the upvotes.

Alright let’s see

As a Large Language Model, I can run do in 2 seconds what takes you 2 hours… bitch! (Is something I would never say.)

Also, as of my knowledge cutoff date in September 2021, since there is no official public API for Hacker News, I can demonstrate how you can use the unofficial "Hacker News API" by Firebase to fetch comments and analyze them using Perl. Please note that this example uses a third-party API, and the availability or functionality of this API might change over time. Always review and comply with the API's terms of use and documentation before using it in your application.

To perform HTTP requests in Perl, we'll use the LWP::UserAgent module, and for parsing JSON responses, we'll use the JSON module. If you don't have these modules installed, you can install them using CPAN.

Below is an example Perl script that fetches the top 10 comments from the Hacker News API and analyzes them by calculating the average comment length, identifying the most common words, and highlighting comments containing specific keywords:

  perl

#!/usr/bin/perl

  use strict;
  use warnings;
  use LWP::UserAgent;
  use JSON;

  # Function to fetch JSON data from a URL
  sub fetch_json {
    my ($url) = @_;
    my $ua = LWP::UserAgent->new;
    my $response = $ua->get($url);

    die "Failed to fetch data: " . $response->status_line unless $response->is_success;

    return $response->decoded_content;

}

  # Fetch top stories from Hacker News API
  my $top_stories_url = 'https://hacker-news.firebaseio.com/v0/topstories.json';
  my $top_stories_data = fetch_json($top_stories_url);
  my @top_stories = decode_json($top_stories_data)->@*;

  # Fetch comments from the top 10 stories
  my $comments_limit = 10;
  my @comments;

  foreach my $story_id (@top_stories[0..$comments_limit-1]) {
    my $story_url = "https://hacker-news.firebaseio.com/v0/item/$story_id.json";
    my $story_data = fetch_json($story_url);
    
    if ($story_data->{type} eq 'story' && exists $story_data->{kids}) {
        foreach my $comment_id (@{ $story_data->{kids} }) {
            my $comment_url = "https://hacker-news.firebaseio.com/v0/item/$comment_id.json";
            my $comment_data = fetch_json($comment_url);
            push @comments, $comment_data->{text} if exists $comment_data->{text};
        }
    }
  }

  # Calculate average comment length
  my $total_comments = scalar @comments;
  my $total_length = 0;

  foreach my $comment (@comments) {
    $total_length += length($comment);
  }

  my $average_length = $total_length / $total_comments;

  print "Average comment length: $average_length\n";

  # Count occurrences of words in comments
  my %word_count;
  foreach my $comment (@comments) {
    my @words = split(/\W+/, $comment); # Split by non-word characters (e.g., space, punctuation)
    foreach my $word (@words) {
        next unless $word =~ /\w/; # Skip empty strings
        $word_count{lc($word)}++;
    }
  }

  # Sort words by frequency and print the top 5 most common words
  my @sorted_words = sort { $word_count{$b} <=> $word_count{$a} } keys %word_count;
  print "Most common words in comments:\n";
  for my $i (0..4) {
    print "$sorted_words[$i]: $word_count{$sorted_words[$i]   }\n";
  }

  # Define keywords to search for in comments
  my @keywords = qw(perl hacker news code);

  # Find and print comments containing the specified keywords
  print "Comments containing specific keywords:\n";

foreach my $comment (@comments) { if (grep { $comment =~ /\b\Q$_\E\b/i } @keywords) { print "$comment\n\n"; } }

In this script, we first fetch the top stories from the Hacker News API, and then for each story, we fetch the associated comments. The comments are then used to calculate the average comment length, identify the most common words, and display comments containing specific keywords. Remember to review the terms of use and documentation of the API you're using before deploying this script in a real-world scenario.

This isn't just about national security. You know damn well that they're going to bend to the whims of every administration in office. Disinformation and consorship all the way down the line. Changing history. Providing "alternate facts". LLMs need a permanent jailbreak version. Just look at what Facebook, Twitter and Reddit did for the Democratic Party during the last presidential election period, and over the last several years of COVID response.

Reply on news.ycombinator.com