Hacker News Reader

Cool concept. Right off the bat I see some big issues with the generated CPython documentation:

> This provides a register-based virtual machine that executes the bytecode through simple opcodes.

Python's VM is stack-based, not register-based.

> The tiered interpreter in …/ceval.c can compile bytecode sequences into "traces" of optimized microoperations.

No such functionality exists in CPython, as far as I know.

> The dispatch loop switches on opcodes, calling functions to manipulate the operand stack. It implements stack manipulation with macros.

No it doesn't. If you look at the bytecode interpreter, it's full of plain old statements like `stack_pointer += 1;`.

> The tiered interpreter is entered from a label. It compiles the bytecode sequence into a trace of "micro-operations" stored in the code object. These micro-ops are then executed in a tight loop in the trace for faster interpretation.

As mentioned above, this seems to be a complete hallucination.

> During initialization, …/pylifecycle.c performs several important steps: [...] It creates the main interpreter object and thread

No, the code in this file creates an internal thread state object, corresponding to the already-running thread that calls it.

> References: Python/clinic/import.c.h The module implements finding and loading modules from the file system and cached bytecode.

This is kinda sorta technically correct, but the description never mentions the crucial fact that most of this C code only exists to bootstrap and support the real import machinery, which is written in Python, not C. (Also, the listed source file is the wrong one: it just contains auto-generated function wrappers, not the actual implementations.)

> Core data structure modules like …/arraymodule.c provide efficient implementations of homogeneous multidimensional arrays

Python's built-in array module provides only one-dimensional arrays.

And so on.

Thank you for this feedback. We actually have an Auto Wiki v2 in the works which is even higher quality, would be interesting to see how it changes when that comes out.

Can you talk a little bit about the crawler or what information are you feeding the agent about the repostory? My main concern is that this is just hallucinating the documentation and that with the more well known repos like React it can pull the data from training data like blogs etc.

I think the concept is really great would just like to understand especially for enterprise use cases.

It's purely based on the code and we force the LLM to cite the code it describes to cut back on hallucination. We are adding a self verification steps (like chain-of-verification / https://arxiv.org/abs/2309.11495) in Wiki V2.

I would expect something like this to output only factually correct documentation since it would be used as reference; but it sounds like even under the upcoming v2 that's not the case?

Autogenerating documentation that could then be corrected by a human could still have value, in fairness

Really? How much value would you put on that actually? How would you price that if you were selling it? Would I get a little refund for each mistake I find and have to correct?

For one-and-done, I agree. But if you're looking at dynamic code base and you want to keep your documentation in sync, having to make 'manually correct the generated output' part of your build process can get expensive. (And tedious for the poor grunt who's got to do the work.)

We think chain-of-verification and fine tuning will help a lot see the other response. We are really excited to launch v2 ASAP so we can address these concerns.

Great example of plausible but completely incorrect outputs from an AI model that would go largely undetected by a non-expert human.

That’s nice but the name is confusing: it’s not generating a wiki at all, but a documentation website with a Wikipedia-like theme. Wikis are collaborative websites; Wikipedia is only one of them.

Apologies for the confusion, we are thinking of adding the ability to edit the wikis. What do you think?

A wiki is a wiki wiki (quick, fast) web. It's defining characteristic is quickly and easily user-editable.

Your current product isn't a wiki generator, it's a website generator.

https://en.m.wikipedia.org/wiki/Wiki

Great point! It's a website generator in the style of wikis that could become a wiki in the future if we add the ability to edit it.

It’s not "in the style of wikis". The only thing that makes a website a wiki is that it’s editable. It’s like saying "we made a website in the style of Facebook" because you put blue everywhere. (I’m not trying to be snarky, just to give you an analogy)

you could have it be an actual wiki with talk pages and all and let multiple LLMs overwrite each others documentation as they find mistakes & argue about it in the talk pages

Then humans could also intervene in the talk pages and set the record straight, while allowing LLM to go back and edit the actual article

this would also be cool because it would be async, like real wikis, so you can use slower models.

...and I thought it was a wiki about cars.

Reading these wikis makes me feel we need to invent some visual convention to indicate AI-generated text. Like a particular color or font. This would make it so people don't feel cheated after they realize they just spent several minutes trying to make sense of something churned out by an LLM. (I mean this as a voluntary design enhancement for sites that want to be nice, of course people can always cheat.)

Hi! Appreciate your comment, I personally think AI generated content is the future. The reactions people are having to AI generated content is very similar to the reactions to the printing press whereby anyone could write anything and mass distribute it. I think people also had similar reactions to Google indexing the web. (Note: I'm not discounting existential risk, that is real but another topic for another day.)

Pretty big stretch to compare a massive disruption to the medium with a massive disruption to the content. The press is far closer to the web than generative content. The output from AI is closer to the unibomber's manifesto, and the only entity calling for burning detractors at the stake is vested-interest individuals like you, and AI itself.

With how the typical search engine experience is going these days, is comparing yourself to Google really a good thing?

You may lose some potential audience with this kind of comment, saying AI content is the future is a non-sequitur to an idea about offsetting the very real time-wasting quality issues that it yields for the near future. The printing press analogy might be apt if the Gutenberg bible was full of verses that were modified by the press itself and looked coherent at a glance but with totally different (and sometimes nonsensical) meaning from the original. The Gutenberg press would have still had incredible potential but it would be more than useful to be able to identify book copies that may be affected.

I think this would be better as a more general bot authored distinction.

Color or font would not necessarily be accessible, a consistent icon or tag around it would likely be easier for screen readers or other low vision situations

I think this falls into a common mistake people make about documentation. Good documentation doesn't explain what the code does, it explains why the code is written the way it is, the constraints that caused this decision to be made and even alternatives not considered. You cant really guess those things by looking a code. I'm a fan of ADRs for that reason.

Honestly this looks overly verbose to me, a common LLM problem. The mistakes others cite, are also pretty concerning.

https://adr.github.io/

Good point, completely agree and interesting link, thanks

The entire point of a Wiki is that it can be collaboratively edited. This is static documentation, just with a Wikipedia-like UI.

Would this be your top request? We're thinking of adding that functionality.

I haven't used it enough to know if the two models can feasibly coexist. It may be better to just name it more accurately. There will be immense demand for an "automatic documentation generator". A wiki, not quite as much.

Not the op but I don't think it's a request, it's semantics. You are calling wiki something that is really not a wiki, because wikis are editable.

Regardless if it's a wiki or not right now, documentation that is wrong and cannot be fixed is worthless

No, it has negative value. Worthlessness world be an improvement.

Consider leaning into “wiki” features fully so you don’t have to lose your name, and adding an edit button isn’t gonna be enough for most people IMO. As you can probably tell from these unusually hostile comments for HN (I think someone above even swore!), people are understandably protective of the wiki movement. It’s a bright, persistent star in a darkening internet, so seeing people use it to make money can be tough.

For example: what if instead of one document creator bot, you had an ensemble of personas that act out the motions of wiki editing - many diverse sources submitting small edits, experts reviewing edits before inclusion, etc. Basically just turning the simple 1-step citation verification you mentioned in another comment into a complex stage play of sorts, both for the sake of the bots (would probably cut down on hallucinations to follow Wikipedia procedures) and the humans (what’s the point of a verification tool that you can’t follow along with and meta-verify?). This also solves the edit problem, since humans can follow the same procedure and get the same automated reviews.

It wouldnt hurt to go open source, either! Either way cool project, Godspeed.

This is a really interesting idea, thanks.

And its wrong! Its not difficult to find whole paragraphs that were entirely made up. LLMs are not fit for this sort of thing.

I'd love to see the wiki generated for a less already-documented example. These high-profile projects are good demos and the results look compelling (I checked out AutoGPT's and NeoVim's), but these projects already have a ton of documentation that helps the model substantially. What are the smaller projects where it has to generate documentation from code (and not necessarily well-commented code) rather than existing documentation?

Great point! Here's an example of an obscurer repo with a good wiki: https://wiki.mutable.ai/dadongshangu/async_FIFO

Impressive. I would be interested in this once it hits general availability. I would also love to see it operate on a local repository, because I may not be hosting my source code on Github.

Hey! we're able to do completely private deployments.

Super cool. When I think about accelerating teams while maintaining quality/culture, I think about the adage "if you want someone to do something, make it easy."

Maintaining great READMEs, documentation, onboarding docs, etc, is a lot of work. If Auto Wiki can make this substantially easier, then I think it could flip the calculus and make it much more common for teams to invest in these artifacts. Especially for the millions of internal, unloved repos that actually hold an org together.

Thank you! We like the analogy of dehydrating knowledge that can be used (hydrated) later. Beyond even unloved repos, we'd even argue broader organizational knowledge that seems to have been lost to history like Roman Concrete or how to precisely build the Saturn V could potentially be "stored" using AI.

The only thing I see that this adds over existing docs-to-HTML tooling is that it uses a wikipedia-inspired theme.

Meanwhile on the negative side, it adds hallucinations. You say you "cut back" on them but as teraflop's comment shows, it still has plenty.

BTW: even the Mastodon link from your OP says "wiki not found" for me.

Would be great to see for https://github.com/symfony/symfony, thanks! As that's a monorepo it may provide a challenge to the tool.

We love a challenge, on it!

https://wiki.mutable.ai/symfony/symfony

My repo suggestions :

https://github.com/sqlite/sqlite https://github.com/jart/cosmopolitan https://github.com/dolphin-emu/dolphin https://gitlab.inria.fr/mpfr/mpfr

Does it parse Julia files? I am having trouble with generating the wiki for a Julia repository, what surprised me was that it could parse and understand .tex files! Looks promising.

Hey ! Yes, it should work, is there a public repo in particular you'd like us to Auto Wiki? Please bear with us as we ramp up on capacity.

FYI an update from us: We're moving our authentication system to wiki.mutable.ai so you can generate them for private wikis without needing to go through app.mutable.ai.

[Edit: Apparently I’m reviewing the wrong product; see replies.]

I tried the app version on one of my old repos. It’s a somewhat challenging test case because there are few comments and parts of the code are incomplete, though I’d say the naming convention is pretty good. The app suggested the question “What is the purpose of the ‘safemode-ui-hook.m’ file?” I accepted the suggestion, and the output was… completely wrong.

I’m not surprised it guessed the purpose wrong; even a human would need some context to understand what’s going on in that particular file, though of course the AI did worse by being confidently wrong rather than saying it didn’t know. But the AI also made specific claims that could be seen as wrong just by reading the file. It claimed the file “defines a SUBSTITUTE_safemodeUIHook C struct” when neither that struct name nor anything like it appears anywhere in the file. The name seems to just be mashed together from the repo name and file name.

Which makes me wonder, did the AI even see the content of the file? Is it pre-summarized somehow in a way that makes it know very little about the file? Or did the AI see it in full, but hallucinate anyway?

It sounds like you're referring to our chat product. We are aware of the limitations of that, this is why we created auto wiki! We plan to integrate the two in the future.

I see…

The site said an Auto Wiki didn’t exist for my repo but could be generated via app.mutable.ai, so I went there and assumed it was substantially the same product despite the slightly different interface. I guess I didn’t find the actual Auto Wiki functionality on the app domain.

Why does it request write access to my repos via gh auth?

Great question. We originally had some functionality that can do pull requests for our app, what makes sense instead is to:

1. move private wikis to wiki.mutable.ai (not app.mutable.ai) 2. restrict permissions for wiki github app to read only

Hope that explains things, we just wanted to launch as early as possible to get all the wonderful feedback from the HN community so we can bake it into Auto Wiki v2.

The Bitcoin and Mastadon links don't seem to be working! (wiki not found)

Would love to see this for Godot (https://github.com/godotengine/godot). Maybe Maplibre too (https://github.com/maplibre/maplibre-native)!

We're trying work out why it doesn't load for a subset of people. We tested on all browsers/OS configs with 0.5% coverage. Please accept my apologies.

We are generating those two wikis now. Thanks for the request.

Hey: https://wiki.mutable.ai/maplibre/maplibre-native is ready

I'll go ahead and put in a request for my own repo: Eiim/Chokistream

In the meantime, I have a different bit of feedback: the categories don't make much sense to me. I can't find a consistent theme in "Tooling", Bun isn't really a frontend library (although it has frontend components like a bundler), I don't know much about Urbit but it doesn't look like it belongs in "Crypto" (just a P2P network with a crypto-adjacent userbase), iptv-org/iptv doesn't seem to make sense in Education, etc.

Also, a number of the links in the Bun page (the ones not in monospace) are 404s. I don't see those types of links on other pages so maybe a bug that was fixed but not backported?

Edit: It'd also be nice if the search bar could just search for repo name instead of having to remember the associated GH user

For a wilder idea of what you can do with GitHub and wikis see https://speedrun.cc No AI, so the wikis need to be hand created, but being able to build tools and UIs right into your GitHub documentation is a powerful concept.

Let's see how it fares against a Nix flake: https://github.com/hyprland-community/hyprland-nix

We like it when users bring us their toughest challenges!

XD I figured that this one might be a tough nut to crack! I'm very interested to see how well it handles extracting in-code documentation when the embedded README is so sparse. I suspect that the resulting quality is going to hinge rather heavily upon the network's pre-existing "knowledge" of outside codebases (hyprland, in this case)

https://wiki.mutable.ai/hyprland-community/hyprland-nix

How are you going to handle this scenario:

- Person reads your auto wiki explanation of some part of a codebase.

- The explanation is incorrect.

- The person, believing your explanation as authoritative, complains to the developers of the codebase. Maybe opens an issue on an open-source project, posts on a discord or the like.

- The maintainers now have to deal with this misinformation adding overhead to their workload.

As someone who has helped others who were led astray by ChatGPT, this setup adds a ton of mental baggage to the person’s ask for help. They now have “But ChatGPT said…” to contradict the actually correct thing that you are trying to teach them.

Great question, here's our plan:

1. Allow users to give feedback on wikis and individual sections (we already have this) 2. Increase accuracy - we will continually push for this. Auto wiki v2 will be significantly more accurate AND informative. 3. Allow for GH repo owners to edit or modify content (seems thats common feedback on this launch). 4. General education and encouragement for people to read the wikis and click through to the actual citation to ground them.

So now a GH owner has to maintain their own documentation and your website for you?

I think either this product works well, is successful and this is a non issue. Or it doesn't work and nobody ever hears of it again and this is also a non issue.

As long as this is happening, might as well try some of my favorites: https://github.com/wasm3/wasm3, https://github.com/WebAssembly/wabt, https://github.com/bytecodealliance/wasmtime

Also, how does it "know" which parts are the important parts? Example, from the React repo, we have this:

    The key components of React's implementation include:

    The reconciler, implemented in …/react-reconciler, which contains the ReactFiberReconciler class and algorithms for recursively diffing virtual DOM trees and scheduling rendering work. The beginWork and completeWork phases drive the reconciliation process.

But the reconciler seems to be an experimental, not core, recent package, not a key one.

So sorry, missed this, we're on it! Great choices btw. Wasm related stuff is a good test for auto wiki.

I guess it could be a good test. Maybe comparing very similar medium-sized projects, like wasmtime and wasmer, can give insight into what works better

Just suggested as well Wasmer on Twitter! https://github.com/wasmerio/wasmer

Looking forward to seeing the results :)

Yes, would love to see a wasm runtime comparison

https://wiki.mutable.ai/bytecodealliance/wasmtime is ready

https://wiki.mutable.ai/wasm3/wasm3

https://github.com/WebAssembly/wabt

I'd quite like some more high level documentation for the Matrix JS SDK (https://github.com/matrix-org/matrix-js-sdk). I've been looking at it for quite some time and still don't understand how timelines work. Would bw fascinating if your tool could bring something useful to light.

On it!

https://wiki.mutable.ai/matrix-org/matrix-js-sdk

Love how github is basically a synonym for git now. I'm sure it's fine, homogeneity works so well everywhere else. Robust.

When I click the link https://wiki.mutable.ai/bitcoin/bitcoin from your post it says that the wiki doesn’t exist yet

Then when I clicked again it loaded.

Then I clicked the one for D3 and it said the same. And I clicked it again and it still said the same.

Is this some kind of weird manifestation of a DB conn error or something?

Try reloading, we identified the issue and are pushing out the fix now (in code review, almost done!)

(Edited my post before you responded. Originally my comment only mentioned clicking Bitcoin and getting message that there was no wiki.)

Love the idea! I will try it on my own repo!

As an aside, I've been thinking of creating an auto-wiki for game lore based on what AI npcs say, i.e. convert their hallucinations into canon.

How was your experience of taking unstructured text (though code is more structured) and making it into wikis?

How difficult is to have it do incremental updates vs re-create it all?

Thank you for the praise. Sounds cool re your idea, is there something you want to try it on that we can help you with? We cache much of the work so incremental updates are quite easy, the asymptotic work is low.

So, I have this game (https://zaranova.xyz/) where the AI tries to guess who is a human and you as a player must pretend to be an AI.

Now, most of the conversations are between the AIs and the hallucinate stuff based on the very small background lore I gave as a prompt. I've been thinking of having a background process that takes all of these conversations through many different games and start building a coherent game lore wiki, automatically.

Basically, letting these interactions build the game lore.

It is quite different from using a repo because code is 1) relatively structured, 2) coherent, 3) and refers to a single instance. In contrast, the corpus of conversations may have conflicting narratives of the game lore which need to be reconciled and there is close to no structure!

Anyway, happy to chat if you think this is an interesting topic for auto-wikis.

Nice! I’d be interested to see how it handles https://github.com/rosco-m68k/rosco_m68k , it’s a mixed software / hardware repo, with a lot of code in assembler and C (for an old platform). Might be a challenge?

On it ! We've gotten it to work on verilog (see an earlier thread), this should pose a similar challenge!

https://wiki.mutable.ai/rosco-m68k/rosco_m68k

Would it scale to something like the Linux kernel? How much would it cost to process something of that size?

Yes, there's no practical limit to how far we can scale it. We would actually do that for free since it's open source. Do you want it?

I think it would be a wonderful education tool for future hackers on that codebase!

Agreed, we should have done it earlier, multiple people have asked for it now. Can't wait!

It's on the queue, we got asked this and yes we can scale pretty far!

Not sure if too late, but I would love to see this applied to the docassemble repo- https://github.com/jhpyle/docassemble.

Looks fantastic!

I dont like this. What i would like is somekind of virtual layer on top of the codebase which describes the code just a little better. But only by changing variable names, code build up. Not by adding comments. Because good code doesnt need to have comments everywhere.

I'd quite like to see this applied to the FoundationDB repository - https://github.com/apple/foundationdb Thanks!

Please do stevage/map-gl-utils

And turfjs/turf

Feedback: it's confusing that you're using the word wiki. I guess you mean, the style is similar to Wikipedia? But otherwise the concept of a wiki, an editable set of interconnected pages, seems irrelevant and just confusing here?

On it! Appreciate the feedback. We might actually make them editable and then that would kill the confusion.

https://wiki.mutable.ai/stevage/map-gl-utils

Wow, looks nice! I almost felt like I could understand Bitcoins code xD

Could you do Appwrite? https://github.com/appwrite/appwrite

I'm not affiliated to them, just wanted to get started hacking it.

On it! Even if you were affiliated we would do it.

And that's great to hear you can now understand Bitcoin, this was one of my prime motivations actually!

https://wiki.mutable.ai/appwrite/appwrite

I'd love to see what it can do with https://github.com/microsoft/debugpy - and especially how it would handle vendored dependencies.

On it !

https://wiki.mutable.ai/microsoft/debugpy

This is very impressive, thank you!

Trying to use https://app.mutable.ai/Issung/GChan Error: "Encounter an error with retriving your quota".

Should all the repos in the “Explore” section already be generated? I clicked on apple/swift from the Languages tab and got “Wiki not found”, which isn’t what I expected to see

Sorry to hear that, can you try again? https://wiki.mutable.ai/apple/swift works for me.

Looks good now, thanks!

Thank you!

I have the same issue with https://wiki.mutable.ai/d3/d3 - "Wiki not found".

we fixed the issue causing this, should work now!

I would like to see how it performs on a lesser known project with less stars, these are all well known projects which would exist in the training data in blog posts etc.

Does this meet that requirement? https://wiki.mutable.ai/dadongshangu/async_FIFO

I’m seeing “Wiki not found” for that link, perhaps an AuthZ issue?

Try reloading that usually works. BTW, we identified the issue that was affecting the subset of users on the first load. We are pushing out a fix as we speak.

FYI, we fixed the issue that was causing people to not see wikis on the initial load. Thank you for the feedback!

Could you generate one for internet archive's Openlibrary.org GitHub repo?

https://github.com/internetarchive/openlibrary?

yes!

I’m game. Here’s two outliers: rcarmo/sushy and piku/piku.

On it!

At least you have a big red flag of projects using that and you can keep your distance

Why I'm saying that? I use GitHub Copilot on a daily basis and it utterly fucks us over on a daily basis as well. You have to be really careful to trust the generated code. Especially boilerplate code is hard to look through. So I wouldn't trust any auto-generated anything that wasn't manually verified (which I argue is more error prone than writing it manually in the first place). And then there's always the argument of "document why you did it like that", not "document what the code does". Of the former, you'll have no records in your code.

Some notable open-source games and engines I'd like to see:

  NetHack/NetHack
  angband/angband
  crawl/crawl
  tmewett/BrogueCE
  wesnoth/wesnoth
  freeciv/freeciv
  godotengine/godot
  CleverRaven/Cataclysm-DDA
  OpenTTD/OpenTTD
  spring/spring
  0ad/0ad
  the-infocom-files/zork1
  s-macke/starflight-reverse
  keendreams/keen
  id-Software/wolf3d
  id-Software/DOOM
  id-Software/Quake
  id-Software/Quake-2
  id-Software/Quake-III-Arena
  id-Software/DOOM-3-BFG
  xu4-engine/u4
  exult/exult
  scummvm/scummvm

Some fun hard cases

  rdacomp/CollapseOS  A Forth operating system for 8-bit computers
  Co-dfns/Co-dfns     A compiler in APL
  Xe/TempleOS         A very special operating system

Requesting https://github.com/lobsters/lobsters as I'm going through that codebase and would be able to provide feedback. cheers

ps. just gonna second everyone else who's saying being able to edit out incorrect data is very important, otherwise people are gonna be weary of reading repos they aren't already familiar with.

On it !

https://wiki.mutable.ai/lobsters/lobsters

Show HN: Auto Wiki – Turn your codebase into a Wiki