rhdunn

Audio transcription is a hard problem that is inherently non-deterministic and probabilistic due to ambiguities in the speech due to:

1. accents -- Especially around mergers (cot-caught [AmE], trap-bath [BrE] vs palm-bath [LondonE], pin-pen [Some AmE]). These can even be hard for native speakers -- try transcribing a broad Scottish, London, Brooklyn, or Indian accent and see how well you do.

2. sound/phoneme variation based on surrounding phonemes -- It is common for the 'n' sound to be realised as an 'ng' sound before a 'k' or 'g' sound due to velarization ('ng' is the velar variant of 'n' and 'k' and 'g' are velar sounds). It is common for vowels to be nasalized before nasal sounds ('n', 'm', 'ng'). It is also common in non-rhotic (don't pronounce the 'r's next to vowels like in 'start' and 'north') to pronounce an 'r' between two adjacent vowels in words ending/beginning with vowels (the "intrusive r", e.g. in "there and back").

3. sound changes due to fast speech ("I'm gonna see 'bout it t'day.", etc.)

4. ambiguity about where words start/end (e.g. "to Damon" vs "today mon" where the "mon" is the variant of "man" in Caribbean English).

5. word play, puns, etc. due to accent and other speech.

6. technical words in a given domain, specific place names, etc.

7. other things that can affect speech such as mumbling, stuttering, or slurred speech.

Speech recognition is a notoriously difficult problem to solve. It can work great as a first pass that someone can fix up, but not on its own. This is why having sentence/clause and word-level transcription markers along side a recording help.

One of the main issues is around homophones in an accent (Adam/Atom in American English, Bath/Barf in London English, etc.). Not to mention pronunciation variations due to fast speech, speech impedements, or parts of words side-by-side that sound like a different word.

Another big issue is around misaligned training data. For example, Whisper is known to hallucinate on silence [1].

[1] Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio (https://arxiv.org/html/2501.11378v1)

This is the same as when using LLMs as chatbots to ask questions.

If you ask another question in the same context it has all the information from that context and will be difficult to stray from anything in that context, e.g. if the LLM has gone down the wrong path or you are doing something slightly differently then it is difficult to steer the LLM away from the old context. This is why I tend to start a new context whenever I ask a question even if related to a previous question/answer. It can also be useful to do if/when the LLM gets stuck as a way of resetting it.

Note: this is probably why sub-agents are useful/work as they have a new context history.

You can incrementally refactor existing code to make it more maintainable and easier to implement the things you are working on.

The issue with Netscape (per that article) was that it was a rewrite, not a refactor. I.e. replacing all the existing code with new code. That's why it took a long time to release a new version: the previous code hadn't changed and the new code wasn't functionally equivalent to the existing code, let alone have any new features.

That's fine if the projects are in a single place (e.g. GitHub) as you only need to create one account.

If every project requires its own username/password just to report a bug, I may do that for the first 2 or 3, but not for every project. I'm also unlikely to create accounts on hosting platforms I don't intend on using outside of reporting the single bug.

Look at even older writing and you'll see em-dashes there too: Edgar Allan Poe, Lucy Maud Montgomery, the Brontës, among others.

> “No, reprobate! you are a castaway—be off, or I’ll hurt you seriously! I’ll have you all modelled in wax and clay! and the first who passes the limits I fix shall—I’ll not say what he shall be done to—but, you’ll see! Go, I’m looking at you!”

-- Wuthering Heights (1847), Chapter 2, Emily Brontë.

I wonder what the frequency of AI common words like delve and other words/phrases are in Project Gutenberg, Wikipedia, and other collections.

That's what gitolite does with some helper scripts for creating/setting up the repositories, adding metadata to the configs, etc. That way your `git` user can be set to reject ssh login attempts. The projects are managed by pushing a config to a gitolite admin repository.

Indeed. I've edited my post to fix the typo.

See also the codeberg blog post (https://blog.codeberg.org/protecting-our-floss-commons-from-...) which has more informal guidelines. That includes projects that are:

1. created by LLM agents -- any vibe-coded projects;

2. mainly written and maintained by LLMs -- this would cover the recent changes to the rsync project;

3. tied to the LLM ecosystem -- this covers pytorch, llama.cpp, cursor, SillyTavern, AI skills repositories, and a whole host of other projects.

Two of the big issues I had were around the user/repository forced structure. That works to some extent when you have multiple users with a small number of repositories. But for my personal projects I wanted more control over the structure and grouping.

I worked around it to some extent by having an organisation per category, but that doesn't work at scale. For example, you can't group Text-to-Speech projects around the different libraries (coqui-ai, Qwen3, parler, etc.), or language tooling installation scripts by programming language. -- Those have a group/subgroup/repository style structure.

The other related feature is having organisation or group-level issues and corresponding tracking features.

Gitolite doesn't impose/enforce a structure on the user. GitWeb/CGit have a free-form category (e.g. "lang/python") that the project can be assigned to. That works, but I'd like it to support multiple tags/labels for that.

It depends on what you are after?

1. Do you want something that works and feels like GitHub? -- Forgejo and Gitea are good for this.

2. Do you want a place to host git repositories with minimal hassle? -- GitLab, CodeBerg, and others are available.

3. Do you have your own hosting infrastructure? You could use gitolite and CGit/GitWeb on that hosting platform or local hardware.

4. Do you just want to host repositories? -- Gitolite can be used to help with SSH/auth/repository creation, and CGit or GitWeb for the frontend.

5. Do you need something like GitHub Actions? -- GitLab, Forgejo, and Gitea offer CI, or use external CI infrastructure.

6. Do you need issue tracking and management? -- GitLab, Forgejo, and Gitea provide these. There are alternatives from Jira to Kanban (including Trello) to Markdown (Obsidian and others) and more.

The watermarking is independent of the model. The model itself has the probability weights to determine the next token. The watermark is similar to things like temperature and top_p/top_k in that the watermark adjusts the probabilities in a deterministic way that changes over time to hide tells from word choices.

The only useful benchmarks are those you've created for your specific workflow. Only then can you assess whether a given model is better or worse for what you are using it for.

There are tools like promptfoo designed for this.

What do you test? How much time/effort do you spend testing each of those things? How do you know those will be performance issues?

There's only so much you can do -- making reasonable assumptions, choosing suitable data structures, database indices, testing various workloads, etc. -- in a limited test environment.

You may spend a week optimizing a feature that only 10 people use or that never gets to the point where it becomes an issue. Or you may have something that can't easily be tested at scale, such as various counts or other dynamic data that are determined by complex queries that you may (likely) find you need to cache but don't necessarily know which values will become issues until you start using the system.

It can sometimes be difficult to predict what will cause performance regressions when developing an application or web server with test data. So you do your best with what you know at the time of release, and then deal with performance issues when you know what is causing regressions (e.g. saved searches) and can then make measurements and profiling to see what exactly is causing the issue.

Likewise, improving performance pushes the limit at which performance regressions happen allowing more data (documents, triangles/pixels, etc.) to be processed. This allows things like more complex game graphics. That in turn makes it harder to improve performance for the next round (more advanced triangle/face culling and pre-processing).

There can also be trade-offs with things like data layout, memory usage (caching and memoization), or implementation. For example, when processing XML/HTML data you could use DOM (more memory and upfront parsing, but easier to perform complex queries across the data), SAX (less memory, but more complex to process due to tracking state), or reader API (similar to SAX but a different processing model).

There can be challenges with various features, such as type checking with higher-order generic types. Others add various levels of overhead, such as parsing a program, constructing an AST (Abstract Syntax Tree), generating an IR (Intermediate Representation), the optimization passes and final code generation.

JavaScript evaluation for example is complex. Before Chrome the approach was to use a slow interpreter. IIRC, Chrome was the first browser to introduce JIT (Just-in-Time) compilation, leading to faster JavaScript and eventually more complex applications running in that language. Modern browser JavaScript pipelines are complex in order to achieve and maintain performance:

1. start interpreting the code on the AST so it is run immediately (or only rely on (2));

2. generate a machine code equivalent of that interpreted code (for faster baseline performance);

3. run more aggressive optimizations on known types based on profiling/analysis for even faster performance, including special handling of things like asm.js.

[1] https://v8.dev/blog/ignition-interpreter

[2] https://benediktmeurer.de/2016/11/25/v8-behind-the-scenes-no...

[3] https://www.cs.cornell.edu/courses/cs6120/2020fa/blog/tracem...

[4] https://webkit.org/blog/3362/introducing-the-webkit-ftl-jit/

LLMs have been trained on a lot of data and have built representations of various concepts and links between them from that training data. This is why they are good when seeing names of things like `Semigroup` as they can draw on all the information about that concept from multiple sources (mathematics papers and discussions, programming articles and code, etc.).

There was an article or comment somewhere that said that when an LLM was prompted in comments/variable names about a complex algorithm or something like that in a very specific domain it was very knowledgeable. However, when given different names/terminology for the same thing it didn't understand the problem it was prompted to solve.

This is also why LLMs are good at generating code using Bootstrap, Tailwind or any other technologies as they can relate terms like `flex` and `grid` to various layout patterns. So using sensible names from the literature/specifications/etc. helps both developers and LLMs.

I knew it... Wuthering Heights was written by AI and Lucy Maud Montgomery, Edgar Allan Poe, et. al. were AI bots churning out content!

Or maybe -- just maybe -- using dashes isn't a sign of content being AI written, just a style it picked up from the training data.

The what happened section mentions the addition of the `setup.mjs` and `Math_Symbol.js`, along with a change in `package.json` to add `"preinstall": "node setup.mjs"`, so presumably checking for any of those would be a good indication to check further.

For example:

    find . -type f | grep -P "/Math_Symbol\.js$"

On my 4090 setup I'm getting 86t/s on a 12B Q6_K quantized model running entirely in VRAM.

The current GPUs are optimized for processing huge numbers of triangles per second. There are three things at play here:

1. the organization of the data being sent to the GPU to optimize throughput;

2. the speed at which the GPU can read that data from its VRAM;

3. how many triangles it can process in parallel by using individual compute units.

I suspect that given parallel improvements for neural networks, we'll see similar improvements:

1. optimizing the structure of the weights in the model for efficient access by the CPU/GPU/NPU/TPU;

2. efficient access of data strides (matrix rows) in the memory, e.g. being able to read multiple 2x2 matrix values in one clock cycle, or stepwise pairs of values (a(i,j), b(j,k)) needed for matrix multiplication;

3. parallel compute for matrix and tensor multiplication and other operations needed by neural networks.

There are several aspects to this:

1. Better hardware optimized for matrix operations and other ML/NN specific workflows (NVIDIA hardware, Apple M4/M5 chips, Intel AI Boost, other NPU/TPU chips, etc.).

2. Bigger RAM/VRAM/unified for loading models in memory -- with 3090/4090 you can run 22-24B models in VRAM; with 5090 you can run 30-32B models; with Apple hardware with 128GB unified RAM you can run 70-128B models.

3. Better quantization (smaller, better value representations/lower errors/loss, optimized CUDA/etc. kernels) will lead to larger models being able to be run on the lower VRAM hardware at (with optimized kernels) decent tokens/s.

4. Better optimizations in general taking advantage of the hardware (coupled with the hardware and driver advances) and the NN architectures (NOTE: I saw a huge performance boost a while back after upgrading my llama.cpp version).

5. Better model architectures optimized/designed for running on lower-spec hardware, like Mixture-of-Experts which make it easier to run on a single GPU or distribute the weights more efficiently across multiple GPUs (i.e. you know that specific weights will be used together so you can keep them resident on a specific GPU and not have to swap the weights in/out of the various GPUs or transfer intermediate results of a single layer, only the result from that block/expert).

5. Improved NVME or similar SSD technology which a) has a large amount of storage, and b) has access efficiency on par with RAM/VRAM, especially if/when coupled with efficient DMA access to the GPU/NPU.

Various libraries (e.g. Python's `re` library) support comments and whitespace as an option allowing you to format the regex on multiple lines with commenting to document what each part does.

I'm not sure if there are any regex libraries that support DSLs and easy composability (e.g. the email RFC regex would be easier to read/maintain if you could specify the individual parts like are defined in the RFCs).

Unless they are building a podcast application which is predominantly RSS 2.0 based with some extensions from itunes and others to provide additional podcast-specific metadata such as episode and season information.

XSLT 3 (via XPath 3.1) has support for maps, arrays, and parsing JSON to/from those or an XML representation. The XML representation is easier to work with in 3.

There's a draft version of XSLT 4 that Saxon supports which adds more capabilities for working with map and array types.

C++ has had smart pointers for memory (and other resource) management for a long time now (see e.g. the Windows ATL classes for working with COM objects and resources).

There are a number of challenges that make browsers more challenging (even in memory safe languages like Swift and Rust).

1. Back references/pointers like `parentElement` to other objects in the graph that create dependency cycles (where traditional/simple reference counting will prevent the objects being deallocated).

2. Interacting with (and creating resources in) a garbage collector when running/evaluating JavaScript code.

3. Just-in-Time (JIT) compilation of JavaScript and other complicated interpretation-compilation pipelines that can allocate and transfer objects between the different stages.

A common writing tool is to use a story grid. You have chapters/similar along the Y axis and title, characters, plot elements, etc. along the X axis. That way you can keep track of what information you are revealing at each story beat/chapter, make sure that plot twists work (e.g. when outlining something like The Sixth Sense).

Depending on how big the story is and how many characters/plot threads there are, I can see the Excel files getting large.

This has happened with other accelerants in various media/fields:

1. easy access to video recording and editing equipment has made it a lot easier to produce videos on sites like YouTube;

2. easy access to audio recording and editing (Audacity, etc.) has made it a lot easier to produce podcasts and other audio content (via LibriVox, Apple Podcasts, etc.);

3. easy access to writing software and self-publishing (Amazon, fan fiction sites, blogging platforms) has made it a lot easier to produce stories and books;

4. easy access to game engines and tools (Unity, Unreal, Godot, Twine, RenPy, etc.) have made it a lot easier to produce games.

IIUC, the main problem with the current Li batteries is that the two plates can over time grow material that will 1) degrade performance; and 2) make it more likely to short circuit and catch fire. Similarly with electric car batteries after accidents where the battery is damaged, short circuits, and then catches fire.

So the main risk here would be the likelyhood of short circuiting under different failure scenarios.

In the linked "Kimi-K3 Technical Report [pdf]" paper, section 2.3 (Stable LatentMoE, p6) has the table with those equations on (top of p7, using β_1 for the gate branch and β_2 for the up branch). They talk specifically about the function in section 2.3.2 (Sigmoid Tanh Unit GLU, bottom of p7).

Figure 2 (p3) has the architecture diagram for K3 (which is the same one on the Kimi K3 blog post https://www.kimi.com/blog/kimi-k3). AFAICT, that diagram along with section 2.3 should answer your question.

Note: I'm not familiar with LLM/NN architecture to answer it more precisely than that; I only have a surface level understanding from watching various YouTube channels like 3B1B and Welch Labs on the subject.

From the paper (page 6 with a comparison to GLU and SwiGLU) they are not using tanh directly (i.e. f(x) = tanh(x)) but:

    f_gate(b,x) = b * tanh(x / b) * sigmoid(x)

    f_up(b,x) = b * tanh(x / b)

Looking at the graph I wonder if this is to try and get the best of both GLU (better representation at higher values of x >~ 5) and SwiGLU (the value bump just before 0).

Maybe they are in the process of uploading the weights and git history and have taken down the holding page/project to not have the "coming soon" in the git history.