Hacker News Reader

bt1aon The quiet art of attention

This is beautiful! It is how I look at mindfulness mediation, and I will use some of the clear writing from this piece when trying to convey it to others

bt1aon Lm.rs: Minimal CPU LLM inference in Rust with no dependency

You are correct. This project is "on the CPU", so it will not utilize your GPU for computation. If you would like to try out a Rust framework that does support GPUs, Candle https://github.com/huggingface/candle/tree/main may be worth exploring

bt1aon An n-ball Between n-balls

I am struggling to juggle the balls in my mind. Are there any stepping-stone visual pieces like this to hopefully get me there? Very neat write-up, but I can't wait to share the realized absurdity of the red ball's green box eclipsing in our 3D intersection of the fully diagonalized 10D construct

bt1aon Google Listed as Sponsor, Host of Israeli Military Tech Conference

Archive link?

bt1aon NASA confirms space station cracking a "highest" risk and consequence problem

Non-expert, pure speculator here: I cannot help but think of the Titan submersible and its minor flaw in the hull that slowly became more and more of an issue with each dive and pressurization cycle. I wonder if the 'untenable' leak rate that isn't agreed upon is the rate at which an accelerated and catastrophic runaway event occurs?

bt1aon Three Mile Island nuclear plant restart in Microsoft AI power deal

As an AI assistant, I'm not equipped or trained to handle an ongoing nuclear meltdown..

bt1aon Show HN: Wealthfolio: Private, open-source investment tracker

I am currently working on configuring a similar private, open source portfolio tracker built on top of ledger (a double-entry accounting system). I was drawn to it because of its yml config that I can version control easily

If anyone has set up Paisa (successfully or unsuccessfully) and has anything to share, I'd love to hear it.

https://paisa.fyi/ https://demo.paisa.fyi/ https://ledger-cli.org/

When comparing the two programs here, I can't immediately see any big differences. Sorry if this reads like a shallow plug

bt1aon CrowdStrike accepting the PwnieAwards for "most epic fail" at defcon

Yes

bt1aon CrowdStrike accepting the PwnieAwards for "most epic fail" at defcon

Probably because our incredibly inefficient, burdened, and splintered healthcare system barely functions as is, and they do not have the time nor resources to pause and put in place an emergency downtime operating protocol that works as well as their 15 year old windows cobweb

bt1aon CrowdStrike accepting the PwnieAwards for "most epic fail" at defcon

I am positive that people in hospitals died as a direct result of this incident.

bt1aon PSA: Internet Archive "glitch" deletes years of user data and accounts

> The Internet Archive serves millions of people each day and is one of the top 300 web sites in the world. A single copy of the Internet Archive library collection occupies 145+ Petabytes of server space (and we store at least 2 copies of everything). We are funded through donations, grants, and by providing web archiving and book digitization services for our partners. As with most libraries we value the privacy of our patrons, so we avoid keeping the IP (Internet Protocol) addresses of our readers and offer our site in https (secure) protocol.

QFT

What an unbelievably pissy and deserving tone from the author. If you want to be an archivist and you've permanently lost data that was stored in a single, external location, hopefully it's a learning experience

bt1aon Import and Export Markdown in Google Docs

Why not a markdown doc + git with each colleague using their own text editor/IDE of choice?

bt1aon Show HN: a Rust lib to trigger actions based on your screen activity (with LLMs)

Maybe a jest at those who are self-employed..?

bt1aon Language models on the command line

(aichat in tangential comment looks much better) Besides aider and ollama, I think shell_gpt https://github.com/TheR1D/shell_gpt is great for quick chats / lookups. Being able to quickly cat files to repl sessions saves a lot of time.

I need to integrate distil whisper large v3, aider, and shell_gpt to tidy up a lot of my disjointed LLM use. As someone else mentioned, the commits created by aider allow me to "skip" some intermediate steps that would be required when working on coding tasks with other frameworks.

bt1aon Apple's On-Device and Server Foundation Models

This, plus metal acceleration works quite well. 7~8B parameter models quantized to 3bpw or so run with good tok/s on my iphone 15 pro

bt1aon Koheesio: Nike's Python-based framework to build advanced data-pipelines

I'll agree that the pages about Koheesio hosted on nike's website are strangely empty of persuasive specifics (seems like generative AI used by someone who was in charge of writing it up lol..), but maybe the authors believe that someone who's seriously considering a data pipeline library would want to look at the actual code (which by the way, is structured quite well). Python doesn't "inspire confidence"? Think of the target audience. The frameworks claims to take typing and tests quite seriously, so perhaps your confidence is gained elsewhere. I didn't go through the codebase because I'm not interested, but your dismissive attitude doesn't radiate good faith

bt1aon Apple News+ subscription growth blows away major media sites

bypass paywalls clean aint bad either

bt1aon Self-hosted offline transcription and diarization service with LLM summary

I do what you're describing with VB-CABLE on Windows (and Blackhole-2ch on macOS) + OBS

https://vb-audio.com/Cable/index.htm

So, if you wanted to transcribe the audio from one of your physical output devices (say you had your meeting software outputting on external headphones), you could set the virtual audio device as the monitoring device on the physical external headphones. Therefore, you end up with a virtual audio input device containing the audio from your meeting software. I also do this to apply a chain of filters to my condenser mic in OBS because it picks up everything.

bt1aon Phi-3 Technical Report

This won't dethrone Llama 3, but it's equally impressive.

They mention this model's relative weakness in the TruthfulQA eval, since it's more lossy trying to pack 'knowledge' into a small model relative to problem-solving skills (which shine on MMLU)

Regardless - still a very useful thing to have offline and on the fly. Those scores are nothing to scoff at.

Given that these pipelines are likely harder harder to imitate than new architectures like Transformers, I assume there has been and will be an intense focus on synthetic data generation and cleansing. Llama 3 used 15T of tokens in its training corpus vs 4.8T in the "scaled-up" version of phi-3. If you made it to the end of this disjointed ramble I'm sorry

bt1aon Phi-3 Technical Report

I tore my hair out developing a SwiftUI app that could run llama.cpp and whisper.cpp simultaneously. Was able to run a Q3_K Mistral 7B along with a smaller whisper model eventually, but grinding through XCode is a nightmare.

They're working on MLX but it only recently got swift bindings. They just don't have the DEVELOPERS DEVELOPERS DEVELOPERS coked out attitude i guess

bt1aon More Agents Is All You Need: LLMs performance scales with the number of agents

If I'm reading this correctly, they had to discard Llama 2 answers and only use GPT-3.5 given answers to test the hypothesis.

GPT-3.5 answering questions through the OAI API alone is not an acceptable method of testing problem solving ability across a range of temperatures. OpenAI does some blackbox wizardry on their end.

There are many complex and clever sampling techniques for which temperature is just one (possibly dynamic) component

One example from the llama.cpp codebase is dynamic temperature sampling

https://github.com/ggerganov/llama.cpp/pull/4972/files

Not sure what you mean by whole model state given that there are tens of thousands of possible tokens and the models have billions of parameters in XX,XXX-dimensional space. How many queries across how many sampling methods might you need? Err..how much time? :)

bt1aon PyTorch Library for Running LLM on Intel CPU and GPU

I think there's more than a few enthusiasts who would be very interesting in buying 1 or more of these cards (if they had 32+ GB of memory), but I don't have any data to back that opinion up. It is not only those who can't afford a 4090 though.

While the 4090 can run models that use less than 24GB of memory at blistering speeds, models are going to continue to scale up and 24GB is fairly limiting. Because LLM inference can take advantage of splitting the layers among multiple GPUs, high memory GPUs that aren't super expensive are desirable.

To share a personal perspective, I have a desktop with a 3090 and an M1 Max Studio with 64GB of memory. I use the M1 for local LLMs because I can use up to 57~GB of memory, even though the output (in terms of tok/s) is much slower than ones I can fit on a 3090.

bt1aon LLM4Decompile: Decompiling Binary Code with LLM

I would also think that generating a very large amount of C code using coding LLMs (using deepseek, for example, + verifying that the output compiles) as synthetic training data would be quite beneficial in this situation. Generally the quality of synthetic training data is one of the main concerns, but in this case, the ability for the code to compile is the crux.

bt1aon Building an Open Source Decentralized E-Book Search Engine

Perhaps the initial creation of the index is indeed something that an average laptop could accomplish, but I'd imagine that frequently updating the index and serving requests against it would be compute-intensive. I have nothing to back this up but speculation. Would love to learn more!

bt1aon The Pile is a 825 GiB diverse, open-source language modelling data set (2020)

Pouring one out for the future litigators, jurors, and judges who will have to pore over this inextricable web of legal and technical details

bt1aon The Pile is a 825 GiB diverse, open-source language modelling data set (2020)

It is not.

In related news, v2 of the "stack" dataset was recently released

> 3.28B unique files belonging to 104.2M github repositories were collected by traversing the Software Heritage 2023-09-06 graph dataset. Additional repository-level metadata was collected from GitHub Archive data up to 2023-09-14. The total uncompressed size of all files is 67.53TB. Near-deduplication was implemented in the pre-processing pipeline on top of exact deduplication.

V1 vs V2 by Deduped Size Tokens

V1: 2.9TB and 200B

V2: 32.1TB and 900B

I imagine we'll see some fairly powerful open coding models soon. The ones I'm looking at testing are:

dolphincoder-starcoder2-15b-iMat.GGUF

CodeFuse-DeepSeek-33B-iMat.GGUF

OpenCodeInterpreter-DS-33B-iMat.GGUF

starcoder2-15b-instruct-iMat.GGUF

more info

dataset https://huggingface.co/datasets/bigcode/the-stack-v2

gguf quants https://huggingface.co/dranger003

bt1aon WyGPT: Minimal mature GPT model in C++