eyegor

You are describing the ebay market for used nvidia tesla cards. The k80, p40, or m40 are widely available and sell for ~$100 with 24gb vram. The m10 even has 32gb! The problem for ai hobbyists is it won't take long to realize how many apis use the "optical flow" pathways and so on nvidia they'll only run at acceptable speeds on rtx hardware, assuming they run at all. Cuda versions are pinned to hardware to some extent.

What about mozjpeg?

The worst part about avif support in edge is it was added as an optional feature flag ~8 months ago but it still isn't enabled as a default. Nearly every other browser supports avif by default these days.

https://winaero.com/avif-support-is-now-available-in-microso...

The lazy way is to use text-generation-webui, use an exllamav2 conversion of your model, and turn down context length until it fits (and tick the 8 bit cache option). If you go over your vram it will cut your speed substantially. Like 60/s down to 15/s for an extra 500 context length over what fits. Similar idea applies to any other backends, but you need to shove all the layers into vram if you want decent tok/s. To give you a starting point, typically for 7b models I have to use 4k-6k context length and I use 4-6 bit quantizations for an 8gb gpu. So start at 4 bit, 4k context and adjust up as you can.

You can find most popular models converted for you on huggingface.co if you add exl2 to your search and start with the 4 bit quantized version. Don't bother going above 6 bits even if you have spare vram, practically it doesn't offer much benefit.

For reference I max out around 60 tok/s at 4bit, 50 tok/s at 5bit, 40 at 6bit for some random 7b parameter model on a rtx 2070.

Any 7b can run well (~50 tok/s) on an 8gb gpu if you tune the context size. 13b can sometimes run well but typically you'll end up with a tiny context window or slow inference. For cpu, I wouldn't recommend going above 1.3b unless you don't mind waiting around.

Comcast may be complete garbage to work with, but you can always get a support contact. For any issue. Google is opposed to support.

For anyone looking to try this out on consumer grade hardware here's a q4 version [0]. From initial testing it's subjectively a bit behind deepseek-coder-instruct [1] at the same size and quantization. Deepseeks model is near magic when it behaves (1-2 tries), spitting out nicely formatted code fence blocks in markdown that I can naively render in real time to get a local chatgpt like experience. Magicoder can do this too, but it usually takes 3-4 tries and it tends to "opt out" and ask for more info pretty frequently. Of course if you have a ton of vram, use the 33b-instruct variant instead.

After more testing, I think it's a toss up on most coding tasks but Magicoder tends to give subjectively better responses to "bad prompts". That is, prompts where you don't put effort into writing clear instructions. For example, one of my "bad prompt" tests is

> how to enable shared gpu memory in wsl2 docker container

A good response to this would discuss the nvidia container toolkit, maybe something about port forwarding, etc. But this isn't a prompt most models can give good responses to. Both of these models can handle it, even at 7b, but Magicoder gives more information.

[0] https://huggingface.co/LoneStriker/Magicoder-S-DS-6.7B-4.0bp...

[1] https://huggingface.co/bartowski/deepseek-coder-6.7b-instruc...

I'd defer to the llama.cpp or exllamav2 repos for quantizing help, I just use their guides myself. For practical use, I'd say searching for the quantized model on huggingface.co is the way to go. Chances are high that someone else already did what you're interested in and it's a lot faster to download a pre quantized model.

Thebloke, turboderp, lonestriker, bartowski. Between these 4 users on huggingface you can find pretty much every common model already quantized.

I use either llama.cpp or exllamav2 as the backend, but speed is mostly dependent on how many layers fit in your gpu. I'm not sure which gpu I was thinking of for that 50-100 tok/s, on my 2070 I can only get around 15 tok/s on 7b models. I have to use 1.3b models to get to 100 tok/s on this gpu.

7b can run fine on consumer hardware if you quantize to 4bit. I get around 50-100 token/s on 7b q4 models on older 8gb gpus I have lying around, which is plenty for chat. I don't know of any good resource for "consumer hardware leaderboard" though. In general:

1.3b is about the max you can get for 4gb vram

7b will work on 8gb vram if you quantize

13b will work on 16gb vram if you quantize

33b can work on 24gb if you quanitze

If you lack vram but have ram and infinite patience you can do offloading shenanigans with cpu but it tanks performance to unusable levels in my experience.

There are several web apis that Mozilla refuses to support, for example webhid or webusb, which all chromium browsers have had for years at this point. Same for little css features and some bits in the webaudio and webrtc implementations.

Mozilla refusing to support webhid in any way: https://mozilla.github.io/standards-positions/#webhid

Chrome adding it to stable in 2021: https://developer.chrome.com/blog/new-in-chrome-89/

This happens periodically, where google services act funny with a Firefox user agent. Just use edge or chromes user agent string, and Google sites will work again.

Hmm, I just assumed people knew about it. Submitted https://news.ycombinator.com/item?id=38495176

Self hosted 6.7b model is phenomenal in my experience, I actually prefer it to chatgpt a lot of the time. Similar quality code but fewer disclaimers and #todo in my experience. Gpt4 is still a bit better at coding but not by much and it's much slower for me.

Quantized 6.7b - https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct...

1.3b - https://huggingface.co/TheBloke/deepseek-coder-1.3b-instruct...

In my experience, these models are at least as good as gpt3.5-turbo and they're very fast on consumer grade gpus (if you use quantized q3_m, q4_m, q5_m). The 1.3b is fast and good enough on consumer gpus to be used for completely local/offline code autocomplete.

With an 8gb vram gpu, I tend to prefer these models over chatgpt. Granted I'm a bit strapped for vram so I'm stuck at 6.7b q3_m or q4_m, so I often drop to 1.3b q6. It gives marginally worse quality answers than gpt3.5-turbo for me, but it's so stupid fast I don't care. On my 2070 it generates ~a dozen or more lines of code + prose per second.

If you only have a cpu, the 1.3b model is plenty fast enough for chat although it's definitely too large for real time / low latency tasks like code autocomplete.

In my experience, the deepseek-coder-instruct family is at least as good as gpt-3.5-turbo on python. Even the 1.3b models are very good (and run okay on cpu), although you should use larger if you have the vram. There are even larger models (30b+) if you are drowning in vram, but I don't think they perform much better at coding than deepseek-coder-instruct 6.7b.

3-4gb vram or cpu (1.3b): https://huggingface.co/TheBloke/deepseek-coder-1.3b-instruct...

Alternative for chat (1.3b): https://huggingface.co/TheBloke/evolvedSeeker_1_3-GGUF

Alternative for chat (3b): https://huggingface.co/TheBloke/open-llama-3b-v2-wizard-evol...

6-8gb vram (6.7b): https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct...

It was fixed in 2019 though? I don't understand your point.

https://github.com/jqlang/jq/issues/1380

Dropbox seems to still work fine in my experience.

Considering how poorly it seems to support cuda as a backend [0], I wouldn't hold my breath about non intel vendor support (amd cpu or gpu). If you want to try anyway, here's the benchmark suite [1] and I couldn't find sample results available.

As for less common gpus, there really is no good support in any library. If you ever want to go down a fun rabbit hole, try to use the gpu in a raspberry pi for something. You'll eventually find one guy who reverse engineered the drivers to make a compiler but that's it. That's also the story for most uncommon chips with gpus (amlogic, broadcom, etc) where the real vendor gives you a driver blob for linux and no documentation. On the other hand rockchip has some good things available [2].

[0] https://github.com/IntelPython/dpctl/discussions/1124

[1] https://github.com/IntelPython/dpbench/blob/main/README.md

[2] https://github.com/rockchip-linux/rknn-toolkit2

My understanding is they reduced the number of ensembles feeding gpt4 so they could support more customers. I want to say they cut it from 16 to 8. Take that with a grain of salt, that comes through the rumor telephone.

Are you prompting it with instructions about how it should behave at the start of a chat, or just using the defaults? You can get better results by starting a chat with "you are an expert X developer, with experience in xyz and write full and complete programs" and tweak as needed.

https://katex.org/ may work?

Yeah all the pieces are already there, but I was trying to make something like a player replacer that would let the user load arbitrary glsl shaders to use. The idea being to provide usable upscaling or filters for weak connections or old videos, correct shaky videos, etc. in real time.

I just found this for fsr [0] which might work for the upscaling use case.

[0] https://github.com/Hajime-san/web-fsr/tree/main/browser-exte...

Kind of unrelated, but how difficult do you think it would be to hack support for glsl shaders in a browser? I tried to look into it once, but got a bit lost in the media source side of things. My idea was to try to add glsl shaders as post processing to video streams like in mpv but without having to jump through all the hurdles of passing data to mpv.

Example of a shader I was playing with https://github.com/TianZerL/ACNetGLSL

Wow, faststream works great for normal web players. Doesn't seem to work on any youtube videos when using the ff extension in the store though. Gets stuck loading forever.

Edit: fixed, works well in every test case

How do you work with so much python code with no type annotations? I get it for smaller projects but isn't half this stuff meant to be library code?

See also

https://github.com/cupy/cupy

https://github.com/inducer/pyopencl

This is true, but no one in the industrial world is using macs. You even address this in your first paragraph. No IT group in their right mind is going to add a 3rd environment to manage (apple). Beyond being a headache to have multiple operating systems, update orchestrators, etc. it's also a nightmare because you can't get a 24h on site support contract from Apple. I'm surprised labview was supporting mac users in the first place, that's just a waste of development resources.

Most of the time an easy optimization is to pad fields of your struct to a 32 bit boundary. Almost any compiler will do this for you (look up "struct alignment / padding"). If the compiler is going to do this anyway, might as well use the memory yourself instead of letting it be empty space. If it doesn't happen, you leave performance on the table, so doing this raises the chance that your struct/fields will be aligned.

Nuance is that each field should be at an address divisible by the fields size or wordline size, not some magic 32 constant. The entire struct should also be padded to a multiple of the largest fields size. In practice this usually means 32 bit alignment.

Ref http://www.catb.org/esr/structure-packing/

You should watch this: https://youtu.be/cD7YFUYLpDc?si=rUm6IR3IKXyzcaDB to better understand why high sample rates are a waste of time, instead of just reading about nyquist. "accurately preserving a 6kHz sine wave" sounds a lot like you think that sample points are reproduced 1-1 from the digital to the analog domain.

This just builds on the xiph video someone else linked but essentially

- sine waves are fine as long as you have points for rising and falling edge (nyquist, 44k guarantees 22k sine wave reproduction)

- bit depth only really affects noise floor, so it depends on your audios dynamic range

Intel has been running oneapi for several years, and as a long time user I assure you it's horrible to deal with. It's the only software I've dealt with that breaks other software. Every year or two I'd have to fully reinstall visual studio because it hopped in, messed up a bunch of files, and the uninstaller never works. It will also happily break your system python environment if you let it try. And did I mention it takes over an hour to run through their horrible installer that tends to break itself? Even under linux it would try to hop in and screw up system /bin and /sbin links because why not. They also shut down their old license validation portals so their older, working versions can no longer be installed. Intels dev tooling is absolutely the worst tooling around.

If that's true, you should probably update the docs. Everything I could find implied dotnet, jvm, python were still unsupported. For example, the roadmap section of the readme mentions most of these as future targets but nothing even mentions dotnet. However I did find your tickets and a demo being merged in which makes dotnet seem maybe supported?

Ticket: https://github.com/parca-dev/parca-agent/issues/161

Demo: https://github.com/parca-dev/parca-demo/pull/18