ianpurton

Salsa classes.

Kubernetes means you don't have to learn each clouds way of doing a deployment. You just learn the k8s way then use that with Google, Azure or whatever.

So your skillset is reusable.

You can use reddit to find open source reddit clones https://www.reddit.com/r/opensource/comments/vog8mg/is_there...

But like other posters have already said you would need a strategy to get people to use it.

It's the classic chicken and egg, if you have no content why would I visit, and if you have no visitors why would I post content.

I'm the author of a guide to web development with rust https://rust-on-nails.com/

It gives you an idea of what a full stack in Rust can look like.

I have a dockerfile here https://github.com/purton-tech/rust-llm-guide/tree/main/llm-... for running mpt-7b

docker run -it --rm ghcr.io/purton-tech/mpt-7b-chat

It's a big download due to the model size i.e. 5GB. The model is quantized and runs via the ggml tensor library. https://ggml.ai/.

Reddit wants to charge for it's API.

Some users are annoyed as they are using 3rd party apps to help with moderation.

They can protest by making subreddits private.

But Reddit has been a clunky, ugly, painful user experience for many years and has succeeded despite that, so I guess Reddit will win.

> and would market it more aggressively,

You could set up a landing page and test your marketing, if you get the traction you hope for, build the product, if not then let it die.

Docker images run native on Linux. The virtual part is just to get Docker working on MacOs/Windows

> I’m not smart enough to figure out docker confiuration, so instead I compile my web application into a self contained statically linked binary executable file.

Good luck getting that to run in different environments because static or not, nearly every executable has dependencies.

> I’m not smart enough to figure out cloud services and auto scaling groups, so I just upload my static binary file using scp to a linux server I rent from a VPS provider.

Which of course you'll need to setup, patch, figure out how to restart apps when they go down and on and on. There's no easy out.

> I’m not smart enough to figure out how to setup all the databases and firewalls and load balancers, so I just embed the http server and data storage engine as libraries in my static binary.

And how do you back that up?

I get the gist of the article, but some things are not always replaceable with an easy alternative. Most alternatives also have disadvantages as well as their own learning curves.

Postgres, Docker, Infra as Code and Kubernetes are pretty much best practices on any development.

Don't kid yourself that the learning curve is not worth it.

So Rolls Royce want to build small modular reactors https://www.rolls-royce.com/innovation/small-modular-reactor...

It will be interesting to see if the smaller scale can avoid some of the issues mentioned in the article.

ONNX doesn't support the same level of quantization as GGML.

So basically GGML will run on hardware with less memory.

Assuming you have 24GB of VRAM, then you should be able to run something like 7 billion parameters i.e. MPT-7b and quantized perhaps you can get up to 13 billion.

Note: I have no practical experience of this, just reading around.

You can compress LLM models and run them in less ram. This matters because most people don't have access to powerful GPU clusters.

You might want to try some of the discord channels connected to some of the repos. i.e. GPT4All https://github.com/nomic-ai/gpt4all scroll down for the discord link.

I can already imagine people lining up to get the best time on strava.

Don't worry all the panels will be stolen soon anyway.

It's too late, it's out there now.

It's just a matter of time before we see a HN Launch for a startup that's going to disrupt the medical industry with LLM.

How does the MLIR that Mojo generates differ from https://openxla.github.io/iree/

Or does Mojo sit on top of iree?

Will the inference engine support multi GPU setups?

To get an idea where WebGPU is heading a couple of projects worth looking at.

- The Bevy game engine, using WebGPU on the backend. https://bevyengine.org/

- Wonnx, A WebGPU inference engine for running AI compute on the server or in the browser. https://github.com/webonnx/wonnx

> GPU in other ways, such as training ML models and then using them via an inference engine all powered by your local GPU?

Have a look at wonnix https://github.com/webonnx/wonnx

A WebGPU-accelerated ONNX inference run-time written 100% in Rust, ready for native and the web

> We are currently focused on completing the training process on the entire RedPajama dataset.

So that's 1.2 trillion tokens. Nice.

Monolith is not a best practice.

With most web applications you'll end up with a web part and at some point some crob jobs. In this case they also have a service that processes a queue.

They should in be in the same repo, that's a Monorepo.

A web app with some cronjobs is best deployed as separate executables (ideally docker containers). In this case they had everything in one exe, including threads to run the queue etc.

So it looks like there solution was more complicated by aiming to be a Monolith.

I mainly test with a combination of unit tests and using selenium https://github.com/stevepryde/thirtyfour for integration tests

I wrote it up here https://rust-on-nails.com/docs/continuous-integration/integr...

One of the libraries from the article which I have experience with is https://dioxuslabs.com/

It has a server side rendering mode, which for most projects will be all you need.

So then if you follow the Islands Architecture https://www.patterns.dev/posts/islands-architecture means you can sprinkle some typescript when you need more dynamic functionality.

I'd love to use Rust for front end enhancement in this way, but the problem is you have to completely open up your content security policy https://github.com/WebAssembly/content-security-policy/blob/...

Which is going to get picked up in any security review.

I've written up how I develop web apps with rust here https://rust-on-nails.com/

Question. I can see in the code the WGSL that's needed to implement inference on the GPU. https://github.com/0hq/WebGPT/blob/main/kernels.js

Could this code also be used to train models or only for inference?

What I'm getting at, is could I take the WGSL and using rust wgpu create a mini ChatGPT that runs on all GPU's?

I use zola a static site generator and deploy the pages (for free) to cloudflare pages.

So there's no impact on my servers.

For a product you will get a lot of people from HN reading the landing page but relatively few registrations.

Practice generating traffic.

An example could be a blog. Start with 1 article and promote it.

This is something you could do today, then you'll have an idea what it's like to do marketing.

> I do not allow my team to work remotely from wherever they want and require them to come to the office.

Why?

It's renewable if Greenpeace can somehow release the CO2 from our childrens lungs.

Germany is running on coal. It's by far the worst example of energy vs pollution in Europe.