Hacker News Reader

kiratpon Llama 2 on ONNX runs locally

Nobody is deploying 3+GB models to iOS beyond some enthusiast “because you can” apps. Amazing tech but not feasible for any mainstream use yet.

Eg: https://apps.apple.com/app/id6444050820

kiratpon Llama 2 on ONNX runs locally

When hardware is so expensive and so difficult to obtain, performance starts to trump the cost of “single model implementation” pretty quickly.

It can be cheaper to deploy Llama.cpp and then foobar.cpp (6 months from now) than it is to have inference that is 2x slower.

Interestingly in the LLM space all these model servers seem to be converging to using the same API as OpenAI, making it easy to swap containers to get a different model+inference server with 0 code change.

kiratpon Llama 2 on ONNX runs locally

If you’re working with LLMs, just use this - https://github.com/ggerganov/llama.cpp

It has Metal support.

kiratpon Apple reports third quarter results

This all sounds extremely miserable from an end user perspective. We are all going to have 15 app stores on our phones and they will all be sucking up every piece of identifying information they can.

The EU is going to ruin app stores like they ruined the web with cookie notices.

kiratpon The underground world of credit card network exploitation

I'll take 3% over having a single payment processor I can't swap out.

The change to the Indian economy was going to come through private investment anyway but the government blocked that. So saying the change came because of UPI is disingenuous at best because it never had to compete with any other offerings. No other entity was even allowed to compete.

kiratpon SMS traffic pumping fraud

India has been enacting regulation enshrining SMS as 2FA for their entire Digital payments infrastructure. They even require storing all biometrics (FaceID etc.) on gov servers and not on-device. We are in for a decade+ of this at minimum.

kiratpon The underground world of credit card network exploitation

No thank you. This is how you get something like UPI in India.

https://news.ycombinator.com/item?id=36980279

kiratpon The underground world of credit card network exploitation

UPI is a terrible thing.

0) makes every transaction a trivial SQL query away for the government.

1) everything needs an SMS code. Just as we are trying to get everyone off SMS 2FA

2) doesn’t work for non-Indian numbers or roaming devices

3) can’t get an Indian SIM without proof of address etc. No burners in India

4) regulation expressly forbids devic-local biometrics. This is why there is no Apple Pay in India.

5) Biometrics must be stored with the government. “Unique Identification Authority (UIDAI)” - https://studentbriefs.law.gwu.edu/ilpb/2022/03/22/regulating...

kiratpon If we want a shift to walking we need to prioritize dignity

This is not true in the US, at least in Washington state for example. Honking = speech

https://law.justia.com/cases/washington/supreme-court/2011/8...

kiratpon Which GPU(s) to Get for Deep Learning

The bar for the AI ecosystem to get models running is git clone + <20 lines of Python.

If AMD can’t make that work on consumer GPUs, then the only option they have is to undercut Nvidia so deeply that it makes sense for the big players to hire a team to write AMD’s software for them.

kiratpon Apple already shipped attestation on the web, and we barely noticed

Explanation in relatively straightforward form: https://developer.apple.com/wwdc22/10077

Don’t shoot the messenger!

kiratpon A simple guide to fine-tuning Llama 2

If you want high performance inference us this: https://vllm.ai/

For local batch stuff: https://huggingface.co/docs/transformers/main_classes/pipeli...

kiratpon I stopped buying new laptops (2020)

Because it’s not faster. Well at least the 3990x wasn’t.

Compilation rapidly drops from parallel to single thread bound for most workloads.

https://twitter.com/kiratpandya/status/1457438725680480257

kiratpon Mercedes beats Tesla to autonomous driving in California

https://electrek.co/2019/05/17/tesla-nerfs-autopilot-europe-...

kiratpon Mercedes beats Tesla to autonomous driving in California

People who paid for the FSD package have been given the upgrades for free.

kiratpon Scar tissues make relationships wear out (2013)

The audience is all dreaming of what could have been in another life.

kiratpon [dead]

https://twitter.com/SendGridSupport/status/16615194199089725...

> Ahoy there! We're experiencing a period of maintenance for the SendGrid website (see http://status.sendgrid.com). You can subscribe to status updates which will let you know when the maintenance is complete and you may login.

As far as I can tell, no email heads up was sent out.

kiratpon SR-71 Blackbird Speed Check Story

Concorde was profitable. Accidents + 9/11 pullback killed it. The major factor against supersonic is actually the fact that sonic booms are not allowed over the continental United states. Companies like Boom Supersonic are focussed on eliminating those.

https://simpleflying.com/did-british-airways-make-a-profit-f...

> If British Airways and Air France were looking for more profit, these issues would have to be addressed. And that was the plan. British and French aerospace divisions were looking to create a brand new supersonic aircraft before plans to take the Concorde out of service in 2000 were realized.

> Of course, that never happened. With a plethora of odds stacked against it, the 2000 crash in France, 9/11 affecting interest, and Airbus scrapping Concorde part replacements, the Concorde was shelved before its time.

kiratpon 'BlackBerry' review: The comedy and tragedy of the innovator's dilemma

I was at Windows Phone (WP7/8 days) when “RCS was just a couple of years away”. Good thing I didn’t hold my breath - it didn’t seem to me then that there was any path to success and I was on my first job as a 20 year old.

Google hasn’t managed anything with RCS other than deploy it for their own messaging app that talks to their own infrastructure. The fact that the app uses RCS is an implementation detail - could just as well be carrier pigeons. Doesn’t matter because it isn’t cross-carrier interoperable.

The carriers all moved from Circuit Switched SMS signaling to SMS over IMS (LTE onwards) for interoperable SMS. IMS is the layer under RCS (created originally for cable set top boxes - go figure).

This is basically SMS over IP which means your device is doing all the legacy stuff like 7bit ASCII encoding to pack messages into 160 characters etc. These requirements won’t go away because, the moment a carrier tries to do cross-carrier interop there is a whole sea of legacy translations that would have to be managed by the message originating carrier. This would require real-time figuring out the capabilities of a receiving carrier based on phone number homing. It’s a massive undertaking that’s $$$.

On top of this The carriers all fought including E2EE in the spec to avoid picking a fight with the government. So anything Google implemented is DOA.

RCS has failed.

kiratpon Google “We have no moat, and neither does OpenAI”

You can do it on CPU now.

Benchmarks: https://github.com/ggerganov/llama.cpp/issues/34

kiratpon Google “We have no moat, and neither does OpenAI”

Benchmarks for what you can do on CPU alone.

https://github.com/ggerganov/llama.cpp/issues/34

An M1 Max does 100ms per token. A 64 core threadripper about 33ms per token.

kiratpon Google “We have no moat, and neither does OpenAI”

And here are the benchmarks

https://github.com/ggerganov/llama.cpp/issues/34

kiratpon Google “We have no moat, and neither does OpenAI”

You need roughly (model size + (n * (prompt + generated text)) where n. Is the number of parallel users/ request.

kiratpon Graviton 3, Apple M2 and Qualcomm 8cx 3rd gen: a URL parsing benchmark

Case in point

> Rust, cargo build for our "test world binary".: 64 core Threadripper 3990x @ PBO level 3, ~400w, with Optane drive - 1 min 37s M1 Max, 64 GB, on battery @ 30% charge - 1:34s

https://twitter.com/kiratpandya/status/1457438725680480257

kiratpon Maybe you should store passwords in plaintext

> And our only real change is to find yet another slaver, oh I mean capitalist to work for.

Can we please tone down the rhetoric? This is quite disrespectful to the millions who suffered through actual slavery.

kiratpon Maybe you should store passwords in plaintext

Same energy as “I pirate stuff because it’s not provided to me exactly when, how and at the exact price I want”. “Information wants to be free!”

This entire thread is depressing.

kiratpon MLC-LLM: GPT/Llama on consumer-class GPUs and phones

That’s not an argument against letting the GUI experience be downloadable from the App Store. You just have to flip a flag.

kiratpon Maybe you should store passwords in plaintext

For nothing else than the sake of “life is about the journey”, please consider changing your job. It is absolutely possible to align paid employment with fulfillment.

kiratpon MLC-LLM: GPT/Llama on consumer-class GPUs and phones

Why not let this be installed on Mac devices via test flight?

kiratpon MLC-LLM: GPT/Llama on consumer-class GPUs and phones

You can do Stable Diffusion on your phone too, fully local.

https://apps.apple.com/app/id6444050820