Hacker News Reader

kiratpon What are the best options for Amazon SDEs thinking about leaving over RTO policy

It is the GP’s problem when this coworkers don’t want to work with them again.

The tech hubs (SF, Seattle, NYC are pretty small communities.

kiratpon Clearview AI faces $45.6M fine in the Netherlands for 'illegal database of faces

Are you sure you haven't violated the laws of any of the other 190+ countries? sure enough to risk taking another vacation?

kiratpon Tesla’s TTPoE at Hot Chips 2024: Replacing TCP for Low Latency Applications

Current public SOTA seems to be “no congestion control”

> We proceeded without DCQCN for our 400G deployments. At this time, we have had over a year of experience with just PFC for flow control, without any other transport-level congestion control. We have observed stable performance and lack of persistent congestion for training collectives.

https://engineering.fb.com/2024/08/05/data-center-engineerin...

kiratpon Ex-Kansas police chief who raided local newspaper criminally charged

You’re already paying for it via settlements.

Requiring officer-named insurance would create immediate pressure to weed out bad behavior and prevent bad cops from being re-hired one county over.

kiratpon Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

The Bitter Lesson -> http://www.incompleteideas.net/IncIdeas/BitterLesson.html

kiratpon Structured Outputs in the API

Thank you! This confirms my intuition!

Structured generation seems counter to every other signal we have that chain of thought etc improves performance.

kiratpon You can opt out of airport face scans

The peer responses pretty much cover most reasons.

kiratpon You can opt out of airport face scans

As an Indian American, I opt out of this photo scanning and the mm wave scanners (and get a pat down) every single time.

The system needs active backpressure.

kiratpon You can opt out of airport face scans

https://theintercept.com/2015/05/27/tsa-body-scanner-lobbyis...

https://www.propublica.org/article/tsa-removes-x-ray-body-sc...

The whole implementation of these scanners was layers of corruption.

Still worth it IMO - as long as they have to train people to maintain the manual process, its a signal that there are people out there who won't just fold.

kiratpon FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision

> However, a hardware-naive implementation is approximately worthless.

This isn’t true when there is one vendor that’s 90% of the market and 2 maybe 3 generations of hardware to consider. Support A100, H100 and you are supporting most of the current market.

kiratpon Metal thieves in America's cities

Its really grating to see headlines phrased this way, implying extreme outcomes. To someone not living in a city, it paints an immediate picture of dystopia. It pushes the rural-city divide further - the last thing we need more of.

kiratpon Show HN: Edna, note taking app for developers

I can’t back up the data in a PWA.

kiratpon Testing AMD's Bergamo: Zen 4c

IMO if the trade off is cheaper, faster hardware iteration then Nvidia’s strategy makes a lot of sense.

kiratpon Cost of self hosting Llama-3 8B-Instruct

By using the drivers you agree to their TOS. So yes, it applies even on your private network.

kiratpon Cost of self hosting Llama-3 8B-Instruct

3 year commit pricing with Jetstream + Maxtext on TPU v5e is $0.25 per million tokens.

On demand pricing put it at about $0.45 per million tokens.

Source: We use TPUs at scale at https://osmos.io

Google Next 2024 session going into detail: https://www.youtube.com/watch?v=5QsM1K9ahtw

https://github.com/google/JetStream

https://github.com/google/maxtext

kiratpon Cost of self hosting Llama-3 8B-Instruct

Nvidia EULA prevents you from using consumer gaming GPUs in datacenters so 4xxx cards are a non-starter for any service usecases

EDIT: TOS -> EULA per comments below

kiratpon cuDF – GPU DataFrame Library

Jax is the closest thing at the moment.

https://developer.apple.com/metal/jax/

And MLX

https://github.com/ml-explore/mlx

kiratpon I'm a Former Surgeon General and I Couldn't Believe My $10k Medical Bill

Helicopter rides are pretty much never covered.

> The average cost of life flight within the U.S. ranges between $12,000 and $25,000, according to NAIC (National Association of Insurance Commissioners). This is based on a 52-mile trip, which is also the average distance. This figure represents an out-of-pocket cost when not covered by insurance or calculated before an insurance company steps in. International flights can easily cost 3 to 5 times that amount. … In December 2020, Sean Deines was diagnosed with acute lymphoblastic leukemia (a fast-growing blood cancer) and took an air ambulance from Colorado to North Carolina, which also included ground transportation between hospitals and airports. His total bill was $489,000.

https://www.emergencyassistanceplus.com/resources/what-is-th...

kiratpon Snowflake Arctic Instruct (128x3B MoE), largest open source model

> 2. Small number of activated parameters shine in batch inference case for cloud providers

Could you elaborate more please? Batch inference activates pretty much all the experts since token in every sequence in a batch could hit a different expert. So at Bs=128 you’re not really getting a sparsity win.

kiratpon Phi-3 Technical Report

The technique to use is to give the model an “out” for the missing/negative case.

"{text}\n\n###\n\nPlease summarize the text above. The text is a video transcript. It may not have the names of the speakers in it. If you need to refer to an unnamed speaker, call them Speaker_1, Speaker_2 and so on."

kiratpon LLaMA now goes faster on CPUs

It fascinating to me that coming up on a year since Sapphire Rapids has been available in the public cloud, developers are still targeting AVX512 when they should be targeting VNNI and AMX.

https://github.com/ggerganov/llama.cpp/issues/2555

kiratpon Apple's punitive compliance will burn its political and developer goodwill

> I don't care how much Apple hand holds grandma. Mobile is a hellscape for developers and other companies.

As a developer and founder, grandma before developers. Every time. iOS is the first platform my family can just run with safely.

kiratpon Efficient LLM inference solution on Intel GPU

+1 to this.

I’ve been working with their research teams on this and credit due indeed.

Some context: https://github.com/ggerganov/llama.cpp/issues/2555#issuecomm...

https://github.com/ggerganov/llama.cpp/discussions/3965

kiratpon Our Generation Ships Will Sink (2015)

One thing I firmly believe - the people who will pull this off will be ones who just aren’t willing or able to see all the possible failure paths.

Unreasonable optimism is the only path to success at this scale.

kiratpon I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

No Nvidia drivers for MacOS.

kiratpon I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

We should be happy that compute is once again improving and machines are getting outdated rapidly. Which is better - a world where your laptop is competitive for 5+ years but everything stays the same? Or one where entire new realms of advancement open up every 18 months?

It’s a no contest option 2 for me.

Just use llama.cpp with any of the available UIs. It will be usable with 4 but quantization on CPU. You can use any of the “Q4_M” “GGUF” models that TheBloke puts out on Huggingface.

https://github.com/ggerganov/llama.cpp

UI projects in description.

https://huggingface.co/TheBloke

A closed source option is LMStudio.

https://lmstudio.ai/

kiratpon TinyLlama: An Open-Source Small Language Model

I guess it comes down to whether your usecase has a single correct answer vs multiple possible ones. For example a lot of what we do has one and only one correct sequence of tokens. Need to look at both but so much of the learning material out there just focuses on loss. YMMV.

kiratpon TinyLlama: An Open-Source Small Language Model

> For any model, the loss curve going down could mean it’s learning, or could mean it’s overfitting, we don’t know which without looking at validation loss, which is like a second set of test data the model hasn’t seen before.

You want to look at validation accuracy.

kiratpon A Lufthansa A350's frustrating Oakland diversion

The FAA themselves recommend that foreign pilots do not use visual approaches at SFO.

https://www.cbsnews.com/sanfrancisco/news/faa-wants-foreign-...

kiratpon A new old kind of R&D lab

Because we don’t have a clear understanding of how the things work yet.