jimmyl02

the most unintuitive part is that from my understanding, individual tokens are routed to different experts. this is hard to comprehend with "experts" as that means two you can have different experts for two sequential tokens right?

I think where MoE is misleading is that the experts aren't what we would call "experts" in the normal world but rather they are experts for a specific token. that concept feels difficult to grasp.

the large context windows generally involve RoPE[0] which is a trick that allows the training window to be smaller but expand larger during inference. it seems like they have a new "iRoPE" which might have better performance?

[0]https://arxiv.org/pdf/2104.09864

the needle in a haystack benchmark looks good but at this point I think we need new benchmarks to test actual understanding of content in such a large window.

I feel like understanding this difference is what a good product manager should be responsible for. Not just optimizing any metric that is available but understanding the meaning behind them and choosing the push them the right direction.

I wonder at which scale spark fits into this picture and what the tradeoffs / benefits would be

pretty cool and thanks for the details about TUN devices!

I believe wireguard runs over UDP and while you still need a TUN device, it has kernel implementations to handle encrypting the traffic.

Based on the status report, it seems like it was related to a database disaster and mention recovering shards. Publicly they say they are using vitess[0] so I wonder if there was some issues there.

I wonder if there will be a more detailed post-mortem, would be super interesting to learn more!

[0] https://slack.engineering/scaling-datastores-at-slack-with-v...

don't most architectures resolve this via superscaling / some up scaling pipeline after that adds the details?

iirc stable diffusion xl uses a "refiner" after initial generation

one small idea I read somewhere that I found interesting for thought

at a startup: you might work on an entire page that impacts thousands of people

at a larger firm: you might work on a single form that impacts millions

wouldn't it be coredns caches the information and records from API server for X amount of time (it seems like this might be 20 minutes?) then once the 20 minutes expired coredns would query api server, receive no response, then fail?

I think the idea of just serving cached responses indefinitely when api server is unreachable is what you're describing but not sure if this is default. (and probably has other tradeoffs that I'm not sure about too)

splitting the control and data plane is a great way to improve resilience and prevent everything from being hard down. I wonder how it could be accomplished with service discovery / routing.

maybe instead of relying on kubernetes DNS for discovery it can be closer to something like envoy.the control plane updates configs that are stored locally (and are eventually consistent) so even if the control plane dies the data plane has access to location information of other peer clusters.

pretty new to the large scale data processing space so not sure if this is a known question but isn't the purpose of spark that it can be distributed across many workers and parallelized?

I guess the scale of data here ~100GB is manageable with something like DuckDB but once data gets past a certain scale, wouldn't single machine performance have no way of matching a distributed spark cluster?

this is a really neat writeup! the design choice to make each "exit node" control the local wireguard connections instead of a global control plane is pretty neat.

an unfinished project I worked on (https://github.com/redpwn/rvpn) was a bit more ambitious with a global control plane and I quickly learned supporting multiple clients especially anything networking related is a tarpit. the focus on linux / aws specifically here and the results achievable from it are nice to see.

networking is challenging and this was a nice deep dive into some networking internals, thanks for sharing the details :)

I think a good way of looking at it is risk. Is the change (whether it is code or configuration, etc.) worth the risk it brings on.

For example if it's a small feature then it probably makes sense to wait and keep things stable. But, if it's something that itself causes larger imminent danger like security patches / hard disk space constraints, then it's worth taking on the risk of change to mitigate the risk of not doing it.

At the end of the day no system is perfect and it ends up being judgement calls but I think viewing it as a risk tradeoff is helpful to understand.

One thing I appreciate very much about this article is that they describe things they tried and didn't work as well. It's becoming increasingly rare (and understandably why) for articles to describe failed attempts but it's very interesting and helpful as someone unfamiliar with the space!

the view I now have is that for a relational table, yes you have to suffer through migrations but at least they are in sql. for document based stores, you still have to have migrations, but they are just implemented in code

json documents sound great, especially initially, but end up being a maintenance nightmare

I don't know if the tone selector UX was used by other companies before but it is so simple and intuitive I'm surprised I haven't seen it more. I wouldn't be shocked if it becomes a standard within all the AI tools

Great stuff from Figma, many people I know have already been using it for slides and this is a great next step

curious what are the performance characteristics here? I would assume something like Nginx that has been optimized over a longer period of time / a more specific use case would have non-negligible performance benefits at scale?

From what I know about RWKV, it's mostly a one man effort and doesn't have the same data pipeline / resources as most major labs. It's a bit unfortunate but I'm curious about the performance given the same training corpus as OpenAI's GPTs. Maybe some labs have tried internally but haven't released results? On the other hand it makes sense to invest more money into transformer training runs as they have been proven to work.

They really burst onto the scene and brought back RNNs in the world of transformers. The claim that RWKV isn't paralleizable during training also seems to be refuted in their readme. I'd guess it's generalizable performance as there is a difference between doing well on benchmarks and being usable. Personally I've tried running the weights a long time ago when it was first released and the results weren't usable but I'm sure there has been considerable progress since then.

curious to hear does anyone know what the mentioned alternatives are? a super simple to use wireguard control plane is super valuable and PiVPN seemed to fit that gap perfectly

unfortunate that it's come to an end but it's nice to hear the maintainer moving on in such a positive way :)

on the fault tolerance side - if it's really this fast why not just store the data in a high-availability database / data lake then run batch processing? it doesn't seem like this use case requires real-time indexing and searching (which may be why elastic is slower as it supports it)

modern mitigations have made exploits much more difficult, however researchers continue to find bypasses to these mitigations.

there are "classic" techniques to bypass most of the modern protections and if there isn't then researchers often come up with novel attacks / bypasses. for example for heap protections you can see how2heap to bypass heap protections[0]. another example is an exploit which allows bypass of KASLR (which has since been patched)[1]. it looks like this exploit comes up with a "dirty pagetable" technique[2].

it's always a game of cat and mouse with more mitigations always being added while researchers constantly look to bypass them.

[0] https://github.com/shellphish/how2heap

[1] https://www.willsroot.io/2022/12/entrybleed.html

[2] https://pwning.tech/nftables/#452-the-technique

this will always be true of jailbreaks as they fundamentally break the security of the device. this means that it can be used for cool things like shown in the article but it could also be used maliciously by an attacker to completely take over the device which the manufacturer should definitely patch.

It seems like you can disable the data being used from training by turning off gemini app activity.

> You can turn Gemini Apps Activity off If you don’t want future conversations reviewed or used to improve machine-learning models, turn off Gemini Apps Activity Opens in a new window .

maybe this is similar to google's patenting of dropout for neural networks? you can never know but so far there haven't been many adverse effects and they claim that they patent it so others can't maliciously patent and enforce it.

dropout patent link: https://patents.google.com/patent/US9406017B2/en

I think it's mainly the change in behavior that could be viewed as concerning. As a user of a website, if the order of websites you visited after browser startup impacts the success or invalid certificate error I feel like it's a pretty confusing experience.

I definitely didn't know this before and if I saw this behavior I would be pretty confused.

the entire open source AI space feels like this right now. basically every day there is some new advancement that either makes something deemed impossible achievable and it's actually really hard to keep up with all the changes.

from what I remember bard should be able to browse the internet and write code internally to better answer queries. I feel like these abilities are just improved with Gemini as a better language model.

to be fair, I think the ability of these models to actually use these contexts beyond the standard 8k / 16k tokens is pretty weak. RAG based methods are probably a better option for these ultra long contexts

One small difference is that the GPT architecture is just the decoder stack of the original transformer as opposed to the full encoder decoder stack in the original.

I agree the branding play on GPTs in general is pretty smart and strong from OpenAI though.