Ask HN: What can the ChatGPT history bug teach us?
Scenarios for the bug:
1. heavy load that made some service(s) to fail unpredictably
2. dev team tried to scale and provisioning/deployment went wrong
3. some other software/logic bug
Useful info to consider:
- before the bug, many people including myself complained that they couldn't see the chat history, and there was a message saying: "please sign out and log in again if you experience this", and it worked, i could see my previous chats after logging back in
- this indicates that the SSO JWT token was used to determine the chats history owner
- it's known that horizontal websocket scaling patterns introduce the risk of syncing the wrong data if not handled properly, but i doubt they used websockets to retrieve the chat history, and i guess they used HTTP for it
- multi-tenancy per user is out of discussion if this bug occurred
- there is the question if their DB was vertically or horizontally scaled. it feels like a WHERE clause was made based on some ID (that proved non-unique) and returned the wrong data. if there was the case of a big shared DB, the WHERE clause would've correctly returned the right data, so now it feels they horizontally scaled the DB and the WHERE clause matched an ID from another instance / partition (PostgreSQL for example guarantees uniqueness of indexes across partitions from what i know)
- most probably they also used a load-balancer. i imagine they made the user session sticky and tied it to a particular backend instance (which in turn might had used a specific DB instance)
- maybe the auth service failed to decode the JWT correctly and the history service failed to check/authorize chat retrievals based on the decoded user ID
- what happens if the load balancer fails in unexpected ways and starts stickying users to the wrong backend instances that are tied to their own DB instances/partitions where chat/user IDs stop corresponding to their original owner? or what happens if some replicated backend instances go down and the load balancer sticks sessions to the remaining available instances?
- what failed? the cloud architecture, the app implementation, the cloud provider, the 3rd party software?
Maybe these are the wrong question, but let's find out!
Note: i want to level things out once again and say that if this post gets upvoted, it might end up putting stress on the business and dev teams. this is not my intention and in reality these problems are common, regardless of company size. i'm an OpenAI customer, I deeply respect their work and will continue using their services