ChatGPT Is Down | Hacker News Reader

ChatGPT Is Down

21 pointsnomilk15 comments

https://chat.openai.com/ isn't loading past conversations and won't allow new ones to be created. Anyone else?

Have been having intermittent issues the past couple days. It's times like these I'm happy to have llama running in k8s on GPUs (https://microk8s.io/docs/addon-gpu https://hub.docker.com/r/pytorch/pytorch).

By the way, has anyone noticed they're starting to have an "AI-dependence"?

These AI tools are infiltrating my workflow, and it's becoming more noticeable when they go down...

I have an AI-dependence in the same way that I had a Stack Overflow dependence, and before that a Google dependence, and before that a reference manual dependence, and with that a (book) library dependence.

> It's times like these I'm happy to have llama running in k8s on GPUs

I'd love to know more about this. Do you have your own hardware, or do you run it in the cloud. Did you set everything up yourself. I'm somewhat intimidated by k8s, let alone the other stuff on top, but very curious.

> has anyone noticed they're starting to have an "AI-dependence"?

Yup +1

> Do you have your own hardware, or do you run it in the cloud

Running two AI-based startups. We use GPU compute from "everywhere possible" including cloud and on-prem. Cloud is a combo of auto-scaling clusters with spot instances (GKE, EKS, etc.).

Have done some proofs of concept on serverless (runpod, modal), but they are very unpredictable in terms of latency and scalability.

How do we make it all work? K8s is the magic. The auto-scaling cloud clusters and the local cloud clusters all just look like k8s endpoints.

The configs are deployed with basic k8s yaml + jinja2, but helm is an option (we're trying to keep it simple.)

What are you using for K8s autoscaling? We initially tried a few standard K8s scaling mechanisms and found that they didn't work well for GPU workloads. For example, if we were serving a low-RAM Huggingface model on GPU, it wouldn't trigger autoscaling. But since the GPU can only process one request at a time, the system would get bottlenecked while it waited to process requests one-by-one.

We wrote a bit about this here, if anyone is interested: https://www.beam.cloud/blog/serverless-autoscaling

If you aren’t fully utilizing gpu memory, can you use time slicing? Or MIG for A100?

Orthogonal solution but could help.

Sharing GPUs only really makes sense for GPUs that are large enough to share. MIGs can work for 80Gi A100s but won't work with smaller cards like T4s. It also adds latency to the GPU operations. Unfortunately there's not yet a silver bullet for this stuff.

That’s why I was curious about utilization since you mentioned low memory usage. I believe time slicing can work on those smaller cards these days. Did you explore any other optimizations like batching or concurrency for same model?

Model heterogeneity seems like a real challenge there — you could optimize usage if you know all the sizes ahead of times and actually have gpu capacity to do efficient allocations, but it’s way harder than just doling 1 gpu per pod.

e: also, latency because of reduced resources? Or what do you mean?

yeah I have been using it so much that I definitely know when it's down.

As a paying user, I better get a discount. I use this for certain tedious tasks at work.

Today's tedious task:

A low budget, garbage software company I'm forced to integrate with for billing purposes is complaining about the format of a new CSV file compared to the previous exports being sent. ChatGPT is perfect for identifying the exact format discrepancies, and naturally it's down the day I need it to be up.

Just go to the Playground: https://platform.openai.com/playground

All the same functionality, without the downtime. Has 100% uptime for the past 90 days: https://status.openai.com/

Try Claude or Bard, I am also an AI CSV-parsing enjoyer and they both work well. Claude has size limits unlike ChatGPT’s Continue” button thiugh

I just switch over to using the API, which sucks because then I’m paying for the subscription and the API usage, but I don’t really have a choice. OpenAI uptime isn’t great for ChatGPT.

From the OpenAI status page:

> Degraded performance on ChatGPT

> Aug 31, 2023 - 07:58 PDT: Update - We are continuing to work on a remediation and recover service.

> Aug 31, 2023 - 06:52 PDT: Identified - Some users are experiencing issues loading conversation history and creating new conversations. We have identified the issue and are working on a remediation.

[1] https://status.openai.com/

Same here

Reply on news.ycombinator.com