Ask HN: AI infrastructure in production – what is your tech stack? | Hacker News Reader

Ask HN: AI infrastructure in production – what is your tech stack?

5 pointsk92941 comment

I'm working at an AI startup, and our infrastructure has grown increasingly complex. What started as a few simple model calls has evolved into systems that limit us and slow us down.

Our current setup requires touching multiple different components whenever we add a new feature. We spend more time maintaining infrastructure than building actual products.

The real pain comes when stakeholders get excited about new capabilities. Last week, they saw Google Gemini's new native image generation and editing features and wanted them integrated ASAP. This requires:

- Adding new API integration code to handle LLMs returning images

- Updating a cost calculator to support this case

- Updating our storage system to automatically handle images in LLM responses and store them in S3

- Probably something will blow up that I haven't thought about yet

I'm spending about 70% of development time (if not more) on infrastructure related tasks rather than product features, and this ratio is getting worse as we add more capabilities.

- How is your company handling this complexity? Any recommendations?

- Any open-source tools/frameworks that have significantly reduced your AI infrastructure complexity that more teams should know about?

- Which tools or approaches did you try that became maintenance nightmares and you'd warn others to avoid?

- Any AI-related third-party services you would recommend?

p.s. I will share our stack in the comments. Hit HN's max character limit.

Can't share the company name, but it's a relatively small startup with a few thousand monthly active users.

Here's what our current stack looks like:

- LLM Access: Custom proxy layer to an OpenAI-compatible gateway for chat completion, transcriptions, image generation, embeddings (we started with LangChain but were fighting with it more than it helped us). Now we use the OpenAI SDK + a custom API gateway that handles cost tracking, metered billing, API with model fallbacks, aliases, and feature flags (e.g., supports_structured_output, supports_reasoning, modality: text, image, audio, etc.).

- Workflow Orchestration: We started with a very simple task executor + background job queue to handle retries and durable executions. Now it has become a real limitation, and we are looking for a complete replacement, considering Temporal, trigger.dev, and Conductor. I don't see an obvious winner here, everything looks like a really sophisticated piece of tech that will have a learning curve and require some rethinking of our decisions and serious refactoring to support new flows.

- Observability: OTEL + https://signoz.io, we don't track LLM outputs for security reasons (consumer product, a lot of private stuff).

- Cost Tracking: We embedded cost tracking into our API proxy layer. For each request, we estimate usage and push it into ClickHouse (via TinyBird) to provide analytics of product usage and to our metering/billing provider (Stripe meters, but evaluating Orb to replace Stripe).

- Agent Memory / Chat History / Persistence: Postgres for everything + S3 for files and images. Frankly, it looks like a chat app schema (chat threads, messages, attachments). Relatively simple and straightforward.

- Billing: This was a real pain - a hybrid subscription with a monthly fee + pay-as-you-go for overages. Stripe offers limited support for subscriptions with usage billing and does not support automatic top-ups. Implementing a free plan + paid plan with metered usage required a lot of code and even more tests. I thought billing providers had already solved this?

- RAG: Custom document ingestion service based on LlamaIndex (document extraction, indexing, querying), managed Qdrant for vector search. Pros - works well, quite simple, scales well. Cons - requires an API boundary layer and more work upfront but was quite low-maintenance in the long run.

- Integrations (Tools, MCPs): LLM tools wrappers + user credentials management (API keys, OAuth) to connect users' CRM, Notion, whatever. That's a big pain; we have accumulated quite some debt here because we can't decide on a layer to manage this. There is no open-source "go-to" solution.

Reply on news.ycombinator.com