Adding an OpenAI call to a SaaS product is easy. Adding one that survives real usage — variable latency, rate limits, unpredictable costs, and users who paste in ten thousand words — is a different problem. I've integrated OpenAI and other LLM providers into production products across fintech, logistics, and data-analysis platforms, and the failure modes are consistent enough to write down.
Here's what a production-grade integration actually requires, beyond the first chat.completions.create call that works in a demo.
Never call OpenAI directly from the client
This should be obvious, but it's the single most common mistake in early-stage AI products: an API key embedded in frontend code, or a serverless function that just proxies the request unchanged. Anyone can extract that key from the network tab and run up your bill.
The API key belongs on your backend only. In a NestJS service, that means a dedicated module that owns all OpenAI calls, reads the key from environment configuration, and exposes only the operations your frontend actually needs — never a raw pass-through.
Treat cost as a first-class engineering concern
LLM costs scale with usage in a way most engineers aren't used to reasoning about. A feature that costs $40/month in testing can cost $4,000/month once real users touch it, because token cost compounds with every input token, every output token, and every retry.
Concrete things that keep this under control:
- Cap max output tokens explicitly on every request — don't rely on the model to stop at a "reasonable" length.
- Truncate or summarize long inputs before they reach the model, rather than sending an entire document on every request.
- Cache repeated queries where the input is likely to recur — this alone can cut costs substantially for FAQ-style or analysis features.
- Track token usage per user or per account, not just in aggregate, so you can see which usage patterns are actually driving cost before they become a problem.
- Choose the smallest model that meets the quality bar for each specific task — not every feature needs your most capable model.
Design for latency, not just correctness
A model response can take anywhere from a few hundred milliseconds to tens of seconds, and that variance is normal, not a bug to route around. Two things matter here:
Stream the response. For anything user-facing — chat, report generation, summarization — stream tokens back to the client as they're generated rather than waiting for the full completion. This alone is the biggest perceived-performance improvement you can make, and OpenAI's streaming API supports it natively.
Move long-running generation off the request/response cycle. For anything that takes more than a few seconds — a multi-step report, a batch analysis — use an async job pattern: kick off the job, return immediately, and let the frontend poll or subscribe for the result. Blocking an HTTP request for 30 seconds is fragile and doesn't survive a client disconnect or a proxy timeout.
Handle failure like it's the common case, not the edge case
OpenAI's API returns rate limit errors, timeout errors, and occasional malformed or incomplete responses. In a low-traffic demo, you rarely see these. In production, at real usage volume, you'll see them daily.
A resilient integration includes:
- Exponential backoff with jitter on rate-limit and transient errors, not a naive single retry.
- A fallback response for when the model call fails entirely — never let an AI feature return a raw 500 to the end user.
- Output validation — if you're asking the model for structured data (JSON, a specific format), validate the response before using it. Models occasionally produce output that doesn't match the schema you asked for, and your code needs to handle that without crashing.
- Timeouts on every call — an OpenAI request with no timeout can hang a worker indefinitely under the wrong conditions.
Don't let the model own decisions it shouldn't
For SaaS products handling anything sensitive — financial data, user-facing recommendations, anything with legal or compliance weight — the model should generate content, not make unchecked decisions. Put a validation or business-rule layer between the model's output and any action your system takes automatically. This is the difference between an AI feature that's a delight and one that quietly makes a wrong call nobody notices until a customer complains.
The integration pattern I use by default
For most SaaS features I build, the shape looks like this: a NestJS AiModule that wraps the OpenAI SDK, exposes typed methods per use case (not a generic "send prompt" method), applies per-user rate limiting and token budgets, streams results back through Server-Sent Events or WebSockets where the UX benefits from it, and logs every request's token usage for cost visibility. That structure has held up across every AI feature I've shipped, from conversational finance insights to multi-LLM data analysis pipelines — because it treats OpenAI as a dependency to be engineered around, not a magic function call.
Get this layer right once, and every future AI feature in the product becomes cheaper and safer to add. Skip it, and every new AI feature inherits the same fragility as the first one.
