Hybrid AI Infrastructure

The Challenge

Cloud AI APIs are seductive. They're easy to start with, require no hardware investment, and scale effortlessly. But that convenience comes with strings attached—strings I kept tripping over.

Beyond rate limits, I was dealing with unpredictable costs that made budgeting a nightmare, latency that varied based on server load I couldn't control, and the nagging concern about sensitive data leaving our network. Something had to change.

My Approach

I decided to build infrastructure where owned hardware handles everything it can, and cloud models are a deliberate choice rather than a dependency. Not a toy setup for experimentation, but infrastructure my production applications actually run on.

My requirements were clear:

• Run capable open-weight models locally, including tool-calling, coding, reasoning, vision, embedding, and speech-to-text models
• Maintain API compatibility so the same code can talk to local or cloud models
• Reach the GPU from a public server without exposing it to the internet
• Keep it maintainable—I'm not running a data center here

After evaluating several options, I standardized on LM Studio as the inference backbone. It struck the right balance between power and usability, and critically, it exposes an OpenAI-compatible API endpoint. That last point was huge—it meant I could point existing applications at my local infrastructure with minimal code changes.

The Solution

The infrastructure has two halves. A home GPU server runs LM Studio, and a public VPS runs my applications. An always-on WireGuard tunnel connects them, and a Python layer routes each request to the local model or a cloud API. Here's what makes it work:

Hardware Foundation

The inference GPU is an NVIDIA RTX 5060 Ti with 16GB of VRAM. It comfortably runs quantized models in the 8B–20B range, including open-weight tool-calling models, coding models, vision-language models, embeddings, and Whisper for transcription. The key constraint is that only one model fits in VRAM at a time, so the software has to treat "model not loaded" as a normal condition, not an exception.

LM Studio Configuration

LM Studio handles model management, quantization options, and serves the API endpoint. I configured it to auto-load specific models on startup and tuned context window sizes for our typical use cases. The OpenAI-compatible endpoint means our applications don't know (or care) whether they're talking to cloud or local infrastructure.

Private Tunnel

The GPU lives on a home network behind NAT. Instead of opening ports, the home server keeps an always-on WireGuard tunnel to the VPS, sending keepalives so the connection survives router changes. To the applications, the GPU is just a private address with tens of milliseconds of latency. The tunnel is the access boundary, and its addresses never reach a browser.

Python Bridge Layer

I built a Python layer that sits between applications and the models. It routes each request to LM Studio or a cloud API, sets sensible timeouts (first model loads can take most of a minute), and handles the tunnel or the GPU being briefly unavailable without breaking the request. Both backends stream through the same parser, so the application above doesn't care which one answered. My agentic chat platform runs on top of it.

Lessons Learned

This project taught me several things I'd do from day one on similar builds:

• VRAM planning is everything. Understand your model sizes and leave 20-30% headroom for concurrent requests and future growth.
• API compatibility saves massive time. Choosing tools that match existing standards means faster adoption and easier migration paths.
• Start with your actual use cases. I initially over-provisioned for scenarios that never materialized. Let real usage patterns guide optimization.
• Document the setup thoroughly. Future-you (or your replacement) will thank present-you when something needs adjusting at 2 AM.