Scalable FastAPI AI Service & Gateway

Deploying large language model APIs and retrieval pipelines into production environments requires a robust microservice architecture. This project builds a high-performance, containerized inference gateway and RAG service designed to handle scalable agent traffic while protecting the system from security threats.

FastAPI & Agentic Workflow Orchestration

The core service is developed in Python using FastAPI, leveraging asynchronous event loops to manage concurrent API requests and external LLM calls. The system integrates LangChain and LlamaIndex to structure complex, multi-turn agentic workflows. By separating routing logic, prompt templates, and backend inference endpoints, the service handles high-concurrency workloads with minimum overhead.

Hybrid RAG: pgvector & Elasticsearch

To optimize retrieval precision, a hybrid RAG pipeline was designed. Large text corpora are chunked and converted to dense embeddings using OpenAI/Anthropic/Hugging Face models, then indexed in PostgreSQL via pgvector. Sparse keyword retrieval is handled concurrently by Elasticsearch. Query results are merged using Reciprocal Rank Fusion (RRF), establishing a balanced search combining semantic relevance and keyword matches.

Kubernetes Orchestration & LLM Guardrails

The architecture is containerized with Docker and orchestrated on a local Kubernetes cluster, enabling automated horizontal pod scaling (HPA) based on CPU/memory usage. To secure the AI supply chain, a proxy layer driven by NeMo Guardrails validates all incoming user prompts and outgoing LLM completions, preventing malicious prompt injections and toxic outputs from affecting the service.