In this blog, I’ll walk you through the design, development, and lessons learned while building a multi-agent AI customer support assistant using the LangChain framework and related AI tools. 🎮💬
🎯 Motivation: Why Build This?
At KGeN, a game aggregation platform connecting publishers and gamers, our primary users are gamers and clan chiefs (micro-community leaders).
These users often ask questions about:
- Platform features
- Game-specific achievements
- Player and clan statistics
Some answers come from a static knowledge base, while others depend on dynamic user-specific data.
⚡ We wanted an intelligent, scalable AI assistant that could:
- Understand natural language queries
- Route them to the appropriate data sources
- Continuously improve through feedback
Based on the poc feedback, I wanted to take this to production.
🧠 The use case generalizes to any industry with static documentation and dynamic user data—only the context changes.
🔗 Application & Code
The poc application is deployed in Render, you can try this out. The Github also contains instructions to run it locally or in cloud.
- 🌐 Live Demo (Needs pre-heating, if the query does not work, execute this curl:
curl -s https://support-assistant.onrender.com/ > /dev/null) - 💻 GitHub Repository
📌 Business Goals
I wanted a system with following business goals:
- 🗣 Answer queries conversationally
- ⚙️ Route questions to the right agent (static/dynamic/hybrid)
- 🧾 Escalate unresolved issues via Jira tickets
- ⭐ Collect feedback for iterative improvements
- 📚 Learn from feedback to enhance performance
🧪 Prototyping Approach
I followed a “vibe coding” model:
- Start fast with a working prototype
- Use AI to assist (ChatGPT + Cursor editor)
- Iterate with real feedback
💡 Tools Used:
- ChatGPT to generate mock data (static + SQL)
- Langchain/Langsmith as agentic framework
- Cursor for AI-assisted coding
- Render for cloud deployment
⚠️ Tip: Feed detailed requirements to AI code editors. Without clarity, they produce unreliable or messy code.
🧭 Agent Flow: How It Works
Each user query is first routed by a Main AI Agent, which classifies the query as:
- 📘 Static: Uses vector search on documentation (FAISS)
- 🗄 Dynamic: Converts to SQL query on structured data
- 🔁 Hybrid: Mixes both static + dynamic sources
- 📥 Follow-Up: Needs more user input
- 🚨 Escalation: Routed to a human via Jira
Each type has a specialized agent with its own system prompt.
🧠 LangChain powers the routing, agents, and execution logic.
📌 Architecture Diagram: Agent Flow
🧱 Architecture & Tech Stack
| Component | Tool / Framework | Reasoning |
|---|---|---|
| Agent Framework | LangChain | Modular, battle-tested |
| Monitoring | LangSmith | Easy trace/debug for agents |
| Vector DB | FAISS | Simple to set up for POC |
| LLMs | OpenAI (pluggable) | Can switch to others like Claude |
| Backend API | FastAPI | Lightweight, async-friendly |
| Frontend (POC) | Streamlit | Quick prototyping |
| Deployment | Render | Easy cloud deployment |
| Ticketing | Jira API | For support escalations |
| DB (Local/Test) | SQLite | Lightweight |
| DB (Production) | Postgres | Scalable |
🐞 Issues & Learnings
This whole application took me around 8-10 hours over a period of 2 weeks. I got time to spend only on weekends to do this.. Following are some issues I faced:
- 🧩 Dependency Hell: LangChain and LLM libs change fast. Cursor couldn’t resolve pip issues well. I had to request cursor to get latest details in internet to resolve it.
- 🧪 Streamlit Cloud Problems: Ended up moving to Render for better compatibility.
- 🌍 Env File Confusion: Environment-specific bugs were hard to debug in prod as Cursor does not integrate with Render deployment.
🔍 Debugging with LangSmith
Langsmith is great to understand if the agentic workflow is working as expected. I was able to fix the following issues with Langsmith:
- 🔎 Identified issue that search from vector database is giving the whole static knowledge base instead of giving the specific context. Adding semantic analysis to vector database match helped solve this.
- 🧩 Fix hybrid agent’s output merging logic
- 🔁 Debug why hybrid/support queries didn’t escalate to Jira
📂 Sample Queries along with Langsmith trace
Static query example: What are legendary items?
From the above trace, we can see that there are 2 LLM chain calls and 1 call to vector database. The first chain call is to identity the type of query and the second chain call is to summarize the response from vector database.
Hybrid query example: How many gold achievements has DragonSlayer99 earned and what rewards do they give?
From the above trace, we can see that there are 6 chain calls in the above query:
- first to identity type of query
- second to summarize results of vector database
- third to check if there is username in the query
- fourth to generate sql query and get results from postgres db
- fifth to take the results from sql query and generate summarized response
- sixth to combine the summarized static data and dynamic data to give response to the user
📋 Requirements Summary (Generated via ChatGPT)
I fed these requirements as initial prompt into cursor after few iterations of discussions with chatgpt.
💳 Business Requirements
- Build an AI system that can:
- 📘 Answer static queries from documentation
- 🗄 Query live backend data
- 🔁 Combine static + dynamic sources
- 📥 Handle follow-up interactions
- 🚨 Escalate to Jira when needed
- Serve two user roles:
- 👤 Gamers (general players)
- 👑 Clan Chiefs (advanced users)
- Goals:
- Reduce manual tickets by 70%
- Improve first-response time
- Maintain conversational accuracy
📈 Functional Requirements
- Static Question Answering
- Vectorize and index knowledge base with FAISS
- Use RAG (Retrieval-Augmented Generation) to answer
- Dynamic Question Answering
- Use LangChain SQL Agent to convert natural language to SQL
- Query SQLite (for testing) and Postgres (in prod)
- Hybrid Handling
- Mix RAG results with SQL data for composite answers
- Follow-Up Logic
- Prompt for missing data (e.g., usernames)
- Escalation
- Auto-create Jira ticket with conversation context if unresolved
- Multi-Agent System
- Router → specialized agents (static, dynamic, hybrid, etc.)
- API & UI
- FastAPI for backend
- Streamlit for POC frontend; React for future UI
🚀 Technical Requirements
- LangChain (Python)
- FAISS or ChromaDB for vector storage
- OpenAI or Claude LLMs
- SQLite (via LangChain SQL agent) as simulated backend
- JIRA API for ticket creation
- FastAPI (backend API layer)
- Streamlit (prototyping UI)
- React (future UI)
🔐 Security & Testing
- Role-based access (gamers vs. clan chiefs)
- Environment variable protection
- Unit tests, evaluation prompts, and simulated load
✅ Success Criteria
- 90%+ accurate responses in test cases ✅
- Sub-3-second latency ✅
- Smooth Jira escalation pipeline ✅
- API ready for frontend integrations ✅
🚀 Final Thoughts
This project demonstrates how AI agents, vector databases, LLMs, and good system design can solve real-world support problems.
There are many improvements needed to take this into production. Following are some of them:
- Use Langchain conversation memory. This is maintained at streamlit level to stitch a conversation.
- RBAC based on user login and queries based on the user.
- Performance improvement using caching at different levels, database connection optimisation.
- Agent self learnings from user feedback
- Improve UI/UX
🔄 This customer support agent is a template that can be adapted across industries—from gaming to banking to e-commerce.
Sreenivas Makam's Blog 