Conversational AI Referral Chatbot
An Conversational AI assistant that helps Los Angeles County residents get social services referrals faster using LLM and Retrieval-Augmented Generation (RAG).
Timeline: 3 months
Key Responsibilities:
• Customer interviews
• User data analysis
• Product requirements
• AI workflow design
• Evaluation metrics
• MVP prioritization
• Sandbox testing and evalution
• Roadmapping
Impact:
- Reduced Average Call Handling Time (AHT) by 32%
- Increased Customer Satisfaction Rate (CSAT) by 11%
Tech:
LLM
RAG
Problem
During the COVID-19 pandemic, demand for 211 LA's social services surged as more residents sought help with food assistance, housing, healthcare, and mental health resources. At the same time, the nonprofit faced limited staffing and rising operational costs, making it increasingly difficult to meet community needs.
Residents often waited on hold for long periods before reaching an agent, only to repeat basic intake questions before receiving referrals. For call center agents, handling repetitive information collection left less time for complex cases and contributed to heavy workloads and burnout.
The challenge was finding a scalable way to help more people while maintaining service quality without continuously increasing staffing costs.
Customer Segments
The AI Referral Assistant serves multiple stakeholders across the social service ecosystem. Through customer discovery, we identified two primary user groups and two secondary stakeholders whose needs shaped product priorities and success metrics.
01
Residents Seeking Social Services (Primary)
Who they are:
-
Individuals seeking food, housing, healthcare, or financial assistance
-
Seniors, veterans, people with disabilities, and single parents
-
Residents experiencing housing or utility insecurity
Needs:
-
Find the right services quickly
-
Receive trustworthy step-by-step guidance during stressful situations
Pain Points:
-
Long wait times before speaking with an agent
-
Difficult website navigation and information overload
-
Language and literacy barriers
-
Difficulty explaining complex situations under stress
02
Call Center Agents (Primary)
Who they are:
-
New and experienced 211 call center agents
Needs:
-
Help more residents
-
Reduce repetitive work
-
Get instruction when case is extremely complicated
Pain Points:
-
Repetitive intake questions every call
-
High call volume and emotional burnout
-
Manual eligibility lookup and referral matching
-
Long training time
-
Lack of support support
03
Community Partners (Secondary)
Who they are:
-
Social workers
-
Family members or friends supporting residents
-
School counselors and nonprofit organizations
Needs:
-
Connect clients with appropriate services
-
Access up-to-date service information
Pain Points:
-
Difficult to identify the right programs
-
Information scattered across multiple organizations
-
Eligibility requirements constantly changing
04
211 LA Leadership (Secondary)
Who they are:
-
Stakeholders and Leaders
-
Finance management teams
-
Training and quality teams
Needs:
-
Scale services sustainably
-
Improve operational efficiency
-
Maintain service quality
-
Improve customer satisfaction
-
Ensure regulatory and organizational compliance
Pain Points:
-
Rising staffing and training costs
-
Inconsistent service quality across agents
-
Maintaining compliance while scaling services
Persona
Customer Discovery
To validate the problem and identify the highest-impact opportunities, I conducted stakeholder interviews, call center agent interviews, call recording analysis, post-call surveys, and website analytics. Combining qualitative research with five years of Google Analytics data allowed us to uncover user needs, operational bottlenecks, and opportunities where AI could create the greatest value.
What we Learned
Operations Leadership
“We’re not trying to replace humans, since people do need to talk with someone when they are in between something — we just can’t keep scaling with more people. Our agents are exhausted, costs are rising, and we need a smarter way to serve more people, better.”
Call Center Agent
“If the chatbot can just handle the basic intake — household size, ZIP code, what they’re looking for — that would save me so much time. Right now, we’re asking the same questions over and over.”
Community Members
“When I finally got through, the agent was great. But I wish there was a faster way. It’s hard when you’re in crisis and you’re just waiting and waiting.”
Customer Experience
with
211 LA
AI referral Assistant
1️⃣ Resident looks for help
An LA resident opens the 211 LA website looking for food assistance, "I don't have money to buy food"
2️⃣ AI-guided intake
The chatbot asks a few simple questions, "Let me help you. What's your zip code? How many people are there in your family? Do you have children or seniors at home? Do you have a job? Do you have transportation?"
3️⃣ Search most-relevant social service resources
Based on the conversation, the chatbot recognize it is a food shortage problem. It retrieves current community resources and eligibility requirements and generate answer based on the protocols
4️⃣ Give step-by-step guidance
The chatbot explains the available options using simple, empathetic language, "There are two options. Option one, go to the food bank. The nearest one is at [address]. Open time is [time]. Option two for you is call xxx-xxx-xxxx, tell them your situation, and they will send food to your place if you don't have a way to transport."
5️⃣ Human support when needed
If the chatbot detects sensitive word / uncertainty or a complex situation / the resident asks for an escalate, it transfers the conversation to a live agent.
Instead of starting over, the agent already receives: conversation summary, user profile and suggested protocols
Feature Prioritization
Feature Frequency Impact Engineering Effort Priority
----------------------------------------------------------------------------------------------------------------------------
Mobile Accessible High High Medium P0
Human escalation Low Very High Low P0
Multilingual support Medium Medium Low P1
Personalized referral High High High P0
AI-guided intake High High Low P0
System Design
AI Input & AI Output
User Input
Unstructured:
- Free-text questions (e.g., “I need help paying rent”)
- Follow-up context (“I have two kids”, “I lost my job”)
- Language tone (urgency, confusion, distress)
Structured:
- ZIP code
- Household size and income
- Demographic flags (e.g., veteran, disability)
- Preferred contact method
System Input
Real-time service availability from referral database
Policy/eligibility rules (via vector store + documents)
Past chatbot interaction logs
System Output
- Clarification Questions: “Are you currently at risk of eviction?” or “Do you have a regular source of income?”
- Referral Recommendations: Names, contact info, and descriptions of services
Justifications for matches
- User Summary & Agent Handoff Payload: A structured summary of user needs + pre-filled fields for CRM/agent tools
Data Pipeline
The data pipeline handles real-time interactions, retrieval, and feedback capture, with an emphasis on privacy and resilience.
1. Data Ingestion
User messages captured via chatbot UI (web, mobile)
User metadata collected through structured form fills and intent parsing
2. Preprocessing
NLP-based entity recognition and intent classification
Normalization (e.g., converting “I lost my job” → unemployment_status = true)
3. Retrieval Layer
Embedding-based search against vector store containing:
Service descriptions
Eligibility policies
Past cases/anonymized session summaries
Service Taxonomy + LLM-Generated Tags
FAQs and scripts
Real-time queries to 211 referral database
4. LLM Invocation
Prompt construction using:
User intent + structured metadata
Retrieved service data
GPT 4 generates output with custom constraints (e.g., max reading level)
5. Output Delivery
UI response (chat message, call transcript)
Optional SMS/email follow-up
6. Feedback & Logging
User satisfaction rating (thumbs up/down)
Tech Stack
------------------------------------------------------------
Frontend :React, Webchat SDK (custom UI layer)
----------------------------------------------------
Backend: Node.js / Express API Gateway
-----------------------------------------------------------
AI Model: GPT-4 + custom RAG module
--------------------------------------------------------
Database: PostgreSQL (user sessions), Redis (state mgmt), iCarol API (referrals)
------------------------------------------------
Hosting & Infra: Azure Cloud
----------------------------------------------------------
Architecture
AI Evaluation
Success was measured across three layers: product, business, and AI model performance.
The goal is to ensure the referral assistant delivered meaningful user outcomes: increase 211 LA's referral efficiency while offering a more reliable, stable and trustworthy referral service.
Experiment Results:
- CSAT increased by 11% from 3.6/5 to 4/5.
- Average Call Handling Time was reduced by 32%, from 14.7 mins to less than 10 mins.
- Hallucination rate < 5%
- Referral Accuracy > 85%
1 / Product Success
-
CSAT Measures user satisfaction after completing the conversation.
-
Average Call Handling Time Measures reduction in human agent workload
2 / Business Impact
-
Referral Completion Rate Measures whether users successfully connect with recommended services
-
Containment Rate Measures the percentage of conversations resolved without human agent intervention
3 / Model Evaluation
-
Referral Accuracy How many referrals given by the model align with what human agent would do
-
Hallucination Rate How many referrals are generated without grounded protocol from knowledge base
-
Retrieval Recall How many relevant information segments have been retrieved for answer generation
Product Evolution and Roadmap
Phase 1 - Sandbox
Goal: Validate AI quality before exposing users to the system.
Scope: Build and validate RAG pipeline; Build knowledge base and refine internal protocol; Optimize prompt engineering and retrieval quality
Success: We improved document chunking strategy and retrieval pipeline; refined the internal protocols; built up test set and finally made Hallucination Rate < 5%, referral accuracy > 85%; kept a stable end-to-end inference pipeline
Phase 2 - MVP
Goal: Demonstrate measurable improvements in user experience and efficiency.
Scope: AI-guided Intake, personalized referral, human escalation, Internal pilot using historical and simulated cases
Success: The historical data indicated the average call handling time could be decrease by 32% (from 14.7 min to <10 min). The customer satisfaction rate could be increased to 4/5 (from 3.6/5 to 4.0/5).
Phase 3 - 50% Rollout
Goal: Validate production stability with real users while minimizing deployment risk.
Scope: Add multilingual support and partially rollout to 50% of live traffic; Feedback-driven optimization
Success: Pending Launch
Phase 4 - Full Rollout
Goal: Scale the assistant across the organization while maintaining product quality.
Scope: Full production rollout; Continuous monitoring and model improvement; Knowledge base maintenance
Success: Pending Launch