Conversational AI Referral Chatbot
The AI Referral Assistance Chatbot is a natural language AI system designed to collect user needs, create service profiles, and match users with appropriate resources across LA County’s social support ecosystem. Built using cutting-edge RAG techniques and powered by Llama 3, the chatbot understands user intent, retrieves real-time and curated information from 211 LA’s service databases, and presents accurate, eligibility-matched referrals through a conversational interface.
The solution is integrated with internal referral platforms and call center tools. It handles high-volume, routine queries 24/7, freeing up human agents to focus on crisis cases and deeper service navigation. This hybrid approach scales support without scaling costs, ensuring that more Angelenos get the help they need, faster.
Overview
The COVID-19 pandemic exposed a critical gap in access to social services across Los Angeles County. As thousands of residents faced food insecurity, housing instability, and urgent mental health needs, the demand for 211 LA's referral services surged beyond sustainable levels.
​
Residents often found themselves waiting on hold for long periods — only to receive partial or outdated information. For 211 LA staff, the rising volume of emotionally complex calls, combined with repetitive intake tasks, led to skyrocketing stress and burnout. Meanwhile, increasing labor and training costs strained the nonprofit’s already tight budget.​​
​
To address both sides of the problem — overwhelmed users and overworked agents — 211 LA launched an AI Referral Assistance Chatbot, a digital assistant trained using retrieval-augmented generation (RAG) and powered by OpenAI’s models. The chatbot helps residents navigate services with speed and clarity while reducing the burden on live agents, resulting in faster service, higher satisfaction, and smarter use of organizational resources.
AI Hypothesis
If the AI Referral Assistant can accurately extract user intent and household context from natural language conversations, and match users with relevant, real-time service referrals with at least 85% accuracy, then users will be able to access helpful services without waiting for a human agent, and will complete their referral journey in under 5 minutes.
​
This will lead to a measurable reduction in Average Handle Time (AHT) by at least 30% and an increase in user satisfaction ratings by 10% or more. As a result, 211 LA will experience lower call volume, reduced agent workload, and increased capacity to serve more residents — all while containing operational costs and improving overall service delivery efficiency.
Customer Research
To deeply understand both the organizational and community needs driving this product, I led a multi-pronged research effort that included stakeholder interviews, agent conversations, call analysis, and direct outreach to 211 LA callers. This qualitative discovery work gave our team a grounded, empathetic view of the real-world pain points the AI Referral Assistant needed to solve.
​
Stakeholder Insight
“We’re not trying to replace humans, since people do need to talk with someone when they are in between something — we just can’t keep scaling with more people. Our agents are exhausted, costs are rising, and we need a smarter way to serve more people, better.” — Senior Operations Lead, 211 LA
​
Agent Insight
“If the chatbot can just handle the basic intake — household size, ZIP code, what they’re looking for — that would save me so much time. Right now, we’re asking the same questions over and over.” — Call Center Agent, 211 LA
​
Caller Follow-up Insight
“When I finally got through, the agent was great. But I wish there was a faster way. It’s hard when you’re in crisis and you’re just waiting and waiting.” — 211 County Resident (From post-call survey)
Customer Segments
While the 211 LA Referral Assistant Chatbot ultimately serves a diverse population, I highlight four core customer segments and subcategories of each that represent large groups of users and stakeholders.
This is a strong foundation for understanding the chatbot’s impact across key user types.
01
Callers / Help Seekers
-
Low-income individuals
-
Seniors
-
People with disabilities
-
Single parents
-
People experiencing housing insecurity or utility shut-offs
-
People facing food insecurity​
​
Pain Points:
1. In urgent situations but struggle with long wait times to reach live assistance
2. Overwhelming online information make it hard to find the right service
3. Language barriers
4. Feel difficult to clearly explain the situation on the phone due to stress
03
Call Center Agents
-
Full-time 211 agents
-
Part-time / temporary call center staff
-
Trainee or recently onboarded agents​​​​​​
​
​
​
Pain Points:
1. High call volume and emotional labor leading to stress and burnout
2. Inconsistent referral quality due to varied agent experience levels
3. Limited bandwidth for follow-ups or deeper engagement with callers
​
02
Social Helpers
-
Social workers at nonprofits
-
Caseworkers
-
Family members or friends assisting someone
-
School counselors and teachers​​​​
​
​
​
Pain Points:
1. Inefficiency due to lacking a centralized, up-to-date database for services and eligibility
2. Limited training in navigating public assistance systems
3. Difficulty matching the right resources to specific client needs
​​​
04
211 LA Organization
-
Stakeholders and Leaders
-
Finance management teams
-
Agent trainers and quality assurance specialists
​​​​​​​​​​​​​​​​​​​​
Pain Points:
1. Difficulty scaling services to meet rising demand with the limited budget
2. Locking tools for standardized agent onboarding, performance monitoring
3. High operational costs in both time and finance from training new staff
Persona 1
Persona 2
Customer Journey
Based on our interviews and research, we identified several primary customer segments, including low-income families seeking basic needs, digitally underserved residents, and 211 LA agents managing high call volumes. We also identified secondary customer segments such as multilingual callers, residents seeking mental health services, and partner organizations referring clients.
We chose to focus our MVP on low-educated callers and exhausted frontline agents, a group with an urgent, clearly defined problem: navigating complex service systems under stress, with limited access to timely help.
This group is often juggling multiple life pressures — child care, housing instability, language barriers — while trying to make sense of public resources. They need simple, fast, human-like interactions that don't require them to understand complex systems or wait on hold for long periods.
By targeting them first, we can maximize early impact, reduce friction in the highest-volume use cases, and validate our AI system’s ability to handle real-world needs in a compassionate and scalable way — laying the groundwork for broader expansion in future phases.
AI Input & AI Output
User Input
Unstructured:
- Free-text questions (e.g., “I need help paying rent”)
- Follow-up context (“I have two kids”, “I lost my job”)
- Language tone (urgency, confusion, distress)
Structured:
- ZIP code
- Household size and income
- Demographic flags (e.g., veteran, disability)
- Preferred contact method
​
System Input
Real-time service availability from referral database
Policy/eligibility rules (via vector store + documents)
Past chatbot interaction logs
​
System Output
- Clarification Questions: “Are you currently at risk of eviction?” or “Do you have a regular source of income?”
- Referral Recommendations: Names, contact info, and descriptions of services
Justifications for matches
- User Summary & Agent Handoff Payload: A structured summary of user needs + pre-filled fields for CRM/agent tools
Data Pipeline
The data pipeline handles real-time interactions, retrieval, and feedback capture, with an emphasis on privacy and resilience.
​
1. Data Ingestion
User messages captured via chatbot UI (web, mobile)
User metadata collected through structured form fills and intent parsing
​
2. Preprocessing
NLP-based entity recognition and intent classification
Normalization (e.g., converting “I lost my job” → unemployment_status = true)
​
3. Retrieval Layer
Embedding-based search against vector store containing:
Service descriptions
Eligibility policies
Past cases/anonymized session summaries
Service Taxonomy + LLM-Generated Tags
FAQs and scripts
Real-time queries to 211 referral database
​
4. LLM Invocation
Prompt construction using:
User intent + structured metadata
Retrieved service data
Llama 3 generates output with custom constraints (e.g., max reading level)
​
5. Output Delivery
UI response (chat message, call transcript)
Optional SMS/email follow-up
​
6. Feedback & Logging
User satisfaction rating (thumbs up/down)
Tech Stack
------------------------------------------------------------
Frontend :React, Webchat SDK (custom UI layer)
----------------------------------------------------
Backend: Node.js / Express API Gateway
-----------------------------------------------------------
AI Model: Llama 3 + custom RAG module
--------------------------------------------------------
Database: PostgreSQL (user sessions), Redis (state mgmt), iCarol API (referrals)
------------------------------------------------
Hosting & Infra: Azure Cloud
----------------------------------------------------------
Architecture
User Stories & Acceptance Criteria
Success Metrics
AI Product Scability
Vector Store + HFRL + Corner Case Hardening
The AI chatbot is designed with scalability at its core, ensuring it continues to perform effectively as user volume and complexity grow. Inspired by the concept of Human Feedback Reinforcement Learning (HFRL), a specialty human review loop is introduced, allowing 211 experts to audit and correct low-confidence or inaccurate responses. This process reinforces the model with high-quality domain knowledge and significantly reduces hallucinations. Additionally, by refining the service taxonomy and continuously updating the vector database, the chatbot's knowledge base stays current and context-aware. Together, these systems ensure the product is not only scalable, but also increasingly reliable, safe, and adaptive as it evolves.