What we learned building an agent to automate expense approvals.
Overview
The article discusses the development of LLM-backed agents at Ramp, focusing on building trust through transparency, user control, and collaborative feedback. It outlines strategies for optimizing AI performance in finance-related tasks, emphasizing the importance of clear reasoning and user engagement.
What You'll Learn
1
How to identify finance problems suitable for LLM application
2
Why clear reasoning is critical for LLM trustworthiness
3
How to implement user-driven control mechanisms in AI systems
4
How to create collaborative feedback loops for LLM improvement
Key Questions Answered
What are the best applications for LLMs in finance?
LLMs excel in finance when applied to ambiguous problems that require high volume processing and where the value of automation outweighs the cost of occasional errors. Examples include approving expenses, identifying credit card merchants, and parsing receipts.
How can LLMs build user trust?
User trust can be built by clearly explaining the reasoning behind LLM decisions, allowing users to modify context, and providing mechanisms for users to control agent actions. This transparency and collaboration enhance reliability and user confidence.
What role do escape hatches play in LLM systems?
Escape hatches allow LLMs to express uncertainty, which helps prevent hallucinations. By informing users when the AI is unsure, it encourages a feedback loop that improves decision-making and builds trust.
How should confidence scores be handled in LLM outputs?
Instead of providing confidence scores, which can be misleading, LLMs should categorize outcomes into actionable states like 'Approve', 'Reject', or 'Needs review'. This approach is clearer and more useful for users.
Key Statistics & Figures
Percentage of expense approvals handled by the agent
65%
Since enabling the policy agent at Ramp, more than 65% of approvals have been fully handled by the agent.
Key Actionable Insights
1Implement clear reasoning in LLM outputs to enhance user trust.By explaining the rationale behind decisions, users can better understand and verify outcomes, which is crucial in finance where trust is paramount.
2Create escape hatches in your LLM systems to handle uncertainty effectively.Allowing LLMs to indicate when they are unsure can prevent erroneous outputs and foster a more reliable user experience.
3Encourage user collaboration in defining context for LLM decisions.This not only improves the accuracy of the LLM but also empowers users, making them feel more in control of the AI's actions.
4Utilize evals as a continuous improvement tool for LLM systems.Regularly assessing LLM performance through evals helps identify weaknesses and ensures the system evolves effectively over time.
Common Pitfalls
1
Relying too heavily on LLM confidence scores can lead to misleading outcomes.
Confidence scores from LLMs are often not numerically relevant and can create a false sense of certainty. Instead, categorizing outcomes into actionable states is more effective.