The FCA has published a research note on the effectiveness of large language models (LLMs) like OpenAI’s GPT series in consumer-facing financial services.
The research conducted two pilot projects: asking GPT models to generate simplified definitions of complex financial terms, tailored to specific reading ages and supported by examples; and comparing the effectiveness of consumer guidance on cash savings queries generated by LLMs with responses under a traditional website FAQ format.
The key findings were as follows:
- While LLMs have strong potential to simplify complex information, enhance readability and accessibility, validating their outputs requires a robust evaluation framework that combines human judgment with automation.
- LLM effectiveness is dependent on context – outcomes like user comprehension and engagement were influenced by how the model was embedded within the customer journey, including design and delivery.
- There is a strong appetite for AI-drive assistance, with many users responding positively to automated support.
The FCA has also published an engagement paper which outlines proposals for live AI model testing pilots.