FCA tests AI large language models on consumer guidance

The FCA has published a research note�on the effectiveness of large language models (LLMs) like OpenAI’s GPT series in consumer-facing financial services.

The research conducted two pilot projects: asking GPT models to generate simplified definitions of complex financial terms, tailored to specific reading ages and supported by examples; and comparing the effectiveness of consumer guidance on cash savings queries generated by LLMs with responses under a traditional website FAQ format.

The key findings were as follows:

• While LLMs have strong potential to simplify complex information, enhance readability and accessibility, validating their outputs requires a robust evaluation framework that combines human judgment with automation.
• LLM effectiveness is dependent on context – outcomes like user comprehension and engagement were influenced by how the model was embedded within the customer journey, including design and delivery.
• There is a strong appetite for AI-drive assistance, with many users responding positively to automated support.

The FCA has also published an engagement paper which outlines proposals for live AI model testing pilots.