AI Financial Bots Fail to Answer SEC-based Queries
A startup, Patronus AI, discovered that large language models, such as the one used in ChatGPT, struggle to answer questions based on Securities and Exchange Commission filings. Even the best AI model tested, GPT-4-Turbo, was only able to get 79% of answers right, when given nearly the entire filing alongside the question. The models often refused to answer or provided incorrect information not found in the SEC filings. This presents challenges for AI models as companies in finance hope to incorporate cutting-edge technology into their operations.
FinanceBench: A New Test for AI Models
Patronus AI developed FinanceBench, a test consisting of over 10,000 questions and answers from SEC filings of major companies. The test aims to set a “minimum performance standard” for large language models in the financial sector. The test found GPT-4-Turbo unable to answer 88% of questions without access to underlying SEC documents, and even when given access, the model’s performance was still suboptimal.
AI Models Need to Improve
Patronus AI’s findings revealed that AI models are not performing well in the financial sector, even with access to relevant source text. For regulated industries, even a small margin of error is unacceptable. The cofounders believe there is great potential for language models like GPT to help people in the finance industry, but that improvements need to be made for automation to be successful. What do you think? Do you believe AI bots will be successful in automating processes in the finance industry? Let us know your thoughts in the comments!
IntelliPrompt curated this article: Read the full story at the original source by clicking here