# Financial Question Answering on S&P 500 benchmark Source: [Financial Question Answering on S&P 500 benchmark]() Collection index: [llms.txt]() Data updated: 2026-05-22T08:03:10.555Z Evaluation of Financial Question Answering performance on the S&P 500 benchmark dataset, covering metrics like Forecast QA, Event Detection, and Multi-signal Reasoning. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Financial Question Answering | | Dataset (including any stated split) | S&P 500 benchmark | | Metric | Forecast QA Score | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 14 of 14 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Forecast QA Score](): Higher is better; unit: PERCENT. - [Event Detection Score](): Higher is better; unit: PERCENT. - [Fundamental QA Score](): Higher is better; unit: PERCENT. - [News Analysis Score](): Higher is better; unit: PERCENT. - [Multi-signal Reasoning Score](): Higher is better; unit: PERCENT. - [Average Score](): Higher is better; unit: PERCENT. ## Recorded results | Method | Forecast QA Score | Result date | Report paper | | --- | --- | --- | --- | | STOCK-R1-8B | 56.78 | Not recorded | 6a0fe326f984739ed6739e83 | | STOCK-R1-4B | 51 | Not recorded | 6a0fe326f984739ed6739e83 | | Fin-R1-8B | 40.28 | Not recorded | 6a0fe326f984739ed6739e83 | | DianJin-R1-32B | 38.42 | Not recorded | 6a0fe326f984739ed6739e83 | | Fin-o1-14B | 32.86 | Not recorded | 6a0fe326f984739ed6739e83 | | Qwen3-30B | 10.34 | Not recorded | 6a0fe326f984739ed6739e83 | | GPT-5 | 8.16 | Not recorded | 6a0fe326f984739ed6739e83 | | Qwen3-4B | 7.92 | Not recorded | 6a0fe326f984739ed6739e83 | | Qwen3-8B | 6.23 | Not recorded | 6a0fe326f984739ed6739e83 | | Llama3-70B | 5.35 | Not recorded | 6a0fe326f984739ed6739e83 | | DeepSeek-V3.2 | 3.37 | Not recorded | 6a0fe326f984739ed6739e83 | | Llama3-8B | 3.21 | Not recorded | 6a0fe326f984739ed6739e83 | | OpenAI-o3 | 2.78 | Not recorded | 6a0fe326f984739ed6739e83 | | Llama3-3B | 2.53 | Not recorded | 6a0fe326f984739ed6739e83 | ### STOCK-R1-8B Forecast QA Score: 56.78. Result ID: 6a100dbafbc373592c08d6cd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Financial LLM","Model Size":"8B","Base Model":"Qwen3-8B"}. This record does not assert independent reproduction. ### STOCK-R1-4B Forecast QA Score: 51. Result ID: 6a100dbafbc373592c08d6cc. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Financial LLM","Model Size":"4B","Base Model":"Qwen3-4B"}. This record does not assert independent reproduction. ### Fin-R1-8B Forecast QA Score: 40.28. Result ID: 6a100dbafbc373592c08d6c9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Financial LLM","Model Size":"8B"}. This record does not assert independent reproduction. ### DianJin-R1-32B Forecast QA Score: 38.42. Result ID: 6a100dbafbc373592c08d6cb. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Financial LLM","Model Size":"32B"}. This record does not assert independent reproduction. ### Fin-o1-14B Forecast QA Score: 32.86. Result ID: 6a100dbafbc373592c08d6ca. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Financial LLM","Model Size":"14B"}. This record does not assert independent reproduction. ### Qwen3-30B Forecast QA Score: 10.34. Result ID: 6a100dbafbc373592c08d6c4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"General-purpose LLM","Model Size":"30B"}. This record does not assert independent reproduction. ### GPT-5 Forecast QA Score: 8.16. Result ID: 6a100dbafbc373592c08d6c7. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"General-purpose LLM"}. This record does not assert independent reproduction. ### Qwen3-4B Forecast QA Score: 7.92. Result ID: 6a100dbafbc373592c08d6c2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"General-purpose LLM","Model Size":"4B"}. This record does not assert independent reproduction. ### Qwen3-8B Forecast QA Score: 6.23. Result ID: 6a100dbafbc373592c08d6c3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"General-purpose LLM","Model Size":"8B"}. This record does not assert independent reproduction. ### Llama3-70B Forecast QA Score: 5.35. Result ID: 6a100dbafbc373592c08d6c5. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Category":"General-purpose LLM","Model Size":"70B"}. This record does not assert independent reproduction. ### DeepSeek-V3.2 Forecast QA Score: 3.37. Result ID: 6a100dbafbc373592c08d6c6. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"General-purpose LLM"}. This record does not assert independent reproduction. ### Llama3-8B Forecast QA Score: 3.21. Result ID: 6a100dbafbc373592c08d6c1. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Category":"General-purpose LLM","Model Size":"8B"}. This record does not assert independent reproduction. ### OpenAI-o3 Forecast QA Score: 2.78. Result ID: 6a100dbafbc373592c08d6c8. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"General-purpose LLM"}. This record does not assert independent reproduction. ### Llama3-3B Forecast QA Score: 2.53. Result ID: 6a100dbafbc373592c08d6c0. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Category":"General-purpose LLM","Model Size":"3B"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.