# Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT

Source: [Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT](<https://www.sota2.com/research/paper/68fa6ea91e209131cea41332>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:21:27.168Z

Authors: Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, Dacheng Tao

Published: 2023-02-19T12:29:33.000Z

## Abstract

Recently, ChatGPT has attracted great attention, as it can generate fluent and high-quality responses to human inquiries. Several prior studies have shown that ChatGPT attains remarkable generation ability compared with existing models. However, the quantitative analysis of ChatGPT's understanding ability has been given little attention. In this report, we explore the understanding ability of ChatGPT by evaluating it on the most popular GLUE benchmark, and comparing it with 4 representative fine-tuned BERT-style models. We find that: 1) ChatGPT falls short in handling paraphrase and similarity tasks; 2) ChatGPT outperforms all BERT models on inference tasks by a large margin; 3) ChatGPT achieves comparable performance compared with BERT on sentiment analysis and question-answering tasks. Additionally, by combining some advanced prompting strategies, we show that the understanding ability of ChatGPT can be further improved.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2302.10198>)

[PDF](<https://arxiv.org/pdf/2302.10198.pdf>)
