# Tokenization Preference for Human and Machine Learning Model: An Annotation Study

Source: [Tokenization Preference for Human and Machine Learning Model: An Annotation Study](<https://www.sota2.com/research/paper/68fa72351e209131cea54f03>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:20:14.513Z

Authors: Tatsuya Hiraoka, Tomoya Iwakura

Published: 2023-04-21T08:49:06.000Z

## Abstract

Is preferred tokenization for humans also preferred for machine-learning (ML) models? This study examines the relations between preferred tokenization for humans (appropriateness and readability) and one for ML models (performance on an NLP task). The question texts of the Japanese commonsense question-answering dataset are tokenized with six different tokenizers, and the performances of human annotators and ML models were compared. Furthermore, we analyze relations among performance of answers by human and ML model, the appropriateness of tokenization for human, and response time to questions by human. This study provides a quantitative investigation result that shows that preferred tokenizations for humans and ML models are not necessarily always the same. The result also implies that existing methods using language models for tokenization could be a good compromise both for human and ML models.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2304.10813>)

[PDF](<https://arxiv.org/pdf/2304.10813.pdf>)
