# Do We Need Zero Training Loss After Achieving Zero Training Error? Source: [Do We Need Zero Training Loss After Achieving Zero Training Error?]() Collection index: [llms.txt]() Data updated: 2025-10-25T23:05:39.232Z Authors: Takashi Ishida, Ikko Yamane, Tomoya Sakai, Gang Niu, Masashi Sugiyama Published: 2020-02-20T12:50:49.000Z ## Abstract Overparameterized deep networks have the capacity to memorize training data with zero \\emph{training error}. Even after memorization, the \\emph{training loss} continues to approach zero, making the model overconfident and the test performance degraded. Since existing regularizers do not directly aim to avoid zero training loss, it is hard to tune their hyperparameters in order to maintain a fixed/preset level of training loss. We propose a direct solution called \\emph{flooding} that intentionally prevents further reduction of the training loss when it reaches a reasonably small value, which we call the \\emph{flood level}. Our approach makes the loss float around the flood level by doing mini-batched gradient descent as usual but gradient ascent if the training loss is below the flood level. This can be implemented with one line of code and is compatible with any stochastic optimizer and other regularizers. With flooding, the model will continue to "random walk" with the same non-zero training loss, and we expect it to drift into an area with a flat loss landscape that leads to better generalization. We experimentally show that flooding improves performance and, as a byproduct, induces a double descent curve of the test loss. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [AI-generated text detection on Cross-genre AIGT Overall (test)](): AI-generated text detection; Cross-genre AIGT Overall (test); OA: 85. - [AI-generated text detection on HC3 (test)](): AI-generated text detection; HC3 (test); F1 (Overall): 99.89. - [AIGT detection on HC3 Deep-Word-Bug attack Overall (in-domain)](): AIGT detection; HC3 Deep-Word-Bug attack Overall (in-domain); OA: 100. - [AIGT detection on HC3 Pruthi attack Overall (in-domain)](): AIGT detection; HC3 Pruthi attack Overall (in-domain); Overall Accuracy: 100. - [AIGT detection on HC3 PWWS attack, AI to Human (in-domain)](): AIGT detection; HC3 PWWS attack, AI to Human (in-domain); Overall Accuracy: 100. - [AI-Generated Text Detection on mixed-source AI -\> Human GPT-2, GPT-Neo, GPT-J, LLaMa, GPT-3](): AI-generated text detection; mixed-source AI -\> Human GPT-2, GPT-Neo, GPT-J, LLaMa, GPT-3; Overall Accuracy: 96. - [AIGT detection on HC3 Deep-Word-Bug attack AI to Human (in-domain)](): AIGT detection; HC3 Deep-Word-Bug attack AI to Human (in-domain); Overall Accuracy: 100. - [AIGT detection on HC3 Pruthi attack AI to Human (in-domain)](): AIGT detection; HC3 Pruthi attack AI to Human (in-domain); Overall Accuracy: 100. - [AI-Generated Text Detection on mixed-source Human -\> AI GPT-2, GPT-Neo, GPT-J, LLaMa, GPT-3](): AI-generated text detection; mixed-source Human -\> AI GPT-2, GPT-Neo, GPT-J, LLaMa, GPT-3; Overall Accuracy: 95.5. - [AI-Generated Text Detection on SeqXGPT-bench (test)](): AI-generated text detection; SeqXGPT-bench (test); Precision (AI): 94.91. - [AIGT Detection on Cross-Domain (AI -\> Human)](): AIGT detection; cross-domain AIGT detection AI -\> Human; Overall Accuracy: 90. - [AIGT detection on HC3 PWWS attack, Human to AI (in-domain)](): AIGT detection; HC3 PWWS attack, Human to AI (in-domain); OA: 100. - [AIGT detection on HC3 Deep-Word-Bug attack Human to AI (in-domain)](): AIGT detection; HC3 Deep-Word-Bug attack Human to AI (in-domain); Overall Accuracy: 100. - [AIGT detection on HC3 Pruthi attack Human to AI (in-domain)](): AIGT detection; HC3 Pruthi attack Human to AI (in-domain); OA: 1. - [AI-Generated Text detection on SeqXGPT-Bench cross-genre](): AI-generated text detection; SeqXGPT-Bench cross-genre; Precision (AI): 83.75. - [AIGT Detection on Cross-Domain Overall](): AIGT detection; cross-domain AIGT detection Overall; OA: 0.84. - [AI-Generated Text Detection on Cross-genre (test)](): AI-generated text detection; Cross-genre (test); OA: 87.5. - [AIGT Detection on Cross-domain (Human -\> AI)](): AIGT detection; cross-domain AIGT detection Human -\> AI; Overall Accuracy: 78. - [AIGT Detection on SeqXGPT-Bench cross-domain](): AIGT detection; SeqXGPT-Bench cross-domain; Precision (AI): 84.19. ## Original sources and implementation [Original paper]() [PDF]()