# High Performance I/O For Large Scale Deep Learning

Source: [High Performance I/O For Large Scale Deep Learning](<https://www.sota2.com/research/paper/68fa66e21e209131cea12cbe>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:00:21.133Z

Authors: Alex Aizman, Gavin Maltby, Thomas Breuel

Published: 2020-01-07T02:35:42.000Z

## Abstract

Training deep learning (DL) models on petascale datasets is essential for achieving competitive and state-of-the-art performance in applications such as speech, video analytics, and object recognition. However, existing distributed filesystems were not developed for the access patterns and usability requirements of DL jobs. In this paper, we describe AIStore, a highly scalable, easy-to-deploy storage system, and WebDataset, a standards-based storage format and library that permits efficient access to very large datasets. We compare system performance experimentally using image classification workloads and storing training data on a variety of backends, including local SSDs, single-node NFS, and two identical bare-metal clusters: HDFS and AIStore.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2001.01858>)

[PDF](<https://arxiv.org/pdf/2001.01858.pdf>)