# Big Data at HPC Wales

Source: [Big Data at HPC Wales](<https://www.sota2.com/research/paper/68fa61ff1e209131ce9f708a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T20:10:28.856Z

Authors: Sidharth N. Kashyap, Ade J. Fewings, Jay Davies, Ian Morris, Andrew Thomas Thomas Green, Martyn F. Guest

Published: 2015-06-30T00:18:11.000Z

## Abstract

This paper describes an automated approach to handling Big Data workloads on HPC systems. We describe a solution that dynamically creates a unified cluster based on YARN in an HPC Environment, without the need to configure and allocate a dedicated Hadoop cluster. The end user can choose to write the solution in any combination of supported frameworks, a solution that scales seamlessly from a few cores to thousands of cores. This coupling of environments creates a platform for applications to utilize the native HPC solutions along with the Big Data Frameworks. The user will be provided with HPC Wales APIs in multiple languages that will let them integrate this flow into their environment, thereby ensuring that the traditional means of HPC access do not become a bottleneck. We describe the behavior of the cluster creation and performance results on Terasort.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1506.08907>)

[PDF](<https://arxiv.org/pdf/1506.08907.pdf>)