Hammerspace unveils the world’s fastest file system for enterprise AI models

In order to meet the demands of enterprise AI, machine learning, and deep learning (AI/ML/DL) initiatives, Hammerspace unveiled the high-performance NAS architecture.

Hyperscale NAS is a new class of storage architecture that offers the speed necessary to power GPU clusters of any size for GenAI, rendering, and enterprise high-performance computing. It is based on the principles needed for large language model (LLM) training.

For the purpose of feeding the GPU clusters that process the data, huge performance is needed for both building and training AI models. Large-file streaming, read-intensive apps, and sporadic read-write workloads for checkpointing and scratch space are among the many performance needs. Traditional scale-out NAS designs, including all-flash systems, are unable to meet the scalability and performance needs of these applications. It was previously only possible to get consistent performance at this scale with HPC parallel file systems, which are difficult to set up and maintain and don't fit enterprise needs.

For enterprise and web-scale AI training, the Hyperscale NAS architecture has now been shown to be the fastest file system globally. With systems based on about 1,000 storage nodes and up to 30,000 GPUs fed at an aggregate performance of 80 terabits per second across TCP/IP and conventional Ethernet, it is in production.

Hyperscale NAS is repurposing big tech tactics for commercial applications. Hyperscale NAS is developing in the same way that Amazon Web Services (AWS) developed S3 for large-scale, efficient storage, which went on to become a model for object storage in businesses. Large language models (LLMs) are trained using this approach, which is also being utilized in enterprises to train generative AI models and do GPU computing. This strategy is expanding, offering cutting-edge tech firm practices to businesses of all kinds.