IBM Spectrum Scale (formerly known as General Parallel File System – GPFS) is a highly scalable, flexible, reliable, and high performance clustered file system.
While many clustered file systems like Lustre, HDFS (Hadoop Distributed File System) and Redhat GFS (Global File System) provide similar availability, reliability, and performance, the amazing flexibility offered by Spectrum Scale makes it stand out from the rest of the file systems. Spectrum Scale allows can be configured almost to simulate HDFS functionality using FPO (File Placement Optimizer) feature and Lustre file systems using data and metadata separation in majority of cases.
The main objective of this blog is to provide a quick understanding of IBM Spectrum Scale implementation options as a Software-Only Solution or as a Customized Pre-Built Appliance Solution.
Software Only Solution
Spectrum Scale can be purchased from IBM as a software only solution which provides maximum flexibility to design and implement Spectrum Scale Clusters to meet exact business requirements on any existing or new Server and Storage hardware.
Spectrum Scale runs on virtually any x86 or power hardware that supports AIX, Linux, and Windows operating systems. It is supported on many of the commercial storage sub-systems from IBM, EMC, Hitachi etc. It also supports many of the available virtualization technologies like Power-VM, VMWare, Power-KVM, and KVM with few limitations.
As a software-only-solution Spectrum Scale is available in Express, Standard, and Advanced editions from IBM. While the Express edition provides the basic features Spectrum Scale, the Standard edition includes in addition to the basic features, majority of the features like Information Life-cycle Management(ILM), Active File Monitoring (AFM), OpenStack Swift, etc., and Advanced edition includes all Standard Edition features plus Encryption, secure data storage, and deletion.
Spectrum Scale is so flexible that it can be configured on a single server with a single file system of a few gigabytes of storage initially and can be scaled to hundreds of nodes with a single or multiple file systems of petabytes of storage.
Since this is a software only solution, as with any large scale complex High Performance Computing (HPC) or File Storage clusters, a complete understanding of the business requirements, careful planning of Server and Storage Architecture is required for optimal performance the Spectrum Scale cluster to avoid potential performance bottlenecks as the cluster eventually expands into hundreds of nodes and petabytes of storage. Some of the parameters like GPFS Filesystem block size, Inode Size, Data and Metadata separation, AIX/Linux CPU/network/multipath device settings, RAID configuration, Read/Write cache settings on the backend storage can severely impact the cluster performance if not correctly tuned.
Customized Pre-Built Appliance Solutions
As configuring large scale HPC and File Storage clusters involves extensive panning, testing and needs expert knowledge of various Server, Storage, SAN, Network, and Application architectures, there are few custom Spectrum Scale and Hardware bundled solutions currently available which are optimized for certain HPC and File Storage clusters. These customized Appliances can be deployed very quickly without spending a lot of time and effort in testing and planning.
Scale Out Network Attached Storage (SONAS – IBM ), V7000-Unified, GPFS Storage Server (GSS – Lenovo ) and Elastic Storage Server (ESS – IBM ) are the Spectrum Scale Software and hardware bundled as an appliance that are available from IBM as file storage solutions. These appliances can scale from entry level systems storing few terabytes of data to enterprise level systems that can store up to petabytes of data.
Few other vendors like DDN and RAIDINC also offer custom-built GPFS solutions for optimized performance.
SONAS is pre-built file serving solution with IBM-3650 servers as dedicated Storage, Interface, and Management nodes that are inter-connected with Infiniband and Ethernet Switches. The SONAS gateway option allows to connect IBM XIV, V7000, or DCS 3700 Storage Systems for the Spectrum Scale file system storage. Hundreds or Thousands of Linux, Windows nodes can access the the Spectrum Scale File system data on SONAS using a variety of protocols like CIFS, NFS, HTTP, etc. via either a 10 Gbps and/or 1 Gbps Ethernet network. The backend infiniband interconnect provides high speed data transfers between the SONAS cluster nodes.
V7000-Unified is a pre-built File and Block serving solution with storage virtualization, real-time compression, easy tiring, replication, and many additional features. The entry level system can store up to few terabytes of data and can fully configured system can scale up to few petabytes of data. File data in the Spectrum Scale file system can be accessed using similar protocols like CIFS, NFS, HTTP, etc. like SONAS. External storage from multiple vendors like IBM, EMC, and Hitachi can be virtualized behind V7000-Unified and can serve as block level storage using fiber channel or iSCSI.
GPFS Storage Server is a pre-built Appliance from Lenovo with Spectrum Scale and x86 hardware bundle. It supports 10 Gbps/40 Gbps Ethernet and multiple JBOD drives with IBM de-clustered software RAID. The GSS is capable of serving as dedicated backend storage for large scale HPC clusters.
Elastic Storage Server (ESS) is similar to GPFS Storage Server, but uses Power-8 hardware instead of x86. ESS also uses IBM native de-clustered software RAID on JBOD disks. IBM ESS can provide up to 25 GBps sustained throughput. It uses flash devices for internal and external cache to mask the latency of SAS drives. The de-clustered RAID also helps to reduce the re-build times in case of a disk failure to minutes compared to hours on traditional RAID implementation.
