Memory-Like Storage Means File Systems Must Change

The term #softwaredefinedstorage is in the new job title that Eric Barton has at #DataDirect Networks, and he is a bit amused by this. As one of the creators of early parallel file systems for supercomputers and one of the people who took the #Lustre file systems from a handful of #supercomputing centers to one of the two main data management platforms for high performance computing, to a certain way of looking at it, Barton has always been doing software-defined storage. The world has just caught up with the idea. Now Barton, who is leaving Intel in the wake of the chip giant ceasing its commercialized Lustre business, wants to help diversify and commercialize the file system that is at the heart of DDN’s Infinite Memory Engine ( #IME ) burst buffer, and so he has taken the job of chief technical officer for #SDN at the high performance storage company. Barton has spent a career that spans more than three decades on the bleeding edge of high-end storage, starting in 1985 with co-founding #MeikoScientific, a maker of parallel #supercomputer s based on transputers (remember those?) rather than ordinary vector or single-threaded processors as we know them. Lawrence Livermore National Lab stepped up and bought one of the Meiko Computing Surface clusters rather than the Connection Machine from Thinking Machines. The Computing Surface needed a file system, and Barton tells The Next Platform that Meiko was so short of staff that he had to write the parallel file system, obviously called PFS, himself. “I didn’t know quite what to do,” Barton recalls. “Solaris had this lovely virtual file system concept, and I thought I could just stripe a file system across other file systems and that will be it. I had one file that I used as the namespace and other file systems where the data was striped across.” This sounds simple enough, but getting a namespace to scale and therefore provide access to data chunks both large and small across the multiple nodes that comprise a parallel file system is a tricky business, indeed. That is why the Lustre file system was born, and that is why #IBM, #SGI, #SunMicrosystems and others also spent a fortune developing parallel file systems over the decades when supercomputer was new and cool.

https://www.nextplatform.com/2017/05/24/memory-like-storage-means-file-systems-must-change/