Hadoop and Spark Get RADICAL at SC15

The rapid maturation of the #Apache #Hadoop ecosystem has caught the eyes of HPC professionals who are eager to take advantage of emerging big data tools, such as #Spark. One #HPC group presenting on the topic at the SC15 show this week in Austin, Texas, is Rutgers University’s RADICAL team.

The Research in Advanced Distributed Cyberinfrastructure and Applications Laboratory ( #RADICAL ) duo of professors Shantenu Jha (Rutgers) and Andre Luckow (Clemson University) are giving a three-and-a-half hour tutorial Sunday morning demonstrating how the power of Hadoop-resident frameworks, such as MapReduce and Spark–as well as the group’s own RADICAL-Cybertools suite–can further the analytical goals of the HPC professional.

In his introduction to the course, Professor Luckow discusses about how the HPC world can learn and benefit from the tools and analytic approaches that have been championed in the Hadoop world. “High performance computing (HPC) environments have traditionally been designed to meet the compute demands of scientific applications; data has only been a second order concern,” Professor Luckow says in his tutorial introduction, which can be viewed on YouTube.

“However, with science moving toward data-driven discoveries relying on correlations and patterns in data to form scientific hypotheses,” Professor Luckow continues, “the limitations of HPC approaches become apparent. Low-level abstractions and architectural paradigms, such as the separation of storage and compute, are not optimal for data-intensive applications.”

While there are powerful kernels and libraries available for traditional HPC, the lack of “functional completeness” of analytical libraries is holding them back, the professor says. “In contrast, the Apache Hadoop ecosystem has grown to be rich with analytical libraries, e.g. Spark MLlib,” he says. “Bringing the richness of the Hadoop ecosystem to traditional HPC environments will help address some gaps.”

http://www.hpcwire.com/2015/11/13/hadoop-and-spark-get-radical-at-sc15/