Big Data Tech Hadoop and Spark Get Slow Start in Enterprise

There are plenty of success stories about #Hadoop in the enterprise, but those may be the exception and not the rule. @Gartner Research VP @MervAdrian provides a reality check on deployment rates, successes, and failures of big data technologies in the enterprise. You may have seen quite a few success stories about enterprises implementing Hadoop and related technologies as part of companies' big data analytics programs. You may wonder if you are behind the curve if you haven't implemented this open source big data technology yet. Here's some reassurance for you -- while there may be plenty of success stories to tell, they are the exception. Many projects don't even get started, and many of the projects that do get started will fail. Gartner is forecasting that through 2018, 70% of Hadoop deployments will fail to meet cost savings and revenue generation objectives due to skills and integration challenges. It's a prediction the company actually made back in 2016, and according to Gartner Research VP Merv Adrian, it's a prediction that's held up pretty well. Adrian provided an update for enterprise organizations -- Hadoop and Spark: Understanding Open Source Opportunities and Risks -- at the Gartner Data and Analytics Summit in Grapevine, Texas this month.

https://www.informationweek.com/big-data/software-platforms/big-data-tech-hadoop-and-spark-get-slow-start-in-enterprise/d/d-id/1331285?

Cloudera founder Mike Olson: ‘We’re moving from automating processes to automating decisions’

The growing momentum of big data in the cloud has been described as a threat to @Cloudera Inc., which is ironic given that the company’s original business plan was to sell big data as a cloud service. The market wasn’t ready in 2008, so Cloudera shifted to selling an integrated platform that combines various open-source projects with proprietary extensions. Of the three prominent startups to emerge from the @Hadoop ecosystem, the other two being @MapR Technologies Inc. and @Hortonworks Inc., Cloudera was the most prominent, in no small part because of the $740 million investment it received from @Intel Corp. in 2014. Its stock-market performance since going public last April has been underwhelming, but few would deny that it’s a market leader. Chief Strategy Officer Mike Olson, one of Cloudera’s four founders, joined SiliconANGLE recently for a telephone interview on where big data’s going next. Gartner has estimated that 85 percent of enterprise big data projects failed. Does that surprise you? I don’t understand who they’re talking to. We’re a high-growth company in the $300 million-plus forecast range and most of our business is still on-prem. People aren’t shutting down large football fields of Teradata [Corp.], but the opportunity was never to displace data warehouses. It was to capture more data than we could before and see what we could do if we had better tools to derive value. I think Gartner is comparing the wrong things. The growth rate of machine learning is zero if you compare it against traditional markets because there is no traditional market. But it has created value in huge new ways. It’s true that the biggest growth area has been in cloud databases like Cloudera Altus and Redshift on Amazon Web Services. We’ve believed since the early days that much of the action would be in cloud services, and that’s why we named the company Cloudera. We still believe that, but I think there’s enormous potential and success in on-prem deployments. Three years ago, Cloudera defined its mission around a unified platform built on Hadoop, the pioneering framework for managing big data. Today, you don’t even mention Hadoop in your description. What changed? We talked about Hadoop in the early days because people needed to know what the platform was and what we had. Today it’s a much richer environment. We’re seeing native machine learning using Spark integrated with AWS storage buckets. There’s no Hadoop in there. This platform is doing more than just what Hadoop did. Our original platform was Hadoop and MapReduce. Today, we ship 26 different open-source projects, 18 of which were created by Clouderans. Hadoop is always going to be part of the foundation of the company — and we’re proud that we spotted it so early — but it’s only a small part of what we do today.

https://siliconangle.com/blog/2018/03/13/cloudera-founder-mike-olson-moving-automating-processes-automating-decisions/

Hadoop and Spark: Synergy Is Possible

If somebody mentions #Hadoop and #Spark together, they usually contrast these two popular big data frameworks. According to Ahrefs, 1,200 #Google visitors are searching for Spark vs. Hadoop each month, while only 90 are inquiring about Spark and Hadoop. It looks like the frameworks have gradually gained a reputation of being mutually exclusive. But this is not always the case. There are multiple ways for businesses to benefit from their synergy. Let’s take a closer look at Hadoop and Spark and discover scenarios where they can work together. #ApacheHadoop Defined Apache Hadoop is an open-source framework for data storage and parallel processing. Initially released in 2011, Hadoop triggered #bigdata evolvement. Distributed #datastorage allowed companies to cope with big data volumes. They didn’t need to buy extremely expensive custom hardware anymore. Instead, they could use multiple affordable computers to store data. Besides, this approach enabled the much-needed scalability of the solution. When the amount of data to be stored and processed increased, companies could solve that challenge by adding extra computers. With such an approach to data storage, parallel data processing was required. This became another distinctive feature of Hadoop. Apache Spark Defined Apache Spark is an open-source framework for parallel processing. Released in 2014, Spark was designed to cope with the shortcomings of Hadoop MapReduce, which was mainly the speed of processing. Unlike Hadoop MapReduce, which has to write interim analysis results back to the disk and then read the data again and again, Spark processes data in-memory. As a result, it is up to 100 times faster than Hadoop MapReduce. By the way, if we talk about real alternatives, there are Apache Spark and Hadoop MapReduce (not entire Hadoop). And this is evident already from Spark’s definition. Using Hadoop and Spark Together One important remark: the Hadoop ecosystem consists of several components, among which is Hadoop Distributed File System (or HDFS for short), Apache Hive (a query engine), Hadoop MapReduce (a framework for the parallel processing of distributed large datasets), and more. With this information in mind, let’s take a look at possible synergy scenarios. HDFS + Apache Spark We have already clarified that Apache Spark’s intended purpose is data processing. But to process data, the engine needs to take it from some storage first. HDFS is not the only option available, but it's a quite frequent one. The reason is simple: both belonging to Apache Foundation family, HDFS and Spark are highly compatible. An illustrative example of such a synergy is a word count (you can find the code example here). The sequence of operations is as follows: Apache Spark takes a text file from HDFS, divides each line into separate words, sets the value 1 for each word, calculates the sum of values for each word, and records the result to HDFS. Apache Hive + Apache Spark The combination of Apache Spark and Apache Hive (that is based on HDFS) allows solving many business tasks, for example, conducting customer behavior analytics. Imagine a company that cumulates data from multiple sources: clickstream data, comments, and posts on social media, data from customer mobile apps, etc. Let’s say that the company has chosen HDFS to store their data and Apache Hive to act as an intermediary between HDFS and Spark. Apache Hive makes it possible to query the data using a SQL-like language. As a result, Spark that has special support for Hive could easily access the data and process it. In the end, the company can understand the preferences and behavior patterns of each customer. Real-Life Examples of Spark and Hadoop Duets Real-life examples of using Hadoop and Spark together are not rare in big data consulting practices. The list of companies that adopt such an approach includes many well-known names. Undoubtedly, their solutions are of different complexity. And this is understandable, as these companies strive to solve different business tasks. Still, there is one thing that unites them: their big data technology stack includes both Hadoop and Spark. Let’s look at the following two examples. TripAdvisor uses Hadoop and Spark together to deliver a seamless customer experience. They introduced auto-tagging, which is based on the analysis of visitors’ reviews and tags. This feature allows TripAdvisor to predict whether a visitor’s impression of a particular location will be the same as that of the other visitors. Another interesting feature is improved photo selection. Now, a website visitor can get a more precise picture of any location thanks to a better choice of visuals. For instance, if a hotel has a pool, machine learning algorithms will pick the photo of the pool and show it to the visitor. Uber is doing a great job of managing their big data to improve their service. They know the typical behavior of each customer (starting and destination points, usual day and time of their journeys, etc.). The company also uses real-time traffic situations to adjust the number of drivers needed at a particular time and in a particular location. To make this possible, Uber uses HDFS for loading raw data onto Hive for SQL-powered analysis and Spark for processing of millions of events. Conclusion Now, you can see that Hadoop and Spark can smoothly work together. We have supplied the article with real-life examples so that you can see that the synergy of these big data frameworks is possible not only in theory but also in practice. When making a choice, just remember one of the maxims of big data: your big data technology stack should suit your business goals.

https://dzone.com/articles/hadoop-and-spark-synergy-is-real

Microsoft's cloud Big Data service cuts prices up to 52 percent

@Microsoft has decided to get down to business with @HDInsight (HDI), its #Azure cloud-hosted #BigData offering, based on @Apache #Hadoop, @HBase, @Spark, @Storm, @Kafka, @Hive LLAP, and #MicrosoftRServer. Ostensibly, Microsoft previously considered its competition to be on-premises Hadoop implementations. But it now offers pricing that is far more competitive with @Amazon Web Services' (AWS') @Elastic @MapReduce (EMR), while still offering a 3-nines service level agreement (SLA) as a differentiator. Details, details The pricing changes, highlighted in a blog post by Microsoft's Rimma Nehme and detailed on a separate page, offer varying price cuts depending on the virtual machine type used for the head and worker nodes in the HDInsight cluster. Price cuts are up to 52 percent, Microsoft says, while the service itself remains largely the same. In addition, for those customers wishing to run data science workloads with code written in R, the surcharge for running R Server in a distributed fashion on an HDI cluster has been cut by 80 percent, down to just $0.016 (i.e. 1.6 US cents) per CPU core, per hour. Microsoft points out that because of Azure's numerous global data centers (regions), HDI is available at more points of presence than any other cloud Hadoop service. In addition to Azure's mainstream cloud, the service is also available on its US government cloud and on so-called sovereign clouds, including those in Germany and China. Per various regulatory requirements, the sovereign clouds run in facilities operated by local partners, rather than Microsoft itself. In other news Microsoft has a few other announcements to go with the price change: The introduction, available in preview, of HDInsight Enterprise Security Package, which integrates Microsoft Active Directory with Apache Ranger. This is essentially a re-branding of HDInsight's Premium cluster tier General availability of its Apache Kafka cluster type, which had been in preview until recently General availability of HDInsight's integration with Azure Log Analytics Public preview of support for Power BI DirectQuery, specifically against Hive LLAP, used in HDI "Interactive Query" cluster types New HDInsight add-in developer tools for IntelliJ, Eclipse and Visual Studio Code (Microsoft's cross-platform code editor for MacOS, Linux and Windows). The IntelliJ and Eclipse tooling include the ability to submit and debug distributed Spark code right from those development environments. The VS Code tooling allows for interactive execution of PySpark (a Python library for Apache Spark) code.

http://www.zdnet.com/article/microsofts-cloud-big-data-service-cuts-prices-up-to-52/

Syncsort’s Fourth Annual Big Data Survey Uncovers Five Key Data Lake Trends for 2018

PEARL RIVER, N.Y.--(BUSINESS WIRE)--@Syncsort, a global leader in #Big ron to #BigData solutions, today announced the results from its fourth annual Big Data survey. Results revealed the top use cases and challenges faced by organizations as they progress their modern data architecture and data lake initiatives, and see significant benefits from #Hadoop/@Spark, with nearly 60% citing both increased productivity across the organization and improved efficiency to reduce costs as their biggest gains. “But those benefits can only be unlocked if organizations have access to enterprise data, can create trusted data sets and establish effective data governance practices. This propels them to a place where they can not only adapt to digital disruption, but take advantage of it so their businesses thrive.” Tweet this Compared with last year’s survey, the most dramatic increase in reported benefits was higher revenue and accelerated growth, which 55% named as a benefit this year compared with 37% last year. And though organizations are using Big Data insights in more sophisticated ways to improve revenue and customer service, they continue to face some of the same challenges reported in past years, including keeping up with rapidly changing technologies and tools. The survey also found that Hadoop and Spark, which had high interest but low adoption at the time Syncsort launched the survey in 2014, are now in production or test at 70% of responding organizations. Specifically, this year, more than 40% of respondents say they are in production with either Hadoop or Spark, and 30% say they are engaged in a proof of concept or pilot program. Based on the survey results, the five key trends in Data Lake initiatives that organizations need to monitor in 2018 include: 1. The composition of the data lake shifts. Traditional sources remain most popular for filling the data lake. Relational Database Management Systems (RDBMS) were chosen as the top source at 69%, up from 61% last year, surpassing enterprise data warehouses (EDW) at 63%. But newer sources grew, with NoSQL databases identified by 46% of respondents compared to 35% last year. Cloud repositories are also gaining strength as a data source (accounting for 40%) as more organizations leverage the cloud as a deployment platform. 2. Legacy platforms continue to play a significant role. Data from legacy platforms (such as the mainframe and IBM i) also make major contributions to the data lake: over 97% of respondents with mainframe believe it’s valuable to access and integrate mainframe data into the data lake for real-time analytics – a 27% increase over last year. Over 90% of those with IBM i believe it’s valuable to access and integrate that data into the data lake, not surprising as leaving behind decades of valuable data stored on these systems would seriously hamper their companies’ analytics. 3. Data quality and regulatory compliance challenges are top of mind. While the skills shortage had been ranked the top challenge for three consecutive years, this year it fell to number two, replaced by mounting concerns over improving the quality of data in the data lake. Indeed, 40% said data quality was a significant struggle for their organization, likely a result of expanded use of data lakes driving an emphasis on improving data quality. But the survey also showed not everyone is making the connection between data quality and ROI. 60% of Financial Services and Insurance professionals said ensuring data quality is a top priority, compared with just 40% of respondents from other industries. Nearly 40% of respondents also cited meeting regulatory compliance mandates as a significant challenge – one that will only increase as GDPR and other compliance mandates become a pressing reality for executives. Therefore, the scope of data governance is projected to expand as organizations will place a higher priority on putting processes in place that allow them to understand what their data is and where it has been. 4. Data lakes will be kept fresher to support data use. This year, 71% of respondents named ETL as their most compelling data lake use case – a notable increase from last year, when it came in third at 53%. The second and third-highest use case involved analytics, with advanced/predictive analytics at 64% and real-time analytics at 60%. All of these use cases require fresh, up-to-date data; however, more than 75% of respondents report having difficulty keeping their data lake in sync with changing data sources, particularly when the source is hard to access, like mainframe. 5. Organizations will continue to invest in Big Data. Around 90% of organizations found that leveraging Hadoop and Spark and shifting away from legacy systems proved valuable not only in driving data insights, but also in saving money (21% reported freeing up mainframe resources reduced costs). As more companies discover the cost savings resulting from optimizing traditional platforms, more money will in turn be funneled into funding more Big Data projects. “We are seeing increased adoption of data lake initiatives where organizations are very focused on governance of the data in the data lakes, increasing benefits through advanced analytics and machine learning and deployment of hybrid environments including cloud,” said Tendü Yoğurtçu, CTO, Syncsort. “But those benefits can only be unlocked if organizations have access to enterprise data, can create trusted data sets and establish effective data governance practices. This propels them to a place where they can not only adapt to digital disruption, but take advantage of it so their businesses thrive.”

https://www.businesswire.com/news/home/20171218005197/en/Syncsort%E2%80%99s-Fourth-Annual-Big-Data-Survey-Uncovers

Apache Bigtop Adds OpenJDK 8 Support

@Apache has released #Bigtop 1.2.1 with support for #OpenJDK 8, and a new sandbox feature that lets you run big data pseudo clusters on @Docker. Bigtop is an Apache Foundation project that you can use for packaging, testing, and configuration of the big name open source big data components that make up the #Hadoop infrastructure. Bigtop supports a wide range of components and projects, including Hadoop, @HBase and @Spark. The primary goal of Bigtop is to build a community around the packaging, deployment and interoperability testing of Hadoop-related projects. This includes testing at various levels, including packaging, platform, runtime, and upgrade, and focussing on the system as a whole, rather than individual projects. While Hadoop is generally used to refer to the central collection of tools, Bigtop looks at the wider selection that makes up the Hadoop-related projects, including Hbase, Pig, Hive MapReduce, Zookeeper and Avro among others.  Bigtop packages Hadoop RPMs and DEBs, so that you can manage and maintain your Hadoop cluster, and it provides an integrated smoke testing framework, alongside a suite of 50 test files. It also helps with virtualization testing, with vagrant recipes, raw images, and (work-in-progress) docker recipes for deploying Hadoop from zero, and you can use Bigtop Provisioner to spin-up a virtual cluster with a single command. The new release of Bigtop adds a Sandbox feature that lets you use Docker to run pseudo clusters. Creating it is a single line command, and for HDFS the process takes around 30 seconds. You then get a local WebUI to play around it. You can run HDFS and Spark standalong, or HDFS, Yarn, Hive and Pig - there are simple instructions on the Bigtop site. This presentation from DataWorks Summit 2017 on using the Sandbox comes from Apache Bigtop Project Committer and PMC member, Evans Ye:   The new release also includes a faster Docker Provisioner which has been rewritten to fully embrace the Docker ecosystem. The OpenJDK support in the new release means that all the components are now built on JDK8. The major components have been updated to recent versions, including Hadoop 2.7.3, Spark 2.1.1, HBase 1.1.9, and Zeppelin 0.72, and most of the ecosystem projects have also been updated, including Apex, Crunch, Flume, Ignite, Mahout, Oozie, and Phoenix, among others.

http://www.i-programmer.info/news/197-data-mining/11374-apache-bigtop-adds-openjdk-8-support.html

Cloudera Bringing Impala to AWS Cloud

#Apache #Impala, the #SQL-based #analytical #database that originated at #Cloudera, will soon be available as a managed service on the @Amazon Web Services cloud, the #Hadoop software distributor announced today at #AWS Re:Invent. Cloudera Altus Analytic DB, as the hosted cloud version of Impala will be known, will be available as a beta service on AWS by the end of the year, with general availability expected in 2018. The company says support for the #Microsoft #Azure cloud will follow, but no timeline was given. Impala is one of the most popular engines in Cloudera’s Distribution of Hadoop (CDH), and the open source software is also offered in other Hadoop distributions. The software essentially allows customers to run a host of standard SQL queries against massive stores of relational data stored in Parquet, an optimized Hadoop file format. It’s not the only parallel SQL data warehouse designed to run atop Hadoop, but it is one of the most mature. Getting customers the capability to run SQL workloads against data hosted in cloud-based object storage repositories was a big priority for Cloudera, says Alex Gutow, a product manager with the Palo Alto, California-based company. “When we look around at our customers and the types of workloads that are really best suited to take advantage of the agility and cost efficiencies of the cloud, BI and analytics is one of the key workloads,” Gutow tells Datanami. Specifically, BI workloads running on Impala will benefit tremendously from features customers can find on AWS cloud, such as multi-tenant isolation and workload elasticity, Gutow says. Instead of leaning on in-house administrators to procure computational resources and then manage it for the user on an on-going basis, the Altus Analytic DB service allows customers to lean on Amazon and Cloudera to do that heavy lifting for them, she says. “So you can have a very specific cluster to run reporting workloads, and you can have another to run ad hoc queries or self-service BI,” Gutow continues. “It allows for much more of that agility, giving all different types of analysts access to shared data very quickly, giving them much more flexibility, and being able to elastically scale up those resources as you need to meet different performance requirements, or to make sure there’s predicable performance for those workloads.” Altus Analytic DB will access data stored in customers’ Simple Storage Service (S3) accounts. Impala has been able to access data stored in the S3 object store via an HDFS API for about a year, says Greg Rahn, a Cloudera product manager for Impala and Altus. “HDFS provides an API to S3 known as an S3A connector. Impala uses this,” Rahn says. “It looks to Impala as if it’s kind of the HDFS file system, or the abstraction thereof.” There’s a similar API that exposes data stored in Microsoft’s cloud object store, ALDS, through HDFS, and the company will use that connector when Altus Analytics DB is supported on the Azure cloud in the future. “So at the end of the day, whether the data is in S3 or ALDS or HDFS itself, it all kind of looks the same in terms of the visibly it to impala,” Rahn says. Cloudera has pre-selected certain Elastic Cloud Compute (EC2) instances that Altus Analytic DB will be allowed to run upon. Customers will be able to spin up Altus Analytic DB clusters with just a few clicks of the mouse, Rahn says.  Cloudera is positioning its SDX as a key product uniting customers’ cloud and on-premise Hadoop deployments “The Altus deployment makes it quite trivial to start up these things, probably on the order of three to four clicks to provision a cluster,” he says. “You log in, name the cluster, pick the size of the instance, the number you want, then you hit ‘create cluster.’ So it’s very simple.” Customers will be able to quickly spin up analytic clusters on AWS, run a workload, and then quickly dispose of it. None of the data, metadata, or state information for these jobs will be lost when the cloud cluster is deleted because it’s all managed centrally under Cloudera’s Shared Data Experience (SDX), which the company announced at the Strata Data Conference in September. The SDX provides a way to manage data access and permissions for on-premise and cloud environments from a central console. The software sports hooks into core management tools, including Cloudera Navigator, Cloudera Manager, and Sentry for on-premise implementations and Altus controls for cloud-based environments. “Not only can you provide these different isolated resources for each of the different workloads,” Gutow explains, “but from the management side of things, these all benefit from having shared security, shared governance, and shared metadata as running actors shared data layer in the cloud, the shared object storage. So each time any of these different workloads are provisioned or run for different self-service workloads, you don’t have to go and redefine the different security policies. You can easily manage them from an enterprise standpoint.” Altus Analytic DB will be the second hosted offering under the Altus banner since Cloudera announced its new platform as a service (PaaS) in May. The first offering, Altus Data Engineering, was focused on data ingest and transformation tasks, and includes Spark, Hive, Hive on Spark, and MapReduce2 engines. Cloudera was mum on what engines will come next for Altus. Kudu, its fast-data layer, is one obvious candidate. Cloudera is currently in a quiet period before it announces financial results on December 7.

https://www.datanami.com/2017/11/28/cloudera-bringing-impala-aws-cloud/

BlueData and Dell EMC Expand Relationship to Deliver Big-Data-as-a-Service in the Enterprise

Dell EMC to Resell BlueData EPIC Software for Containerized Big Data Workloads SANTA CLARA, CA--(Marketwired - Nov 14, 2017) - @BlueData ®, provider of a leading #BigData -as-a-Service ( #BDaaS ) software platform, today announced that it has signed a reseller agreement with Dell EMC, becoming a Select partner in Dell EMC's #TechnologyConnect #PartnerProgram. The two companies will deepen their go-to market relationship to deliver the BlueData EPIC software platform to Dell EMC's enterprise customers across the globe. BlueData EPIC software uses @Docker #container technology to help simplify and accelerate the deployment of #BigData #analytics and #datascience environments in the enterprise. Together, Dell EMC and BlueData provide a comprehensive solution to accelerate time-to-value and reduce TCO for Big Data infrastructure. With Dell EMC and BlueData, enterprises can deliver exceptional simplicity, speed, efficiency, and performance for their Big Data implementations. Dell Technologies Capital is an early investor in BlueData, and an active member of their Board of Directors, providing valuable business guidance and access to the broad Dell Technologies ecosystem. This latest agreement deepens the go-to-market relationship between BlueData and Dell Technologies and is an important step in helping the company to continue to scale its business. BlueData's EPIC software platform achieved certification for Dell EMC PowerEdge Servers in the Dell Technology Partner Program last year - and earlier this fall, BlueData was announced as a key component of the new Dell EMC Elastic Data Platform, along with Dell EMC Isilon and Dell EMC services. As part of this new reseller agreement, BlueData will be included in the Dell EMC price list and configurator for its global sales teams and channel partners. "This is a critical milestone for BlueData," said Kumar Sreekanti, CEO of BlueData. "This new reseller relationship allows BlueData to expand our reach into the global Dell EMC customer base, helping both new and existing customers to scale their Big Data deployments. We are proud to work with world-class technology leaders like Dell EMC and we look forward to this expanded partnership." BlueData enables enterprises to implement Big-Data-as-a-Service running on their own data center infrastructure or in a hybrid architecture. Using BlueData EPIC software, these organizations can provide on-demand agility and elasticity for their analytics and data science teams -- with the ability to spin up instant clusters for Hadoop, Spark, and other Big Data tools in a secure multi-tenant environment. And with BlueData, they can save up to 75 percent on the infrastructure and operational costs for their Big Data deployments. "BlueData is disrupting the traditional deployment model for Big Data analytics, with a fundamentally new and more cost-effective approach," said Keith Manthey, CTO of Analytics at Dell EMC. "Now enterprises can get cloud-like agility and flexibility with containerized Big Data applications, while ensuring comparable performance and security to that of a bare-metal environment. We look forward to working with BlueData to bring these benefits to our joint customers." The expanded relationship with @Dell EMC is another factor in the continued momentum for @BlueData, which achieved a 426% yearly increase in sales in 2016. #BlueData attributes its success to the increased adoption of #Hadoop, @Spark, and other #BigData technologies; growing interest in @Docker containers, cloud computing, and #BigDataAsAService among enterprise customers; the strength of its technology partner ecosystem; as well as the ground-breaking innovation of BlueData's #EPIC software.

http://m.marketwired.com/press-release/bluedata-dell-emc-expand-relationship-deliver-big-data-as-service-enterprise-2240634.htm

Should Spark In-Memory Run Natively On IBM i?

There’s a revolution happening in the field of data analytics, and an #opensource computing framework called Apache Spark is right smack in the middle of it. Spark is such a powerful tool that IBM elected to create a distribution of it that runs natively on its System z mainframe. Will it do the same for its baby mainframe, the IBM i? So, what is Apache Spark, and why should you care? Great questions! Let’s introduce you to Spark. Spark came out of UC Berkeley’s AMPLab about five years ago to provide a faster and easier-to-use alternative to MapReduce, which at that point was the primary computational engine for running big data processing jobs on Apache Hadoop. While Spark has a learning curve of its own, the Scala-based framework has not only replaced Java-based MapReduce, but also eclipsed Hadoop in importance in the emerging big data ecosystem. Spark is useful for developing and running all sorts of data-intensive applications, including familiar programs like ETL jobs and SQL analytics, as well as more advanced approaches like real-time stream processing, machine learning, and graph analytics. This versatility, as well as well-documented APIs for developers working in Java, Scala, Python, and R languages and its familiar DataFrame construct, have fueled Spark’s meteoritic rise in the emerging field of big data analytics. IBM took notice of Spark several years ago, and has since worked on several fronts to help accelerate the maturation of Spark on the one hand, and to embed Spark within its various products on the other, including: ML for z/OS, which executes Watson machine learning functions in a Spark runtime in the mainframe’s Linux-based System z Integrated Information Processor (zIIP). Integrated Analytics System, which combines Spark, Db2 Warehouse, and its Data Science Experience, a Jupyter-based data science “notebook” for data scientists to quickly iterate with Spark scripts. Project DataWorks, which brings Spark and Watson analytics together on the Bluemix cloud. Open Data Analytics for z/OS, a runtime that combines Spark, Python, and the Anaconda package of (mostly) Python-based data science libraries from Anaconda. And Spark running directly on its Bluemix cloud. And considering that IBM opened a Spark Technology Center in 2015, it’s safe to say that IBM is quite bullish on Spark. (That’s a major understatement, actually.) But perhaps the most interesting data point for this discussion came in 2016, when Big Blue launched its z/OS Platform for Apache Spark, which is a native distribution of Spark for the System z mainframe. Native Spark On The Mainframe IBM received kudos for the work from various industry insiders who participated in this video on the z/OS Platform for Apache Spark webpage. Among those singing IBM’s praise was Bryan Smith, the former CTO and VP of R&D at Rocket Software. “IBM did a really good job in porting Apache Spark to z/OS,” Smith says. “They could have just done a very simple port. But they didn’t. They didn’t cut any corners. They really exploited the underlying hardware architecture. They’re using specialty engines. They’re using the hardware compression facilities. They’re able to leverage the 10 TB of memory that you have on a z13 machine and the . . . processors, so you can actually run those Apache Spark clusters on z/OS.” Another software vendor that appreciates having Spark running natively on z/OS is Jack Henry & Associates, the Missouri banking software developer that also has a fairly big IBM i business. “The pain point to us is getting the data out to our customers,” Todd Hill, Jack Henry’s direct of card processing, says in the video. “Currently we have data on the mainframe. We have a distributed stack for across many types of applications. What Apache Spark does for us is to keep your data centralized in the one location. So instead of moving all that data off from multiple platforms into other applications, I can run Apache Spark directly on the mainframe, at low cost, and get it built out, and get the data to the people that need it.” Mike Rohrbaugh, zSystem lead for Accenture, says having Spark on the mainframe helps by automating the generation of intelligence and reducing the complexity. “It’s just so simple to bring the analytics engine back to the data to do intelligent automation,” he says in the video. IBM i Versus The Mainframe So how does this relate to the question in the headline of this story? For starters, let’s compare the similarities and differences between the IBM i and the z/OS mainframe platforms. First, the similarities. Both the IBM i server and the z/OS mainframe are relied upon to run transactional applications that are core to the businesses that use them. Both of them are used to store structured data that’s arguably the most critical data for the businesses that use them. They both store data in the EBCDIC format, and are heralded for best-in-class reliability and security. The also both run proprietary operating systems as well as open OSes like Linux, mostly utilize older languages (RPG and Cobol, respectively), and sport text-based interfaces that use the 5250 and 3270 datastreams, respectively. Now, the differences. Mainframes have their own processor type, while IBM i runs on the more popular Power processor. The mainframe stores data many different data stores (Db2 for z, copy books, etc.), while most IBM i data is stored in Db2 or IFS. Demographically, mainframe customers tend to be the largest companies in the world, whereas IBM i has a bigger installed base among small and midsized business. There’s also a large concentration of mainframes in banking, insurance, and healthcare, whereas IBM i has a stronger foothold in manufacturing, distribution, and retail. IBM i and mainframes are strong transactional systems, and are less known for their analytical prowess. However, data analytics are becoming increasingly important in this day and age, especially as part of a company’s digital transformation strategy. The pundits often say that all companies will need data analytics strategies to effectively compete in the coming decades. That’s probably a bit of an exaggeration, but only for the timing. The question, then, becomes the places where this analytical processing is going to take happen. Today, most mainframe and IBM i shops offload it to another system. It’s fairly common for users of both mainframes and IBM i servers to set up elaborate workflows to move data from the “big iron” transactional systems to dedicated analytical systems, including massively parallel processing (MPP) column-oriented systems like Teradata, Netezza, or Vertica. With the advent of Apache Hadoop clusters running on commodity X86 processors, many companies started experimenting with Hadoop computing, which invariably introduced them to the in-memory Spark framework. IBM wants to keep those analytic workloads on the mainframe if at all possible, which is why it made Spark run natively. This not only keeps costs down for its customers, but it also make the mainframe more “sticky” and lessens the urgency to migrate data and workloads off its biggest cash cow. The question, then, is whether IBM sees similar dynamics at play for the average IBM i user. Mainframe customers, owing to their size and tendency to be in financial services, are early adopters of new technologies, like Spark. They’re arguably closer to the cutting edge than the average IBM i shop, and the dollars at stake for each mainframe client are much larger. It’s safe to say that IBM i members of the Large User Group (LUG) probably are more closely resemble their mainframe brethren, and could benefit from having a powerful, cutting-edge tool like Spark running natively on the IBM i. They’re more apt to have a bigger investment in separate analytical environments, be it a Teradata machine or a Hadoop cluster. They’re also more likely to have some data science Skunk Works project running somewhere in their shop, and are more likely to already be running Spark in Linux, which is where it was originally developed to run. Spark On IBM i While Spark may not be on the radar of the average IBM i shop yet, folks within IBM are starting to ask questions about whether Spark will impact the IBM i installed base, and if it’s going to be important to them, how it ought to be introduced. If the company is planning to support Spark natively on IBM i, the company isn’t saying publically, which is not surprising. What we do know, however, is that IBM executives are at least talking about the prospect of bringing Spark to IBM i in some way, shape, or form. “It’s part of some discussions,” IBM’s product development manager for Db2 Web Query Robert Bestgen recently told IT Jungle. There are two general options for bringing Spark to the platform: porting Spark to run natively on IBM i or running in a Linux partition running on Power Systems. Spark was written in Scala, and therefore can run within a Java virtual machine (JVM), which the IBM i platform obviously runs. It may not be a stretch to get it running there, but there could be other factors that come into play, such as IBM i’s single level storage architecture, and how that maps to how Spark tries to keep everything in RAM (but will spill out to disk if needed). Should the Spark port be native? “Depends on who you talk to,” Bestgen said. The widely held thinking within IBM is that the Linux route makes more practical sense – if Spark is to come to IBM i at all (which, as far as we know, hasn’t been decided). “If you back up [and look at it] from an IBM i perspective, IBM would say that IBM i is part of the Power Systems portfolio, or what we call Cognitive Systems now,” Bestgen says. “For Power Systems, those platforms [like Hadoop and Spark] tend to run best on a… Linux kind of environment. That’s what folks think about it.” Few IBM i shops today are even running Linux partitions. According to HelpSystems‘ 2017 IBM i Marketplace study, fewer than 8 percent of organizations are running Linux next to IBM i on a Power Systems box, while about 9 percent are running Linux on other Power boxes. AIX’s penetration is about 50 percent higher, for what it’s worth. There’s a case to be made that IBM i shops are lousy at figuring out how to leverage the wealth of available tools for Linux, even after IBM went through the trouble of supporting little endian, X86-style Linux to go along with its existing support for big endian Linux within Power. “One of the areas that IBM could do a better job selling is saying, you seem to be willing to run Linux on a different platform. Why not run it on the platform that you have in your system now?” Bestgen says. At the end of the day, there are a lot of unanswered questions, including whether the IBM i installed base needs or wants such a powerful tool as Spark, let alone how it should run. So the question to the answer in the headline is no. “I don’t think we’re there yet in terms of running those things natively on i,” Bestgen says.

https://www.itjungle.com/2017/11/06/spark-memory-run-natively-ibm/

IBM Unveils a New High-Powered Analytics System for Fast Access to Data Science

ARMONK, N.Y., Sept. 26, 2017 /PRNewswire/ -- IBM (NYSE: #IBM ) today announced the Integrated Analytics System, a new unified data system designed to give users fast, easy access to advanced data science capabilities and the ability to work with their data across private, public or hybrid cloud environments.  The system, which comes with a variety of data science tools built-in, allows data scientists to get up and running quickly to develop and deploy their advanced analytics models in-place, directly where the data resides for greater performance. And because it is based on the IBM common SQL engine, clients can use the system to easily move workloads to the public cloud to begin automating their businesses with machine learning. In fact, because the popular database engine is used across both hosted and cloud-based databases, users can move and query data across multiple data stores, such as the Db2 Warehouse on Cloud, or #Hortonworks Data Platform. At the heart of the Integrated Analytics System are the #IBM Data Science Experience, #Apache #Spark and the #Db2 Warehouse, all of which have been optimized to work together with straight forward management. The Data Science Experience provides a set of critical data science tools and a collaborative work space through which data scientists can create new analytic models that developers can use to build intelligent applications quickly and easily. The inclusion of Apache Spark, the popular open source framework, enables in-memory data processing, which speeds analytic applications by allowing analytics to be processed directly where the data resides. New to this class of offering are the machine learning capabilities that come with both the Data Science Experience and Spark embedded on the system. Having machine learning processing embedded means that data does not need to be moved to the analytics processing, reducing the associated processes and wait times for analytics to run and respond. This simplifies the process of training and evaluating predictive models, as well as the testing, deployment and training as it is all done in-place. "The combination of high performance and advanced analytics – from the Data Science Experience to the open Spark platform – gives our business analysts the ability to conduct intense data investigations with ease and speed," said Vitaly Tsivin, Executive Vice President, at AMC Networks, who has been testing the system for several months. "The Integrated Analytics System is positioned as an integral component of an enterprise data architecture solution, connecting IBM Netezza Data Warehouse and IBM PureData System for Analytics, cloud-based Db2 Warehouse on Cloud clusters, and other data sources." "Today's announcement is a continuation of our aggressive strategy to make data science and machine learning more accessible than ever before and to help organizations like AMC, begin harvesting their massive data volumes – across infrastructures – for insight and intelligence," said Rob Thomas, General Manager, IBM Analytics. Seamless Expansion to the Cloud The integrated architecture of the new system combines software enhancements such as asymmetric massively parallel processing (AMPP) with IBM Power® technology and flash memory storage hardware and builds on the IBM PureData System for Analytics, and the previous IBM Netezza data warehouse offerings. It also supports a wide range of data types and data services, including everything from the Watson Data Platform and IBM Db2 Warehouse On Cloud, to Hadoop and IBM BigSQL. Like these solutions, the Integrated Analytics System is built with the IBM common SQL engine, enabling users to seamlessly integrate the unit with cloud-based warehouse solutions. In addition, industry standard tools and the common SQL engine provide users with an option to also move these workloads seamlessly to public or private cloud environments with Spark clusters, based on the user's requirements. Like IBM's existing data warehouse products, the Integrated Analytics System is designed to provide built-in data virtualization and compatibility with Netezza®, Db2®, and IBM PureData System for Analytics. Among these capabilities, the new system also incorporates hybrid transactional analytical processing (HTAP). In contrast to typical business environments where transaction processing and analytics are run on distinct architectures, HTAP runs predictive analytics, transactional and historical data on the same database at accelerated response times. Later this year, the company plans to add support for HTAP with IBM Db2 Analytics Accelerator for z/OS, which will enable the system to transparently integrate with IBM z Systems infrastructures.

http://markets.businessinsider.com/news/stocks/IBM-Unveils-a-New-High-Powered-Analytics-System-for-Fast-Access-to-Data-Science-1002571958

BlueData Provides Ability to Deploy Containerized Hadoop and Spark for Dell EMC Elastic Data Platform

SANTA CLARA, CA--(Marketwired - Sep 26, 2017) - #BlueData®, a leading #BigDataasAService company, announced that #DellEMC is using BlueData EPIC software to help power its new Elastic Data Platform. Using BlueData EPIC and the #DellEMCElasticDataPlatform, customers can spin up instant clusters for #Hadoop, #Spark, and other Big Data tools running on #Docker #containers. "We've been working together with Dell EMC across our product, services, and sales teams to help our joint customers get the most out of their Big Data investments," said Kumar Sreekanti, CEO of BlueData. "This latest collaboration is further evidence that our software has become the solution of choice for containerized Big Data deployments in the enterprise." The Dell EMC Elastic Data Platform provides on-demand access to Big Data analytics and data science workloads (e.g. Hadoop, Spark, machine learning, and other use cases) in a highly scalable, flexible, and secure multi-tenant architecture. The platform delivers fast and easy provisioning, simplified deployments, cost-efficiency, and assurance that governance and security requirements are being met for Big Data. The Dell EMC Elastic Data Platform includes Dell EMC infrastructure, software from BlueData as well as BlueTalon for data-centric security, and Dell EMC Professional Services: Deploying Big Data Environments: BlueData provides the ability to quickly create elastic, multi-tenant Big Data environments for data science and analytics using Docker containers -- for Big-Data-as-a-Service whether on-premises, in the cloud, or in a hybrid architecture. Separating Compute and Storage: When aggregate datasets grow larger than a few hundred terabytes, it makes sense to separate compute from storage to allow both to scale independently. For enterprises needing scale out storage, Dell EMC Isilon offers a compelling ROI and ease of use with scalability. Enforcing Centralized Policy: BlueTalon provides the consistent creation and enforcement of data access policies across environments supporting a diverse set of users, tools and data systems. Automating and Integrating: Dell EMC Professional Services have automated the deployment of the above components and provided an open and flexible set of interfaces to integrate into existing Big Data environments. To learn more about the Elastic Data Platform and BlueData's collaboration with Dell EMC, visit the BlueData booth (# 433) at the Strata Data Conference in New York City this week, September 26th to 28th.

http://m.marketwired.com/press-release/bluedata-provides-ability-deploy-containerized-hadoop-spark-dell-emc-elastic-data-platform-2235027.htm

Actian Vector in Hadoop Turbocharges Spark Performance

PALO ALTO, Calif.--(BUSINESS WIRE)--#Actian, the hybrid #datamanagement, #analytics and integration company, today announced support for #Apache® #Spark™ in the latest release of Actian Vector in #Hadoop ( #VectorH ), providing enterprises with access to an expansive range of data types and faster and richer analytic functionality than ever before. Actian’s pioneering Vector technology exploits vectorized processing and multi-level, in-memory acceleration to deliver industry leading performance on Hadoop data stores. The Actian Vector family of analytic databases offers innovative, flexible, best-fit solutions for single server, clustered and hybrid transactional/analytic computing environments, spanning on-premise and cloud deployments. @ActianCorp announces latest support for #ApacheSpark w/ #VectorH. Learn more here: actian.com/vectorh Tweet this “Customers facing the challenges of today’s dynamic, hybrid data environments are demanding faster, more nimble solutions to activate that data, to unlock the business value and insights, using the skills and tools they already know,” said Rohit De Souza, CEO of Actian. “Actian now offers the power of VectorH through Spark to support a diverse range of file formats and workloads including machine learning, delivering performance and operational advantages over open source and proprietary alternatives. Imagine how much more productive your data scientists and business analysts can be with results in seconds, not hours.” Actian VectorH opens up rich analytics functionalities for developers and data scientists seeking to harness the power of Apache® Spark™ capabilities including machine learning, streaming and predictive analytics, and creates a modern production platform that offers security, real-time data updates with zero performance penalty, resource management in the Hadoop cluster, query optimization and industry standard SQL support. Highlights of the Actian VectorH analytics database: Scale-out Hadoop performance – Dramatically improved performance with Actian Vector technology – VectorH enables extensibility of the world’s fastest database from single node processing to accelerate performance in Hadoop clusters. Workloads of standard benchmark queries that typically take over two hours with traditional SQL solutions on Hadoop finish in less than a minute running on VectorH. Zero-penalty, real-time data updates – Unlike traditional Hadoop analytics solutions, VectorH can process real-time data updates without any associated performance penalty, an essential capability to ensure that an organization’s analytic insight is always current, leveraging its freshest data. Organizations no longer have to give up consistency for performance. Spark powered direct query access – Through its innovative native Spark support, VectorH delivers optimized access to Hadoop data file formats including Parquet and ORC, circumventing the need to translate and store data separately into the Vector file format. This direct access includes the ability to perform functions like SQL joins across different table types. Native Spark DataFrame support – Direct connection to Spark functionality via DataFrames, enabling VectorH to serve as a faster query execution engine for SparkSQL and Spark R applications. “As an innovative provider of leading network monitoring solutions to manage global transportation and mobile systems, Expandium pushes the edge of Big Data technologies, starting with Actian Vector,” said Rodolphe Guillard, software team leader, Expandium. “With explosive growth in mobile data, we’ve developed our new network intelligence platform to scale up on Actian VectorH to perform near real-time data ingestion in a production environment. We’re excited about employing Actian’s new native Spark integration to stream data to machine learning solutions to sustain our technical leadership.” The new version of Actian VectorH will be released at the end of October. To learn more, visit here or stop by booth #350 at the Strata Data Conference taking place September 25-28 at Javits Center in New York, NY. You can also explore the single-server version of Actian Vector running on Linux, distributed free as a community edition, available for download here. About Actian -- Activate your Data™ Actian, the hybrid data management, analytics and integration company, delivers data as a competitive advantage to thousands of customers worldwide. Through the deployment of innovative hybrid data technologies and solutions Actian ensures that business critical systems can transact and integrate at their very best – on premise, in the cloud or both. Thousands of forward-thinking organizations around the globe trust Actian to help them solve the toughest data challenges to transform how they run their businesses, today and in the future. For more, visit http://www.actian.com.

http://www.businesswire.com/news/home/20170919005531/en/Actian-Vector-Hadoop-Turbocharges-Spark-Performance

DriveScale and Dell EMC Expand Partnership to Provide Enterprise Solutions for Data Center Operators

SUNNYVALE, CA--(Marketwired - Sep 7, 2017) - #DriveScale, the leader in #SoftwareComposableInfrastructure ( #SCI ) for modern workloads, today announced that #DellEMC will resell DriveScale's solution to help customers dramatically improve the efficiency and agility of their data center infrastructure for cloud native applications. By offering DriveScale's SCI technology, Dell EMC can help meet the growing demand amongst its customer base for a more flexible architecture to run distributed workloads such as #Hadoop, #Cassandra and other scale-out systems. "Through this strategic partnership, DriveScale and Dell EMC will work toward a common goal of providing the enterprise with efficient and cost-effective infrastructure solutions," said Gene Banman, CEO of DriveScale. "By expanding our reach into the Dell EMC customer base, we are well-positioned to help combat resource over-provisioning in the data center and help new and existing customers scale their businesses. We are proud to partner with world-class technology solution providers like Dell EMC and look forward to creating new opportunities for the channel." Supporting Modern Workloads with Software Composable Infrastructure Modern workloads, such as Hadoop, NoSQL databases and other big data technologies, are dynamic and unpredictable, with data growing exponentially as enterprises find new ways to mine this information for business value. This has resulted in IT administrators struggling to correctly size physical infrastructure for each application or respond to changing workloads that are experiencing a spike in demand. DriveScale's unique SCI solution provides organizations with a next-generation data center architecture that treats hardware infrastructure as code -- delivering significant gains in efficiency and agility while lowering both equipment and operating costs. DriveScale's SCI solution allows data center administrators to disaggregate their infrastructure into separate pools of compute and storage resources, and quickly compose and re-compose them to meet the specific needs of a workload. IT teams can achieve higher utilization levels, reduce over-provisioning and respond much more quickly to changing business requirements. The DriveScale System does not require any changes to an existing application stack and delivers equivalent performance to a bare metal server with internal storage. Expanded Partnership with New Reseller Agreement Extends SCI to Dell EMC Customer Base Earlier this year, DriveScale joined the Dell EMC Tech Connect Program and was certified on Dell EMC PowerEdge Servers, Ethernet Switches and Direct Attached Storage. This new reseller agreement deepens the relationship between the two companies and provides customers with an easier path to manage big data in private and hybrid cloud environments. Recognized as a leader in Software Composable Infrastructure, DriveScale provides Dell EMC partners with a powerful addition to their data center offerings that enables their customers to meet the challenges of today's dynamic workloads. "We chose DriveScale as our solution because its software-defined approach helps us scale our data center quickly and easily while reducing resources trapped in siloed clusters," said Tim Smith, SVP of technical operations at AppNexus. "This expanded relationship between Dell EMC and DriveScale allows for a streamlined technology engagement with an efficient acquisition model." About DriveScale DriveScale is the leader in Software Composable Infrastructure for modern workloads. Our innovative data center solution empowers IT to disaggregate compute and storage resources and quickly and easily recompose them to meet the needs of the business. Enterprises can respond faster to changing application environments, maximize the efficiency of their assets, and save on equipment and operating expenses. DriveScale supports modern workloads such as Hadoop, Spark, Kafka, NoSQL, Cassandra, Docker, Kubernetes and other distributed applications at a fraction of the cost of alternative platforms. DriveScale, based in Sunnyvale, CA, is founded by technologists with deep roots in IT architecture that built enterprise-class systems for Cisco and Sun Microsystems. Investors include Pelion Venture Partners, Nautilus Venture Partners and Ingrasys, a wholly owned subsidiary of Foxconn. Visit www.drivescale.com or follow us on Twitter at @DriveScale_Inc.

http://m.marketwired.com/press-release/drivescale-dell-emc-expand-partnership-provide-enterprise-solutions-data-center-operators-2232829.htm

Apache Spark and Apache Hadoop on Google Cloud Platform Documentation

You can run powerful and cost-effective #Apache #Spark and Apache #Hadoop clusters on #GoogleCloudPlatform. The easiest way to do this is with #GoogleCloudDataproc, a managed Spark and Hadoop service that allows you to create clusters quickly, and then hand off cluster management to the service. If you want to set up and manage a Spark/Hadoop cluster on your own, see Cloud Dataproc downloads to obtain resources to help you accomplish these tasks.

https://cloud.google.com/hadoop/?hl=zh-CN

Big data integration tool targets Hadoop skills gap

The software company last week debuted a developer edition of #iWay #BigData Integrator, which it is offering at no cost for six months. The tool could boost the productivity of channel partners working on big data projects and could help other partners launch businesses in the growing market, industry executives said. #BigDataIntegrator, an evolution of the #iWay #Hadoop Data Manager, initially launched in March 2017 on the #Microsoft #Azure Marketplace Cloud. According to Information Builders, Big Data Integrator simplifies Hadoop, providing a graphical interface for Hadoop and #Apache #Spark data ingestion and transformation that reduces the need for coding. Hadoop, a software framework for storing and processing large data sets, provides a foundation for developing big data applications. Apache Spark is a data processing engine that supports in-memory computing and can run on Hadoop. Jake Freivald, vice president of product marketing at Information Builders, based in New York, said harnessing the power of Hadoop is difficult and involves complicated programming, adding the task of managing Hadoop environments is somewhat arduous, as well.
http://searchitchannel.techtarget.com/news/450424293/Big-data-integration-tool-targets-Hadoop-skills-gap

MapR Technologies Launches Top-Tier Channel Program To Recruit, Enable Systems Integrators

#Bigdata platform developer #MapR Technologies is expanding its channel partner initiatives with a new Elite Premier program for the company's global and regional systems integrator partners. The main focus of the new Elite Premier program, part of the company's larger MapR Converge Partners program, is to recruit and enable systems integrators to work with the vendor's MapR Converged Data Platform, said Geneva Lake, vice president of worldwide alliances.  "Systems integrators are becoming a more critical part of MapR's business," Lake said in an interview with CRN, noting that the company began a push to recruit systems integrators last year. [Related: The 2017 Big Data 100] "We really need to make sure they understand the product and how to implement it. It's very important for us to have that high-quality, high-performing group of systems integrators." The #MapR Converged Data Platform integrates #Hadoop, #Spark and #Apache Drill with a real-time database, event streaming and other capabilities for running big data applications. The company, based in San Jose, Calif., competes with #Cloudera and #Hortonworks in the Hadoop/big data platform arena. MapR currently works with one global systems integrator and is developing enablement plans with two more. Lake said MapR would probably cap the number of global systems integrators in the program at three. The vendor is working with about 10 additional boutique, regional and super-regional systems integrators and is looking to recruit more, especially in geographies and vertical industries where the company doesn't have a presence. Some systems integrators focus on implementation work while others provide additional services or have consulting practices in areas such as business analytics and business process management. Enabling them with the proper training and resources is key, Lake said, because projects incorporating the MapR Converged Data Platform tend to be complex. Through the Elite Premier program systems integrators will be offered comprehensive training on MapR's software, including a boot camp training session on installing and deploying the product. One systems integrator MapR is already working with is RCG Global Services, based in Edison, N.J. The company incorporates MapR's software into its data analytics practice, which includes big data strategic planning, systems development and implementation services, said John Bender, senior vice president and general manager. The company develops use cases and analytical applications for system maintenance, health care and other tasks. The new Elite Premier program signals a "greater level of commitment" by MapR to its systems integration partners and will "further the strategic alignment between our two companies," Bender said. The program will provide RCG with access to more training and certification opportunities "to keep our skills up to date," he added, and help the company stay on top of new software releases. And he expects the program to provide more joint sales and marketing investments, although he said the two already hold joint marketing events for prospects. Lake noted that while MapR Technologies does have a professional services business, it only accounts for about 15 percent of the company's revenue and it has no plans to expand that business – hence the need for experienced systems integrators. MapR has a range of consulting/integration, ISV, OEM and distribution partners. More than 50 percent of the vendor's sales involve a partner in some way, according to Lake. Systems integrators play an influencer role in about 20 percent of MapR's sales – a number that Lake said has been increasing

http://m.crn.com/news/channel-programs/300089346/mapr-technologies-launches-top-tier-channel-program-to-recruit-enable-systems-integrators.htm

IBM Combines All-Flash and Storage Software Optimized for Hortonworks

ARMONK, N.Y., July 24, 2017 /PRNewswire/ -- #IBM (NYSE: IBM) today announced a new #allflash, high-performance data and file management solution for enterprise clients running exabyte-scale #bigdata analytics, cognitive and #AI applications. The combined flash and storage software solution has been certified with the #Hortonworks Data Platform (HDP) to provide clients with more choice in selecting the right platform for their big data analytics on data processing engines like Hadoop and Spark. The new IBM All-Flash #ElasticStorage Server ( #ESS ), powered by IBM #SpectrumScale exabyte-scale software defined storage, increases data throughput by 60 percent1 over previous solutions. Performance improvements produce a storage solution powerful enough to run complex applications from machine and deep learning clusters to image recognition of cancer cells. The #IBM #SpectrumScale storage software combines data from silos across the enterprise into a single view in order to run analytics on #Hadoop and #Spark. Through unifying file, object and Hadoop #HDFS data storage, #IBMESS with IBM Spectrum Scale combines to shorten time to results while contributing to reducing storage capacity requirements by more than 55 percent2 for Hadoop and Spark workload data. Beyond performance and economic benefits, the solution provides user data with essential features such as redundant data paths, distributed erasure coding and end-to-end checksums. Based upon millions of hours of current production systems, reported over the past two years the IBM ESS design has delivered five nines of availability against unscheduled downtime. "We're seeing an increasing appetite on the part of clients to unlock value within their own business data where competitive advantage can be found but too often they face challenges that inhibit ready access to this information," said Ed Walsh, general manager, IBM Storage and Software Defined Infrastructure. "IBM ESS with Spectrum Scale has been developed to make data available by reducing obstacles that prevent users from quickly accessing and processing the growing oceans of differing data types stored within their disparate, isolated storage systems." IBM All-Flash ESS represents a new generation of data and file management. To help clients more effectively manage their hot and cold data. IBM ESS with IBM Spectrum Scale, an exabyte-scale software defined storage solution, supports a wide variety of network protocols while providing clients the elasticity to transparently tier data across flash, disk, tape and cloud storage systems. With all IBM ESS and IBM Spectrum Scale models certified on Hortonworks, we fulfill a commitment to deliver higher exabyte scale storage densities and related performance throughput required for programs like IBM's support for the Department of Energy CORAL project. IBM ESS integration with IBM Spectrum Scale, which is being developed to support anticipated NVMe-based storage solutions from IBM, sets the stage for increased proficiency in managing throughput of Big Data analytics workloads. Availability IBM will be taking orders on the new IBM All-Flash Elastic Storage Server (ESS) today but the solution will be generally available on August 25, 2017. To learn more about the IBM ESS portfolio please visit our website. For more details about the latest IBM All-Flash ESS, please see our blog "Fast data: the future of big data." 1 - Comparing the IBM ESS GS6 25GB/s against the new IBM ESS GS6S 40GB/s maximum throughput 2 - Comparing open source HDFS 3x data replication v. IBM Spectrum Scale RAID 1.22x erasure coded data protection.
http://www.prnewswire.com/news-releases/ibm-combines-all-flash-and-storage-software-optimized-for-hortonworks-300492966.html

“Hide the Tech” to Take Big Data Mainstream

#Bigdata is a big deal – and a big opportunity. The challenge is that most big data progress has been limited to big companies with big engineering and data science teams. The systems can be complex, immature and hard to manage. That might be OK if you’re a well-trained developer in Silicon Valley, but it doesn’t play well if you’re a jack-of-all-trades IT leader at a bank in Atlanta or an on-the-move business leader in Amsterdam. How can you tap into big data if you don’t have an army of engineers who stay steeped in the latest tech? Fortunately, help is on the way thanks to several trends that move tech from the foreground to the background. First and foremost, big data is moving into the cloud. #Amazon, #Microsoft and #Google now deliver #Hadoop and #Spark “as a service,” eliminating the need to spin up a cluster, research new software products or worry about version management. Companies are increasingly moving their big data workloads into the cloud, hiding complexity from their users while relying on the world’s best data center professionals to manage the tech. They want access to infrastructure when and how they want it, with just as much as they need but more.

https://insidebigdata.com/2017/07/21/hide-tech-take-big-data-mainstream/

OpenText vows to take on IBM Watson with new Magellan AI platform

IEnterprise information management company #OpenText has announced the general availability of its new #artificialintelligence ( #AI ) and cognitive computing platform, #Magellan, at its Enterprise World conference in Toronto. Attendees at the show were left in little doubt as to how senior executives at the Waterloo, Ontario-based company would like them to view Magellan: as a direct competitor to #IBM ’s #Watson. OpenText isn’t targeting Salesforce’s Einstein, nor similar AI-based initiatives from #SAP or #Pegasystems, CEO Mark Barrenechea stressed in a press conference at the event: It’s Watson [that we’re targeting]. What differentiates us is that the information-centric applications that we already have will be integrated directly with Magellan. We don’t want to extract information – from contracts, for example – out of applications and into an AI platform. The fact that you have to feed Watson with data extracted from multiple resources, I think, is a weakness. In other words, OpenText is aiming to embed Magellan’s AI capabilities in some of the tools that comprise its vast portfolio of software products. These typically focus on unstructured and semi-structured information, in areas such as enterprise content management, customer experience and data discovery. Where the company believes Magellan can stand out is in its understanding of human language (as opposed to transactional data), as it is expressed in documents such as reports, memos, emails, contracts and requests for information, via text mining and semantic analysis. Built by OpenText? This looks likely to be a significant challenge, because OpenText has largely been built through acquisition. In the last 20 or so years, it’s acquired over 50 different information management companies. Most recently, these have included its $1.62 billion purchase of #DellEMC s enterprise content software, which included former arch-rival #Documentum, in a deal that closed in January, and its April acquisition of customer experience software and service assets from #HPE, for $170 million. In fact, #Magellan itself has its roots in technology acquired in two acquisitions: the 2010 purchase of #NsteinTechnologies, which brought text mining and auto classification capabilities to OpenText; and the 2015 purchase of Actuate, which brought it analytics capabilities. On top of these technologies, OpenText has introduced integration with two open source products: big data framework #ApacheSpark and #ApacheMLlib, a machine learning library. Algorithms for Magellan can be programmed in languages including #Scala, #Python and R. But as Barrenechea stressed: There’s also a whole ton of our own engineering in there. We may start with M&A, but if we can’t grow a product, if we can’t innovate, if we can’t get it to evolve, then we’re not interested in buying it. Magellan is built by OpenText. Challenges ahead It’s no surprise that Barrenechea is keen to stress OpenText’s engineering and innovation credentials. With the launch of Magellan and the shift into AI, the company will find itself in a fiercely competitive and fast-moving field of technology. It could prove to be an unforgiving environment for a company built on acquisition, as opposed to organic growth fuelled by new products. As Barrenechea told analysts on the company’s third quarter results call: M&A remains our leading growth driver, and we will continue to grow and invest in our M&A team. It’s hard, then, not to draw parallels with the kind of strategy made famous by CA Technologies during the 1990s and early 2000s: buy up intellectual property from other software companies, provide basic maintenance and milk the installed base. The fact that Barrenechea worked at CA Technologies (then known as Computer Associates) in various roles during those years – as product development head, president of technology strategy and chief technology officer – simply reinforces that comparison. To sweeten the deal for prospective Magellan customers, the product will be sold at a much lower price point than IBM Watson, will be available as an appliance that runs on preconfigured, commodity hardware and be available on premises or via the cloud. A lower cost of entry could appeal to a wider range of customers than might consider IBM Watson a viable investment. As Barrenechea put it: With OpenText Magellan, we are moving towards a more open, scalable and affordable future for AI. OpenText is committed to ensuring AI is designed into its offerings from day one.

http://diginomica.com/2017/07/14/opentext-vows-take-ibm-watson-new-magellan-ai-platform/

Look for advances in serving the Hadoop data scientist

#Hadoop and its proponents have been stymied in bringing the distributed data framework to wider use, as very special, end-to-end skills are required to get this style of distributed big data application up and running.

However, changes are afoot. The original style of Hadoop based on MapReduce and the Hadoop Distributed File System (HDFS) has given way to interchangeable configurations that may use neither MapReduce, nor HDFS. Cloud-based Hadoop is on the upswing. And vendors are trying to bring self-service capabilities to the Hadoop data scientist.

To some extent, #Spark was a reaction to Hadoop's initial complexity. Spark set about to improve upon original Hadoop's MapReduce data processing model, and also upped the level of abstraction for programmers. Building systems still required Java programmers to do much of the work, but they did not have to deal with the down-and-dirty camshafts and flywheels of fairly raw Java. Similar motivation was also behind the tools created to open up Hadoop to SQL and a wider programming audience.

Yet, the problem remained: Hadoop and Spark in production called for near-superhero users who possessed skills covering a range of jobs -- including system admin, system programmer, Java developer and data engineer. Why not throw in domain expert, statistician and Hadoop data scientist, too?

http://searchdatamanagement.techtarget.com/opinion/Look-for-advances-in-serving-the-Hadoop-data-scientist