Hadoop and Spark: Synergy Is Possible

If somebody mentions #Hadoop and #Spark together, they usually contrast these two popular big data frameworks. According to Ahrefs, 1,200 #Google visitors are searching for Spark vs. Hadoop each month, while only 90 are inquiring about Spark and Hadoop. It looks like the frameworks have gradually gained a reputation of being mutually exclusive. But this is not always the case. There are multiple ways for businesses to benefit from their synergy. Let’s take a closer look at Hadoop and Spark and discover scenarios where they can work together. #ApacheHadoop Defined Apache Hadoop is an open-source framework for data storage and parallel processing. Initially released in 2011, Hadoop triggered #bigdata evolvement. Distributed #datastorage allowed companies to cope with big data volumes. They didn’t need to buy extremely expensive custom hardware anymore. Instead, they could use multiple affordable computers to store data. Besides, this approach enabled the much-needed scalability of the solution. When the amount of data to be stored and processed increased, companies could solve that challenge by adding extra computers. With such an approach to data storage, parallel data processing was required. This became another distinctive feature of Hadoop. Apache Spark Defined Apache Spark is an open-source framework for parallel processing. Released in 2014, Spark was designed to cope with the shortcomings of Hadoop MapReduce, which was mainly the speed of processing. Unlike Hadoop MapReduce, which has to write interim analysis results back to the disk and then read the data again and again, Spark processes data in-memory. As a result, it is up to 100 times faster than Hadoop MapReduce. By the way, if we talk about real alternatives, there are Apache Spark and Hadoop MapReduce (not entire Hadoop). And this is evident already from Spark’s definition. Using Hadoop and Spark Together One important remark: the Hadoop ecosystem consists of several components, among which is Hadoop Distributed File System (or HDFS for short), Apache Hive (a query engine), Hadoop MapReduce (a framework for the parallel processing of distributed large datasets), and more. With this information in mind, let’s take a look at possible synergy scenarios. HDFS + Apache Spark We have already clarified that Apache Spark’s intended purpose is data processing. But to process data, the engine needs to take it from some storage first. HDFS is not the only option available, but it's a quite frequent one. The reason is simple: both belonging to Apache Foundation family, HDFS and Spark are highly compatible. An illustrative example of such a synergy is a word count (you can find the code example here). The sequence of operations is as follows: Apache Spark takes a text file from HDFS, divides each line into separate words, sets the value 1 for each word, calculates the sum of values for each word, and records the result to HDFS. Apache Hive + Apache Spark The combination of Apache Spark and Apache Hive (that is based on HDFS) allows solving many business tasks, for example, conducting customer behavior analytics. Imagine a company that cumulates data from multiple sources: clickstream data, comments, and posts on social media, data from customer mobile apps, etc. Let’s say that the company has chosen HDFS to store their data and Apache Hive to act as an intermediary between HDFS and Spark. Apache Hive makes it possible to query the data using a SQL-like language. As a result, Spark that has special support for Hive could easily access the data and process it. In the end, the company can understand the preferences and behavior patterns of each customer. Real-Life Examples of Spark and Hadoop Duets Real-life examples of using Hadoop and Spark together are not rare in big data consulting practices. The list of companies that adopt such an approach includes many well-known names. Undoubtedly, their solutions are of different complexity. And this is understandable, as these companies strive to solve different business tasks. Still, there is one thing that unites them: their big data technology stack includes both Hadoop and Spark. Let’s look at the following two examples. TripAdvisor uses Hadoop and Spark together to deliver a seamless customer experience. They introduced auto-tagging, which is based on the analysis of visitors’ reviews and tags. This feature allows TripAdvisor to predict whether a visitor’s impression of a particular location will be the same as that of the other visitors. Another interesting feature is improved photo selection. Now, a website visitor can get a more precise picture of any location thanks to a better choice of visuals. For instance, if a hotel has a pool, machine learning algorithms will pick the photo of the pool and show it to the visitor. Uber is doing a great job of managing their big data to improve their service. They know the typical behavior of each customer (starting and destination points, usual day and time of their journeys, etc.). The company also uses real-time traffic situations to adjust the number of drivers needed at a particular time and in a particular location. To make this possible, Uber uses HDFS for loading raw data onto Hive for SQL-powered analysis and Spark for processing of millions of events. Conclusion Now, you can see that Hadoop and Spark can smoothly work together. We have supplied the article with real-life examples so that you can see that the synergy of these big data frameworks is possible not only in theory but also in practice. When making a choice, just remember one of the maxims of big data: your big data technology stack should suit your business goals.

https://dzone.com/articles/hadoop-and-spark-synergy-is-real

Microsoft's cloud Big Data service cuts prices up to 52 percent

@Microsoft has decided to get down to business with @HDInsight (HDI), its #Azure cloud-hosted #BigData offering, based on @Apache #Hadoop, @HBase, @Spark, @Storm, @Kafka, @Hive LLAP, and #MicrosoftRServer. Ostensibly, Microsoft previously considered its competition to be on-premises Hadoop implementations. But it now offers pricing that is far more competitive with @Amazon Web Services' (AWS') @Elastic @MapReduce (EMR), while still offering a 3-nines service level agreement (SLA) as a differentiator. Details, details The pricing changes, highlighted in a blog post by Microsoft's Rimma Nehme and detailed on a separate page, offer varying price cuts depending on the virtual machine type used for the head and worker nodes in the HDInsight cluster. Price cuts are up to 52 percent, Microsoft says, while the service itself remains largely the same. In addition, for those customers wishing to run data science workloads with code written in R, the surcharge for running R Server in a distributed fashion on an HDI cluster has been cut by 80 percent, down to just $0.016 (i.e. 1.6 US cents) per CPU core, per hour. Microsoft points out that because of Azure's numerous global data centers (regions), HDI is available at more points of presence than any other cloud Hadoop service. In addition to Azure's mainstream cloud, the service is also available on its US government cloud and on so-called sovereign clouds, including those in Germany and China. Per various regulatory requirements, the sovereign clouds run in facilities operated by local partners, rather than Microsoft itself. In other news Microsoft has a few other announcements to go with the price change: The introduction, available in preview, of HDInsight Enterprise Security Package, which integrates Microsoft Active Directory with Apache Ranger. This is essentially a re-branding of HDInsight's Premium cluster tier General availability of its Apache Kafka cluster type, which had been in preview until recently General availability of HDInsight's integration with Azure Log Analytics Public preview of support for Power BI DirectQuery, specifically against Hive LLAP, used in HDI "Interactive Query" cluster types New HDInsight add-in developer tools for IntelliJ, Eclipse and Visual Studio Code (Microsoft's cross-platform code editor for MacOS, Linux and Windows). The IntelliJ and Eclipse tooling include the ability to submit and debug distributed Spark code right from those development environments. The VS Code tooling allows for interactive execution of PySpark (a Python library for Apache Spark) code.

http://www.zdnet.com/article/microsofts-cloud-big-data-service-cuts-prices-up-to-52/

Cloudera Bringing Impala to AWS Cloud

#Apache #Impala, the #SQL-based #analytical #database that originated at #Cloudera, will soon be available as a managed service on the @Amazon Web Services cloud, the #Hadoop software distributor announced today at #AWS Re:Invent. Cloudera Altus Analytic DB, as the hosted cloud version of Impala will be known, will be available as a beta service on AWS by the end of the year, with general availability expected in 2018. The company says support for the #Microsoft #Azure cloud will follow, but no timeline was given. Impala is one of the most popular engines in Cloudera’s Distribution of Hadoop (CDH), and the open source software is also offered in other Hadoop distributions. The software essentially allows customers to run a host of standard SQL queries against massive stores of relational data stored in Parquet, an optimized Hadoop file format. It’s not the only parallel SQL data warehouse designed to run atop Hadoop, but it is one of the most mature. Getting customers the capability to run SQL workloads against data hosted in cloud-based object storage repositories was a big priority for Cloudera, says Alex Gutow, a product manager with the Palo Alto, California-based company. “When we look around at our customers and the types of workloads that are really best suited to take advantage of the agility and cost efficiencies of the cloud, BI and analytics is one of the key workloads,” Gutow tells Datanami. Specifically, BI workloads running on Impala will benefit tremendously from features customers can find on AWS cloud, such as multi-tenant isolation and workload elasticity, Gutow says. Instead of leaning on in-house administrators to procure computational resources and then manage it for the user on an on-going basis, the Altus Analytic DB service allows customers to lean on Amazon and Cloudera to do that heavy lifting for them, she says. “So you can have a very specific cluster to run reporting workloads, and you can have another to run ad hoc queries or self-service BI,” Gutow continues. “It allows for much more of that agility, giving all different types of analysts access to shared data very quickly, giving them much more flexibility, and being able to elastically scale up those resources as you need to meet different performance requirements, or to make sure there’s predicable performance for those workloads.” Altus Analytic DB will access data stored in customers’ Simple Storage Service (S3) accounts. Impala has been able to access data stored in the S3 object store via an HDFS API for about a year, says Greg Rahn, a Cloudera product manager for Impala and Altus. “HDFS provides an API to S3 known as an S3A connector. Impala uses this,” Rahn says. “It looks to Impala as if it’s kind of the HDFS file system, or the abstraction thereof.” There’s a similar API that exposes data stored in Microsoft’s cloud object store, ALDS, through HDFS, and the company will use that connector when Altus Analytics DB is supported on the Azure cloud in the future. “So at the end of the day, whether the data is in S3 or ALDS or HDFS itself, it all kind of looks the same in terms of the visibly it to impala,” Rahn says. Cloudera has pre-selected certain Elastic Cloud Compute (EC2) instances that Altus Analytic DB will be allowed to run upon. Customers will be able to spin up Altus Analytic DB clusters with just a few clicks of the mouse, Rahn says.  Cloudera is positioning its SDX as a key product uniting customers’ cloud and on-premise Hadoop deployments “The Altus deployment makes it quite trivial to start up these things, probably on the order of three to four clicks to provision a cluster,” he says. “You log in, name the cluster, pick the size of the instance, the number you want, then you hit ‘create cluster.’ So it’s very simple.” Customers will be able to quickly spin up analytic clusters on AWS, run a workload, and then quickly dispose of it. None of the data, metadata, or state information for these jobs will be lost when the cloud cluster is deleted because it’s all managed centrally under Cloudera’s Shared Data Experience (SDX), which the company announced at the Strata Data Conference in September. The SDX provides a way to manage data access and permissions for on-premise and cloud environments from a central console. The software sports hooks into core management tools, including Cloudera Navigator, Cloudera Manager, and Sentry for on-premise implementations and Altus controls for cloud-based environments. “Not only can you provide these different isolated resources for each of the different workloads,” Gutow explains, “but from the management side of things, these all benefit from having shared security, shared governance, and shared metadata as running actors shared data layer in the cloud, the shared object storage. So each time any of these different workloads are provisioned or run for different self-service workloads, you don’t have to go and redefine the different security policies. You can easily manage them from an enterprise standpoint.” Altus Analytic DB will be the second hosted offering under the Altus banner since Cloudera announced its new platform as a service (PaaS) in May. The first offering, Altus Data Engineering, was focused on data ingest and transformation tasks, and includes Spark, Hive, Hive on Spark, and MapReduce2 engines. Cloudera was mum on what engines will come next for Altus. Kudu, its fast-data layer, is one obvious candidate. Cloudera is currently in a quiet period before it announces financial results on December 7.

https://www.datanami.com/2017/11/28/cloudera-bringing-impala-aws-cloud/

Define and Process Data Pipelines in Hadoop With Apache Falcon

#ApacheFalcon is a framework to simplify data pipeline processing and management on #Hadoop clusters. It makes it much simpler to onboard new workflows/pipelines, with support for late data handling and retry policies. It allows you to easily define relationships between various data and processing elements and integrate with metastore/catalog such as #Apache #Hive/ #HCatalog. Finally it also lets you capture lineage information for feeds and processes. In this tutorial we are going to walkthrough the process of: Defining the feeds and processes Defining and executing a data pipeline to ingest, process and persist data continuously Prerequisites Download #Hortonworks Sandbox Complete the Learning the Ropes of the Hortonworks Sandbox tutorial, you will need it for logging into ambari as an administrator user. Complete the Creating Falcon Cluster tutorial to start the falcon service, prepare HDFS directories for Falcon cluster and to create Falcon cluster entities. Once you have downloaded the Hortonworks sandbox and run the VM, navigate to the Ambari interface on port 8080 of the host IP address of your Sandbox VM. Login with the username of admin and password that you set for the Ambari admin user as part of the Learning the Ropes of the Hortonworks Sandbox tutorial:

https://community.hortonworks.com/articles/110399/define-and-process-data-pipelines-in-hadoop-with-a-1.html

Yahoo’s Massive Hadoop Scale on Display at Dataworks Summit

#Yahoo put its massive #Hadoop investment on display this week at #Dataworks Summit, the semi-annual big data conference that it co-hosts with #Hortonworks. While Hadoop is no longer the conference headliner that it once was, the platform is still critical for the daily operations of Yahoo, which officially became part of #Verizon Communications this week when the $4.5 billion acquisition finally closed. With 120,000 servers and 800 PB of in storage, few companies have the computing scale of Yahoo. And as the birthplace of this distributed computing platform called Apache Hadoop, it’s worth keeping an eye on what Yahoo is doing with its collection of big data tech. Sumeet Singh, the senior director of cloud and big data platforms at Yahoo, took to the Dataworks Summit stage on Wednesday to describe how the technological makeup of Yahoo’s massive cloud platform has evolved over the years. For starters, the company is moving solidly away from #MapReduce. Over the past 17 months, Tez has replaced MapReduce as the underlying engine for many of the batch-oriented #Pig and #Hive workloads that Yahoo relies on to serve its 1 billion monthly users. Today, 70% of the Hadoop workloads and Yahoo run under Tez, according to Singh. Use of Apache Spark has also grown, but not nearly as quickly as Tez, he says. Singh referred to this switch from MapReduce to Tez and Spark as “compute shaping.” “What compute shaping does, it allows us to make better use of the platform,” he says. “This is fantastic for the company and our customers because they can make better use of the capacity.”

https://www.datanami.com/2017/06/16/yahoos-massive-hadoop-scale-display-dataworks-summit/

Streaming Live Data and the Hadoop Ecosystem

Oleg Zhurakousky discusses the #Hadoop ecosystem – Hadoop, #HDFS, #Yarn -, and how projects such as #Hive, #Atlas, #NiFi interact and integrate to support the variety of data used for analytics.

https://www.infoq.com/presentations/streaming-hadoop

APACHE BIG DATA SEVILLE 2016 – HADOOP, HIVE, SPARK AND OBJECT STORES – STEVE LOUGHRAN

#Hadoop, #Hive, #Spark and #ObjectStores – Steve Loughran Cloud deployments of #Apache Hadoop are becoming more commonplace. Yet Hadoop and it’s applications don’t integrate that well äóîsomething which starts right down at the file IO operations. This talk looks at how to make use of cloud object stores in Hadoop applications, including Hive and Spark. It will go from the foundational “what’s an object store?” to the practical “what should I avoid” and the timely “what’s new in Hadoop?” äóî the latter covering the improved S3 support in Hadoop 2.8+. I’ll explore the details of benchmarking and improving object store IO in Hive and Spark, showing what developers can do in order to gain performance improvements in their own code äóîand equally, what they must avoid. Finally, I’ll look at ongoing work, especially “S3Guard” and what its fast and consistent file metadata operations promise.

http://feathercast.apache.org/apache-big-data-seville-2016-hadoop-hive-spark-and-object-stores-steve-loughran/

Tips for populating Big SQL and Hive Hadoop tables with DATE types

When creating external #Hive tables defined with DATE columns, ensure that the values in the data files on #HDFS ( #Hadoop ) correspond to DATE values and not a mix of DATE and TIMESTAMP values. The same is true for when creating Hive tables and using the Hive INSERT or INSERT…SELECT commands to add data to tables. When Hive expects a DATE type, but instead finds a TIMESTAMP type in the data file, then a NULL value is inserted to the table. NULL values can have a negative impact on query performance especially for queries performed against partitioned tables where the partitioning keys are NULL values. This is because Hive will put all NULL values into one partition.

https://developer.ibm.com/hadoop/2016/12/16/tips-for-populating-big-sql-and-hive-hadoop-tables-with-date-types/

BI On Hadoop Success: 7 Things To Know

#Businessintelligence is the top use-case for IT organizations implementing #Hadoop, according to a large industry survey last year. Now a new benchmark study shows which Hadoop ecosystem tools are best for particular types of BI queries. The recently released study's findings offer perspective for IT organizations on a handful of SQL-on-Hadoop engines, such as #Hive, #Impala, #Presto, and #Spark SQL. They provide insights on their performance for particular kinds of analytic jobs. The benchmark comes from #AtScale, a company that is focused on helping organizations make business intelligence work on Hadoop. "Different engines continue to perform well for different jobs," said Josh Klahr, VP of products at AtScale, in an interview with InformationWeek. "IT organizations should probably be wary about making a bet on just one engine -- like putting everything on Hive or on Impala." This new benchmark released this week is the second edition. It provides insight into how the performance of each of these engines has improved since the last report, released 6 months ago.
http://www.informationweek.com/big-data/software-platforms/bi-on-hadoop-success-7-things-to-know/d/d-id/1327225

Building a common data platform for the enterprise on Apache Hadoop

To become a data-driven enterprise, ( #Bigdata ) organisations must process all types of data, whether it be structured transactions or unstructured file server data such as social, #IoT or machine data. Competitive advantage is at stake, and companies failing to evolve into data-driven organisations risk serious business disruption from competitors and startups.

Fortunately, we live in a time of unprecedented innovation in enterprise software and enterprise data has finally become manageable on a large scale. Thanks to the #Apache #Hadoop open source framework delivering enterprise archives, data lakes and advanced analytics applications, enterprise data management solutions are now able to turn the tide on data growth challenges.

Enter the Common Data Platform (CDP): a uniform data collection system for structured and unstructured data featuring low-cost data storage and advanced analytics. In this article, I’m going to define the components of a CDP, and where it stands alongside the traditional enterprise data warehouse.

1. #Apache Hadoop

Apache Hadoop is the backbone of the CDP. Hadoop is an open-source data management system that distributes and processes large amounts of data in parallel (across multiple servers and distributed nodes). It’s engineered with scalability and efficiency in mind, and designed to run on low-cost commodity hardware. Using the Hadoop Distributed File System ( #HDFS;), #Hive and #MapReduce or #Spark programming model, Apache Hadoop is able to service most any enterprise workload.

Hadoop supports any data whether structured or unstructured in many different formats making it ideal as a uniform data collection system across the enterprise. By denormalising data into an Enterprise Business Record (EBR), all enterprise data may be text searched and processed through queries and reports. Unstructured data from file servers, email systems, machine logs and social sources is easily ingested and retrieved as well.

2. Data lake

A Hadoop data lake functions as a central repository for data. Data is either transformed as required prior to ingestion or stored “as is,” eliminating the need for heavy extract, transform and load (ETL) processes. Data needed to drive the enterprise may be queried, text searched or staged for further processing by downstream NOSQL analytics or applications and systems. 

Data lakes also significantly reduce the high cost of interface management and data conversion between production systems. Data conversion and interface management may be centralised with a data lake deployed as a data hub to decouple customisations and point to point interfaces from production systems.

http://www.itproportal.com/features/building-a-common-data-platform-for-the-enterprise-on-apache-hadoop/

Unraveling Hadoop and Spark Performance Mysteries

What do you do when your #Spark or #Hive job runs like molasses? If you’re like most end-users who lack in-depth technical skills, the answer is “not much.” Now a startup named #UnravelData is working to show you what’s actually going on in the cluster, and provide some configuration recommendations and automatic fixes as well. “Big data operations is usually considered a black art,” says Kunal Agarwal, the co-founder and CEO of Unravel Data, which came out of stealth mode today three years after its founding. “People don’t usually understand what’s happening in the stack, and this impedes its performance.” Agarwal studied parallel programming and distributed systems while enrolled in Duke University’s computer science program, and got a taste for how complex #Hadoop and #Spark clusters can be. When he left North Carolina to enter the big data space, he looked for ways he could make the biggest impact. His focus quickly turned to the operations side of things, which he terms “Data Ops.” “End users are spending more than half of their day trying to solve these issues and getting productive on the big data stack,” he tells Datanami. “People don’t necessarily understand distributed computing and parallel processing, and getting performance and reliability out of these applications is super hard.”

https://www.datanami.com/2016/09/13/unraveling-hadoop-spark-performance-mysteries/

The Role of Hadoop in Digital Transformations and Managing the IoT

The digital transformation underway at Under Armour is erasing any stale stereotypes that athletes and techies don’t mix. While hardcore runners sporting the company’s latest microthread singlet can’t see #Hadoop, #Apache #Hive, Apache #Spark, or #Presto, these technologies are teaming up to track some serious mileage. Under Armour is working on a “connected fitness” vision that connects body, apparel, activity level, and health. By combining the data from all these sources into an app, consumers will gain a better understanding of their health and fitness, and Under Armour will be able to identify and respond to customer needs more quickly with personalized services and products. The company stores and analyzes data about food and nutrition, recipes, workout activities, music, sleep patterns, purchase histories, and more. Compiling, storing, and analyzing these types of structured and unstructured data at this scale would have been nearly impossible a decade ago. Today, companies can use Hadoop to merge their data from business applications, business analytics, web logs, the Internet of Things (IoT), and many other sources to deliver context-relevant insights. When companies collect data from all sources to augment the core of their business, they often realize real-time business insights that give them a competitive edge.

http://data-informed.com/the-role-of-hadoop-in-digital-transformations-and-managing-the-iot/

SnapLogic Announces Additions to its Library of Connectors

#SnapLogic features a hybrid cloud architecture and a library of more than 400 Snaps. Snaps are pre-built integration components that can be used to connect to a specific data source or application. The new #Hive Snap is a Snap for the #Apache Hive data warehouse. According to the press release, the new Hive Snap "automates execution of Data Manipulation Language (DML) and Data Definition Language (DDL) statements for rapid queries on either #Cloudera- or #Hortonworks -based Hadoop clusters." The new #Teradata Snap is used to connect to a Teradata database in order to perform business analytics. The SnapLogic #Hadooplex is a data processing engine that is deployed on a Hadoop cluster. The SnapLogic Hadooplex has been enhanced so that data processing in Hadoop clusters is more flexible and secure.

http://www.programmableweb.com/news/snaplogic-announces-additions-to-its-library-connectors/brief/2016/08/22

Cloudera makes Navigator Optimizer available, releases Enterprise 5.8

California-based #Cloudera that designs open source technologies and develops apache hadoop-based enterprise analytic data management software, has announced the general availability of Cloudera Navigator Optimizer, alongside the release of Cloudera Enterprise 5.8. Since its limited beta release in November 2015, Cloudera Navigation Optimizer aims to aid customers augment their analytic database or data warehouse solution with #Hadoop and help offload SQL workloads for more efficient analytics processing. Posing as the newest update to Cloudera’s Big Data solutions and offerings, the optimization tool claims to provide database administrators (DBAs) the usage visibility to manage Hadoop data models and guidance to optimize query performance. In a company blog, Cloudera stated that it has helped customers “offload ETL (extract, transform and load) processes, Business Intelligence (BI) reports, ad-hoc workloads to Cloudera’s Hadoop platform,” after profiling and analyzing workloads for customers in identifying potential risks and optimizing data model to support various data needs. Reportedly, it evaluates structured query language (SQL) text in large, complex #SQL workloads so users can gain an in-depth understanding of their workloads, identify queries best-suited for Hadoop and modify them as needed for optimal efficiency via an easy-to-use web user interface. Its built-in optimization guidance for #Apache #Hive and Apache #Impala (incubating), a SQL engine also enables peak in performance for faster data preparation, BI and exploratory analytics, claims the company.

http://techseen.com/2016/07/25/cloudera-navigator-optimizer-enterprise-5-8/

HDInsight Tool in Azure Toolkit for Eclipse Goes GA

#Microsoft has added to its growing list of Eclipse-based product offerings with the general availability release of the #HDInsight Tool for #Eclipse. This cloud implementation on #Azure of the #Apache #Hadoop technology stack is part of the Azure Toolkit for Eclipse, which Microsoft open sourced under the MIT License. It includes implementations of Apache #Spark, #HBase, #Storm, #Pig, #Hive, #Sqoop, #Oozie, #Ambari and other Hadoop tech. It also integrates with Power BI, Excel, SQL Server Analysis Services and SQL Server Reporting Services. The company released a version of the HDInsight Tool for the IntelliJ integrated development environment (IDE) last month. Since that announcement "... we are maintaining our strong momentum to serve the open source community and expanding our support for Eclipse," wrote Jenny Jiang, principal program manager in Microsoft's Big Data Team, in a blog post.

https://adtmag.com/articles/2016/07/13/hdinsight-azure-eclipse.aspx?m=1

JSON in Hadoop and Hive

In this episode of Data Exposed Scott welcomes Rashim Gupta to the show. Rashim, a Senior Program Manager in the SQL Server Big Data group is in the studio today to show us how to query #JSON in #Hadoop using #HIVE and #HiveQL. Rashim begins by first providing an overview of Hadoop and #HDInsight and the scenarios in which you would use these technologies.

 
At the 3:00 mark Rashim discusses the motivation behind the use and benefits of JSON, including the different optimizations and options avaialble when using JSON with Hive.

 

Then at the 4:45 mark Rashim jumps right into the demo and spends some quality time showing the differnces between the the built-in Hive UDFs and the custom SerDe for working with JSON. He shows the peformance differences of each and discusses the scenarios for using each and the recommended method for querying JSON. Awesome demos showing the how easy it is to work with JSON in Hadoop and Hive.

Great insight from Rashim and we look forward to future shows with him!

https://channel9.msdn.com/Shows/Data-Exposed/JSON-in-Hadoop-and-Hive

Big Data Benchmark Gauges Hadoop Platforms

In another indication of a maturing technology and growing demand, an industry group has released a big data analytics benchmark designed to gauge the performance of #Hadoop -based systems.

The Transaction Processing Performance Council said this week its TPCx-BB benchmark for big data analytics systems covers systems such as #MapReduce, #Apache #Hive, Apache #Spark and Machine Learning Library, or #MLib.

According to the TPC website, the “express” benchmark measures the performance of Hadoop-based systems, including hardware and software components. The benchmark executes 30 frequently performed analytical queries in the context of “retailers with physical and online store presence.”

The queries are expressed in SQL for structured data and in machine learning algorithms for semi-structured and unstructured data. SQL queries can use Hive or Spark while machine learning algorithms use MLib along with “user defined functions and a procedural program,” the benchmark group added.

Along with representing the three data types, the new benchmark simulates big data processing, analytics and reporting for the 30 use cases. Runtimes for the big data simulations range from seconds to hours.

The benchmark workload also addresses data set scaling and can run concurrent threads supporting multiple jobs with different characteristics running on a single cluster or via node scaling. The metric supports Hive on MapReduce as well as Hive running on both Spark and Apache Tez, the framework for building high-performance batch and interactive data processing applications.

The benchmark characteristics ultimately provide performance and price metrics for determining the tradeoffs between data analytics performance and cost, the council said.

http://www.datanami.com/2016/06/01/big-data-benchmark-gauges-hadoop-platforms/

What you must know about Hadoop and Spark right now

The biggest thing you need to know about #Hadoop is that it isn’t Hadoop anymore.

Between #Cloudera sometimes swapping out HDFS for Kudu while declaring #Spark the center of its universe (thus replacing #MapReduce everywhere it is found) and Hortonworks joining the Spark party, the only item you can be sure of in a “Hadoop” cluster is YARN. Oh, but Databricks, aka the Spark people, prefer Mesos over YARN -- and by the way, Spark doesn’t require HDFS.

http://www.infoworld.com/article/3019754/application-development/16-things-you-should-know-about-hadoop-and-spark-right-now.html

5 words of advice on going to cloud, from a very busy federal agency

What does the federal agency charged with tracking more than 30 billion financial trades a day know about cloud computing? The Financial Industry Regulatory Authority (FINRA) recently moved its mission-critical systems to the cloud, and is sharing details about how it went. At the recent Strata + #Hadoop World conference, Jaipaul Agonus, technology director at FINRA, provided the six key lessons FINRA learned as it moved into the cloud world.

http://www.zdnet.com/article/migrate-with-caution-a-busy-federal-agency-describes-its-cloud-journey/