Hadoop Servers Expose Over 5 Petabytes of Data

Improperly configured #HDFS -based servers, mostly #Hadoop installs, are exposing over five petabytes of information, according to John Matherly, founder of Shodan, a search engine for discovering Internet-connected devices. The expert says he discovered 4,487 instances of HDFS-based servers available via public IP addresses and without authentication, which in total exposed over 5,120 TB of data. According to Matherly, 47,820 #MongoDB servers exposed only 25 TB of data. To put things in perspective, HDFS servers leak 200 times more data compared to MongoDB servers, which are ten times more prevalent. A report from Binary Edge from 2015 revealed that at the time, Redis, MongoDB, Memcached, and ElasticSearch servers put together exposed a tota of only 1.1 PB of data.

https://www.bleepingcomputer.com/news/security/hadoop-servers-expose-over-5-petabytes-of-data/

Committers Talk Hadoop 3 at Apache Big Data

The upcoming delivery of #Apache #Hadoop 3 later this year will bring big changes to how customers store and process data on clusters. Here at the annual Apache #BigData show in Miami, Florida, a pair of Hadoop project committers from #Cloudera shared details on how the changes will impact #YARN and #HDFS. The biggest change coming to HDFS with Hadoop 3 is the addition of erasure coding, says Cloudera engineer Andrew Wang, who is the Hadoop 3 release manager for the Apache Hadoop project at the Apache Software Foundation. HDFS historically has replicated each piece of data three times to ensure reliability and durability. However, all those replicas come at a big cost to customers, Wang says. “Many clusters are HDFS-capacity bound, which means that they’re always adding more nodes to clusters, not for CPU or more processing, but just to store more data,” he tells Datanami. “That means this 3x replication overhead is very substantial from a cost point-of-view.” The Apache Hadoop community considered the problem, and decided to pursue erasure coding, a data-striping method similar to RAID 5 or 6 that has historically been used in object storage systems. It’s a technology we first told you was coming to Hadoop 3 exactly one year ago, during last year’s Apache Big Data shindig. “The benefit of using a scheme like erasure coding is you can gain much better storage efficiency,” Wang says. “So instead of paying a 3x cost, you’re paying a 1.5x cost. So you’re saving 50% compared to the 3x replication, when you look at purely disk expenditure. Many of our Hadoop customers are storage bound, so being able to save them half their money in hard disk cost is pretty huge.”

https://www.datanami.com/2017/05/18/committers-talk-hadoop-3-apache-big-data/

Streaming Live Data and the Hadoop Ecosystem

Oleg Zhurakousky discusses the #Hadoop ecosystem – Hadoop, #HDFS, #Yarn -, and how projects such as #Hive, #Atlas, #NiFi interact and integrate to support the variety of data used for analytics.

https://www.infoq.com/presentations/streaming-hadoop

MongoDB, ElasticSearch hackers now target Hadoop with ransomware

Following recent cyber attacks on #MongoDB and #ElasticSearch, hackers are now targeting Internet-facing #Hadoop Distributed File System (HDFS) installations. As with the attacks on MongoDB and ElasticSearch, hackers are holding databases for ransom and, in many reported cases, simply deleting the data. It has now been confirmed by Fidelis Cybersecurity Threat Research that these sort of attacks are happening on HDFS instances, with the company estimating that the potential exposure of this attack is around 8,000-10,000 HDFS installations worldwide. In one incident, Fidelis observed an attacker erasing most of the directories and creating a single directory called “NODATA4U_SECUREYOURSHIT”. There was no attempt to claim a ransom or any other communication — the data was simply deleted and the directory name was left as a calling card. Further investigation saw a core issue similar to MongoDB, namely the default configuration can allow “access without authentication.”
http://www.cbronline.com/news/cybersecurity/breaches/mongodb-elasticsearch-hackers-now-target-hadoop-ransomware/

Tips for populating Big SQL and Hive Hadoop tables with DATE types

When creating external #Hive tables defined with DATE columns, ensure that the values in the data files on #HDFS ( #Hadoop ) correspond to DATE values and not a mix of DATE and TIMESTAMP values. The same is true for when creating Hive tables and using the Hive INSERT or INSERT…SELECT commands to add data to tables. When Hive expects a DATE type, but instead finds a TIMESTAMP type in the data file, then a NULL value is inserted to the table. NULL values can have a negative impact on query performance especially for queries performed against partitioned tables where the partitioning keys are NULL values. This is because Hive will put all NULL values into one partition.

https://developer.ibm.com/hadoop/2016/12/16/tips-for-populating-big-sql-and-hive-hadoop-tables-with-date-types/