Syncsort quality manager aims to purify Hadoop data lakes

#Syncsort Inc. is extending the data quality features of the #TrilliumSoftware Inc. subsidiary it acquired last November to native #Hadoop environments with #TrilliumQuality for #BigData. The offering combines Trillium’s data quality features with its Intelligent Execution data integration platform to enable information technology organizations to normalize and integrate data at the same time. The Trillium platform was previously available in native format only on #Linux, #Unix and Windows operating systems. The Hadoop support is the first time Syncsort has applied its data quality features to applications. Data quality is about identifying inconsistencies, errors or duplication. Examples include a ZIP code entered in a date field or duplicate customer records that appear to be different because of misspellings. Normalizing data is a tricky process. For example, different countries have different address and date formats and two people with the same name in the same ZIP Code may or may not be the same person. Users are rushing to extract data from production systems and load it into analytics engines, but are discovering that quality problems limit their effectiveness. “Everybody is trying to govern the data once it’s in the data lake so it doesn’t turn into a data swamp,” said Tendü Yoğurtçu, Syncsort’s chief technology officer. “The volume and variety of data makes it complex.” Trillium has hundreds of matching algorithms to identify such problems, and can be configured to automatically apply corrective algorithms, Yoğurtçu said. The offering includes address- and name-matching data for 150 countries as well as postal directories and geocoding. Intelligent Execution examines the topology of a data flow and optimizes resources for the job without changes to the application. It supports both new and existing Trillium data quality projects across Hadoop, MapReduce and Apache Spark on-premises or in the cloud. “Once you understand the data you can create the rules to cleanse that data,” Yoğurtçu said. “For example, if you have duplicates you can specify a process to flag them or get rid of them.” Trillium Quality for Big Data is available on all Hadoop distributions including Cloudera Inc.’s CDH, Hortonworks Inc.’s HDP and MapR Technologies Inc.’s Converged Data Platform. It deploys and installs via Cloudera Manager and Apache Ambari. Pricing is on a per-node basis or cloud subscription, but Sync

https://siliconangle.com/blog/2017/09/20/syncsort-quality-manager-aims-purify-hadoop-data-lakes/

Syncsort Delivers Native Mainframe Hadoop, Spark Data

#Syncsort, a provider of big data and mainframe software, has upgraded its DMX-hdata integration software to enable enterprise organizations to work with mainframe data in #Hadoop or #Spark in its native format.

Syncsort delivered the new capabilities because some of its large enterprise customers—particularly those in financial services, banking, insurance and health care—needed to maintain their mainframe data in its native format for compliance purposes, the company said.

Tendu Yogurtcu, general manager of Syncsort's big data business, told eWEEK that while many of Syncsort's large enterprise customers want the scalability and cost benefits of Hadoop and Spark for their mainframe data, converting that data for the big data platforms presents compliance challenges because they are required to preserve the data in its original EBCDICformat.

http://mobile.eweek.com/database/syncsort-delivers-native-mainframe-hadoop-spark-data.html

New Hadoop Survey Identifies Big Data Trends to Watch in 2016

WOODCLIFF LAKE, N.J., Jan. 19, 2016/PRNewswire/ -- #Syncsort, a global leader in #BigData and mainframe software, today announced the results from its second annual #Hadoop survey, showing that as more organizations are moving from Hadoop experimentation to production, realizing the full potential of big data analytics, there are three top areas they will focus on in 2016.

Syncsort polled over 250 respondents including data architects, IT managers, developers, business intelligence/data analysts and data scientists, with a majority (66 percent) coming from organizations with revenues over $100 million. Participants were from a broad range of industries including financial services, healthcare, government and retail.

Based on the survey results, there are three key trends Syncsort anticipates in 2016:

1. Apache Spark will move from a talking point into deployment. Nearly 70 percent of respondents are most interested in Apache Spark, surpassing interest in all other compute frameworks, including the recognized incumbent, MapReduce (55 percent). While Syncsort expects MapReduce will still be the prevalent compute framework in production, the high level of interest should translate into more Spark deployments, mostly running on Hadoop.

2. Offloading from expensive platforms into Hadoop will continue to increase in numbers and scope. 63 percent of respondents feel Hadoop will help them increase business/IT agility, 55 percent expect to increase operational efficiency and reduce costs, and over 51 percent want to leverage it to make more data available for business use across the entire organization. These findings are consistent with Syncsort customer use cases that should continue to gain steam in 2016, including Mainframe and Enterprise Data Warehouse (EDW) offload to Hadoop. 

3. A growing number of companies will look to leverage Hadoop for advanced use cases. More than half of respondents see Hadoop as a way to innovate, using data from social media and IoT, and applying predictive analytics and visualization for greater insights about their business. Hadoop is yet to be leveraged for mobile apps and software, as only 4.9 percent reported utility for these use cases.

"As Hadoop adoption becomes mainstream, the number of applications in production increases and the use cases, frameworks and data sources become more varied and complex. Organizations realize significant benefits from Hadoop; however, they also cite challenges in keeping up with new tools and skills, connectivity and data movement, and unforeseen costs," said Tendü Yoğurtçu, General Manager of Syncsort's Big Data business. "A single software environment to access all enterprise data and manage the entire data pipeline will be critical for organizations to maximize the ROI on their Big Data projects, especially as the demand for real-time analytics in industries such as financial services, healthcare, telecommunications, and retail increases."

Based on additional customer feedback, Syncsort predicts two additional trends in 2016:

4. More organizations will leverage streaming, real-time data sources. The best business decisions often require the most recent data available. Popular use cases include fraud detection, analytics on telemetry and security data, insurance claim validation, and the IoT.

5. Data governance and security will be major areas of focus as organizations move to production deployments. More organizations will move towards adopting a "Hadoop first" approach to data management – skipping traditional and more expensive platforms and applying metadata, lineage, security, and other data management measures on Hadoop from the start.

"The ability to combine real-time data sources with batch data will create even more insights for businesses, and predictive analytics will play a critical role in this," continued Yoğurtçu. "The ability to transform and prepare data in flight will be more important, eliminating the need for staging increasing volumes of data. Though challenging, this will also create an opportunity to deliver next generation data integration products, future proofing user's applications while taking advantage of highly scalable and distributed platforms like Apache Hadoop and Spark, on-premise or in the cloud."

For more information on the Hadoop study,click here.

About Syncsort 
Syncsort provides enterprise software that allows organizations to collect, integrate, sort and distribute more data in less time, with fewer resources and lower costs.  Thousands of customers in more than 85 countries, including 87 of the Fortune 100 companies, use our fast and secure software to optimize and offload data processing workloads. Syncsort software provides specialized solutions spanning "Big Iron to Big Data," including next gen analytical platforms such as Hadoop, cloud, and Splunk. For more than 40 years customers have turned to Syncsort's software and expertise to dramatically improve performance of their data processing environments, while reducing hardware and labor costs.

http://www.prnewswire.com/news-releases/new-hadoop-survey-identifies-big-data-trends-to-watch-in-2016-300205457.html

Q&A: Why Syncsort introduced the mainframe to Hadoop

When you think of leaders in big data and analytics, you’d be forgiven for not listing #Syncsort among them. But this nearly 50-year-old company, which began selling software for the decidedly unglamorous job of optimizing mainframe sorting, has refashioned itself into a critical conduit by which core corporate data flows into #Hadoop and other key big data platforms. Syncsort labels itself "a freedom fighter" liberating data and dollars -- sometimes millions of dollars -- from the stranglehold of big iron and traditional data warehouse/analytics systems.

In this installment of the IDG CEO Interview Series, Chief Content Officer John Gallant spoke with Josh Rogers, who was named CEO this week, as well as outgoing CEO Lonne Jaffe, who remains as Senior Advisor to Syncsort’s board. Among other topics, the pair talked about why Syncsort was recently acquired by Clearlake Capital Group, and how Syncsort’s close partnership with Splunk is dramatically improving security and application performance management.

http://www.infoworld.com/article/3020532/security/qa-why-synscort-introduced-mainframe-to-hadoop.html

14 Big Data acquisitions and why they happened

France-based BI startup #BIME Analytics announced Tuesday its acquisition by cloud-based customer service platform vendor #Zendesk. A day later, #Syncsort, a company focused on the integration of #BigData and mainframe technologies, announced its own acquisition by #ClearlakeCapital. And just a week earlier, #IBM announced that it would be acquiring Big Data storage vendor #Cleversafe.

http://www.zdnet.com/article/14-big-data-acquisitions-and-why-they-happened/