No one likes waiting those long milliseconds for their data to thaw out of an archival storage system. Then again, no one likes paying high-performance prices for long-term storage either. One solution is a streaming storage system that combines low latency with High Throughput File System storage, according to Tom Kaitchuck (pictured), senior consultant software engineer at #DellEMC. “The difference between what we’re doing and #Kafka fundamentally comes down to the model,” said Kaitchuck, who spoke with George Gilbert (@ggilbert41), co-host of theCUBE, SiliconANGLE’s mobile live-streaming studio. (*Disclosure below.) Kaitchuck joined theCUBE at the Flink Forward conference last week in San Francisco to talk about Pravega, a new open-source stream storage system that Dell EMC designed and built from the ground up for modern-day stream processors like Apache Flink, an open-source stream processing framework. Streaming to storage Kaitchuck outlined how Pravega differs from Apache Kafka, explaining that Kafka is a messaging system, while Pravega is a streaming system. When a client sends bytes over the wire, Pravega doesn’t interpret them at all. This allows the system to channel that data into a long-term archival system. By comparison, Kafka systems don’t share that capability, meaning companies must turn to other solutions for their storage needs, according to Kaitchuck. The advantage for Pravega lies in being able to stream data and send it into durable storage with one package. The project’s goal is to get very low latency, Kaitchuck stated. Companies can’t stream off HTFS because the latency is too high. Pravega offers a system with low latency, while also ultimately storing data in HTFS. It does this by aggregating a bunch of little writes into one or two big writes. Kaitchuck also mentioned applications, saying anything that requires strong consistency would be a good fit for Pravega. The system also handles tasks that require transactional semantics, he added. Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of Flink Forward 2017. (*Disclosure: TheCUBE is a paid media partner at Flink Forward. The conference sponsor, data Artisans, does not have editorial oversight of content on theCUBE or SiliconANGLE.)
No comments:
Post a Comment