Container storage: Docker in flash, HCI and software-defined

We found the big storage players all have @Docker Volume Plugins for their products and some are building out container storage management platforms. But what about other storage players – the flash specialists, the #softwaredefinedstorage makers, and the #hyperconverged infrastructure suppliers? To run containers – usually Docker, which has emerged as a de facto standard – you do not have to provision persistent storage because any capacity required lives and dies with the container. But that hasn’t been sufficient for many use cases to which containers have been put, so efforts have been made to develop ways of providing persistent storage for Docker. That is, storage that remains once the container has been spun down, which is needed for all sorts of reasons in enterprise scenarios. To provide persistent storage for Docker means, at the least, having Docker Volume plugins available, APIs (application programming interfaces) written to work with the storage product in question. It could also mean having integration with higher-level containers management platforms, such as Docker @Swarm, @Kubernetes or @Mesosphere Marathon. Efforts are being made to develop standardised ways to integrate plugins/APIs with these platforms, such as with the Container Storage Interface. All the biggest storage suppliers – @DellEMC, @HPE, @Hitachi Vantara, @IBM and @NetApp – have container storage covered by at least Docker Volume Plugins. Read more on containers and storage We look at what the big five storage array makers – Dell EMC, Hitachi, HPE, IBM and NetApp – are doing to provide persistent storage for container-based deployments. Containers such as those from Docker are agile, lightweight, and can be short-lived, but they and their data often need to be protected. We look at the key options available. Outside the top five, the picture is more patchy, as you might expect. On the one hand, Pure Storage, one of the all-flash array market leaders, provides plugins as well as its own software that integrates with the leading orchestration platforms. Other all-flash array, hyper-converged and software-defined storage makers offer Docker Volume Plugins. And yet others go a bit further with, for example, Kubernetes integration or products aimed at container environments. Meanwhile, some mid-range storage array makers appear not to address container storage at all, but presumably containers are just not in the plans of their customer base. Flash specialists’ container capabilities Kaminario All-flash array maker Kaminario has Docker plugins for its hardware via the Kaminario K2 Flocker Plugin. Pure Storage Part of the leading group in the all-flash array market, Pure Storage has recently added to its container storage capabilities with the release of Pure Service Orchestrator. This allows storage-as-a-service-style management of container storage in Pure FlashArray and FlashBlade arrays to provide persistent storage for Docker containers, with on-demand provisioning based on policies. Pure Service Orchestrator integrates with Kubernetes and Mesosphere Marathon container environments. Pure also has Docker storage plugins for FlashArray and FlashBlade that allow for persistent storage on those arrays. Western Digital WD recently introduced Docker container support “for select customers” in its ActiveScale 5.3 Object Storage System. ActiveScale is an archiving product that uses the S3 protocol to access a single namespace cluster that can scale up to tens of petabytes. WD also has its Intelliflash arrays, taken on when it acquired Tegile in 2017, but doesn’t appear to have addressed container storage capability in that line of products. Software-defined storage sellers Datacore Datacore doesn’t seem to have Docker Volume Plugins, but does offer a Kubernetes PersistentVolume API to orchestrate creation and operation of containerised operations and to integrate storage into Kubernetes, as well as working with Docker Swarm. Hedvig This software-defined storage maker offers Docker support via Docker Volume Plugins to provide persistent storage. Nexenta Nexenta has Docker Volume Plugins for its NexentaStor and NexentaEdge block, file and object storage software products. It also offers a NexentaEdge DevOps Edition, which is a small-scale free product that allows developers to experiment. It allows containerised applications access to persistent storage. Veritas Veritas offers its Hyperscale for Containers platform which delivers container workload deployment across software-defined storage to create a kind of hyper-converged infrastructure platform for containers. It delivers a so-called “intelligent workload deployment” model that aims to provide quality of service for container storage. Container capabilities in hyper-converged infrastructure Cisco Cisco’s Springpath hyper-converged products run Microsoft Hyper-V hypervisor and it offers Docker Volume Plugins for them. Nutanix This hyper-converged pioneer offers a Docker Volume Plugin that allows customers to attach persistent storage to Docker volumes created in its Acropolis hypervisor.

https://www.computerweekly.com/feature/Container-storage-Docker-in-flash-HCI-and-software-defined

Data protection for containers: Why, and how to do Docker backup

Containers have been around for many years, but the use of container technology has been popularised in the last five years by Docker. The Docker platform provides a framework to create, configure and launch applications in a much simpler way than in the native features of the Linux and Windows operating systems on which they run. An application is a set of binary files that run on top of an operating system. The application makes calls via the operating system to read and write data to persistent storage or to respond to requests from across the network. Over the past 15 years, the typical method of application deployment has been to run applications within a virtual machine (VM). VMs take effort to build and manage. They need patching and have to be upgraded. Virtual machines can attract licensing charges, such as operating system licences and application licences per VM, so have to be managed efficiently. Containers provide a much more lightweight way to run applications. Rather than dedicate an entire VM for each application, containers allow multiple applications to run on the same operating system instance, and these are isolated from each other by segregating the set of processes that make up each application. Containers were designed to run microservices, be short-lived and not require persistent storage. Data resiliency was meant to be handled by the application, but in practice, this has proved impractical. As a result, containers can now be easily launched with persistent storage volumes or made to work with other forms of shared storage.  Container data protection A container is started from a container image that contains the binary files needed to run the application. At launch, time parameters can be passed to the container to configure components such as databases or network ports. This includes attaching persistent data volumes to the container or mapping file shares. In the world of virtual machines, the VM and the data are backed up. Backup of a virtual machine is for convenience and other potential uses. So, for example, if the VM is corrupted or individual files are deleted they can be recovered. Alternatively, the whole VM and its data can be brought back quickly. In practice though, with a well configured system, it may be quicker to rebuild the VM from a gold master and configure it using automation or scripts. With containers, rebuilding the application from code is even quicker, making it unnecessary to backup the container itself. In fact, because of the way containers are started by platforms such as Docker, the effort to recover a container backup would probably be much greater than simply restarting a new container image. The platform simply isn’t designed to recover pre-existing containers. So, while a running container instance doesn’t need to be backed up, the base image and configuration data does. Without this the application can’t be restarted. Equally, this applies to implementing a disaster recovery strategy. Restarting an application elsewhere (eg, in the public cloud or another datacentre) also needs access to the container image and runtime configuration. These components need to be highly available and replicated or accessible across locations. 

https://www.computerweekly.com/feature/Data-protection-for-containers-Why-and-how-to-do-Docker-backup

Cumulus Networks Bolsters Cumulus NetQ with Kubernetes Integration to Provide Network Operators with Actionable Insight into Container Networking

MOUNTAIN VIEW, Calif.--(BUSINESS WIRE)--@Cumulus Networks, the leaders in building web-scale networks for the digital era with open and modern solutions, today announced @Kubernetes support with Cumulus #NetQ, a network operations tool that provides visibility and intelligence into the health of the entire network. The support for Kubernetes builds on existing integration with @Docker Swarm to enhance visibility, streamline troubleshooting, and help network operators avoid costly downtime in container environments. The company also introduced a new global leader of engineering, @Partho Mishra, and the appointment of @Shrijeet Mukherjee as Chief Architect. Designed to work hand in hand with Cumulus Linux, Cumulus NetQ provides actionable insight into every trace and hop in the Linux-based data center—from the container, virtual machine, or host, all the way to the switch and port. The metadata it collects across these elements ensures visibility and intelligence into the health of the network, making sure everything is behaving as intended. Cumulus NetQ 1.3, available now, adds direct integration with Kubernetes, the popular container orchestration engine in use by 71% of enterprises.1 This release also supports the Container Networking Interface (CNI) ecosystem, specifically across the two most popular CNIs of Calico and Flannel. Both CNIs are maintained by Tigera, which has partnered with Cumulus on the integration. Enterprise adoption of Linux-based containers is increasing rapidly, due to containers’ ability to dramatically improve flexibility when running cloud-native applications on a physical or virtual infrastructure. 451 Research predicts2 containers will grow to become a $2.7 billion market by 2020; 3.5 times greater than the $762 million container market in 2016. Yet with all their benefits for speed and efficiency, containers can cause major challenges for network operators. Because containers are ephemeral, the traffic patterns are constantly changing, so being able to view history and changes to the network becomes more important than ever. According to Gartner, multihost networking is important because “the portability and ephemeral life cycle of containers will overwhelm the traditional networking stack. The native container networking stack doesn't have robust access and policy management capabilities.”

https://www.businesswire.com/news/home/20180418005410/en/Cumulus-Networks-Bolsters-Cumulus-NetQ-Kubernetes-Integration

Singularity Containers for HPC & Deep Learning

#Containerization as a concept of isolating application processes while sharing the same operating system (OS) kernel has been around since the beginning of this century. It started its journey from as early as Jails from the @FreeBSD era. Jails heavily leveraged the chroot environment but expanded capabilities to include a virtualized path to other system attributes such as storage, interconnects and users. #SolarisZones and #AIX Workload Partitions also fall into a similar category. Since then, the advent and advancement in technologies such as cgroups, systemd and user-namespaces greatly improved the security and isolation of containers when compared to their initial implementations. The next step was to abstract the complexity involved in containerization by adding user-friendly features such as templates, libraries and language bindings. LXC, an OS container meant to run multiple services, accomplished this task. In the late 2000s, @Docker, an application container that was meant to run a single service, took it one step further by creating an entire ecosystem of developer tools and registries for container images, improving the user experience of containers and making containerization technology accessible to novice Linux users. This led to widespread adoption. The evolving nature of this domain led to the creation of a standardization process called Open Container Initiative in 2015. Containerization drastically improves the “time to results” metric of many applications by eliminating several roadblocks in the path to production. These roadblocks arise from issues pertaining to portability, reproducibility, dependency hell, isolation, configurability and security. For example, applications in fast-paced growth areas, such as high performance computing (HPC) which include verticals and workloads like molecular dynamics, computational fluid dynamics, MIMD Lattice Computation codes, life sciences and deep learning (DL), are very complicated to build and run optimally, due to very frequent updates to the application codes, its dependencies and supporting libraries. Container images include the applications themselves and their development environments, which aids developers in porting their applications from their desktops to datacenters. This also helps with version control, making it easier to understand which versions of the software package and dependencies lead to a particular result. This, in turn, helps to manage dependency hell, which refers to the entire application failing if one of its dependencies fails. Another real-world example is collaboration between researchers, where containers help with reproducibility of the results. In short, instead of spending time debugging the software environment, researchers can focus on the science. Containers are orders of magnitude faster to spin up than virtual machines (VM) since they do not have a hypervisor providing a virtual hardware layer and a guest OS on top of the host OS as an independent entity. For a majority of HPC and DL applications, an un-tuned full VM may experience performance degradation when compared to bare metal. The overhead and performance degradation of a container when compared to an equivalent fully subscribed bare metal workload performance is nonexistent. All of these features have promoted the deep proliferation of container technologies.

https://www.nextplatform.com/2018/03/19/singularity-containers-for-hpc-deep-learning/

​Former Docker CEO takes over Storj Labs' top spot

Top executives who get open source are rare and far between. So it is that @StorjLabs, the leader in decentralized cloud storage, must be pleased as punch to announce that @BenGolub, former CEO and co-founder of @Docker, is now Storj's executive chairman and interim CEO. Storj Labs' program is Storj. This is an open-source distributed, encrypted, and fast object storage. It's based on #blockchain and peer-to-peer protocols to provide secure, private cloud storage. This isn't a new idea. Other peer-to-peer cloud storage products include Resilo's @BitTorrent Sync, @Tresorit and, more recently, @ProtocolLabs' @FileCoin, which is also taking a blockchain approach to shared storage.

http://www.zdnet.com/article/former-docker-ceo-takes-over-storj-labs-top-spot/

IBM Java CTO: Devs shouldn't have to learn Docker, K8s, 30 other things to deploy an app

Index At @IBM 's Index developer conference in San Francisco, on Tuesday, @The Register sat down with Big Blue's @Java CTO @John Duimovich to talk about the Java programming language, IBM, the cloud and other developer-oriented concerns. Duimovich made the case for IBM as a cloud platform partner, based on the company's Java expertise. He also argued for Java – 22 years old – as a vital, evolving language. Oracle, Java's meddling uncle, last year said it would hand over governance of Java EE to the Eclipse Foundation. For enterprise Java, that means a chance to accelerate the development and deployment of new features and to become more responsive to the needs of corporate developers. "What they did is they gave control of the evolution of enterprise Java to a foundation," Duimovich explained. "They moved the code itself and the governance. And I think that was in part that acknowledgement of some of the momentum that MicroProfile has had." MicroProfile is an initiative to make Java EE – or Jakarta EE, assuming the name change vote goes as expected – more suitable for the development of lightweight, composable microservices, as opposed to large monolithic applications. "This is where the compatible future of cloud Java is being defined," he said. "You're going to see EE evolve to adopt many of these new technologies." We sent a vulture to IBM's new developer conference to find an answer to the burning question: Why Big Blue? READ MORE IBM, said Duimovich, is interested providing support for any language that enough developers want. He said the company sees quite a bit of interest in Node.js, particularly for front-end applications, and more modest interest in Swift now that its ecosystem has become more mature. "If you have any serious performance needs, Java's your story," he said. "If you want to integrate with all the systems you're going to have to integrate with terms of an enterprise, no question." Duimovich contends enterprise Java is "actually a pretty interesting space" these days, pointing to developments like reactive programming, where you're dealing with asynchronous code and high event rates for IoT applications. "Enterprise Java at the Eclipse Foundation is going to drive new features more quickly," he said. "And I think it will also be a more inclusive community that's easier to join." For IBM, that's an opportunity to build relationships with the open source development community. "Now that IBM is open source, we have a much greater chance to actually engage with developers," he said. Where once IBM kept its cards close to its vest, now as developer of open source technology, there's far more contact with the development community, which has the potential to translate into new customers. "This excites me because I've much better contact points, a broader set of developers," he said. "People who may not be IBM customers still are able to interact with us and work with us and then, and hopefully from our perspective, turn into IBM customers because they can buy support and other things." Embracing open source means opportunities for technical interaction between IBM personnel and outside developers. Duimovich recounts how IBM gave a university course in how to use the OMR technology to build a virtual machine called Base9. "We got six people instantly who want to work with us because they knew we were doing this," he said, adding that being involved in open source means a better reception from developers. Obscure bug reports Open sourcing its JVM hasn't translated into a significant increase in community code contributions yet, but Duimovich voiced appreciation for the obscure bug reports where someone will fix some edge case with a line or two of code. Seeing Java EE develop at a more rapid pace has pluses and minuses, Duimovich said. "Developers who are waiting for language features and other stuff won't have to wait three years," he explained, pointing to aspects of Kotlin, Scala, or other languages that may be incorporated into Java EE sooner rather than later. "Java this year got a shell, a read-eval-print loop (REPL)," he said. "Every other scripting language had that. Java is finally there." The downside, he said, is that everyone has to move faster just to keep up. "It's a great time to be a Java developer," he said, "and a VM developer," adding that competition between virtual machines benefits everyone. Duimovich sees IBM playing a starring role in the cloud optimization of Java. As an example he pointed to VM startup time. A 40-second startup time didn't matter as much when app servers stayed up for months, but you can't have that for containers. "So we've made both our startup time and ramp up time over 2x better than existing JVMs and we're continuing to focus on that," he said. IBM's Java VM has also been optimized to return memory when it's not being used. "Our VMs will automatically give the system back memory when we go idle," explained Duimovich. Reducing a system's memory footprint in the cloud translates to reducing the system's cost. That's not so much of an issue with JVMs that are maxed out all the time. "But I can tell you from our mainframes and from our deployments, we've seen there's a lot of idle VMs that wake up once in a while," he said. The most pressing technical issue for Java and containers, said Duimovich, is having Java memory adapt to container sizes, because if you exceed that, it kills the container. Low level "You want them to dance as one, so to speak," he explained. "If you reduce the memory in the container, you reduce the memory in Java. Those are the kinds of things we're working on at the low level." In its effort to reduce the footprint of Java, IBM is also working on moving its JIT compiler outside of the address space and have it as a service. "That opens up a huge bunch of opportunities for us," said Duimovich, noting that each instance saves 200 to 400 MB of footprint, which is what you want for microservices and moves Java closer to the memory requirements of other languages. And as a service, he said, it has more visibility across instances, which provides the opportunity to compare performance data in A/B testing. "The easy way to think about it is if I'm bringing up 100 or 1000 microservices, did every one of them need to compile Object?" he said. "Clearly not. And you can get that with AOT – ahead-of-time compilation static – but then you've giving up some of the things Java can do like dynamic bytecode modification." Beyond @Java, Duimovich expressed interest in making development easier overall by making it more dev and less ops. "The notion that as a developer you'll have to learn @Docker, @Kubernetes, and 30 other things before you can even deploy an app is something I'd like to get rid of," he said. Maybe there's a cloud for that. ®

https://www.theregister.co.uk/2018/02/22/ibm_java_cto_john_duimovich_interview/

Sylabs launches Singularity Pro, a container platform for high-performance computing

@Sylabs, the commercial company behind the open source #Singularity #container engine, announced its first commercial product today, #SingularityPro. Sylabs was launched in 2015 to create a container platform specifically designed for scientific and high performance computing use cases, two areas that founder and CEO @Gregory Kurtzer, says were left behind in the containerization movement over the last several years. (For an explanation of containers, see this article.) @Docker emerged as the container of engine of choice for developers, but Kurtzer says the container solutions developed early on focused on #microservices. He says there’s nothing inherently wrong with that, but it left out some types of computing that relied on processing jobs instead of services, specifically high performance computing. Kurtzer, who didn’t exactly just fall off the open source turnip truck, had more than 20 years of experience as a high performance computing architect working at the US Department of Energy Lab, where he founded CentOS, an open source enterprise Linux project and Warewulf, which he says has become the most utilized stateless HPC cluster provisioner. He decided to shift his attention to containers when founded Sylabs and launched the first open source version of Singularity in April, 2016. Even then, he had a vision of creating a commercial version of the product. He saw Singularity as a Docker for HPC environments, and would run his company in a similar fashion to Docker, leading with the open source project, then building a commercial business on top of it — just as Docker had done. Kurtzer now wants to bring Singularity to the enterprise with a focus not just on the HPC commercial market, but other high performance computing workloads such as artificial intelligence, machine learning, deep learning and advanced analytics. “These applications carry data-intensive workloads that demand HPC-like resources, and as more companies leverage data to support their businesses, the need to properly containerize and support those workflows has grown substantially,” Kurtzer wrote in a blog post announcing the enterprise product. Even though Singularity is designed to handle different kinds of workloads, it still works with container orchestration tools, specifically Kubernetes and Mesos, and it is also compatible with Microsoft’s Azure Batch tool and other cloud tools. Kurtzer indicated Sylabs currently has 12 employees, and is operating on an undisclosed amount of seed money. It was funded by RStor, a startup itself currently operating in stealth mode.
https://techcrunch.com/2018/02/08/sylabs-launches-singularity-pro-a-container-platform-for-high-performance-computing/?ncid=mobilenavtrend

Serverless Framework for Real-Time Apps Emerges

Serverless computing can advance real-time applications by providing developers with access to a framework employing an event-driven architecture. Serverless computing has a lot of potential to advance real-time applications by providing developers with access to a framework employing an event-driven architecture that makes IT infrastructure resources seamlessly available on demand. The trouble is that the existing serverless computing frameworks are too slow to support real-time applications. At the Kubecon + CloudNativeCon 2017 conference this week iguazio moved to address that issue by making available a high-speed serverless computing framework as an open source project that it developed in support of the company’s real-time analytics database See also: Why putting #IoT into @Docker containers will unlock it Dubbed nuclio and written in the #Go programming language, iguazio CTO @Yaron Haviv says this #serverlesscomputing framework is unique because it allows developers to employ function within the context of a Docker container running on a @Kubernetes cluster or a software development kit ( #SDK ) that iguazio developed. That approach differs considerably from a proprietary serverless framework such as the @Lambda service developed by @Amazon Web Services ( #AWS), which Haviv also notes requires organizations to also consume a raft of complimentary AWS compute and storage services to work. Supported programing language include supports Golang and Python, with support for Java and Node.js coming soon. Haviv says the high-speed function-as-a-service (FaaS) layer of software iguazio developed for nuclio makes it possible for real-time applications to invoke IT infrastructure in parallel on a truly elastic basis. “Nuclio uses Flash directly,” says Haviv. “In an on-premises environment, it can fit in a 2u box.” In addition, Haviv says nuclio includes tools to enable simple debugging, regression testing and even a multi-versioned continuous integration/continuous development (CI/CD) pipeline to drive DevOps processes. Function images can also be stored in a shared repository.

https://www.rtinsights.com/serverless-framework-for-real-time-apps-emerges/

USING DOCKER TO SCALE OPERATIONAL INTELLIGENCE AT SPLUNK

USING DOCKER TO SCALE OPERATIONAL INTELLIGENCE AT SPLUNK By @JennyFong January 2, 2018

34 22 0 0 14 0 Docker Customers, Docker EE, Docker Enterprise Edition, scalability, Splunk

@Splunk wants to make #machinedata accessible, usable and valuable to everyone. With over 14,000 customers in 110 countries, providing the best software for #visualizing machine data involves hours and hours of testing against multiple supported platforms and various configurations. For Mike Dickey, Sr. Director in charge of engineering infrastructure at Splunk, the challenge was that 13 different engineering teams in California and Shanghai had contributed to test infrastructure sprawl, with hundreds of different projects and plans that were all being managed manually. At DockerCon Europe, Mike and Harish Jayakumar, Docker Solutions Engineer, shared how Splunk leveraged Docker Enterprise Edition (Docker EE) to dramatically improve build and deployment times on their test infrastructure, converge on a unified Continuous Integration (CI) workflow, and how they’ve now grown to 600 bare-metal servers deploying tens of thousands of Docker containers per day. You can watch the entire session here:

https://blog.docker.com/2018/01/docker-splunk/

The Evolution and Future of Hypervisors

#Hypervisors are a #virtualization technique that powers #cloudcomputing infrastructure like @Amazon #EC2 and @Google #ComputeEngine. Although #container virtualization technology like @Docker and @Kubernetes have taken the spotlight recently, #containers are often deployed on top of hypervisors on the cloud. In this article, we will first outline the architecture of classical trap-and-emulate hypervisors that were invented in the 1970s. We then describe how hypervisors evolved from the 1970s to today’s cloud computing era. Finally, we will look at future trends that affect hypervisor design. (This article was inspired by an awesome talk on Amazon’s Nitro project by Anthony Liguori, which I highly recommend everyone interested in hypervisors and cloud infrastructure to watch.) Architecture A hypervisor is a piece of system software that provides virtual machines (VMs), which users can use to run their OS and applications on. The hypervisor provides isolation between VMs, which run independent of each other, and also allows different VMs to run their own OS. Like other virtualization techniques, hypervisors provide multitenancy, which simplifies machine provision and administration. One of the main criticisms against hypervisors is that they tend to usually be heavy-weight compared to other virtualization techniques like containers (Morabito et al., 2015). However, it’s also possible to build hypervisors that are light-weight (Manco et al., 2017) and also make the guest OS more light-weight when running in under a hypervisor (Madhavapeddy, 2013). A hypervisor can be decomposed into two major parts: the virtual machine monitor (VMM) and the device model. The VMM is responsible for setting up VMs and handling traps (a.k.a VM exits) caused by the guest OS executing privileged instructions like I/O access. The device model, on the other hand, is responsible for implementing I/O interfaces for all the devices like networking cards, storage, and so on, the hypervisor supports. Hypervisor architecture is illustrated in the following diagram. Hypervisor architecture. The hypervisor can be decomposed into two parts: the virtual machine monitor (VMM) and the device model. (The terms hypervisor and VMM are often used interchangeably. However, we refer to hypervisor as the combination of a VMM and a device model.) Virtual machine monitor (VMM) A VMM must satisfy three properties (Popek and Goldberg, 1973): Equivalence property states that program execution has identical observable behavior on bare metal and under VMM, except for timing and resource availability, which are difficult to preserve because of shared physical hardware. Efficiency property states that the majority of program instructions are executed directly on a physical CPU without interference from the hypervisor. Resource control property states that the VMM manages all hardware resources. Virtual machines require permission from the hypervisor to directly access hardware. As a side note, it’s worth noting that emulators satisfy both equivalence and resource control properties, but does not satisfy the efficiency property. The KVM subsystem in the Linux kernel (and other OS’es it has been ported to), for example, provides the building blocks for implementing a VMM. The KVM subsystem is effectively a portable abstraction over CPU hardware virtualization capabilities, which can be leveraged by userspace applications like QEMU to implement a VMM or a full hypervisor. Device Model The device model is the part of a hypervisor, which provides I/O interfaces for virtual machines. While the VMM is responsible for handling traps, it delegates I/O requests to the appropriate device model. Examples of device models are virtualized NICs and storage devices. Device models can either provide interface for a real hardware device or a paravirtualized device. The device model can be implemented either using software, like the virtio family of drivers, or in hardware, using SR-IOV, for example. I/O Virtualization To implement a device model, I/O virtualization is needed. The two approaches for I/O virtualization are software-based and hardware-assisted. Software-based I/O virtualization implements I/O interfaces in software to allow sharing the same physical devices across multiple virtual machines. Software-based I/O virtualization can be implemented on top of various different backends. For example, a software-based storage device can be layered on top of a block device or a filesystem. One issue with software-based approach is that the device model uses the same CPU resources that the vCPUs, which reduces available CPU capacity and causes jitter. Hardware-assisted I/O virtualization implements I/O interfaces in hardware. This approach requires hardware support for sharing the same physical device across multiple virtual machines. SRV-IO is a PCI extension, which allows a physical PCI function to be partitioned into multiple virtual PCI functions. Evolution The semantics of a trap-and-emulate VMM was formalized in the early 1970s (Popek and Goldberg, 1973) and made popular again in the mid-1990s for running commodity OS’es on multicore machines (Bugnion et al., 1997). However, the most popular machine architecture at the time, Intel x86, was not virtualizable because some of it’s privileged instructions did not trap. The VMware hypervisor, which targeted x86, was first released in 1999. It used binary translation to replace privileged instructions to trap into the hypervisor, while still running unprivileged instructions directly on the physical CPU, which solved x86’s virtualization issues (Adams and Agesen, 2006). This allowed the VMware hypervisor to run unmodified commodity OS’es on x86 hardware in virtual machines without the performance penalty of emulation. The Xen hypervisor released first in 2003 took a different approach to solving the x86 virtualization issue. Instead of binary translation, they modified the source code of the guest OS to trap to the hypervisor instead of executing non-trapping privileged instructions. Intel and AMD released x86 CPUs with virtualization extensions in 2005 and 2006, which made classic trap-and-emulate virtualization possible. KVM, initially developed for Linux, implements a kernel subsystem that in combination with QEMU’s device model provides a full hypervisor. Initially, the KVM project provided software-based device model that emulated full hardware devices, but later acquired paravirtualized I/O device model when the virtio device model was introduced. Future The classic hypervisor architecture has stood the test of time but there some trends that affect hypervisor design. Hardware virtualization is becoming more wide-spread. For example, the Amazon Nitro project (talk by Anthony Liguori) takes an unconventional approach to hypervisor design, which replaces all of the software-based device model with hardware virtualization as illustrated in this diagram. Amazon’s Nitro hypervisor also uses a custom designed VMM that leverage’s Linux KVM. Operating systems have also started to evolve to accommodate hypervisors better. Unikernels are an interesting OS design approach that packages the OS and the application into one bundle, which runs in the same CPU protection level (Madhavapeddy, 2013). This eliminates the traditional separation between kernel and user space, which reduces context switch and system call overheads at the expense of losing some OS functionality. The basic idea was already pioneered earlier in the form of library OSes, but the much simpler device model of a hypervisor compared to bare metal made the idea much more feasible for real world use. Light-weight virtualization is becoming more and more important as the use of cloud computing grows. Containers are excellent technology for providing light-weight virtualization. However, containers are unable to provide the full isolation capabilities of VMs, and have various security problems because containers share the same host OS and have access to the large OS system call interface (Manco et al., 2017). Hypervisors can be slimmed down significantly (Manco et al., 2017) and unikernels provide even larger opportunity to optimize the hypervisor if we relax the equivalence property requirement of VMMs (Williams, 2016). Serverless computing is a new computing model, better described as Functions as a Service (FaaS), that allows application developers to deploy functions instead of applications to a managed platform. One approach to serverless computing is to use hypervisors and unikernels for packaging and deploying the functions (Koller and Williams, 2017). Energy efficiency is another important future direction for hypervisor design. Communications technology, which cloud computing is a large part of, is forecasted to consume around 20% of global electricity by 2030, or as much as 50% in the worst case (Andrae and Edler, 2015)! The energy overhead of a hypervisor can be extremely high depending on workload. One experiment reports between 59% and 273% energy overhead for KVM (Jin et al., 2012)! Kernel-bypass networking has become important recently because NICs are getting faster and traditional TCP/IP and POSIX socket abstraction is proving to have high overheads (Han et al., 2012; Young et al., 2014; Yasukata et al., 2016). Hypervisors that implement the device model using I/O paravirtualized effectively introduce another layer to the networking data path, which increases networking overheads. In Linux, the vhost architecture is one solution to the problem. Vhost moves the virtio paravirtualized I/O device model from QEMU (which is the VMM userspace kernel) to the host kernel (which also hosts the KVM module), which eliminates the exit from host kernel to userspace VMM. Another solution is full hypervisor kernel-bypass using hardware NIC virtualization introduced by the Arrakis project (Peter et al., 2014). Summary The hypervisor architecture invented in the 1970s has stood the test of time. The x86 architecture quirks meant that the first successful hypervisors had to resort into binary translation to handle privileged instructions. Binary translation solutions were followed by paravirtualization (popularized by Xen) but hypervisor architectures were consolidated to the classic model as Intel and AMD added virtualization extensions to the x86 architecture. Although containers have recently become a very popular virtualization technique, emerging computing paradigms like serverless computing could make hypervisors an attractive technique again. Light-weight hypervisor designs, unikernels, and hardware-assisted virtualization all reduce hypervisor overheads, which also makes hypervisors more competitive against containers.
https://medium.com/@penberg/the-evolution-and-future-of-hypervisors-999f568f9a5d

iXsystems: 'weird' but profitable NAS vendor

Here’s something you rarely hear from high tech companies today: “We’re a hardware company in our heart and soul.” That is how executive vice president Brett Davis introduced iXsystems during a press tour in early December at the NAS vendor’s San Jose, California headquarters. The company sells open-source based TrueNAS enterprise hardware and FreeNAS desktop systems.

Self-identifying as a hardware company in this software-defined world is only one reason why iXsystems seems out of place in Silicon Valley. The vendor also bootstrapped its financing, turning a profit without accepting outside investment.

“We’re private, profitable and self-funded,” Davis said. “Our heritage goes back to the ‘90s. We’ve just been here. We say we’re unique, but you can say we’re weird.”

But it’s the hardware tag that provides most of the weirdness these days. The iXsystems headquarters includes a manufacturing facility in the back, where Davis said the 130-person company can fulfill 3,000 orders in a day.

The vendor claims more than 4,000 customers, including Sony, NBC, Duke University and NASA.

But the iXsystems strategy of bundling open-source storage software on commodity hardware isn’t that unusual.  Plenty of others do that, and label it software-defined storage. But only iXsytems boasts it’s a hardware company even if its value comes from open-source projects.

The company’s roots date to Berkeley Software Design, Inc. (BSDi), which started in 1991. iXsystems founders founders Mike Lauth and Matt Olander acquired the hardware business from BSDi in 2002. Lauth has been the CEO and Olander the CTO since then. From the start, iXsystems was heavily involved in the FreeBSD project and is the project leader for FreeNAS Storage and TrueOS Desktop open-source operating systems.

Davis said 70% of @iXsystems appliances are custom configurations. The vendor uses @Intel, @AMD and #ARM chips inside. The systems support @VMware, @Microsoft #HyperV, #Citrix, @KVM and @OpenStack #virtualization software, #Hadoop, @Docker and #MySQL data and container platforms, and @FreeNAS, @FreeBSD, @CentOS, @RedHat @Linux and  @Ubuntu #opensource software.

“We’ve been doing open source since way before it was cool,” Davis said. “We give away the number one software-defined storage (FreeNAS), but software and hardware are inseparable.”

iXsystems re-sold storage systems from Dot Hill, Infortrend and others in the late 2000s, but had to rely on those vendors for support. Now iXsystems provides end-to-end support for its storage. The company acquired the FreeNAS project in late 2009, and then spent two years re-writing the operating system before making it commercially available. iXsystems ported the OpenZFS open-source enterprise file system to FreeNAS. That gave FreeNAS file, block and object support, triple parity RAID, support for flash and unlimited instant snapshots.

@FreeNAS is file-only storage while @TrueNAS is unified storage with #FibreChannel networking support. TrueNAS competes with the likes of @DellEMC #VNXe and #Unity, @NetApp #FAS, @Hewlett Packard Enterprise’s #MSA2040 and @Nimble arrays, and @Western Digital’s @Tegile platform.

A more souped-up @iXsystems platform is planned for March with #NVMe support, but the vendor is staying quietly publicly about that for now.

Davis said iXsystems is staying out of hyper-convergence, even though it has all the pieces – including FreeBSD’s open-source bhyve hypervisor.

“We can run virtual machines and containers in FreeNAS,” he said. “We have the capabilities to do it, and we have our own hypervisor. But it’s a competitive space, and we have other plans.”

http://searchstorage.techtarget.com/blog/Storage-Soup/iXsystems-weird-but-profitable-NAS-vendor

Is Docker Dead?

Summary Nic Jackson compares and contrasts the development and deployment flow for both a @Docker and a #Serverless project, attempting to see if Docker has already been made obsolete by serverless. Bio Nic Jackson is a developer advocate and polyglot programmer working for HashiCorp, and the author of “Building #Microservices in Go”. In his spare time, Nic coaches and mentors at Coder Dojo teaching kids 7–14 Swift with Playgrounds, teaches at Women Who Go and GoBridge, speaks and evangelizes good coding practice, process, and technique and works to raise money for a charity he runs with his wife. It's all about #Containers, Serverless and Reactive Programming right now! ProgSCon London will explore these trends through engaging talks delivered by leading industry experts. Several talks will also feature various aspect of Blockchain, Microservices and Big Data. If you are a software developer looking to sharpen your skills and learn from the best in the industry, then ProgSCon London 2017 is the place you need to be at!

https://www.infoq.com/presentations/docker-serverless

Researchers Advance User-Level Container Solution for HPC

Most scientific computing facilities, such us #HPC or #gridinfrastructures, are shared among different research disciplines, and thus the system software environment needs to be generic enough to accommodate different user and applications profiles; they are multi-user environments. Because of managerial and technical constraints, such infrastructures cannot afford offering every research project a tailored environment in their machines. Therefore the interest of exploring the applicability of containers technology on such systems is rather evident from the end-user point of view. Researchers need then to customize their applications software to fit the computing center environment at the level of system software and batch system. #Containers provide a way to pack and deploy software including all the dependencies in a way that can be executed in a seamless way, independently of the underlying #Linux Operating System and environment. The main benefit of integrating the execution of containers in #HPC systems would then be to provide a way to execute applications homogeneously across different resource centers. The flagship container software, @Docker, cannot be used in a satisfactory way on HPC systems, grids and in general multi-user oriented infrastructures. Deploying Docker on such facilities presents a number of problems related to the fact that within the container, processes are executed with the root id. This raises security concerns among system managers, as the Docker root might be able to gain access to root privileges in the host machine. Also, when executed as root, the processes escape from the usual managerial limits on resource consumption or accounting, imposed on regular users at shared facilities. User-level tools The user-level tool udocker provides a layer for users to execute Docker containers, that by definition, does not require the intervention of the system administrators. Udocker combines the pulling, extraction and execution of Docker containers without requiring privileges. The Docker image is extracted on a user-space filesystem area, and from there on, it is executed in an chroot-like environment. udocker provides a command line interface that mimics Docker, providing a subset of its commands to be able to handle Docker images at the level of pulling, extracting and execute containers “á la Docker”. Processes are run without privileges under the regular user id, under the same process tree, thus facilitating the enforcement of the managerial limits imposed to regular users in HPC or grid resource centers. udocker provides several ways, depending on the application and host environment, to execute containerized applications. It is also possible to access specialized hardware like Infiniband for MPI jobs, or GPGPUs, making it adequate to execute containers in batch systems and HPC infrastructures. udocker enables the execution of Docker containers with different engines based on intercepting system calls. Depending on the application requirements the user may choose to run in one execution mode or another. For instance CPU-intensive applications may use udocker in the ptrace execution mode, to intercept and modify pathnames; if the application is I/O intensive the interception of system calls via library pre-loading using the Fakechroot execution mode is a more adequate way to run the container. All the tools and libraries required by udocker and its execution modes are provided with udocker itself. The udocker execution mode RunC employs the technology of user namespaces to run the containers in rootless mode. This feature can be used with modern Linux distributions with kernels from 3.9 on. However most HPC systems are conservative environments and it will take some time until they will be able to support this execution mode. Regarding impact in performance, in the figure presented below we have plotted the weak scaling performance of openQCD, a comprehensive software package to run Lattice QCD simulations (a CPU-intensive application) from 8 to 256 cores. As we see, the performance of the containerized version of openQCD is slightly higher than the one on the host itself. This is especially so when the execution takes place within a single node (the test machine has 24-core nodes). This behavior has been reported consistently by container users across different hardware and system software settings, and it is related to the better libraries available in the more advanced versions of the operating systems inside the container. Clearly this feature opens the door to container exploitation in HPC mainframes since there the software system is by necessity very conservative.  Figure Caption: Weak Scaling performance of openQCD with a local lattice of Volume=32^4. The tests have been performed on the Finisterrae-II HPC system at CESGA (Spain). Since its first release in June 2016 udocker expanded quickly in the open source community. It is being used in large international collaborations like the case of MasterCode, a leading particle physics phenomenology collaboration, which uses udocker to handle the library complexity of the set of codes included in the MasterCode. It has also been adopted by a number of software projects to complement Docker. Among them openmole, bioconda, Common Workflow Language or SCAR. System Administration level Beyond the user level, several solutions have been developed in recent times to support system administrators in deploying customized containers for their users. These solutions rely on the installation of system software by the system administrator, which also is in charge of preparing the containers that the users are authorized to run on the system. The most popular of these tools is Singularity. Singularity can be downloaded and installed from source or binaries, and must be installed by root for the software to have all the functionalities. Singularity binaries are therefore installed with SUID and need be deployed in a filesystem that allows SUID. Given the security concerns on network filesystems regarding SUID, Singularity is normally installed in a directory locally accessible to the users (i.e., not network-mounted). Singularity offers its own containers registry, the Singularity Hub, and its own specification to create containers, the Singularity Recipe (i.e., the Singularity equivalent of the Dockerfile specification). The default container format is squashfs, which is a compressed read-only Linux file system, where the images need to be created by root. It also supports a sandbox format, in which the container is deployed inside a standard Unix directory, much like udocker. In particular, executing udocker in Singularity execution mode will cause the container to be executed via Singularity if installed in the system. In order to do this udocker exploits the sandbox mode. The container building environment of Singularity belongs to root. Containers may be built either from a Singularity recipe, from a previous container coming from the Singularity Hub, or importing a container from the Docker repository. Notice that the Singularity format for containers is not compatible with Docker; therefore, in the latter case the container needs to be converted to the Singularity format. Once the container exists, it can be executed by a regular user in a way analogous to Docker. These containers can also be checked at the binary level, at the level of sensitive content of the filesystem for example, or even for particular features defined by the system administrator. The comparison of the most popular tools, udocker and Singularity, shows that they have a completely different scope, and the selection of one solution or another depends on the priorities at the user level and the computing center management policies. Singularity is a system administration level tool, to be installed at this level, giving the managers of the infrastructure full control of which containers are run into the system or not. Udocker however is a user tool that acts as a layer over different execution methods, enabling regular users to run containers in their own user space, much in the philosophy of the jailed systems.

https://www.hpcwire.com/2017/12/18/researchers-advance-user-level-container-solution-hpc/

Apache Bigtop Adds OpenJDK 8 Support

@Apache has released #Bigtop 1.2.1 with support for #OpenJDK 8, and a new sandbox feature that lets you run big data pseudo clusters on @Docker. Bigtop is an Apache Foundation project that you can use for packaging, testing, and configuration of the big name open source big data components that make up the #Hadoop infrastructure. Bigtop supports a wide range of components and projects, including Hadoop, @HBase and @Spark. The primary goal of Bigtop is to build a community around the packaging, deployment and interoperability testing of Hadoop-related projects. This includes testing at various levels, including packaging, platform, runtime, and upgrade, and focussing on the system as a whole, rather than individual projects. While Hadoop is generally used to refer to the central collection of tools, Bigtop looks at the wider selection that makes up the Hadoop-related projects, including Hbase, Pig, Hive MapReduce, Zookeeper and Avro among others.  Bigtop packages Hadoop RPMs and DEBs, so that you can manage and maintain your Hadoop cluster, and it provides an integrated smoke testing framework, alongside a suite of 50 test files. It also helps with virtualization testing, with vagrant recipes, raw images, and (work-in-progress) docker recipes for deploying Hadoop from zero, and you can use Bigtop Provisioner to spin-up a virtual cluster with a single command. The new release of Bigtop adds a Sandbox feature that lets you use Docker to run pseudo clusters. Creating it is a single line command, and for HDFS the process takes around 30 seconds. You then get a local WebUI to play around it. You can run HDFS and Spark standalong, or HDFS, Yarn, Hive and Pig - there are simple instructions on the Bigtop site. This presentation from DataWorks Summit 2017 on using the Sandbox comes from Apache Bigtop Project Committer and PMC member, Evans Ye:   The new release also includes a faster Docker Provisioner which has been rewritten to fully embrace the Docker ecosystem. The OpenJDK support in the new release means that all the components are now built on JDK8. The major components have been updated to recent versions, including Hadoop 2.7.3, Spark 2.1.1, HBase 1.1.9, and Zeppelin 0.72, and most of the ecosystem projects have also been updated, including Apex, Crunch, Flume, Ignite, Mahout, Oozie, and Phoenix, among others.

http://www.i-programmer.info/news/197-data-mining/11374-apache-bigtop-adds-openjdk-8-support.html

Hadoop 3.0 Likely to Arrive Before Christmas

It’s looking like #bigdata developers will get an early holiday present as work on #Hadoop version 3.0 nears completion. And while #Hadoop3.0 brings compelling new features, including a 50% increase in capacity and upwards of a 4x improvement in scalability, more exciting stuff – like support for @Docker, support for #GPU s, and an S3-compatible storage API — is slated for versions 3.1 and 3.2 next year. After years of work, the #Apache Hadoop community is now putting the finishing touches on a release candidate for Hadoop 3.0 and, barring any unforeseen occurrences, will deliver it by the middle of December, according to Vinod Kumar Vavilapalli, a committer on the Apache Hadoop project and director of engineering at Hortonworks. “We can’t set the dates in stone, but it’s looking like we’ll get something out by mid-December,” Vavilapalli told Datanami in an interview last week. The current plans for Hadoop 3.0 haven’t changed dramatically since the last time we covered the open source project, back in May, when we talked with Andrew Wang, the Hadoop 3 release manager for Apache Hadoop and an engineer at Cloudera. Here’s a rundown on what’s coming in Hadoop 3.0: Erasure Coding Erasure coding is a data protection method that up to this point has mostly been found in object stores. With erasure coding in place, Hadoop 3.0 will no longer default to storing three full copies of each piece of data across its clusters. Instead it will use a data striping method that’s similar in some ways to RAID 5 or 6. Instead of that 3x hit on storage, the erasure encoding method in Hadoop 3.0 will incur an overhead of 1.5x while maintaining the same level of data recoverability from disk failure. “HDFS has always been a system based on replication,” Vavilapalli said. “Now that organizations have put a lot of data in their big data clusters… it’s time to think back and look at how to efficiently manage the storage and get more out of the Hadoop cluster. That’s where erasure coding comes in.” YARN Federation Hadoop 3 will include new features in the YARN resource manager that will open the door to customers running single clusters with tens of thousands of nodes – and possibly even to hundreds of thousands of nodes. “YARN was originally designed to scale to 10,000 machines,” Vavilapalli said. “Our friends at Microsoft have been contributing this feature called YARN Federation that will scale YARN to multiple tens of thousands of machines.” Conceptually, YARN Federation works similarly to HDFS Federation, a major Hadoop 2.x delivery that has multiple namenodes working in concert. With YARN Federation, each sub-cell would be responsible for a group of machines while allowing multiple sub-cells to work together to build massive clusters in a single namespace. YARN Federation should allow Hadoop clusters to scale to 40,000 nodes without too much trouble, and even allow it to scale beyond 100,000 nodes, Vavilapalli says. Resource Types Hadoop 3.0 will bring an extensible new framework to YARN that lets it mange additional resource types beyond memory and CPU. This will provide the basis for supporting GPUs in Hadoop clusters with version 3.1 and FPGAs ostensibly in version 3.2. It will also allow YARN to control another important resource in a big data cluster that up until now has not been directly controllable: disk. Java 8 Apache Hadoop 2 and many extended members of the Hadoop family currently run on version 7 of the Java Developers Kit (JDK). With support for JDK7 waning, and JDK8 being the optimal route forward, the folks running the project are making the call to enforce a switch to JDK8 starting with Apache Hadoop 3.0. The good news is the wider big data community is working together to ensure that other animals on the big data farm are also moving up to JDK8. The releases of Hadoop 3.0, HBase 2.0, Hive 3.0, and Phoenix 3.0 will coincide, more or less, in time and in Java. “JDK is one of the things that does tie all these communities together, because at the end of the day, you’re running all the software on the same cluster, so running different pieces on different JDKs is going to be a problem,” Vavilapalli said. After Hadoop 3.0 becomes generally available this month, there will be new releases of HBase, Hive, and Phoenix, he said. “So the first half of next year is when all these things will come together,” he said. “It’s not just about Hadoop 3.0.” Accelerated Release Cycle Work is already underway on Hadoop 3.1 and Hadoop 3.2, and the plan calls for delivering these releases three months apart, Vavilapalli said. The Apache Hadoop community will be accelerating its release cycle to get more capabilities into the hands of its users more quickly, he said. The releases “have been spaced out too much. Every six months you used to get a new release,” he said. “In 3.0 we’ve proposed…the community generally moving to faster release. So the community is moving in that direction.” Vavilapalli also discussed features slated for Hadoop 3.1 and Hadoop 3.2, which we’ll cover in a future article.

https://www.datanami.com/2017/12/05/hadoop-3-0-likely-arrive-christmas/

BlueData and Dell EMC Expand Relationship to Deliver Big-Data-as-a-Service in the Enterprise

Dell EMC to Resell BlueData EPIC Software for Containerized Big Data Workloads SANTA CLARA, CA--(Marketwired - Nov 14, 2017) - @BlueData ®, provider of a leading #BigData -as-a-Service ( #BDaaS ) software platform, today announced that it has signed a reseller agreement with Dell EMC, becoming a Select partner in Dell EMC's #TechnologyConnect #PartnerProgram. The two companies will deepen their go-to market relationship to deliver the BlueData EPIC software platform to Dell EMC's enterprise customers across the globe. BlueData EPIC software uses @Docker #container technology to help simplify and accelerate the deployment of #BigData #analytics and #datascience environments in the enterprise. Together, Dell EMC and BlueData provide a comprehensive solution to accelerate time-to-value and reduce TCO for Big Data infrastructure. With Dell EMC and BlueData, enterprises can deliver exceptional simplicity, speed, efficiency, and performance for their Big Data implementations. Dell Technologies Capital is an early investor in BlueData, and an active member of their Board of Directors, providing valuable business guidance and access to the broad Dell Technologies ecosystem. This latest agreement deepens the go-to-market relationship between BlueData and Dell Technologies and is an important step in helping the company to continue to scale its business. BlueData's EPIC software platform achieved certification for Dell EMC PowerEdge Servers in the Dell Technology Partner Program last year - and earlier this fall, BlueData was announced as a key component of the new Dell EMC Elastic Data Platform, along with Dell EMC Isilon and Dell EMC services. As part of this new reseller agreement, BlueData will be included in the Dell EMC price list and configurator for its global sales teams and channel partners. "This is a critical milestone for BlueData," said Kumar Sreekanti, CEO of BlueData. "This new reseller relationship allows BlueData to expand our reach into the global Dell EMC customer base, helping both new and existing customers to scale their Big Data deployments. We are proud to work with world-class technology leaders like Dell EMC and we look forward to this expanded partnership." BlueData enables enterprises to implement Big-Data-as-a-Service running on their own data center infrastructure or in a hybrid architecture. Using BlueData EPIC software, these organizations can provide on-demand agility and elasticity for their analytics and data science teams -- with the ability to spin up instant clusters for Hadoop, Spark, and other Big Data tools in a secure multi-tenant environment. And with BlueData, they can save up to 75 percent on the infrastructure and operational costs for their Big Data deployments. "BlueData is disrupting the traditional deployment model for Big Data analytics, with a fundamentally new and more cost-effective approach," said Keith Manthey, CTO of Analytics at Dell EMC. "Now enterprises can get cloud-like agility and flexibility with containerized Big Data applications, while ensuring comparable performance and security to that of a bare-metal environment. We look forward to working with BlueData to bring these benefits to our joint customers." The expanded relationship with @Dell EMC is another factor in the continued momentum for @BlueData, which achieved a 426% yearly increase in sales in 2016. #BlueData attributes its success to the increased adoption of #Hadoop, @Spark, and other #BigData technologies; growing interest in @Docker containers, cloud computing, and #BigDataAsAService among enterprise customers; the strength of its technology partner ecosystem; as well as the ground-breaking innovation of BlueData's #EPIC software.

http://m.marketwired.com/press-release/bluedata-dell-emc-expand-relationship-deliver-big-data-as-service-enterprise-2240634.htm

Windocks releases first Data Delivery Platform based on Docker's Container Technology and SQL Server Containers

@Windocks announced the Beta release of Windocks 3.0, the first data delivery platform built on @Docker ’s container technology, with support for #SQLServer #containers. Windocks presents a new option for enterprises as a modern, open, data delivery platform. Modern: Windocks is the first enterprise data delivery platform built on Docker’s container technology. In contrast with outdated, expensive @Solaris #UNIX systems that require dedicated storage administrators, Windocks runs on Windows and provides Developers and DBAs self-service data access and management. Windocks is priced for wide adoption, being available at less than $20/month/data environment. Windocks runs wherever Windows is supported, for use on any public cloud and on premise. Open: Windocks integrates with existing enterprise storage systems, providing simplified access to existing enterprise data. Windocks also adds support for creating and managing SQL Server clones on Windows fileshares. Windocks also supports delivery of cloned data environments to any enterprise application, and is the first to provide support for delivery of clones to Microsoft’s new Docker containers. Windocks 3.0 also adds support for MySQL, and other enterprise data environments are pending. Enterprise Data Delivery: Windocks delivers Terabyte class data environments in seconds, with incremental instantly available updates. Teams simplify operations by using environments on a shared server, reducing VM usage by 5:1 or more. Windocks is popular for support of Jenkins and Continuous Integration strategies, simplifying the delivery of data environments with multi-tier application environments, including .NET, Java, with SQL Server and other databases. Finally, Windocks enhances Data Governance and Regulatory Compliance with data images stored in a versioned, auditable data image repositories. “Enterprise customers are asking for an alternative to expensive, complex solutions built on Solaris UNIX.” said Paul Stanton, Windocks co-founder. “Windocks 3.0 delivers the first container native data delivery solution that fits any budget. Windocks empowers software developers and database administrators with tools to create, manage, and deliver data environments more simply, and affordably than ever.    In a single step SQLServer DBAs create clonable images, and users self-service environments with one click on the Windocks web application.” About Windocks: Windocks delivers database cloned environments for organizations around the globe, enabling organizations to access, manage, and protect data faster, more efficiently, and simply than existing data systems. Database cloning combined with Docker based containers enables modern software development and delivery, and reporting. Windocks is the first open, modern platform for enterprise data delivery.

http://www.prweb.com/releases/2017/10/prweb14854867.htm

Containers will oust VMs and storage arrays, says Red Hat

@RedHat launches storage delivered via #containers and predicts a future in which costly and inflexible #storagehardware and pricey #hypervisors will be a thing of the past The #traditionalstorage array will become a thing of the past – and so will the virtual machine and hypervisor. Costly and cumbersome and lacking the flexibility to support the unpredictable and “bursty” requirement of applications in the cloud era, they will be replaced by containers.

That is the view of Red Hat, which earlier this month launched Container-Native Storage 3.6 for Red Hat #OpenShift Container Platform 3.6, its distribution of the @Kubernetes container #orchestration software.

In Openshift Container Platform, customers can deploy applications via container, rapidly scaling instances according to workload requirements.

Container-Native Storage allows storage for container-based apps to be spun up, run and be decommissioned as required to support those applications and can run on-premise or in the cloud and with service levels set by policy.

New features in Container-Native Storage include the addition of support for block (via iSCSI) and object storage.

Containers – such as Docker – are emerging as an alternative to server virtualisation. They are effectively a form of software-defined server/storage and offer a form of virtualisation, but more lightweight, that runs directly on the operating system.

Containers lack the hypervisor layer, and also the often-duplicated data in virtual machine images.

Containers excel where the rapid scaling of web serving requests is met by the creation, use and dying out of many containers managed by an automated orchestration platform.

http://www.computerweekly.com/news/450428848/Containers-will-oust-VMs-and-storage-arrays-says-Red-Hat

Virtuozzo simplifies persistent data storage for Kubernetes clusters

#Hyperconverged infrastructure software provider @Virtuozzo Inc. has punted out a new #persistentstorage product for @Kubernetes, which is a popular orchestration tool that’s used to manage software application #containers. The move comes following this summer’s release of Virtuozzo Storage for @Docker, a #softwaredefinedstorage platform designed for managing Docker containers with persistent data in production. Persistent storage, sometimes called nonvolatile storage, relates to storage devices that retain data after the power is switched off. With regard to application containers, the term more generally refers to storage volumes or individual clusters that remain available and retain data after the life of individual containers. Being able to retain data after an application container has shut down is obviously desirable, but Virtuozzo Chief Executive Officer George Karidis said it’s not so easy to set up this kind of persistent storage in container deployments that span across multiple servers. That’s where the new Virtuozzo Storage for Kubernetes platform should prove to be helpful, as it provides customers with a simplified way to set up and manage these persistent storage volumes. Basically, Virtuozzo Storage for Kubernetes is a set of easy-to-use management tools built atop of the company’s flagship software-defined storage platform. With the new product, users can set up, monitor and manage their Kubernetes storage clusters and other services such as file and object storage more easily. In addition, the platform allows users to define storage classes and automatic snapshot policies to ensure everything is safely backed up. Virtuozzo Storage for Kubernetes can be run on any kind of commodity hardware, the company said. “With our latest Virtuozzo Storage for Kubernetes solution, we’re delivering a tightly integrated, production-ready solution for K8s that makes it very easy for any enterprise, small business or service provider to put containerized apps into production,” Karidis said. Virtuozzo said its new Kubernetes storage platform, and also its Virtuozzo Storage for Docker, are both being exhibited this week at the DockerCon EU ‘17 event taking place in Copenhagen.

https://siliconangle.com/blog/2017/10/16/virtuozzo-simplifies-persistent-data-storage-kubernetes-clusters/

A Brave New World: Pat Gelsinger Leads VMware Into The Multi-Cloud Era

As @VMware prepared to enter the @DellTechnologies orbit in the wake of the biggest merger in industry history, CEO @PatGelsinger grappled with weighty decisions that would reshape the destiny of the company. Rarely before had such an established software giant faced disruptive threats of such severity on multiple fronts. Public cloud operators were proclaiming the end of private infrastructure; #container enthusiasts were dubbing @Docker a VMware killer that would supplant virtualization. After more than a decade of dominance in the data center, with VMware's core business under siege, Gelsinger knew he and the board faced some tough decisions. There were seven "final" meetings of VMware's management team, three "final" board meetings, before consensus formed around a bold new strategy: Rather than battle emerging technologies, VMware would embrace them with breakthrough products and partnerships. It would leverage its existing technology to become a connective fabric, the central nervous system linking the heterogenous platforms many expected to undercut its business. [VMware Cloud On AWS: Vendors And Partners Weigh The Opportunity And Risks] "It's a multi-cloud world and I think we're uniquely positioning VMware to be an enabler in the multi-cloud world that really benefits the core VMware customers," Gelsinger told CRN in an exclusive interview detailing its remarkable transformation. "Any technology company, if you're fi ghting the wave of technology, you're likely to become driftwood. If you're riding that wave of technology, you're likely to become a very powerful and customer-valued entity in the future. You either are on the right side of it or you're not." [Sponsored Suggested Post: Free IoT Virtual Conference Learn how to monetize IoT solutions. Discover new IoT technologies. Build strategic partnerships.] Getting on the right side of the public cloud wave, however, posed a daunting challenge fraught with risk. Could VMware partner with Amazon Web Services? Could it trust Microsoft? Would Google emerge as a serious enterprise player? As difficult as those questions were, the private cloud kingpin knew it had to position itself to thrive in an IT landscape increasingly controlled by those companies. Solution providers say the multi-cloud strategy is a winning hand for them and VMware. "We are seeing very wide adoption of multi-cloud, public and private, on-prem and off. We're seeing customers starting to embrace it more and more every day," Chuck Farrow, vice president of strategic partner alliances at New York-based Logicalis, told CRN. "The way Pat's taken the company and acted upon their vision, making that turn, their investments to make them more complete around the data center, tremendous things are going to come from that," Farrow said. Investors, for their part, also are buying into VMware's transformation story. VMware shares closed trading at $109.20 on Sept. 28 — up 50 percent since Sept. 7, 2016, the day Dell Technologies completed its $67 billion acquisition of EMC and became VMware's de facto owner. That surge has translated into nearly $14 billion in additional shareholder value. One beneficiary of the soaring stock price has been Michael Dell, who took over as VMware's chairman the same day the deal closed. "I think Dell EMC and VMware go together like peanut butter and chocolate," Dell told attendees of VMware's annual conference, VMworld, during a question-and-answer session. "We've got a great thing going here. We've talked a lot about revenue synergies. And that was driven not only by cross-selling, but also a deep level of technical integration innovation in creating new products and solutions.

http://www.crn.com/news/cloud/300093516/a-brave-new-world-pat-gelsinger-leads-vmware-into-the-multi-cloud-era.htm?itc=ticker