Transforming Cloud Infrastructure to Support Big Data Storage and Workflows
Cloud systems promise virtually unlimited, on-demand increases in storage, computing, and bandwidth. As companies have turned to cloud-based services to store, manage and access big data, it has become clear that this promise is tempered by a series of technical bottlenecks: transfer performance over the WAN, HTTP throughput within remote infrastructures, and size limitations of the cloud object stores.
This session will discuss principles of cloud object stores, using examples of Amazon S3, Microsoft Azure, and OpenStack Swift, and performance benchmarks of their native HTTP I/O. It will share best practices in orchestration of complex, large-scale big data workflows. It will also examine the requirements and challenges of such IT infrastructure designs (on-premise, in the cloud or hybrid), including integration of necessary high-speed transport technologies to power ultra-high speed data movement, and adoption of appropriate high-performance network-attached storage systems.
The session will also explore how organizations across different industries are using big data in the cloud for ever-greater efficiencies and innovation, including those in the media and entertainment industry and in the field of life sciences.