Durability and performance in an S3 alternative storage platform are complex problems, but not owning the hard drives brings another level of difficulty. At Storj, we have 13,500+ independent node operators putting their unutilized hard drive space to work by joining our decentralized network. Learn how Storj developed an architecture that could deliver the demands of an S3 workload and also ensure durability. Assuming any of the node operators could be malicious, required a focus on encryption. We’ll go over how encrypting files the typical way (the whole thing) makes it difficult to support video streaming and what we did instead. What if there’s a blackout bringing thousands of nodes offline or a country deciding to block internet access? We’ll share the details about how these scenarios played out and what happened with our network. Spoiler, we didn’t lose a file. How do we maintain a system that is fault tolerant despite being susceptible to a large amount of threats? Short answer: erasure coding, encryption, and repair! We’ll show you how it works. Dan will discuss how Storj addresses these complexities and more. Join us for this casual, yet technical discussion: - Erasure codes for fun and profit - Streaming 4k video from encrypted content across thousands of nodes - Ensuring performance and eliminating long-tail effects - Managing data audits, repair, and reputation at scale
You are here
Lessons Learned (the hard way) from Building a Global, Decentralized Storage Network
With Amazon S3 celebrating its sixteenth birthday this year, it's easy to forget just how revolutionary it was at its release.
Azure Disks provide block storage for Azure Virtual Machines and are a core pillar of the Azure IaaS platform.
Azure Block Storage, also referred to as Azure Disks, is the persistent block storage for Azure Virtual Machines and a core pillar for Azure IaaS infrastructure.
- Greg KramerMicrosoft
Edge is becoming the new core .More and more data will live in edge environments rather than in traditional data centers/cloud due to reasons ranging from data gravity to data sovereignty.Different
- Subhadip DasHewlett packard enterprise
Cloud service providers offer a range of storage services to enterprises, such as block storage and object storage. Currently, more data is being created, stored, and analyzed than ever before.
- Kapil KarkraIntel Corporation
Many years ago SPDK introduced the “Copy Framework”, a light weight framework for supporting one HW accelerator, Intel® IOAT, behind a generic API so that acceleration would be used if available an
Samsung recently contributed Poseidon project, which is an OCP-based industry collaboration between component vendor (Samsung), system vendor(Inspur) and data center.
The major Cloud Providers such as VMWare, Microsoft and Google provide workload suites to deploy and manage container microservices and cloud storage through data models.
- Ravishankar NDell