Introducing infinite file size and universal upload resume for all cloud storage
Spend any time moving data to the cloud and you will undoubtedly hit a few snags, especially with large file uploads. Depending on the size of the file and the speed/stability of your internet connection, you can find yourself in a state of perpetual upload.
Sisyphean upload
In Greek mythology, Sisyphus was forced to push an enormous boulder up a steep hill. Before reaching the top, the boulder would always roll all the way back down to the bottom. The poor guy would then have to start the whole thing over again, ad infinitum.
Sound familiar?
Many folks have found themselves stuck in this situation when trying to push their large files up to the cloud. Something bad will happen, mid-transfer, and the upload will need to restart from the beginning. Some user’s files will just never, ever make it … and it really sucks.
The problem arises when a file is large enough that the length of time it takes to successfully upload exceeds the length of time your internet connection can maintain adequate stability.
The convergence of ever-decreasing cloud storage costs, ever-increasing file size, and always-available content creation devices (like smartphones) is triggering a Sisyphean upload epidemic. The main issue is that consumer internet upload speeds are increasing at a snail’s pace compared to the rate at which users are creating new (and exceedingly large) content; and the longer a file takes to upload, the more likely it is to fail.
The FCC defines “broadband” upload speed at a pitiful 3Mb/sec. The average consumer internet upload speed is somewhere between 3Mb(0.38MB)/sec and 8Mb(1MB)/sec. Compressed 4K video, which is becoming a standard for new smart devices, typically comes in at between 350 and 400 MB/minute (Apple cites 375MB/minute for the iPhone 6).
Rough estimates for common large file types
Just a 15 minute home video on your iPhone 6 is going result in a file that is ~6GB in size. Given the average consumer upload speed, this one file, consisting of a mere 15 minutes of your life, is going to take 2–5 hours to upload. That is 2–5 hours of sustained, uninterrupted network connectivity, amid a myriad of potential variables that can cause a break in the transfer.
If you find yourself on the lower end of the speed spectrum and have multi-gigabyte files to upload, you are in trouble… or rather, you were in trouble.
Universal upload resume
With over 20 different types of storage integrations under our belt, we’ve seen this issue come up a lot. It presents an interesting problem to solve because there have been no universal methods for resuming uploads. Many services do not offer any form of upload resume, and those few that do vary wildly in their implementation.
The solution? Create our own, of course! Our design needed to satisfy the following requirements:
- Resume from any network/system disruption
- Accommodate files of any size, regardless of storage file size limits
- Usable across all current and future odrive integrations
- Transparent to the user via the odrive desktop client and all normal file operations (move/rename, copy, edit, delete) work as expected
- Portable. Files can not only move within the service where they reside, but they can be moved to other services, as well.
- Non-proprietary
- Configurable
- Reliable
The result? odrive "Infinite File Size".
With IFS you will always be able to successfully upload your files to any odrive linked storage. No more file size limitations. No more perpetual upload.
We accomplish this by performing on-the-fly upload chunking. The chunked segment size is user-specified, with a range between 100MB and 2GB. The option specified also acts as an automatic threshold for any future content. If you have specified 100MB, as we have in the example image below, any new files that exceed 100MB in size will be chunked using 100MB segments. If you select “Never”, then IFS is not used.
Each segment is hashed and uploaded to a special directory.
Inside the directory you will find the individual file segments named with the MD5 hash of that segment.
When all parts have been uploaded, a meta file is created (.meta). This file contains details about the chunked file segments, including the size, hash, and order of each segment.
Through odrive, the file looks like any other, both in its locally cached mode and its placeholder mode:
If the upload of an IFS file is interrupted, odrive will automatically resume the upload from the last successfully uploaded segment. It does this by comparing the calculated hash of the currently processing segment with the hash-based file names inside the .xlarge directory on the remote storage. If the current hash is found within the enumerated list, the segment is skipped and the next one begins processing.
DIY
Considering the fact that we are creating a brand new system to facilitate unlimited file sizes and universal upload resume capabilities, it is important to us that this system be completely transparent, and independent of odrive. To demonstrate this, we created some simple shell scripts that anyone can use to re-assemble the raw odrive Infinite File Size files.
Powershell:
Bash:
Struggle no longer
This is just one of the many ways that we are transforming how users interact with their storage. odrive Infinite File Size is available now, for all odrive, users via the odrive desktop client.
Infinite sync.
Infinite file size.
Zero struggle.
Get everything into the cloud today!