Storage
Intermediate 11 min read Storage

Data Transfer and Import

Getting data in, out and between clouds

Moving terabytes into and out of the cloud needs the right tool: SDK/uploads for online streaming, transfer services for big buckets, and physical appliances (Snowball, Data Box) when the pipe is too slow.

data transferimportSnowballAWS DataSyncAzure Data BoxTransfer Servicebottoms
Prerequisites: Object Storage
Before we start

What you will learn

  • Choose between network, streaming and physical transfer
  • Use transfer services for bulk and scheduled sync
  • Estimate transfer time and know when to ship a device
In 60 seconds

Small data rides the network (CLI/SDK, DataSync/Storage Transfer Service). Very large data is shipped physically (Snowball, Data Box, offline-transfer). The decide metric is time: at 100 Mbps, 100 TB takes ~100 days — ship it.

Introduction

What is this?

Online: s3 sync/aws S3 transfer acceleration, az storage copy (azcopy), gsutil rsync; managed: AWS DataSync (schedule, delta), Azure Storage Transfer Service, GCP Storage Transfer Service. Offline: AWS Snowball (TB-PB), Azure Data Box, GCP Transfer Appliance. Costs: egress (out of cloud) is billed per GB; ingress free — plan the direction.

Motivation

Why do you need it?

  • Recovering from an on-prem datacenter over slow links is painfully slow.
  • Replicating between regions/buckets can piggyback transfers instead of local moves.
  • Physical devices beat networks for multi-TB/PB workloads by orders of magnitude.
  • Data classification (PII, EU data residency) decides which carrier/region is allowed.
Think about it

Mental model

Imagine

Shipping a pallet vs emailing a spreadsheet

Think of

Physical vs network transfer

Nicely-formatted CSVs of a few MB can be emailed (online). A warehouse of tapes needs a freight truck (appliance) — velocity, not bandwidth, is the deciding factor.

Mechanics

How it works

  • Online: parallel HTTP/S3 multipart uploads; tools parallelize (azcopy -j, gsutil -m).
  • Managed sync: agents run deltas, schedule, retry and compress at the edges.
  • Physical: you order a device, copy to it, return it; the provider imports into your bucket safe room.
  • Egress: leaving the cloud costs per-GB; ingress generally free — direct bulk outbound cheaply (private links/VPN).
  • Latency/consistency: object stores are eventually consistent for many ops; verify the final sync.

Choosing the transfer path.

Size the job
Time estimate ok?
Online
Offline
Verify + reconcile
Hands-on

Examples

Estimate then run a transfer

Estimate then run a transfer
# estimate: 10 TB at 200 Mbps sustained
python3 -c "print('days:', 10*1024*1024*1024*8 / (200*1e6) / 86400)"
# run
aws s3 sync ./backups/ s3://my-bucket/backups/ --sse AES256
gsutil -m rsync -d ./export gs://my-bucket/export
azcopy copy ./export "https://acct.blob.core.windows.net/export?sas" --recursive

Estimate the network ceiling first; then let the tooling parallelize the actual copy.

Real world

In production

  • A hospital migrated 40 TB of PACS images via Data Box for residency-reasoned offline import.
  • A media house uses AWS DataSync for nightly delta-sync of on-prem renders to the cloud.
  • GCP Storage Transfer Service replicated a 2 PB data lake between regions with checksums.
Guidelines

Best practices

  • Always checksum/reconcile (aws s3 sync reports, gsutil rsync -n live preview).
  • Begin with a small pilot transfer to validate tools and permissions.
  • Plan for egress costs when pulling out of the cloud — often the largest hidden bill.
  • Encrypt in transit and at rest for sensitive datasets; enable object-lock/versioning during import.
Watch out

Common mistakes

  • Kicking off a full copy before a pilot; permission errors surface after hours.
  • Ignoring egress pricing when mirroring to other clouds.
  • Using a single-threaded upload for huge trees (you get 1/10 of the pipe you bought).
  • Forgetting object metadata/ACLs on re-uploads (ownership/timestamps lost).
Root cause analysis

Common issues

Error
Transfer finishes but etags don't match
Root cause

Multipart chunking vs single-part checksums differ

Fix

Compare total-object hashes after finalize; some tools use composite checks.

Error
Data Sync keeps overwriting
Root cause

No -d/--delete flag or case mismatch in keys

Fix

Use the tool's preview/dry-run and explicit delete semantics, or compare listings upfront.

When things break

Debugging tips

  • Slow sync: check the number of parallel workers vs target config — check order of magnitude.
  • Failures mid-copy: the tools resume (s3 sync/azcopy checkpoint) — verify partial upload listing.
  • Pilot permissions: test one folder before the whole prefix.
  • Monitor bandwidth: an egress spike on huge transfers is also a billing spike.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/1
1

Egress (upload) from a data center into a cloud is typically free; downloads out carry a per-GB charge.

Go deeper

Official documentation

AWS — DataSync
docs
Azure — Data Box
docs
GCP — Storage Transfer Service
docs