Resilient Large File Uploads with Signed URLs and ETags

A large file upload reaches 93 percent. The connection drops. The user retries, reloads the page, then presses Cancel. A few minutes later, the object appears in storage anyway.

A retry loop around one HTTP request does not define what should happen in that sequence. Resilient large file uploads need a durable upload session, independently retryable parts, an integrity contract, and explicit rules for completion and cancellation.

I separate the application that authorizes the upload from the service that receives its bytes. This article uses Amazon S3 multipart upload, a browser client, and an application backend. It covers S3 general-purpose buckets; other storage providers and directory buckets need their own API and limit checks.

The JavaScript examples show the critical boundaries rather than a complete web application. The goal is to know what can be retried, what must be reconciled, and when the product can honestly say the file is ready.

Keep control requests separate from file bytes

The application backend authenticates the user, creates an upload session, chooses a storage key, signs individual part requests, records progress, and owns completion or abort. The browser sends the file bytes directly to storage using those signed URLs.

This keeps a multi-gigabyte body away from the application server’s request buffers and proxy timeouts. It does not remove the backend from the reliability design. The backend remains the authority for ownership, quotas, the intended file, and whether an upload has become a usable object.

Multipart upload has three storage operations: initiate, upload parts, and complete. An uploaded part is not yet the finished object. A completed object can still require validation before the application exposes it. AWS describes the underlying mechanism in its multipart upload overview.

Define a durable session before starting workers

Give the application session its own opaque ID. Store the authenticated owner, server-generated object key, provider upload ID, declared size, approved content type, part plan, expected checksums, receipts, state, revision, and expiry. The application session ID and S3’s UploadId are different identifiers.

A compact control API can look like this:

POST /uploads                         create or recover a session
PUT  /uploads/{id}/manifest            seal the file's part plan
POST /uploads/{id}/parts/{n}/url       authorize one planned part
PUT  /uploads/{id}/parts/{n}/receipt   record an accepted part
GET  /uploads/{id}                    read authoritative progress
POST /uploads/{id}/complete           request finalization
POST /uploads/{id}/cancel             request provider cleanup

Every endpoint reloads and authorizes the session. The signing endpoint derives the bucket, key, provider upload ID, and permitted part number from that record. A browser-supplied key or upload ID is not proof of ownership. Use idempotency keys for session creation and finalization requests so an HTTP retry returns the same operation rather than starting another one. Apply the normal CSRF protections to cookie-authenticated control endpoints.

Persist initiation intent before calling S3, then persist the returned provider ID. A crash between those steps can leave an orphaned multipart upload. A database transaction cannot include that storage operation atomically. A recovery job and lifecycle cleanup must cover this gap.

Split the file into a stable part plan

For an illustrative 20 GiB file, 64 MiB parts produce 320 parts. Number them consecutively from 1. Store each part’s start offset, expected length, and SHA-256 checksum. Keep this mapping unchanged for the life of the session.

The byte range for part n is:

// Part numbers are one-based; Blob.slice uses an exclusive end.
const start = (partNumber - 1) * partSize;
const end = Math.min(start + partSize, file.size);
const body = file.slice(start, end);

Only the final part may be shorter. Current S3 limits allow up to 10,000 parts, with regular parts from 5 MiB to 5 GiB. Derive or reject the plan before starting: a fixed 64 MiB choice cannot handle arbitrarily large files. Handle empty files separately. Check the current multipart upload limits when defining the product’s maximum file size.

Start with a small bounded worker pool, such as four concurrent parts, then measure on the connections and devices the product serves. That is an example policy, not a universal optimum. More workers mean more network contention, hashing work, and memory pressure.

Each worker gets one immutable slice. Do not intentionally run two attempts for the same part at once. A retry keeps the same session, part number, bytes, and checksum.

S3 replaces an earlier part when the same part number is uploaded again for the same upload ID. This makes an uncertain part result recoverable without restarting acknowledged parts. The constraint matters: retrying different bytes under the same number would replace the intended content. See the UploadPart API.

ETags, checksums, and If-Match solve different problems

Keep a returned part ETag exactly as received, including quotes. It is an opaque receipt used when completing that upload. Do not reconstruct it, strip characters for convenience, or assume it is an MD5 digest.

For this design, initiate S3 multipart upload with ChecksumAlgorithm: "SHA256" and ChecksumType: "COMPOSITE". Hash each part and bind its signed request to that digest. Send each part’s checksum alongside its ETag during completion.

A multipart SHA-256 checksum is composite. It is not SHA256 of the concatenated file bytes. Compute and verify the expected composite value using the provider’s definition and the ordered part digests. Do not compare a composite checksum with an ordinary whole-file hash. AWS documents these distinctions in checking upload integrity.

If-Match is an HTTP precondition. A resource API can reject a write when the client’s revision no longer matches the current representation, usually with 412 Precondition Failed. That is useful for concurrent updates; it is not an upload retry mechanism. S3 also supports conditional completion, such as If-None-Match: * to avoid replacing an existing key. Handle its documented precondition and conflict outcomes. A 409 conflict during conditional multipart completion can require a new multipart upload and reuploading parts. A 412 following an uncertain completion should first be reconciled against the existing object. Do not treat every precondition failure as a reason to resend blindly. This design additionally uses unique server-generated keys. See S3 conditional writes.

A checksum proves consistency with the expected bytes. If the uploader supplied those expected bytes, it does not prove the file is trustworthy, harmless, or authorized for someone else to read.

Issue signed URLs just before the part runs

A presigned URL is a reusable bearer capability until it expires. Anyone holding it can exercise the signed operation within its restrictions. Keep credentials on the backend and remove signed query strings from logs, analytics, and error reports.

Sign only an authorized part of an active session, with its fixed checksum. Issue URLs in small batches or as workers need them. A URL may expire earlier than its requested lifetime when the underlying temporary credentials expire. Renew through the authenticated control API. These behaviors are covered in the presigned URL documentation.

Here is an AWS SDK for JavaScript v3 excerpt. session and part are loaded, authorized server records. The backend has already sealed the part plan; the browser cannot choose another key or change the expected hash.

import {
  S3Client, CreateMultipartUploadCommand, UploadPartCommand,
} from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const s3 = new S3Client({ region: process.env.AWS_REGION });

// Initiate once, then persist UploadId on the application session.
const created = await s3.send(new CreateMultipartUploadCommand({
  Bucket: session.bucket,
  Key: session.key,
  ContentType: session.approvedContentType,
  Metadata: { "app-upload-id": session.id },
  ChecksumAlgorithm: "SHA256",
  ChecksumType: "COMPOSITE",
}));

// A later request signs a part from the sealed, authorized plan.
const url = await getSignedUrl(s3, new UploadPartCommand({
  Bucket: session.bucket,
  Key: session.key,
  UploadId: session.s3UploadId,
  PartNumber: part.number,
  ChecksumSHA256: part.sha256Base64,
}), {
  expiresIn: 300,
  unhoistableHeaders: new Set(["x-amz-checksum-sha256"]),
});

const response = {
  url,
  headers: { "x-amz-checksum-sha256": part.sha256Base64 },
};

Set the object content type and session metadata when initiating the multipart upload. The presigner returns both the URL and the headers the client must send. Its unhoistableHeaders option keeps the checksum as a required signed header rather than moving it into the query string. See the official S3 request presigner documentation.

A 403 response does not automatically mean expiry. It can indicate incorrect headers, permissions, policy, or credentials. Reauthorize and renew when expiry is established; surface persistent signature or authorization failures instead of retrying them indefinitely.

CORS is part of the receipt contract

The browser must be allowed to send the checksum header and read the response ETag. A successful PUT without a readable ETag leaves the client unable to record the receipt, even though storage may have accepted the part.

An illustrative S3 CORS configuration for one application origin is:

[
  {
    "AllowedOrigins": ["https://app.example.com"],
    "AllowedMethods": ["PUT"],
    "AllowedHeaders": ["x-amz-checksum-sha256", "content-type"],
    "ExposeHeaders": ["ETag", "x-amz-checksum-sha256", "Retry-After"],
    "MaxAgeSeconds": 300
  }
]

Replace the example origin. Allow the actual request headers your signing contract uses. The browser performs preflight; OPTIONS is not an S3 AllowedMethods entry. CORS controls browser access to responses. Authorization still comes from the signed request and storage policy. See S3 CORS configuration.

Send the raw Blob slice as the PUT body. Wrapping it in FormData changes the payload. Do not manually set the browser’s Content-Length header or alter signed headers.

Retry one part with a budget and a cancellation signal

A fetch promise usually resolves for HTTP errors, so inspect the status. Network errors and timeouts leave the remote result unknown. Retry suitable transient failures with capped exponential backoff and full jitter. Honor a readable Retry-After as a minimum delay; stop if it exceeds the remaining operation budget.

The following worker hashes one bounded slice, obtains a fresh signed request for each attempt, and returns its receipt. Its issuePartUrl callback calls the authenticated backend, validates the response, and passes the supplied signal to its own fetch. It must reject a changed checksum or an inactive session.

const RETRYABLE = new Set([408, 429, 500, 502, 503, 504]);

class RetryableNetwork extends Error {}

async function networkRequest(send) {
  try { return await send(); }
  catch (error) {
    if (error instanceof TypeError) throw new RetryableNetwork(error.message, { cause: error });
    throw error;
  }
}

class RetryableResponse extends Error {
  constructor(response) {
    super(`Part upload returned HTTP ${response.status}`);
    const value = response.headers.get('Retry-After');
    const seconds = value !== null && /^\d+$/.test(value) ? Number(value) : NaN;
    const date = value ? Date.parse(value) : NaN;
    this.delayMs = Number.isFinite(seconds) ? seconds * 1000
      : Number.isFinite(date) ? Math.max(0, date - Date.now()) : 0;
  }
}

function wait(ms, signal) {
  signal.throwIfAborted();
  return new Promise((resolve, reject) => {
    const stop = () => {
      clearTimeout(timer);
      signal.removeEventListener('abort', stop);
      reject(signal.reason);
    };
    const timer = setTimeout(() => {
      signal.removeEventListener('abort', stop);
      resolve();
    }, ms);
    signal.addEventListener('abort', stop, { once: true });
  });
}

// issuePartUrl honors its signal. HTTP/auth/schema failures must use non-TypeError errors;
// reserve TypeError for fetch transport failures, including CORS failures.
export async function uploadPart({ blob, sessionId, partNumber, signal, issuePartUrl }) {
  signal.throwIfAborted();
  if (!(blob instanceof Blob) || blob.size === 0
      || !Number.isInteger(partNumber) || partNumber < 1 || partNumber > 10_000) {
    throw new Error('Expected a nonempty Blob and a valid part number');
  }
  const deadline = performance.now() + 5 * 60_000;
  // WebCrypto buffers one part. Choose a part size that fits browser memory.
  const digest = await crypto.subtle.digest('SHA-256', await blob.arrayBuffer());
  const checksumSHA256 = btoa(String.fromCharCode(...new Uint8Array(digest)));

  for (let attempt = 0; attempt < 4; attempt++) {
    signal.throwIfAborted();
    const remaining = deadline - performance.now();
    if (remaining <= 0) throw new Error('Part retry budget exhausted');
    const controller = new AbortController();
    const stop = () => controller.abort(signal.reason);
    signal.addEventListener('abort', stop, { once: true });
    const timer = setTimeout(() => controller.abort(
      new DOMException('Part attempt timed out', 'TimeoutError')
    ), Math.min(90_000, remaining));
    let delayMs = 0;
    try {
      const plan = await networkRequest(() => issuePartUrl({ sessionId, partNumber,
        checksumSHA256, signal: controller.signal }));
      controller.signal.throwIfAborted();
      const headers = new Headers(plan.headers);
      if (headers.get('x-amz-checksum-sha256') !== checksumSHA256) {
        throw new Error('Signing response does not match the part checksum');
      }
      const response = await networkRequest(() => fetch(plan.url, { method: 'PUT', body: blob,
        headers, signal: controller.signal, credentials: 'omit' }));
      if (response.body) await response.body.cancel().catch(() => {});
      controller.signal.throwIfAborted();
      if (RETRYABLE.has(response.status)) throw new RetryableResponse(response);
      if (!response.ok) throw new Error(`Part upload returned HTTP ${response.status}`);
      const etag = response.headers.get('ETag');
      if (!etag) throw new Error('ETag unavailable: check CORS and reconcile the part');
      return { partNumber, etag, checksumSHA256 };
    } catch (error) {
      signal.throwIfAborted();
      const timedOut = controller.signal.reason?.name === 'TimeoutError';
      if (!(error instanceof RetryableResponse || error instanceof RetryableNetwork || timedOut)
          || attempt === 3) throw error;
      const jitter = Math.random() * Math.min(8_000, 250 * 2 ** attempt);
      delayMs = Math.max(jitter, error instanceof RetryableResponse ? error.delayMs : 0);
    } finally {
      clearTimeout(timer);
      signal.removeEventListener('abort', stop);
    }
    if (delayMs >= deadline - performance.now()) throw new Error('Part retry budget exhausted');
    await wait(delayMs, signal);
  }
}

The callback must distinguish transport failures from HTTP or validation errors. Its authenticated control requests also need a bounded policy for transient HTTP responses; authorization and schema errors remain terminal. The timeout covers signing and transfer for an attempt. The overall deadline covers this part operation, not the entire large-file session. The example values need tuning for the minimum connection speed the product supports. A deadline that cannot accommodate one part on that connection will make every attempt fail; choose part size and concurrency together before sealing the plan. A paused browser can also delay timers; this is not a hard real-time deadline.

SHA-256 hashing reads one part into memory. SubtleCrypto.digest does not stream its input, and this hashing call is not itself abortable. Check cancellation before and after it, bound the part size, and move expensive hashing to a worker when needed. Never load the whole multi-gigabyte file into one ArrayBuffer. See Web Crypto digest.

The retry categories should remain explicit:

A deliberate user abort must not be retried. An attempt timeout may be retried if the session remains active and its budget allows it. CORS failures can resemble network failures in JavaScript; the attempt limit prevents a permanent configuration problem from becoming an endless loop. MDN explains fetch error behavior and abort signals.

Persist acknowledged progress and make resume verify identity

After a worker succeeds, report the part number, ETag, checksum, and expected length to the backend. Make the receipt endpoint idempotent. If its response is lost, retry that control request without uploading the bytes again. The backend validates receipts against the sealed plan and provider observations.

Keep a durable application manifest. S3’s ListParts is useful for verification, but it is not a substitute for the intended manifest. Follow pagination through every page; one response contains at most 1,000 parts. A lost upload acknowledgment can be resolved conservatively by reuploading the same validated slice to obtain a fresh ETag. Never adopt unexpected provider parts as intended content. See the ListParts API.

After a page reload, the browser still needs the original file bytes. Require file reselection unless a supported persisted file handle and its permission are available. Filename, size, and lastModified help the user choose; they do not prove content identity.

For strict resume across reloads, one straightforward design is to compute and seal a complete ordered manifest of part hashes before uploading. Recompute it for a reselected file and reject mismatches before combining old and new parts. This requires an initial pass over the whole file, but only one bounded part buffer at a time. A system that hashes only parts already attempted cannot prove the identity of unexamined bytes after reselection.

The worker above rechecks a slice before sending it. That can mean a second hashing pass after manifest creation. This is a deliberate correctness baseline with a CPU and I/O cost, not a claim that prehashing is free.

Show acknowledged bytes as durable progress. You may display in-flight transport progress separately, but bytes sent are not a storage receipt. Browser fetch does not offer the same upload progress events as XMLHttpRequest.upload. Reserve “Ready” for the completed and validated object, rather than reaching 100 percent transmitted bytes.

Completion is a backend operation with an uncertain-result path

Stop scheduling parts and await every local worker before requesting completion. The backend must also serialize its state transitions and account for other tabs or clients. Verify every planned part’s number, size, checksum, and current receipt; reject missing or unexpected content.

Freeze the application’s completion manifest in ascending part-number order. Include each part’s PartNumber, ETag, and ChecksumSHA256. The chosen composite checksum path requires consecutive numbers beginning with 1.

Call CompleteMultipartUpload from the backend through the SDK. S3 can initially return HTTP 200 and later put an error in the response body. A raw response.ok check is insufficient for this operation; SDKs handle that embedded error condition. See the completion API reference.

transactionally claim UPLOADING -> FINALIZING
persist the sorted completion manifest and operation identity
stop signing new parts
verify provider parts against the sealed application plan
call CompleteMultipartUpload through the backend SDK
if the outcome is unknown:
    keep FINALIZING and reconcile the exact object key
verify session metadata, length, and the correct checksum type
run required validation while the object remains private
mark READY only after the checks succeed

If completion times out, keep the session in a resolving state. Check the exact server-generated key with HeadObject. Match the backend-set session metadata, actual content length, and the expected checksum using its correct algorithm and type. Request checksum metadata explicitly, and provide the necessary KMS permissions when applicable. Retain the returned object version ID when bucket versioning is enabled.

A matching object can repair application state after storage succeeded but the database update failed. A 403 is not proof the object is absent. NoSuchUpload is also ambiguous: the upload might have completed, been aborted, or never existed. Keep retrying bounded reconciliation or raise an operational incident instead of guessing. AWS documents the metadata and permission behavior in HeadObject.

Keep the object private or quarantined until validation finishes. Depending on the product, readiness may require file signature checks, malware scanning, parsing, or a domain-specific import. The browser’s MIME type and declared size are claims to verify.

Pause and cancel are different operations

Pause stops scheduling and aborts local transfers while preserving the multipart session and acknowledged parts. Resume uses a new controller and reconciles uncertain parts. An already aborted signal cannot be reused.

Cancel asks the backend to terminate the session and release its provider resources. The UI can stop transferring immediately, but should show cleanup pending until the backend reaches a terminal result.

// This controller belongs to one active local upload run.
uploadController.abort(new DOMException("User cancelled", "AbortError"));

// Do not reuse uploadController.signal for the control request.
const result = await fetch(`/uploads/${sessionId}/cancel`, {
  method: "POST",
  credentials: "same-origin",
  signal: AbortSignal.timeout(10_000),
  headers: { "Idempotency-Key": cancelOperationId },
});
if (!result.ok) {
  // Keep cleanup pending; reconcile through GET /uploads/{id}.
  throw new Error(`Cancellation requires follow-up: ${result.status}`);
}
// Read the returned state: ABORTING is not yet ABORTED.

The cancel request must use a fresh signal. Reusing the upload’s aborted signal would cancel the cleanup request before it reaches the backend. If the device is offline or the tab closes, a session expiry job still needs to finish cleanup.

Make completion and cancellation compete for a conditional transition from UPLOADING. Only one intent wins. Once either begins, stop issuing new part URLs. Persist the winning operation before making the external storage call.

If completion wins, cancellation must report that finalization has already started or return the current completed result. It must not claim the file was rolled back. Deleting an already completed object is a separate product operation with its own authorization and version checks.

This database decision does not create a transaction with S3. An expired worker lease, a crash after an external call, or an already issued URL can still complicate the result. Serialize finalization work, keep requests repeatable, and reconcile the provider state. A URL already handed out does not become invalid merely because the application row changed.

Aborting requests is not the same as aborting multipart storage

AbortController.abort() stops browser work. It does not invoke S3’s AbortMultipartUpload, remove acknowledged parts, or undo a completed object.

After cancellation wins, the backend aborts the provider upload. In-flight part requests can race that operation and still succeed or fail. AWS notes that repeated abort calls may be needed, followed by checking parts to verify cleanup. Treat NoSuchUpload as evidence to reconcile with the session’s completion history, not as universal proof of cancellation. See the abort API reference.

Configure AbortIncompleteMultipartUpload lifecycle cleanup as a safety net. Its retention period must exceed the resume window the product promises, with operational margin. Lifecycle is asynchronous cleanup for abandoned parts; it is not the immediate Cancel interaction, and it does not delete completed objects. See the incomplete upload lifecycle guide.

Bound resource use and test the inconvenient failures

Set limits for declared and actual file size, active sessions per user, part count, concurrent workers, URL issuance, retries, and session lifetime. Verify actual part sizes before completion and the final object length afterward. Signing an UploadPart URL alone does not enforce the application’s total-size quota. Strict limits on temporary storage exposure need provider-enforced size controls or a suitable gateway as well as application policy.

Track acknowledged throughput, retry causes, URL refresh failures, unknown completion outcomes, age of resolving sessions, abandoned part storage, and cleanup backlog. Log the application session ID and part number, never the signed URL.

I would test the following before calling the feature ready:

• Drop a response after storage accepts a part. Retry the same slice and number without duplicating or changing content.
• Remove ETag from exposed CORS headers. The UI must not report a durable receipt it cannot read.
• Expire a URL and its underlying credentials. Check renewal and persistent authorization errors separately.
• Cancel during hashing, PUT, retry backoff, and receipt reporting. Stop local work, preserve uncertainty, and verify provider cleanup.
• Reload and select a different file with matching name and size. The integrity check must reject mixing files.
• Use more than 1,000 parts. Verify pagination and the complete intended manifest.
• Race complete and cancel from separate tabs. One backend intent must win and the other must report the actual outcome.
• Crash after S3 completion but before the application update. Reconciliation must recover the correct object and readiness state.
• Exercise embedded completion errors, failed scans, and abandoned sessions. Confirm they cannot become silently available files.

The practical design rule

Multipart upload reduces the amount of work a failed transfer must repeat. Signed URLs authorize direct data transfer. ETags record accepted parts. Checksums verify byte integrity. The backend’s session and finalization protocol make those pieces recoverable.

Retry an operation only when its identity is stable. Reconcile when its result is unknown. Treat cancellation as a durable request to clean up, not as proof that the server did nothing.

That is the foundation I would build before adding smoother progress animations, adaptive concurrency, or more storage providers. For related backend design articles, browse Best Practices and all engineering articles.