S3多段上传Multipart与完整性校验实践

概述目标：以多段上传并行加速大文件传输，结合校验和与ETag确保数据完整与可靠存储。适用：视频/归档/模型等大对象上传。核心与实战CLI多段上传（`s3api`）：aws s3api create-multipart-upload --bucket my-bucket --key big.bin --checksum-algorithm SHA256

# 返回 UploadId，随后分段上传：

aws s3api upload-part --bucket my-bucket --key big.bin --part-number 1 --body part1.bin --upload-id <UploadId> --checksum-sha256 <base64sha256>

aws s3api upload-part --bucket my-bucket --key big.bin --part-number 2 --body part2.bin --upload-id <UploadId> --checksum-sha256 <base64sha256>

# 汇总各分段ETag与Checksum完成上传：

aws s3api complete-multipart-upload --bucket my-bucket --key big.bin --upload-id <UploadId> --multipart-upload '{"Parts":[{"ETag":"\"etag1\"","PartNumber":1},{"ETag":"\"etag2\"","PartNumber":2}]}'

Node.js SDK v3并行上传：import { S3Client, Upload } from "@aws-sdk/lib-storage"

const client = new S3Client({ region: "us-east-1" })

const upload = new Upload({

client,

params: { Bucket: "my-bucket", Key: "big.bin", Body: fs.createReadStream("big.bin") },

queueSize: 4,

partSize: 8 * 1024 * 1024,

leavePartsOnError: false

})

upload.on("httpUploadProgress", (p) => console.log(p))

await upload.done()

示例完整性校验：aws s3api head-object --bucket my-bucket --key big.bin

# 检查ETag与校验和字段（ChecksumSHA256/CRC32C）

失败重试：-- 根据返回的分段信息仅重传失败段，避免重复上传

验证与监控传输性能：调整`partSize/queueSize`平衡并发与内存；观察上传速率与失败重试。完整性：对照本地分段校验和与S3对象校验和；确保一致。成本与存储：避免长时间未完成导致保留分段占用；定期`abort-multipart-upload`清理。常见误区误用ETag当作SHA1/MD5；多段上传的ETag不是简单的MD5，需使用校验和字段。分段过小导致请求过多与开销；应选择合理分段大小（≥8MB）。忽视`abort`导致未完成分段长期占用成本；需清理失败会话。结语通过多段上传与校验和验证可实现大对象高效可靠传输，并以监控与清理策略保障成本与可维护性。