aws-s3 plugin reports bytesUploaded larger than bytesTotal

[thecodewarrior]

Initial checklist

• I understand this is a bug report and questions should be posted in the Community Forum
• I searched issues and couldn’t find anything (or linked relevant results below)

Link to runnable example

No response

Steps to reproduce

1. Create an uploader using aws-s3 (I used companion but you can use whatever you want)
2. Add the event hooks listed below
3. Upload a small file (the effect still occurs for large files, but is much more pronounced for small ones)

const PROGRESS_FORMAT = "%c%s - %c%s%c%s %c%s%c%s %c%s%c%s %c%s%c%s %c%s%c%s"
function progressLog(source, progress, bytesPadding) {
  return [
    "font-weight: bold;", source,
    "color: #888;", "bytesTotal: ", "", (progress.bytesTotal?.toString() ?? '-').padStart(bytesPadding),
    "color: #888;", "bytesUploaded: ", "", (progress.bytesUploaded?.toString() ?? '-').padStart(bytesPadding),
    "color: #888;", "percentage: ", "", (progress.percentage?.toString() ?? '-').padStart(3),
    "color: #888;", "uploadComplete: ", "", (progress.uploadComplete?.toString() ?? '-').padEnd(5),
    "color: #888;", "uploadStarted: ", "", progress.uploadStarted?.toString() ?? '-',
  ]
}

uppyDashboard.on('upload-progress', (file, argProgress) => {
  const fileProgress = file.progress
  const lookupFile = uppyDashboard.getFile(file.id)
  const lookupProgress = lookupFile.progress
  const bytesPadding = file.size.toString().length

  console.log(`%c%s%c%s\n${PROGRESS_FORMAT}\n${PROGRESS_FORMAT}\n${PROGRESS_FORMAT}`,
      "font-weight: bold; color: #55a64a", "upload-progress: ", "", file.name,
      ...progressLog("progress arg", argProgress, bytesPadding),
      ...progressLog("    file arg", fileProgress, bytesPadding),
      ...progressLog("   getFile()", lookupProgress, bytesPadding),
  )
})

uppyDashboard.on('upload-success', (file, response) => {
  const fileProgress = file.progress
  const lookupFile = uppyDashboard.getFile(file.id)
  const lookupProgress = lookupFile.progress
  const bytesPadding = file.size.toString().length

  console.log(`%c%s%c%s\n${PROGRESS_FORMAT}\n${PROGRESS_FORMAT}`,
      "font-weight: bold; color: #55a64a", "upload-success: ", "", file.name,
      ...progressLog("    file arg", fileProgress, bytesPadding),
      ...progressLog("   getFile()", lookupProgress, bytesPadding),
  )
});

Expected behavior

• The progress object should be the same no matter how you access it (possibly excluding the getFile(), if that's being updated less frequently)
• The progress object should be complete
• bytesUploaded should never be larger than bytesTotal

Actual behavior

• The progress value is different depending on how it's accessed, with both the progress argument and file.progress being in incomplete states. (This is probably related to Outdated file state during upload-progress event due to setState implementation #4593)
• bytesUploaded reliably exceeds bytesTotal, with the effect being especially pronounced for small files.

Here's an example for a 3.5kb file:

upload-progress: image_000.jpg
  progress arg - bytesTotal: 3574 bytesUploaded: 6163 percentage:   - uploadComplete: -     uploadStarted: -
      file arg - bytesTotal: 3574 bytesUploaded:    0 percentage:   0 uploadComplete: false uploadStarted: -
     getFile() - bytesTotal: 3574 bytesUploaded: 6163 percentage: 172 uploadComplete: false uploadStarted: 1706642090496

upload-progress: image_000.jpg
  progress arg - bytesTotal: 3574 bytesUploaded: 3574 percentage:   - uploadComplete: -     uploadStarted: -
      file arg - bytesTotal: 3574 bytesUploaded:    0 percentage:   0 uploadComplete: false uploadStarted: -
     getFile() - bytesTotal: 3574 bytesUploaded: 3574 percentage: 100 uploadComplete: false uploadStarted: 1706642090496

upload-success: image_000.jpg
      file arg - bytesTotal: 3574 bytesUploaded: 3574 percentage: 100 uploadComplete: false uploadStarted: 1706642090496
     getFile() - bytesTotal: 3574 bytesUploaded: 3574 percentage: 100 uploadComplete: true  uploadStarted: 1706642090496

This has been reproduced in Firefox, Chrome, and Safari.

I think the issue is caused by this code, which directly interprets the ProgressEvent.loaded value as the currently uploaded byte count, which apparently is incorrect in this case.

Looking in the network inspector it seems like the loaded value corresponds to the size of the POST request body, which is larger than the file due to ~2.5kb of extra data:

-----------------------------325560411535172556181726533394
Content-Disposition: form-data; name="success_action_status"

201
-----------------------------325560411535172556181726533394
Content-Disposition: form-data; name="content-type"

image/jpeg
-----------------------------325560411535172556181726533394
Content-Disposition: form-data; name="x-amz-meta-name"

image_000.jpg
-----------------------------325560411535172556181726533394
Content-Disposition: form-data; name="x-amz-meta-type"

image/jpeg
-----------------------------325560411535172556181726533394
Content-Disposition: form-data; name="x-amz-meta-dfs_uploadName"

352ec934-bddb-4526-9486-a20653d88b55.jpg
-----------------------------325560411535172556181726533394
Content-Disposition: form-data; name="bucket"

<...>
-----------------------------325560411535172556181726533394
Content-Disposition: form-data; name="X-Amz-Algorithm"

AWS4-HMAC-SHA256
-----------------------------325560411535172556181726533394
Content-Disposition: form-data; name="X-Amz-Credential"

<...>
-----------------------------325560411535172556181726533394
Content-Disposition: form-data; name="X-Amz-Date"

20240130T194952Z
-----------------------------325560411535172556181726533394
Content-Disposition: form-data; name="key"

352ec934-bddb-4526-9486-a20653d88b55.jpg
-----------------------------325560411535172556181726533394
Content-Disposition: form-data; name="Policy"

<...>
-----------------------------325560411535172556181726533394
Content-Disposition: form-data; name="X-Amz-Signature"

9b95764409c02985d7fa3b9c5696b5ee95d46630d5c3a8382bfb353fbf5c94ce
-----------------------------325560411535172556181726533394
Content-Disposition: form-data; name="file"; filename="image_000.jpg"
Content-Type: image/jpeg

If we assume all the extra data comes before the file (pretty safe imo) the calculation could change to this:

const metadataSize = ev.total - fileSize
{
  bytesUploaded: Math.max(0, ev.loaded - metadataSize),
  bytesTotal: fileSize
}

That or it could be computed using the fraction:

{
  bytesUploaded: Math.ceil(ev.loaded / ev.total * fileSize),
  bytesTotal: fileSize
}