Local to S3 upload limitation. 5G #2

STGSC · 2023-02-27T02:48:40Z

Thank you very much for providing this project. I have encountered an issue where it prompts me that the file I am trying to upload exceeds the size limit. After reviewing the AWS documentation, I found that using the SDK for uploading supports a single file size of 5G, but it also provides a method for segmented uploading. I am looking forward to the implementation of this feature in the project. Cheers!

https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/php/example_code/s3/MultipartUpload.php

Next Aws\S3\Exception\S3Exception: Error executing "PutObject" on "https://xxxx-nextcloud.s3-xxxxxx-1.amazonaws.com/urn%3Aoid%3A261344"; AWS HTTP error: Client error: `PUT https://xxxx-nextcloud.s3-xxxxxx-1.amazonaws.com/urn%3Aoid%3A261344` resulted in a `400 Bad Request` response:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>EntityTooLarge</Code><Message>Your proposed upload exceeds the maxim (truncated...)
 EntityTooLarge (client): Your proposed upload exceeds the maximum allowed size - <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>EntityTooLarge</Code><Message>Your proposed upload exceeds the maximum allowed size</Message><ProposedSize>6161602382</ProposedSize><MaxSizeAllowed>5368709120</MaxSizeAllowed><RequestId>XW298HH7XXXXST9A</RequestId><HostId>1PcidcefrBv3q5KxpYb+v0qWyeZJr1tXBqAIpx1vbEwPQxB666JgckDsTNHP8VUbIZtLzcosuJ0=</HostId></Error> in /root/nextcloud-S3-local-S3-migration-main/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:195
Stack trace:
#0 /root/nextcloud-S3-local-S3-migration-main/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php(97): Aws\WrappedHttpHandler->parseError()
#1 /root/nextcloud-S3-local-S3-migration-main/vendor/guzzlehttp/promises/src/Promise.php(204): Aws\WrappedHttpHandler->Aws\{closure}()
#2 /root/nextcloud-S3-local-S3-migration-main/vendor/guzzlehttp/promises/src/Promise.php(169): GuzzleHttp\Promise\Promise::callHandler()
#3 /root/nextcloud-S3-local-S3-migration-main/vendor/guzzlehttp/promises/src/RejectedPromise.php(42): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}()
#4 /root/nextcloud-S3-local-S3-migration-main/vendor/guzzlehttp/promises/src/TaskQueue.php(48): GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}()
#5 /root/nextcloud-S3-local-S3-migration-main/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(159): GuzzleHttp\Promise\TaskQueue->run()
#6 /root/nextcloud-S3-local-S3-migration-main/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(184): GuzzleHttp\Handler\CurlMultiHandler->tick()
#7 /root/nextcloud-S3-local-S3-migration-main/vendor/guzzlehttp/promises/src/Promise.php(248): GuzzleHttp\Handler\CurlMultiHandler->execute()
#8 /root/nextcloud-S3-local-S3-migration-main/vendor/guzzlehttp/promises/src/Promise.php(224): GuzzleHttp\Promise\Promise->invokeWaitFn()
#9 /root/nextcloud-S3-local-S3-migration-main/vendor/guzzlehttp/promises/src/Promise.php(269): GuzzleHttp\Promise\Promise->waitIfPending()
#10 /root/nextcloud-S3-local-S3-migration-main/vendor/guzzlehttp/promises/src/Promise.php(226): GuzzleHttp\Promise\Promise->invokeWaitList()
#11 /root/nextcloud-S3-local-S3-migration-main/vendor/guzzlehttp/promises/src/Promise.php(269): GuzzleHttp\Promise\Promise->waitIfPending()
#12 /root/nextcloud-S3-local-S3-migration-main/vendor/guzzlehttp/promises/src/Promise.php(226): GuzzleHttp\Promise\Promise->invokeWaitList()
#13 /root/nextcloud-S3-local-S3-migration-main/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#14 /root/nextcloud-S3-local-S3-migration-main/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(58): GuzzleHttp\Promise\Promise->wait()
#15 /root/nextcloud-S3-local-S3-migration-main/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(86): Aws\AwsClient->execute()
#16 /root/nextcloud-S3-local-S3-migration-main/localtos3.php(793): Aws\AwsClient->__call()
#17 /root/nextcloud-S3-local-S3-migration-main/localtos3.php(556): S3put()
#18 {main}
  thrown in /root/nextcloud-S3-local-S3-migration-main/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php on line 195

The text was updated successfully, but these errors were encountered:

STGSC · 2023-02-27T07:30:05Z

Although I am not familiar with the PHP language, ChatGPT and NewBing helped me complete the following content, and I am running it now. I am not sure if there are any other issues.

Add

use Aws\S3\MultipartUploader;

Set S3put()

function S3put($s3, $bucket, $vars = array()) {
  if (is_string($vars)) {
    if (file_exists($vars)) {
      $vars = array('SourceFile' => $vars);
    } else {
      return 'ERROR: S3put($cms, $bucket, $vars)';
    }
  }
  if (empty($vars['Bucket'])) {
    $vars['Bucket'] = $bucket;
  }
  if (empty($vars['Key']) && !empty($vars['SourceFile'])) {
    $vars['Key'] = $vars['SourceFile'];
  }

  if (empty($vars['Bucket'])) {
    return 'ERROR: no Bucket';
  }
  if (empty($vars['Key'])) {
    return 'ERROR: no Key';
  }

  if (empty($vars['ACL'])) {
    $vars['ACL'] = 'private';
  }

  try {
    $uploader = new MultipartUploader(
      $s3,
      $vars['SourceFile'],
      [
        'bucket' => $bucket,
        'key' => $vars['Key'],
        'acl' => 'private'
      ]
    );

    $result = $uploader->upload();

    if (!empty($result['ObjectURL'])) {
      return 'OK: ' . 'ObjectURL: ' . $result['ObjectURL'];
    } else {
      return 'ERROR: ' . $vars['Key'] . ' was not uploaded';
    }
  } catch (MultipartUploadException | S3Exception | Exception $e) {
    return 'ERROR: ' . $e->getMessage();
  }
}

mrAceT · 2023-02-27T13:31:26Z

added support for 'MultipartUploader', could you test?

(un-comment line 17 & 18 and keep/set the rest of your config naturally)

PS: let me know how it went (in general)

STGSC · 2023-02-28T01:41:54Z

added support for 'MultipartUploader', could you test? > > (un-comment line 17 & 18 and keep/set the rest of your config naturally) > > > PS: let me know how it went (in general)

Thanks for the update, I'm testing it on a 2Tb sized Nextcloud project
I'm not sure how to quickly verify that it works. I have to run Test=0 to see if it works after database changes, the wait is long .....

mrAceT · 2023-02-28T08:52:25Z

Yes it'll be a serious wait!

When the script runs and you 'at most' see a warning here and there and (eventually :P) it completes, then you should be able to simply set $TEST to 'zero'. then the database settings will be converted to S3 usage.

It shouldn't, but if it does fail, simply restore the SQL-backup (do first go to maintenance mode ON!) and you are reset to "local usage".. so your data should be safe :)

And, well, yea.. you will only know if it all went well, when 'test=1' has completed.. (and only then re run with test=0)
It might take a few days ;)
And if it hangs for some reason (that's why i built a simple progress indication) simply cancel it and restart.. it'll check what has been done and continue where it left off..)

mrAceT · 2023-03-01T10:48:21Z

Let me know if you run in any (more) trouble

STGSC · 2023-03-02T01:24:03Z

Why does running this script on a suspended project prompt some files on S3, but is older then local, upload...
and that these files are fixed.They are the ones every time.

mrAceT · 2023-03-02T09:00:03Z

Hmm, if the upload succeed then there must be a discrepancy in your database? The script uploads the file again if the timestamp of the file is more recent then the timestamp of that file in your database..

have you set $SHOWINFO = 1 and $TEST anything but 0?

I suspect that's it, since it's the message "XX/YY.zz on S3, but is older then local, upload..."

You could do 'occ files:scan --all' (or set $DO_FILES_SCAN=1), also setting $DO_FILES_CLEAN =1 (occ files:cleanup) might be a good idea.. those are "check & clean" operations, that can help in cleaning up discrepancies..

AD: Running those two "set to 1" once should be enough.. it'll fix discrepancies.. once fixed..

STGSC · 2023-03-02T09:08:29Z

I didn't pay too much attention to this error, because he would just make me waste some traffic.
All data is now synced and finished. Great, I found it working properly!
If I want to migrate my nextclou web service, in theory I just need to import SQL and overwrite this config.php in the new environment, right?

mrAceT · 2023-03-02T09:13:27Z

Ehm, I don't understand that question completely..

If everything went well with $TEST=1, you can set $TEST=0.. then it'll do a final check & sync (which should go quick) and perform all the database changes to point your Nextcloud from local to S3 and the migration is completed!

AD: do keep in mind that the script will put your instance in maintenance mode! Users can not use the instance while in maintenance mode!

Don't do this "by hand", the order "of things" is important! The script does that for you! There are a few ways you can use the script, so do carefully read what the script tells you.. sometimes you need to do something by hand (again, the order of things is important!!)

AD: this is "the big one".. if by any chance it fails, simply restore the backup and you will have reverted to a working local operation (and tell me what went wrong ;)

STGSC · 2023-03-02T09:18:37Z

I have verified that the MultipartUpload is valid. Thank you again for providing this project.

mrAceT · 2023-03-02T09:20:13Z

Migration completed? You are now "live with S3"?

STGSC · 2023-03-02T11:44:48Z

Yes, the migration is now complete, and the data is in S3.

…

----------Reply to Message---------- On Thu, Mar 2, 2023 17:20 PM ***@***.***> wrote: Migration completed? You are now "live with S3"? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

mrAceT closed this as completed Mar 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Local to S3 upload limitation. 5G #2

Local to S3 upload limitation. 5G #2

STGSC commented Feb 27, 2023 •

edited

Loading

STGSC commented Feb 27, 2023

mrAceT commented Feb 27, 2023 •

edited

Loading

STGSC commented Feb 28, 2023

mrAceT commented Feb 28, 2023 •

edited

Loading

mrAceT commented Mar 1, 2023

STGSC commented Mar 2, 2023

mrAceT commented Mar 2, 2023 •

edited

Loading

STGSC commented Mar 2, 2023

mrAceT commented Mar 2, 2023 •

edited

Loading

STGSC commented Mar 2, 2023

mrAceT commented Mar 2, 2023

STGSC commented Mar 2, 2023 via email

Local to S3 upload limitation. 5G #2

Local to S3 upload limitation. 5G #2

Comments

STGSC commented Feb 27, 2023 • edited Loading

https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/php/example_code/s3/MultipartUpload.php

STGSC commented Feb 27, 2023

Although I am not familiar with the PHP language, ChatGPT and NewBing helped me complete the following content, and I am running it now. I am not sure if there are any other issues.

Add

Set S3put()

mrAceT commented Feb 27, 2023 • edited Loading

STGSC commented Feb 28, 2023

mrAceT commented Feb 28, 2023 • edited Loading

mrAceT commented Mar 1, 2023

STGSC commented Mar 2, 2023

mrAceT commented Mar 2, 2023 • edited Loading

STGSC commented Mar 2, 2023

mrAceT commented Mar 2, 2023 • edited Loading

STGSC commented Mar 2, 2023

mrAceT commented Mar 2, 2023

STGSC commented Mar 2, 2023 via email

STGSC commented Feb 27, 2023 •

edited

Loading

mrAceT commented Feb 27, 2023 •

edited

Loading

mrAceT commented Feb 28, 2023 •

edited

Loading

mrAceT commented Mar 2, 2023 •

edited

Loading

mrAceT commented Mar 2, 2023 •

edited

Loading