[Bug]: Syncing to S3 Storage (e.g. minio server) syncs folders twice into a subtree

[Section1977]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

look below

Steps to reproduce

1.Minio S3 Storage provide a bucket
2. Syncing to bucket with NExtcloud 27 syncs folders twice into a subtree
3. means /example plus content of folder syncs to /example plus content of folder AND /example/example plus content of folder
4. you see the bug only on the S3 storage side, not within Nextcloud !
5. pydio client e.g. does not make this bug in S3 structure, so it is a bug of nextcloud

Expected behavior

syncs only once with mirroring exactly the folder structure

Installation method

Community Docker image

Nextcloud Server version

27

Operating system

Debian/Ubuntu

PHP engine version

None

Web server

Nginx

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

No response

List of activated Apps

No response

Nextcloud Signing status

No response

Nextcloud Logs

No response

Additional info

No response

[joshtrichards]

What do you mean by "syncing"? Please describe how, specifically, you have integrated S3 into your Nextcloud environment.

E.g.

• As External Storage: https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/external_storage/amazons3.html

• As Primary Storage: https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/primary_storage.html

It would also be helpful if you fill out the entire issue template as requested.

[Section1977]

hi, minio s3 storage bucket is connected by external storage app.
Syncing with Nextcloud Client PC folder directly to external storage S3.
The problematic occurs when creating a folder on synced PC folder. This makes the folder and a ghost folder in subtree on S3

[joshtrichards]

This sounds like a configuration matter or, I suppose, possibly an issue with whichever client you're using.

We do all our testing primarily with MinIO so it's unlikely a bug like this would occur and make it to release (but anything is possible).

You've still failed to provide enough information to attempt to reproduce this. There are thousands of possible configuration combinations.

Please provide the configuration of your External Storage mount. Either a screen shot from your External Storage config section or the output of occ config:list files_external (or equivalent).

Please also provide screenshot from the MinIO Console Object Browser with "the folder and a ghost folder in subtree" that you're seeing.

EDIT: Oops I mean the output of occ files_external:list --output=json_pretty.

[Section1977]

{
    "apps": {
        "files_external": {
            "enabled": "yes",
            "installed_version": "1.19.0",
            "types": "filesystem"
        }
    }
}

[Section1977]

[joshtrichards]

Sorry my mistake: Please send the output of occ files_external:list --output=json_pretty (I gave you the wrong command)

[Section1977]

thanks for your help

[
    {
        "mount_id": 2,
        "mount_point": "\/test",
        "storage": "\\OCA\\Files_External\\Lib\\Storage\\AmazonS3",
        "authentication_type": "amazons3::accesskey",
        "configuration": {
            "bucket": "test",
            "hostname": "minio.server.lan",
            "port": "80",
            "region": "eu-central-1",
            "storageClass": "",
            "use_ssl": false,
            "use_path_style": true,
            "legacy_auth": false,
            "key": "secret",
            "secret": "secret"
        },
        "options": {
            "encrypt": true,
            "previews": true,
            "enable_sharing": false,
            "filesystem_check_changes": 1,
            "encoding_compatibility": false,
            "readonly": false
        },
        "applicable_users": [],
        "applicable_groups": []
    }
]

[joshtrichards]

Reproduced. Thanks for the report! 👍

Only the folder placeholder appears to get created. I don't see duplicated content fortunately.

Did you notice this behavior after an upgrade? (If so it might help us figure out what version of NC introduced this regression).

[Section1977]

if you download the content of the "empty" folder you get content :-)
it is twice.
never seen this before because I am new to S3

[joshtrichards]

Actually I think this is just the 0 byte placeholder object created by the S3 client to represent folders. Despite how the MinIO browser shows it, I see no indication of duplicate content. If you download the parent (in MinIO) there is no duplicated content.

Since S3 doesn't support folders natively, there are several ways of simulating the concept of folders in it. And, in turn, several ways of displaying in in S3 explorer clients. The core functionality is working appropriately here best as I can tell. This is merely a cosmetic misdirection within the MinIO browser. The extra folder is just the placeholder object that represents the folder.

Technically it only needs to exist until the folder has something else in it... In theory we could clear out the placeholder after the "folder" gets populated with files I guess... but that's not now AWS does it AFAIK. And we follow their convention. Also it would mean that if you delete the files in the folder, the folder would suddenly magically disappear. That wouldn't be very friendly IMO.

The reason this doesn't happen when creating a folder in the MinIO browser itself is because the approach they take seems to be to not really create a placeholder for a folder. Instead, if you create a "folder" in the browser and don't immediately put something in it, it never truly gets created. It doesn't look that way, but try to create a folder in the MinIO browser then go back to the parent of your bucket. The folder won't exist.

We actually create a placeholder.

Not familiar with pydio directly, but seems they suggest doing something similar if you want empty folders:

e.g.
https://forum.pydio.com/t/how-to-create-a-folder-without-files-s3-postman/2293

[nextcloud-command]

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.

[Section1977]

s3 doesnt support folders
so when the folder is empty you can automatically delete the folder and follow s3 convention.
please make another solution for creating folders but not this way. it is more than irritating.
pydio use hidden file to leqve folder but removing them if empty is also very useful.