Nextcloud and S3

[ldapomni]

Nextcloud version: 12.0.0.3

Nextcloud is connected to S3. When copying files using aws s3 php sdk in a directory bucket,Nextcloud does not see them. But if you look through the content of S3 Browser, everything is there. If the files to burn through nextcloud, no problem. What could be the problem? Some kind of analogue occ filesystem_external:scan ?

[ldapomni]

ps: Premissions is full.
ps2: Directory and files visible if create path with step by step
league\flysystem adapter s3

$fs_s3->createDir("../dir1");
$fs_s3->createDir("../dir1/subdir2");
$fs_s3->createDir("../dir1/subdir2/subdir3/");
$fs_s3->write("../dir1/subdir2/subdir3/test.txt");
all well and nextcloud displays file

if create file as $fs_s3->write("../dir2/subdir5/subdir6/test.txt")
the file is created but nextcloud does not display (s3 browser sees it)

[nickvergessen]

So you are uploading files directly to S3 and not through Nextcloud?

If so, in the options of the external storage definition there is a dropdown with different options:

[ldapomni]

Options "Check for changes" set.
also php function aws s3 sdk $cont=$fs_s3->ListContents('',true); work fine and return list of items.

when NC refers to the directory in ceph radosgw log shows that reading the contents of the directory is, but for some reason, NC doesn't display them

[ldapomni]

For test: added
file_put_contents ('dir to file test',print_r($object,true),FILE_APPEND); 240 line opendir() function
/apps/file_external/lib/Lib/Storage/AmazonS3.php.
result - directory present. NC doesn't display them.

Array
(
    [Key] => file1.txt
    [LastModified] => 2017-10-26T05:34:39.735Z
    [ETag] => "355884b5c38c11828b51aafb7c543581"
    [Size] => 8
    [StorageClass] => STANDARD
    [Owner] => Array
        (
            [ID] => rados
            [DisplayName] => S3 cloud
        )

)
Array
(
    [Key] => test_20171026093814.txt
    [LastModified] => 2017-10-26T05:38:14.789Z
    [ETag] => "23a398553ab3b06032dc7badcca0eaf4"
    [Size] => 3508
    [StorageClass] => STANDARD
    [Owner] => Array
        (
            [ID] => rados
            [DisplayName] => S3 cloud
        )

)
Array
(
    [Prefix] => dir4/
)
Array
(
    [Prefix] => test1/
)
Array
(
    [Prefix] => test2/
)

dir4 and subdir write external scripts

[ldapomni]

for some reason, the function filetype($path) AmazonS3.php return false on 'dir4'
$this->getConnection()->doesObjectExist($this->bucket,$path.'/') return false, but true for subdir and dir created with NC

NC log:
Debug OC\Files\Cach.. !!! Path 'dir4' is not accessible or present !!!
Why?

[ldapomni]

If use code from league\flysystem AwS3Adapter.php to determine directory -

 protected function doesDirectoryExist($location)
    {
        // Maybe this isn't an actual key, but a prefix.
        // Do a prefix listing of objects to determine.
        $command = $this->s3Client->getCommand(
            'listObjects',
            [
                'Bucket' => $this->bucket,
                'Prefix' => rtrim($location, '/') . '/',
                'MaxKeys' => 1,
            ]
        );

        try {
            $result = $this->s3Client->execute($command);

            return $result['Contents'] || $result['CommonPrefixes'];
        } catch (S3Exception $e) {
            if ($e->getStatusCode() === 403) {
                return false;
            }

            throw $e;
        }
    }

and change AmazonS3.php function filetype (file/dir)
work is ok.

[default50]

I'm having the exact same issue (NC 12.0.3). I do have the Check for changes: Once every direct access like @nickvergessen mentioned, which makes access much slower, but I do get to see the files/dirs I create straight in S3.

However, a whole tree that I synced (i.e. aws s3 sync ...) from a different bucket goes unnoticed in NC.

• If I add new files to one of those ignored folders, nothing changes.
• If I move one of those folders under another one previously created from S3 (and present in NC), still not visible.
• If I copy files inside one of the invisible folders into a visible one, they show up fine.

---

@ldapomni I've spotted the filetype() function in file /var/www/nextcloud/apps/files_external/lib/Lib/Storage/AmazonS3.php

288         public function filetype($path) {
289                 $path = $this->normalizePath($path);
290 
291                 if ($this->isRoot($path)) {
292                         return 'dir';
293                 }
294 
295                 try {
296                         if ($this->getConnection()->doesObjectExist($this->bucket, $path)) {
297                                 return 'file';
298                         }
299                         if ($this->getConnection()->doesObjectExist($this->bucket, $path.'/')) {
300                                 return 'dir';
301                         }
302                 } catch (S3Exception $e) {
303                         \OCP\Util::logException('files_external', $e);
304                         return false;
305                 }
306 
307                 return false;
308         }

However, I'm not sure which changes I should made based on your prior comment to test. Also, how to debug this further and gather the needed info?

[MorrisJobke]

cc @icewind1991

[icewind1991]

I can't reproduce the problem with master (to be NC13) and amazon S3, when creating a folder and adding a file to it trough the aws console, and running occ files:scan it properly detects the newly added folder and file.

Can anyone experiencing this problem try either the lastest NC13 beta (as always when using beta software: backup) or 12.0.4 which should include the same s3 changes.

[ldapomni]

After update to NC13, problem still exists.
@sporykhin some error after occ files:scan.

[velimirg]

Same problem here, I work normally with S3 folder/files created through NC or directly via AWS S3 GUI, but not with files uploaded with the aws cli. I believe it's somehow connected to the error I keep seeing in the log:

[kruffin]

I have a very similar error when trying to delete a folder from an S3 bucket within NC.

• Invalid argument supplied for foreach() at /snap/nextcloud/5132/htdocs/apps/files_external/lib/Lib/Storage/AmazonS3.php#291
• [Delete] is missing and is a required parameter

Nextcloud 12.0.5

"Exception": "InvalidArgumentException", "Message": "Found 1 error while validating the input provided for the DeleteObjects operation:\\n[Delete] is missing and is a required parameter",
#0 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/Middleware.php(79): AwsApiValidator->validate('DeleteObjects', Object(AwsApiStructureShape), Array)
#1 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php(421): AwsMiddleware::Aws{closure}(*** sensitive parameters replaced ***)
#2 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php(444): AwsS3S3Client::AwsS3{closure}(*** sensitive parameters replaced ***)
#3 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php(378): AwsS3S3Client::AwsS3{closure}(*** sensitive parameters replaced ***)
#4 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php(397): AwsS3S3Client::AwsS3{closure}(*** sensitive parameters replaced ***)
#5 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/Middleware.php(55): AwsS3S3Client::AwsS3{closure}(*** sensitive parameters replaced ***)
#6 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/S3/SSECMiddleware.php(59): AwsMiddleware::Aws{closure}(*** sensitive parameters replaced ***)
#7 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/IdempotencyTokenMiddleware.php(77): AwsS3SSECMiddleware->__invoke(Object(AwsCommand), NULL)
#8 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php(65): AwsIdempotencyTokenMiddleware->__invoke(Object(AwsCommand))
#9 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php(59): AwsAwsClient->executeAsync(Object(AwsCommand))
#10 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php(78): AwsAwsClient->execute(Object(AwsCommand))
#11 /snap/nextcloud/5132/htdocs/apps/files_external/lib/Lib/Storage/AmazonS3.php(255): AwsAwsClient->__call('deleteObjects', Array)
#12 /snap/nextcloud/5132/htdocs/apps/files_external/lib/Lib/Storage/AmazonS3.php(224): OCAFiles_ExternalLibStorageAmazonS3->batchDelete('2018/02')
#13 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Wrapper.php(87): OCAFiles_ExternalLibStorageAmazonS3->rmdir('2018/02')
#14 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/PermissionsMask.php(106): OCFilesStorageWrapperWrapper->rmdir('2018/02')
#15 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Wrapper.php(87): OCFilesStorageWrapperPermissionsMask->rmdir('2018/02')
#16 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Availability.php(95): OCFilesStorageWrapperWrapper->rmdir('2018/02')
#17 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Encryption.php(298): OCFilesStorageWrapperAvailability->rmdir('2018/02')
#18 [internal function]: OCFilesStorageWrapperEncryption->rmdir('2018/02')
#19 /snap/nextcloud/5132/htdocs/apps/files_trashbin/lib/Storage.php(258): call_user_func_array(Array, Array)
#20 /snap/nextcloud/5132/htdocs/apps/files_trashbin/lib/Storage.php(189): OCAFiles_TrashbinStorage->doDelete('2018/02', 'rmdir')
#21 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Encryption.php(617): OCAFiles_TrashbinStorage->rmdir('2018/02')
#22 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Wrapper.php(569): OCFilesStorageWrapperEncryption->moveFromStorage(Object(OCAFiles_TrashbinStorage), '2018/02', 'files_trashbin/...')
#23 /snap/nextcloud/5132/htdocs/apps/files_trashbin/lib/Trashbin.php(251): OCFilesStorageWrapperWrapper->moveFromStorage(Object(OCAFiles_TrashbinStorage), '2018/02', 'files_trashbin/...')
#24 /snap/nextcloud/5132/htdocs/apps/files_trashbin/lib/Storage.php(247): OCAFiles_TrashbinTrashbin::move2trash('AWS/2018/02', false)
#25 /snap/nextcloud/5132/htdocs/apps/files_trashbin/lib/Storage.php(189): OCAFiles_TrashbinStorage->doDelete('2018/02', 'rmdir')
#26 /snap/nextcloud/5132/htdocs/lib/private/Files/View.php(1159): OCAFiles_TrashbinStorage->rmdir('2018/02')
#27 /snap/nextcloud/5132/htdocs/lib/private/Files/View.php(348): OCFilesView->basicOperation('rmdir', '/AWS/2018/02', Array)
#28 /snap/nextcloud/5132/htdocs/apps/dav/lib/Connector/Sabre/Directory.php(303): OCFilesView->rmdir('/AWS/2018/02')
#29 /snap/nextcloud/5132/htdocs/3rdparty/sabre/dav/lib/DAV/Tree.php(179): OCADAVConnectorSabreDirectory->delete()
#30 /snap/nextcloud/5132/htdocs/3rdparty/sabre/dav/lib/DAV/CorePlugin.php(287): SabreDAVTree->delete('AWS/2018/02')
#31 [internal function]: SabreDAVCorePlugin->httpDelete(Object(SabreHTTPRequest), Object(SabreHTTPResponse))
#32 /snap/nextcloud/5132/htdocs/3rdparty/sabre/event/lib/EventEmitterTrait.php(105): call_user_func_array(Array, Array)
#33 /snap/nextcloud/5132/htdocs/3rdparty/sabre/dav/lib/DAV/Server.php(479): SabreEventEventEmitter->emit('method:DELETE', Array)
#34 /snap/nextcloud/5132/htdocs/3rdparty/sabre/dav/lib/DAV/Server.php(254): SabreDAVServer->invokeMethod(Object(SabreHTTPRequest), Object(SabreHTTPResponse))
#35 /snap/nextcloud/5132/htdocs/apps/dav/appinfo/v1/webdav.php(76): SabreDAVServer->exec()
#36 /snap/nextcloud/5132/htdocs/remote.php(162): require_once('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/snap/nextcloud...')
#37 {main}
"File": "/snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/Api/Validator.php", "Line": 65