[Bug]: TarStreamer library is incompatible, downloads fail.

[AdamVenn]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• Nextcloud Server is running on 64bit capable CPU, PHP and OS.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

When downloading largeish folders (>90 files, subfolders, etc.) on MacOS, Nextcloud fails to create the tar file of the folder.

Nextcloud uses the TarStreamer library here:
https://github.com/owncloud/TarStreamer

But it is apparently not compatible with current versions of PHP:

owncloud/TarStreamer#23 (comment)

The library needs to be brought up to date in order to use Nextcloud with secure versions of PHP/MacOS.

Steps to reproduce

1. Create a folder with 90 files in a subfolder
2. Open a browser on MacOS
3. Attempt to download the folder
4. Receive a truncated tar file

Expected behavior

Receive a complete tar file of the folder.

Installation method

Community Manual installation with Archive

Nextcloud Server version

26

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Updated from a minor version (ex. 22.2.3 to 22.2.4)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "10.150.64.10",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "26.0.0.11",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "htaccess.RewriteBase": "\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "skeletondirectory": "\/var\/www\/skeleton",
        "trashbin_retention_obligation": "1, 2",
        "ldapUserCleanupInterval": "30",
        "log_type": "syslog",
        "logfile": "",
        "loglevel": 1,
        "syslog_tag": "Nextcloud",
        "enable_previews": false,
        "log.condition": {
            "apps": [
                "admin_audit"
            ]
        },
        "default_phone_region": "CA",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "filelocking.enabled": "true",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "dbindex": 0,
            "password": "***REMOVED SENSITIVE VALUE***",
            "timeout": 1.5
        },
        "maintenance": false,
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory"
    }
}

List of activated Apps

Enabled:
  - activity: 2.18.0
  - admin_audit: 1.16.0
  - bruteforcesettings: 2.6.0
  - cloud_federation_api: 1.9.0
  - contacts: 5.2.0
  - dav: 1.25.0
  - federatedfilesharing: 1.16.0
  - files: 1.21.1
  - files_downloadactivity: 1.16.0
  - files_external: 1.18.0
  - files_pdfviewer: 2.7.0
  - files_rightclick: 1.5.0
  - files_sharing: 1.18.0
  - files_trashbin: 1.16.0
  - lookup_server_connector: 1.14.0
  - oauth2: 1.14.0
  - password_policy: 1.16.0
  - privacy: 1.10.0
  - provisioning_api: 1.16.0
  - related_resources: 1.1.0-alpha1
  - serverinfo: 1.16.0
  - settings: 1.8.0
  - text: 3.7.2
  - theming: 2.1.1
  - twofactor_backupcodes: 1.15.0
  - user_ldap: 1.16.0
  - viewer: 1.10.0
  - workflowengine: 2.8.0
Disabled:
  - circles: 26.0.0 (installed 22.1.1)
  - comments: 1.16.0 (installed 1.12.0)
  - contactsinteraction: 1.7.0 (installed 1.3.0)
  - dashboard: 7.6.0 (installed 7.2.0)
  - encryption: 2.14.0
  - federation: 1.16.0 (installed 1.12.0)
  - files_versions: 1.19.1 (installed 1.15.0)
  - firstrunwizard: 2.15.0 (installed 2.11.0)
  - group_everyone: 0.1.11 (installed 0.1.11)
  - logreader: 2.11.0 (installed 2.7.0)
  - nextcloud_announcements: 1.15.0 (installed 1.11.0)
  - notifications: 2.14.0 (installed 2.10.1)
  - photos: 2.2.0 (installed 1.4.0)
  - recommendations: 1.5.0 (installed 1.1.0)
  - sharebymail: 1.16.0 (installed 1.12.0)
  - support: 1.9.0 (installed 1.5.0)
  - survey_client: 1.14.0 (installed 1.10.0)
  - suspicious_login: 4.4.0
  - systemtags: 1.16.0 (installed 1.12.0)
  - twofactor_totp: 8.0.0-alpha.0
  - updatenotification: 1.16.0 (installed 1.15.0)
  - user_status: 1.6.0 (installed 1.2.0)
  - weather_status: 1.6.0 (installed 1.2.0)

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

Relevant log lines:

nc_msg_Exception
    TypeError
nc_msg_File
    /var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php
nc_msg_Line
    72
nc_msg_Message
    decoct(): Argument #1 ($num) must be of type int, string given
nc_msg_Trace
    {file=/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php, line=72, function=decoct},
    {file=/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php, line=200, function=getHeader, class=ownCloud\TarStreamer\TarHeader, type=->},
    {file=/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php, line=166, function=writeLongName, class=ownCloud\TarStreamer\TarStreamer, type=->},
    {file=/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php, line=102, function=initFileStreamTransfer, class=ownCloud\TarStreamer\TarStreamer, type=->},
    {file=/var/www/nextcloud/lib/private/Streamer.php, line=167, function=addFileFromStream, class=ownCloud\TarStreamer\TarStreamer, type=->},
    {file=/var/www/nextcloud/lib/private/Streamer.php, line=136, function=addFileFromStream, class=OC\Streamer, type=->},
    {file=/var/www/nextcloud/lib/private/Streamer.php, line=141, function=addDirRecursive, class=OC\Streamer, type=->},
    {file=/var/www/nextcloud/lib/private/legacy/OC_Files.php, line=217, function=addDirRecursive, class=OC\Streamer, type=->},
    {file=/var/www/nextcloud/apps/files/ajax/download.php, line=77, function=get, class=OC_Files, type=::},
    {file=/var/www/nextcloud/lib/private/Route/Route.php, line=155, args=[/var/www/nextcloud/apps/files/ajax/download.php], function=require_once},
    {function=OC\Route\{closure}, class=OC\Route\Route, type=->, args=[*** sensitive parameters replaced ***]},
    {file=/var/www/nextcloud/lib/private/Route/Router.php, line=324, function=call_user_func},
    {file=/var/www/nextcloud/lib/base.php, line=1055, function=match, class=OC\Route\Router, type=->},
    {file=/var/www/nextcloud/index.php, line=36, function=handleRequest, class=OC, type=::}






Full Logs:

application_name
    Nextcloud
facility
    user-level
facility_num
    1
level
    3
message
    {"reqId":"DxUGPZDJ1qX1WR8O81ll","level":3,"time":"2023-04-14T20:21:01+00:00","remoteAddr":"10.150.64.1","user":"***REMOVED SENSITIVE VALUE***","app":"index","method":"GET","url":"/index.php/apps/files/ajax/download.php?***REMOVED SENSITIVE VALUE***","message":"{\"Exception\":\"TypeError\",\"Message\":\"decoct(): Argument #1 ($num) must be of type int, string given\",\"Code\":0,\"Trace\":[{\"file\":\"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php\",\"line\":72,\"function\":\"decoct\"},{\"file\":\"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php\",\"line\":200,\"function\":\"getHeader\",\"class\":\"ownCloud\\\\TarStreamer\\\\TarHeader\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php\",\"line\":166,\"function\":\"writeLongName\",\"class\":\"ownCloud\\\\TarStreamer\\\\TarStreamer\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php\",\"line\":102,\"function\":\"initFileStreamTransfer\",\"class\":\"ownCloud\\\\TarStreamer\\\\TarStreamer\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/lib/private/Streamer.php\",\"line\":167,\"function\":\"addFileFromStream\",\"class\":\"ownCloud\\\\TarStreamer\\\\TarStreamer\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/lib/private/Streamer.php\",\"line\":136,\"function\":\"addFileFromStream\",\"class\":\"OC\\\\Streamer\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/lib/private/Streamer.php\",\"line\":141,\"function\":\"addDirRecursive\",\"class\":\"OC\\\\Streamer\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/lib/private/legacy/OC_Files.php\",\"line\":217,\"function\":\"addDirRecursive\",\"class\":\"OC\\\\Streamer\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/apps/files/ajax/download.php\",\"line\":77,\"function\":\"get\",\"class\":\"OC_Files\",\"type\":\"::\"},{\"file\":\"/var/www/nextcloud/lib/private/Route/Route.php\",\"line\":155,\"args\":[\"/var/www/nextcloud/apps/files/ajax/download.php\"],\"function\":\"require_once\"},{\"function\":\"OC\\\\Route\\\\{closure}\",\"class\":\"OC\\\\Route\\\\Route\",\"type\":\"->\",\"args\":[\"*** sensitive parameters replaced ***\"]},{\"file\":\"/var/www/nextcloud/lib/private/Route/Router.php\",\"line\":324,\"function\":\"call_user_func\"},{\"file\":\"/var/www/nextcloud/lib/base.php\",\"line\":1055,\"function\":\"match\",\"class\":\"OC\\\\Route\\\\Router\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/index.php\",\"line\":36,\"function\":\"handleRequest\",\"class\":\"OC\",\"type\":\"::\"}],\"File\":\"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php\",\"Line\":72,\"CustomMessage\":\"--\"}","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/109.0","version":"26.0.0.11"}
nc_app
    index
nc_level
    3
nc_message
    {"Exception":"TypeError","Message":"decoct(): Argument #1 ($num) must be of type int, string given","Code":0,"Trace":[{"file":"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php","line":72,"function":"decoct"},{"file":"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php","line":200,"function":"getHeader","class":"ownCloud\\TarStreamer\\TarHeader","type":"->"},{"file":"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php","line":166,"function":"writeLongName","class":"ownCloud\\TarStreamer\\TarStreamer","type":"->"},{"file":"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php","line":102,"function":"initFileStreamTransfer","class":"ownCloud\\TarStreamer\\TarStreamer","type":"->"},{"file":"/var/www/nextcloud/lib/private/Streamer.php","line":167,"function":"addFileFromStream","class":"ownCloud\\TarStreamer\\TarStreamer","type":"->"},{"file":"/var/www/nextcloud/lib/private/Streamer.php","line":136,"function":"addFileFromStream","class":"OC\\Streamer","type":"->"},{"file":"/var/www/nextcloud/lib/private/Streamer.php","line":141,"function":"addDirRecursive","class":"OC\\Streamer","type":"->"},{"file":"/var/www/nextcloud/lib/private/legacy/OC_Files.php","line":217,"function":"addDirRecursive","class":"OC\\Streamer","type":"->"},{"file":"/var/www/nextcloud/apps/files/ajax/download.php","line":77,"function":"get","class":"OC_Files","type":"::"},{"file":"/var/www/nextcloud/lib/private/Route/Route.php","line":155,"args":["/var/www/nextcloud/apps/files/ajax/download.php"],"function":"require_once"},{"function":"OC\\Route\\{closure}","class":"OC\\Route\\Route","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/lib/private/Route/Router.php","line":324,"function":"call_user_func"},{"file":"/var/www/nextcloud/lib/base.php","line":1055,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/nextcloud/index.php","line":36,"function":"handleRequest","class":"OC","type":"::"}],"File":"/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php","Line":72,"CustomMessage":"--"}
nc_method
    GET
nc_msg_Code
    0
nc_msg_CustomMessage
    --
nc_msg_Exception
    TypeError
nc_msg_File
    /var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php
nc_msg_Line
    72
nc_msg_Message
    decoct(): Argument #1 ($num) must be of type int, string given
nc_msg_Trace
    {file=/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php, line=72, function=decoct},
    {file=/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php, line=200, function=getHeader, class=ownCloud\TarStreamer\TarHeader, type=->},
    {file=/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php, line=166, function=writeLongName, class=ownCloud\TarStreamer\TarStreamer, type=->},
    {file=/var/www/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php, line=102, function=initFileStreamTransfer, class=ownCloud\TarStreamer\TarStreamer, type=->},
    {file=/var/www/nextcloud/lib/private/Streamer.php, line=167, function=addFileFromStream, class=ownCloud\TarStreamer\TarStreamer, type=->},
    {file=/var/www/nextcloud/lib/private/Streamer.php, line=136, function=addFileFromStream, class=OC\Streamer, type=->},
    {file=/var/www/nextcloud/lib/private/Streamer.php, line=141, function=addDirRecursive, class=OC\Streamer, type=->},
    {file=/var/www/nextcloud/lib/private/legacy/OC_Files.php, line=217, function=addDirRecursive, class=OC\Streamer, type=->},
    {file=/var/www/nextcloud/apps/files/ajax/download.php, line=77, function=get, class=OC_Files, type=::},
    {file=/var/www/nextcloud/lib/private/Route/Route.php, line=155, args=[/var/www/nextcloud/apps/files/ajax/download.php], function=require_once},
    {function=OC\Route\{closure}, class=OC\Route\Route, type=->, args=[*** sensitive parameters replaced ***]},
    {file=/var/www/nextcloud/lib/private/Route/Router.php, line=324, function=call_user_func},
    {file=/var/www/nextcloud/lib/base.php, line=1055, function=match, class=OC\Route\Router, type=->},
    {file=/var/www/nextcloud/index.php, line=36, function=handleRequest, class=OC, type=::}
nc_remoteAddr
    10.150.64.1
nc_reqId
    DxUGPZDJ1qX1WR8O81ll
nc_time
    2023-04-14 16:21:01.000
nc_url
    /index.php/apps/files/ajax/download.php?***REMOVED SENSITIVE VALUE***
nc_user
    ***REMOVED SENSITIVE VALUE***
nc_userAgent
    Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/109.0
nc_version
    26.0.0.11
process_id
    1822
source
    nc
timestamp
    2023-04-14 16:21:01.331

Additional info

No response

[pjft]

I get this as well on Linux. Tried to download 5GB of multiple folders with smaller files, and the tar file comes back with 257MB only.

If it's downloaded as a ZIP file (i.e. smaller download), it works as intended. The moment it crosses the threshold and it becomes a TAR file, the TAR file gets downloaded, but it's incomplete. I cannot replicate it consistently, though, but will happily troubleshoot this further. It might be related to the size of the files it's archiving? I'll look into this and try to come up with an example that's replicable.

This is the error:

[index] Error: TypeError: decoct(): Argument #1 ($num) must be of type int, string given at <<closure>>

 0. /var/www/html/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php line 72
    decoct()
 1. /var/www/html/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php line 200
    ownCloud\TarStreamer\TarHeader->getHeader()
 2. /var/www/html/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php line 166
    ownCloud\TarStreamer\TarStreamer->writeLongName()
 3. /var/www/html/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarStreamer.php line 102
    ownCloud\TarStreamer\TarStreamer->initFileStreamTransfer()
 4. /var/www/html/nextcloud/lib/private/Streamer.php line 168
    ownCloud\TarStreamer\TarStreamer->addFileFromStream()
 5. /var/www/html/nextcloud/lib/private/Streamer.php line 137
    OC\Streamer->addFileFromStream()
 6. /var/www/html/nextcloud/lib/private/legacy/OC_Files.php line 215
    OC\Streamer->addDirRecursive()
 7. /var/www/html/nextcloud/apps/files/ajax/download.php line 77
    OC_Files::get()
 8. /var/www/html/nextcloud/lib/private/Route/Route.php line 155
    require_once("/var/www/html/n ... p")
 9. <<closure>>
    OC\Route\Route->OC\Route\{closure}("*** sensitive parameters replaced ***")
10. /var/www/html/nextcloud/lib/private/Route/Router.php line 306
    call_user_func()
11. /var/www/html/nextcloud/lib/base.php line 1048
    OC\Route\Router->match()
12. /var/www/html/nextcloud/index.php line 36
    OC::handleRequest()

GET /index.php/apps/files/ajax/download.php?dir=***redacted***&downloadStartSecret=w1riopogit8
from XXX at 2023-06-07T06:00:59+00:00

[pjft]

After further investigation, I am not convinced that this has anything to do with a PHP version incompatibility - at least not directly. The error does not happen all the time - I'm just finishing downloading a 4GB TAR archive and all is well.

EDIT: and, on the flipside, I can consistently replicate this with a specific folder, particularly in a file that does not get added.

It crashes when decoding the mtime of the file being compressed, so I will look into that and report back in the coming weeks.

[pjft]

In my case, this is fixed with the following change:

In
/var/www/html/nextcloud/3rdparty/deepdiver1975/tarstreamer/src/TarHeader.php

Change:

                        ['a12', str_pad(decoct($this->size), 11, '0', STR_PAD_LEFT)],
                        ['a12', str_pad(decoct($this->mtime), 11, '0', STR_PAD_LEFT)],

To:

                        ['a12', str_pad(decoct((int)$this->size), 11, '0', STR_PAD_LEFT)],
                        ['a12', str_pad(decoct((int)$this->mtime), 11, '0', STR_PAD_LEFT)],

to force the explicit int cast.

@szaimen since you tagged this issue, I'd appreciate guidance here. I'm happy to submit a PR, but should I do it upstream, assuming they'll take it, or do you have any other recommendation?

Thanks all.

[szaimen]

Best to open a PR here then: https://github.com/owncloud/TarStreamer

also cc @nextcloud/server-backend

[pjft]

Thank you. I submitted the PR here:

owncloud/TarStreamer#25

I'll keep you posted.

[pjft]

Thanks for the guidance - the PR has been approved.

Is there anything that needs to be done on the Nextcloud front (a PR or something) to incorporate those changes here?

[szaimen]

I guess they would need to publish a new release. Afterwards we can update the dependency here.

[come-nc]

The cast to int will fail on 32bit for files larger than 4GB, not sure if this is a big deal or not (and whether without the cast it will work or not on 32bits).

-> I tested and it seems even without the cast this will crash on 32bit for sizes above 4GB, so the PR does not break anything.

[pjft]

@come-nc Thanks for the heads up.

Just to make sure I'm reading this correctly, what you're saying is that TarStreamer fails on 32bit systems on sizes above 4GB, with or without the patch. Is that correct?

There's a lot of nuances on 32bit systems and 4GB file sizes indeed. Well spotted.

[come-nc]

Just to make sure I'm reading this correctly, what you're saying is that TarStreamer fails on 32bit systems on sizes above 4GB, with or without the patch. Is that correct?

Yes I think so, because those sizes on 32bits will always appears as float and decoct expects an int.