PILOT-4152: Update cli logic to fit dynamic chunk size when uploading

[hugoantunes]

Summary

• update GET /v1/files/chunks/presigned api to pass the chunk_size as parameter.
• set chunk_size to upload_chunk in stream_upload

https://indocconsortium.atlassian.net/browse/PILOT-4152

Type of Change

Please delete options that are not relevant.

• Refactor or reformatting

Testing

Are there any new or updated tests to validate the changes?

• Yes
• No

Test Directions

N/A

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
__init__.py	0	0	100%
pilotcli.py	0	31	0%	5, 7–8, 10–14, 17–21, 23–28, 31–39, 42–44
commands
__init__.py	0	0	100%
container_registry.py	0	19	59%	16, 22–24, 31–33, 41–44, 50–53, 62–65
dataset.py	0	48	44%	19, 36–38, 41–48, 50–60, 78–80, 83–91, 93–103, 123, 127
entry_point.py	0	10	84%	37, 43, 85–87, 89–93
file.py	0	67	67%	38, 133–135, 139, 144, 147–149, 167–168, 210, 292–299, 301, 303, 317–323, 325, 354, 356, 359, 395–401, 404–405, 407–408, 410–422, 424–426, 428, 431–433, 435–437
project.py	0	1	97%	17
user.py	0	6	85%	23, 40, 48, 56, 72–73
configs
__init__.py	0	0	100%
app_config.py	0	0	100%
config.py	0	0	100%
user_config.py	0	17	84%	61, 70, 114, 124, 131, 135, 139, 143, 147, 151, 155, 159, 163, 167, 171, 175, 179
models
__init__.py	0	0	100%
enums.py	0	0	100%
service_meta_class.py	0	1	80%	9
singleton.py	0	0	100%
upload_form.py	0	6	25%	30, 42–46
resources
custom_error.py	0	0	100%
custom_help.py	0	0	100%
services
__init__.py	0	0	100%
services/container_registry_manager
container_registry_manager.py	0	108	17%	19–20, 23–26, 29–33, 36–44, 48–60, 62–64, 68–82, 84–86, 90–99, 101–103, 107–110, 125–136, 140–143, 147–162, 164, 166–169
services/crypto
__init__.py	0	0	100%
crypto.py	0	19	40%	35, 43–47, 49, 59–61, 69–75, 77, 79
services/dataset_manager
dataset_detail.py	0	23	70%	42, 44, 61, 68–73, 75–77, 79–88, 90
dataset_download.py	0	19	83%	56, 72–73, 84, 96, 98–99, 101–102, 104, 106–108, 116–119, 153, 165
dataset_list.py	0	6	83%	37, 43–44, 46–48
model.py	0	0	100%
services/file_manager
__init__.py	0	0	100%
file_list.py	0	19	71%	29, 48, 50, 57–58, 63–65, 73–76, 78, 84–89
file_manifests.py	0	95	25%	19–24, 39–43, 47–48, 51–59, 61, 65–68, 71–75, 77, 81–82, 85–87, 91–92, 95–102, 104, 107, 109–110, 112–113, 115–118, 123–129, 134–139, 142–148, 150–152, 155–161, 163–165, 167–170
file_tag.py	0	39	31%	23, 27–31, 33, 36–48, 50–52, 54–55, 59–61, 64–68, 70–73, 75–76
services/file_manager/file_download
__init__.py	0	0	100%
download_client.py	0	187	17%	29–41, 44–45, 47–53, 56–57, 59–60, 69, 71–72, 76–79, 86, 91–94, 96, 98–106, 108, 110, 114–119, 121, 124–125, 128–130, 132–134, 136–137, 139–141, 145–153, 156–161, 165–174, 182–184, 186–193, 195–199, 204–214, 216–217, 220–221, 226–227, 231–236, 240–241, 244–246, 248–259, 263–266, 269–273, 275–277, 279–281, 283–284, 286, 290–292, 294–307, 309
model.py	0	1	87%	15
services/file_manager/file_upload
__init__.py	0	0	100%
exception.py	0	1	75%	10
file_upload.py	0	29	83%	35–42, 76, 98–100, 112, 143–144, 148, 153, 190, 210–212, 253–254, 256–258, 332, 335, 339
models.py	0	5	91%	24, 44, 150–152
upload_client.py	0	77	60%	101–104, 207–209, 223–229, 231–235, 237–249, 251, 295, 298, 302, 304–306, 308–311, 315–319, 321, 325, 327, 329, 331, 350, 380–381, 386, 390–393, 397, 413, 415–417, 424, 429–430, 432, 434–435, 437–440, 442
upload_validator.py	0	24	52%	26–28, 31–36, 39–48, 51–52, 57, 59, 61
services/logger_services
__init__.py	0	0	100%
log_functions.py	0	0	100%
services/output_manager
__init__.py	0	0	100%
error_handler.py	0	0	100%
help_page.py	0	0	100%
message_handler.py	0	56	68%	23, 45, 50, 65–67, 72, 82, 87, 100, 105, 112, 117, 122, 126, 142, 177, 187, 196, 201, 220, 242–253, 264–269, 271–277, 288, 292–293, 297–298, 300–301, 305, 309, 313
services/project_manager
__init__.py	0	0	100%
project.py	0	6	83%	38, 44–45, 47–49
services/user_authentication
__init__.py	0	0	100%
decorator.py	0	2	93%	27, 30
token_manager.py	0	15	76%	25, 40–41, 72, 75, 82–83, 89–90, 92–95, 97, 102
user_login_logout.py	0	42	66%	30–31, 41, 122, 126–132, 134–136, 140–141, 145–152, 154–156, 160–161, 168–173, 175–177, 181–184
utils
__init__.py	0	0	100%
aggregated.py	0	35	65%	72–74, 76–77, 84–85, 87, 91–94, 96–97, 111–118, 120–121, 125–132, 134–136
TOTAL	2841	1014	64%

[colorzzr]

Hey @hugoantunes! Please check the comments below when you have time thx!

[colorzzr]

can you also add the default as .get(str(count + 1), {}). because if we reach the unfinished chunks, the variable chunk will be None and line 305, 306 will give a error.