Staging release 18/04/2023

[colorzzr]

Summary

• add new command for resumable upload as pilotcli file resume -r manifest.json. the logic of normal upload and resumble will be split. the manifest file will be output for both file/folder upload. The resumable upload operation will be per batch action with size 100.
• add a new constraint to block tagging/manifest when folder uploading.
• skip and raise the warning for file with size 0.
• update error message.
• adapt cli with new lineage workflow.Temporary disable the lineage in this stating release since backend is not merged yet.
• add token refresh in a dedicate thread.
• update upload bar logic which will block the multithread.

JIRA Issues

(What JIRA issues this merge request is related to)

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update
• Refactor or reformatting

Testing

Are there any new or updated tests to validate the changes?

• Yes
• No

Test Directions

(Additional instructions for how to run tests or validate functionality if not covered by unit tests)

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
__init__.py	0	0	100%
pilotcli.py	0	29	0%	5–6, 8–12, 15–19, 21–26, 29–37, 40–41
commands
__init__.py	0	0	100%
container_registry.py	0	47	0%	5, 7–10, 13–14, 16, 19–24, 27–33, 36–44, 47–53, 56–65
dataset.py	0	86	0%	5–6, 8–13, 16–17, 19, 22–25, 34–38, 41–48, 50–60, 63–67, 76–80, 83–91, 93–103, 106–109, 117–127, 129–130
entry_point.py	0	102	0%	5, 7, 9–10, 12–25, 28–39, 41–42, 45–51, 54–56, 59–63, 66–70, 73–77, 80–83, 86–88, 91–94, 97–114, 117, 119–121, 123–129, 131, 133–135, 137
file.py	0	68	64%	36, 126, 138, 142, 160–161, 203, 278–285, 287, 289, 303–309, 311, 338–345, 347, 381–387, 390–391, 393–394, 396–408, 410–412, 414, 417–419, 421–423
hpc.py	0	123	0%	5, 7–17, 20–21, 23, 26–29, 36–40, 42–45, 47–49, 51, 53–61, 64–72, 75–84, 87–100, 103–117, 120–132, 134–137, 140–153, 155–157
kg_resource.py	0	13	0%	5, 7–9, 12–13, 15, 18–23
project.py	0	39	0%	5–6, 8–11, 14–15, 17, 20–23, 31, 39, 48–52, 55–62, 64–74
use_config.py	0	13	0%	5, 7–9, 12–13, 15, 18–20, 27–29
user.py	0	30	0%	5, 7–9, 12–15, 18–19, 21, 42–48, 50, 52, 55–56, 58, 61–62, 71–75
configs
__init__.py	0	0	100%
app_config.py	0	0	100%
user_config.py	0	15	79%	49, 59, 63, 67, 71, 75, 79, 83, 87, 91, 95, 99, 103, 107, 111
models
__init__.py	0	0	100%
convert_type.py	0	8	0%	5, 7, 10–15
service_meta_class.py	0	2	80%	9, 16
singleton.py	0	0	100%
upload_form.py	0	26	61%	10, 25, 29, 33, 37, 41, 45, 49, 53, 57, 61, 65, 69, 73, 77, 81, 85, 89, 93, 97, 122, 134–138
resources
custom_error.py	0	0	100%
custom_help.py	0	0	100%
services
__init__.py	0	0	100%
services/crypto
__init__.py	0	0	100%
crypto.py	0	19	40%	35, 43–47, 49, 59–61, 69–75, 77, 79
services/dataset_manager
dataset_detail.py	0	23	70%	42, 44, 61, 68–73, 75–77, 79–88, 90
dataset_list.py	0	6	83%	37, 43–44, 46–48
services/file_manager
__init__.py	0	0	100%
file_list.py	0	48	27%	25–29, 31–34, 37, 45–60, 63–65, 69–75, 77–88
file_manifests.py	0	95	25%	19–24, 39–43, 47–48, 51–59, 61, 65–68, 71–75, 77, 81–82, 85–87, 91–92, 95–102, 104, 107, 109–110, 112–113, 115–118, 123–129, 134–139, 142–148, 150–152, 155–161, 163–165, 167–170
file_tag.py	0	39	31%	23, 27–31, 33, 36–48, 50–52, 54–55, 59–61, 64–68, 70–73, 75–76
services/file_manager/file_download
__init__.py	0	0	100%
download_client.py	0	187	17%	29–41, 44–45, 47–53, 56–57, 59–60, 69, 71–72, 76–79, 86, 91–94, 96, 98–106, 108, 110, 114–119, 121, 124–125, 128–130, 132–134, 136–137, 139–141, 145–153, 156–161, 165–174, 182–184, 186–193, 195–199, 204–214, 216–217, 220–221, 226–227, 231–236, 240–241, 244–246, 248–259, 263–266, 269–273, 275–277, 279–281, 283–284, 286, 290–292, 294–307, 309
model.py	0	1	87%	15
services/file_manager/file_upload
__init__.py	0	0	100%
exception.py	0	1	75%	10
file_upload.py	0	30	81%	33–40, 74, 104, 135–136, 140, 145, 175, 183–185, 189, 206, 208, 212, 223–224, 226–228, 306, 309, 313
models.py	0	5	91%	23, 41, 147–149
upload_client.py	0	75	56%	98–101, 163–165, 179–185, 187–191, 193–205, 207, 249, 252, 256, 258–261, 265–270, 272, 276, 278, 280, 282, 300, 325–326, 331, 335–338, 342, 358, 360–362, 369, 374–375, 377, 379–380, 382–385, 387
upload_validator.py	0	24	52%	26–28, 31–36, 39–48, 51–52, 57, 59, 61
services/hpc_manager
hpc_auth.py	0	0	100%
hpc_cluster.py	0	68	32%	34–37, 39–40, 42, 46–58, 60–64, 66–67, 72–74, 76–77, 81–93, 95–99, 101–102, 106–118, 120–121, 123
services/kg_manager
kg_resource.py	0	11	85%	33–34, 36–37, 65–66, 81–82, 87–89
services/logger_services
__init__.py	0	0	100%
log_functions.py	0	0	100%
services/output_manager
__init__.py	0	0	100%
error_handler.py	0	1	98%	105
help_page.py	0	14	84%	27–28, 36–37, 46–47, 59–60, 106–107, 116–117, 129–130
message_handler.py	0	65	59%	17–22, 27, 32, 37, 47, 52–54, 59, 64, 69, 74, 87, 92, 99, 104, 109, 113, 129, 149, 159, 169, 188, 192, 196, 210–221, 232–237, 239–245, 256, 260–261, 265–266, 268–269, 273, 277, 281
response_handler.py	0	53	0%	5–6, 9–12, 14–15, 17–21, 23–24, 26–34, 36–39, 42–45, 47–48, 50–56, 58–59, 61–63, 65–68, 71–73
services/project_manager
__init__.py	0	0	100%
project.py	0	6	83%	38, 44–45, 47–49
services/user_authentication
__init__.py	0	0	100%
decorator.py	0	13	67%	29, 32, 49–53, 55, 59–62, 64
token_manager.py	0	21	57%	25, 28–30, 33, 36–37, 40–41, 61, 65, 69–70, 76–77, 79–82, 84–85
user_login_logout.py	0	43	65%	40–43, 114, 118–124, 126–128, 132–133, 137–144, 146–148, 152–153, 160–165, 167–169, 173–176
user_set_config.py	0	9	47%	15–18, 22–24, 26–27
utils
__init__.py	0	0	100%
aggregated.py	0	41	55%	22, 46, 62–73, 75–77, 83–84, 86, 90–93, 95–96, 115–117, 119, 124–131, 133–135
TOTAL	2938	1499	48%