Pilot 2392: refactor the upload logic

[colorzzr]

Summary

in the old logic, the normal upload and resumable upload are mixed together, which make the logic very complex. This PR will do following updates:

• remove the resumable upload logic out of normal upload
• clean up the path assembly logic to make it at one place

JIRA Issues

Pilot 2392

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update
• Refactor or reformatting

Testing

Are there any new or updated tests to validate the changes?

• Yes
• No

Test Directions

update test cases with new logic and add new test cases

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
__init__.py	0	0	100%
pilotcli.py	0	29	0%	5–6, 8–12, 15–19, 21–26, 29–37, 40–41
commands
__init__.py	0	0	100%
container_registry.py	0	47	0%	5, 7–10, 13–14, 16, 19–24, 27–33, 36–44, 47–53, 56–65
dataset.py	0	86	0%	5–6, 8–13, 16–17, 19, 22–25, 34–38, 41–48, 50–60, 63–67, 76–80, 83–91, 93–103, 106–109, 117–127, 129–130
entry_point.py	0	102	0%	5, 7, 9–10, 12–25, 28–39, 41–42, 45–51, 54–56, 59–63, 66–70, 73–77, 80–83, 86–88, 91–94, 97–114, 117, 119–121, 123–129, 131, 133–135, 137
file.py	0	106	47%	37, 120–130, 132, 134, 136–137, 141–142, 145–146, 148–151, 161–165, 167, 169–172, 187, 190, 192, 200, 213–216, 218, 220–221, 293–300, 302, 304, 318–324, 326, 353–360, 362, 396–402, 405–406, 408–409, 411–423, 425–427, 429, 432–434, 436–438
hpc.py	0	123	0%	5, 7–17, 20–21, 23, 26–29, 36–40, 42–45, 47–49, 51, 53–61, 64–72, 75–84, 87–100, 103–117, 120–132, 134–137, 140–153, 155–157
kg_resource.py	0	13	0%	5, 7–9, 12–13, 15, 18–23
project.py	0	39	0%	5–6, 8–11, 14–15, 17, 20–23, 31, 39, 48–52, 55–62, 64–74
use_config.py	0	13	0%	5, 7–9, 12–13, 15, 18–20, 27–29
user.py	0	30	0%	5, 7–9, 12–15, 18–19, 21, 42–48, 50, 52, 55–56, 58, 61–62, 71–75
configs
__init__.py	0	0	100%
app_config.py	0	0	100%
user_config.py	0	15	79%	49, 59, 63, 67, 71, 75, 79, 83, 87, 91, 95, 99, 103, 107, 111
models
__init__.py	0	0	100%
convert_type.py	0	8	0%	5, 7, 10–15
service_meta_class.py	0	2	80%	9, 16
singleton.py	0	0	100%
upload_form.py	0	28	60%	12, 27, 31, 35, 39, 43, 47, 51, 55, 59, 63, 67, 71, 75, 79, 83, 87, 91, 95, 99, 127, 140–146
resources
custom_error.py	0	0	100%
custom_help.py	0	0	100%
services
__init__.py	0	0	100%
services/crypto
__init__.py	0	0	100%
crypto.py	0	19	40%	35, 43–47, 49, 59–61, 69–75, 77, 79
services/dataset_manager
dataset_detail.py	0	23	70%	42, 44, 61, 68–73, 75–77, 79–88, 90
dataset_list.py	0	6	83%	37, 43–44, 46–48
services/file_manager
__init__.py	0	0	100%
file_lineage.py	0	7	41%	13–14, 23, 26–28, 30
file_list.py	0	48	27%	25–29, 31–34, 37, 45–60, 63–65, 69–75, 77–88
file_manifests.py	0	96	25%	19–24, 35, 39–43, 47–48, 51–59, 61, 65–68, 71–75, 77, 81–82, 85–87, 91–92, 95–102, 104, 107, 109–110, 112–113, 115–118, 123–129, 134–139, 142–148, 150–152, 155–161, 163–165, 167–170
file_tag.py	0	39	31%	23, 27–31, 33, 36–48, 50–52, 54–55, 59–61, 64–68, 70–73, 75–76
services/file_manager/file_download
__init__.py	0	0	100%
download_client.py	0	186	17%	29–41, 44–45, 47–53, 56–57, 59–60, 69, 71–72, 76–79, 86, 91–94, 96, 98–105, 107, 109, 113–118, 120, 123–124, 127–129, 131–133, 135–136, 138–140, 144–152, 155–160, 164–173, 181–183, 185–192, 194–198, 203–213, 215–216, 219–220, 225–226, 230–235, 239–240, 243–245, 247–258, 262–265, 268–272, 274–276, 278–280, 282–283, 285, 289–291, 293–306, 308
model.py	0	1	87%	15
services/file_manager/file_upload
__init__.py	0	0	100%
exception.py	0	1	75%	10
file_upload.py	0	61	59%	32–39, 73, 103, 134–135, 139, 141, 143–145, 147, 149, 161–163, 165–167, 171, 173–177, 181, 189–193, 196, 200, 204–207, 209–210, 212–214, 216–221, 223–224, 282, 285, 289, 294–295
models.py	0	5	91%	23, 41, 147–149
upload_client.py	0	86	55%	102–105, 169–171, 184, 186–191, 193–197, 200, 202–214, 216, 258, 261, 265, 267–270, 274–279, 281, 285, 287, 289, 291, 309, 334–335, 340, 344–347, 351, 367–369, 371–373, 382, 387–388, 390, 392–393, 395–398, 400, 414–420, 430
upload_validator.py	0	26	50%	27–29, 32–34, 38–42, 45–54, 57–58, 63, 65, 67
services/hpc_manager
hpc_auth.py	0	0	100%
hpc_cluster.py	0	68	32%	34–37, 39–40, 42, 46–58, 60–64, 66–67, 72–74, 76–77, 81–93, 95–99, 101–102, 106–118, 120–121, 123
services/kg_manager
kg_resource.py	0	11	85%	33–34, 36–37, 65–66, 81–82, 87–89
services/logger_services
__init__.py	0	0	100%
log_functions.py	0	0	100%
services/output_manager
__init__.py	0	0	100%
error_handler.py	0	4	95%	88–90, 104
help_page.py	0	14	84%	27–28, 36–37, 46–47, 59–60, 106–107, 116–117, 129–130
message_handler.py	0	66	57%	17–22, 27, 32, 37, 47, 52–54, 59, 64, 69, 74, 82, 87, 92, 99, 104, 109, 113, 129, 144, 154, 164, 183, 187, 191, 205–216, 227–232, 234–240, 251, 255–256, 260–261, 263–264, 268, 272, 276
response_handler.py	0	53	0%	5–6, 9–12, 14–15, 17–21, 23–24, 26–34, 36–39, 42–45, 47–48, 50–56, 58–59, 61–63, 65–68, 71–73
services/project_manager
__init__.py	0	0	100%
project.py	0	6	83%	38, 44–45, 47–49
services/user_authentication
__init__.py	0	0	100%
decorator.py	0	13	67%	29, 32, 49–53, 55, 59–62, 64
token_manager.py	0	21	57%	25, 28–30, 33, 36–37, 40–41, 61, 65, 69–70, 76–77, 79–82, 84–85
user_login_logout.py	0	43	65%	40–43, 114, 118–124, 126–128, 132–133, 137–144, 146–148, 152–153, 160–165, 167–169, 173–176
user_set_config.py	0	9	47%	15–18, 22–24, 26–27
utils
__init__.py	0	0	100%
aggregated.py	0	43	53%	22, 46, 62–73, 75–77, 81–84, 86, 90–93, 95–96, 115–117, 119, 124–131, 133–135
TOTAL	2968	1596	46%