Pilot 2999: fixup the dataset download issue

[colorzzr]

Summary

The bug was caused by the legacy code. update the dataset download logic which will be the same as portal:

1. send pre-download request to download service to pack the zip
2. wait the status to be SUCCEED
3. download the dataset zip

JIRA Issues

Pilot 2999

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update
• Refactor or reformatting

Testing

Are there any new or updated tests to validate the changes?

• Yes
• No

Test Directions

Add a new set of download test cases

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
__init__.py	0	0	100%
pilotcli.py	0	29	0%	5–6, 8–12, 15–19, 21–26, 29–37, 40–41
commands
__init__.py	0	0	100%
container_registry.py	0	47	0%	5, 7–10, 13–14, 16, 19–24, 27–33, 36–44, 47–53, 56–65
dataset.py	0	48	44%	19, 36–38, 41–48, 50–60, 78–80, 83–91, 93–103, 123, 127
entry_point.py	0	102	0%	5, 7, 9–10, 12–25, 28–39, 41–42, 45–51, 54–56, 59–63, 66–70, 73–77, 80–83, 86–88, 91–94, 97–114, 117, 119–121, 123–129, 131, 133–135, 137
file.py	0	61	68%	36, 133, 137, 155–156, 198, 273–280, 282, 284, 298–304, 306, 335, 337, 340, 376–382, 385–386, 388–389, 391–403, 405–407, 409, 412–414, 416–418
hpc.py	0	123	0%	5, 7–17, 20–21, 23, 26–29, 36–40, 42–45, 47–49, 51, 53–61, 64–72, 75–84, 87–100, 103–117, 120–132, 134–137, 140–153, 155–157
kg_resource.py	0	13	0%	5, 7–9, 12–13, 15, 18–23
project.py	0	39	0%	5–6, 8–11, 14–15, 17, 20–23, 31, 39, 48–52, 55–62, 64–74
use_config.py	0	13	0%	5, 7–9, 12–13, 15, 18–20, 27–29
user.py	0	30	0%	5, 7–9, 12–15, 18–19, 21, 42–48, 50, 52, 55–56, 58, 61–62, 71–75
configs
__init__.py	0	0	100%
app_config.py	0	0	100%
user_config.py	0	15	79%	49, 59, 63, 67, 71, 75, 79, 83, 87, 91, 95, 99, 103, 107, 111
models
__init__.py	0	0	100%
convert_type.py	0	8	0%	5, 7, 10–15
service_meta_class.py	0	2	80%	9, 16
singleton.py	0	0	100%
upload_form.py	0	26	61%	10, 25, 29, 33, 37, 41, 45, 49, 53, 57, 61, 65, 69, 73, 77, 81, 85, 89, 93, 97, 122, 134–138
resources
custom_error.py	0	0	100%
custom_help.py	0	0	100%
services
__init__.py	0	0	100%
services/crypto
__init__.py	0	0	100%
crypto.py	0	19	40%	35, 43–47, 49, 59–61, 69–75, 77, 79
services/dataset_manager
dataset_detail.py	0	23	70%	42, 44, 61, 68–73, 75–77, 79–88, 90
dataset_download.py	0	19	83%	56, 71–72, 83, 95, 97–98, 100–101, 103, 105–107, 115–118, 152, 164
dataset_list.py	0	6	83%	37, 43–44, 46–48
model.py	0	0	100%
services/file_manager
__init__.py	0	0	100%
file_list.py	0	19	71%	29, 48, 50, 57–58, 63–65, 73–76, 78, 84–89
file_manifests.py	0	95	25%	19–24, 39–43, 47–48, 51–59, 61, 65–68, 71–75, 77, 81–82, 85–87, 91–92, 95–102, 104, 107, 109–110, 112–113, 115–118, 123–129, 134–139, 142–148, 150–152, 155–161, 163–165, 167–170
file_tag.py	0	39	31%	23, 27–31, 33, 36–48, 50–52, 54–55, 59–61, 64–68, 70–73, 75–76
services/file_manager/file_download
__init__.py	0	0	100%
download_client.py	0	187	17%	29–41, 44–45, 47–53, 56–57, 59–60, 69, 71–72, 76–79, 86, 91–94, 96, 98–106, 108, 110, 114–119, 121, 124–125, 128–130, 132–134, 136–137, 139–141, 145–153, 156–161, 165–174, 182–184, 186–193, 195–199, 204–214, 216–217, 220–221, 226–227, 231–236, 240–241, 244–246, 248–259, 263–266, 269–273, 275–277, 279–281, 283–284, 286, 290–292, 294–307, 309
model.py	0	1	87%	15
services/file_manager/file_upload
__init__.py	0	0	100%
exception.py	0	1	75%	10
file_upload.py	0	30	81%	33–40, 74, 104, 135–136, 140, 145, 176, 184–186, 190, 207, 209, 213, 224–225, 227–229, 307, 310, 314
models.py	0	5	91%	23, 41, 147–149
upload_client.py	0	75	56%	98–101, 163–165, 179–185, 187–191, 193–205, 207, 250, 253, 257, 259–262, 266–271, 273, 277, 279, 281, 283, 301, 326–327, 332, 336–339, 343, 359, 361–363, 370, 375–376, 378, 380–381, 383–386, 388
upload_validator.py	0	24	52%	26–28, 31–36, 39–48, 51–52, 57, 59, 61
services/hpc_manager
hpc_auth.py	0	0	100%
hpc_cluster.py	0	68	32%	34–37, 39–40, 42, 46–58, 60–64, 66–67, 72–74, 76–77, 81–93, 95–99, 101–102, 106–118, 120–121, 123
services/kg_manager
kg_resource.py	0	11	85%	33–34, 36–37, 65–66, 81–82, 87–89
services/logger_services
__init__.py	0	0	100%
log_functions.py	0	0	100%
services/output_manager
__init__.py	0	0	100%
error_handler.py	0	1	98%	104
help_page.py	0	12	86%	36–37, 46–47, 59–60, 106–107, 116–117, 129–130
message_handler.py	0	63	60%	17–22, 27, 32, 37, 47, 52–54, 59, 64, 69, 74, 87, 92, 99, 104, 109, 113, 129, 149, 159, 169, 188, 210–221, 232–237, 239–245, 256, 260–261, 265–266, 268–269, 273, 277, 281
response_handler.py	0	53	0%	5–6, 9–12, 14–15, 17–21, 23–24, 26–34, 36–39, 42–45, 47–48, 50–56, 58–59, 61–63, 65–68, 71–73
services/project_manager
__init__.py	0	0	100%
project.py	0	6	83%	38, 44–45, 47–49
services/user_authentication
__init__.py	0	0	100%
decorator.py	0	13	67%	29, 32, 49–53, 55, 59–62, 64
token_manager.py	0	21	57%	25, 28–30, 33, 36–37, 40–41, 61, 65, 69–70, 76–77, 79–82, 84–85
user_login_logout.py	0	43	65%	40–43, 114, 118–124, 126–128, 132–133, 137–144, 146–148, 152–153, 160–165, 167–169, 173–176
user_set_config.py	0	9	47%	15–18, 22–24, 26–27
utils
__init__.py	0	0	100%
aggregated.py	0	31	66%	22, 46, 71–73, 75–76, 83–84, 86, 90–93, 95–96, 115–117, 119, 124–131, 133–135
TOTAL	3061	1430	53%