Skip to content

Training details #48

Closed
Closed
Training details#48

Description

@twinkleyang1
twinkleyang1
opened on Oct 6, 2024

Thank you for your contribution, may I ask how much graphics memory was used when training csc-tuner?

### My graphics card parameters are as follows:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01 Driver Version: 535.183.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3090 Off | 00000000:18:00.0 Off | N/A |
| 36% 46C P8 24W / 350W | 358MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce RTX 3090 Off | 00000000:3B:00.0 Off | N/A |
| 36% 51C P8 39W / 350W | 346MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA GeForce RTX 3090 Off | 00000000:86:00.0 Off | N/A |
| 36% 46C P8 31W / 350W | 346MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 3 NVIDIA GeForce RTX 3090 Off | 00000000:AF:00.0 Off | N/A |
| 36% 45C P8 23W / 350W | 346MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
Can I train the model?

### I am using gradio interface for training, and the running interface is as follows:
interface

### The terminal runs as follows：
start task Swift@stable_diffusion@SD1.5@SCE@202410062018589311915910
PYTHONPATH=. python /data/twinkle/app/scepter/scepter/studio/self_train/scripts/run_task.py --cfg=/data/twinkle/app/scepter/cache/scepter_ui/self_train/Swift@stable_diffusion@SD1.5@SCE@202410062018589311915910/train.yaml 2> ./error_logs/1728217143.log
[Info]: Loading config from /data/twinkle/app/scepter/cache/scepter_ui/self_train/Swift@stable_diffusion@SD1.5@SCE@202410062018589311915910/train.yaml
[Info]: System take /data/twinkle/app/scepter/cache/scepter_ui/self_train/Swift@stable_diffusion@SD1.5@SCE@202410062018589311915910/train.yaml as yaml, because we find yaml in this file
[Info]: SEED is not set and will use default SEED as 2023; If want to change this value, please set it in your config.
[Info]: Parse cfg file as
{
"ENV": {
"BACKEND": "nccl",
"SEED": 2023
},
"SOLVER": {
"ACCU_STEP": 1,
"CHANNELS_LAST": true,
"DTYPE": "float16",
"EVAL_HOOKS": [
{
"NAME": "ProbeDataHook",
"PROB_INTERVAL": 25,
"SAVE_LAST": true,
"SAVE_NAME_PREFIX": "step",
"SAVE_PROBE_PREFIX": "image"
}
],
"EVAL_INTERVAL": -1,
"FILE_SYSTEM": {
"NAME": "ModelscopeFs",
"TEMP_DIR": "./cache/cache_data"
},
"FREEZE": null,
"LOAD_MODEL_ONLY": true,
"LOG_FILE": "std_log.txt",
"MAX_EPOCHS": 200,
"MAX_STEPS": 1000,
"MODEL": {
"COND_STAGE_MODEL": {
"FREEZE": true,
"LAYER": "last",
"NAME": "FrozenCLIPEmbedder",
"PRETRAINED_MODEL": "ms://AI-ModelScope/clip-vit-large-patch14",
"USE_GRAD": false
},
"DEFAULT_N_PROMPT": null,
"DIFFUSION_MODEL": {
"ATTENTION_RESOLUTIONS": [
4,
2,
1
],
"CHANNEL_MULT": [
1,
2,
4,
4
],
"CONTEXT_DIM": 768,
"CONV_RESAMPLE": true,
"DIMS": 2,
"DISABLE_MIDDLE_SELF_ATTN": false,
"IGNORE_KEYS": [],
"IN_CHANNELS": 4,
"MODEL_CHANNELS": 320,
"NAME": "DiffusionUNet",
"NUM_HEADS": 8,
"NUM_RES_BLOCKS": 2,
"OUT_CHANNELS": 4,
"RESBLOCK_UPDOWN": false,
"TRANSFORMER_DEPTH": 1,
"USE_CHECKPOINT": false,
"USE_LINEAR_IN_TRANSFORMER": false,
"USE_SCALE_SHIFT_NORM": false,
"USE_SPATIAL_TRANSFORMER": true
},
"FIRST_STAGE_MODEL": {
"BATCH_SIZE": 4,
"DECODER": {
"ATTN_RESOLUTIONS": [],
"CH": 128,
"CH_MULT": [
1,
2,
4,
4
],
"DROPOUT": 0.0,
"GIVE_PRE_END": false,
"IN_CHANNELS": 3,
"NAME": "Decoder",
"NUM_RES_BLOCKS": 2,
"OUT_CH": 3,
"RESAMP_WITH_CONV": true,
"TANH_OUT": false,
"Z_CHANNELS": 4
},
"EMBED_DIM": 4,
"ENCODER": {
"ATTN_RESOLUTIONS": [],
"CH": 128,
"CH_MULT": [
1,
2,
4,
4
],
"DOUBLE_Z": true,
"DROPOUT": 0.0,
"IN_CHANNELS": 3,
"NAME": "Encoder",
"NUM_RES_BLOCKS": 2,
"OUT_CH": 3,
"RESAMP_WITH_CONV": true,
"Z_CHANNELS": 4
},
"IGNORE_KEYS": [],
"NAME": "AutoencoderKL"
},
"IGNORE_KEYS": [],
"LOSS": {
"LOSS_TYPE": "l2",
"NAME": "ReconstructLoss"
},
"MIN_SNR_GAMMA": null,
"NAME": "LatentDiffusion",
"PARAMETERIZATION": "eps",
"PRETRAINED_MODEL": "ms://AI-ModelScope/stable-diffusion-v1-5@v1-5-pruned-emaonly.safetensors",
"SCALE_FACTOR": 0.18215,
"SCHEDULE_ARGS": {
"BETA_MAX": 0.012,
"BETA_MIN": 0.00085,
"NAME": "scaled_linear"
},
"SIZE_FACTOR": 8,
"TIMESTEPS": 1000,
"TOKENIZER": {
"CLEAN": true,
"LENGTH": 77,
"NAME": "ClipTokenizer",
"PRETRAINED_PATH": "ms://AI-ModelScope/clip-vit-large-patch14"
},
"USE_EMA": false,
"ZERO_TERMINAL_SNR": false
},
"NAME": "LatentDiffusionSolver",
"NUM_FOLDS": 1,
"OPTIMIZER": {
"AMSGRAD": false,
"BETAS": [
0.9,
0.999
],
"EPS": "1e-8",
"LEARNING_RATE": 0.0001,
"NAME": "AdamW",
"WEIGHT_DECAY": "1e-2"
},
"RESCALE_LR": false,
"RESUME_FROM": null,
"SAMPLE_ARGS": {
"DISCRETIZATION": "trailing",
"GUIDE_RESCALE": null,
"GUIDE_SCALE": 7.5,
"IMAGE_SIZE": [
512,
512
],
"RUN_TRAIN_N": false,
"SAMPLER": "ddim",
"SAMPLE_STEPS": 50,
"SEED": 2023
},
"SHARDING_STRATEGY": null,
"TRAIN_DATA": {
"BATCH_SIZE": 1,
"MODE": "train",
"MS_DATASET_NAME": "/data/twinkle/app/scepter/cache/scepter_ui/datasets/scepter_img2img_hed_example",
"MS_DATASET_NAMESPACE": "",
"MS_DATASET_SPLIT": "train",
"MS_DATASET_SUBNAME": "",
"MS_REMAP_KEYS": null,
"MS_REMAP_PATH": "/data/twinkle/app/scepter/cache/scepter_ui/datasets/scepter_img2img_hed_example",
"NAME": "ImageTextPairMSDataset",
"NUM_WORKERS": 4,
"OUTPUT_SIZE": [
512,
512
],
"PIN_MEMORY": true,
"PROMPT_PREFIX": "",
"REPLACE_KEYWORDS": "",
"REPLACE_STYLE": false,
"SAMPLER": {
"NAME": "LoopSampler"
},
"TRANSFORMS": [
{
"BACKEND": "pillow",
"NAME": "LoadImageFromFile",
"RGB_ORDER": "RGB"
},
{
"BACKEND": "pillow",
"INPUT_KEY": [
"img"
],
"INTERPOLATION": "bilinear",
"NAME": "FlexibleResize",
"OUTPUT_KEY": [
"img"
],
"SIZE": [
512,
512
]
},
{
"BACKEND": "pillow",
"INPUT_KEY": [
"img"
],
"NAME": "FlexibleCenterCrop",
"OUTPUT_KEY": [
"img"
],
"SIZE": [
512,
512
]
},
{
"BACKEND": "pillow",
"INPUT_KEY": [
"img"
],
"NAME": "ImageToTensor",
"OUTPUT_KEY": [
"img"
]
},
{
"BACKEND": "torchvision",
"INPUT_KEY": [
"img"
],
"MEAN": [
0.5,
0.5,
0.5
],
"NAME": "Normalize",
"OUTPUT_KEY": [
"image"
],
"STD": [
0.5,
0.5,
0.5
]
},
{
"KEYS": [
"image",
"prompt"
],
"META_KEYS": [
"data_key"
],
"NAME": "Select"
}
]
},
"TRAIN_HOOKS": [
{
"NAME": "BackwardHook",
"PRIORITY": 0
},
{
"LOG_INTERVAL": 10,
"NAME": "LogHook",
"SHOW_GPU_MEM": true
},
{
"NAME": "TensorboardLogHook"
},
{
"DISABLE_SNAPSHOT": true,
"HUB_MODEL_ID": "",
"INTERVAL": 25,
"NAME": "CheckpointHook",
"PRIORITY": 200,
"PUSH_TO_HUB": false,
"SAVE_LAST": true,
"SAVE_NAME_PREFIX": "step"
}
],
"TUNER": [
{
"DIMS": [
1280,
1280,
1280,
1280,
1280,
640,
640,
640,
320,
320,
320,
320
],
"DOWN_RATIO": 1,
"NAME": "SwiftSCETuning",
"TARGET_MODULES": "model.lsc_identity\.\d+$",
"TUNER_MODE": "identity"
}
],
"USE_AMP": true,
"USE_FSDP": false,
"WORK_DIR": "/data/twinkle/app/scepter/cache/scepter_ui/self_train/Swift@stable_diffusion@SD1.5@SCE@202410062018589311915910"
}
}
[Info]: Now running in the distributed environment with world size 4!
[Info]: PMI rank 0!
[Info]: Nums of gpu 4!
[Info]: Current rank 0 current devices num 4 current machine rank 0 and all world size 4
[Info]: Now running in the distributed environment with world size 4!
[Info]: PMI rank 0!
[Info]: Nums of gpu 4!
[Info]: Current rank 1 current devices num 4 current machine rank 0 and all world size 4
[Info]: Now running in the distributed environment with world size 4!
[Info]: PMI rank 0!
[Info]: Nums of gpu 4!
[Info]: Current rank 3 current devices num 4 current machine rank 0 and all world size 4
scepter [INFO] 2024-10-06 20:19:15,985 [File: run_task.py Function: run_task at line 22] Pytorch version: 2.0.1+cu117
scepter [INFO] 2024-10-06 20:19:15,986 [File: run_task.py Function: run_task at line 22] Pytorch version: 2.0.1+cu117
scepter [INFO] 2024-10-06 20:19:15,989 [File: base_solver.py Function: init at line 308] Parse work dir /data/twinkle/app/scepter/cache/scepter_ui/self_train/Swift@stable_diffusion@SD1.5@SCE@202410062018589311915910's prefix is /data/twinkle/app/scepter
scepter [INFO] 2024-10-06 20:19:15,989 [File: diffusion_solver.py Function: init at line 113] Use default backend.
scepter [INFO] 2024-10-06 20:19:15,989 [File: base_solver.py Function: init at line 308] Parse work dir /data/twinkle/app/scepter/cache/scepter_ui/self_train/Swift@stable_diffusion@SD1.5@SCE@202410062018589311915910's prefix is /data/twinkle/app/scepter
scepter [INFO] 2024-10-06 20:19:15,989 [File: diffusion_solver.py Function: init at line 113] Use default backend.
scepter [INFO] 2024-10-06 20:19:15,989 [File: run_task.py Function: run_task at line 22] Pytorch version: 2.0.1+cu117
scepter [INFO] 2024-10-06 20:19:15,990 [File: logger.py Function: init_logger at line 86] Running task with log file: /data/twinkle/app/scepter/cache/scepter_ui/self_train/Swift@stable_diffusion@SD1.5@SCE@202410062018589311915910/std_log.txt
[Info]: Now running in the distributed environment with world size 4!
[Info]: PMI rank 0!
[Info]: Nums of gpu 4!
[Info]: Current rank 2 current devices num 4 current machine rank 0 and all world size 4
scepter [INFO] 2024-10-06 20:19:15,994 [File: base_solver.py Function: init at line 308] Parse work dir /data/twinkle/app/scepter/cache/scepter_ui/self_train/Swift@stable_diffusion@SD1.5@SCE@202410062018589311915910's prefix is /data/twinkle/app/scepter
scepter [INFO] 2024-10-06 20:19:15,994 [File: diffusion_solver.py Function: init at line 113] Use default backend.
scepter [INFO] 2024-10-06 20:19:15,994 [File: run_task.py Function: run_task at line 22] Pytorch version: 2.0.1+cu117
scepter [INFO] 2024-10-06 20:19:15,997 [File: base_solver.py Function: init at line 308] Parse work dir /data/twinkle/app/scepter/cache/scepter_ui/self_train/Swift@stable_diffusion@SD1.5@SCE@202410062018589311915910's prefix is /data/twinkle/app/scepter
scepter [INFO] 2024-10-06 20:19:15,997 [File: diffusion_solver.py Function: init at line 113] Use default backend.
scepter [INFO] 2024-10-06 20:19:16,102 [File: registry.py Function: init at line 185] Built dataloader with len 9223372036854775807
scepter [INFO] 2024-10-06 20:19:16,107 [File: registry.py Function: init at line 185] Built dataloader with len 9223372036854775807
scepter [INFO] 2024-10-06 20:19:16,191 [File: registry.py Function: init at line 185] Built dataloader with len 9223372036854775807
scepter [INFO] 2024-10-06 20:19:16,204 [File: registry.py Function: init at line 185] Built dataloader with len 9223372036854775807
scepter [INFO] 2024-10-06 20:19:22,880 [File: ldm.py Function: construct_network at line 147] all parameters:859.52M
scepter [INFO] 2024-10-06 20:19:23,002 [File: ldm.py Function: construct_network at line 147] all parameters:859.52M
scepter [INFO] 2024-10-06 20:19:23,369 [File: ldm.py Function: construct_network at line 147] all parameters:859.52M
scepter [INFO] 2024-10-06 20:19:23,441 [File: ldm.py Function: construct_network at line 147] all parameters:859.52M
scepter [INFO] 2024-10-06 20:19:23,731 [File: ae_module.py Function: construct_model at line 76] AE Module XFORMERS_IS_AVAILBLE : True
scepter [INFO] 2024-10-06 20:19:23,809 [File: ae_module.py Function: construct_model at line 76] AE Module XFORMERS_IS_AVAILBLE : True
scepter [INFO] 2024-10-06 20:19:24,129 [File: ae_module.py Function: construct_model at line 76] AE Module XFORMERS_IS_AVAILBLE : True
scepter [INFO] 2024-10-06 20:19:24,207 [File: ae_module.py Function: construct_model at line 76] AE Module XFORMERS_IS_AVAILBLE : True
scepter [INFO] 2024-10-06 20:19:25,563 [File: ldm.py Function: init_from_ckpt at line 218] Restored from ./cache/cache_data/95c8c1dcca842facbdcb9606c314f7ae/AI-ModelScope/stable-diffusion-v1-5/v1-5-pruned-emaonly.safetensors with 0 missing and 15 unexpected keys
scepter [INFO] 2024-10-06 20:19:25,564 [File: ldm.py Function: init_from_ckpt at line 224]
Unexpected Keys:
['alphas_cumprod', 'alphas_cumprod_prev', 'betas', 'log_one_minus_alphas_cumprod', 'model_ema.decay', 'model_ema.num_updates', 'posterior_log_variance_clipped', 'posterior_mean_coef1', 'posterior_mean_coef2', 'posterior_variance', 'sqrt_alphas_cumprod', 'sqrt_one_minus_alphas_cumprod', 'sqrt_recip_alphas_cumprod', 'sqrt_recipm1_alphas_cumprod', 'cond_stage_model.transformer.text_model.embeddings.position_ids']
scepter [INFO] 2024-10-06 20:19:29,802 [File: diffusion_solver.py Function: print_model_params_status at line 745] Load trainable params 19680000 / 1085915307 = 1.81%, train part: {'model.lsc_identity': 19680000}.
scepter [INFO] 2024-10-06 20:19:29,803 [File: diffusion_solver.py Function: print_model_params_status at line 749] Load forzen params 1066235307 / 1085915307 = 98.19%, forzen part: {'model': 859520964, 'first_stage_model': 83653863, 'cond_stage_model': 123060480}.
scepter [INFO] 2024-10-06 20:19:32,473 [File: diffusion_solver.py Function: print_model_params_status at line 745] Load trainable params 19680000 / 1085915307 = 1.81%, train part: {'model.lsc_identity': 19680000}.
scepter [INFO] 2024-10-06 20:19:32,474 [File: diffusion_solver.py Function: print_model_params_status at line 749] Load forzen params 1066235307 / 1085915307 = 98.19%, forzen part: {'model': 859520964, 'first_stage_model': 83653863, 'cond_stage_model': 123060480}.
scepter [INFO] 2024-10-06 20:19:32,541 [File: diffusion_solver.py Function: print_model_params_status at line 745] Load trainable params 19680000 / 1085915307 = 1.81%, train part: {'model.lsc_identity': 19680000}.
scepter [INFO] 2024-10-06 20:19:32,542 [File: diffusion_solver.py Function: print_model_params_status at line 749] Load forzen params 1066235307 / 1085915307 = 98.19%, forzen part: {'model': 859520964, 'first_stage_model': 83653863, 'cond_stage_model': 123060480}.
scepter [INFO] 2024-10-06 20:19:32,628 [File: diffusion_solver.py Function: print_model_params_status at line 745] Load trainable params 19680000 / 1085915307 = 1.81%, train part: {'model.lsc_identity': 19680000}.
scepter [INFO] 2024-10-06 20:19:32,628 [File: diffusion_solver.py Function: print_model_params_status at line 749] Load forzen params 1066235307 / 1085915307 = 98.19%, forzen part: {'model': 859520964, 'first_stage_model': 83653863, 'cond_stage_model': 123060480}.
scepter [INFO] 2024-10-06 20:19:36,920 [File: run_task.py Function: run_task at line 37] max step is changed from 1000 to 500 according to the setting epoches 200 and dataset size 10
scepter [INFO] 2024-10-06 20:19:36,920 [File: run_task.py Function: run_task at line 48] checkpoint save interval is changed from 25 to 62 according to the setting epoches interval 25
scepter [INFO] 2024-10-06 20:19:36,920 [File: run_task.py Function: run_task at line 60] prob interval is changed from 25 to 62 according to the setting epoches interval 25
scepter [INFO] 2024-10-06 20:19:36,920 [File: run_task.py Function: run_task at line 37] max step is changed from 1000 to 500 according to the setting epoches 200 and dataset size 10
scepter [INFO] 2024-10-06 20:19:36,920 [File: run_task.py Function: run_task at line 48] checkpoint save interval is changed from 25 to 62 according to the setting epoches interval 25
scepter [INFO] 2024-10-06 20:19:36,920 [File: run_task.py Function: run_task at line 60] prob interval is changed from 25 to 62 according to the setting epoches interval 25
scepter [INFO] 2024-10-06 20:19:36,922 [File: log.py Function: before_solve at line 248] Tensorboard: save to /data/twinkle/app/scepter/cache/scepter_ui/self_train/Swift@stable_diffusion@SD1.5@SCE@202410062018589311915910/tensorboard
scepter [INFO] 2024-10-06 20:19:36,923 [File: run_task.py Function: run_task at line 37] max step is changed from 1000 to 500 according to the setting epoches 200 and dataset size 10
scepter [INFO] 2024-10-06 20:19:36,924 [File: run_task.py Function: run_task at line 48] checkpoint save interval is changed from 25 to 62 according to the setting epoches interval 25
scepter [INFO] 2024-10-06 20:19:36,924 [File: run_task.py Function: run_task at line 60] prob interval is changed from 25 to 62 according to the setting epoches interval 25
scepter [INFO] 2024-10-06 20:19:36,924 [File: run_task.py Function: run_task at line 37] max step is changed from 1000 to 500 according to the setting epoches 200 and dataset size 10
scepter [INFO] 2024-10-06 20:19:36,924 [File: run_task.py Function: run_task at line 48] checkpoint save interval is changed from 25 to 62 according to the setting epoches interval 25
scepter [INFO] 2024-10-06 20:19:36,924 [File: run_task.py Function: run_task at line 60] prob interval is changed from 25 to 62 according to the setting epoches interval 25
scepter [INFO] 2024-10-06 20:19:51,817 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [10/500], data_time: 1.0402(1.0402), time: 1.4696(1.4696), loss: 0.0978(0.0978), nvidia-smi: 9885.4000(9885.4000), pg0_lr: 0.000100, scale: 65536.000000, [35secs 2.00%(29mins 15secs)]
scepter [INFO] 2024-10-06 20:19:51,839 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [10/500], data_time: 1.0282(1.0282), time: 1.4694(1.4694), loss: 0.0978(0.0978), nvidia-smi: 9885.4000(9885.4000), pg0_lr: 0.000100, scale: 65536.000000, [35secs 2.00%(29mins 16secs)]
scepter [INFO] 2024-10-06 20:19:51,865 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [10/500], data_time: 1.0281(1.0281), time: 1.4692(1.4692), loss: 0.0978(0.0978), nvidia-smi: 9885.4000(9885.4000), pg0_lr: 0.000100, scale: 65536.000000, [35secs 2.00%(29mins 17secs)]
scepter [INFO] 2024-10-06 20:19:51,904 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [10/500], data_time: 1.0530(1.0530), time: 1.4691(1.4691), loss: 0.0978(0.0978), nvidia-smi: 9885.4000(9885.4000), pg0_lr: 0.000100, scale: 65536.000000, [35secs 2.00%(29mins 19secs)]
scepter [INFO] 2024-10-06 20:19:57,188 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [20/500], data_time: 0.2501(0.6392), time: 0.5352(1.002325), loss: 0.0645(0.0811), nvidia-smi: 9886.000000(9885.7000), pg0_lr: 0.000100, scale: 65536.000000, [41secs 4.00%(16mins 28secs)]
scepter [INFO] 2024-10-06 20:19:57,247 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [20/500], data_time: 0.2476(0.6439), time: 0.5353(1.002450), loss: 0.0645(0.0811), nvidia-smi: 9886.000000(9885.7000), pg0_lr: 0.000100, scale: 65536.000000, [41secs 4.00%(16mins 30secs)]
scepter [INFO] 2024-10-06 20:19:57,278 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [20/500], data_time: 0.2841(0.6561), time: 0.5354(1.002290), loss: 0.0645(0.0811), nvidia-smi: 9886.000000(9885.7000), pg0_lr: 0.000100, scale: 65536.000000, [41secs 4.00%(16mins 30secs)]
scepter [INFO] 2024-10-06 20:19:57,296 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [20/500], data_time: 0.2969(0.6749), time: 0.5353(1.002225), loss: 0.0645(0.0811), nvidia-smi: 9886.000000(9885.7000), pg0_lr: 0.000100, scale: 65536.000000, [41secs 4.00%(16mins 31secs)]
scepter [INFO] 2024-10-06 20:20:02,547 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [30/500], data_time: 0.2788(0.5303), time: 0.5348(0.8464), loss: 0.1063(0.0895), nvidia-smi: 9886.000000(9885.8000), pg0_lr: 0.000100, scale: 65536.000000, [46secs 6.00%(12mins 9secs)]
scepter [INFO] 2024-10-06 20:20:02,566 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [30/500], data_time: 0.2970(0.5489), time: 0.5348(0.8464), loss: 0.1063(0.0895), nvidia-smi: 9886.000000(9885.8000), pg0_lr: 0.000100, scale: 65536.000000, [46secs 6.00%(12mins 9secs)]
scepter [INFO] 2024-10-06 20:20:02,597 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [30/500], data_time: 0.2505(0.5096), time: 0.5349(0.8465), loss: 0.1063(0.0895), nvidia-smi: 9886.000000(9885.8000), pg0_lr: 0.000100, scale: 65536.000000, [46secs 6.00%(12mins 10secs)]
scepter [INFO] 2024-10-06 20:20:02,622 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [30/500], data_time: 0.2486(0.5121), time: 0.5348(0.8466), loss: 0.1063(0.0895), nvidia-smi: 9886.000000(9885.8000), pg0_lr: 0.000100, scale: 65536.000000, [46secs 6.00%(12mins 10secs)]
scepter [INFO] 2024-10-06 20:20:07,786 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [40/500], data_time: 0.2539(0.4457), time: 0.5251(0.7662), loss: 0.1449(0.1034), nvidia-smi: 9886.000000(9885.8500), pg0_lr: 0.000100, scale: 65536.000000, [51secs 8.00%(9mins 55secs)]
scepter [INFO] 2024-10-06 20:20:07,835 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [40/500], data_time: 0.2501(0.4466), time: 0.5251(0.7662), loss: 0.1449(0.1034), nvidia-smi: 9886.000000(9885.8500), pg0_lr: 0.000100, scale: 65536.000000, [51secs 8.00%(9mins 56secs)]
scepter [INFO] 2024-10-06 20:20:07,837 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [40/500], data_time: 0.2520(0.4608), time: 0.5251(0.7661), loss: 0.1449(0.1034), nvidia-smi: 9886.000000(9885.8500), pg0_lr: 0.000100, scale: 65536.000000, [51secs 8.00%(9mins 56secs)]
scepter [INFO] 2024-10-06 20:20:07,876 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [40/500], data_time: 0.2966(0.4859), time: 0.5251(0.7661), loss: 0.1449(0.1034), nvidia-smi: 9886.000000(9885.8500), pg0_lr: 0.000100, scale: 65536.000000, [51secs 8.00%(9mins 56secs)]
scepter [INFO] 2024-10-06 20:20:13,191 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [50/500], data_time: 0.2550(0.4083), time: 0.5387(0.7207), loss: 0.1633(0.1154), nvidia-smi: 9886.000000(9885.8800), pg0_lr: 0.000100, scale: 65536.000000, [57secs 10.00%(8mins 34secs)]
scepter [INFO] 2024-10-06 20:20:13,219 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [50/500], data_time: 0.2823(0.4251), time: 0.5387(0.7206), loss: 0.1633(0.1154), nvidia-smi: 9886.000000(9885.8800), pg0_lr: 0.000100, scale: 65536.000000, [57secs 10.00%(8mins 34secs)]
scepter [INFO] 2024-10-06 20:20:13,249 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [50/500], data_time: 0.2484(0.4062), time: 0.5387(0.7207), loss: 0.1633(0.1154), nvidia-smi: 9886.000000(9885.8800), pg0_lr: 0.000100, scale: 65536.000000, [57secs 10.00%(8mins 35secs)]
scepter [INFO] 2024-10-06 20:20:13,277 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [50/500], data_time: 0.2944(0.4476), time: 0.5389(0.7206), loss: 0.1633(0.1154), nvidia-smi: 9886.000000(9885.8800), pg0_lr: 0.000100, scale: 65536.000000, [57secs 10.00%(8mins 35secs)]
scepter [INFO] 2024-10-06 20:20:18,492 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [60/500], data_time: 0.2386(0.3800), time: 0.5300(0.6889), loss: 0.1802(0.1262), nvidia-smi: 9886.000000(9885.9000), pg0_lr: 0.000100, scale: 65536.000000, [1mins 2secs 12.00%(7mins 38secs)]
scepter [INFO] 2024-10-06 20:20:18,550 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [60/500], data_time: 0.2662(0.3986), time: 0.5300(0.6889), loss: 0.1802(0.1262), nvidia-smi: 9886.000000(9885.9000), pg0_lr: 0.000100, scale: 65536.000000, [1mins 2secs 12.00%(7mins 38secs)]
scepter [INFO] 2024-10-06 20:20:18,553 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [60/500], data_time: 0.2609(0.3820), time: 0.5300(0.6889), loss: 0.1802(0.1262), nvidia-smi: 9886.000000(9885.9000), pg0_lr: 0.000100, scale: 65536.000000, [1mins 2secs 12.00%(7mins 38secs)]
scepter [INFO] 2024-10-06 20:20:18,576 [File: log.py Function: _print_iter_log at line 71] Stage [train] iter: [60/500], data_time: 0.2986(0.4227), time: 0.5301(0.6889), loss: 0.1802(0.1262), nvidia-smi: 9886.000000(9885.9000), pg0_lr: 0.000100, scale: 65536.000000, [1mins 2secs 12.00%(7mins 38secs)]
scepter [INFO] 2024-10-06 20:20:19,544 [File: checkpoint.py Function: after_iter at line 110] Saving checkpoint after 62 steps
scepter [INFO] 2024-10-06 20:20:19,571 [File: checkpoint.py Function: after_iter at line 110] Saving checkpoint after 62 steps
scepter [INFO] 2024-10-06 20:20:19,598 [File: checkpoint.py Function: after_iter at line 110] Saving checkpoint after 62 steps
scepter [INFO] 2024-10-06 20:20:19,625 [File: checkpoint.py Function: after_iter at line 110] Saving checkpoint after 62 steps
Terminate 955768 ...
sh: 1: kill: No such process

Terminate 955768 success.
sh: 1: kill: No such process

error_logs is as follows:

2024-10-06 20:19:08,609 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:08,611 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:08,644 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
2024-10-06 20:19:15,110 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:15,111 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:15,115 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:15,116 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:15,147 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:15,149 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:15,220 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:15,222 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:15,255 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:15,272 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:15,273 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:15,309 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/datasets/load.py:2524: FutureWarning: 'ignore_verifications' was deprecated in favor of 'verification_mode' in version 2.9.1 and will be removed in 3.0.0.
You can remove this warning by passing 'verification_mode=no_checks' instead.
warnings.warn(
2024-10-06 20:19:24,854 - modelscope - WARNING - Model revision not specified, use revision: v1.0.9
/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/datasets/load.py:2524: FutureWarning: 'ignore_verifications' was deprecated in favor of 'verification_mode' in version 2.9.1 and will be removed in 3.0.0.
You can remove this warning by passing 'verification_mode=no_checks' instead.
warnings.warn(
2024-10-06 20:19:25,096 - modelscope - WARNING - Model revision not specified, use revision: v1.0.9
/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/datasets/load.py:2524: FutureWarning: 'ignore_verifications' was deprecated in favor of 'verification_mode' in version 2.9.1 and will be removed in 3.0.0.
You can remove this warning by passing 'verification_mode=no_checks' instead.
warnings.warn(
2024-10-06 20:19:25,328 - modelscope - WARNING - Model revision not specified, use revision: v1.0.9
/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/datasets/load.py:2524: FutureWarning: 'ignore_verifications' was deprecated in favor of 'verification_mode' in version 2.9.1 and will be removed in 3.0.0.
You can remove this warning by passing 'verification_mode=no_checks' instead.
warnings.warn(
2024-10-06 20:19:25,444 - modelscope - WARNING - Model revision not specified, use revision: v1.0.9
2024-10-06 20:19:25,811 - swift - INFO - Matching target module [model.lsc_identity.0] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:25,811 - swift - INFO - Matching target module [model.lsc_identity.1] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:25,811 - swift - INFO - Matching target module [model.lsc_identity.2] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:25,811 - swift - INFO - Matching target module [model.lsc_identity.3] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:25,811 - swift - INFO - Matching target module [model.lsc_identity.4] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:25,811 - swift - INFO - Matching target module [model.lsc_identity.5] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:25,811 - swift - INFO - Matching target module [model.lsc_identity.6] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:25,811 - swift - INFO - Matching target module [model.lsc_identity.7] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:25,811 - swift - INFO - Matching target module [model.lsc_identity.8] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:25,812 - swift - INFO - Matching target module [model.lsc_identity.9] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:25,812 - swift - INFO - Matching target module [model.lsc_identity.10] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:25,812 - swift - INFO - Matching target module [model.lsc_identity.11] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,445 - swift - INFO - Matching target module [model.lsc_identity.0] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,445 - swift - INFO - Matching target module [model.lsc_identity.1] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,445 - swift - INFO - Matching target module [model.lsc_identity.2] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,445 - swift - INFO - Matching target module [model.lsc_identity.3] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,445 - swift - INFO - Matching target module [model.lsc_identity.4] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,445 - swift - INFO - Matching target module [model.lsc_identity.5] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,445 - swift - INFO - Matching target module [model.lsc_identity.6] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,445 - swift - INFO - Matching target module [model.lsc_identity.7] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,445 - swift - INFO - Matching target module [model.lsc_identity.8] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,446 - swift - INFO - Matching target module [model.lsc_identity.9] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,446 - swift - INFO - Matching target module [model.lsc_identity.10] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,446 - swift - INFO - Matching target module [model.lsc_identity.11] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,532 - swift - INFO - Matching target module [model.lsc_identity.0] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,533 - swift - INFO - Matching target module [model.lsc_identity.1] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,533 - swift - INFO - Matching target module [model.lsc_identity.2] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,533 - swift - INFO - Matching target module [model.lsc_identity.3] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,533 - swift - INFO - Matching target module [model.lsc_identity.4] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,533 - swift - INFO - Matching target module [model.lsc_identity.5] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,533 - swift - INFO - Matching target module [model.lsc_identity.6] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,533 - swift - INFO - Matching target module [model.lsc_identity.7] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,533 - swift - INFO - Matching target module [model.lsc_identity.8] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,533 - swift - INFO - Matching target module [model.lsc_identity.9] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,533 - swift - INFO - Matching target module [model.lsc_identity.10] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,533 - swift - INFO - Matching target module [model.lsc_identity.11] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,681 - swift - INFO - Matching target module [model.lsc_identity.0] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,681 - swift - INFO - Matching target module [model.lsc_identity.1] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,681 - swift - INFO - Matching target module [model.lsc_identity.2] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,681 - swift - INFO - Matching target module [model.lsc_identity.3] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,681 - swift - INFO - Matching target module [model.lsc_identity.4] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,681 - swift - INFO - Matching target module [model.lsc_identity.5] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,681 - swift - INFO - Matching target module [model.lsc_identity.6] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,681 - swift - INFO - Matching target module [model.lsc_identity.7] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,681 - swift - INFO - Matching target module [model.lsc_identity.8] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,681 - swift - INFO - Matching target module [model.lsc_identity.9] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,681 - swift - INFO - Matching target module [model.lsc_identity.10] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:30,681 - swift - INFO - Matching target module [model.lsc_identity.11] of type <class 'torch.nn.modules.linear.Identity'>
2024-10-06 20:19:44,013 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:44,015 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:44,051 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:44,069 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:44,070 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:44,087 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:44,089 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:44,106 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:44,110 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:44,111 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:44,111 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:44,113 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:44,125 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:44,148 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:44,148 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:44,187 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:44,188 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:44,188 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:44,189 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:44,223 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:44,224 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:44,226 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:44,226 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:44,227 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:44,228 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:44,228 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:44,229 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:44,261 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:44,265 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:44,265 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:44,309 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:44,311 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:44,348 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:44,383 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:44,385 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:44,395 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:44,396 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:44,422 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:44,433 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:44,505 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:44,506 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:44,530 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:44,532 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:44,543 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:44,549 - modelscope - INFO - PyTorch version 2.0.1 Found.
2024-10-06 20:19:44,550 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
2024-10-06 20:19:44,568 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
2024-10-06 20:19:44,587 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
[W reducer.cpp:1300] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
[W reducer.cpp:1300] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
[W reducer.cpp:1300] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
[W reducer.cpp:1300] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
Traceback (most recent call last):
File "/data/twinkle/app/scepter/scepter/studio/self_train/scripts/run_task.py", line 180, in
we.init_env(cfg, logger=None, fn=run_task)
File "/data/twinkle/app/scepter/scepter/modules/utils/distribute.py", line 431, in init_env
mp.spawn(mp_worker,
File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 239, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 197, in start_processes
while not context.join():
File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 160, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
fn(i, *args)
File "/data/twinkle/app/scepter/scepter/modules/utils/distribute.py", line 314, in mp_worker
fn(cfg)
File "/data/twinkle/app/scepter/scepter/studio/self_train/scripts/run_task.py", line 66, in run_task
solver.solve()
File "/data/twinkle/app/scepter/scepter/modules/solver/diffusion_solver.py", line 328, in solve
self.run_train()
File "/data/twinkle/app/scepter/scepter/modules/solver/diffusion_solver.py", line 367, in run_train
self.after_iter(self.hooks_dict[self._mode])
File "/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py", line 613, in after_iter
[t.after_iter(self) for t in hooks]
File "/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py", line 613, in
[t.after_iter(self) for t in hooks]
File "/data/twinkle/app/scepter/scepter/modules/solver/hooks/checkpoint.py", line 140, in after_iter
ckpt, cfg = solver.save_pretrained()
File "/data/twinkle/app/scepter/scepter/modules/solver/diffusion_solver.py", line 311, in save_pretrained
ckpt = self.model.module.save_pretrained()
TypeError: save_pretrained() missing 1 required positional argument: 'save_directory'

/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
/data/twinkle/app/scepter/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
warnings.warn(f'{e}')
/data/twinkle/anaconda3/envs/scepter/lib/python3.8/multiprocessing/resource_tracker.py:203: UserWarning: resource_tracker: There appear to be 143 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

Activity

  1. jiangzeyinzi commented on Oct 6, 2024

    @jiangzeyinzi
    jiangzeyinzi
    on Oct 6, 2024
    Collaborator
    1. To train csc-tuner, we used bs=64, 34G memory as shown in table2 in the paper. you can lower the bs appropriately to meet your training needs.
    2. The problem that occurs with training on gradio is caused by improper operation of saving ckpt when training sc-tuner with multiple GPUs, we will fix it in the next version, in the meantime you can set up to use a single card mode to circumvent the problem temporarily.

  2. twinkleyang1 commented on Oct 8, 2024

    @twinkleyang1
    twinkleyang1
    on Oct 8, 2024
    Author

    @jiangzeyinzi Thanks for your reply, I use single GPU training, but I still have the following problems:(scepter) (base) twinkle@login-SYS-7049GP-TRT:/data/twinkle/app/scepter$ CUDA_VISIBLE_DEVICES=0 python scepter/tools/run_train.py --cfg scepter/methods/scedit/ctr/sd21_768_sce_ctr_canny.yaml
    2024-10-08 13:16:32,173 - modelscope - INFO - PyTorch version 2.0.1 Found.
    2024-10-08 13:16:32,175 - modelscope - INFO - Loading ast index from /home/twinkle/.cache/modelscope/ast_indexer
    2024-10-08 13:16:32,209 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1966a94e015adba32f1fb346597a21b0 and a total number of 976 components indexed
    /data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/scepter/modules/solver/base_solver.py:185: UserWarning: No module named 'pytorch_lightning'
    warnings.warn(f'{e}')
    [Info]: Loading config from scepter/methods/scedit/ctr/sd21_768_sce_ctr_canny.yaml
    [Info]: System take scepter/methods/scedit/ctr/sd21_768_sce_ctr_canny.yaml as yaml, because we find yaml in this file
    [Info]: SEED is not set and will use default SEED as 2023; If want to change this value, please set it in your config.
    [Info]: Parse cfg file as
    {
    "ENV": {
    "BACKEND": null,
    "SEED": 2023
    },
    "SOLVER": {
    "NAME": "LatentDiffusionSolver",
    "RESUME_FROM": null,
    "LOAD_MODEL_ONLY": true,
    "USE_FSDP": false,
    "SHARDING_STRATEGY": null,
    "USE_AMP": true,
    "DTYPE": "float16",
    "CHANNELS_LAST": true,
    "MAX_STEPS": 200,
    "MAX_EPOCHS": -1,
    "NUM_FOLDS": 1,
    "ACCU_STEP": 1,
    "EVAL_INTERVAL": 100,
    "RESCALE_LR": false,
    "WORK_DIR": "./cache/save_data/sd21_768_sce_ctr_canny",
    "LOG_FILE": "std_log.txt",
    "FILE_SYSTEM": {
    "NAME": "ModelscopeFs",
    "TEMP_DIR": "./cache/cache_data"
    },
    "FREEZE": {
    "FREEZE_PART": [
    "first_stage_model",
    "cond_stage_model",
    "model"
    ],
    "TRAIN_PART": [
    "control_blocks"
    ]
    },
    "MODEL": {
    "NAME": "LatentDiffusionSCEControl",
    "PARAMETERIZATION": "v",
    "TIMESTEPS": 1000,
    "MIN_SNR_GAMMA": null,
    "ZERO_TERMINAL_SNR": false,
    "PRETRAINED_MODEL": "ms://AI-ModelScope/stable-diffusion-2-1@v2-1_768-ema-pruned.safetensors",
    "IGNORE_KEYS": [],
    "SCALE_FACTOR": 0.18215,
    "SIZE_FACTOR": 8,
    "DEFAULT_N_PROMPT": null,
    "SCHEDULE_ARGS": {
    "NAME": "scaled_linear",
    "BETA_MIN": 0.00085,
    "BETA_MAX": 0.012
    },
    "USE_EMA": false,
    "DIFFUSION_MODEL": {
    "NAME": "DiffusionUNet",
    "IN_CHANNELS": 4,
    "OUT_CHANNELS": 4,
    "MODEL_CHANNELS": 320,
    "NUM_HEADS_CHANNELS": 64,
    "NUM_RES_BLOCKS": 2,
    "ATTENTION_RESOLUTIONS": [
    4,
    2,
    1
    ],
    "CHANNEL_MULT": [
    1,
    2,
    4,
    4
    ],
    "CONV_RESAMPLE": true,
    "DIMS": 2,
    "USE_CHECKPOINT": false,
    "USE_SCALE_SHIFT_NORM": false,
    "RESBLOCK_UPDOWN": false,
    "USE_SPATIAL_TRANSFORMER": true,
    "TRANSFORMER_DEPTH": 1,
    "CONTEXT_DIM": 1024,
    "DISABLE_MIDDLE_SELF_ATTN": false,
    "USE_LINEAR_IN_TRANSFORMER": true,
    "PRETRAINED_MODEL": null,
    "IGNORE_KEYS": []
    },
    "FIRST_STAGE_MODEL": {
    "NAME": "AutoencoderKL",
    "EMBED_DIM": 4,
    "PRETRAINED_MODEL": null,
    "IGNORE_KEYS": [],
    "BATCH_SIZE": 4,
    "ENCODER": {
    "NAME": "Encoder",
    "CH": 128,
    "OUT_CH": 3,
    "NUM_RES_BLOCKS": 2,
    "IN_CHANNELS": 3,
    "ATTN_RESOLUTIONS": [],
    "CH_MULT": [
    1,
    2,
    4,
    4
    ],
    "Z_CHANNELS": 4,
    "DOUBLE_Z": true,
    "DROPOUT": 0.0,
    "RESAMP_WITH_CONV": true
    },
    "DECODER": {
    "NAME": "Decoder",
    "CH": 128,
    "OUT_CH": 3,
    "NUM_RES_BLOCKS": 2,
    "IN_CHANNELS": 3,
    "ATTN_RESOLUTIONS": [],
    "CH_MULT": [
    1,
    2,
    4,
    4
    ],
    "Z_CHANNELS": 4,
    "DROPOUT": 0.0,
    "RESAMP_WITH_CONV": true,
    "GIVE_PRE_END": false,
    "TANH_OUT": false
    }
    },
    "TOKENIZER": {
    "NAME": "OpenClipTokenizer",
    "LENGTH": 77
    },
    "COND_STAGE_MODEL": {
    "NAME": "FrozenOpenCLIPEmbedder",
    "ARCH": "ViT-H-14",
    "PRETRAINED_MODEL": null,
    "LAYER": "penultimate"
    },
    "LOSS": {
    "NAME": "ReconstructLoss",
    "LOSS_TYPE": "l2"
    },
    "CONTROL_MODEL": {
    "NAME": "CSCTuners",
    "PRE_HINT_IN_CHANNELS": 3,
    "PRE_HINT_OUT_CHANNELS": 256,
    "DENSE_HINT_KERNAL": 3,
    "SCALE": 1.0,
    "SC_TUNER_CFG": {
    "NAME": "SCTuner",
    "TUNER_NAME": "SCEAdapter",
    "DOWN_RATIO": 1.0
    }
    },
    "CONTROL_ANNO": {
    "NAME": "CannyAnnotator",
    "LOW_THRESHOLD": 100,
    "HIGH_THRESHOLD": 200
    }
    },
    "SAMPLE_ARGS": {
    "SAMPLER": "ddim",
    "SAMPLE_STEPS": 50,
    "SEED": 2023,
    "GUIDE_SCALE": 7.5,
    "GUIDE_RESCALE": 0.5,
    "DISCRETIZATION": "trailing",
    "IMAGE_SIZE": [
    768,
    768
    ],
    "RUN_TRAIN_N": false
    },
    "OPTIMIZER": {
    "NAME": "AdamW",
    "LEARNING_RATE": 0.0001,
    "BETAS": [
    0.9,
    0.999
    ],
    "EPS": "1e-8",
    "WEIGHT_DECAY": "1e-2",
    "AMSGRAD": false
    },
    "TRAIN_DATA": {
    "NAME": "ImageTextPairMSDataset",
    "MODE": "train",
    "MS_DATASET_NAME": "style_custom_dataset",
    "MS_DATASET_NAMESPACE": "damo",
    "MS_DATASET_SUBNAME": "3D",
    "PROMPT_PREFIX": "",
    "MS_DATASET_SPLIT": "train_short",
    "MS_REMAP_KEYS": {
    "Image:FILE": "Target:FILE"
    },
    "REPLACE_STYLE": false,
    "PIN_MEMORY": true,
    "BATCH_SIZE": 1,
    "NUM_WORKERS": 4,
    "SAMPLER": {
    "NAME": "LoopSampler"
    },
    "TRANSFORMS": [
    {
    "NAME": "LoadImageFromFile",
    "RGB_ORDER": "RGB",
    "BACKEND": "pillow"
    },
    {
    "NAME": "Resize",
    "SIZE": 768,
    "INTERPOLATION": "bilinear",
    "INPUT_KEY": [
    "img"
    ],
    "OUTPUT_KEY": [
    "img"
    ],
    "BACKEND": "pillow"
    },
    {
    "NAME": "CenterCrop",
    "SIZE": 768,
    "INPUT_KEY": [
    "img"
    ],
    "OUTPUT_KEY": [
    "img"
    ],
    "BACKEND": "pillow"
    },
    {
    "NAME": "ToNumpy",
    "INPUT_KEY": [
    "img"
    ],
    "OUTPUT_KEY": [
    "image_preprocess"
    ]
    },
    {
    "NAME": "ImageToTensor",
    "INPUT_KEY": [
    "img"
    ],
    "OUTPUT_KEY": [
    "img"
    ],
    "BACKEND": "pillow"
    },
    {
    "NAME": "Normalize",
    "MEAN": [
    0.5,
    0.5,
    0.5
    ],
    "STD": [
    0.5,
    0.5,
    0.5
    ],
    "INPUT_KEY": [
    "img"
    ],
    "OUTPUT_KEY": [
    "img"
    ],
    "BACKEND": "torchvision"
    },
    {
    "NAME": "Rename",
    "INPUT_KEY": [
    "img",
    "image_preprocess"
    ],
    "OUTPUT_KEY": [
    "image",
    "image_preprocess"
    ]
    },
    {
    "NAME": "Select",
    "KEYS": [
    "image",
    "prompt",
    "image_preprocess"
    ],
    "META_KEYS": [
    "data_key"
    ]
    }
    ]
    },
    "EVAL_DATA": {
    "NAME": "ImageTextPairMSDataset",
    "MODE": "eval",
    "MS_DATASET_NAME": "style_custom_dataset",
    "MS_DATASET_NAMESPACE": "damo",
    "MS_DATASET_SUBNAME": "3D",
    "PROMPT_PREFIX": "",
    "MS_DATASET_SPLIT": "train_short",
    "MS_REMAP_KEYS": {
    "Image:FILE": "Target:FILE"
    },
    "REPLACE_STYLE": false,
    "PIN_MEMORY": true,
    "BATCH_SIZE": 10,
    "NUM_WORKERS": 4,
    "TRANSFORMS": [
    {
    "NAME": "LoadImageFromFile",
    "RGB_ORDER": "RGB",
    "BACKEND": "pillow"
    },
    {
    "NAME": "Resize",
    "SIZE": 768,
    "INTERPOLATION": "bilinear",
    "INPUT_KEY": [
    "img"
    ],
    "OUTPUT_KEY": [
    "img"
    ],
    "BACKEND": "pillow"
    },
    {
    "NAME": "CenterCrop",
    "SIZE": 768,
    "INPUT_KEY": [
    "img"
    ],
    "OUTPUT_KEY": [
    "img"
    ],
    "BACKEND": "pillow"
    },
    {
    "NAME": "ToNumpy",
    "INPUT_KEY": [
    "img"
    ],
    "OUTPUT_KEY": [
    "image_preprocess"
    ]
    },
    {
    "NAME": "ImageToTensor",
    "INPUT_KEY": [
    "img"
    ],
    "OUTPUT_KEY": [
    "img"
    ],
    "BACKEND": "pillow"
    },
    {
    "NAME": "Normalize",
    "MEAN": [
    0.5,
    0.5,
    0.5
    ],
    "STD": [
    0.5,
    0.5,
    0.5
    ],
    "INPUT_KEY": [
    "img"
    ],
    "OUTPUT_KEY": [
    "img"
    ],
    "BACKEND": "torchvision"
    },
    {
    "NAME": "Rename",
    "INPUT_KEY": [
    "img",
    "image_preprocess"
    ],
    "OUTPUT_KEY": [
    "image",
    "image_preprocess"
    ]
    },
    {
    "NAME": "Select",
    "KEYS": [
    "image",
    "prompt",
    "image_preprocess"
    ],
    "META_KEYS": [
    "data_key"
    ]
    }
    ]
    },
    "TRAIN_HOOKS": [
    {
    "NAME": "BackwardHook",
    "PRIORITY": 0
    },
    {
    "NAME": "LogHook",
    "LOG_INTERVAL": 50
    },
    {
    "NAME": "CheckpointHook",
    "INTERVAL": 100
    },
    {
    "NAME": "ProbeDataHook",
    "PROB_INTERVAL": 100
    }
    ],
    "EVAL_HOOKS": [
    {
    "NAME": "ProbeDataHook",
    "PROB_INTERVAL": 100
    }
    ]
    }
    }
    scepter [INFO] 2024-10-08 13:16:32,896 [File: base_solver.py Function: init at line 308] Parse work dir ./cache/save_data/sd21_768_sce_ctr_canny's prefix is /data/twinkle/app/scepter
    scepter [INFO] 2024-10-08 13:16:32,896 [File: diffusion_solver.py Function: init at line 113] Use default backend.
    scepter [INFO] 2024-10-08 13:16:32,896 [File: logger.py Function: init_logger at line 86] Running task with log file: /data/twinkle/app/scepter/./cache/save_data/sd21_768_sce_ctr_canny/std_log.txt
    2024-10-08 13:16:33,386 - modelscope - INFO - dataset_type: 2
    2024-10-08 13:16:34,603 - modelscope - WARNING - Reusing dataset dataset_builder (/home/twinkle/.cache/modelscope/hub/datasets/damo/style_custom_dataset/master/data_files)
    2024-10-08 13:16:34,604 - modelscope - INFO - Generating dataset dataset_builder (/home/twinkle/.cache/modelscope/hub/datasets/damo/style_custom_dataset/master/data_files)
    2024-10-08 13:16:34,604 - modelscope - INFO - Reusing cached meta-data file: /home/twinkle/.cache/modelscope/hub/datasets/damo/style_custom_dataset/master/data_files/20773d60e56183e65c9ac537a8adcd83
    scepter [INFO] 2024-10-08 13:16:35,231 [File: registry.py Function: init at line 185] Built dataloader with len 9223372036854775807
    2024-10-08 13:16:35,599 - modelscope - INFO - dataset_type: 2
    2024-10-08 13:16:36,447 - modelscope - WARNING - Reusing dataset dataset_builder (/home/twinkle/.cache/modelscope/hub/datasets/damo/style_custom_dataset/master/data_files)
    2024-10-08 13:16:36,447 - modelscope - INFO - Generating dataset dataset_builder (/home/twinkle/.cache/modelscope/hub/datasets/damo/style_custom_dataset/master/data_files)
    2024-10-08 13:16:36,447 - modelscope - INFO - Reusing cached meta-data file: /home/twinkle/.cache/modelscope/hub/datasets/damo/style_custom_dataset/master/data_files/20773d60e56183e65c9ac537a8adcd83
    scepter [INFO] 2024-10-08 13:16:37,072 [File: registry.py Function: init at line 185] Built dataloader with len 3
    scepter [INFO] 2024-10-08 13:16:43,411 [File: ldm.py Function: construct_network at line 147] all parameters:865.91M
    scepter [INFO] 2024-10-08 13:16:43,412 [File: ae_module.py Function: construct_model at line 76] AE Module XFORMERS_IS_AVAILBLE : True
    2024-10-08 13:16:55,021 - modelscope - WARNING - Model revision not specified, use revision: v1.0.0
    scepter [INFO] 2024-10-08 13:16:55,894 [File: ldm.py Function: init_from_ckpt at line 218] Restored from ./cache/cache_data/601abc72ccca722467955bb9c36da4b3/AI-ModelScope/stable-diffusion-2-1/v2-1_768-ema-pruned.safetensors with 86 missing and 14 unexpected keys
    scepter [INFO] 2024-10-08 13:16:55,894 [File: ldm.py Function: init_from_ckpt at line 222] Missing Keys:
    ['model.control_blocks.0.pre_hint_blocks.0.weight', 'model.control_blocks.0.pre_hint_blocks.0.bias', 'model.control_blocks.0.pre_hint_blocks.2.weight', 'model.control_blocks.0.pre_hint_blocks.2.bias', 'model.control_blocks.0.pre_hint_blocks.4.weight', 'model.control_blocks.0.pre_hint_blocks.4.bias', 'model.control_blocks.0.pre_hint_blocks.6.weight', 'model.control_blocks.0.pre_hint_blocks.6.bias', 'model.control_blocks.0.pre_hint_blocks.8.weight', 'model.control_blocks.0.pre_hint_blocks.8.bias', 'model.control_blocks.0.pre_hint_blocks.10.weight', 'model.control_blocks.0.pre_hint_blocks.10.bias', 'model.control_blocks.0.pre_hint_blocks.12.weight', 'model.control_blocks.0.pre_hint_blocks.12.bias', 'model.control_blocks.0.dense_hint_blocks.0.1.weight', 'model.control_blocks.0.dense_hint_blocks.0.1.bias', 'model.control_blocks.0.dense_hint_blocks.1.1.weight', 'model.control_blocks.0.dense_hint_blocks.1.1.bias', 'model.control_blocks.0.dense_hint_blocks.2.1.weight', 'model.control_blocks.0.dense_hint_blocks.2.1.bias', 'model.control_blocks.0.dense_hint_blocks.3.1.weight', 'model.control_blocks.0.dense_hint_blocks.3.1.bias', 'model.control_blocks.0.dense_hint_blocks.4.1.weight', 'model.control_blocks.0.dense_hint_blocks.4.1.bias', 'model.control_blocks.0.dense_hint_blocks.5.1.weight', 'model.control_blocks.0.dense_hint_blocks.5.1.bias', 'model.control_blocks.0.dense_hint_blocks.6.1.weight', 'model.control_blocks.0.dense_hint_blocks.6.1.bias', 'model.control_blocks.0.dense_hint_blocks.7.1.weight', 'model.control_blocks.0.dense_hint_blocks.7.1.bias', 'model.control_blocks.0.dense_hint_blocks.8.1.weight', 'model.control_blocks.0.dense_hint_blocks.8.1.bias', 'model.control_blocks.0.dense_hint_blocks.9.1.weight', 'model.control_blocks.0.dense_hint_blocks.9.1.bias', 'model.control_blocks.0.dense_hint_blocks.10.1.weight', 'model.control_blocks.0.dense_hint_blocks.10.1.bias', 'model.control_blocks.0.dense_hint_blocks.11.1.weight', 'model.control_blocks.0.dense_hint_blocks.11.1.bias', 'model.control_blocks.0.lsc_tuner_blocks.0.tuner_op.ln1.weight', 'model.control_blocks.0.lsc_tuner_blocks.0.tuner_op.ln1.bias', 'model.control_blocks.0.lsc_tuner_blocks.0.tuner_op.ln2.weight', 'model.control_blocks.0.lsc_tuner_blocks.0.tuner_op.ln2.bias', 'model.control_blocks.0.lsc_tuner_blocks.1.tuner_op.ln1.weight', 'model.control_blocks.0.lsc_tuner_blocks.1.tuner_op.ln1.bias', 'model.control_blocks.0.lsc_tuner_blocks.1.tuner_op.ln2.weight', 'model.control_blocks.0.lsc_tuner_blocks.1.tuner_op.ln2.bias', 'model.control_blocks.0.lsc_tuner_blocks.2.tuner_op.ln1.weight', 'model.control_blocks.0.lsc_tuner_blocks.2.tuner_op.ln1.bias', 'model.control_blocks.0.lsc_tuner_blocks.2.tuner_op.ln2.weight', 'model.control_blocks.0.lsc_tuner_blocks.2.tuner_op.ln2.bias', 'model.control_blocks.0.lsc_tuner_blocks.3.tuner_op.ln1.weight', 'model.control_blocks.0.lsc_tuner_blocks.3.tuner_op.ln1.bias', 'model.control_blocks.0.lsc_tuner_blocks.3.tuner_op.ln2.weight', 'model.control_blocks.0.lsc_tuner_blocks.3.tuner_op.ln2.bias', 'model.control_blocks.0.lsc_tuner_blocks.4.tuner_op.ln1.weight', 'model.control_blocks.0.lsc_tuner_blocks.4.tuner_op.ln1.bias', 'model.control_blocks.0.lsc_tuner_blocks.4.tuner_op.ln2.weight', 'model.control_blocks.0.lsc_tuner_blocks.4.tuner_op.ln2.bias', 'model.control_blocks.0.lsc_tuner_blocks.5.tuner_op.ln1.weight', 'model.control_blocks.0.lsc_tuner_blocks.5.tuner_op.ln1.bias', 'model.control_blocks.0.lsc_tuner_blocks.5.tuner_op.ln2.weight', 'model.control_blocks.0.lsc_tuner_blocks.5.tuner_op.ln2.bias', 'model.control_blocks.0.lsc_tuner_blocks.6.tuner_op.ln1.weight', 'model.control_blocks.0.lsc_tuner_blocks.6.tuner_op.ln1.bias', 'model.control_blocks.0.lsc_tuner_blocks.6.tuner_op.ln2.weight', 'model.control_blocks.0.lsc_tuner_blocks.6.tuner_op.ln2.bias', 'model.control_blocks.0.lsc_tuner_blocks.7.tuner_op.ln1.weight', 'model.control_blocks.0.lsc_tuner_blocks.7.tuner_op.ln1.bias', 'model.control_blocks.0.lsc_tuner_blocks.7.tuner_op.ln2.weight', 'model.control_blocks.0.lsc_tuner_blocks.7.tuner_op.ln2.bias', 'model.control_blocks.0.lsc_tuner_blocks.8.tuner_op.ln1.weight', 'model.control_blocks.0.lsc_tuner_blocks.8.tuner_op.ln1.bias', 'model.control_blocks.0.lsc_tuner_blocks.8.tuner_op.ln2.weight', 'model.control_blocks.0.lsc_tuner_blocks.8.tuner_op.ln2.bias', 'model.control_blocks.0.lsc_tuner_blocks.9.tuner_op.ln1.weight', 'model.control_blocks.0.lsc_tuner_blocks.9.tuner_op.ln1.bias', 'model.control_blocks.0.lsc_tuner_blocks.9.tuner_op.ln2.weight', 'model.control_blocks.0.lsc_tuner_blocks.9.tuner_op.ln2.bias', 'model.control_blocks.0.lsc_tuner_blocks.10.tuner_op.ln1.weight', 'model.control_blocks.0.lsc_tuner_blocks.10.tuner_op.ln1.bias', 'model.control_blocks.0.lsc_tuner_blocks.10.tuner_op.ln2.weight', 'model.control_blocks.0.lsc_tuner_blocks.10.tuner_op.ln2.bias', 'model.control_blocks.0.lsc_tuner_blocks.11.tuner_op.ln1.weight', 'model.control_blocks.0.lsc_tuner_blocks.11.tuner_op.ln1.bias', 'model.control_blocks.0.lsc_tuner_blocks.11.tuner_op.ln2.weight', 'model.control_blocks.0.lsc_tuner_blocks.11.tuner_op.ln2.bias']
    scepter [INFO] 2024-10-08 13:16:55,894 [File: ldm.py Function: init_from_ckpt at line 224]
    Unexpected Keys:
    ['alphas_cumprod', 'alphas_cumprod_prev', 'betas', 'log_one_minus_alphas_cumprod', 'model_ema.decay', 'model_ema.num_updates', 'posterior_log_variance_clipped', 'posterior_mean_coef1', 'posterior_mean_coef2', 'posterior_variance', 'sqrt_alphas_cumprod', 'sqrt_one_minus_alphas_cumprod', 'sqrt_recip_alphas_cumprod', 'sqrt_recipm1_alphas_cumprod']
    scepter [INFO] 2024-10-08 13:16:56,475 [File: diffusion_solver.py Function: print_model_params_status at line 745] Load trainable params 99111760 / 1402708988 = 7.07%, train part: {'control_blocks.0': 99111760}.
    scepter [INFO] 2024-10-08 13:16:56,475 [File: diffusion_solver.py Function: print_model_params_status at line 749] Load forzen params 1303597228 / 1402708988 = 92.93%, forzen part: {'time_embed': 2050560, 'input_blocks': 251918080, 'middle_block': 97693440, 'output_blocks': 514236480, 'out': 12164, 'encoder': 34163592, 'decoder': 49490179, 'conv1': 72, 'conv2': 20, 'model': 354032641}.
    0it [00:02, ?it/s]
    Traceback (most recent call last):
    File "scepter/tools/run_train.py", line 62, in
    run()
    File "scepter/tools/run_train.py", line 58, in run
    we.init_env(cfg, logger=None, fn=run_task)
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/scepter/modules/utils/distribute.py", line 363, in init_env
    fn(config)
    File "scepter/tools/run_train.py", line 26, in run_task
    solver.solve()
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/scepter/modules/solver/diffusion_solver.py", line 328, in solve
    self.run_train()
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/scepter/modules/solver/diffusion_solver.py", line 341, in run_train
    self.run_eval()
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/scepter/modules/solver/diffusion_solver.py", line 389, in run_eval
    results = self.run_step_eval(transfer_data_to_cuda(batch_data),
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/scepter/modules/solver/base_solver.py", line 559, in run_step_eval
    results = self.model(**batch_data)
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/scepter/modules/model/network/ldm/ldm.py", line 288, in forward
    return self.forward_test(**kwargs)
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/scepter/modules/model/network/ldm/ldm_sce.py", line 130, in forward_test
    return super().forward_test(**kwargs)
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/scepter/modules/model/network/ldm/ldm.py", line 318, in forward_test
    context = self.encode_condition(self.tokenizer(prompt).to(
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/scepter/modules/model/network/ldm/ldm.py", line 228, in encode_condition
    return getattr(self.cond_stage_model,
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/scepter/modules/model/embedder/embedder.py", line 303, in encode_text
    z = self.encode_with_transformer(tokens.to(we.device_id))
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/scepter/modules/model/embedder/embedder.py", line 278, in encode_with_transformer
    x = self.text_transformer_forward(x,
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/scepter/modules/model/embedder/embedder.py", line 294, in text_transformer_forward
    x = r(x, attn_mask=attn_mask)
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/open_clip/transformer.py", line 263, in forward
    x = q_x + self.ls_1(self.attention(q_x=self.ln_1(q_x), k_x=k_x, v_x=v_x, attn_mask=attn_mask))
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/open_clip/transformer.py", line 250, in attention
    return self.attn(
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 1205, in forward
    attn_output, attn_output_weights = F.multi_head_attention_forward(
    File "/data/twinkle/anaconda3/envs/scepter/lib/python3.8/site-packages/torch/nn/functional.py", line 5251, in multi_head_attention_forward
    raise RuntimeError(f"The shape of the 2D attn_mask is {attn_mask.shape}, but should be {correct_2d_size}.")
    RuntimeError: The shape of the 2D attn_mask is torch.Size([77, 77]), but should be (10, 10).

  3. jiangzeyinzi commented on Oct 8, 2024

    @jiangzeyinzi
    jiangzeyinzi
    on Oct 8, 2024
    Collaborator

    This issue has some confusing aspects.
    SCEdit design the sc-tuner for fine-tuning txt2img task, as mentioned in the first comment, and it is integrated into the studio to allow users to fine-tune text-to-image generation. The answer above can address this issue.
    Your second comment refers to another module of the method, the csc-tuner, which is used for conditional image generation. I could not replicate the issues you encountered during the training process of canny-to-image in my own environment. Additionally, I searched for this problem, and the answers I found may be related to the version of open-clip-torch you are using. I am currently using open-clip-torch==2.20.0, or you may want to try other versions compatible with your environment.

  4. twinkleyang1 commented on Oct 8, 2024

    @twinkleyang1
    twinkleyang1
    on Oct 8, 2024
    Author

    Thanks for your answer, the solution worked according to what you said

  5. twinkleyang1
    closed this as completedon Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions