Repository navigation
FlaxDDPMScheduler produces black images #2179
Description
Describe the bug
FlaxDDPMScheduler produces black images, even after fixing #2176.
I checked in bfloat16, bfloat32 and at revision f106ab4. It failed in all cases.
Reproduction
import jax
import numpy as np
from flax.jax_utils import replicate
from flax.training.common_utils import shard
from diffusers import FlaxStableDiffusionPipeline, FlaxDDPMScheduler
model_id = "runwayml/stable-diffusion-v1-5"
scheduler, scheduler_state = FlaxDDPMScheduler.from_pretrained(
model_id,
subfolder="scheduler"
)
pipeline, params = FlaxStableDiffusionPipeline.from_pretrained(
model_id,
scheduler=scheduler,
revision="bf16",
dtype=jax.numpy.bfloat16,
)
params["scheduler"] = scheduler_state
prompt = "a photo of an astronaut riding a horse on mars"
prng_seed = jax.random.PRNGKey(0)
num_inference_steps = 50
num_samples = jax.device_count()
prompt = num_samples * [prompt]
prompt_ids = pipeline.prepare_inputs(prompt)
params = replicate(params)
prng_seed = jax.random.split(prng_seed, jax.device_count())
prompt_ids = shard(prompt_ids)
images = pipeline(prompt_ids, params, prng_seed, num_inference_steps, jit=True).images
images = pipeline.numpy_to_pil(np.asarray(images.reshape((num_samples,) + images.shape[-3:])))
for i, image in enumerate(images):
image.save(f"flax_{i}.png")Logs
No response
System Info
Diffusers 0.13.0.dev0. Also tested on previous revisions as indicated above.
Activity
pcuenca commentedon Jan 31, 2023
Initially reported as part of #2155.
patrickvonplaten commentedon Feb 3, 2023
Interesting, are we sure that it's not a bug that exists from the very beginning? If DDIM and DPM works maybe this one is also not super important to fix?
github-actions commentedon Mar 3, 2023
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
kxhit commentedon May 3, 2023
@pcuenca @patrickvonplaten
I found a similar bug when I set bf16. I traced the bug and found the scheduler.step() will return .prev_sample full of nan.
The bug is here, when casting
self.alphas_cumprodto bf16, thealpha_prod_twill be 1 whent=0. Thus, thepred_original_sample_coeff = (alpha_prod_t_prev ** (0.5) * current_beta_t) / beta_prod_tresult will be nan.Need to force the value to be float32 or avoid divided by 0.
The value of alphas_cumprod in float32 vs bfloat16:
self.alphas_cumprod tensor([0.9991, 0.9983, 0.9974, 0.9966, 0.9957, 0.9948, 0.9940, 0.9931, 0.9922, 0.9913, 0.9904, 0.9895, 0.9886, 0.9877, 0.9868, 0.9859, 0.9850, 0.9841, 0.9832, 0.9822, 0.9813, 0.9804, 0.9794, 0.9785, 0.9776, 0.9766, 0.9757, 0.9747, 0.9737, 0.9728, 0.9718, 0.9708, 0.9698, 0.9689, 0.9679, 0.9669, 0.9659, 0.9649, 0.9639, 0.9629, 0.9619, 0.9609, 0.9599, 0.9588, 0.9578, 0.9568, 0.9557, 0.9547, 0.9537, 0.9526, 0.9516, 0.9505, 0.9495, 0.9484, 0.9473, 0.9463, 0.9452, 0.9441, 0.9430, 0.9420, 0.9409, 0.9398, 0.9387, 0.9376, 0.9365, 0.9354, 0.9343, 0.9332, 0.9320, 0.9309, 0.9298, 0.9287, 0.9275, 0.9264, 0.9252, 0.9241, 0.9229, 0.9218, 0.9206, 0.9195, 0.9183, 0.9171, 0.9160, 0.9148, 0.9136, 0.9124, 0.9112, 0.9100, 0.9089, 0.9077, 0.9065, 0.9052, 0.9040, 0.9028, 0.9016, 0.9004, 0.8992, 0.8979, 0.8967, 0.8955, 0.8942, 0.8930, 0.8917, 0.8905, 0.8892, 0.8880, 0.8867, 0.8854, 0.8842, 0.8829, 0.8816, 0.8804, 0.8791, 0.8778, 0.8765, 0.8752, 0.8739, 0.8726, 0.8713, 0.8700, 0.8687, 0.8674, 0.8661, 0.8647, 0.8634, 0.8621, 0.8607, 0.8594, 0.8581, 0.8567, 0.8554, 0.8540, 0.8527, 0.8513, 0.8500, 0.8486, 0.8473, 0.8459, 0.8445, 0.8431, 0.8418, 0.8404, 0.8390, 0.8376, 0.8362, 0.8348, 0.8334, 0.8320, 0.8306, 0.8292, 0.8278, 0.8264, 0.8250, 0.8236, 0.8221, 0.8207, 0.8193, 0.8179, 0.8164, 0.8150, 0.8136, 0.8121, 0.8107, 0.8092, 0.8078, 0.8063, 0.8049, 0.8034, 0.8019, 0.8005, 0.7990, 0.7975, 0.7960, 0.7946, 0.7931, 0.7916, 0.7901, 0.7886, 0.7871, 0.7856, 0.7842, 0.7827, 0.7812, 0.7796, 0.7781, 0.7766, 0.7751, 0.7736, 0.7721, 0.7706, 0.7690, 0.7675, 0.7660, 0.7645, 0.7629, 0.7614, 0.7599, 0.7583, 0.7568, 0.7552, 0.7537, 0.7521, 0.7506, 0.7490, 0.7475, 0.7459, 0.7444, 0.7428, 0.7412, 0.7397, 0.7381, 0.7365, 0.7350, 0.7334, 0.7318, 0.7302, 0.7286, 0.7271, 0.7255, 0.7239, 0.7223, 0.7207, 0.7191, 0.7175, 0.7159, 0.7143, 0.7127, 0.7111, 0.7095, 0.7079, 0.7063, 0.7047, 0.7031, 0.7015, 0.6999, 0.6982, 0.6966, 0.6950, 0.6934, 0.6918, 0.6901, 0.6885, 0.6869, 0.6852, 0.6836, 0.6820, 0.6803, 0.6787, 0.6771, 0.6754, 0.6738, 0.6722, 0.6705, 0.6689, 0.6672, 0.6656, 0.6639, 0.6623, 0.6606, 0.6590, 0.6573, 0.6557, 0.6540, 0.6524, 0.6507, 0.6490, 0.6474, 0.6457, 0.6441, 0.6424, 0.6407, 0.6391, 0.6374, 0.6357, 0.6341, 0.6324, 0.6307, 0.6291, 0.6274, 0.6257, 0.6241, 0.6224, 0.6207, 0.6190, 0.6174, 0.6157, 0.6140, 0.6123, 0.6107, 0.6090, 0.6073, 0.6056, 0.6039, 0.6023, 0.6006, 0.5989, 0.5972, 0.5955, 0.5939, 0.5922, 0.5905, 0.5888, 0.5871, 0.5855, 0.5838, 0.5821, 0.5804, 0.5787, 0.5770, 0.5754, 0.5737, 0.5720, 0.5703, 0.5686, 0.5669, 0.5652, 0.5636, 0.5619, 0.5602, 0.5585, 0.5568, 0.5551, 0.5535, 0.5518, 0.5501, 0.5484, 0.5467, 0.5450, 0.5434, 0.5417, 0.5400, 0.5383, 0.5366, 0.5350, 0.5333, 0.5316, 0.5299, 0.5282, 0.5266, 0.5249, 0.5232, 0.5215, 0.5199, 0.5182, 0.5165, 0.5148, 0.5132, 0.5115, 0.5098, 0.5082, 0.5065, 0.5048, 0.5032, 0.5015, 0.4998, 0.4982, 0.4965, 0.4948, 0.4932, 0.4915, 0.4898, 0.4882, 0.4865, 0.4849, 0.4832, 0.4816, 0.4799, 0.4782, 0.4766, 0.4749, 0.4733, 0.4716, 0.4700, 0.4684, 0.4667, 0.4651, 0.4634, 0.4618, 0.4601, 0.4585, 0.4569, 0.4552, 0.4536, 0.4520, 0.4503, 0.4487, 0.4471, 0.4455, 0.4438, 0.4422, 0.4406, 0.4390, 0.4374, 0.4357, 0.4341, 0.4325, 0.4309, 0.4293, 0.4277, 0.4261, 0.4245, 0.4229, 0.4213, 0.4197, 0.4181, 0.4165, 0.4149, 0.4133, 0.4117, 0.4101, 0.4086, 0.4070, 0.4054, 0.4038, 0.4022, 0.4007, 0.3991, 0.3975, 0.3960, 0.3944, 0.3928, 0.3913, 0.3897, 0.3882, 0.3866, 0.3850, 0.3835, 0.3819, 0.3804, 0.3789, 0.3773, 0.3758, 0.3742, 0.3727, 0.3712, 0.3697, 0.3681, 0.3666, 0.3651, 0.3636, 0.3621, 0.3605, 0.3590, 0.3575, 0.3560, 0.3545, 0.3530, 0.3515, 0.3500, 0.3485, 0.3470, 0.3456, 0.3441, 0.3426, 0.3411, 0.3396, 0.3382, 0.3367, 0.3352, 0.3338, 0.3323, 0.3308, 0.3294, 0.3279, 0.3265, 0.3250, 0.3236, 0.3222, 0.3207, 0.3193, 0.3178, 0.3164, 0.3150, 0.3136, 0.3122, 0.3107, 0.3093, 0.3079, 0.3065, 0.3051, 0.3037, 0.3023, 0.3009, 0.2995, 0.2981, 0.2967, 0.2954, 0.2940, 0.2926, 0.2912, 0.2899, 0.2885, 0.2871, 0.2858, 0.2844, 0.2831, 0.2817, 0.2804, 0.2790, 0.2777, 0.2763, 0.2750, 0.2737, 0.2723, 0.2710, 0.2697, 0.2684, 0.2671, 0.2658, 0.2645, 0.2631, 0.2618, 0.2606, 0.2593, 0.2580, 0.2567, 0.2554, 0.2541, 0.2528, 0.2516, 0.2503, 0.2490, 0.2478, 0.2465, 0.2453, 0.2440, 0.2428, 0.2415, 0.2403, 0.2391, 0.2378, 0.2366, 0.2354, 0.2341, 0.2329, 0.2317, 0.2305, 0.2293, 0.2281, 0.2269, 0.2257, 0.2245, 0.2233, 0.2221, 0.2209, 0.2198, 0.2186, 0.2174, 0.2163, 0.2151, 0.2139, 0.2128, 0.2116, 0.2105, 0.2093, 0.2082, 0.2071, 0.2059, 0.2048, 0.2037, 0.2026, 0.2014, 0.2003, 0.1992, 0.1981, 0.1970, 0.1959, 0.1948, 0.1937, 0.1926, 0.1915, 0.1905, 0.1894, 0.1883, 0.1872, 0.1862, 0.1851, 0.1841, 0.1830, 0.1820, 0.1809, 0.1799, 0.1788, 0.1778, 0.1768, 0.1757, 0.1747, 0.1737, 0.1727, 0.1717, 0.1707, 0.1696, 0.1686, 0.1677, 0.1667, 0.1657, 0.1647, 0.1637, 0.1627, 0.1618, 0.1608, 0.1598, 0.1589, 0.1579, 0.1569, 0.1560, 0.1550, 0.1541, 0.1532, 0.1522, 0.1513, 0.1504, 0.1494, 0.1485, 0.1476, 0.1467, 0.1458, 0.1449, 0.1440, 0.1431, 0.1422, 0.1413, 0.1404, 0.1395, 0.1386, 0.1378, 0.1369, 0.1360, 0.1352, 0.1343, 0.1334, 0.1326, 0.1317, 0.1309, 0.1301, 0.1292, 0.1284, 0.1276, 0.1267, 0.1259, 0.1251, 0.1243, 0.1235, 0.1227, 0.1219, 0.1211, 0.1203, 0.1195, 0.1187, 0.1179, 0.1171, 0.1163, 0.1155, 0.1148, 0.1140, 0.1132, 0.1125, 0.1117, 0.1110, 0.1102, 0.1095, 0.1087, 0.1080, 0.1073, 0.1065, 0.1058, 0.1051, 0.1044, 0.1036, 0.1029, 0.1022, 0.1015, 0.1008, 0.1001, 0.0994, 0.0987, 0.0980, 0.0973, 0.0967, 0.0960, 0.0953, 0.0946, 0.0940, 0.0933, 0.0926, 0.0920, 0.0913, 0.0907, 0.0900, 0.0894, 0.0887, 0.0881, 0.0875, 0.0868, 0.0862, 0.0856, 0.0850, 0.0844, 0.0837, 0.0831, 0.0825, 0.0819, 0.0813, 0.0807, 0.0801, 0.0795, 0.0789, 0.0784, 0.0778, 0.0772, 0.0766, 0.0761, 0.0755, 0.0749, 0.0744, 0.0738, 0.0732, 0.0727, 0.0721, 0.0716, 0.0711, 0.0705, 0.0700, 0.0694, 0.0689, 0.0684, 0.0679, 0.0673, 0.0668, 0.0663, 0.0658, 0.0653, 0.0648, 0.0643, 0.0638, 0.0633, 0.0628, 0.0623, 0.0618, 0.0613, 0.0608, 0.0604, 0.0599, 0.0594, 0.0589, 0.0585, 0.0580, 0.0575, 0.0571, 0.0566, 0.0562, 0.0557, 0.0553, 0.0548, 0.0544, 0.0539, 0.0535, 0.0531, 0.0526, 0.0522, 0.0518, 0.0514, 0.0509, 0.0505, 0.0501, 0.0497, 0.0493, 0.0489, 0.0485, 0.0481, 0.0477, 0.0473, 0.0469, 0.0465, 0.0461, 0.0457, 0.0453, 0.0450, 0.0446, 0.0442, 0.0438, 0.0435, 0.0431, 0.0427, 0.0424, 0.0420, 0.0416, 0.0413, 0.0409, 0.0406, 0.0402, 0.0399, 0.0395, 0.0392, 0.0389, 0.0385, 0.0382, 0.0379, 0.0375, 0.0372, 0.0369, 0.0365, 0.0362, 0.0359, 0.0356, 0.0353, 0.0350, 0.0347, 0.0343, 0.0340, 0.0337, 0.0334, 0.0331, 0.0328, 0.0325, 0.0323, 0.0320, 0.0317, 0.0314, 0.0311, 0.0308, 0.0305, 0.0303, 0.0300, 0.0297, 0.0295, 0.0292, 0.0289, 0.0286, 0.0284, 0.0281, 0.0279, 0.0276, 0.0274, 0.0271, 0.0268, 0.0266, 0.0264, 0.0261, 0.0259, 0.0256, 0.0254, 0.0251, 0.0249, 0.0247, 0.0244, 0.0242, 0.0240, 0.0237, 0.0235, 0.0233, 0.0231, 0.0229, 0.0226, 0.0224, 0.0222, 0.0220, 0.0218, 0.0216, 0.0214, 0.0212, 0.0210, 0.0207, 0.0205, 0.0203, 0.0201, 0.0200, 0.0198, 0.0196, 0.0194, 0.0192, 0.0190, 0.0188, 0.0186, 0.0184, 0.0182, 0.0181, 0.0179, 0.0177, 0.0175, 0.0174, 0.0172, 0.0170, 0.0168, 0.0167, 0.0165, 0.0163, 0.0162, 0.0160, 0.0158, 0.0157, 0.0155, 0.0154, 0.0152, 0.0151, 0.0149, 0.0147, 0.0146, 0.0144, 0.0143, 0.0142, 0.0140, 0.0139, 0.0137, 0.0136, 0.0134, 0.0133, 0.0132, 0.0130, 0.0129, 0.0127, 0.0126, 0.0125, 0.0123, 0.0122, 0.0121, 0.0120, 0.0118, 0.0117, 0.0116, 0.0115, 0.0113, 0.0112, 0.0111, 0.0110, 0.0109, 0.0107, 0.0106, 0.0105, 0.0104, 0.0103, 0.0102, 0.0101, 0.0100, 0.0098, 0.0097, 0.0096, 0.0095, 0.0094, 0.0093, 0.0092, 0.0091, 0.0090, 0.0089, 0.0088, 0.0087, 0.0086, 0.0085, 0.0084, 0.0083, 0.0082, 0.0082, 0.0081, 0.0080, 0.0079, 0.0078, 0.0077, 0.0076, 0.0075, 0.0074, 0.0074, 0.0073, 0.0072, 0.0071, 0.0070, 0.0070, 0.0069, 0.0068, 0.0067, 0.0066, 0.0066, 0.0065, 0.0064, 0.0063, 0.0063, 0.0062, 0.0061, 0.0061, 0.0060, 0.0059, 0.0058, 0.0058, 0.0057, 0.0056, 0.0056, 0.0055, 0.0054, 0.0054, 0.0053, 0.0053, 0.0052, 0.0051, 0.0051, 0.0050, 0.0049, 0.0049, 0.0048, 0.0048, 0.0047, 0.0047], device='cuda:0') (Pdb) self.alphas_cumprod.dtype torch.float32 (Pdb) tmp = self.alphas_cumprod.to(torch.bfloat16) (Pdb) tmp tensor([1.0000, 1.0000, 0.9961, 0.9961, 0.9961, 0.9961, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922, 0.9883, 0.9883, 0.9883, 0.9883, 0.9844, 0.9844, 0.9844, 0.9844, 0.9805, 0.9805, 0.9805, 0.9805, 0.9766, 0.9766, 0.9766, 0.9766, 0.9766, 0.9727, 0.9727, 0.9727, 0.9727, 0.9688, 0.9688, 0.9688, 0.9688, 0.9648, 0.9648, 0.9648, 0.9648, 0.9609, 0.9609, 0.9609, 0.9570, 0.9570, 0.9570, 0.9570, 0.9531, 0.9531, 0.9531, 0.9531, 0.9492, 0.9492, 0.9492, 0.9492, 0.9453, 0.9453, 0.9453, 0.9414, 0.9414, 0.9414, 0.9414, 0.9375, 0.9375, 0.9375, 0.9336, 0.9336, 0.9336, 0.9336, 0.9297, 0.9297, 0.9297, 0.9258, 0.9258, 0.9258, 0.9258, 0.9219, 0.9219, 0.9219, 0.9180, 0.9180, 0.9180, 0.9141, 0.9141, 0.9141, 0.9141, 0.9102, 0.9102, 0.9102, 0.9062, 0.9062, 0.9062, 0.9023, 0.9023, 0.9023, 0.8984, 0.8984, 0.8984, 0.8984, 0.8945, 0.8945, 0.8945, 0.8906, 0.8906, 0.8906, 0.8867, 0.8867, 0.8867, 0.8828, 0.8828, 0.8828, 0.8789, 0.8789, 0.8789, 0.8750, 0.8750, 0.8750, 0.8711, 0.8711, 0.8711, 0.8672, 0.8672, 0.8672, 0.8633, 0.8633, 0.8633, 0.8594, 0.8594, 0.8594, 0.8555, 0.8555, 0.8555, 0.8516, 0.8516, 0.8516, 0.8477, 0.8477, 0.8477, 0.8438, 0.8438, 0.8398, 0.8398, 0.8398, 0.8359, 0.8359, 0.8359, 0.8320, 0.8320, 0.8320, 0.8281, 0.8281, 0.8281, 0.8242, 0.8242, 0.8203, 0.8203, 0.8203, 0.8164, 0.8164, 0.8164, 0.8125, 0.8125, 0.8125, 0.8086, 0.8086, 0.8047, 0.8047, 0.8047, 0.8008, 0.8008, 0.8008, 0.7969, 0.7969, 0.7930, 0.7930, 0.7930, 0.7891, 0.7891, 0.7891, 0.7852, 0.7852, 0.7812, 0.7812, 0.7812, 0.7773, 0.7773, 0.7734, 0.7734, 0.7734, 0.7695, 0.7695, 0.7656, 0.7656, 0.7656, 0.7617, 0.7617, 0.7617, 0.7578, 0.7578, 0.7539, 0.7539, 0.7539, 0.7500, 0.7500, 0.7461, 0.7461, 0.7461, 0.7422, 0.7422, 0.7383, 0.7383, 0.7383, 0.7344, 0.7344, 0.7305, 0.7305, 0.7305, 0.7266, 0.7266, 0.7227, 0.7227, 0.7227, 0.7188, 0.7188, 0.7148, 0.7148, 0.7109, 0.7109, 0.7109, 0.7070, 0.7070, 0.7031, 0.7031, 0.7031, 0.6992, 0.6992, 0.6953, 0.6953, 0.6953, 0.6914, 0.6914, 0.6875, 0.6875, 0.6836, 0.6836, 0.6836, 0.6797, 0.6797, 0.6758, 0.6758, 0.6719, 0.6719, 0.6719, 0.6680, 0.6680, 0.6641, 0.6641, 0.6641, 0.6602, 0.6602, 0.6562, 0.6562, 0.6523, 0.6523, 0.6523, 0.6484, 0.6484, 0.6445, 0.6445, 0.6406, 0.6406, 0.6406, 0.6367, 0.6367, 0.6328, 0.6328, 0.6289, 0.6289, 0.6289, 0.6250, 0.6250, 0.6211, 0.6211, 0.6172, 0.6172, 0.6172, 0.6133, 0.6133, 0.6094, 0.6094, 0.6055, 0.6055, 0.6055, 0.6016, 0.6016, 0.5977, 0.5977, 0.5938, 0.5938, 0.5938, 0.5898, 0.5898, 0.5859, 0.5859, 0.5820, 0.5820, 0.5820, 0.5781, 0.5781, 0.5742, 0.5742, 0.5703, 0.5703, 0.5703, 0.5664, 0.5664, 0.5625, 0.5625, 0.5586, 0.5586, 0.5586, 0.5547, 0.5547, 0.5508, 0.5508, 0.5469, 0.5469, 0.5469, 0.5430, 0.5430, 0.5391, 0.5391, 0.5352, 0.5352, 0.5352, 0.5312, 0.5312, 0.5273, 0.5273, 0.5234, 0.5234, 0.5234, 0.5195, 0.5195, 0.5156, 0.5156, 0.5117, 0.5117, 0.5117, 0.5078, 0.5078, 0.5039, 0.5039, 0.5000, 0.5000, 0.4980, 0.4961, 0.4941, 0.4922, 0.4922, 0.4902, 0.4883, 0.4863, 0.4844, 0.4824, 0.4824, 0.4805, 0.4785, 0.4766, 0.4746, 0.4727, 0.4707, 0.4707, 0.4688, 0.4668, 0.4648, 0.4629, 0.4609, 0.4609, 0.4590, 0.4570, 0.4551, 0.4531, 0.4512, 0.4512, 0.4492, 0.4473, 0.4453, 0.4434, 0.4414, 0.4414, 0.4395, 0.4375, 0.4355, 0.4336, 0.4316, 0.4316, 0.4297, 0.4277, 0.4258, 0.4238, 0.4238, 0.4219, 0.4199, 0.4180, 0.4160, 0.4141, 0.4141, 0.4121, 0.4102, 0.4082, 0.4062, 0.4062, 0.4043, 0.4023, 0.4004, 0.3984, 0.3984, 0.3965, 0.3945, 0.3926, 0.3906, 0.3906, 0.3887, 0.3867, 0.3848, 0.3828, 0.3828, 0.3809, 0.3789, 0.3770, 0.3750, 0.3750, 0.3730, 0.3711, 0.3691, 0.3672, 0.3672, 0.3652, 0.3633, 0.3613, 0.3613, 0.3594, 0.3574, 0.3555, 0.3555, 0.3535, 0.3516, 0.3496, 0.3477, 0.3477, 0.3457, 0.3438, 0.3418, 0.3418, 0.3398, 0.3379, 0.3359, 0.3359, 0.3340, 0.3320, 0.3301, 0.3301, 0.3281, 0.3262, 0.3242, 0.3242, 0.3223, 0.3203, 0.3184, 0.3184, 0.3164, 0.3145, 0.3145, 0.3125, 0.3105, 0.3086, 0.3086, 0.3066, 0.3047, 0.3027, 0.3027, 0.3008, 0.2988, 0.2988, 0.2969, 0.2949, 0.2949, 0.2930, 0.2910, 0.2891, 0.2891, 0.2871, 0.2852, 0.2852, 0.2832, 0.2812, 0.2812, 0.2793, 0.2773, 0.2754, 0.2754, 0.2734, 0.2715, 0.2715, 0.2695, 0.2676, 0.2676, 0.2656, 0.2637, 0.2637, 0.2617, 0.2598, 0.2598, 0.2578, 0.2559, 0.2559, 0.2539, 0.2520, 0.2520, 0.2500, 0.2490, 0.2480, 0.2461, 0.2451, 0.2441, 0.2432, 0.2412, 0.2402, 0.2393, 0.2383, 0.2363, 0.2354, 0.2344, 0.2334, 0.2314, 0.2305, 0.2295, 0.2285, 0.2266, 0.2256, 0.2246, 0.2236, 0.2217, 0.2207, 0.2197, 0.2188, 0.2178, 0.2158, 0.2148, 0.2139, 0.2129, 0.2119, 0.2109, 0.2090, 0.2080, 0.2070, 0.2061, 0.2051, 0.2041, 0.2021, 0.2012, 0.2002, 0.1992, 0.1982, 0.1973, 0.1963, 0.1943, 0.1934, 0.1924, 0.1914, 0.1904, 0.1895, 0.1885, 0.1875, 0.1865, 0.1855, 0.1836, 0.1826, 0.1816, 0.1807, 0.1797, 0.1787, 0.1777, 0.1768, 0.1758, 0.1748, 0.1738, 0.1729, 0.1719, 0.1709, 0.1699, 0.1689, 0.1680, 0.1670, 0.1660, 0.1650, 0.1641, 0.1631, 0.1621, 0.1611, 0.1602, 0.1592, 0.1582, 0.1572, 0.1562, 0.1553, 0.1543, 0.1533, 0.1523, 0.1514, 0.1504, 0.1494, 0.1484, 0.1475, 0.1465, 0.1455, 0.1445, 0.1436, 0.1436, 0.1426, 0.1416, 0.1406, 0.1396, 0.1387, 0.1377, 0.1367, 0.1357, 0.1348, 0.1348, 0.1338, 0.1328, 0.1318, 0.1309, 0.1299, 0.1289, 0.1279, 0.1279, 0.1270, 0.1260, 0.1250, 0.1245, 0.1235, 0.1226, 0.1221, 0.1211, 0.1201, 0.1196, 0.1187, 0.1177, 0.1172, 0.1162, 0.1157, 0.1147, 0.1138, 0.1133, 0.1123, 0.1118, 0.1108, 0.1104, 0.1094, 0.1089, 0.1079, 0.1074, 0.1064, 0.1060, 0.1050, 0.1045, 0.1035, 0.1030, 0.1021, 0.1016, 0.1006, 0.1001, 0.0996, 0.0986, 0.0981, 0.0972, 0.0967, 0.0962, 0.0952, 0.0947, 0.0938, 0.0933, 0.0928, 0.0918, 0.0913, 0.0908, 0.0898, 0.0894, 0.0889, 0.0879, 0.0874, 0.0869, 0.0864, 0.0854, 0.0850, 0.0845, 0.0835, 0.0830, 0.0825, 0.0820, 0.0815, 0.0806, 0.0801, 0.0796, 0.0791, 0.0781, 0.0776, 0.0771, 0.0767, 0.0762, 0.0757, 0.0747, 0.0742, 0.0737, 0.0732, 0.0728, 0.0723, 0.0718, 0.0713, 0.0703, 0.0698, 0.0693, 0.0688, 0.0684, 0.0679, 0.0674, 0.0669, 0.0664, 0.0659, 0.0654, 0.0649, 0.0645, 0.0640, 0.0635, 0.0630, 0.0623, 0.0618, 0.0613, 0.0608, 0.0603, 0.0598, 0.0593, 0.0588, 0.0583, 0.0581, 0.0576, 0.0571, 0.0566, 0.0562, 0.0557, 0.0552, 0.0549, 0.0544, 0.0540, 0.0535, 0.0530, 0.0527, 0.0522, 0.0518, 0.0513, 0.0510, 0.0505, 0.0500, 0.0498, 0.0493, 0.0488, 0.0486, 0.0481, 0.0476, 0.0474, 0.0469, 0.0464, 0.0461, 0.0457, 0.0454, 0.0449, 0.0447, 0.0442, 0.0439, 0.0435, 0.0430, 0.0427, 0.0422, 0.0420, 0.0417, 0.0413, 0.0410, 0.0405, 0.0403, 0.0398, 0.0396, 0.0393, 0.0388, 0.0386, 0.0381, 0.0378, 0.0376, 0.0371, 0.0369, 0.0366, 0.0361, 0.0359, 0.0356, 0.0352, 0.0349, 0.0347, 0.0344, 0.0339, 0.0337, 0.0334, 0.0332, 0.0330, 0.0325, 0.0322, 0.0320, 0.0317, 0.0315, 0.0311, 0.0309, 0.0305, 0.0303, 0.0300, 0.0297, 0.0294, 0.0292, 0.0289, 0.0287, 0.0284, 0.0281, 0.0278, 0.0276, 0.0273, 0.0271, 0.0269, 0.0266, 0.0264, 0.0261, 0.0259, 0.0256, 0.0254, 0.0251, 0.0249, 0.0247, 0.0244, 0.0242, 0.0239, 0.0238, 0.0236, 0.0233, 0.0231, 0.0228, 0.0226, 0.0225, 0.0222, 0.0220, 0.0217, 0.0216, 0.0214, 0.0211, 0.0210, 0.0208, 0.0205, 0.0204, 0.0201, 0.0199, 0.0198, 0.0195, 0.0194, 0.0192, 0.0190, 0.0188, 0.0187, 0.0184, 0.0183, 0.0181, 0.0179, 0.0177, 0.0176, 0.0173, 0.0172, 0.0170, 0.0168, 0.0167, 0.0165, 0.0164, 0.0161, 0.0160, 0.0159, 0.0156, 0.0155, 0.0154, 0.0152, 0.0151, 0.0149, 0.0148, 0.0146, 0.0145, 0.0143, 0.0142, 0.0140, 0.0139, 0.0137, 0.0135, 0.0134, 0.0133, 0.0132, 0.0130, 0.0129, 0.0128, 0.0126, 0.0125, 0.0123, 0.0122, 0.0121, 0.0120, 0.0118, 0.0117, 0.0116, 0.0115, 0.0114, 0.0112, 0.0111, 0.0110, 0.0109, 0.0107, 0.0106, 0.0105, 0.0104, 0.0103, 0.0102, 0.0101, 0.0099, 0.0098, 0.0098, 0.0096, 0.0095, 0.0094, 0.0093, 0.0092, 0.0091, 0.0090, 0.0089, 0.0088, 0.0087, 0.0086, 0.0085, 0.0084, 0.0084, 0.0082, 0.0082, 0.0081, 0.0080, 0.0079, 0.0078, 0.0077, 0.0076, 0.0075, 0.0074, 0.0074, 0.0073, 0.0072, 0.0071, 0.0070, 0.0070, 0.0069, 0.0068, 0.0067, 0.0067, 0.0066, 0.0065, 0.0064, 0.0063, 0.0063, 0.0062, 0.0061, 0.0060, 0.0060, 0.0059, 0.0058, 0.0058, 0.0057, 0.0056, 0.0056, 0.0055, 0.0054, 0.0054, 0.0053, 0.0052, 0.0052, 0.0051, 0.0051, 0.0050, 0.0049, 0.0049, 0.0048, 0.0048, 0.0047, 0.0047], device='cuda:0', dtype=torch.bfloat16)