Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hanging on 100% issue #15744

Open
CoddedApple opened this issue May 9, 2024 · 4 comments
Open

Hanging on 100% issue #15744

CoddedApple opened this issue May 9, 2024 · 4 comments
Labels
bug-report Report of a bug, yet to be confirmed

Comments

@CoddedApple
Copy link

CoddedApple commented May 9, 2024

What happened?

So the issue of the program getting stuck at 100% has been reported multiple times. In my case it seems to happen after changing the prompt, it'll run a hundred of the same prompt just fine but if I change it even slightly it'll have a chance to delay the final step for who knows how long (it took an hour once).

I only just started to use live preview (it existed before I did) and can see that every step before the final is done quickly, so can I just change the code on my end so that it preemptively stops 1 step before the final and then save it as it would normally, skipping the 100% delay issue?

I'm not familiar with the project's code so I have no idea what would need changing to do this.

Steps to reproduce the problem

Generate
Change prompt.
Generate with a chance for delay.

What should have happened?

No delay at 100%. What is usually a minute per gen can take upwards of 10 minutes at times.

SysInfo

SysInfo

{
    "Platform": "Windows-10-10.0.19045-SP0",
    "Python": "3.10.11",
    "Version": "v1.7.0",
    "Commit": "cf2772fab0af5573da775e7437e6acdca424f26e",
    "Script path": "X:\\stable-diffusion-webui",
    "Data path": "X:\\stable-diffusion-webui",
    "Extensions dir": "X:\\stable-diffusion-webui\\extensions",
    "Checksum": "ecd5c9bebbb2be1e878bbaf7a6c61e12ec2ff24f240964e4170521e23feaa64a",
    "Commandline": [
        "launch.py",
        "--xformers"
    ],
    "Torch env info": {
        "torch_version": "2.0.1+cu118",
        "is_debug_build": "False",
        "cuda_compiled_version": "11.8",
        "gcc_version": null,
        "clang_version": null,
        "cmake_version": null,
        "os": "Microsoft Windows 10 Home",
        "libc_version": "N/A",
        "python_version": "3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)",
        "python_platform": "Windows-10-10.0.19045-SP0",
        "is_cuda_available": "True",
        "cuda_runtime_version": "11.8.89\r",
        "cuda_module_loading": "LAZY",
        "nvidia_driver_version": "546.17",
        "nvidia_gpu_models": "GPU 0: NVIDIA GeForce RTX 2060",
        "cudnn_version": null,
        "pip_version": "pip3",
        "pip_packages": [
            "numpy==1.23.5",
            "open-clip-torch==2.20.0",
            "pytorch-lightning==1.9.4",
            "torch==2.0.1+cu118",
            "torchdiffeq==0.2.3",
            "torchmetrics==1.2.0",
            "torchsde==0.2.6",
            "torchvision==0.15.2+cu118"
        ],
        "conda_packages": null,
        "hip_compiled_version": "N/A",
        "hip_runtime_version": "N/A",
        "miopen_runtime_version": "N/A",
        "caching_allocator_config": "",
        "is_xnnpack_available": "True",
        "cpu_info": [
            "Architecture=9",
            "CurrentClockSpeed=3000",
            "DeviceID=CPU0",
            "Family=198",
            "L2CacheSize=2048",
            "L2CacheSpeed=",
            "Manufacturer=GenuineIntel",
            "MaxClockSpeed=3000",
            "Name=Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz",
            "ProcessorType=3",
            "Revision="
        ]
    },
    "Exceptions": [
        {
            "exception": "A tensor with all NaNs was produced in VAE. This could be because there's not enough precision to represent the picture. Try adding --no-half-vae commandline argument to fix this. Use --disable-nan-check commandline argument to disable this check.",
            "traceback": [
                [
                    "X:\\stable-diffusion-webui\\modules\\processing.py, line 600, decode_latent_batch",
                    "devices.test_for_nans(sample, \"vae\")"
                ],
                [
                    "X:\\stable-diffusion-webui\\modules\\devices.py, line 150, test_for_nans",
                    "raise NansException(message)"
                ]
            ]
        }
    ],
    "CPU": {
        "model": "Intel64 Family 6 Model 158 Stepping 13, GenuineIntel",
        "count logical": 8,
        "count physical": 8
    },
    "RAM": {
        "total": "48GB",
        "used": "24GB",
        "free": "24GB"
    },
    "Extensions": [
        {
            "name": "a1111-sd-webui-tagcomplete",
            "path": "X:\\stable-diffusion-webui\\extensions\\a1111-sd-webui-tagcomplete",
            "version": "3eef536b",
            "branch": "main",
            "remote": "https://github.com/DominikDoom/a1111-sd-webui-tagcomplete.git"
        },
        {
            "name": "sd-dynamic-prompts",
            "path": "X:\\stable-diffusion-webui\\extensions\\sd-dynamic-prompts",
            "version": "1567e787",
            "branch": "main",
            "remote": "https://github.com/adieyal/sd-dynamic-prompts.git"
        },
        {
            "name": "sd-webui-controlnet",
            "path": "X:\\stable-diffusion-webui\\extensions\\sd-webui-controlnet",
            "version": "aa3e2055",
            "branch": "main",
            "remote": "https://github.com/Mikubill/sd-webui-controlnet.git"
        },
        {
            "name": "sd-webui-regional-prompter",
            "path": "X:\\stable-diffusion-webui\\extensions\\sd-webui-regional-prompter",
            "version": "50493ec0",
            "branch": "main",
            "remote": "https://github.com/hako-mikan/sd-webui-regional-prompter.git"
        }
    ],
    "Inactive extensions": [
    ],
    "Environment": {
        "COMMANDLINE_ARGS": "--xformers",
        "GRADIO_ANALYTICS_ENABLED": "False"
    },
    "Config": {
        "samples_save": true,
        "samples_format": "png",
        "samples_filename_pattern": "[seed]",
        "save_images_add_number": true,
        "grid_save": true,
        "grid_format": "jpg",
        "grid_extended_filename": false,
        "grid_only_if_multiple": true,
        "grid_prevent_empty_spots": false,
        "grid_zip_filename_pattern": "",
        "n_rows": -1,
        "font": "",
        "grid_text_active_color": "#000000",
        "grid_text_inactive_color": "#999999",
        "grid_background_color": "#ffffff",
        "enable_pnginfo": true,
        "save_txt": false,
        "save_images_before_face_restoration": false,
        "save_images_before_highres_fix": false,
        "save_images_before_color_correction": false,
        "save_mask": false,
        "save_mask_composite": false,
        "jpeg_quality": 80,
        "webp_lossless": false,
        "export_for_4chan": true,
        "img_downscale_threshold": 4.0,
        "target_side_length": 4000,
        "img_max_size_mp": 200,
        "use_original_name_batch": true,
        "use_upscaler_name_as_suffix": false,
        "save_selected_only": true,
        "save_init_img": false,
        "temp_dir": "",
        "clean_temp_dir_at_start": false,
        "outdir_samples": "",
        "outdir_txt2img_samples": "X:\\txt2img",
        "outdir_img2img_samples": "X:\\img2img",
        "outdir_extras_samples": "outputs/extras-images",
        "outdir_grids": "",
        "outdir_txt2img_grids": "X:\\txt2img-grids",
        "outdir_img2img_grids": "X:\\img2img-grids",
        "outdir_save": "log/images",
        "outdir_init_images": "outputs/init-images",
        "save_to_dirs": false,
        "grid_save_to_dirs": false,
        "use_save_to_dirs_for_ui": false,
        "directories_filename_pattern": "[date]",
        "directories_max_prompt_words": 8,
        "ESRGAN_tile": 192,
        "ESRGAN_tile_overlap": 8,
        "realesrgan_enabled_models": [
            "R-ESRGAN 4x+",
            "R-ESRGAN 4x+ Anime6B"
        ],
        "upscaler_for_img2img": null,
        "face_restoration_model": "CodeFormer",
        "code_former_weight": 0.5,
        "face_restoration_unload": false,
        "show_warnings": true,
        "memmon_poll_rate": 8,
        "samples_log_stdout": false,
        "multiple_tqdm": true,
        "print_hypernet_extra": true,
        "list_hidden_files": true,
        "disable_mmap_load_safetensors": false,
        "unload_models_when_training": false,
        "pin_memory": false,
        "save_optimizer_state": false,
        "save_training_settings_to_txt": true,
        "dataset_filename_word_regex": "",
        "dataset_filename_join_string": " ",
        "training_image_repeats_per_epoch": 1,
        "training_write_csv_every": 500,
        "training_xattention_optimizations": false,
        "training_enable_tensorboard": false,
        "training_tensorboard_save_images": false,
        "training_tensorboard_flush_every": 120,
        "sd_model_checkpoint": "",
        "sd_checkpoint_cache": 0,
        "sd_vae_checkpoint_cache": 0,
        "sd_vae": "nai.vae.pt",
        "sd_vae_as_default": true,
        "sd_unet": "Automatic",
        "inpainting_mask_weight": 1.0,
        "initial_noise_multiplier": 1.0,
        "img2img_color_correction": false,
        "img2img_fix_steps": false,
        "img2img_background_color": "#ffffff",
        "enable_quantization": false,
        "enable_emphasis": true,
        "enable_batch_seeds": true,
        "comma_padding_backtrack": 20,
        "CLIP_stop_at_last_layers": 2,
        "upcast_attn": false,
        "auto_vae_precision": true,
        "randn_source": "GPU",
        "sdxl_crop_top": 0,
        "sdxl_crop_left": 0,
        "sdxl_refiner_low_aesthetic_score": 2.5,
        "sdxl_refiner_high_aesthetic_score": 6.0,
        "cross_attention_optimization": "Automatic",
        "s_min_uncond": 0,
        "token_merging_ratio": 0,
        "token_merging_ratio_img2img": 0,
        "token_merging_ratio_hr": 0,
        "pad_cond_uncond": false,
        "experimental_persistent_cond_cache": false,
        "use_old_emphasis_implementation": false,
        "use_old_karras_scheduler_sigmas": false,
        "no_dpmpp_sde_batch_determinism": false,
        "use_old_hires_fix_width_height": false,
        "dont_fix_second_order_samplers_schedule": false,
        "hires_fix_use_firstpass_conds": false,
        "interrogate_keep_models_in_memory": false,
        "interrogate_return_ranks": false,
        "interrogate_clip_num_beams": 1,
        "interrogate_clip_min_length": 24,
        "interrogate_clip_max_length": 48,
        "interrogate_clip_dict_limit": 1500,
        "interrogate_clip_skip_categories": [],
        "interrogate_deepbooru_score_threshold": 0.5,
        "deepbooru_sort_alpha": true,
        "deepbooru_use_spaces": true,
        "deepbooru_escape": true,
        "deepbooru_filter_tags": "",
        "extra_networks_show_hidden_directories": true,
        "extra_networks_hidden_models": "When searched",
        "extra_networks_default_multiplier": 1.0,
        "extra_networks_card_width": 0,
        "extra_networks_card_height": 0,
        "extra_networks_card_text_scale": 1.0,
        "extra_networks_card_show_desc": true,
        "extra_networks_add_text_separator": " ",
        "ui_extra_networks_tab_reorder": "",
        "textual_inversion_print_at_load": false,
        "textual_inversion_add_hashes_to_infotext": true,
        "sd_hypernetwork": "None",
        "localization": "None",
        "gradio_theme": "Default",
        "img2img_editor_height": 720,
        "return_grid": true,
        "return_mask": false,
        "return_mask_composite": false,
        "do_not_show_images": false,
        "send_seed": true,
        "send_size": true,
        "js_modal_lightbox": true,
        "js_modal_lightbox_initially_zoomed": true,
        "js_modal_lightbox_gamepad": false,
        "js_modal_lightbox_gamepad_repeat": 250,
        "show_progress_in_title": true,
        "samplers_in_dropdown": true,
        "dimensions_and_batch_together": true,
        "keyedit_precision_attention": 0.1,
        "keyedit_precision_extra": 0.05,
        "keyedit_delimiters": ".,\\/!?%^*;:{}=`~()",
        "keyedit_move": true,
        "quicksettings_list": [
            "sd_model_checkpoint"
        ],
        "ui_tab_order": [],
        "hidden_tabs": [],
        "ui_reorder_list": [],
        "hires_fix_show_sampler": true,
        "hires_fix_show_prompts": true,
        "disable_token_counters": false,
        "add_model_hash_to_info": true,
        "add_model_name_to_info": true,
        "add_user_name_to_info": false,
        "add_version_to_infotext": true,
        "disable_weights_auto_swap": true,
        "infotext_styles": "Apply if any",
        "show_progressbar": true,
        "live_previews_enable": true,
        "live_previews_image_format": "jpeg",
        "show_progress_grid": false,
        "show_progress_every_n_steps": 2,
        "show_progress_type": "Approx NN",
        "live_preview_content": "Prompt",
        "live_preview_refresh_period": 1000,
        "hide_samplers": [],
        "eta_ddim": 0.0,
        "eta_ancestral": 1.0,
        "ddim_discretize": "uniform",
        "s_churn": 0.0,
        "s_tmin": 0.0,
        "s_noise": 1.0,
        "k_sched_type": "Automatic",
        "sigma_min": 0.0,
        "sigma_max": 0.0,
        "rho": 0.0,
        "eta_noise_seed_delta": 0,
        "always_discard_next_to_last_sigma": false,
        "uni_pc_variant": "bh1",
        "uni_pc_skip_type": "time_uniform",
        "uni_pc_order": 3,
        "uni_pc_lower_order_final": true,
        "postprocessing_enable_in_main_ui": [],
        "postprocessing_operation_order": [],
        "upscaling_max_images_in_cache": 5,
        "disabled_extensions": [],
        "disable_all_extensions": "none",
        "restore_config_state_file": "",
        "sd_checkpoint_hash": "",
        "ldsr_steps": 100,
        "ldsr_cached": false,
        "SCUNET_tile": 256,
        "SCUNET_tile_overlap": 8,
        "SWIN_tile": 192,
        "SWIN_tile_overlap": 8,
        "lora_functional": false,
        "sd_lora": "None",
        "lora_preferred_name": "Alias from file",
        "lora_add_hashes_to_infotext": false,
        "lora_show_all": false,
        "lora_hide_unknown_for_versions": [],
        "extra_options": [],
        "extra_options_accordion": false,
        "canvas_hotkey_zoom": "Alt",
        "canvas_hotkey_adjust": "Ctrl",
        "canvas_hotkey_move": "F",
        "canvas_hotkey_fullscreen": "S",
        "canvas_hotkey_reset": "R",
        "canvas_hotkey_overlap": "O",
        "canvas_show_tooltip": true,
        "canvas_blur_prompt": false,
        "canvas_disabled_functions": [
            "Overlap"
        ],
        "sd_vae_overrides_per_model_preferences": false,
        "hypertile_enable_unet": false,
        "hypertile_enable_unet_secondpass": false,
        "hypertile_max_depth_unet": 3,
        "hypertile_max_tile_unet": 256,
        "hypertile_swap_size_unet": 3,
        "hypertile_enable_vae": false,
        "hypertile_max_depth_vae": 3,
        "hypertile_max_tile_vae": 128,
        "hypertile_swap_size_vae": 3,
        "tac_tagFile": "danbooru.csv",
        "tac_active": true,
        "tac_activeIn.txt2img": true,
        "tac_activeIn.img2img": true,
        "tac_activeIn.negativePrompts": true,
        "tac_activeIn.thirdParty": true,
        "tac_activeIn.modelList": "",
        "tac_activeIn.modelListMode": "Blacklist",
        "tac_slidingPopup": true,
        "tac_maxResults": 5,
        "tac_showAllResults": false,
        "tac_resultStepLength": 100,
        "tac_delayTime": 100,
        "tac_useWildcards": true,
        "tac_sortWildcardResults": true,
        "tac_useEmbeddings": true,
        "tac_includeEmbeddingsInNormalResults": false,
        "tac_useHypernetworks": true,
        "tac_useLoras": true,
        "tac_useLycos": true,
        "tac_showWikiLinks": false,
        "tac_showExtraNetworkPreviews": true,
        "tac_replaceUnderscores": true,
        "tac_escapeParentheses": true,
        "tac_appendComma": true,
        "tac_appendSpace": true,
        "tac_alwaysSpaceAtEnd": true,
        "tac_modelKeywordCompletion": "Never",
        "tac_modelKeywordLocation": "Start of prompt",
        "tac_wildcardCompletionMode": "To next folder level",
        "tac_alias.searchByAlias": true,
        "tac_alias.onlyShowAlias": false,
        "tac_translation.translationFile": "None",
        "tac_translation.oldFormat": false,
        "tac_translation.searchByTranslation": true,
        "tac_translation.liveTranslation": false,
        "tac_extra.extraFile": "extra-quality-tags.csv",
        "tac_extra.addMode": "Insert before",
        "tac_chantFile": "demo-chants.json",
        "tac_keymap": "{\n    \"MoveUp\": \"ArrowUp\",\n    \"MoveDown\": \"ArrowDown\",\n    \"JumpUp\": \"PageUp\",\n    \"JumpDown\": \"PageDown\",\n    \"JumpToStart\": \"Home\",\n    \"JumpToEnd\": \"End\",\n    \"ChooseSelected\": \"Enter\",\n    \"ChooseFirstOrSelected\": \"Tab\",\n    \"Close\": \"Escape\"\n}",
        "tac_colormap": "{\n    \"danbooru\": {\n        \"-1\": [\"red\", \"maroon\"],\n        \"0\": [\"lightblue\", \"dodgerblue\"],\n        \"1\": [\"indianred\", \"firebrick\"],\n        \"3\": [\"violet\", \"darkorchid\"],\n        \"4\": [\"lightgreen\", \"darkgreen\"],\n        \"5\": [\"orange\", \"darkorange\"]\n    },\n    \"e621\": {\n        \"-1\": [\"red\", \"maroon\"],\n        \"0\": [\"lightblue\", \"dodgerblue\"],\n        \"1\": [\"gold\", \"goldenrod\"],\n        \"3\": [\"violet\", \"darkorchid\"],\n        \"4\": [\"lightgreen\", \"darkgreen\"],\n        \"5\": [\"tomato\", \"darksalmon\"],\n        \"6\": [\"red\", \"maroon\"],\n        \"7\": [\"whitesmoke\", \"black\"],\n        \"8\": [\"seagreen\", \"darkseagreen\"]\n    }\n}",
        "tac_refreshTempFiles": "Refresh TAC temp files",
        "save_images_replace_action": "Replace",
        "save_incomplete_images": false,
        "notification_audio": true,
        "notification_volume": 100,
        "use_old_scheduling": false,
        "extra_networks_dir_button_function": false,
        "extra_networks_card_order_field": "Path",
        "extra_networks_card_order": "Ascending",
        "lora_in_memory_limit": 0,
        "persistent_cond_cache": true,
        "batch_cond_uncond": true,
        "s_tmax": 0,
        "sgm_noise_multiplier": false,
        "sd_checkpoints_limit": 1,
        "sd_checkpoints_keep_in_cpu": true,
        "tiling": false,
        "hires_fix_refiner_pass": "second pass",
        "sd_vae_encode_method": "Full",
        "sd_vae_decode_method": "Full",
        "img2img_extra_noise": 0,
        "img2img_sketch_default_brush_color": "#ffffff",
        "img2img_inpaint_mask_brush_color": "#ffffff",
        "img2img_inpaint_sketch_default_brush_color": "#ffffff",
        "img2img_batch_show_results_limit": 32,
        "gallery_height": "",
        "add_vae_name_to_info": true,
        "add_vae_hash_to_info": true,
        "infotext_skip_pasting": [],
        "live_preview_allow_lowvram_full": false,
        "live_preview_fast_interrupt": false,
        "js_live_preview_in_modal_lightbox": false,
        "keyedit_delimiters_whitespace": [
            "Tab",
            "Carriage Return",
            "Line Feed"
        ],
        "extra_options_txt2img": [],
        "extra_options_img2img": [],
        "extra_options_cols": 1,
        "compact_prompt_box": false,
        "sd_checkpoint_dropdown_use_short": false,
        "txt2img_settings_accordion": false,
        "img2img_settings_accordion": false,
        "gradio_themes_cache": true,
        "api_enable_requests": true,
        "api_forbid_local_requests": true,
        "api_useragent": "",
        "auto_launch_browser": "Disable",
        "enable_console_prompts": false,
        "show_gradio_deprecation_warnings": true,
        "hide_ldm_prints": true,
        "dump_stacks_on_signal": false,
        "face_restoration": false,
        "postprocessing_existing_caption_action": "Ignore",
        "canvas_auto_expand": true,
        "tac_wildcardExclusionList": "",
        "tac_skipWildcardRefresh": false,
        "tac_useLoraPrefixForLycos": true,
        "tac_modelSortOrder": "Name",
        "tac_useStyleVars": false,
        "tac_frequencySort": true,
        "tac_frequencyFunction": "Logarithmic (weak)",
        "tac_frequencyMinCount": 3,
        "tac_frequencyMaxAge": 30,
        "tac_frequencyRecommendCap": 10,
        "tac_frequencyIncludeAlias": false,
        "dp_ignore_whitespace": false,
        "dp_write_raw_template": false,
        "dp_write_prompts_to_file": false,
        "dp_parser_variant_start": "{",
        "dp_parser_variant_end": "}",
        "dp_parser_wildcard_wrap": "__",
        "dp_limit_jinja_prompts": false,
        "dp_auto_purge_cache": false,
        "dp_wildcard_manager_no_dedupe": false,
        "dp_wildcard_manager_no_sort": false,
        "dp_wildcard_manager_shuffle": false,
        "dp_magicprompt_default_model": "Gustavosta/MagicPrompt-Stable-Diffusion",
        "dp_magicprompt_batch_size": 1,
        "control_net_detectedmap_dir": "detected_maps",
        "control_net_models_path": "",
        "control_net_modules_path": "",
        "control_net_unit_count": 3,
        "control_net_model_cache_size": 2,
        "control_net_inpaint_blur_sigma": 7,
        "control_net_no_detectmap": false,
        "control_net_detectmap_autosaving": false,
        "control_net_allow_script_control": false,
        "control_net_sync_field_args": true,
        "controlnet_show_batch_images_in_ui": false,
        "controlnet_increment_seed_during_batch": false,
        "controlnet_disable_openpose_edit": false,
        "controlnet_disable_photopea_edit": false,
        "controlnet_photopea_warning": true,
        "controlnet_ignore_noninpaint_mask": false,
        "controlnet_clip_detector_on_cpu": false,
        "controlnet_control_type_dropdown": false,
        "regprp_debug": false,
        "regprp_hidepmask": false
    },
    "Startup": {
        "total": 20.960586547851562,
        "records": {
            "initial startup": 0.0553586483001709,
            "prepare environment/checks": 0.010970830917358398,
            "prepare environment/git version info": 0.06526303291320801,
            "prepare environment/torch GPU test": 2.168682098388672,
            "prepare environment/clone repositores": 0.23125052452087402,
            "prepare environment/run extensions installers/a1111-sd-webui-tagcomplete": 0.0,
            "prepare environment/run extensions installers/sd-dynamic-prompts": 0.3091740608215332,
            "prepare environment/run extensions installers/sd-webui-controlnet": 0.2828691005706787,
            "prepare environment/run extensions installers/sd-webui-regional-prompter": 0.0,
            "prepare environment/run extensions installers": 0.5920431613922119,
            "prepare environment": 3.1100974082946777,
            "launcher": 0.0019943714141845703,
            "import torch": 7.1912078857421875,
            "import gradio": 1.2739925384521484,
            "setup paths": 1.5650112628936768,
            "import ldm": 0.006981372833251953,
            "import sgm": 0.0,
            "initialize shared": 0.32887840270996094,
            "other imports": 0.919750452041626,
            "opts onchange": 0.0,
            "setup SD model": 0.0029926300048828125,
            "setup codeformer": 0.17453384399414062,
            "setup gfpgan": 0.02892160415649414,
            "set samplers": 0.0,
            "list extensions": 0.004987001419067383,
            "restore config state file": 0.0,
            "list SD models": 0.7826824188232422,
            "list localizations": 0.0009980201721191406,
            "load scripts/custom_code.py": 0.008975982666015625,
            "load scripts/img2imgalt.py": 0.000997304916381836,
            "load scripts/loopback.py": 0.0,
            "load scripts/outpainting_mk_2.py": 0.0,
            "load scripts/poor_mans_outpainting.py": 0.000997304916381836,
            "load scripts/postprocessing_caption.py": 0.0,
            "load scripts/postprocessing_codeformer.py": 0.0,
            "load scripts/postprocessing_create_flipped_copies.py": 0.0,
            "load scripts/postprocessing_focal_crop.py": 0.0019953250885009766,
            "load scripts/postprocessing_gfpgan.py": 0.0009970664978027344,
            "load scripts/postprocessing_split_oversized.py": 0.0,
            "load scripts/postprocessing_upscale.py": 0.0,
            "load scripts/processing_autosized_crop.py": 0.0009975433349609375,
            "load scripts/prompt_matrix.py": 0.0,
            "load scripts/prompts_from_file.py": 0.0,
            "load scripts/sd_upscale.py": 0.0009970664978027344,
            "load scripts/xyz_grid.py": 0.0029909610748291016,
            "load scripts/ldsr_model.py": 2.0528385639190674,
            "load scripts/lora_script.py": 0.6170594692230225,
            "load scripts/scunet_model.py": 0.04787158966064453,
            "load scripts/swinir_model.py": 0.04606056213378906,
            "load scripts/hotkey_config.py": 0.0009968280792236328,
            "load scripts/extra_options_section.py": 0.0,
            "load scripts/hypertile_script.py": 0.0937502384185791,
            "load scripts/hypertile_xyz.py": 0.0,
            "load scripts/model_keyword_support.py": 0.004293203353881836,
            "load scripts/shared_paths.py": 0.0009911060333251953,
            "load scripts/tag_autocomplete_helper.py": 0.45704221725463867,
            "load scripts/tag_frequency_db.py": 0.0010280609130859375,
            "load scripts/dynamic_prompting.py": 0.19117522239685059,
            "load scripts/adapter.py": 0.0009984970092773438,
            "load scripts/api.py": 0.27554821968078613,
            "load scripts/batch_hijack.py": 0.0009677410125732422,
            "load scripts/cldm.py": 0.0,
            "load scripts/controlnet.py": 0.2533755302429199,
            "load scripts/controlnet_diffusers.py": 0.0009980201721191406,
            "load scripts/controlnet_lllite.py": 0.0,
            "load scripts/controlnet_lora.py": 0.0009961128234863281,
            "load scripts/controlnet_model_guess.py": 0.0,
            "load scripts/controlnet_sparsectrl.py": 0.0,
            "load scripts/controlnet_version.py": 0.0009982585906982422,
            "load scripts/enums.py": 0.0010223388671875,
            "load scripts/external_code.py": 0.0,
            "load scripts/global_state.py": 0.0010027885437011719,
            "load scripts/hook.py": 0.0,
            "load scripts/infotext.py": 0.0,
            "load scripts/logging.py": 0.0009908676147460938,
            "load scripts/lvminthin.py": 0.0,
            "load scripts/movie2movie.py": 0.0010008811950683594,
            "load scripts/supported_preprocessor.py": 0.0010008811950683594,
            "load scripts/utils.py": 0.0,
            "load scripts/xyz_grid_support.py": 0.0009942054748535156,
            "load scripts/attention.py": 0.0,
            "load scripts/latent.py": 0.004989147186279297,
            "load scripts/regions.py": 0.0009663105010986328,
            "load scripts/rp.py": 0.050436973571777344,
            "load scripts/rps.py": 0.010969877243041992,
            "load scripts/refiner.py": 0.0,
            "load scripts/seed.py": 0.0,
            "load scripts": 4.139312267303467,
            "load upscalers": 0.0029921531677246094,
            "refresh VAE": 0.006013631820678711,
            "refresh textual inversion templates": 0.0,
            "scripts list_optimizers": 0.0009970664978027344,
            "scripts list_unets": 0.0,
            "reload hypernetworks": 0.0019659996032714844,
            "initialize extra networks": 0.029920339584350586,
            "scripts before_ui_callback": 0.0,
            "create ui": 1.082993507385254,
            "gradio launch": 0.27393460273742676,
            "add APIs": 0.005983829498291016,
            "app_started_callback/lora_script.py": 0.0009968280792236328,
            "app_started_callback/tag_autocomplete_helper.py": 0.005984067916870117,
            "app_started_callback/api.py": 0.0029921531677246094,
            "app_started_callback": 0.00997304916381836
        }
    },
    "Packages": [
        "-rotobuf==4.25.3",
        "absl-py==2.0.0",
        "accelerate==0.21.0",
        "addict==2.4.0",
        "aenum==3.1.15",
        "aiofiles==23.2.1",
        "aiohttp==3.9.0",
        "aiosignal==1.3.1",
        "albumentations==1.4.3",
        "altair==5.1.2",
        "antlr4-python3-runtime==4.9.3",
        "anyio==3.7.1",
        "astunparse==1.6.3",
        "async-timeout==4.0.3",
        "attrs==23.1.0",
        "basicsr==1.4.2",
        "beautifulsoup4==4.12.2",
        "blendmodes==2022",
        "blis==0.7.11",
        "boltons==23.1.1",
        "cachetools==5.3.2",
        "catalogue==2.0.10",
        "certifi==2023.11.17",
        "cffi==1.16.0",
        "chardet==5.2.0",
        "charset-normalizer==3.3.2",
        "clean-fid==0.1.35",
        "click==8.1.7",
        "clip==1.0",
        "cloudpathlib==0.16.0",
        "colorama==0.4.6",
        "coloredlogs==15.0.1",
        "colorlog==6.8.2",
        "confection==0.1.3",
        "contourpy==1.2.0",
        "cssselect2==0.7.0",
        "cycler==0.12.1",
        "cymem==2.0.8",
        "cython==3.0.9",
        "deepdanbooru==1.0.2",
        "deprecation==2.1.0",
        "depth-anything==2024.1.22.0",
        "dsine==2024.3.23",
        "dynamicprompts==0.31.0",
        "easydict==1.13",
        "einops==0.4.1",
        "embreex==2.17.7.post4",
        "exceptiongroup==1.2.0",
        "facexlib==0.3.0",
        "fake-useragent==1.4.0",
        "fastapi==0.94.0",
        "ffmpy==0.3.1",
        "filelock==3.13.1",
        "filterpy==1.4.5",
        "flatbuffers==24.3.7",
        "fonttools==4.45.0",
        "frozenlist==1.4.0",
        "fsspec==2023.10.0",
        "ftfy==6.1.1",
        "future==0.18.3",
        "fvcore==0.1.5.post20221221",
        "gast==0.5.4",
        "gdown==4.7.1",
        "geffnet==1.0.2",
        "gfpgan==1.3.8",
        "gitdb==4.0.11",
        "gitpython==3.1.32",
        "glob2==0.5",
        "google-auth-oauthlib==1.1.0",
        "google-auth==2.23.4",
        "google-pasta==0.2.0",
        "gradio-client==0.5.0",
        "gradio==3.41.2",
        "grpcio==1.59.3",
        "h11==0.12.0",
        "h5py==3.10.0",
        "handrefinerportable==2024.2.12.0",
        "httpcore==0.15.0",
        "httpx==0.24.1",
        "huggingface-hub==0.19.4",
        "humanfriendly==10.0",
        "idna==3.4",
        "imageio==2.33.0",
        "importlib-metadata==6.8.0",
        "importlib-resources==6.1.1",
        "inflection==0.5.1",
        "insightface==0.7.3",
        "iopath==0.1.9",
        "jax==0.4.25",
        "jinja2==3.1.2",
        "joblib==1.3.2",
        "jsonmerge==1.8.0",
        "jsonschema-specifications==2023.11.1",
        "jsonschema==4.20.0",
        "keras==3.1.1",
        "kiwisolver==1.4.5",
        "kornia==0.6.7",
        "langcodes==3.3.0",
        "lark==1.1.2",
        "lazy-loader==0.3",
        "libclang==18.1.1",
        "lightning-utilities==0.10.0",
        "llvmlite==0.41.1",
        "lmdb==1.4.1",
        "lpips==0.1.4",
        "lxml==5.1.0",
        "mapbox-earcut==1.0.1",
        "markdown-it-py==3.0.0",
        "markdown==3.5.1",
        "markupsafe==2.1.3",
        "matplotlib==3.8.2",
        "mdurl==0.1.2",
        "mediapipe==0.10.11",
        "ml-dtypes==0.3.2",
        "mpmath==1.3.0",
        "multidict==6.0.4",
        "murmurhash==1.0.10",
        "namex==0.0.7",
        "networkx==3.2.1",
        "numba==0.58.1",
        "numpy==1.23.5",
        "oauthlib==3.2.2",
        "omegaconf==2.2.3",
        "onnx==1.16.0",
        "onnxruntime-gpu==1.17.1",
        "open-clip-torch==2.20.0",
        "opencv-contrib-python==4.9.0.80",
        "opencv-python-headless==4.9.0.80",
        "opencv-python==4.8.1.78",
        "opt-einsum==3.3.0",
        "optree==0.10.0",
        "orjson==3.9.10",
        "packaging==23.2",
        "pandas==2.1.3",
        "piexif==1.1.3",
        "pillow==9.5.0",
        "pip==23.0.1",
        "platformdirs==4.0.0",
        "portalocker==2.8.2",
        "preshed==3.0.9",
        "prettytable==3.10.0",
        "protobuf==3.20.3",
        "psutil==5.9.5",
        "pyasn1-modules==0.3.0",
        "pyasn1==0.5.1",
        "pycollada==0.8",
        "pycparser==2.21",
        "pydantic==1.10.13",
        "pydub==0.25.1",
        "pygments==2.17.2",
        "pyparsing==3.1.1",
        "pyreadline3==3.4.1",
        "pysocks==1.7.1",
        "python-dateutil==2.8.2",
        "python-multipart==0.0.6",
        "pytorch-lightning==1.9.4",
        "pytz==2023.3.post1",
        "pywavelets==1.5.0",
        "pywin32==306",
        "pyyaml==6.0.1",
        "realesrgan==0.3.0",
        "referencing==0.31.0",
        "regex==2023.10.3",
        "reportlab==4.1.0",
        "requests-oauthlib==1.3.1",
        "requests==2.31.0",
        "resize-right==0.0.2",
        "rich==13.7.1",
        "rpds-py==0.13.1",
        "rsa==4.9",
        "rtree==1.2.0",
        "safetensors==0.3.1",
        "scikit-image==0.21.0",
        "scikit-learn==1.4.1.post1",
        "scipy==1.11.4",
        "semantic-version==2.10.0",
        "send2trash==1.8.0",
        "sentencepiece==0.1.99",
        "setuptools==65.5.0",
        "shapely==2.0.3",
        "six==1.16.0",
        "smart-open==6.4.0",
        "smmap==5.0.1",
        "sniffio==1.3.0",
        "sounddevice==0.4.6",
        "soupsieve==2.5",
        "spacy-legacy==3.0.12",
        "spacy-loggers==1.0.5",
        "spacy==3.7.2",
        "srsly==2.4.8",
        "starlette==0.26.1",
        "svg.path==6.3",
        "svglib==1.5.1",
        "sympy==1.12",
        "tabulate==0.9.0",
        "tb-nightly==2.16.0a20231121",
        "tensorboard-data-server==0.7.2",
        "tensorboard==2.16.2",
        "tensorflow-intel==2.16.1",
        "tensorflow-io-gcs-filesystem==0.31.0",
        "tensorflow==2.16.1",
        "termcolor==2.4.0",
        "tf-keras-nightly==2.16.0.dev2023112110",
        "thinc==8.2.1",
        "threadpoolctl==3.4.0",
        "tifffile==2023.9.26",
        "timm==0.9.2",
        "tinycss2==1.2.1",
        "tokenizers==0.13.3",
        "tomesd==0.1.3",
        "tomli==2.0.1",
        "toolz==0.12.0",
        "torch==2.0.1+cu118",
        "torchdiffeq==0.2.3",
        "torchmetrics==1.2.0",
        "torchsde==0.2.6",
        "torchvision==0.15.2+cu118",
        "tqdm==4.66.1",
        "trampoline==0.1.2",
        "transformers==4.30.2",
        "trimesh==4.2.2",
        "typer==0.9.0",
        "typing-extensions==4.10.0",
        "tzdata==2023.3",
        "urllib3==2.1.0",
        "uvicorn==0.24.0.post1",
        "vhacdx==0.0.6",
        "wasabi==1.1.2",
        "wcwidth==0.2.11",
        "weasel==0.3.4",
        "webencodings==0.5.1",
        "websockets==11.0.3",
        "werkzeug==3.0.1",
        "wheel==0.43.0",
        "wrapt==1.16.0",
        "xatlas==0.0.9",
        "xformers==0.0.20",
        "xxhash==3.4.1",
        "yacs==0.1.8",
        "yapf==0.40.2",
        "yarl==1.9.3",
        "zipp==3.17.0",
        "zipunicode==1.1.1" 
    ]
}

@CoddedApple CoddedApple added the bug-report Report of a bug, yet to be confirmed label May 9, 2024
@w-e-w
Copy link
Collaborator

w-e-w commented May 9, 2024

I don't know what's causing issue but you got to realize 100% is not 100%

the progress while we showing is only the sampling steps
but there's also other things that happen after
such as the VAE or potentially other post processing
so the true progress is 100%+vae+other post processing

so if you "preemptively stops 1 step before the final"
all that dose is make 100%+vae+other post processing in to 95%+vae+other post processing

if it gets stuck afterwards like in VAE, it's still stuck

I'm guessing the issue is during VAE your system doesn't have enough vram and start using system ram which causes massive slow down
see #11063 (comment)

live preview works differently from normal image, (it essentially decodes the latent space using a different method by default using a cheaper method so it's quicker)

2024-05-09 15_57_18_786 chrome

if the issue is with VRAM usage during VAE pahse
there's a couple of things you can try
you can try using TAESD as the final VAE, using TAESD will result in image quality drop but should be much faster and use less VRAM
2024-05-09 15_58_10_921 chrome
or you can try https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111
this can reduce the vram usage during VAE pahse


if the issue is it getting stuck during other post processing then more investigation needs to be done


you should really fill in the issue report with the proper information we requested
like the Sysinfo, if you uploaded that then I can see your settings and what system you're on what graphics card you're using
which may give us clue on what is causing the issue

@CoddedApple
Copy link
Author

CoddedApple commented May 9, 2024

@w-e-w Updated Sysinfo
Just checked a few generations, vram is around 4.6~/6.0 during hires then spikes to 5.7~/6.0 on final processing for a 576x640 image upscaled by 2x.

@w-e-w
Copy link
Collaborator

w-e-w commented May 9, 2024

as expected RTX 2060 with 6GB vram
if it spikes up to 6 then then it's likely that it's actually going into System RAM which can cause slow down

so I think it's highly likely it's caused by not enough VRAM
if you want to test i if this is the case you can you fellow #11063 (comment)
and set CUDA - Sysmem Fallback Policy.to Prefer No Sysmem Fallback
this ment the driver will not try to use system RAM but just ERROR when not enouth VRAM (image gen will fail)

it is possible in your situation --medvram or --lowvram may help
enableing FP8 weight in setting may also help
basically all the settings that reduces vram use may help

@CoddedApple
Copy link
Author

CoddedApple commented May 10, 2024

@w-e-w I don't know if it's actually a vram issue. For a normal 50-90 second gen the vram will spike to 5.8~ on final and normal ram will bump up a bit. I just got a generation that took 8+ minutes but it also only spiked to 5.8 and bumped up normal ram a little, so there's no difference there besides the time it takes to finish. Between the 8 minute gen and the gen before it which took 54 seconds, the only difference was the seed and a slight change made to the prompt, the size and everything else was kept the same.

I followed the instructions and turned CUDA - Sysmem Fallback Policy but it didn't do anything different when generating. I even tried making a couple images twice as large as I normally do, capped at vram 5.8 and uses a bunch of ram but no delay and no failure.

@CoddedApple CoddedApple changed the title A workaround for the stuck at 100% issue? Stuck at 100% issue May 10, 2024
@CoddedApple CoddedApple changed the title Stuck at 100% issue Hanging on 100% issue May 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug-report Report of a bug, yet to be confirmed
Projects
None yet
Development

No branches or pull requests

2 participants