Unable to compile ESP32-S3-box-3 project with update 2024.5.0

[redwngsrul]

The problem

As the title states, since upgrading my ESPHome base add-on, I am unable to compile the file for my ESP32-S3-Box-3. I have several other ESPHome devices that upgraded just fine. They are not Box-3 units, but rather other ESP8266 and ESP32 dev board devices.

Which version of ESPHome has the issue?

2024.5.0

What type of installation are you using?

Home Assistant Add-on

Which version of Home Assistant has the issue?

2024.5.3

What platform are you using?

ESP32-IDF

Board

ESP32-S3-Box-3

Component causing the issue

tflite-micro

Example YAML snippet

substitutions:
  name: esp32-s3-box-3-6412b8
  friendly_name: House-y
packages:
  esphome.voice-assistant: github://esphome/firmware/wake-word-voice-assistant/esp32-s3-box-3.yaml@main
esphome:
  name: ${name}
  name_add_mac_suffix: false
  friendly_name: ${friendly_name}
api:
  encryption:
    key: !secret api_enc_key


wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

switch:
  - platform: template
    name: "Mute when absent"
    id: mute_when_absent
    icon: mdi:account-right-arrow
    optimistic: true
    entity_category: config
    restore_mode: RESTORE_DEFAULT_OFF

number:
  - platform: template
    name: "Presence duration"
    id: radar_delayed_off
    icon: mdi:account-clock
    optimistic: true
    restore_value: true
    initial_value: 60
    min_value: 0
    step: 5
    max_value: 300
    unit_of_measurement: s
    entity_category: config
    mode: box

binary_sensor:
  - platform: gpio
    pin:
      number: GPIO21
    name: "Presence detect"
    disabled_by_default: false
    device_class: "occupancy"
    filters:
      - delayed_off: !lambda return id(radar_delayed_off).state * 1000;
    on_release:
      then:
        - if:
            condition: 
              switch.is_on: mute_when_absent
            then:
              - switch.turn_on: mute
              - light.turn_off: led
    on_press:
      then:
        - if:
            condition: 
              switch.is_on: mute_when_absent
            then:
              - switch.turn_off: mute
              - light.turn_on: led

Anything in the logs that might be useful for us?

Here is the last line of my compile where it reaches error and stops:

Compiling .pioenvs/esp32-s3-box-3-6412b8/components/esp-tflite-micro/tensorflow/lite/micro/micro_context.o
xtensa-esp32s3-elf-g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
*** [.pioenvs/esp32-s3-box-3-6412b8/components/esp-tflite-micro/tensorflow/lite/micro/micro_allocation_info.o] Error 1

Additional information

Additional warnings in the log file:

src/esphome/components/esp_adf/microphone/esp_adf_microphone.cpp: In static member function 'static void esphome::esp_adf::ESPADFMicrophone::read_task(void*)':
src/esphome/components/esp_adf/microphone/esp_adf_microphone.cpp:110:3: warning: missing initializer for member 'i2s_driver_config_t::chan_mask' [-Wmissing-field-initializers]
   };
   ^
src/esphome/components/esp_adf/microphone/esp_adf_microphone.cpp:110:3: warning: missing initializer for member 'i2s_driver_config_t::total_chan' [-Wmissing-field-initializers]
src/esphome/components/esp_adf/microphone/esp_adf_microphone.cpp:110:3: warning: missing initializer for member 'i2s_driver_config_t::left_align' [-Wmissing-field-initializers]
src/esphome/components/esp_adf/microphone/esp_adf_microphone.cpp:110:3: warning: missing initializer for member 'i2s_driver_config_t::big_edin' [-Wmissing-field-initializers]
src/esphome/components/esp_adf/microphone/esp_adf_microphone.cpp:110:3: warning: missing initializer for member 'i2s_driver_config_t::bit_order_msb' [-Wmissing-field-initializers]
src/esphome/components/esp_adf/microphone/esp_adf_microphone.cpp:110:3: warning: missing initializer for member 'i2s_driver_config_t::skip_msk' [-Wmissing-field-initializers]
Compiling .pioenvs/esp32-s3-box-3-6412b8/src/esphome/components/gpio/binary_sensor/gpio_binary_sensor.o
Compiling .pioenvs/esp32-s3-box-3-6412b8/src/esphome/components/ili9xxx/ili9xxx_display.o
src/esphome/components/esp_adf/speaker/esp_adf_speaker.cpp: In static member function 'static void esphome::esp_adf::ESPADFSpeaker::player_task(void*)':
src/esphome/components/esp_adf/speaker/esp_adf_speaker.cpp:77:3: warning: missing initializer for member 'i2s_driver_config_t::chan_mask' [-Wmissing-field-initializers]
   };
   ^
src/esphome/components/esp_adf/speaker/esp_adf_speaker.cpp:77:3: warning: missing initializer for member 'i2s_driver_config_t::total_chan' [-Wmissing-field-initializers]
src/esphome/components/esp_adf/speaker/esp_adf_speaker.cpp:77:3: warning: missing initializer for member 'i2s_driver_config_t::left_align' [-Wmissing-field-initializers]
src/esphome/components/esp_adf/speaker/esp_adf_speaker.cpp:77:3: warning: missing initializer for member 'i2s_driver_config_t::big_edin' [-Wmissing-field-initializers]
src/esphome/components/esp_adf/speaker/esp_adf_speaker.cpp:77:3: warning: missing initializer for member 'i2s_driver_config_t::bit_order_msb' [-Wmissing-field-initializers]
src/esphome/components/esp_adf/speaker/esp_adf_speaker.cpp:77:3: warning: missing initializer for member 'i2s_driver_config_t::skip_msk' [-Wmissing-field-initializers]

Additionally, I have had recent issue with my ESP32-S3-Box-3 compiling any new ESPHome updates. I am forced to Clean All Build Files for the device before installing the update. This above error occurs after I clean all build files and then try to compile. If I proceed with another attempt to compile/install without cleaning build files immediately after this first attempt, I receive a much shorter compile log which is concluded with a different error as shown here:

Compiling .pioenvs/esp32-s3-box-3-6412b8/components/esp-tflite-micro/tensorflow/lite/micro/micro_allocator.o
Compiling .pioenvs/esp32-s3-box-3-6412b8/components/esp-tflite-micro/tensorflow/lite/micro/micro_interpreter.o
Compiling .pioenvs/esp32-s3-box-3-6412b8/components/esp-tflite-micro/tensorflow/lite/micro/micro_interpreter_context.o
Compiling .pioenvs/esp32-s3-box-3-6412b8/components/esp-tflite-micro/tensorflow/lite/micro/micro_interpreter_graph.o
Compiling .pioenvs/esp32-s3-box-3-6412b8/components/esp-tflite-micro/tensorflow/lite/micro/micro_log.o
Compiling .pioenvs/esp32-s3-box-3-6412b8/components/esp-tflite-micro/tensorflow/lite/micro/micro_op_resolver.o
xtensa-esp32s3-elf-g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
*** [.pioenvs/esp32-s3-box-3-6412b8/components/esp-tflite-micro/tensorflow/lite/micro/micro_interpreter_context.o] Error 1

Here is associated error log from the ESPHome add-on to HA at the time of my attempt to compile/install:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/tornado/ioloop.py", line 750, in _run_callback
    ret = callback()
          ^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/tornado/ioloop.py", line 774, in _discard_future_result
    future.result()
  File "/usr/local/lib/python3.11/dist-packages/tornado/gen.py", line 786, in run
    yielded = self.gen.send(value)
              ^^^^^^^^^^^^^^^^^^^^
  File "/esphome/esphome/dashboard/web_server.py", line 254, in _redirect_stdout
    self.write_message({"event": "line", "data": text})
  File "/usr/local/lib/python3.11/dist-packages/tornado/websocket.py", line 332, in write_message
    raise WebSocketClosedError()
tornado.websocket.WebSocketClosedError

edited for markdown formatting issues.

[cryptk]

I ran into a similar issue updating with 2024.5.0 with builds that previously succeeded on an esp32-s3-box now failing. I found that it was due to the system running out of memory. Luckily for me I was able to work around the issue by increasing the RAM on the VM. I was also able to work around the issue by reducing the number of build tasks that ESP home uses, but this obviously caused the build to run much slower than previously as it wasn't using all of the cores.

Check the console/dmesg output and see if you have any lines related to processes being OOMKilled. I found that in order to run the build with 3 threads reliably, I needed five to six GB of RAM allocated to the VM. I never required this much RAM for a build on previous versions.

[redwngsrul]

Thanks for the feedback, I did recently spin up a new VM and imported my backup, but the VM is identical as the old one, it's running 6 cores and 12 GB of RAM. I am seeing OOM killed processes in the console output though. Very weird that 12GB isn't enough. I'll start by trying to limit the # of build tasks before adding more memory, but 12GB seems plenty. What is the default # of build tasks? The field is empty by default and I have no clue what a nominal value would be.

Was anything changed with 2024.5.0 regarding memory usage during compiles? is there a leak?

[redwngsrul]

I set the build task limit to 2 tasks and it did complete the compile process finally. Thank you again for the feedback. I'm going to leave the thread open in case the devs want to review for a potential issue, but otherwise they can close it or let it go stale.

[cryptk]

The default number of build threads is the number of CPU cores, so on your VM it would be 6. I found that I had to give at least 2GB of ram per CPU core on 2024.5 in order to reliably prevent out-of-memory issues. Prior to this version, I never had issues with memory.

[cryptk]

Looked into this a bit more and when building the wake-word-voice-assistant firmware for my ESP32-S3-Box with 3 build threads, on 2024.4.2 the esphome container was typically around 1.5GiB memory used and the highest spike was 2.05GiB, and it was pretty infrequent that it went over 2GiB. When I updated to 2024.5 the highest ram usage I saw was 2.6GiB and it was FREQUENTLY well over 2GiB.

The ram usage has definitely increased on 2024.5, and for users on RPi systems (which is a lot of users) the ~0.5-0.75GiB ram increase can be enough to make the builds not work, especially since on most RPi4 systems that are running Home Assistant have 4 cores, so their ram usage would likely be even higher than it is on my 3 core VM.

[krangchen]

Interesting, I'm able to compile after upgrading ESPHome to 2024.5, however the device will be stuck in a reboot loop without output on the console. Didn't change anything between versions.