Binaries incompatible with Arduino-CLI for OTA

[marchingband]

I have a custom OTA process, using the Arduino Updates library.
https://github.com/marchingband/wvr
If I use Arduino-CLI, then any binary produced from any version 2.x.x will be compatible.
If I use PlatformIIO to produce the binaries, they often cause the device to be bricked, if it was initially flashed using Arduino-CLI.
I have many such boards in the hands of users, as well as some that were flashed using PIO. So I am in a precarious position.

rst:0x10 (RTCWDT_RTC_RESET),boot:0x3f (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1420
ho 0 tail 12 room 4
load:0x40078000,len:13540
load:0x40080400,len:3604
entry 0x400805f0
ets Jun  8 2016 00:22:57

I have done a great deal of regression testing to narrow down the issue, what versions break it.

The most obvious case is:

• Flash a board using Arduino-cli with Arduino-ESP32 v2.0.1
• Compile a binary using PIO for the same version platform = espressif32@4.1.0
• Run my custom OTA code, device gets bricked.

Strangely, an option that works is:

• Flash a board using Arduino-cli with Arduino-ESP32 v2.0.1
• Compile a binary using PIO with Arduino-ESP32 v2.0.8 platform = espressif32@6.2.0

I am hoping that someone can help me identify the issue, and ideally patch it.
Are binaries meant to be compatible for OTA across versions? And if so, shouldn't they be compatible with those that Arduino-CLI produces?
Thanks.

[valeros]

Hi @marchingband,

Are binaries meant to be compatible for OTA across versions? And if so, shouldn't they be compatible with those that Arduino-CLI produces?

Arduino-CLI is a completely different project, so there is no guarantee that binaries are compatible in any way.

Compile a binary using PIO for the same version platform = espressif32@4.1.0

v4.1.0 is quite old version, many changes and bug fixes have been added since that release. The fact that somewhat recent v6.2.0 works as expected indirectly explains this behavior. You've mentioned that you already had done some regression testing, so have you figured out what's the lowest version of the espressif32 dev platform works as expected?

[marchingband]

Arduino-CLI is a completely different project, so there is no guarantee that binaries are compatible in any way.

That seems odd, I assume that Arduino-CLI produces identical binaries to Arduino IDE, and I would assume PIO were also the same, since it is using Arduino as the framework ... and there is no indication in the docs otherwise.

have you figured out what's the lowest version of the espressif32 dev platform works as expected

I have not

[Jason2866]

The build process between ArduinoIDE and Platformio IS different. It can be that the result is the same. In my experience there is code compability, but there is no guarantee and nowhere this is mentioned. So if you provide both ways of building, you have to do compability tests. One more reason to drop ArduinoIDE built support.

One more thing, updates between major core versions are not recommended.
Different IDF versions used. There is no OTA code compability guaranteed.
Even between different Arduino Cores minor versions OTA update issues can happen.

There is a known OTA bug in older IDF 4.4 versions. This bug is solved in newer version (IDF 4.4.6). So it is explainable that a older built version is successful with OTA to an actual version and it fails when trying to update to an not so new version.

Arduino core 2.0.1 and 2.0.8 are both old. Better do your OTA update tests with actual version 2.0.14 (Platformio "6.5.0")

The used partition schemes are identic? Arduino espressif32 changed them a while ago by adding the core dump section. The name of the partition scheme is not changed!!
So an issue will happen when trying to update between older and newer built versions.

Another reason why i choose the way to use ONLY partition schemes defined in the project. Both ArduinoIDE and Platformio does support this.

[Jason2866]

In short the issues you encounter are most probably not Platformio related. -> Bugs and incompatibities under the hood (IDF).
This is one reason why we try to use latest released Arduino Cores with Tasmota. In our experience upgrades from one core version to the next one is the most painless way.

[marchingband]

@Jason2866 maybe re-read my bug report?

If I use Arduino-CLI, then any binary produced from any version 2.x.x will be compatible.
If I use PlatformIIO to produce the binaries, they often cause the device to be bricked, if it was initially flashed using Arduino-CLI.

the issue is with PIO.

[joba-1]

I don't see a contradiction between your statements.

You flash partitions with one tool that assumes partitions layout a. Then you flash with another tool that assumes layout b.

That may or may not work depending on the exact settings you use.

Do you want help or blame?

If it is help, better answer questions and provide verbose flash logs, ideally from both tools. If anyone then Jason can spot the problem :)

[marchingband]

@joba-1 I am making a bug report. If I wanted help I would ask in a forum or on Reddit.
If the repo maintainers feel it is not a bug, feel free to close the issue.

[Jason2866]

the issue is with PIO.

okay, so you already identified. Seems you are a step ahead now. It would help to fix the issue to explain and show what goes wrong.

[marchingband]

@Jason2866 I wish I had time for that, but I do not, and wouldn't even know where to begin.
let me know if I can help in ways that are within my means and capacity as a user.

[Jason2866]

@marchingband So we agree we don't know where the issue is coming from. It is complex with to many variables to say who is responsible for.
Since the issue you encounter is not a feature anywhere mentioned it could be closed from a maintainer of Platformio. But that's not the way of the cool Platformio guys. The wanna help.
Same intention why i answered (i am not related to Platformio).

As mentioned earlier, do the builds have same setups? Partition schemes, Flash mode, flash size? All MCU variants (esp32, esp32-s2, esp32-c3 etc.) affected? Build on same system? Windows, Linux or Mac? Version of Arduino CLI?

To make stuff even more complicated. Arduino Core had issues with flash modes until core 2.0.4. Bootloader was built not correctly. Problems will arrive when an OTA update is done with a build which uses a different flash mode than the original was used.

So there are known bugs which affects or prevents a successful OTA update with later versions. The only solution i see for you users, to find (if possible!) a setup with a actual build where OTA is working. Maybe an intermediate build is needed to reach this goal. Since only you have or know the details asked earlier the help we can provide is to discuss the findings you have with you.

[Jason2866]

Tthe answer to this question:

Are binaries meant to be compatible for OTA across versions?

The should, but bug(s) in Arduino/IDF do prevent this.

[marchingband]

@Jason2866 once again:

The most obvious case is:
Flash a board using Arduino-cli with Arduino-ESP32 v2.0.1
Compile a binary using PIO for the same version platform = espressif32@4.1.0
Run my custom OTA code, device gets bricked.

Moving from Arduino CLI to PIO, even using the SAME VERSION of Arduno-ESP32 bricks the device.

Once again I am not looking for help, I have a working solution for my users.
I am making a bug report for the benefit of the repo maintainers.

If you want to help the maintainers, I imagine you can reproduce the issue locally based on the information I provided.
My repo is public if you would like to clone it and test with my custom OTA procedure.

Happy to help you get up and running but I imagine you will be able to with ease.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Please provide more details or it will be closed if no further activity occurs. Thank you for your contributions.