Updater.cpp support for encrypted flash

[robertpoll]

Background

The current implementation of Update() uses the spi_flash_* api to write and read from flash. These functions ignore the partition->encrypted flag and always write raw data to flash even if the partition is marked as encrypted.

Changes in this PR

1. Update() now uses the esp_partition_* api.
2. Wrapper functions for esp_partition_* added to ESP.cpp. This was done to maintain a consistent approach to the way the spi_flash_* functions were used. I note though that not all of the esp-idf functions are used are wrapped, for example esp_ota_get_next_update_partition() so it may be that these should not be added?
3. The current implementation of Update() changes the first (magic) byte of firmware to 0xFF on write, and then when the firmware is completely written changes it back to ESP_IMAGE_HEADER_MAGIC. This works without erasing the sector because flash bits can be changed from 1->0 (but not 0->1). If the flash is encrypted then the actual data written to flash will not be all ones, so this approach will not work. In addition, encrypted flash must be written in 16 byte blocks. So, instead of changing the first byte the changed code stashes the first 16 bytes, and starts writing at the 17th byte, leaving the first 16 bytes as 0xFF. Then, in _enablePartition() the stashed bytes can be successfully written.

Benefits

1. Whilst it's not possible to use encrypted flash directly from either the Arduino IDE or PIO it's reasonably straightforward to compile and flash a bootloader with the necessary support from a simple esp-idf project and then use ArduinoOTA for subsequent updates. This PR enables the use of this workflow until such time as encrypted flash is supported, and is a first (small) step toward adding support.
2. Regardless of the above, the esp_partition_* api is recommended over the api_flash_* api.

Application code should mostly use these esp_partition_* API functions instead of lower level spi_flash_* API functions. Partition table API functions do bounds checking and calculate correct offsets in flash, based on data stored in a partition table.

Questions

1. IDF-4.0 makes the flash APIs non-atomic in some circumstances. How will this be handled in the Arduino core, and does this affect this PR?

[atanisoft]

@robertpoll which flash APIs are non-atomic in IDF v4?

[robertpoll]

@atanisoft I haven't looked in detail, but the idf4 docs say:

Flash APIs after IDF v4.0 are no longer atomic. A writing operation during another on-going read operation, on the overlapped flash address, may cause the return data from the read operation to be partly same as before, and partly updated as new written.

There's a Config option CONFIG_SPI_FLASH_USE_LEGACY_IMPL to use the old implementations and I'm sure I read that writes to encrypted partitions are still atomic, but can't find the reference now.

[jingramwright]

I'm wondering if you might be able to shed some light on a question I've got.

Do you know if this PR update will also work for pre-encrypted binaries?

I'm looking to use the Arduino Update libraries to write a pre-encrypted binary to encrypted flash. The Espress-If docs say to use esp_flash_write_encrypted(...) for this purpose rather than esp_partition_write(...). When I dig deeper the former calls spi_flash_write_encrypted(...), the same as esp_partition_write(...), and so I can't see what the purpose of what is outlined in the documentation is?

[robertpoll]

Do you know if this PR update will also work for pre-encrypted binaries?

Short answer is no unfortunately. Pre-encrypted binaries are tied to a specific memory location, and with OTA you don't know which of the two partitions will be used. If you really need to do it it is possible to disable the address specific part of the encryption/decryption process using FLASH_CRYPT_CONFIG. I haven't tried it but from the documentation it looks like it might work, but I assume the address based key 'tweaking' is there to increase security so to turn it off might be counter-productive. If the intent is to protect the firmware which it is on-the-wire the an SSL connection is likely a better option.

[robertpoll]

The Espress-If docs say to use esp_flash_write_encrypted(...) for this purpose rather than esp_partition_write(...). When I dig deeper the former calls spi_flash_write_encrypted(...), the same as esp_partition_write(...), and so I can't see what the purpose of what is outlined in the documentation is?

The docs recommend using the esp_partition_* calls as they 1) do some bounds checking and 2) check whether the partition is encrypted and pick either the encrypted or unencrypted esp_flash_* function.

Application code should mostly use these esp_partition_* API functions instead of lower level spi_flash_* API functions. Partition table API functions do bounds checking and calculate correct offsets in flash, based on data stored in a partition table.

[jingramwright]

Ah okay, I see.

The ESP32 I'm working with will be receiving the binary via a BLE connection with a mobile app. I wanted to pre-encrypt it so the original binary isn't accessible to the user.

I'll have to do a bit more research.

Cheers for your help!

[sigi77777]

So basically, we can do OTA update which should be NOT encrypted, and when it is placed in the partition, it will get encrypted by esp32 if FLASH_CRYPT_CNT is set to odd number?
Am i getting this right,

[robertpoll]

Yes, that's it.

[robertpoll]

@me-no-dev Thanks!