ESP32 S3 with a custom partition (IDE 2.3.2)

[stef-ladefense]

Board

ESP32S3 DEV MODULE

Device Description

The board used is an ESP32S3 DEV MODULE.
It is equipped with ESP32-S3-WROOM-1 MCN16R8.
Configured correctly with QIO80MHZ,16MB, QSPI_PSRAM.

Hardware Configuration

yellow card with lcd 170x320 ESP32-1732S019

Version

v3.0.4

IDE Name

Arduino IDE 2.3.2

Operating System

Windows 11 pro

Flash frequency

80 MHZ DIO

PSRAM enabled

yes

Upload speed

921600

Description

I would like some help on using a partitions.csv file.

The first default partition scheme works (4MB with spiffs), but does not use all the onboard memory.

So I have a partitions.csv file that is in my sketch directory.

My question:
If I believe this Using a Custom Partition Scheme
but it does not work in my sketch directory.

I have nothing against modifying the board.txt except that the modifications will be overwritten by the next update

How do I tell the IDE to use my partition file in sketch directory?

thank you

partitions.csv

Sketch

It doesn't matter because the csv file is not used

Debug Message

NA

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[robertlipe]

Looks like your question is more an operational one of your IDE than anything about arduino-esp32 itself. While I don't do Arduino IDE, the good news for you is that you're not the first to be stumped by this. It looks like there's some pointy clicky stuff in the IDE itself to set this:

https://forum.arduino.cc/t/esp32-how-to-change-partition-table-with-ide-2-0/971252/9

[stef-ladefense]

hi,
i post here because in the Expressif Arduino esp32 doc, it is written there:

Using a Custom Partition Scheme
To create your own partition table, you can create the partitions.csv file in the same folder you created your sketch. The build system will automatically pick the partition table file and use it instead of the predefined ones.

But it does not work
problem expressif ?
problem Arduino IDE 2.x ?
or it works only in VScode ?

but yes, it's very complicated when it should be simple

[Xylopyrographer]

@stef-ladefense - Placing a partitions.csv file in the same folder as the sketch does work. However if the partitions file has an error, the default partition table will be used in its place.

See this link for the format:
https://docs.espressif.com/projects/esp-idf/en/release-v5.1/esp32/api-guides/partition-tables.html#partition-tables

The gen_esp32part.py tool can be used to check the .csv file for errors.

[stef-ladefense]

I finally figured it out! Thanks

I copied the following csv into my sketch folder with renaming it to partitions.csv.
"\AppData\Local\Arduino15\packages\esp32\hardware\esp32\3.0.4\variants\esp32_s3r8n16\gen4esp32_4MBapp_4MBota_7MBspiffs.csv"
I had already tested it with the board: 4D Systems gen4-ESP32 16MB Modules (ESP32-S3R8n16).

I configured the IDE to: ESP32S3 Dev Module, QIO 80MHz, QSPI PSRAM, 16MB (128Mb) and Partition Scheme Custom.

Ethoura !

Same when looking at boards.txt that the Partition Scheme Custom had no file registered, in this case it takes the included .csv.

If we could configure the IDE from a file stored in the sketch directory, that would be great, but that is not your task.

Stéphane

[robertlipe]

Cool. It WAS an IDE configuration issue. We sort of knew this since it works for so many of us.

Please consider closing the ticket to help the maintainers keep some clarity on the backlog of 500+ issues.

[aIecxs]

Originally posted by @robertlipe in #10120 (comment)
Looks like your question is more an operational one of your IDE than anything about arduino-esp32 itself.

Well, the Arduino IDE project says different. arduino/arduino-ide/issues/376
I am not sure where to open issue, so I just post here.

Originally posted by @Xylopyrographer in #10120 (comment)
Placing a partitions.csv file in the same folder as the sketch does work. However if the partitions file has an error, the default partition table will be used in its place.

I have checked the partitions.csv and the file is fine. However it does not work anymore (IDE 2.3.x)

Originally posted by @robertlipe in #10120 (comment)
Cool. It WAS an IDE configuration issue. We sort of knew this since it works for so many of us.

It's unclear to me where it breaks now, in Arduino IDE or in ESP32 platform, but this solution doesn't work any longer.

Originally posted by @5chufti in #9572 (comment)
solution:
add a file "partitions.csv" to your sketch directory with following content:

and select "partition scheme = custom" in settings

@supiiik has tracked it down to somewhere between 2.0.11 to 2.0.16.

Originally posted by @supiiik in #9643
Since upgrade from 2.0.11 to 2.0.16 is custom partitions table file partitions.csv located in source code parent directory ignored.

Please note some say selecting "partition scheme = custom" in Arduino IDE settings is necessary. But according to the espressif documentation, it should be sufficient to only place partitions.csv in Sketch directory. The file is correctly copied to %TEMP%/arduino/sketches/*/ dir for compiling, however it is compiled with ARDUINO_PARTITION_default

ARDUINO_FQBN="esp32:esp32:esp32:UploadSpeed=921600,CPUFreq=240,FlashFreq=80,FlashMode=qio,FlashSize=4M,PartitionScheme=default,DebugLevel=none,PSRAM=disabled,LoopCore=1,EventsCore=1,EraseFlash=none,JTAGAdapter=default,ZigbeeMode=default"

[robertlipe]

Pushing back on a closed issue is unlikely to get any attention from developers.

There is often ambiguity between Arduino (an 8-bit AVR SOC) or the boards around those or the company that makes those, the API that was built around those board that has since been used on other boards (in this case, "arduino-esp32"), and the IDE editor made by that company that calls other tools (GCC for code generation, GNU ld for linking, other tools for serial port monitoring, debugging, stack decoding, etc.) that's used to to edit code, send code to boards, etc. It can often be difficult to figure out which "arduino" is being discussed in any context. It's particularly annoying that "arduino-ide" and "arduino-esp32" have similar version numbers at about the same time. Hopefully, as Arduino-ESP32's recent 3.0 popularity gains traction, they'll gain more recognizeable identities of their own.

Custom partitions on ESP32 using ESP-IDF (thus not using Arduino ESP32) and Platformio (using platform = either ESP-IDF as before or platform = Arduino ESP32 and thus definitely using Arduino ESP32) definitely work. There is a large number of projects using them; I work with a couple different ones per week, so there are no widespread reports of catastrophic failure as reported here. I won't say dumb things like that; there have never been any combinations of partition tables that work on any given ESP32 chip/external flash combination, but they are definitely not totally broken as reported in the OP. That leaves the only definition of "arduino" still in play as the IDE or something used by the ide.

As a test, use the IDE to write the partition table to the chip.

If you can read the table with parttool (https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/partition-tables.html) and get back anything other than what is in your CSV, the IDE didn't write it correctly, and the bugreport should go to the arduino-ide project.

If the partition is written to the board correctly and some aspect isn't working as intended (e.g., unaligned addresses become overlapping partitions, causing corruption of stored data, such as filesystems), it's probably an arduino-esp32 issue or maybe an esp-idf issue. There may be border disputes, such as the IDE allowing an invalid table to be written, which then trips the code up, but the assignees on such issues will be able to identify those and work together. (This would point to an option in or in the call of gen_esp32part.py , for example.)

If changing ONLY the IDE and not the arduino-esp32 installation makes the issue come and go, it's clearly an IDE issue and should be raised there.

The Arduino-ESP32 code runs entirely on the ESP32; it has no concept of c:\anything—that's all on your host computer, so that's the IDE. arduino-esp32 reads the partition table that may have been written by the arduino via a pointy clicky draggy droppy operation, but it doesn't know if that was written by MacOS, Windows, FreeBSD, or a custom system. It's all just a data table for that code.

No arguments that some of the error text and default states in ALL of this software could be improved. That's industry-wide. For example:
E (496) esp_core_dump_f�f6�: No core dump partition found!
isn't really an error. Having no core dump partition is a legit configuration; you "just" can't stick a computer that's already crashed and rebooted in the debugger (something most people don't even know is possible) because there is... no core dump partition. Legit, but it LOOKS like an error.

So, yes, it can be difficult to figure out where to assign an issue. Different parts are made by different companies and different teams. Here, as the issue was "solved" by changing the directory (on the host computer) where the partition table is stored, it's an issue with the code running on the host computer, which in this case is the Arduino IDE. Unfortunately, the trackers have become so overrun by operational issues that they turn into personal tech support and just overwhelm the maintainers.

[aIecxs]

Originally posted by @me-no-dev in #9741 (comment)
it does not ignore it, it just has no way of knowing the max size of firmware. Pick a partition scheme with similar or larger size app partition (like Huge APP for example)

Excuse my ignorance, but what is the purpose of having custom partitions.csv in sketch directory, when it is required to pick any different partition scheme from Arduino IDE config anyway?

that undermines the actual purpose of custom partitions.csv in sketch. you could just drop this broken feature from documentation and rely on adding custom partition scheme placed in variants with modified boards.txt only.

The documentation is misleading, it suggests that partitions.csv from sketch is picked automatically and is overriding default partition scheme config.

[me-no-dev]

the purpose is to be able to have custom partition in your sketch folder. The IDE itself has no way to read what's in that partition scheme and this has nothing to do with us. You can ignore this feature and only use things in boards.txt

[zingzog]

I know that this issue is closed but I think that I have a solution for the original issue as I have just had to fix this in my own system.

It would seem that once upon a time all that you needed to do to use your own partitions.csv file was to put it in the same folder as the sketch. This doesn't work in ESP32 3.1.0 but the documentation still says that it should.

Some boards (e.g. "ESP32S3 Dev Module") have a partition option "custom" in the Tools>Partition scheme menu. I used this and put my partitions.csv in my sketch folder and it worked.

I am now using a bigger version of the same module, "ESP32S3 Dev Module Octal (WROOM2)", and this doesn't have the "Custom" partition option. My partitions.csv is in the sketch folder but it was not used.

The "Custom" partition option is missing in the ESP32 boards.txt file.

I edited the boards.txt file at "C:\Users\Me\AppData\Local\Arduino15\packages\esp32\hardware\esp32\3.1.0"

In the section starting with my module name: esp32s3-octal.name=ESP32S3 Dev Module Octal (WROOM2)

at the end of the block of lines starting: "esp32s3-octal.menu.PartitionScheme."

I added:

esp32s3-octal.menu.PartitionScheme.custom=Custom
esp32s3-octal.menu.PartitionScheme.custom.build.partitions=
esp32s3-octal.menu.PartitionScheme.custom.upload.maximum_size=33554432

Where maximum_size=33554432 is 32MB.

It all works now.

I think that this issue is caused by a combination of two things:

1. The mechanism for using the partitions.csv override described in the ESP32 documentation has been changed needlessly but the documentation has not been updated.
2. Whoever maintains the boards.txt file has not included a "Custom" section for some boards. Without the Custom entry partition schemes cannot be easily modified by the user because of the changed functionality.

Andy

[zingzog]

I forgot to say, in tmy previous post, if you make changes to boards.txt, you need to use "Tools>Reload Board Data" to make the changes active.

Andy

[stef-ladefense]

hello,
i am surprised that it does not work for you, because it works for me, as i wrote on august 8.

the problem was the csv file which contained a mistake, corrected since.
the csv just needs to be in the ino directory.
choose partition custom and that's it
partitions.csv

[zingzog]

Hi, in the midst of the discussion I seem to have missed that you had found a fix.

There is still the issue of the misleading documentation that does not say that you need to select the "Custom'' Partition Scheme.

Also, the ESP32 3.1.0 version boards.txt file is missing the "Custom'' Partition Scheme in the entry for the ESP32-S3-DevKitC-1-N32R8V board (the one that I am trying to use!!).

I looked through the boards file, but I couldn't find any other boards missing the "Custom" scheme information.

[stef-ladefense]

no, the doc is ok