-
Notifications
You must be signed in to change notification settings - Fork 13.3k
Add prebuild script to use additional build options #8095
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add prebuild script to use additional build options #8095
Conversation
Additional compile arguments can be given by a file named "build_opt.h". This enables users to easily extend their arguments on a central place. The script has been taken from stm32duino's Arduino_Core_STM32 package
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like the idea, but there are a few changes needed to get it going. Thanks for the PR!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, LGTM! I was thinking we should add something to the docs about this new option, but I can't find anywhere that it fits in well presently in the doc/*.rst
directory. If you think it might make sense to add a couple lines in the faq.rst
about this, that would be icing on the cake.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I personnally like very much this addition.
It is a long waited feature in Arduino IDE that is still not addressed, I will cite arduino/Arduino#421 and arduino/arduino-cli#846. I also made a proposal that was not dismissed, nor closed nor commented yet.
It's a pity that a global solution is not available. This forces platforms to adopt local solution like this one.
Approving, and thanks for bringing this.
@brainelectronics - This is great! I think, when you create an empty For a while now, I have been doing something similar to this PR. I have noticed that core '.S' files are not rebuilt when, for your case, the
while everything else in core is rebuilt. |
True! |
…r users build options file
…ptions file in faq docs
PR is ready for review again and could be merged if alpha testing has been passed |
I'm using this sketch #ifndef BLAH
#error
#endif and this #define BLAH toto I made a build with
Windows alpha version 0.0.2 contains your PR (thanks to the tag edit: I removed the |
@d-a-v That's strange. As you placed the
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, still. Thanks! Slated for 3.1.0, after the 3.0.x bugfixes are all settled.
The contents of the arduino_buid/sketch directory are unpopulated at the time of failure. Maybe the python is running too soon in the process and needs to be run elsewhere?
|
I'll try to check & fix the issue on the weekend |
…brainelectronics/arduino-esp8266 into feature/allow-custom-build-options-file
I had to do this change to make it work under linux: ( I did not try with windows, but without it, I get the same error as with linux. Then it seems to work well and be used for every single file from core and external libs :) diff --git a/platform.txt b/platform.txt
index 555171d4..551e1a14 100644
--- a/platform.txt
+++ b/platform.txt
@@ -115,13 +115,13 @@ recipe.hooks.linking.prelink.3.pattern="{compiler.path}{compiler.c.cmd}" -CC -E
recipe.hooks.linking.prelink.4.pattern="{compiler.path}{compiler.c.cmd}" -CC -E -P {build.vtable_flags} {build.mmuflags} "{runtime.platform.path}/tools/sdk/ld/eagle.app.v6.common.ld.h" -o "{build.path}/local.eagle.app.v6.common.ld"
## Compile c files
-recipe.c.o.pattern="{compiler.path}{compiler.c.cmd}" {compiler.cpreprocessor.flags} {compiler.c.flags} -D{build.sdk}=1 -DF_CPU={build.f_cpu} {build.lwip_flags} {build.debug_port} {build.debug_level} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} -DARDUINO_BOARD="{build.board}" {build.led} {build.flash_flags} {compiler.c.extra_flags} {build.extra_flags} "@{build.opt.path}" {includes} "{source_file}" -o "{object_file}"
+recipe.c.o.pattern="{compiler.path}{compiler.c.cmd}" {compiler.cpreprocessor.flags} {compiler.c.flags} -D{build.sdk}=1 -DF_CPU={build.f_cpu} {build.lwip_flags} {build.debug_port} {build.debug_level} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} -DARDUINO_BOARD="{build.board}" {build.led} {build.flash_flags} {compiler.c.extra_flags} {build.extra_flags} -include "{build.opt.path}" {includes} "{source_file}" -o "{object_file}"
## Compile c++ files
-recipe.cpp.o.pattern="{compiler.path}{compiler.cpp.cmd}" {compiler.cpreprocessor.flags} {compiler.cpp.flags} -D{build.sdk}=1 -DF_CPU={build.f_cpu} {build.lwip_flags} {build.debug_port} {build.debug_level} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} -DARDUINO_BOARD="{build.board}" {build.led} {build.flash_flags} {compiler.cpp.extra_flags} {build.extra_flags} "@{build.opt.path}" {includes} "{source_file}" -o "{object_file}"
+recipe.cpp.o.pattern="{compiler.path}{compiler.cpp.cmd}" {compiler.cpreprocessor.flags} {compiler.cpp.flags} -D{build.sdk}=1 -DF_CPU={build.f_cpu} {build.lwip_flags} {build.debug_port} {build.debug_level} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} -DARDUINO_BOARD="{build.board}" {build.led} {build.flash_flags} {compiler.cpp.extra_flags} {build.extra_flags} -include "{build.opt.path}" {includes} "{source_file}" -o "{object_file}"
## Compile S files
-recipe.S.o.pattern="{compiler.path}{compiler.c.cmd}" {compiler.cpreprocessor.flags} {compiler.S.flags} -D{build.sdk}=1 -DF_CPU={build.f_cpu} {build.lwip_flags} {build.debug_port} {build.debug_level} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} -DARDUINO_BOARD="{build.board}" {build.led} {build.flash_flags} {compiler.S.extra_flags} {build.extra_flags} "@{build.opt.path}" {includes} "{source_file}" -o "{object_file}"
+recipe.S.o.pattern="{compiler.path}{compiler.c.cmd}" {compiler.cpreprocessor.flags} {compiler.S.flags} -D{build.sdk}=1 -DF_CPU={build.f_cpu} {build.lwip_flags} {build.debug_port} {build.debug_level} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} -DARDUINO_BOARD="{build.board}" {build.led} {build.flash_flags} {compiler.S.extra_flags} {build.extra_flags} -include "{build.opt.path}" {includes} "{source_file}" -o "{object_file}"
## Create archives
recipe.ar.pattern="{compiler.path}{compiler.ar.cmd}" {compiler.ar.flags} {compiler.ar.extra_flags} "{archive_file_path}" "{object_file}" |
I have to explain my last comment.
It was copied to the tmp dir with this content:
And that gave this error message when compiling:
|
I had a try with your PR.
It seems to work locally and this is a great feature. There is a caveat: the project must be cleared and rebuilt when the option file is changed. That was anyway already the case (everything had to be compiled and not only the sketch). Maybe the python script can do something about that. Another caveat is that it becomes incompatible with the STM32 version. Here's my proposal against your current patch. diff --git a/platform.txt b/platform.txt
index 223c2cf5..48d06a7a 100644
--- a/platform.txt
+++ b/platform.txt
@@ -87,7 +87,7 @@ compiler.size.cmd=xtensa-lx106-elf-size
build.extra_flags=-DESP8266
# Pre and post build hooks
-build.opt.name=build_opt.h
+build.opt.name=build_opt
build.opt.path={build.path}/sketch/{build.opt.name}
# Create empty {build.opt} if it does not exist in the output sketch dir
diff --git a/tools/prebuild.py b/tools/prebuild.py
index 3b074ca6..1db9dc20 100644
--- a/tools/prebuild.py
+++ b/tools/prebuild.py
@@ -12,7 +12,7 @@
import os
import sys
-
+from shutil import copyfile
def create_sketch_dir(build_path):
"""
@@ -26,96 +26,21 @@ def create_sketch_dir(build_path):
os.makedirs(sketch_build_path)
-def create_build_options_file(source_path, build_path, build_opt_name="build_opt.h"):
+def create_build_options_file(source_path, build_path, build_opt_name="build_opt"):
"""
- Create the build options file in the build directory.
- The modification time of the build options file is set to the sketch
- modification time in case the file did not exist or the users build options
- modification time to reflect changes which require a recompilation
- :param source_path: The path to the source directory
- :type source_path: str
- :param build_path: The path to the build directory
- :type build_path: str
- :param build_opt_name: The build option file name
- :type build_opt_name: str, optional
+ Copy the build options file to the build directory,
+ or create an empty file there if source does not exist.
+ Creation or modification time does not matter.
"""
build_opt_source_path = os.path.join(source_path, build_opt_name)
build_opt_build_path = os.path.join(build_path, "sketch", build_opt_name)
# check for an existing build options file in the source directory
if os.path.exists(build_opt_source_path):
- # user does have/use an build options file in it's sketch directory
-
- if os.path.exists(build_opt_build_path):
- # set build options file modification time to the same time as the
- # sketch modification time to avoid rebuilding libraries
- set_file_time_to_sketch_time(source_path, build_opt_build_path)
- else:
- # the build options file does not yet exist in the build directory
- # just continue as it will be copied by the IDE automatically on
- # time. This section is only entered on the very first run
- pass
+ copyfile(build_opt_source_path, build_opt_build_path)
else:
- # user does not have a build options file, create empty one to avoid
- # compilation error due to missing file for compilation
- open(build_opt_build_path, 'a').close()
-
- # set build options file modification time to the same time as the
- # sketch modification time to avoid rebuilding libraries
- set_file_time_to_sketch_time(source_path, build_opt_build_path)
-
-
-def set_file_time_to_sketch_time(source_path, destination_path):
- """
- Set the file ctime and mtime to the ctime and mtime of the sketch.
- :param source_path: The source path to the sketch
- :type source_path: str
- :param destination_path: The destination path to the file to modify
- :type destination_path: str
- """
- sketch_path = get_full_sketch_path(source_path)
- sketch_mtime = get_modification_time(sketch_path)
- destination_ctime = get_creation_time(destination_path)
-
- os.utime(destination_path, (destination_ctime, sketch_mtime))
-
-
-def get_full_sketch_path(source_path):
- """
- Get the full sketch path.
- :param source_path: The source path
- :type source_path: str
- :returns: The full sketch path including the '.ino' extension.
- :rtype: str
- """
- base_name = os.path.basename(source_path)
-
- sketch_path = os.path.join(source_path, base_name + '.ino')
-
- return sketch_path
-
-
-def get_modification_time(file_path):
- """
- Get the modification time of a file.
- :param file_path: The file path
- :type file_path: str
- :returns: The modification time.
- :rtype: float
- """
- return os.path.getmtime(file_path)
-
-
-def get_creation_time(file_path):
- """
- Gets the creation time if a file.
- :param file_path: The file path
- :type file_path: str
- :returns: The creation time.
- :rtype: float
- """
- return os.path.getctime(file_path)
-
+ # create empty file
+ open(build_opt_build_path, 'w').close()
def main():
if len(sys.argv) == 3: |
About the
|
Copying the file directly like @d-a-v suggests may be the best way to get around this. Alternatively, would it be possible to just post-process the pre-processed header file to do an equivalent |
Both would work for us. edit |
No not on WSL2, we simply have a busybox.exe for windows provided with the tools 😉 Just FYI, I'm waiting Arduino feedback about this. I would really like they allows some other extension like |
@fpistm |
I did not check your proposal. About breaking change well, I always try to avoid it but sometimes we have no choice. |
One more vote for not using the .h extension. It’s not a real header file, so don’t make users think it is. .opt or something would be very obviously different from the end user perspective. I’d say there is no backward compatibility issue at all because the compiler flags between different architectures will almost always be different… |
Then @brainelectronics @fpistm @earlephilhower, would |
@d-a-v fully agree. Naming the file |
Basically that described issues should have been solved with the timestamp manipulation. At least that was the intension. I'll try to check again next week as I'm currently quite busy with another project |
@d-a-v |
@brainelectronics |
@fpistm 's stm32duino/Arduino_Core_STM32#1442 is merged, |
Never mind, I seem to be out of sync with all the other threads out there, enough so that I am confused as to what they have decided and where it is implemented. In any event, the example I did works; however, it may not be the direction everyone is going. Sorry for the noise. |
Focusing on PR#1442@brainelectronics If you are stuck or have just been too busy, here is my interpretation of the immediate contents of stm32duino/Arduino_Core_STM32#1442 (comment). And, how to resolve some of the shortfalls.
Observations of
An easy solution for a global dependency is to create a proxy .h file and use @d-a-v 's To confirm, that the ideas expressed above work, I updated your script, expanded on the thoughts in PR#1442, and added dependency support. Revised script:#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# prebuild.py — This script manages updates to build.opt in the build directory
# with changes from build_opt.h in the Sketch directory.
#
# Copy build_opt.h from the source sketch folder to build.opt in the build
# sketch folder. If source sketch folder does not have a build_opt.h file,
# create am empty build_opt.h files in the build sketch folder.
#
# For dependencies to work, maintain timestamp on a proxy .h file for build.opt.
#
# Written by brainelectronics, 2021.
#
# Update by M Hightower, 2022.
#
"""
"Aggressively cache compiled core" must be turned off for a reliable build process.
In ~/.arduino15/preferences.txt, to disable the feature:
compiler.cache_core=false
Reference:
https://forum.arduino.cc/t/no-aggressively-cache-compiled-core-in-ide-1-8-15/878954/2
"""
"""
# Updates or Additions for platform.txt or platform.local.txt
runtime.tools.prebuild={runtime.platform.path}/tools/prebuild.py
build.opt.source.path={build.source.path}/build_opt.h
build.opt.path={build.path}/sketch/build.opt
build_opt.proxy.path={build.path}/sketch/PROXY.build_opt.h
recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.prebuild}" "{build.opt.source.path}" "{build.opt.path}" "{build_opt.proxy.path}"
compiler.cpreprocessor.flags=-D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -D_GNU_SOURCE @{build.opt.path} -include "{build_opt.proxy.path}" "-I{compiler.sdk.path}/include" "-I{compiler.sdk.path}/{build.lwip_include}" "-I{compiler.libc.path}/include" "-I{build.path}/core"
"""
import os
import sys
from shutil import copyfile
def main():
if len(sys.argv) == 4:
src_build_opt_path = sys.argv[1]
dst_build_opt_path = sys.argv[2]
dst_build_opt_proxy_path = sys.argv[3]
build_path_sketch, not_used = os.path.split(dst_build_opt_path)
if not os.path.exists(build_path_sketch):
os.makedirs(build_path_sketch)
# Conditionally copy a newer build_opt.h from source directory to the
# build/sketch directory as build.opt. When build_opt.h does not exist
# in the source directory, create an empty build.opt file in the
# build/sketch directory.
if os.path.exists(src_build_opt_path):
if (os.path.exists(dst_build_opt_path)) and \
(os.path.getmtime(dst_build_opt_path) >= os.path.getmtime(src_build_opt_path)):
# only copy newer file - do nothing, all is good
print("Using sketch compiler command-line options in " + dst_build_opt_path)
else:
# The new copy gets stamped with the current time, just as other
# files copied by `arduino-builder`.
copyfile(src_build_opt_path, dst_build_opt_path)
open(dst_build_opt_proxy_path, 'w').close()
print("Updated sketch compiler command-line options in " + dst_build_opt_path)
else: # The Sketch currently does not have a build_opt.h.
if os.path.exists(dst_build_opt_path) and \
os.path.getsize(dst_build_opt_path) == 0:
# All done, the Sketch has not changed.
pass
else:
# Place holder - Create an empty `build.opt` to satisfy the
# `@file` parameter. Create proxy .h to track changes to
# `build.opt`. This will help trigger dependency builds.
open(dst_build_opt_path, 'w').close()
open(dst_build_opt_proxy_path, 'w').close()
else:
print("Wrong argument count")
print(" All arguments FQFN. Source path build_opt.h, Build path build.opt, Build path PROXY.build_opt.h")
sys.exit(1)
if __name__ == '__main__':
sys.exit(main()) Aggressive Caching of
|
closed per merged #8504 |
Additional compile arguments can be given by a file named "build_opt.h" and shall be placed next to the Sketch.ino file.
This enables users to easily extend their arguments on a central place.
The script has been taken from stm32duino's Arduino_Core_STM32 package