In the end the build time is always almost the same, but it's a pain to
change one header and have it check the world. CI continues to use the
check so we can get any mistake before merging.
The original proposal was to add this functionality to the --upload option, but
it turns out that --upload seems something slightly different from what the
functionality added with patch does.
The issue motivating this patch is at
https://github.com/ArduPilot/ardupilot/issues/6283
Command for a local installation under BLD/install, where BLD is the build
variant. This will be useful for commands that depends on the install command.
That makes more sense, since that module is supposed to have basic Ardupilot
specific features. The root wscript should be a mere user of those features.
That way we avoid recompiling source files that don't depend on vehicles.
Change notes:
- UTILITY_SOURCE_EXTS: moved to ap_library
- Task generators indexes: ap_library handles that now and in a better way
- Use of AP_STLIB_FEATURES: it doesn't make sense anymore, since the stlib
taskgen has empty source list
- Flags and defines: passed down to ap_library through AP_LIBRARIES_OBJECTS_KW
- Set use='mavlink' in AP_LIBRARIES_OBJECTS_KW instead of calls to
bld.ap_stlib().
Otherwise, a call like `waf configure --board <foo> <command1>` will make
`<command1>` run on wrong variant if there was a different variant configured
previously, or an empty variant if the project is "distcleaned".
That makes it easier see what callbacks are or not being added and the order
they are executed. While at it, don't add build summary callback when in a
check command.
This allows to completely disable the tests. Even the gtest submodule is
not checked out allowing integration on build servers without needing to
download one more submodule.
This is different from the --notests flag. The latter only disables
executing the tests, but always use the submodule.
Waf's autoconfig feature works only for wscripts loaded by the recurse()
function. This patch extends that functionality to all loaded modules, which
includes Waf tools.
This is also needed while compiling libgtest because it's using our
cmath wrapper header. Otherwise it will end up entering the
"ifndef WAF_BUILD" part and fail to compile on gcc 5.3.1.
Setting the "help" keyword argument may not fit in one line sometimes. This
patch adds the following convention to calls to add_option() in order to
address that issue in a better way:
1) The "help" keyword must always be the last argument to be passed.
2) If the help string is a literal string or a literal string with some
operation (e.g. "%" operator) and setting the "help" keyword in the code
doesn't fit a line (considering the limit of characters in a line), then
the help string must be a triple-quoted string. That has the advantage of
not having to have several "+" operations for long help strings. In that
case, the help message must start on the next line and the closing
triple-quotes must be on a separate line together with the closing
parenthesis.
The requirement (1) makes it easier to make the style exception in (2)
acceptable.
The path for the "non-board" environment cache file was hard-coded, which was
causing the error below when the build was configured with the --out option
value different from the default path.
Traceback (most recent call last):
File "/home/gjsousa/ardupilot/modules/waf/waflib/Scripting.py", line 161, in waf_entry_point
run_commands()
File "/home/gjsousa/ardupilot/modules/waf/waflib/Scripting.py", line 263, in run_commands
ctx = run_command(cmd_name)
File "/home/gjsousa/ardupilot/modules/waf/waflib/Scripting.py", line 247, in run_command
ctx.execute()
File "/home/gjsousa/ardupilot/modules/waf/waflib/Scripting.py", line 598, in execute
return execute_method(self)
File "/home/gjsousa/ardupilot/modules/waf/waflib/Build.py", line 250, in execute
self.execute_build()
File "/home/gjsousa/ardupilot/modules/waf/waflib/Build.py", line 263, in execute_build
self.recurse([self.run_dir])
File "/home/gjsousa/ardupilot/modules/waf/waflib/Context.py", line 304, in recurse
user_function(self)
File "/home/gjsousa/ardupilot/wscript", line 246, in build
config_header = Utils.h_file(bld.bldnode.make_node('ap_config.h').abspath())
File "/home/gjsousa/ardupilot/modules/waf/waflib/Utils.py", line 208, in h_file
f = open(fname, 'rb')
IOError: [Errno 2] No such file or directory: '/tmp/build/ap_config.h'
In several places we use functions like asprintf() that are only
available in libc when _GNU_SOURCE is defined. Always define _GNU_SOURCE
since we always want to use them.
This fixes the build on Windows using cygwin.
The usual name for this header is config.h, but that's already used by
vehicles. Using uppercase could give the impression this is a
file to be modified, but it's not. Use lowercase instead.
Use program group name instead of task generator name. That way all programs
belonging to a product are built. That will be the case for different frames
for copter.
Without this patch, if nodes that don't yet exist in the filesystem in
bld.bldnode were created before calls to this function, they'll be removed from
the tree in memory because "*" is passed to ant_glob() and the "remove" keyword
argument defaults to True.
In the future, gbenchmark will be build using a cmake waf tool, which will
require cmake to be installed. We add this option so that users without cmake
can continue use the build system with no problems.
That allows options being declared where they're really used. Additionally, we
load ardupilotwaf after the other tools so that we can create our groups after
all non-ardupilot option groups are created. That makes our groups appear as
the last ones in the help message, which makes it easier to locate them.
That is useful when the user wants to build with one or more submodules at a
specific revision. In case of a developer, that is useful for when a submodule
is in modification process and she needs to (constantly) test the build.
With this change, there's no need to verify if the submodule is initialized,
because the submodules in GIT_SUBMODULES are automatically initialized and
updated if necessary before the build tasks are performed.
One downside of this change is that Google Benchmark configuration is now done
only during build. However that is minor, since now there are easy ways to
separately build different targets and program groups, so that a fail in
benchmark build doesn't really affect the other targets.
We define three build groups and set post_mode as POST_LAZY, which makes one
group be posted only after the previous one. That enables us to correctly
define dynamic sources with submodules and generators that may or may not
depend on submodules (e.g. mavgen - this one depends on mavlink submodule).
The groups are defined below, in that order:
1. git_submodules: contains taskgens that initialize and update submodules.
2. dynamic_sources: contains taskgens that generate sources used by the build.
3. build: contains the normal taskgens.
That makes the code cleaner and better readable. Futhermore, it's easier to add
a pattern for recursion.
Additionally, this commit remove Tools from the exclusion list as that is not
necessary.
Rationale:
1. That function creates a separate build context class instead of just
creating a wrapper for calling build (previous approach).
2. The check command isn't a build shortcut since there's no way of calling it
without using 'check' explicitly.
Now that the concept of programs group is used and 'bin' is the default program
group, the command check needed to be adapted so that (i) it is consistent with
the other build command and (ii) it defines 'all' as its target programs group.
That will make it possible to replace calls of the form
`ardupilotwaf.<method>(bld, <keyword-args...>)` with
`bld.<method>(<keyword-args...>)` in the wscripts.
Advantages of that approach:
- there is no need to import ardupilotwaf in every single wscript
- it follows the same standard used by c and cxx tools (like bld.program,
bld.stlib etc)
- semantically, ap_program, ap_stlib, example etc are all build related
methods, so it makes sense to bind them to the build context
- from the wscripts' perspective, the code is cleaner, since ardupilotwaf,
which is not specific to just build contexts, isn't *explictly* used
That makes it possible to build tests on distributions that don't provide
official packages for gtest. For example, the travis environment is Ubuntu,
which doesn't have official package for gtest library.
waf's terminology might be a bit confusing regarding the word 'target'. As an
attribute for a task generator, it means the paths of the files supposed to be
built. As a command line option (--target), it means the list of names of the
task generators to be used in the build.
Before this commit, only vehicles programs had their task generators' target
parameter value different from the name parameter. Now, there's no distinction
between those two parameters for the case of programs.
The option target for the waf task generator is meant for actual files that are
going to be build. Since our target files are dynamic, let's change the code to
use a new parameter output_dir instead.
The task generator for the mavlink headers is given a name 'mavlink' and is
added as a dependency to all local static libraries, because all of them use
sources that include mavlink headers. It would be nice to have a good way to
detect sources that use mavlink headers or to declare list of libraries that
use the headers, so that the dependency could be added automatically.
Additionally, there's a minor change in syntax for the task generator: using
strings directly instead of lists.
Several boards require cross compilation. The objective of this change is to
allow to specify the target architecture in the board definition, so that the
correct toolchain is selected. The cross compilation mechanism will be
implemented in the next commits.
The TODOs removed with this patch were already fixed. Below is the explanation
for each.
- TODO: add support for unit tests.
- Supported already added.
- TODO: Check if we should simply use the signed 'waf' "binary" (after
verifying it) instead of generating it ourselves from the sources.
- We're using a submodule for waf.
- TODO: evaluate if we need shortcut commands for the common targets
(vehicles). currently using waf --targets=NAME the target name must contain
the board extension so make it less convenient, maybe hook to support
automatic filling this extension?
- There's no need of adding the extension anymore.
- TODO: Once HAL patches get in, need to filter out the HAL based on the
bld.env.BOARD.
- The board-specific HAL library folders is indicated in
bld.env.AP_LIBRARIES.
Instead of a dictionary of dictionaries, have a dictionary of
ConfigSets. Using ConfigSet have two benefits: (1) allow easily copying
values from other, (2) have syntax for specifying the keys directly as
attributes.
With this change now it's easier to specify minlure without
repetition. New boards can override a value, append or prepend depending
on the need.
DEFINES attribute is treated as a dictionary instead of a list, so
that's easier to override values (at expense of ordering). When reading
the board environment, the code converts back to a list.
The board configuration is now stored in a separate file, there's also a
function to get the boards names.
The command will be used to allow travis script to know which boards are
already supported by the waf build system. This is useful while the transition
to the new build system is on going. The command can be removed when the
transition is done and it's not used anywhere besides travis script.
It was implemented in such a way that gtest is required only if the user wants
to build and run tests. Initially we're considering all tests should be gtests.
We can change that assumption in the future if necessary.
We're currently using the tests standard error for reporting tests. We can add
TAP later to integrate with other tools.
Additionally, this patch simplifies the exclude patterns passed to
collect_dirs_to_recurse.
Lucas De Marchi
Andrew Tridgell
Lucas De Marchi
Gustavo Sousa
Eugene Shamaev
Jacob Walser
Gustavo Jose de Sousa
Ricardo de Almeida Gonzaga
Siddharth Bharat Purohit
Caio Marcelo de Oliveira Filho