Skip to content

Commit 0d42c30

Browse files
ArmRyanArmRyan
authored
Add new scripts to build and run unit tests and check version changes in git commits (ARM-software#137)
* Adds requirements.txt file * Creates a new bash script to download dependencies and build/run unit tests * Creates a new bash script to check Date and Revision changes in git commits Change-Id: I163f98516778bf2bec58dbd0614e91d348437acc Signed-off-by: Ryan O'Shea <[email protected]>
1 parent 6982301 commit 0d42c30

File tree

5 files changed

+356
-20
lines changed

5 files changed

+356
-20
lines changed

‎.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ Tests/UnitTest/TestCases/TestData/*/schema_generated.h
77
Tests/UnitTest/PregeneratedData/*
88
Tests/UnitTest/Output/*
99
Tests/UnitTest/Unity/*
10+
Tests/UnitTest/downloads/*
11+
Tests/UnitTest/build-cortex-m*
1012
Documentation/html/*
1113
Documentation/DoxyGen/src/history.txt
1214
Documentation/DoxyGen/nn.dxy

‎Tests/UnitTest/README.md

Lines changed: 25 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,17 @@ Unit test CMSIS-NN functions on any [Arm Mbed OS](https://os.mbed.com/mbed-os/)
44

55
The [Unity test framework](http://www.throwtheswitch.org/unity) is used for running the actual unit tests.
66

7+
For a quick setup, it is reccomended to the helper script targetting the Arm Corstone-300 softwware. See the section " Using FVP based on Arm Corstone-300 software ".
8+
79
## Requirements
810

11+
The following apt packages are required. Replace python venv version with your python version.
12+
```
13+
sudo apt install ruby-full cmake python3.X-venv pip curl git git-lfs
14+
```
15+
916
Python3 is required.
10-
It has been tested with Python 3.6 and it has been tested on Ubuntu 16, 18 and 20.
17+
It has been tested with Python 3.10 and it has been tested on Ubuntu 22.04.
1118

1219
Make sure to use the latest pip version before starting.
1320
If in a virtual environment just start by upgrading pip.
@@ -16,29 +23,15 @@ If in a virtual environment just start by upgrading pip.
1623
pip install --upgrade pip
1724
```
1825

19-
See below for what pip packages are needed.
20-
21-
### Executing unit tests
22-
23-
If using the script unittest_targets.py for executing unit tests, the following packages are needed.
24-
25-
```
26-
pip install pyserial mbed-ls mbed-cli termcolor mercurial pyelftools==0.29 pyyaml jsonschema jinja2 mbed_host_tests mbed_greentea pycryptodome pyusb cmsis_pack_manager psutil cryptography click cbor
27-
```
28-
29-
Python packages mbed-cli and and mbed-ls are command line tools so it should not matter if those are installed under Python2 or Python3. These packages have been tested for Python2, with the following versions: mbed-ls(1.7.9) and mbed-cli(1.10.1). They have also been tested for Python3, with the following versions: mbed-ls(1.7.12) and mbed-cli(1.10.5). Package mercurial is needed for package mbed-cli.
30-
31-
### Generating new test data
32-
33-
For generating new test data, the following packages are needed.
26+
After upgrading pip, the requirements file found in Tests/UnitTests can be installed. This contains all
27+
python modules required to run all of the scripts. This will install tensorflow and keras to allow the use of
28+
the generate_test_data.py script. If you have version specific requirements, it is reccomended to install this
29+
requirements.txt in a virtual environment.
3430

3531
```
36-
pip install numpy packaging tensorflow tf-keras~=2.16
32+
pip install -r requirements.txt
3733
```
3834

39-
40-
For generating new data, the python3 packages tensorflow, numpy and packaging are required. Most unit tests use a Keras generated model for reference. The SVDF unit test use a json template as input for generating a model. To do so flatc compiler is needed and it requires a schema file.
41-
4235
#### Get flatc and schema
4336

4437
Note this is only needed for generating SVDF unit tests.
@@ -77,6 +70,18 @@ Use the -h flag to get more info.
7770

7871
### Using FVP based on Arm Corstone-300 software
7972

73+
The easiest way to run the unit tests on Corstone-300 is to use the build_and_run_tests.sh script. This script will install required packages, build unit tests and run unit tests. This script has been designed for Linux hosts with both aarch64 and x86_64 architectures. For more help use the '-h' flag on the script.
74+
75+
Sample usage:
76+
```
77+
./build_and_run_tests.sh -c cortex-m3,cortex-m7,cortex-m55 -o '-Ofast'
78+
```
79+
By default the script will download and target gcc. To use arm compiler ensure that arm compilers folder is located in path, export CC and use the -a option on the script.
80+
81+
Downloaded dependencies including python venv can be found in Tests/UnitTests/downloads. Test elfs can be found in Tests/UnitTests/build-($cpu) directories.
82+
83+
Otherwise, you can build it manually:
84+
8085
The build for unit tests differs from the build of CMSIS-NN as a [standalone library](https://github.com/ARM-software/CMSIS-NN/blob/main/README.md#building-cmsis-nn-as-a-library) in that, there is a dependency to [CMSIS](https://github.com/ARM-software/CMSIS_5) project for the startup files from CMSIS-Core. This is specified by the mandatory CMSIS_PATH CMake argument.
8186

8287

‎Tests/UnitTest/build_and_run_tests.sh

Lines changed: 250 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,250 @@
1+
#!/usr/bin/env bash
2+
#
3+
# SPDX-FileCopyrightText: Copyright 2024 Arm Limited and/or its affiliates <[email protected]>
4+
#
5+
# SPDX-License-Identifier: Apache-2.0
6+
#
7+
# Licensed under the Apache License, Version 2.0 (the License); you may
8+
# not use this file except in compliance with the License.
9+
# You may obtain a copy of the License at
10+
#
11+
# www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing, software
14+
# distributed under the License is distributed on an AS IS BASIS, WITHOUT
15+
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
# See the License for the specific language governing permissions and
17+
# limitations under the License.
18+
#
19+
# Version: 1.0
20+
# Date: 2024-05-21
21+
# This bash script downloads unit test dependencies, builds unit tests and also runs the tests.
22+
23+
CPU="cortex-m55"
24+
OPTIMIZATION="-Ofast"
25+
QUIET=0
26+
BUILD=1
27+
RUN=1
28+
SETUP_ENVIRONMENT=1
29+
USE_ARM_COMPILER=0
30+
USE_PYTHON_VENV=1
31+
USE_FVP_FROM_DOWNLOAD=1
32+
USE_GCC_FROM_DOWNLOAD=1
33+
34+
ETHOS_U_CORE_PLATFORM_PATH=""
35+
CMSIS_5_PATH=""
36+
37+
38+
usage="
39+
Helper script to setup, build and run CMSIS-NN unit tests
40+
41+
args:
42+
-h Display this message.
43+
-c Target cpu. Takes multiple arguments as a comma seperated list. eg cortex-m3,cortex-m7,cortex-m55 (default: cortex-m55)
44+
-o Optimization level. (default: '-Ofast')
45+
-q Quiet mode. This reduces the amount of info printed from building and running cmsis-unit tests.
46+
-b Disable CMake build. Only works with previously built targets. Designed to quickly rerun cpu targets.
47+
-r Disable running the unit tests. Designed to test build only or allow user to manually run individual test cases outside of this script.
48+
-e Disable environment setup. This flag will stop the script from attempting to download dependencies. This is just a quiet mode to reduce print outs.
49+
-a Use Arm Compiler that is previously available on machine. Ensure compiler directory is added to path and export CC.
50+
-p Disable the usage of python venv from download directory. Requires dependencies to be install before calling script.
51+
-f Disable the usage of FVP from download directory. Requires FVP to be in path before calling script.
52+
-u Path to ethos-u-core-platform
53+
-g Disable the usage of GCC that is already from download directory. Requires gcc to be in path before calling script.
54+
-C Path to cmsis 5
55+
56+
example usage: $(basename "$0") -c cortex-m3,cortex-m4 -o '-O2' -q
57+
"
58+
59+
while getopts hc:o:qbreapfu:gC: flag
60+
do
61+
case "${flag}" in
62+
h) echo "${usage}";;
63+
c) CPU=${OPTARG};;
64+
o) OPTIMIZATION=${OPTARG};;
65+
q) QUIET=1;;
66+
b) BUILD=0;;
67+
r) RUN=0;;
68+
e) SETUP_ENVIRONMENT=0;;
69+
a) USE_ARM_COMPILER=1;;
70+
p) USE_PYTHON_VENV=0;;
71+
f) USE_FVP_FROM_DOWNLOAD=0;;
72+
u) ETHOS_U_CORE_PLATFORM_PATH="${OPTARG}";;
73+
g) USE_GCC_FROM_DOWNLOAD=0;;
74+
C) CMSIS_5_PATH="${OPTARG}";;
75+
esac
76+
done
77+
78+
Setup_Environment() {
79+
echo "++ Downloading Corstone300"
80+
if [[ -d ${WORKING_DIR}/corstone300_download ]]; then
81+
echo "Corstone300 already installed. If you wish to install a new version, please delete the old folder."
82+
else
83+
if [[ ${UNAME_M} == x86_64 ]]; then
84+
CORSTONE_URL=https://developer.arm.com/-/media/Arm%20Developer%20Community/Downloads/OSS/FVP/Corstone-300/FVP_Corstone_SSE-300_11.24_13_Linux64.tgz
85+
elif [[ ${UNAME_M} == aarch64 ]]; then
86+
CORSTONE_URL=https://developer.arm.com/-/media/Arm%20Developer%20Community/Downloads/OSS/FVP/Corstone-300/FVP_Corstone_SSE-300_11.24_13_Linux64_armv8l.tgz
87+
fi
88+
89+
TEMPFILE=$(mktemp -d)/temp_file
90+
wget ${CORSTONE_URL} -O ${TEMPFILE} >&2
91+
92+
TEMPDIR=$(mktemp -d)
93+
tar -C ${TEMPDIR} -xvzf ${TEMPFILE} >&2
94+
mkdir ${WORKING_DIR}/corstone300_download
95+
${TEMPDIR}/FVP_Corstone_SSE-300.sh --i-agree-to-the-contained-eula --no-interactive -d ${WORKING_DIR}/corstone300_download >&2
96+
fi
97+
98+
echo "++ Downloading GCC"
99+
if [[ -d ${WORKING_DIR}/arm_gcc_download ]]; then
100+
echo "Arm GCC already installed. If you wish to install a new version, please delete the old folder."
101+
else
102+
if [[ ${UNAME_M} == x86_64 ]]; then
103+
GCC_URL="https://developer.arm.com/-/media/Files/downloads/gnu/13.2.rel1/binrel/arm-gnu-toolchain-13.2.rel1-x86_64-arm-none-eabi.tar.xz"
104+
elif [[ ${UNAME_M} == aarch64 ]]; then
105+
GCC_URL="https://developer.arm.com/-/media/Files/downloads/gnu/13.2.rel1/binrel/arm-gnu-toolchain-13.2.rel1-aarch64-arm-none-eabi.tar.xz"
106+
fi
107+
108+
TEMPFILE=$(mktemp -d)/temp_file
109+
wget ${GCC_URL} -O ${TEMPFILE} >&2
110+
mkdir ${WORKING_DIR}/arm_gcc_download
111+
112+
tar -C ${WORKING_DIR}/arm_gcc_download --strip-components=1 -xJf ${TEMPFILE} >&2
113+
fi
114+
115+
echo "++ Cloning CMSIS-5"
116+
if [[ -d ${WORKING_DIR}/CMSIS_5 ]]; then
117+
echo "CMSIS-5 already installed. If you wish to install a new version, please delete the old folder."
118+
else
119+
git clone https://github.com/ARM-software/CMSIS_5.git
120+
fi
121+
122+
echo "++ Cloning Ethos-U core platform"
123+
if [[ -d ${WORKING_DIR}/ethos-u-core-platform ]]; then
124+
echo "Ethos-U core platform already installed. If you wish to install a new version, please delete the old folder."
125+
else
126+
git clone https://review.mlplatform.org/ml/ethos-u/ethos-u-core-platform
127+
fi
128+
129+
echo "++ Setting up python environment"
130+
if [[ -d ${WORKING_DIR}/cmsis_nn_venv ]]; then
131+
echo "Python venv already installed. If you wish to install a new version, please delete the old folder."
132+
else
133+
python3 -m venv cmsis_nn_venv
134+
source cmsis_nn_venv/bin/activate
135+
pip3 install -r ../requirements.txt
136+
deactivate
137+
fi
138+
}
139+
140+
Build_Tests() {
141+
echo "++ Building Tests"
142+
if [[ ${QUIET} -eq 0 ]]; then
143+
cmake -S ./ -B build-${cpu}-${compiler} -DCMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_FILE} -DTARGET_CPU=${cpu} -DCMSIS_PATH=${CMSIS_5_PATH} -DCMSIS_OPTIMIZATION_LEVEL=${OPTIMIZATION}
144+
cmake --build build-${cpu}-${compiler}/
145+
146+
echo "Built successfully into build-${cpu}-${compiler}"
147+
else
148+
cmake_command=$(cmake -S ./ -B build-${cpu}-${compiler} -DCMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_FILE} -DTARGET_CPU=${cpu} -DCMSIS_PATH=${CMSIS_5_PATH} -DCMSIS_OPTIMIZATION_LEVEL=${OPTIMIZATION} 2>&1)
149+
make_command=$(cmake --build build-${cpu}-${compiler}/ 2>&1)
150+
echo "${cmake_command}" > build-${cpu}-${compiler}/cmake_command.txt
151+
echo "${make_command}" > build-${cpu}-${compiler}/make_command.txt
152+
153+
echo "Built successfully into build-${cpu}-${compiler}"
154+
fi
155+
}
156+
157+
Run_Tests() {
158+
echo "++ Running Unit Tests"
159+
readarray -d '' tests < <(find ./build-${cpu}-${compiler}/ -iname "*.elf" -print0)
160+
for test in "${tests[@]}"
161+
do
162+
echo "Test: ${test}"
163+
output=$(FVP_Corstone_SSE-300_Ethos-U55 -C mps3_board.uart0.shutdown_on_eot=1 -C mps3_board.visualisation.disable-visualisation=1 -C mps3_board.telnetterminal0.start_telnet=0 -C mps3_board.uart0.out_file="-" -C mps3_board.uart0.unbuffered_output=1 ${test})
164+
echo "$output" | grep "0 Failures" -vqz
165+
if [[ $? -eq 0 ]]; then
166+
echo "${output}"
167+
echo "${test} failed. Script exiting."
168+
exit 1
169+
elif [[ ${QUIET} -eq 0 ]]; then
170+
echo "${output}"
171+
fi
172+
done
173+
echo "Tests for ${cpu} ran successfully."
174+
}
175+
176+
if [[ ${BUILD} -eq 0 && ${RUN} -eq 0 && ${SETUP_ENVIRONMENT} -eq 0 ]]; then
177+
echo "All script functions are disabled. Script will exit and do nothing."
178+
exit 1
179+
fi
180+
181+
UNAME_M=$(uname -m)
182+
UNAME_S=$(uname -s)
183+
184+
if [[ ${UNAME_S} != Linux ]]; then
185+
echo "Error: This script only supports Linux."
186+
exit 1
187+
fi
188+
189+
mkdir -p downloads
190+
pushd downloads
191+
WORKING_DIR=$(pwd)
192+
193+
if [[ ${SETUP_ENVIRONMENT} -eq 1 ]]; then
194+
echo "++ Setting Environment"
195+
Setup_Environment
196+
fi
197+
198+
if [[ ${USE_PYTHON_VENV} -eq 1 ]]; then
199+
source cmsis_nn_venv/bin/activate
200+
fi
201+
202+
if [[ -z "${ETHOS_U_CORE_PLATFORM_PATH}" ]]; then
203+
ETHOS_U_CORE_PLATFORM_PATH="${WORKING_DIR}/ethos-u-core-platform"
204+
fi
205+
206+
if [[ -z "${CMSIS_5_PATH}" ]]; then
207+
CMSIS_5_PATH="${WORKING_DIR}/CMSIS_5"
208+
fi
209+
210+
popd
211+
IFS=',' read -r -a cpu_array <<< "$CPU"
212+
213+
if [[ ${BUILD} -eq 1 || ${RUN} -eq 1 ]]; then
214+
for cpu in "${cpu_array[@]}"
215+
do
216+
echo "++ Targetting ${cpu}"
217+
if [[ ${USE_ARM_COMPILER} -eq 1 ]]; then
218+
compiler="arm-compiler"
219+
TOOLCHAIN_FILE=${ETHOS_U_CORE_PLATFORM_PATH}/cmake/toolchain/armclang.cmake
220+
else
221+
if [[ ${USE_GCC_FROM_DOWNLOAD} -eq 1 ]]; then
222+
export PATH=${WORKING_DIR}/arm_gcc_download/bin/:${PATH}
223+
fi
224+
compiler="gcc"
225+
TOOLCHAIN_FILE=${ETHOS_U_CORE_PLATFORM_PATH}/cmake/toolchain/arm-none-eabi-gcc.cmake
226+
fi
227+
228+
if [[ $USE_FVP_FROM_DOWNLOAD -eq 1 ]]; then
229+
if [[ ${UNAME_M} == x86_64 ]]; then
230+
export PATH=${WORKING_DIR}/corstone300_download/models/Linux64_GCC-9.3/:${PATH}
231+
elif [[ ${UNAME_M} == aarch64 ]]; then
232+
export PATH=${WORKING_DIR}/corstone300_download/models/Linux64_armv8l_GCC-9.3/:${PATH}
233+
fi
234+
fi
235+
236+
if [[ ${BUILD} -eq 1 ]]; then
237+
Build_Tests
238+
fi
239+
240+
if [[ ${RUN} -eq 1 ]]; then
241+
Run_Tests
242+
fi
243+
done
244+
fi
245+
246+
echo ""
247+
echo "++ Tests for ${CPU} ran successfully"
248+
if [[ ${USE_PYTHON_VENV} -eq 1 ]]; then
249+
deactivate
250+
fi

‎Tests/UnitTest/requirements.txt

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
pyserial
2+
mbed-ls
3+
mbed-cli
4+
termcolor
5+
mercurial
6+
pyelftools == 0.29
7+
pyyaml
8+
jsonschema
9+
jinja2
10+
mbed_host_tests
11+
mbed_greentea
12+
pycryptodome
13+
pyusb
14+
cmsis_pack_manager
15+
psutil
16+
cryptography
17+
click
18+
cbor
19+
numpy
20+
packaging
21+
tensorflow
22+
tf-keras ~= 2.16

0 commit comments

Comments
 (0)