random-mirrors/smaeul-u-boot
1
0
Fork 0
mirror of https://github.com/smaeul/u-boot.git synced 2025-10-14 21:06:01 +01:00
Code Issues Packages Projects Releases Wiki Activity

doc: move test/py/README.md to HTML documentation

Browse Source 
Convert test/py/README.md to restructured text and add it to the generated
HTML documentation.

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Reviewed-by: Simon Glass <sjg@chromium.org>
This commit is contained in:
Heinrich Schuchardt 2021-01-18 20:24:03 +01:00 committed by Tom Rini
parent ca6583d4e0
commit 0157619d5c
2 changed files with 142 additions and 104 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
doc/develop/index.rst
View File

@ -30,3 +30,4 @@ Testing
coccinelle coccinelle
testing testing
py_testing

243
test/py/README.md → doc/develop/py_testing.rst
View File

@ -1,6 +1,8 @@
# U-Boot pytest suite U-Boot pytest suite
===================
## Introduction Introduction
------------
This tool aims to test U-Boot by executing U-Boot shell commands using the This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach to the console interface. A single top-level script exists to execute or attach to the
@ -14,40 +16,42 @@ results. Advantages of this approach are:
flexible than writing it all in C. flexible than writing it all in C.
- It is reasonably simple to interact with U-Boot in this way. - It is reasonably simple to interact with U-Boot in this way.
## Requirements Requirements
------------
The test suite is implemented using pytest. Interaction with the U-Boot console The test suite is implemented using pytest. Interaction with the U-Boot console
involves executing some binary and interacting with its stdin/stdout. You will involves executing some binary and interacting with its stdin/stdout. You will
need to implement various "hook" scripts that are called by the test suite at need to implement various "hook" scripts that are called by the test suite at
the appropriate time. the appropriate time.
In order to run the testsuite at a minimum we require that both python3 and In order to run the test suite at a minimum we require that both Python 3 and
pip for python3 be installed. All of the required python modules are pip for Python 3 are installed. All of the required python modules are
described in the requirements.txt file in this directory and can be installed described in the requirements.txt file in the /test/py/ directory and can be
with the command ```pip install -r requirements.txt``` installed via the command
.. code-block:: bash
pip install -r requirements.txt
In order to execute certain tests on their supported platforms other tools In order to execute certain tests on their supported platforms other tools
will be required. The following is an incomplete list: will be required. The following is an incomplete list:
| Package | * gdisk
| -------------- | * dfu-util
| gdisk | * dtc
| dfu-util | * openssl
| dtc | * sudo OR guestmount
| openssl | * e2fsprogs
| sudo OR guestmount | * util-linux
| e2fsprogs | * coreutils
| util-linux | * dosfstools
| coreutils | * efitools
| dosfstools | * mount
| efitools | * mtools
| mount | * sbsigntool
| mtools | * udisks2
| sbsigntool |
| udisks2 |
Please use the appropriate commands for your distribution to match these tools
Please use the apporirate commands for your distribution to match these tools
up with the package that provides them. up with the package that provides them.
The test script supports either: The test script supports either:
@ -58,28 +62,31 @@ The test script supports either:
physical board, attach to the board's console stream, and reset the board. physical board, attach to the board's console stream, and reset the board.
Further details are described later. Further details are described later.
### Using `virtualenv` to provide requirements Using `virtualenv` to provide requirements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The recommended way to run the test suite, in order to ensure reproducibility The recommended way to run the test suite, in order to ensure reproducibility
is to use `virtualenv` to set up the necessary environment. This can be done is to use `virtualenv` to set up the necessary environment. This can be done
via the following commands: via the following commands:
```bash
$ cd /path/to/u-boot
$ sudo apt-get install python3 python3-virtualenv
$ virtualenv -p /usr/bin/python3 venv
$ . ./venv/bin/activate
$ pip install -r test/py/requirements.txt
```
## Testing sandbox .. code-block:: console
To run the testsuite on the sandbox port (U-Boot built as a native user-space $ cd /path/to/u-boot
$ sudo apt-get install python3 python3-virtualenv
$ virtualenv -p /usr/bin/python3 venv
$ . ./venv/bin/activate
$ pip install -r test/py/requirements.txt
Testing sandbox
---------------
To run the test suite on the sandbox port (U-Boot built as a native user-space
application), simply execute: application), simply execute:
``` .. code-block:: bash
./test/py/test.py --bd sandbox --build
``` ./test/py/test.py --bd sandbox --build
The `--bd` option tells the test suite which board type is being tested. This The `--bd` option tells the test suite which board type is being tested. This
lets the test suite know which features the board has, and hence exactly what lets the test suite know which features the board has, and hence exactly what
@ -95,7 +102,8 @@ will be written to `${build_dir}/test-log.html`. This is best viewed in a web
browser, but may be read directly as plain text, perhaps with the aid of the browser, but may be read directly as plain text, perhaps with the aid of the
`html2text` utility. `html2text` utility.
### Testing under a debugger Testing under a debugger
~~~~~~~~~~~~~~~~~~~~~~~~
If you need to run sandbox under a debugger, you may pass the command-line If you need to run sandbox under a debugger, you may pass the command-line
option `--gdbserver COMM`. This causes two things to happens: option `--gdbserver COMM`. This causes two things to happens:
@ -111,19 +119,21 @@ option `--gdbserver COMM`. This causes two things to happens:
A usage example is: A usage example is:
Window 1: Window 1:
```shell
./test/py/test.py --bd sandbox --gdbserver localhost:1234 .. code-block:: bash
```
./test/py/test.py --bd sandbox --gdbserver localhost:1234
Window 2: Window 2:
```shell
gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234' .. code-block:: bash
```
gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
Alternatively, you could leave off the `-ex` option and type the command Alternatively, you could leave off the `-ex` option and type the command
manually into gdb once it starts. manually into gdb once it starts.
You can use any debugger you wish, so long as it speaks the gdb remote You can use any debugger you wish, as long as it speaks the gdb remote
protocol, or any graphical wrapper around gdb. protocol, or any graphical wrapper around gdb.
Some tests deliberately cause the sandbox process to exit, e.g. to test the Some tests deliberately cause the sandbox process to exit, e.g. to test the
@ -132,30 +142,39 @@ to attach the debugger to the new sandbox instance. If these tests are not
relevant to your debugging session, you can skip them using pytest's -k relevant to your debugging session, you can skip them using pytest's -k
command-line option; see the next section. command-line option; see the next section.
## Command-line options Command-line options
--------------------
- `--board-type`, `--bd`, `-B` set the type of the board to be tested. For --board-type, --bd, -B
example, `sandbox` or `seaboard`. set the type of the board to be tested. For example, `sandbox` or `seaboard`.
- `--board-identity`, `--id` set the identity of the board to be tested.
This allows differentiation between multiple instances of the same type of --board-identity`, --id
physical board that are attached to the same host machine. This parameter is sets the identity of the board to be tested. This allows differentiation
not interpreted by the test script in any way, but rather is simply passed between multiple instances of the same type of physical board that are
to the hook scripts described below, and may be used in any site-specific attached to the same host machine. This parameter is not interpreted by th
way deemed necessary. test script in any way, but rather is simply passed to the hook scripts
- `--build` indicates that the test script should compile U-Boot itself described below, and may be used in any site-specific way deemed necessary.
before running the tests. If using this option, make sure that any
environment variables required by the build process are already set, such as --build
`$CROSS_COMPILE`. indicates that the test script should compile U-Boot itself before running
- `--buildman` indicates that `--build` should use buildman to build U-Boot. the tests. If using this option, make sure that any environment variables
There is no need to set $CROSS_COMPILE` in this case since buildman handles required by the build process are already set, such as `$CROSS_COMPILE`.
it.
- `--build-dir` sets the directory containing the compiled U-Boot binaries. --buildman
If omitted, this is `${source_dir}/build-${board_type}`. indicates that `--build` should use buildman to build U-Boot. There is no need
- `--result-dir` sets the directory to write results, such as log files, to set $CROSS_COMPILE` in this case since buildman handles it.
into. If omitted, the build directory is used.
- `--persistent-data-dir` sets the directory used to store persistent test --build-dir
data. This is test data that may be re-used across test runs, such as file- sets the directory containing the compiled U-Boot binaries. If omitted, this
system images. is `${source_dir}/build-${board_type}`.
--result-dir
sets the directory to write results, such as log files, into.
If omitted, the build directory is used.
--persistent-data-dir
sets the directory used to store persistent test data. This is test data that
may be re-used across test runs, such as file-system images.
`pytest` also implements a number of its own command-line options. Commonly used `pytest` also implements a number of its own command-line options. Commonly used
options are mentioned below. Please see `pytest` documentation for complete options are mentioned below. Please see `pytest` documentation for complete
@ -163,21 +182,26 @@ details. Execute `py.test --version` for a brief summary. Note that U-Boot's
test.py script passes all command-line arguments directly to `pytest` for test.py script passes all command-line arguments directly to `pytest` for
processing. processing.
- `-k` selects which tests to run. The default is to run all known tests. This -k
selects which tests to run. The default is to run all known tests. This
option takes a single argument which is used to filter test names. Simple option takes a single argument which is used to filter test names. Simple
logical operators are supported. For example: logical operators are supported. For example:
- `'ums'` runs only tests with "ums" in their name.
- `'ut_dm'` runs only tests with "ut_dm" in their name. Note that in this - `'-k ums'` runs only tests with "ums" in their name.
- `'-k ut_dm'` runs only tests with "ut_dm" in their name. Note that in this
case, "ut_dm" is a parameter to a test rather than the test name. The full case, "ut_dm" is a parameter to a test rather than the test name. The full
test name is e.g. "test_ut[ut_dm_leak]". test name is e.g. "test_ut[ut_dm_leak]".
- `'not reset'` runs everything except tests with "reset" in their name. - `'-k not reset'` runs everything except tests with "reset" in their name.
- `'ut or hush'` runs only tests with "ut" or "hush" in their name. - `'-k ut or hush'` runs only tests with "ut" or "hush" in their name.
- `'not (ut or hush)'` runs everything except tests with "ut" or "hush" in - `'-k not (ut or hush)'` runs everything except tests with "ut" or "hush" in
their name. their name.
- `-s` prevents pytest from hiding a test's stdout. This allows you to see
-s
prevents pytest from hiding a test's stdout. This allows you to see
U-Boot's console log in real time on pytest's stdout. U-Boot's console log in real time on pytest's stdout.
## Testing real hardware Testing real hardware
---------------------
The tools and techniques used to interact with real hardware will vary The tools and techniques used to interact with real hardware will vary
radically between different host and target systems, and the whims of the user. radically between different host and target systems, and the whims of the user.
@ -187,9 +211,11 @@ via `$PATH`. These scripts implement certain actions on behalf of the test
suite. This keeps the test suite simple and isolated from system variances suite. This keeps the test suite simple and isolated from system variances
unrelated to U-Boot features. unrelated to U-Boot features.
### Hook scripts Hook scripts
~~~~~~~~~~~~
#### Environment variables Environment variables
'''''''''''''''''''''
The following environment variables are set when running hook scripts: The following environment variables are set when running hook scripts:
@ -202,16 +228,18 @@ The following environment variables are set when running hook scripts:
- `UBOOT_RESULT_DIR` the test result directory. - `UBOOT_RESULT_DIR` the test result directory.
- `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory. - `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory.
#### `u-boot-test-console` u-boot-test-console
'''''''''''''''''''
This script provides access to the U-Boot console. The script's stdin/stdout This script provides access to the U-Boot console. The script's stdin/stdout
should be connected to the board's console. This process should continue to run should be connected to the board's console. This process should continue to run
indefinitely, until killed. The test suite will run this script in parallel indefinitely, until killed. The test suite will run this script in parallel
with all other hooks. with all other hooks.
This script may be implemented e.g. by exec()ing `cu`, `kermit`, `conmux`, etc. This script may be implemented e.g. by executing `cu`, `kermit`, `conmux`, etc.
via exec().
If you are able to run U-Boot under a hardware simulator such as qemu, then If you are able to run U-Boot under a hardware simulator such as QEMU, then
you would likely spawn that simulator from this script. However, note that you would likely spawn that simulator from this script. However, note that
`u-boot-test-reset` may be called multiple times per test script run, and must `u-boot-test-reset` may be called multiple times per test script run, and must
cause U-Boot to start execution from scratch each time. Hopefully your cause U-Boot to start execution from scratch each time. Hopefully your
@ -219,10 +247,11 @@ simulator includes a virtual reset button! If not, you can launch the
simulator from `u-boot-test-reset` instead, while arranging for this console simulator from `u-boot-test-reset` instead, while arranging for this console
process to always communicate with the current simulator instance. process to always communicate with the current simulator instance.
#### `u-boot-test-flash` u-boot-test-flash
'''''''''''''''''
Prior to running the test suite against a board, some arrangement must be made Prior to running the test suite against a board, some arrangement must be made
so that the board executes the particular U-Boot binary to be tested. Often, so that the board executes the particular U-Boot binary to be tested. Often
this involves writing the U-Boot binary to the board's flash ROM. The test this involves writing the U-Boot binary to the board's flash ROM. The test
suite calls this hook script for that purpose. suite calls this hook script for that purpose.
@ -248,7 +277,8 @@ hook script appropriately.
This script will typically be implemented by calling out to some SoC- or This script will typically be implemented by calling out to some SoC- or
board-specific vendor flashing utility. board-specific vendor flashing utility.
#### `u-boot-test-reset` u-boot-test-reset
'''''''''''''''''
Whenever the test suite needs to reset the target board, this script is Whenever the test suite needs to reset the target board, this script is
executed. This is guaranteed to happen at least once, prior to executing the executed. This is guaranteed to happen at least once, prior to executing the
@ -261,20 +291,22 @@ relay or electronic switch attached to the board's reset signal.
The semantics of this script require that when it is executed, U-Boot will The semantics of this script require that when it is executed, U-Boot will
start running from scratch. If the U-Boot binary to be tested has been written start running from scratch. If the U-Boot binary to be tested has been written
to flash, pulsing the board's reset signal is likely all this script need do. to flash, pulsing the board's reset signal is likely all this script needs to
However, in some scenarios, this script may perform other actions. For do. However, in some scenarios, this script may perform other actions. For
example, it may call out to some SoC- or board-specific vendor utility in order example, it may call out to some SoC- or board-specific vendor utility in order
to download the U-Boot binary directly into RAM and execute it. This would to download the U-Boot binary directly into RAM and execute it. This would
avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus
saving wear on the flash chip(s). saving wear on the flash chip(s).
#### Examples Examples
''''''''
https://github.com/swarren/uboot-test-hooks contains some working example hook https://github.com/swarren/uboot-test-hooks contains some working example hook
scripts, and may be useful as a reference when implementing hook scripts for scripts, and may be useful as a reference when implementing hook scripts for
your platform. These scripts are not considered part of U-Boot itself. your platform. These scripts are not considered part of U-Boot itself.
### Board-type-specific configuration Board-type-specific configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Each board has a different configuration and behaviour. Many of these Each board has a different configuration and behaviour. Many of these
differences can be automatically detected by parsing the `.config` file in the differences can be automatically detected by parsing the `.config` file in the
@ -286,7 +318,8 @@ defined in these modules is available for use by any test function. The data
contained in these scripts must be purely derived from U-Boot source code. contained in these scripts must be purely derived from U-Boot source code.
Hence, these configuration files are part of the U-Boot source tree too. Hence, these configuration files are part of the U-Boot source tree too.
### Execution environment configuration Execution environment configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Each user's hardware setup may enable testing different subsets of the features Each user's hardware setup may enable testing different subsets of the features
implemented by a particular board's configuration of U-Boot. For example, a implemented by a particular board's configuration of U-Boot. For example, a
@ -304,14 +337,16 @@ U-Boot source tree, and should be installed outside of the source tree. Users
should set `$PYTHONPATH` prior to running the test script to allow these should set `$PYTHONPATH` prior to running the test script to allow these
modules to be loaded. modules to be loaded.
### Board module parameter usage Board module parameter usage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The test scripts rely on the following variables being defined by the board The test scripts rely on the following variables being defined by the board
module: module:
- None at present. - none at present
### U-Boot `.config` feature usage U-Boot `.config` feature usage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The test scripts rely on various U-Boot `.config` features, either directly in The test scripts rely on various U-Boot `.config` features, either directly in
order to test those features, or indirectly in order to query information from order to test those features, or indirectly in order to query information from
@ -328,7 +363,8 @@ instances of:
- `@pytest.mark.buildconfigspec(...` - `@pytest.mark.buildconfigspec(...`
- `@pytest.mark.notbuildconfigspec(...` - `@pytest.mark.notbuildconfigspec(...`
### Complete invocation example Complete invocation example
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and
any required environment configuration Python modules into $HOME/ubtest/py, any required environment configuration Python modules into $HOME/ubtest/py,
@ -336,32 +372,33 @@ then you would likely invoke the test script as follows:
If U-Boot has already been built: If U-Boot has already been built:
```bash .. code-block:: bash
PATH=$HOME/ubtest/bin:$PATH \
PATH=$HOME/ubtest/bin:$PATH \
PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
./test/py/test.py --bd seaboard ./test/py/test.py --bd seaboard
```
If you want the test script to compile U-Boot for you too, then you likely If you want the test script to compile U-Boot for you too, then you likely
need to set `$CROSS_COMPILE` to allow this, and invoke the test script as need to set `$CROSS_COMPILE` to allow this, and invoke the test script as
follows: follows:
```bash .. code-block:: bash
CROSS_COMPILE=arm-none-eabi- \
CROSS_COMPILE=arm-none-eabi- \
PATH=$HOME/ubtest/bin:$PATH \ PATH=$HOME/ubtest/bin:$PATH \
PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
./test/py/test.py --bd seaboard --build ./test/py/test.py --bd seaboard --build
```
or, using buildman to handle it: or, using buildman to handle it:
```bash .. code-block:: bash
PATH=$HOME/ubtest/bin:$PATH \ PATH=$HOME/ubtest/bin:$PATH \
PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
./test/py/test.py --bd seaboard --build --buildman ./test/py/test.py --bd seaboard --build --buildman
```
## Writing tests Writing tests
-------------
Please refer to the pytest documentation for details of writing pytest tests. Please refer to the pytest documentation for details of writing pytest tests.
Details specific to the U-Boot test suite are described below. Details specific to the U-Boot test suite are described below.