Skip to content

Test Plan Development

Overview

FixturFab's Test Runner Application works by running hardware test code with pytest. We call the pytest scripts used by the fixture to test the Device Under Test (DUT) Test Plans. If you can run your test plan with a simple pytest call, the FixturFab Test Runner can too.

pytest -p f3ts run test_plan.py

We believe this approach makes hardware testing as simple as possible for hardware engineers with limited software experience. Our hope is that all you have to do is write a couple python scripts and we handle all the hard stuff to make the frontend, data analytics, and database management work as seamlessly as possible.

Writing A Test Plan

FixturFab offers an open-source pytest plugin called pytest-f3ts that gives you all the tools you need to display and save your test results within our test application framework.

Our plugin documentation will walk you through how to get started writing hardware unit tests with pytest: https://fixturfab.gitlab.io/software/pytest-f3ts/

Software Development

Before deployment of the test plan software to the fixture computer, we reccommend you test and debug your test plan software within a development environment.

Clone the Repo

git clone https://gitlab.com/fixturfab/software/fixtur-ctrl-functional-tests.git
cd fixtur-ctrl-functional-tests

Install Dependencies

Docker Desktop

An easy way to develop test plans without managing dependency conflicts within your own machine is to run the software within a virtual machine using Docker Desktop. Install Docker Desktop to your machine and deploy your container with the following:

You will want to deploy container images from FixturFab's private container registry. In order to do this, you need to login to the registry before building the images.

gcloud auth configure-docker us-west1-docker.pkg.dev
gcloud auth login
docker-compose up -d --build

Linux

You can also install the software dependencies manually within your linux machine

Setup USB Permissions
chmod +x ./pytest-f3ts/app/udev/setup-udev.sh
sudo ./pytest-f3ts/app/udev/setup-udev.sh
Create the Virtual Environment

Now within the test plan directory, use poetry to install the python dependencies in a virtual environment:

poetry install
poetry shell
Hardware Dependencies

Often there may be a bunch of linux setup steps or repositories that must be installed for the hardware to work. These typically get installed and managed within the pytest container's Dockerfile. If you are having trouble installing any of these dependencies on your development machine, check the pytest container's Dockerfile for help figuring out what you may be missing.

arduino-cli

https://arduino.github.io/arduino-cli/0.35/getting-started/

rp2040 arduino library

https://arduino-pico.readthedocs.io/en/latest/install.html#installing-via-arduino-cli

Balena Deployment

Balena is used to manage the deployment and management of test fixtures in the field.

BalenaOS is installed on the Intel NUC that is integrated into the test system. A set of containers is then created containing the devices environment configuration and the application code. These containers can then be pushed to devices in the field.

Creating a new release

After testing your changes locally, you can create a release candidate by running the following:

balena push <balena slug>

This will push the latest code and start the building process for the container. If the build was successful, you can then roll out the update to devices in the field using the Balena Dashboard.

Pulling Private Base Images

You will want to deploy container images from FixturFab's private container registry. In order to do this, you need to enable balena to authenticate with the private registry during the build, which is done by passing the --registry-secrets option with a path to the authentication secrets. An example is shown below:

For balena push: 

$ balena push <balena slug> --registry-secrets ../registry-secrets.yml

and the registry-secrets.yml file is outside of the code repository and has the following format:

'us-west1-docker.pkg.dev':
    username: '_json_key'
    password: '{
  "type": "service_account",
  "project_id": "buoyant-idea-226013",
  "private_key_id": "454ddbbe3b6b31f50dde337f3fed8025871c5b6e",
  "private_key": "-----BEGIN PRIVATE KEY-----\nA+LONG+PRIVATE+KEY+STRING\nWITH+ESCAPED+LINE+BREAKS+AND\NMORE+THAN+A+THOUSAND+CHARACTERS=\n-----END PRIVATE KEY-----\n",
  "client_email": "container-registry-test@buoyant-idea-226013.iam.gserviceaccount.com",
  "client_id": "107317699110545445744",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/container-registry-test%40buoyant-idea-226013.iam.gserviceaccount.com"
}'

See this example for more information on how to format json keys from google cloud registries.

Generating Documentation

mkdocs is utilized along with mkdocs-material to generate documentation for this project.

The docs directory contains the general structure for the documentation in the form of markdown files. To autogenerate additional documentation from docstrings, the mkdocstrings module is used.

Developing documentation with the live server

When creating additional documentation it's useful to run the mkdocs server which will live-reload the webpages as you make changes. To start this server, run the following in a terminal:

poetry run mkdocs serve

Adding a new file for autodocumentation

To add a new python file to the autodocumentation, create a docs/<your-test-plan>.md file. Add a new header to the file, and then add the line ::: {new_file_name}, this will signal to mkdocstrings to process the new files when building the documentation