Testing

Continuous Integration (CI) Testing

In every pull request (PR), the ci.yml GitHub (GH) workflow runs the following tests:

• Linting - We run:

  • Several Python code formatters and linters: black, flake8 and isort).
  • The PowerShell script scripts\test\lint.ps1, which runs PSScriptAnalyzer, a static code checker for PowerShell.
  • The Python script scripts\test\lint.py which checks that some of our coding conventions are followed.

• Package installation - To ensure packages install successfully we run the PowerShell script scripts/test/test_install.ps1 which uses choco install to install all modified packages.

The tests need to succeed to merge the PR. Once the PR is merged, the ci.yml GitHub workflow automatically publishes the packages to the MyGet vm-packages packages feed. This means they're immediately available to be installed in everyone's VM.

If a package breaks between the CI run and the merge of the PR (for example if the installer hashes changes), all the packages in the PR won't be pushed to MyGet. To prevent this issue, ensure that the CI has run recently, specially if the PR includes many package changes. When you modify a big number of packages (> 20), you may want to consider to split the PR in several smaller PRs, so that if there are issues, less packages are affected.

To detect issues publishing packages to MyGet, we run the compare_myget.yml GH workflow daily to generate the MyGet Version Mismatches Wiki page. We also test the package installation daily using the daily.yml GH workflow, that generates the Daily Failures Wiki page.

Local testing

When adding or modifying Python code, you must run the Python code formatters and linters (black, flake8 and isort) locally using the same version and with the same options as in the CI (ci.yml). When addition or modifying PowerShell code, you must run scripts\test\lint.ps1 locally.

When adding or modifying a package (and when reviewing a package addition/modification), it is important to test the changes locally in Windows 10. Our test suite runs several linters and tests that the package is installed, but it doesn't start the tool to ensure it works correctly. Uninstalling is best-effort and we are not testing uninstalling in our our CI, please test it also locally.

The best way to test packages is to install the installer.vm package with few or no packages in a fresh Windows VM. An easy way to achieve this is to perform a minimal installation of FLARE-VM.

Minimal FLARE-VM installation

Follow the following steps to perform a minimal FLARE-VM installation (with very few packages) that you can use for testing.

1. Prepare a Windows 10 VM and disable Windows Updates, Tamper Protection and Windows Defender as explained in the Pre-installation section of the FLARE-VM README. Take a snapshot named BUILD-READY.

2. Download the FLARE-VM configuration file and modify it to remove all <package /> entries except <package name="hashmyfiles.vm"/> and <package name="pma-labs.vm"/>. hashmyfiles.vm installs HashMyFiles, a small utility useful to calculate SHA256 hashes (needed to create some packages). pma-labs.vm downloads binaries that can be useful for testing. These two packages do not install any other tools or dependencies that could affect testing, while are useful to ensure the installer works properly and for development and testing other tools.

   <packages>
       <package name="hashmyfiles.vm"/>
       <package name="pma-labs.vm"/>
   </packages>
   

3. Execute the FLARE-VM PowerShell installation script in an admin PowerShell console using the modified config file with only the packages hashmyfiles.vm and pma-labs.vm as explained in the FLARE-VM installation section of the FLARE-VM README: .\install.ps1 -customConfig <config.xml> -password <password> -noWait -noGui

   

4. After the FLARE-VM installation finishes (which may take some time), copy the test_install.ps1 script to the Desktop and create a packages folder in the Desktop. Open an admin PowerShell console, set the current working location to the Desktop and write (but do not execute): ./test_install.ps1.

   

5. Take a testing snapshot.

Every time you want to test a package or packages, restore the testing snapshot, copy the package/s to the packages folder in the Desktop and run the test_install.ps1 script (by just pressing Enter in the PowerShell console). After the installation completes, ensure the installed tools work as expected. When applicable, test also that the package uninstalls correctly. The test_uninstall.ps1 script may be useful to test uninstallation.

If you need to test the same package/s again, restore to the testing snapshot even if the first installation failed to ensure no artifacts remain that could affect the testing.

Notes

Using a shared folder to copy the packages over

You can store the packages to test in a shared folder and create a link to it in the Desktop to speed the testing process up.

1. Create a testing shared folder. In VirtualBox, you need to create the folder on the host and add it to the Shared Folders in the VM settings. The shared folder will appear in the VBOXSVR virtual network server.

   

2. Create a packages folder inside the testing shared folder.

   

3. Copy the packages folder and paste it as a shortcut in the Desktop. Rename it to packages.

   

4. [Optional] You may want to create a shortcut in the Desktop to the testing shared folder for faster access as well.

5. Take a testing snapshot.

Now you can copy directly packages to test to the packages shared folder, restore to the testing snapshot, and test the packages running the test_install.ps1 script (by just pressing Enter in the PowerShell console).

Testing hashmyfiles.vm or pma-labs.vm

If you need to test hashmyfiles.vm or pma-labs.vm, you would need to perform a minimal installation with different simple packages or without any packages.

Testing tools with dependencies

When testing packages that require extra tools or depend on packages that take long to install, it is recommended that you install all the requirements first and take a snapshot before installing the package that you want to test. This way you can revert to this snapshot if you need to adapt and test the package several times without having to install the requirements again.

To install a package or packages, run choco install <package_name_or_names>.

Examples:

Testing IDA related packages

Before testing IDA related packages, such as IDA plugins:

1. Install libraries.python3.vm to install Python 3 and several useful libraries and idafree.vm to install IDA free, which includes some improvements that works with IDA Pro as well (such as the Open with IDA right click shortcut):

   choco install libraries.python3.vm idafree.vm
   

2. Install IDA Pro (using your installer) which will find the installed Python.

Testing Node tools installed with npm

Install nodejs.vm: choco install nodejs.vm.

Automated packages updates

Our bot sends a PR weekly on Wednesdays (approximately at 22:21 UTC) to update packages and fix broken hashes executing the update_packages.yml GH workflow. This workflow can also be triggered manually by clicking Run workflow:

If the bot's PR hasn't been merged, the bot overwrites it when the update_packages.yml GH workflow is triggered again. If you review the PR, ensure it is merged before the bot overwrites it, as you most likely will have to review most of the changes again. It is recommended to trigger the update manually before start the review to ensure you review the latest updates, if the bot run was a few days ago, especially if you know an important tool update has happened since then. The workflow normally runs fast: between 10 and 60 minutes.

It is recommended to review the automated package update PR in a weekly or bi-weekly basis, as this way the number of packages to test is smaller and the review faster. This also ensures we always have the latest version of the tools!

Testing package updates

To review a PR with several package updates, you need to test every of the packages in the PR individually in a clean minimal Windows 10 FLARE-VM installation as described in the local testing section. Ensure the tool is installed and it executes as expected by:

• Checking the shortcut in the Tools directory starts the tool without issues, or that the tool file(s) are downloaded correctly in case no shortcut is created.
• Quickly (no more than 2 minutes) testing the tool functionality. This is important for packages in the FLARE-VM default configuration, but optional for other packages if you are not familiar with the tool. Examples:

  • Wireshark: Start the tool and start capturing packages. Ensure some packages have been captured (if needed generating some traffic).

    

  • capa: execute in a cmd console capa C:\Windows\System32\HOSTNAME.EXE and check the output includes query environment variable.

    

  • CyberChef: Start CyberChef from the Tools directory and test a simple recipe.

    

As the PR may contain a lot of packages, to speed the process up, you can test up to 5 different packages at the same time (running the script script scripts/test/test_install.ps1 once for all of them) instead of completely independently. In this case, try to select packages that are unrelated (for example a Node and a Python tool are unlikely to require the same dependencies).

Note we do not audit or review the tools, just verify that they appear to work correctly.

What to do if the tool testing fails?

If testing a tool fails (for example if the tool does not run correctly):

• In a clean minimal FLARE-VM installation, install the current version of the package from the MyGet feedback with choco install <package_name>.vm.
• If the issue is present in the current version, you can continue to update the package. Ensure there is an issue (or create one) to track the problem and write a comment in the package update PR saying that you encountered the issue but is unrelated to the update.
• If the issue is NOT present in the current version, you need to remove the commit of the concrete package update from the PR or fix the package (for example adding a missing dependency) in order to merge the rest of the updates. If you remove the commit, ensure there is an issue (or create one) to track the problem and write a comment in the PR explaining it. To remove the commit, check out the PR locally and remove the commit (for example using git rebase -i <NUMBER_OF_COMMITS>).