Functional testing

Section author: Marco Buttu

A test is called functional when it aims to verify the component behavior from the user point of view. The components usually have two kind of users:

1. operators (human beings) who interact with the component through the operator input console

2. programs (clients or other components) that interact with the component by using its API (calling its methods)

In the first case, we should verify the behavior of the commands the operator sends to our component. In other words, we we have to test the component command() method. We call this kind of functional tests by the name of command tests.

In the second case, we should verify all the other methods defined in the component IDL interface. We should prove they give us the expected results, and when required they raise the expected exceptions. We give this functional tests the name of API tests.

Writing functional tests in Python is really straightforward, and of course it is easier than C++, so we will use it in this section. The Python standard library provides a unit test framework, called unittest. Unfortunatly the Python version we can use with ACS-8.2 is the old Python 2.5, and in its unittest module there is a lack of useful assertions. The solution is to install a third part module, unittest2, in order to use it in place of the standard library unittest module:

$ easy_install unittest2 # Install the unittest2 library

Let’s have a real example: the DewarPositioner component. We will see both how to write a command and an API test.

Command tests

In this section we will test the derotatorSetup command. As we said before, we have to think from the user (the astronomer in that case) point of view. So, let’s see what the user wants to obtain after executing the command from the operatorInput console:

> derotatorSetup=KKG
>

Well, we are ready to write our functional test. The file and the methods must start with test, so we can create a file called test_derotatorSetup.py inside the functional/commands directory:

# File: test/functional/commands/test_derotatorSetup.py
import unittest2
from Acspy.Clients.SimpleClient import PySimpleClient
from DewarPositioner.DewarPositionerImpl import DewarPositionerImpl


class DerotatorSetupCmdTest(unittest2.TestCase):
    """Test the derotatorSetup and derotatorGetActualSetup commands"""

    def test_proper_setup(self):
        # Get the component
        client = PySimpleClient()
        dp = client.getComponent('RECEIVERS/DewarPositioner')

        # Send the component a command
        success, answer = dp.command('derotatorSetup=KKG')
        # Verify the command is executed as expected
        self.assertEqual(answer, '')

        # Release the component and disconnect the client
        client.releaseComponent('RECEIVERS/DewarPositioner')
        client.disconnect()

if __name__ == '__main__':
    unittest2.main()

Before implementing the code, we write just the component interface without writing the command execution method. We do not write the method implementation because we want to get an expected failure. In that case, we expect the test fails saying the command does not exist.

Important

One of the most important things about testing is to be sure the tests can fail. It is better do not have a test than to have one that does not fails when it should do.

Let’s start the test:

$ python test_derotatorSetup.py

======================================================================
FAIL: test_proper_setup (__main__.DerotatorSetupCmdTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_derotatorSetup.py", line 19, in test_proper_setup
    self.assertEqual(answer, '')
AssertionError: 'Error - command derotatorSetup does not exist' != ''

----------------------------------------------------------------------
Ran 1 test in 1.388s

FAILED (failures=1)

As we can see from the output message, only one test has been run, and it failed as expected, with a straighforward error message. Now we are ready to write the derotatorSetup command code. Let’s write it, and than we run again the test, of course offline. To spin up the test offline we need to simulate the external resources. The DewarPositioner gets a reference to a derotator component, and this one communicates to the hardware. The best approach is to simulate the external resource API, but in our case, because the derotator protocol is a bit complex, we choose to implement a simulator of the ACS derotator component. To get this component ready, we just have to point to the testing CDB:

$ export ACS_CDB=~/Nuraghe/ACS/trunk/SRT/

We can now start ACS and the required containers:

$ acsStartContainer -py DerotatorsContainer
$ acsStartContainer -py DewarPositionerContainer

Let’s spin up the test:

$ python test_derotatorSetup.py
.
----------------------------------------------------------------------
Ran 1 test in 2.825s

OK

If we write a wrong setup code we want the component to behave this way:

> derotatorSetup=GIGIRIVA
Error - setup GIGIRIVA not available"

So, we write an additional test that verifies this case:

# File: test/functional/commands/test_derotatorSetup.py
import unittest2
from Acspy.Clients.SimpleClient import PySimpleClient
from DewarPositioner.DewarPositionerImpl import DewarPositionerImpl


class DerotatorSetupCmdTest(unittest2.TestCase):
    """Test the derotatorSetup and derotatorGetActualSetup commands"""

    def test_proper_setup(self):
        # Get the component
        client = PySimpleClient()
        dp = client.getComponent('RECEIVERS/DewarPositioner')

        # Send the component a command
        success, answer = dp.command('derotatorSetup=KKG')
        # Verify the command is executed as expected
        self.assertEqual(answer, '')

        # Release the component and disconnect the client
        client.releaseComponent('RECEIVERS/DewarPositioner')
        client.disconnect()

    def test_wrong_setup(self):
        # Get the component
        client = PySimpleClient()
        dp = client.getComponent('RECEIVERS/DewarPositioner')

        # Send the component a command
        success, answer = dp.command('derotatorSetup=GIGIRIVA')
        # Verify the answer starts with 'Error'
        self.assertTrue(answer.startswith('Error'))

        # Release the component and disconnect the client
        client.releaseComponent('RECEIVERS/DewarPositioner')
        client.disconnect()

if __name__ == '__main__':
    unittest2.main()

As we can see, we added a new test, called test_wrong_setup(). In that test we command a derotatorSetup with a wrong error code, and we assert that the answer starts with 'Error'. We also notice that our code smells, because there is a lot of duplication among the two tests. The unittest framework provides special methods, called setUp() and tearDown(), that it calls respectively before and after each test. So, we can refactor our test case moving the common code inside this two methods:

# File: test/functional/commands/test_derotatorSetup.py
import unittest2
from Acspy.Clients.SimpleClient import PySimpleClient
from DewarPositioner.DewarPositionerImpl import DewarPositionerImpl


class DerotatorSetupCmdTest(unittest2.TestCase):
    """Test the derotatorSetup and derotatorGetActualSetup commands"""

    def setUp(self):
        self.client = PySimpleClient()
        self.dp = self.client.getComponent('RECEIVERS/DewarPositioner')

    def tearDown(self):
        self.client.releaseComponent('RECEIVERS/DewarPositioner')
        self.client.disconnect()

    def test_proper_setup(self):
        success, answer = self.dp.command('derotatorSetup=KKG')
        self.assertTrue(success)
        self.assertEqual(answer, '')

    def test_wrong_setup(self):
        success, answer = self.dp.command('derotatorSetup=GIGIRIVA')
        self.assertFalse(success)
        self.assertTrue(answer.startswith('Error'))

if __name__ == '__main__':
    unittest2.main()

API tests

In this section we will see how to write an API test. To follow the previous example, we will test the DewarPositioner.setup() method. Let’s start peeking at the IDL interface:

/* Take a configuration code and configure the component
 *
 * This method takes a configuration code, gets the corresponding
 * derotator component reference and initializes the DewarPositioner.
 * For instance, by giving the code KKG, the DewarPositioner gets the
 * KBandDerotator reference and performs its setup. It also sets the
 * rewinding mode and configuration default values as:
 *
 *     setConfiguration('FIXED')
 *     setRewindingMode('AUTO')
 *
 * @param code the setup mode (for instance: LLP, KKG, CCB, ecc.)
 * @throw ComponentErrors::ComponentErrorsEx in case of wrong
 * configuration code or derotator component not available
 */
void setup(in string code) raises (ComponentErrors::ComponentErrorsEx);

We should write a test similar to the previous one. In particular, we want the the setup() to raise a ComponentErrorsEx in case of wrong code. Our test could be the following one:

# File: test/functional/test_setup.py
from __future__ import with_statement
import unittest2
import time

from ComponentErrors import ComponentErrorsEx
from Acspy.Clients.SimpleClient import PySimpleClient


class SetupTest(unittest2.TestCase):
    """Test the DewarPositioner.setup() method"""

    def setUp(self):
        self.client = PySimpleClient()
        self.dp = self.client.getComponent('RECEIVERS/DewarPositioner')

    def tearDown(self):
        self.client.releaseComponent('RECEIVERS/DewarPositioner')
        self.client.disconnect()

    def test_proper_setup(self):
        self.dp.setup('KKG')
        self.assertEqual(self.dp.getActualSetup(), 'KKG')

    def test_wrong_setup(self):
        with self.assertRaises(ComponentErrorsEx):
            self.dp.setup('GIGIRIVA')


if __name__ == '__main__':
    unittest2.main()

Note

The import from __future__ import with_statement is required in Python 2.5 when, as in this case, we use the with statement.