Unit testing with domains

 Daniil E. Barvitsky, Argos Group, LLC 

Introduction

The Unit Testing stands for software testing of pre-build quality assurance. It runs the particular use cases (either positive or negative) against the product's core APIs. Nowdays, unit testing has become a part of the software development process, and is close to debugging. In plain words, every debugging scenario can be a unit test, if implemented properly. The unit tests are mostly automatic, i.e. they can run without any participation on the part of the developer.

At the same time, developers often use the source code generators or other code-producing or modeling tools, which build pieces of the product's core logics according to some formal definition. Sometimes the generated code is guaranteed to work and does not need any testing. Hoever, in common cases, generated source code needs to be tested because of integration with human-coded components.

This article discusses some techniques for generating unit test source code for object models. It assumes that the developer operates with some formerly defined object model, which includes: the entities, their attributes, constraints and relationships. The environment, where the model runs, is assumed to support at least level-1 transactions (the environment is concurrent, transactions may lead to dirty updates, unattended deadlocks are possible).

Types of unit tests for object model

Depending on the model's implementation specifics, the following test types can be offered:

  • attribute coverage: the simplest, but an important type of unit test checks the attributes of the models for access and persistence (i.e. get/set plus load/save operations);
  • relationships coverage checks the releationship operations (i.e. link create relationship of concrete type between two or more objects, unlink remove relationship of concrete type for two or more objects, query check if two or more objects have a relationship of concrete type, read get objects, which have relationship(s) of concrete type with given set of objects);
  • persistence coverage checks that parts of the object model can be saved and loaded without loosing structure;
  • messaging coverage for XML-services or other messaging services, checks typical operations to be available from messaging interface;
  • transactions coverage testing all above assuming transactional behavior (i.e. ability to start transaction, roll it back without affecting the state or commit it with affecting the state);
  • concurrency coverage testing all above in concurrent environment (i.e. several threads running similar scenarios against one data source or service instance).

The first four types can be seamlessly (more or less) used with domain testing. The other two types of testing can be implemented using simpler tests from first group. The messaging coverage seems to be the most competitive problem of the first group, because it has dependence on messaging protocol, which can be rather sophisticated.

Domains

The domain is a combination of three sets of values. The first set is the median  part of the domain, and contains typical values. The second set is the extreme  part of the domain, which contains boundary values. The third set is the invalid part of the domain, which defines unacceptable values. The idea of using domains is:

  • assume that we have an operation, which depends on several parameters;
  • every parameter has its own domain;
  • tests probe various combinations of values from domains as input parameters.

The concrete set of values the for domain is the test configuration . If the test configuration uses a value from an invalid part of at least one domain, the test is negative . If the test configuration uses a value from extreme part of at least one domain, the test is extreme or load . Otherwise the test is median . If an operation is invalid by itself, the test is always negative , regardless of parameters values.

There are two problems in domains testing technique: defining a domain and combining parameters values to achieve optimal coverage . The first problem cannot be automated; it comes from the subject area and a-priori knowledge about the model. Sometimes we are able to define domains using some knowledge about the database (null/non-null/required/etc); however these are very rare occasions. Therefore domain definitions should be done explicitly by the tester or come from predefined terms of the subject area (name, zip, address).

The second problem can be semi-automated. The tester can define a certain amount of tests per operation and the desired balance between median, extreme and invalid tests. This problem does not appear in the orthogonal case (i.e. every attribute is tested separately), but, usually orthogonal cases produce a significant amount of tests. Orthogonal domains testing is often the first step and good for debugging. Composite testing, which mixes variations of several parameters, is good for smoke and acceptance tests, because it helps to achieve greater coverage with fewer scenarios.

Attribute coverage

Attributes can be tested with domains technique. The simplest way to test attributes is to make a-priori domains depending on attribute types and then extend domains with extra values. As you have the domains, you can try to perform orthogonal testing by generating separate tests for every value of domain. Normally such test fixtures include several hundreds of tests. Such testing assumes that attribute values do not depend on each other and can be tested separately. This technique fails if objects can not exist w/o defining initial values for some attributes. The acceptable workaround is to set values for required parameters from median parts of their domains. However, this does not guarantee correct testing if parameters have dependencies on each other. For instance, object may contain flags, state information and so on. For such cases (normally they are rare) it is acceptable to define ensembles pre-defined sets of parameter values to be probed. Number of ensembles can vary depending on coverage capacity. Ensembles should be defined manually.

Relationships coverage

For relationships we have four generic operations: creating, deleting, checking[1]and querying[2]. We also have three types of the relationships: association, composition and aggregation, which correspond to three types of objects life-time limitation. For relationships poles (ends) we have types of cardinality, which define how many objects can appear on the poles of relationship.

Domains technique is not applicable to relationships; however, there is a trick which allows us to do it. Assume that we have a set of named objects; each object has attributes defined. We can associate several of that objects by the relationship of a certain type.

Therefore we can define the relationship as a string obj1, obj2, obj3<->obj4, obj5, obj6. These strings can be used to build a relationship domain . In certain cases, the relationship domain can be created using the knowledge about the relationship type. For instance, relationship type and cardinality may help to define the median part and invalid part of the relationship domain. The extreme part can be generated for most cases.

At the same time we can generate the objects (with the limitations, described in previous section). There are some complicated scenarios, like relationship constraints or derived attributes, which require us to use ensembles of the relationship configurations. Similar to attributes, these ensembles should be manually defined by the tester.

 

Persistence coverage

The specifics of testing persistence are:

  • we need to perform non-orthogonal tests, because we need a piece of model persisted;
  • we need to check the stored data.

Typically persistence can be tested by create-save-load-check or load-save-load-check , w/o checking the persisted data. For an XML and SQL database, checking persisted data requires the use of OS-independent data source access and mapping, which can be a competitive task for the tests generator, especially if we take into accout transactional logics and concurrency. Therefore it is suggested to reduce save-check-load-check scenarios to the simplest cases, and use the objects comparison feature to compare pieces of the model.

If we use create-save-load-check , then we are able to operate the domains technique to make the model before saving. Of course, this has limitations, but for comprehensive cases it should be fine.

 Load-save-load-check is simpler to implement, but will require developers to manually create datasets.

Concurrency and transactions coverage

The way which concurrency may be tested depends on transaction level, which the object model supports. I can recollect the following classification:

  • level 0:  no transactions  no deadlocks possible, dirty updates are possible
  • level 1:  single object locks  get objects copy, update it and save changes to object, saving changes happens in critical section. No deadlocks possible. Dirty updates are possible in two cases overriding the changes or when updating several objects;
  • level 2:  check in/check out put a lock on the object, when getting its copy. No dirty updates are possible. Deadlocks are possible;
  • level 3: unmanaged transactions check if an object is locked before updating. If the object is already locked, fail current transaction and roll back. No deadlocks possible, no dirty updates are possible. However, cannot guarantee that semantically correct transaction will be committed successfully;
  • level 4: managed transactions every object has a queue of transactions, attempting to perform locking of that object. Transaction manager attempts to rearrange transactions from different threads in queues to avoid deadlocks and dirty updates. No deadlocks are possible (normally), for most implementations there is a possibility of dirty updates, however, the possibility is very low.

Transactions can be tested in two modes the concurrent mode and single mode. Single mode tests transactions for the ability to commit and roll back. We can use attribute, relationships and persistence scenarios for that. The common scenario is update-check-rollback-check- update-commit-check , which includes the following steps:

    1.     start transaction, run a scenario;

    2.     roll it back, check that data is in initial sate;

    3.     run scenario again;

    4.     commit it;

    5.     check results.

Single mode does not depend on the transactions model level; everything above level 0 is acceptable. Concurrent mode is much more difficult. And it depends on the transactions model level, which the object model uses. The common idea is to run many scalar tests against shared objects from multiple threads. There are two common techniques for concurrent mode testing. The first is heterogeneous . It assumes that we have several scenarios and a piece of a model. Normally it can be performed as follows:

  • using domains, create a piece of the model;
  • run different updates (attribute, relationships, persistence tests from the list above) against that piece of the model from several threads;
  • wait for a completion;
  • check that all tests passed OK.

This method is close to reality, but, usually contains randomization inside. Therefore a tests verdict may change on different computers and even from run to run. To avoid this, scalar tests should be thoroughly casted to have acceptable concurrency. For instance, we may have N tests, changingdifferentattributes of an object. This is OK for heterogeneous testing. If we have two tests, updatingsameattribute and checking it, we may get dirty updates or deadlocks, which will evidently fail the test, even if it is initially correct. Yet another example is adding and removing concrete objects from a list or a set. This is good for heterogeneous testing too.

The second technique is synchronized tests . It assumes that you have two sequences of steps, with synchronization marks. You run both sequences against shared objects. If one of the sequences reaches the synchronization mark, it waits for the other sequence to reach the same mark. Therefore the test looks like synchronized iterating through concurrent sections. Unlike heterogeneous testing, such tests should normally run similar, regardless to the systems performance. Note, that every step in the sequence should be checked after the synchronization mark, because checking is concurrent too.

 

Messaging coverage

Two-way messaging is close to persistence testing. In this case we have two instances of application: one performs standard tests, the second handles requests. This technique is applicable if custom format messaging has generated APIs for both client and server.

If there is API only for one side, custom format messaging tests is mostly semi-automated (i.e. developers should supply requests manually).

The language

Almost all object model unit tests depend on some test data, which normally should be provided by the developer. Defining attribute domains, relationship domains, pieces of models etc. requires a handy presentation to be easy to read and write. XML seems to provide enough flexibility, but is significantly redundant and is not comfortable to use, because domains operate with sets. The config-like style language is preferable, because it is compact enough to read and write without special software. In Argos, we have a standard, which defines a language of domain presentation.

Conclusion

The domains testing requires tests to be parameterized. *Unit, however, does not provide parameterized test support. The parameterization for *unit tests is usually done inside the fixtures, assuming a single test as probing a configuration.

It is important to notice, that domain testing is good for static structure testing. Reactive features testing is done in the other way, which will be covered in further articles.


[1]Checking out if objects A and B participate in a certain type of relationships.

[2]Finding objects, which have relationships of type R with object A.