Atheist is a simple framework for running test cases. You write small files in Python language using a set of predefined functions and classes. In many senses, the concept is quite similar to make or the SCons framework although Atheist is not a building system at all.
Features:
ToDo:
The Test object is the minimal testing unit. Each Test instance defines an individual execution (a shell command) that may be checked for success upon termination. The Test constructor accepts many parameters that may change the test’s exception behavior in several ways. The only mandatory parameter is cmd which is the command to execute.
The Test object is responsible for executing the command and checking its termination value. A very basic example:
Test('true')
Test instances need to be defined in text source files (with .test extension). Although these files are written in a subset of the Python language, they may be seen as declarative programs. You tell Atheist what you want to test and even the order, but the decision about when to run the corresponding action is taken by Atheist; some of them may be never done.
The file does not define sequential imperative sentences. For example, if you write this in a .test file:
Test('ls /')
print "hello"
the print statement will be executed when the test is LOADED and the ls command will run later, when the test is EXECUTED. You must take in mind that the atheist .test file defines a set of tests. It is not a conventional python program.
Any Test constructor accepts the next key-val parameters. All of them are optional. Beside the parameter’s name appear its type and default value.
check – type: bool, default: True
If ‘check’ is False there is no consequences when the Task fails and it is not considered in the final stats.
cwd – type: str
Directory for the task execution.
delay – type: int, default: 0
Waits for ‘delay’ seconds before launching the task actions.
desc – type: str
One-liner textual task description.
detach – type: bool, default: False
When detach is False the next task does not start until the current one ends. When detach is True the next task is executed even if the current one is still running.
env – type: str:str map
A dictionary with shell environment variables.
expected – type: int
Expected return code for the command. It may be negative if the process is killed by a signal.
tid – type: str
It is a unique string Task IDentifier. You can get a reference to the task later by giving this value to the get_task() function.
must_fail – type: bool, default: False
When you expect the program to end with an error but the return code is not known (i.e: is different from zero). You should check other conditions (stdout, stderr, generated files, etc) to differentiate alternative fails.
parent – type: CompositeTask
Use this to aggregate the current test to an already defined CompositeTask.
template – type: Template list
A list of templates. See Templates.
timeout – type: int, default: 5
The maximum task execution time (in seconds). When the timeout finishes, Atheist sends the programmed signal to the process. To avoid timed termination (daemonic task) give timeout=0.
save_stderr – type: bool, default: False
Store the process’ stderr in a text file. If the stderr parameter is not set, Atheist will create an unique name for the file.
save_stdout – type: bool, default: False
Store the process’ stdout in a text file. If the stdout parameter is not set, Atheist will create an unique name for the file.
shell – type: bool, default: False
Execute the command within a shell session. bash is the used shell.
signal – type: int, default: SIGKILL
It is the signal that Atheist sends to the process when the timeout finishes.
stdout – type: str
It is the name of the file where to save the process’ stdout. Setting this parameters implies save_stdout = True.
todo – type: bool, default: False
It indicates that the task is not fully verified and it is possible that it fail unduly. This has no effect when the task ends successfully.
Not all of these key-args are available for all Task classes. See Task’s, Test’s, Commands, Daemons....
The execution of any task returns a value, which can be:
The template is a set of predefined values for Test key-values. You may use the same configuration for many tests avoiding to repeat them. This is an example:
t1 = Template(timeout=6, expected=-9)
Test('foo', templates=[t1])
Test('bar', templates=[t1])
Both tests will be automatically killed after 6 seconds and the expected return value is -9. This means that these processes receive the SIGKILL(9) signal. You may specify several templates as a list.
Conditions are predicates (actually, functors) that check for specific conditions. Conditions may be specified to be checked before (pre-conditions) or after (post-conditions) the task execution. If any of the conditions fail, then the task fails. This is an example:
t = Test('foo')
t.pre += FileExists('/path/to/foofile')
t.post += FileContains('path/to/barfile', 'some text')
Checks that the installed version of atheist is equal or newer than the given number. This is useful to assure recent or experimental features.
Call the specified function with the given arguments. You must avoid the use of this condition as much as possible.
Checks that directory path exists.
Checks that the environment variable name exists, and optionally has the value value.
Checks that file filename exists.
Checks that file filename exists and contains val, which may be a string or a string list.
If whole is False (default) val must be a strip and the whole content of file must be equal to val.
Otherwise, the file must contain at least times occurrences of val. The default value for filename is the stdout of the corresponding task, and it implies save_stdout=True. This implies also the automatic creation of a FileExists condition for that file.
The file content may be stripped by means of strip argument. No stripping by default.
Checks that the contents of filename1 and filename2 are identical. The default value for filename2 is the stdout of the current task, and it implies save_stdout=True.
Checks that port number port is open, i.e., a process is listening to it.
Checks that the given PID belongs to a running process.
There are other available conditions as plugins.
It is True when condition is evaluated as False.
Example:
t = Test('rm foo_file')
t.pre += Not(FileExists(foo_file))
Checks condition every interval seconds, stopping after timeout seconds or when its value becomes True.
In the next example, the task waits (a maximum of 5 seconds) for the nc server to become ready before continuing:
t = Test('nc -l -p 2000')
t.post += Poll(OpenPort(2000))
Condition decorators may be combined. The following example shows a task that waits for an environment variable to be removed before executing the command:
t = Test('command')
t.pre += Poll(Not(EnvVarDefined('WAIT_FLAG')), timeout=10)
Note that the effect of Poll(Not(condition)) is not the same that Not(Poll(condition)).
Task is the base class for all executable items. Test is-a Task that runs a shell command but other kind of Task are possible:
It is a non-checked Test. Command is exactly the same that a Test with a check=False parameter.
The Commands (or other non-checked Tasks) are not considered in results counting.
Command shortcut for detached commands. Predefined parameters are:
There are other available Task subclasses as plugins.
Returns the task whose tid attribute is name. ToDo: [include a sample here]
Makes possible to reuse atheist or python code in other files. load() returns a module-like object that may be used to access to functions, classes and variables defined in the “loaded” module. All atheist classes are available within the loaded modules:
common = load("files/common.py")
Test("./server %s" % common.server_params())
Warning
Previous include() function is not supported any more.
Test files may include some substitutable variable. This is useful to locate test-relevant related files. You must write the symbol ‘$’ preceding each one of the next words:
For example, for the following vars.test file:
Test("echo $basedir $dirname $fname $testname")
When you run atheist, you get:
~/sample$ atheist -i2 tests/vars.test
[ OK ] Test case: ./tests/substitutions.test
[ OK ] +- T-1 ( 0: 0) echo . ./tests ./tests/vars vars
Total: ALL OK!! - 0.24s - 1 test
You may write tasks to execute before and after each test file. To do so, just put this tasks in files called _setup.test and _teardown.test.
When your task creates files you may track them for automatic cleaning. Just add their filenames to the task gen attribute. Here’s an example:
t = Test('touch foo')
t.gen += 'foo'
You may specify one o more filenames (as a string list).
If you want the generated files not to be automatically removed for manual inspection of the results, you must specify the --dirty option (see below). To clean-up these files later, specify the -C option.
Show basic help information.
Colon-separated options for the tests.
Change working directory.
Don’t execute anything, only remove generated files.
Describe tasks but don’t execute anything.
Print the test process’ stderr.
Print task output, but only if it fails.
Generate a test file template with default values.
Verbosity level (0:nothing, [1:case], 2:task, 3:composite, 4:condition)
Skip _setup and _teardown files.
Continue despite failed tests.
List tests but do not execute them.
Print the test process’ stdout.
Load plugins from that directory (may be given several times).
Print the test process’ stdout.
Do not show result summary nor warnings, only totals.
Run testcases in random order using the specified seed.
Specifies command line script.
Include time info in the logs.
Repeat tests until something fails.
Increase verbosity.
Number of simultaneous tasks. ‘0’ allows atheist to choose the number. Default is 1.
Exec COMMAND like in “Test(COMMAND, shell=True)”
Print case execution time in reports
Set terminal width (in chars).
Alternate config file.
Don’t remove generated files after test execution.
Don’t show progress bar.
Files to ignore (glob patterns) separated with semicolon.
List tasks but do not execute them.
Avoid color codes in console output.
Save stdout of all tasks
Save stderr of all tasks
Atheist version
Notify failed tests to the given jabber account (may be given several times).
Notify failed tests to the given email address (may be given several times).
[ToDo]
[ToDo]
[ToDo]
You may use Atheist to monitor the proper working of any application. It may send you notifications when something is wrong. The easiest way is to run a testcase with cron specifying one --notify-* command-line argument. Currently, two notificators are implemented:
The destination is a email account using the SMTP protocol. Atheist does not require an SMTP server or smarthost. You only need to configure an email account that will be used by Atheist to send mail to any destination. That information must be written in the ~/.atheist/config configuration file. This is an example:
[smtp]
host = smtp.server.org
port = 587
user = atheist.notify@server.org
pasw = somesecret
You must indicate a jabber account that will be used by Atheist to send notification messages to any other jabber account. The destination account must accept this contact previously. The following is an example for the configuration in the ~/.atheist/config file:
[jabber]
user = atheist.notify@server.org
pasw = anothersecret
To ask for a notification you just need to specify a test file and one or more destinations. For example:
$ atheist --notify-jabber John.Doe@jabber.info test/some_test.test
It is possible to give several –notify-* arguments in the same command-line to send notifications to different destinations.