"SfR Fresh" - the SfR Freeware/Shareware Archive 
"SfR Fresh" - the SfR Freeware/Shareware Archive
|
TestMaker is an open-source framework and utility for building intelligent test agents that check Web-enabled applications for scalability, performance, and function. TestMaker provides a Recorder to create test agent scripts for you and a graphical environment for building and running test agent scripts. These recorded test agent scripts test a Web-enabled application for functionality by making requests to the host and recording the success or failure and timing metrics of the response. XSTest takes these recorded test agent scripts and runs them multiple times concurrently to learn the scalability of the Web-enabled application as load increases. XSTest is distributed under the terms of an Apache-style free open-source license.
Most tests begin with a simple functional test. The user asks "Does this part of the system work?" Later they may test the system for scaleability under increasing load, regression tests, or to monitor a system. For now the system manager or software developer wants a single test agent to run a Web-enabled application through a single series of steps and view the result.
TestMaker provides a Recorder feature that builds functional tests. The recorded test implements a TestCase class. The TestCase has a method to run the test and return results data. The results indicate if the test encountered errors, the timing metrics of the request and response, and the response from the host. TestMaker provides a graphical and command-line utility environment to run TestCases.
In a load test, the software developer, QA technician, and IT manager wants to learn how the Web-enabled application host handles requests from increasing large sets of concurrently running test agents. They seek answers to questions about data center readiness and software quality, including "Have I bought enough servers?" and "Is may application meeting user needs by responding quickly?" TestMaker provides the XSTest object framework to stage and analyze a load test to answer these questions.
Here the user operates the Recorder feature to create a TestCase object. Using the XSTest framework the user identifies a TestCase object and XSTest plans the test, stages the test, and calls the tally module to report the system's throughput as a Transactions Per Second (TPS) value. (Click here for a use case on the test scenario.)
Load testing delivers a single TPS value. For example, a load test states that at 50 concurrent virtual users the system performed 28 transactions per second (TPS). Scalability testing delivers a function of load tests over a series of increasing loads. For example, consider the following results of a scalability test. (This chart was developed by importing textual results from XSTest into a spreadsheet program that displays the results in graphical charts.)
The test scenario ran 4 series of test cases. In each test case the number of concurrently running test agents increased. It started with 50 concurrently running test agents and finished at 300 agents. After each test case concluded the test delivered a throughput measurement of the test case by stating the average number of transactions per second. For example, in the chart at 50 concurrent test agents the system handled 55 TPS. And, at 300 the system handled only 12 TPS.
There are any number of reasons that the throughput as measured in TPS of a test may decrease - or for that matter increase. To learn why your system performs this way we recommend you read Java Testing and Design: From Unit Tests to Automated Web Tests by Frank Cohen (Prentice Hall 2004.)
XSTest stages load tests and provides a textual scalability index report. XSTest takes a list of concurrent user levels to test against. XSTest calculates and operates the test cases automatically. Then it reports the results in throughput measurements.
Download TestMaker. XSTest comes bundled with TestMaker. You will find XSTest and its documentation in TestMaker_home/xstest once you download and unpackage the TestMaker ZIP archive.
Please familiarize yourself with XSTest by opening the properties.py, master.py, and testcase.py files and browsing through their contents. XSTest consists of these files:
Follow these steps to use XSTest:
By default, the functional test needs to be named functiontest.py and the test agent file must be in the XSTest directory. The testcase.py file controls the file and class name of the functional test.
XSTest finishes a test by running the tally.py test agent script. Tally.py parses the recorded log file to determine the transactions per second value for the current level of testing.
By default properties.py names the file test.log. You can browse the contents of the test.log file to see the recorded results. It is a text file containing comma-delimited results data with one entry per completed functional test. For example, below is an example of the contents of the test.log file after a scenario runs.
0,10553,3484816,14168064,3,0.5,15
0,7483,2925216,14168064,9,0.5,15
0,1658,2586952,14168064,14,0.5,15
0,1071,3547176,14168064,13,0.5,15
0,773,2527592,14168064,14,0.5,15
0,20510,3665704,17272832,7,0.5,30
0,7493,3466768,17272832,24,0.5,30
0,10520,1508368,17272832,10,0.5,30
0,17823,1863240,17272832,16,0.5,30
Each line contains the results of a single completion of the functional test. Commas delimit the following fields:
The tally.py test agent parses through the test.log file and produces a scalability index that looks like this:
Results
CUs TPS Count Min Max Errors
15 1.90 57 773 15581 0
30 2.10 63 806 43471 0
Error count = 0
Total count = 120
% of errors = 0
Legend: CUs = Concurrent users, TPS = Transactions Per Second,
Min = Shortest test case time in milliseconds,
Max = Longest test case time in milliseconds,
Errors = Count of all errors
Done. Thank you for using XSTest. Further details
are available at http://www.pushtotest.com
The report shows that this test scenario learned the scalability index of the Web-application under test by running the test at 15 concurrent virtual users first and then at 30 concurrent virtual users. The properties.py file controls what levels of concurrent virtual users is included in the test scenario. The report shows a TPS value at 15 concurrent users is 1.9 transactions per second and at 30 concurrent users it is 2.10 transactions per second. While this is test scenario is limited to only two data points, we can surmise the following:
Chapter 12 of Java Testing and Design: From Unit Tests to Automated Web Tests by Frank Cohen (Prentice Hall 2004) provides extensive detail on how to analyze a scalability index report to turn the results into actionable knowledge.
Read articles about testing in docs.pushtotest.com
Learn about support options for XSTest
|
