|
See the readme.html document for the changes in this version of TestMaker.
Java, .NET, and many other computer programming platforms enable software developers to build software applications that provide services accessible using open standard Internet protocols on routed networks, including the Internet and private Intranets. From Web applications to Service Oriented Architecture (SOA), software developers, QA technicians, and IT managers are challenged to deliver services that are scalable, well performing, and reliable. With all this interoperability, server-to-server and application-to-application communication going on... How do you assure the system will be reliable and scalable?
TestMaker is a framework and utility to assure that your services are meeting your user goals. The TestMaker framework helps you rapidly build unit tests to check a service for proper function. Collections of these unit tests deliver an intelligent test agent that uses open Internet protocols (HTTP, HTTPS, SOAP, XML-RPC, SMTP, POP3, IMAP, etc.) to drive an entire service as a real user would.
TestMaker test agents are easily reused as load tests and service monitors. For instance, run a TestMaker test agent in the XSTest framework for instant scalability and load tests. Additionally, run TestMaker test agents in the Service Monitor System for a live dashboard that shows on-going "live" quality of service monitoring results and alerts your IT team when performance slows to unacceptable levels.
TestMaker is brought to you by PushToTest and an open-source community of 26 engineers and 115,000 users. PushToTest Global Services builds test automation solutions on the TestMaker Platform, certifies services as "ready for production," provides "best practices" research for your engineering team, and provides training to teach your team the best practices to use TestMaker to solve your scalability and performance problems. TestMaker is also the basis for the widely used SOA Performance Kit.
This user guide intends to teach you these facets of TestMaker:
While this documentation is a good reference on TestMaker, it does
not teach how to test. For that, we recommend Java
Testing and Design: From Unit Tests to Automated Web Tests (Prentice Hall
2004). Java Testing and Design is written by Frank
Cohen, the principal maintainer of TestMaker and founder of PushToTest.
The book is 480 pages long and offers our favorite tips, techniques, and example
TestMaker test agent scripts.
This section describes how to install TestMaker, and how to get started with TestMaker quickly, and how TestMaker enables black-box and grey-box testing.
TestMaker
comes as a set of files that are compressed into a ZIP archive. To install TestMaker,
uncompress the archive into a TestMaker directory. The TestMaker documentation
will refer to testmaker_home as the directory into which the TestMaker files
are uncompressed.
The testmaker_home directory contains these files:
TestMaker comes with everything needed to build and run intelligent test agents. TestMaker requires a Java 1.4 or higher compliant virtual machine already installed.
From time-to-time we update TestMaker to solve bugs and introduce new features. The PushToTest Web site home page shows the most recent version number. Downloads are available there too. The latest update to TestMaker is always found at http://downloads.pushtotest.com/TestMaker.zip.
Answers to common installation problems are found on the Frequently Asked Questions page.
TestMaker is appropriate for testing Web applications, Web Services, Email systems, Java applications, and more. The following is a table showing our suggestions on how to get started with TestMaker.
 |
|
|
|
|
|
|
| Web application test | Web Service and SOA test | Email system test | Java application test | |||
|
Web applications provide users with functions accessible through a Web browser. TestMaker features record and playback technology for Web applications. TestMaker watches your use of a browser and writes a test agent script. To the Web application the test agent appears to be a Web browser user. The test agent script is a functional test (also referred to as a unit test.) TestMaker includes HTTP and HTTPs protocol handlers. These handlers provide support for cookies, redirect commands, authentication and HTML parsing. To begin using TestMaker in a Web application test environment follow the tutorial. To learn more about TestMaker in a Web application test environment we recommend any of the following: look at the Web sample test agents, read the tutorial, read the Jython scripting guide, read the XSTest documentation and code. To understand how to test a Web application, read chapter 4 of Java Testing and Design. |
Web Services and Service Oriented Architecture (SOA) provide server-to-server and application-to-application XML communication. TestMaker features a New Agent Wizard that reads a Web Services Description Language (WSDL) definition of a service and writes a test agent script. To the Web Service the test agent appears to be a server or application making a request and expecting a well formed response. TestMaker includes SOAP and XML-RPC protocol handlers. These handlers support SOAP document-style (message) encoding, SOAP RPC-encoding, HTTP authentication, SOAP over HTTPS communication, and cookies. TestMaker provides XML parsing and query capabilities through included JDOM, XPATH, Xerces, and JAXB libraries. To begin using TestMaker in a Web Service test environment, use File -> New Agent Wizard -> WSDL to create a test agent script. To learn more about TestMaker in a Web application test environment: look at the Web Service sample test agents. Then read and understand XSTest to turn the same test into a scalability/load test. Then read the Service Monitor System (SMS) documentation to turn the test into a Quality of Service monitor. To understand how to test a Web Service, read chapters 5 and 6 of Java Testing and Design. |
Email systems provide access to email messages stores through email client software and Web browser based email access portals. TestMaker includes SMTP, POP3 and IMAP protocol handlers to communicate with an email store. TestMaker features record and playback technology to use Web browser-based email access portals. TestMaker email protocol handlers support MIME attachments, UTF encoding support, and authentication. To begin using TestMaker in an email test environment, read the email sample test agent. Then read and understand XSTest to turn the same test into a scalability/load test. Then read the Service Monitor System (SMS) documentation to turn the test into a Quality of Service monitor. To understand how to test an email system, read chapter 13 of Java Testing and Design. |
Many companies deploy Java applications to provide an intelligent graphical interface to a service. For example, Sun Microsystem's provides Java desktop application to its engineers to enable bug tracking and change management in its products. The application makes SOAP requests to a service to manage bugs. TestMaker - being a 100% Java application itself - enables test agent scripts make direct calls to a Java application's methods. The application responds in the same way it would to a real user. TestMaker runs a test of a Java application using the same functional test (unit test) framework as Web application, Web Service, and email tests. To begin using TestMaker to test a Java application, read the Java application sample test agent. Then read and understand XSTest to turn the same test into a scalability/load test. Then read the Service Monitor System (SMS) documentation to turn the test into a Quality of Service monitor. To understand how to test an email system, read chapter 11 of Java Testing and Design.
|
Black Box Versus Grey Box Testing
The test environments described in the previous section show how TestMaker provides a framework and utility for building tests that check services for scalability, performance, and reliability. The tests perform black-box tests of a service where measurements are taken at the protocol level from the consumer (client) of a service.
In a black-box test environment, a test environment generates load on the service by making requests to the service and observing the service's response. Black-box scalability testing observes the service's ability and speed to respond to requests as the number of concurrent requests increases.These observations facilitate capacity planning and identify the service's ability to "go live" in a production environment. The TestMaker test environment conducts black-box tests and goes further.
TestMaker is also appropriate for grey-box tests. In a grey-box test the service under test may be queried for internal information that will help to uncover functional and scalability problems. The service under test provides state information from the internal operation of the service, including performance information from the application and database tiers of the service environment. For example, the service responds to special debugging service requests with CPU utilization, io utilization, memory allocation, and transaction states. This extra information allows the Test Environment to provide actionable knowledge in three categories:
• Measuring throughput from
the test agents (client side) identifies the server system’s readiness
to handle the forecasted population of users.
• The test environment changes the mix and operation of test agents during
a test to identify bottlenecks and isolate scalability problems within the service
architecture and software.
• Measuring CPU, memory, and IO bandwidth usage identifies metrics that
will be used for capacity planning.
This section described how to install TestMaker, how to get started with TestMaker quickly, and how TestMaker enables black-box and grey-box testing.
This section provides a high-level view of TestMaker, introduces the TestMaker Platform and optional commercial modules available from PushToTest. The section after this one describes TestMaker in-depth.
Service environments are a powerful and widely adopted technology that enables software applications to communicate over private intranets and the Internet. The protocols (HTTP, HTTPS, SOAP, XML-RPC, SMTP, POP3, IMAP) are accepted standards and widely supported by the computer industry. Enterprise adoption of Web applications sparked an explosion of new software projects, all of which have interoperability and connectivity requirements. A Web browser may retrieve a Web page, a pager may receive an email, a portal may collect data in XML form from many services to present a single Web portal page, and many systems located in various places may be integrated into a productive system.
TestMaker agents drive a service to check that functions are working and are configured properly, to simulate real-world product environments to test scalability under heavy load, and even to proof a Service Level Agreement (SLA) by periodically monitoring a service for proper function.
TestMaker delivers a rich environment for developing intelligent test agents. TestMaker comes with:
Many first time TestMaker users find that running the included sample test agents is a quick way to learn the TestMaker system. The QuickStart guide, documentation, and tutorials will help you learn TestMaker.
PushToTest products are designed for software developers, QA technicians and IT managers. Software developers need a framework to conduct system-level unit tests to check functionality. TestMaker provides a functionality test framework by delivering the Jython scripting language implemented in Java - so developers have full use of Python objects and any Java object. TestMaker agent scripts developed by software developers for unit and functionality tests are then reused by QA technicians to run scalability and performance tests. IT managers run the same agent scripts in the TestMaker environment as quality-of-service monitors. This unique approach from PushToTest provides a new way to leverage any enterprises' software development, QA and IT efforts resulting in new huge cost-savings.
TestMaker is written in Java and runs everywhere Java runs, including Windows, Linux, Solaris, Macintosh and more.
TestMaker is distributed under a free Apache-style license. There is no cost to download and use TestMaker. TestMaker comes with its source code to allow you to better understand how it works, to fix bugs and contribute improvements to the TestMaker community, and to even build your own custom test environment. The next section describes the commercial-grade versions of TestMaker and TestMaker-based solutions.
PushToTest provides support for TestMaker, offers commercial tools that deliver value-added scalability and performance monitoring solutions, and offers a licensed commercial-grade version called TestMaker Platform. This section describes these commercial options.
While the core of TestMaker is immediately useful and free, PushToTest offers a commercial-grade version called the TestMaker Platform that offers these features:
The TestMaker Platform is available from PushToTest. Pricing begins at $10,000 USD and is based on these factors:
Contact PushToTest for your proposal today. Send email to globalservices@pushtotest.com or call (408) 374-7426 (US, California time) for an immediate proposal.
A tutorial and installation guide for TestMaker
A guide to the example agents that come with TestMaker
A list of frequently asked questions (FAQ)
Known problems and incomplete features in this release of TestMaker
Learn about Web Services and TestMaker from articles from IBM, BEA, Sun and popular magazines
Read the book: Java Testing and Design: From Unit Tests to Automated Web Tests
Read the documentation on the docs.pushtotest.com site
Building test agents to check email services for scalability and functionality
Learn about the changes and new features since the previous TestMaker release.
Additionally, check the testmaker_home/docs/lessons directory for tutorials and articles on using TestMaker.
|
