Revision as of 11:04, 24 September 2009

Overview

CKTester is our internal testing hub, created to provide seamless and integrated testing experience to our developers, been capable of managing and interfacing tests from various projects which may potentially rely on different testing frameworks/runners. Been designed with convenient authoring and ease of viewing the test cases in mind, it could be running independently in all situation from a browser : locally, on a web server or via secure connections.

How It Works

Test Cell

A test cell is the basic unit of a project's testing, a plain HTML file reside in individual project's testing sources, in which test suites/cases are written. It will be loaded and executed inside an iframe or popup window depends on the running of choice, so full page life cycle ( from page load to page destroy ) is guaranteed.

Each cell has one key responsibility of reporting the sucess/failure/time of it's execution to the fort, this is done by cross-frame communication with parent/opener window. For this, the cell is required to include a bootstrap file:

<script type="text/javascript" src="{CKTester_ROOT}/bootstrap.js"></script>

The cell's entire testing environment, including testing target, runtime libraries, etc. will mostly be injected by the fort, which are declared by a meta tag of the HTML file. This inversion of control feature (IoC) make the cell testing framework neutral.

// I'm a unit test for the editor(CKEditor) and the TCs within me are stable( been finalized ). 
<meta name="tags" content="editor,unit,stable">

Profile

A testing profile is a JavaScript configuration file which specify how a project is to be tested by CKTester. It contains a function which returns a configuration object including. 

1. Cells list : A list of key/values which tells where are the cells and what are the tags of each cell.
   Key : The test cell file path, where all the paths are relative to the runner;
   Value : An array of tags of that cell.
   

1. Cell Resolvers : An array of functions which were used to manipulate each cell's attributes, e.g. Inject necessary dependencies for cells which have certain tags. These are the complementary of the default resolvers that the fort already has.
   

The file content will basically looks like:

CKTester.profile = function ()
{
	return
	{
		cells : [
			[ '/dt/unit/plugins/htmldataprocessor/1' ], 		// Implicit tag declaration( each sub path form a tag ).
			[ '/tt/1.html' , [ 'editor', 'unit', 'htmldataprocessor' ] ]  // Explicit tag declaration.
		]
		cellResolvers :
		[
			function( cell )
			{
				var tags = cell.tags, env = cell.environment;

				// Inject CKEditor source.
				if( ( tags.indexOf( 'editor' ) != -1 )
					&& ( tags.indexOf( 'editor-ondemand' ) == -1  ) )
					env.push( '${CKEDITOR_ROOT}/ckeditor_source.js' );
				else if ( tags.indexOf( 'editor-ondemand' ) )
					env.push( '${CKEDITOR_ROOT}/ckeditor_basic_source.js' );

				// Inject CKEditor unit test library.
				if ( tags.indexOf( 'unit' ) != -1  )
					env.push( '${CKEDITOR_TEST_BRANCH_ROOT}/js/unit.js' );

				// Inject CKEditor manual test library.
				if ( tags.indexOf( 'manual' ) != -1  )
					env.push( '${CKEDITOR_TEST_BRANCH_ROOT}/js/manual.js' );
			}
		]
	};
}

Note: the testing profile is recommanded to be generated automatically by the gen-profile tool script, though it could be altered manually.

Tags

Tag is playing an important role as the annotation of a cell with the following aspects:

1. Environments( dependencies ) of a cell, e.g. if the cell has tags 'unit' and 'editor', it denote it must be running under a unit test runner( e.g. YUITest ), and the testing target is CKEditor;
2. Categories of the cell, useful to grouping similar cells. E.g. A cell with a path 'dt/core/dom/documentfragment' denote it's a design test for the core codes of dom module about document-fragment.
3. Any other reasonable denotation.

Fort

The Test Fort is a central hub for registering and running all the cells, which also consists of a UI to specifiy criterias and viewing report.

The runner is driven by a specified profile( use the default one if not specified ), from which all cells are registering. Upon requesting by a specific testing criteria, it filtering down the registered cells to only execute the satisfied ones.

During the loading of each cell before running, an extensible cell resolver will manipulate ( to figure out the required/requested resources of a specific cell ) the cell by analysing it's tags, guarantee the cell always has all the environment it should receive. These resources are typically composed of :

• The testing target
• The testing library
• Any resources dedicated to the cell.
  

At last the cell will be running in one of these two modes depend on the user:

1. Managed Mode : It's the default mode when it's running in a batch of cells by the forge, where the cell is running in an iframe.
2. Standalone Mode : When a cell is running separately, on the opposite of 'Managed Mode', the cell will be running inside a popup window, this is often used when it's a manual test case or detailed information about the TCs is required to be reviewed( e.g. which exactly test method failed in the TC? ).

Testing Criteria

A criteria is simply a URL request to the fort, tells which cells to be run.

1. It could contains a path to a profile to be used.
2. It could contains certain tags, so only with which the cells to be run.
3. It could explicitly specify a single cell location along with certain tags.

Some example criteria look like:

• Running all cells defined in a specified profile file: http://ckeditor.t/cktester/fort.html?profile=../myprofile.js
• Running all the stable unit tests of the default profile: http://ckeditor.t/cktester/fort.html?cells=stable,unit
• Running a specific ticket test with tags 'editor','unit','all' : http://ckeditor.t/cktester/fort.html?cells=http://ckeditor.t/tt/4397/1.html&tags=editor,unit,all
  ( Note: It's a shortcut to achieved the same criteria by just open the url of that individual cell (http://ckeditor.t/tt/4397/1.html) and it will be redirected correctly back to the forge. )
  

Installation

Note: Below we'll take the CKEditor project as an example to illustrate the install process.

First of all, checking out these three component: the staging project, it's test suites and CKTester eventually.

1. Checkout CKEditor's testing branch from SVN ( We name the working copy path as CKEDITOR_TEST_ROOT below ).
   
2. Checkout CKEditor trunk( or any other version are subjected to test ) from SVN into a folder ( e.g. ckeditor ) under CKEDITOR_TEST_ROOT ( We name the working copy path as CKEDITOR_ROOT below ).
   
3. Checkout CKTester from SVN into a folder( e.g. cktester ) under CKEDITOR_TEST_ROOT( We name the working copy path as CKTESTER_ROOT below ).

Open the file {CKTESTER_ROOT}/variables.js.tpl with modification to the following variables:

	// Absolute path point to the root path of CKEditor project.
	'CKEDITOR_ROOT' : '/ckeditor',

        // Point to the root path of CKEditor test branch.
          ( You don't need to alter it if your web server's serving root from CKEDITOR_TEST_ROOT.
       'CKEDITOR_TEST_BRANCH_ROOT' : '/'

At last, run the script {CKTESTER_ROOT}/_dev/gen-profile.bat(.sh) ( you'll need install Apache Ant ) which helps you to generate the testing profile automatically by scanning the resources within {CKEDITOR_TEST_ROOT}. After that, you'll get a file created at {CKTESTER_ROOT}/template.js which looks basically like:

	CKTESTER.fort.getProfile = function ()
	{
		return {
			cells :
			[
			// Generated cell paths...
			],
			cellResolvers :
			[
			// Some Default cell resolvers. 
			]
		};
	};

It's done, now point your web browser at {CKEDITOR_TEST_ROOT}/index.html, a few pilot links there to kick start you testing.

Common Tasks

History

CKTester is evolved from previous project FCKTest.