Image-Based Test Automation

Screen objects can be found and steered by means of images. This requires a control to be created by capturing an image which is used for identification later on. This image is then searched and an according action is performed upon control steering.

Please note that the system starts searching for images on the monitor that is primarily used upon test execution.

Image-Based Test Automation controls are steered like any other Tosca TBox standard controls. For further information on how to steer controls, please refer to chapter "Steering Controls".

Creating controls

1. Select the option Create Image based control from the context menu of a Module folder. A window for creating the image will open.

2. Put the required application (i.e. browser for XBrowser controls) to the foreground.

3. Click on the plus sign in the blue square in order to create the image. The currently open window is then minimized, and the cursor's shape will change to a reticle.

   Alternatively, you can hit the F12 key to create a screenshot in a context menu. You can modify the function key assignment via the setting Hotkey for Wizard.

4. Define the area to be captured with the mouse pointer: click and hold the left mouse button while you mark the selected area with a red rectangle.

   The system searches for this image when the control is steered.

5. Select the control type via the corresponding button, e.g. Button.

   This option is only available for selection if a control is newly created.

6. Press Enter to create the image.

7. The upper section of the window shows the control's image, and the lower section shows the image used for identifying the control area. The images are identical here since the image of the control is used for identification purposes.

8. Assign a unique name to the control in the Name field.

Tosca also allows several images to be stored for a control, for example if the shape of the toggle button is subject to changes according to the context. Identification requires that one of the stored images is found.

9. Click on the Save button to create the control. A click on Save & New enables you to save the current control and to create a new one.

If the image-based control is created via a Module folder, Tosca creates a new Module with a control, whereas it only adds a control if the image-based control is created from a Module.

Identifying controls via an image

When a control is created, Tosca uses an image for identifying controls. The distance between the control and the image is captured (offset).

This is also possible even if the image and the control overlap each other.

1. Select the option Create Image based control from the context menu of a Module or a Module folder.

2. Put the required application (i.e. browser for XBrowser controls) to the foreground.

3. Click on the plus sign in the blue square in order to create the image. The currently open window is then minimized, and the cursor's shape will change to a reticle.

   Alternatively, you can hit the F12 key to create a screenshot in a context menu. You can modify the function key assignment via the setting Hotkey for Wizard.

4. Define the area that includes the control to be steered with the cursor.

5. Define the area to be captured in a screenshot for identifying the control.

   Both areas defined are marked with red rectangles and an arrow pointing from the image to the control. The screenshot is created once the mouse button is released.

6. Assign a unique name to the control in the Name field.

The distance between the control image and the image used for identification is shown as Offset property in the Search Properties section of the window.

Creating a screenshot for an existing control

This option is available for both HTML controls and controls which are identified via images.

1. Select the option Identify by Image from the context menu of the required control and a window will open where you can capture the screenshot.

2. Existing controls already have a pre-defined image for identification. You can remove this image via the x button if required.

3. To continue with the next steps, please click here and follow the steps from Point 3 onwards.

Taking screenshots will be easier if you enable the option View->General Show details->Minimize during Execution/Scan in the Tosca Commander main menu.

Properties and configuration options

Option	Description
ScreenResolution	This shows the image resolution at the time the image is being created. This value cannot be modified.
Accuracy	Indicates up to which accuracy level the found image must match the stored image in percent.
ActionPoint	This shows the exact position where operations are performed upon steering, for instance a click into the center of a button. This value cannot be modified.
ControlArea	Defines the screen area for the control to be steered (see chapter "Identifying controls via an image "). This value cannot be modified.
Mode	This option can be modified, provided that the image to be used is already applied for identifying an existing HTML control (see chapter "Creating a screenshot for an existing control"). The search mode for identifying a control is defined here. The following values are possible: Fullscreen: the system searches for the stored screenshot on the entire screen. Controlbased: the system searches in the vicinity of the control for the stored image. Auto: the system determines automatically the search mode that is most efficient.
Offset	Indicates the distance from the control to the stored screenshot (see chapter "Identifying controls via an image "). This value cannot be modified.

Mapping images used for identifying controls in Tosca Commander™

Image-based controls have an additional tab called Identification images in the details view of Tosca Commander.

Here you can view the control image along with all images used for identifying the control.

• Click on the Edit images button if you would like to edit your images. As a result, the window used for creating new images will open (see Illustration " Capturing images for identifying controls").

• Double-click onto an image to open this in a new window.

Verifying texts

Texts of image-based controls are verified. The optical character recognition engine Tesseract is used for the verify operation.

The image is modified according to the Tesseract default settings in the Settings dialog prior to performing verify operations (see chapter "Settings - OCR").

If the system should be unable to identify the text immediately, these settings will be automatically modified as follows (SmartOCR):

1. The setting UseMonochrom is disabled.

2. The value of the setting ScaleFactor is increased by 1.

3. The setting UseMonochrom is enabled and the gray scale value is set to 0.2 (MonochromBrightnessFactor).

4. The value of the setting MonochromBrightnessFactor is set to 0.05.

5. The setting UseInversion is enabled.

Tesseract settings can also be defined by using either test configuration parameters or Steering Params for one or more objects. As a result, the global settings of the Settings dialog are overwritten.

The test configuration parameter names match the setting names without any spaces, and they have the prefix Tesseract, e.g. TesseractScaleFactor.

ActionMode: Verify

Value: OCRText property and the string to be verified

This example verifies whether the value Google Search exists in the control image to be steered. If the text is not found, the results of the steps taken are shown in the Detail column (SmartOCR). Test result in the Detail column