Steer controls with Tricentis Vision AI

You can steer the controls that you have scanned with Vision AI in the same way as TBox controls.

This chapter provides information on:

You can use Vision AI to steer controls in mobile applications if the Mobile Engine 3.0 doesn't support or recognize them. For detailed information, refer to this Knowledge Base article.

Additional parameters for scrollable controls

You can use these steering parameters to increase the timeouts for long lists, trees or tables. Note that you may also need to adjust the Tosca TBox synchronization timeouts.

The table below lists steering parameters for scrollable controls:

Steering parameter

Description

Default value

ScrollSpeed

Specify the scroll speed of Vision AI. Increase the default value to make scrolling faster (e.g. 2.0). Decrease the default value to make scrolling slower (e.g. 0.5).

1.0

ScrollTimeout

Specify how long Vision AI should scroll up or down looking for controls.

Enter the value in seconds.

10

ScrollDownTimeout

Override the ScrollTimeout for scrolling downwards.

 

ScrollUpTimeout

Override the ScrollTimeout for scrolling upwards.

 

Additional parameters for table steering

If you want to automate tables, you can modify additional parameters that specify how Vision AI detects table content. For information on how to edit these parameters in XScan, see chapter "Change the parameters for table detection and steering".

Steer Method

The Method parameter defines the way in which Vision AI distinguishes table cells from pixels on the screen. You can use the following values for this parameter:

  • Gridlines - Vision AI searches for long vertical and horizontal lines within the table bounds. This method might recognize the bottom of the text in a row as a possible line, which results in detecting an extra empty row. In this case, you can use the MergeEmpty parameter to get rid of the extra empty row.

  • Words - Vision AI searches for pixels that are different from their surrounding pixels but not gridlines. This method uses the vertical and horizontal gaps between words as cell boundaries.

  • CountrolBounds - Vision AI searches for the bounds of each control and a text block. This method uses the vertical and horizontal gaps between these controls as cell boundaries.

  • Auto - Vision AI uses a combination of the Gridlines and Words methods.

Steer ColumnThreshold and RowThreshold

The parameters ColumnThreshold and RowThreshold control how sensitive Vision AI is when it selects a row or column. The values for these parameters range from 0 to 100. They specify the percentage threshold that Vision AI must detect to steer the row or column. Lower values mean that more rows or columns are used while higher values mean that less rows or columns are used.

The default value Auto selects the best value based on the Method parameter:

  • For Gridlines method, it's 30.

  • For Words method, it's 15.

  • For ControlBounds method, it's 1.

Steer MergeEmpty

Vision AI might detect extra empty rows or columns. This happens frequently, if you use the Words or ControlBounds method.

The MergeEmpty parameter specifies when these empty rows and columns are merged into the other content of the table:

  • Set this parameter to Yes to turn merging on or to No to turn it off.

  • Set this parameter to Auto if you want to turn merging off when you use the Gridlines method. For all other methods, it turns merging on.

Steer HeaderColumn and HeaderRow

These two parameters work in the same way as standard TBox tables. You can select which row and/or column number is used for headers.

To set these values, click on the cells in the Content Preview.

Steer HeadersContain

Sometimes the exact header row number varies depending on the detection location. In this case, you can't rely on the HeaderRow parameter. Instead, set the HeadersContain parameter to a text value from the header row of your table. Vision AI searches the table for a row that contains this text value and uses it as the header row.

Steer DecisiveColumns

This parameter works in the same way as standard TBox tables. It specifies the columns for the $lastContentRow and $firstEmptyRow selectors. The default value is * which means that all columns are used.

Steer IgnoreInvisibleTableContent

This parameter has no effect for Vision AI tables.

Steer ColumnMinGap and RowMinGap

These parameters specify the minimum number of pixels between columns and rows. The default value is 0.

Increase the values of these parameters by small amounts to reduce the number of columns or rows that have been split incorrectly.

Additional parameters for typing

After you have scanned your controls with Vision AI, you can add steering parameters that control the typing mechanism. These steering parameters apply only to controls with the ControlType property Input or Dropdown.

Note that Vision AI uses a slightly different syntax for special key combinations. To send special key combinations, use the {VSKEYS} command in your TestSteps instead of {SENDKEYS}. For more information on the VSKEYS syntax, click here.

Add steering parameters for the Input control

The table below lists the possible steering parameters for a control with ControlType property set to Input. If you don't specify these parameters, the typing mechanism behaves according to the default value.

Steering parameter

Description

Default value

KeyDelay

Specify how long Vision AI should wait after each key-up or key-down action.

Enter the value in milliseconds.

10

HoldDelay

Specify how long Vision AI should select and hold the key.

Enter the value in milliseconds.

0

SelectTextKeys

Specify the key sequence that Vision AI should select before it enters the value that you specify in TestStepValue.

By default, Vision AI replaces the existing content of an input box with a new value that you specify in the TestStepValue.

Ensure that you enter the value in double quotes to prevent Tricentis Tosca from recognizing it as a dynamic expression.

"{Home-Ctrl-Home-Shift-End}"

DeleteTextKeys

Specify the key sequence that Vision AI should select if the TestStepValue is empty.

By default, Vision AI deletes the existing content of an input box if a TestStepValue is empty.

Ensure that you enter the value in double quotes to prevent Tricentis Tosca from recognizing it as a dynamic expression.

"{Delete}"

In this example, you want to select the content of an input box with the key sequence Ctrl and A. To do so, you add the steering parameter SelectTextKeys in the Properties of this control and set its value to "{Ctrl-A}".

SelextTextKeys steering parameter

Add steering parameters for the Dropdown control

The table below lists the possible steering parameters for a control with the ControlType property set to Dropdown. If you don't specify these parameters, the typing mechanism behaves according to the default value.

Steering parameter

Description

Default value

KeyDelay

Specify how long Vision AI should wait after each key-up or key-down action.

Enter the value in milliseconds.

10

HoldDelay

Specify how long Vision AI should select and hold the key.

Enter the value in milliseconds.

0

ConfirmItemKeys

Specify the key sequence that Vision AI should select to confirm an item and close the drop-down.

By default, Vision AI selects Tab to confirm an item and close the drop-down.

Ensure that you enter the value in double quotes to prevent Tricentis Tosca from recognizing it as a dynamic expression.

"{Tab}"

In this example, your application requires you to press Enter instead of Tab to select an item from a drop-down. To change the default behavior, you add the steering parameter ConfirmItemKeys in the Properties of this control and set its value to "{Enter}".

ConfirmItemKeys steering parameter

Special value <Foreground> for the Caption parameter

You can edit the Caption parameter of your Module to steer the topmost window. This allows you to steer, for example, pop-up windows or windows without a caption that are in the foreground.

To do so, follow the steps below:

  1. Scan a topmost window that you want to steer with XScan.

  2. Open the Properties pane of your new Module.

  3. Replace the value of the Caption parameter with <Foreground>.

    When you use this Module in a TestCase, Vision AI steers the window that is in the foreground.

Caption parameter with value <Foreground>

Window identification can optionally use the ClassName and ProcessName transition parameters. These parameters are selected if the window has an empty caption. If you use the value <Foreground> for the Caption parameter, Vision AI combines these values, which will select the front-most window that matches the specified ClassName and/or ProcessName.

Limitations in steering tables with Vision AI

Currently, Vision AI doesn't support the following options and properties for steering tables:

  • Horizontal table scrolling within the table. If Vision AI doesn't find a specified row or cell, it scrolls through the table, but only vertically. Horizontal scrolling is not supported.

  • Defining embedded controls in cells via the Create Embedded <Control-Type> control option in the context menu. If you want to use an embedded control, scan it separately in XScan. In the Modules section, drag this control into the Cell ModuleAttribute.

  • Column-first steering doesn't support row labels. This means that you can't use the row label as the cell name if you use the secondary steering under a table control. For example, if you select a column and then a cell, only numeric indexes such as $1, $2, $last are supported.

  • The Content Preview pane doesn't display the cells that are merged. The grid lines in merged cells are visible.

To overcome these limitations, you can insert the VisionScript statements directly between TestSteps in Tricentis Tosca. To do so, use the Run VisionScript Module.

Parameters for Tree steering

If you want to steer Trees, you can modify steering parameters to specify how steers them.

The default steering method is the standard single-line syntax, with the names of each Tree node separated by ->. Example: Level1->Level2->Level3.

Multi-line syntax is also supported, but you need to manually create ModuleAttributes for each level. On the ModuleAttribute, set the following values for the following parameters:

  • BusinessType: VisionAIControl

  • SpecialIcon: TreeNode

  • BusinessAssociation: Descendants

  • Engine: Vision AI

  • ExplicitName: True

  • ControlType: Node

The table below lists steering parameters for Trees:

Steering parameter

Description

Default value

DepthMethod

Specify how different levels are identified in the Tree. Text uses the left bound of the text, Pixels uses the leftmost non-background pixel (e.g. the left edge of the icon or text, whichever is first). Some Trees have different indents on the text, depending on whether there is an expansion icon or not.

Text

MinIndent

Specify the number of pixels a tree node should be indented from the parent node to count as a child.

4

ExpandMethod

Specify how a Tree node should be expanded. Possible values: Click, DoubleClick, IconClick. Can also be followed with keystrokes in curly braces, for instance {Right}. Note that you must quote curly braces, for instance "Click{Right}".

DoubleClick

SelectMethod

Specify how the final Tree node should be selected. Similar to ExpandMethod.

Click

Limitations

  • For single-line steering, Trees are considered to be lists with expandable nodes. There is no check whether each level is strictly below the previous level. This is due to current limitations in scrolling and OCR bounds detection.

  • For multi-line steering, all nodes must be visible on screen at once. If a parent node is scrolled out of view during steering, then the steering will fail.

Parameters for Menu steering

You can steer Menus using the single-line syntax, for instance File->New->Mail Message.

The table below lists steering parameters for Menus:

Steering parameter

Description

Default value

FirstClick

Specify whether to click or hover the mouse over the first item.

True

IntermediateClick

Specify whether to click or hover the mouse over the subsequent items. The only exception is the last item, as this one is always clicked.

True

SeparatorTreshold

Specify the height of the control in percentages.

If a vertical line is detected with a length that exceeds this threshold, e.g. more than half the height of the menu bar, then this is considered a break in the menu items. No menu item will span over the separator. This steering parameter is useful if words that are close together are being merged together into one menu item, but there is still a vertical line between them.

50

Parameters for TabBar steering

To steer a TabBar, use the name of the tab you want to click on. One limitation is that the currently selected tab is not identified and can't be verified.

The table below lists steering parameters for TabBars:

Steering parameter

Description

Default value

SeparatorTreshold

Specify the height of the control in percentages.

If a vertical line is detected with a length that exceeds this threshold, e.g. more than half the height of the TabBar, then this is considered a break in the tabs. No tab will span over the separator. This steering parameter is useful if words that are close together are being merged together into one item, but there is still a vertical line between them.

50

Parameters for Toolbar steering

To steer a Toolbar, specify the text of the item you want to click on.

The table below lists steering parameters for Toolbars:

Steering parameter

Description

Default value

SeparatorTreshold

Specify the height of the control in percentages.

If a vertical line is detected with a length that exceeds this threshold, e.g. more than half the height of the Toolbar, then this is considered a break in the toolbar items. No item will span over the separator. This steering parameter is useful if words that are close together are being merged together into one item, but there is still a vertical line between them.

50