Methodology, recommendations, and limits

Before creating and running automated tests, you need to consider the following questions:

Recommendations

The Sage X3 Automated Test Platformis based on the Cucumber open source component that supports behavior-driven development (BDD). Cucumber allows you to write and execute automated regression tests in a user-oriented language called "Gherkin."

Gherkin language introduction

Gherkin language is a business-readable, domain-specific language that allows you to describe the software’s behavior with an end-user approach. Each action or step in the automated test exactly represents actions a Sage X3 user would perform.

Gherkin reserved keywords

Gherkin primary keywords

Feature: A keyword is used to define the feature name. To ensure consistency, it is recommended to name the feature with the same code as the feature file. An automated test always starts with the feature keyword. A feature can be composed of one or several scenarios. 

Scenario: A keyword is used to define the scenario name. A scenario is a test case aimed at describing a succession of user actions performed by the automated test. A scenario is composed of steps describing the different actions performed by the automated test.

Scenario outline: A keyword is used to define a type of scenario that allows it to be executed several times using a different combination of values for each execution.

Example: A keyword is used to define the different combination of parameters and values used by the scenario outline.

A step can be an action or a verification performed by the automated test. Each step begins with one of the following keywords:

Given: describes the initial context of a step

When : describes an event or an action in a step

Then: describes an expected outcome or result of a step

And: used with other steps, (e.g., Given, Then, etc...) to include something

But: used with other steps, ( e.g., Given, Then, etc...) to exclude something

" " ": (Doc Strings) used to encapsulate the value

| : (Data Tables) used for scenario outlines to define different combinations of values and the parameters they are related to

@ Tags: defined before the feature keyword to categorized them. When a Jenkins pipeline is executed, it is possible to execute all the features related to the same Category (feature defined with the same tag)

#Comments: not executed.

Automated test example

An automated test starts with the feature definition and then adding one or multiple scenarios. The following automated test is composed of three scenarios:

• A scenario to log in

• A scenario to open a specific function

• A scenario to log out of Sage X3

Each scenario is composed of steps describing the different actions to be performed by the automated test.

When the automated test is executed, the actions are executed in the following order:

1. To log in into Sage X3 using a login, password and connection type parameters.

2. To select the proper endpoint and to check that the endpoint has been selected.

3. To open the opens the company functionality and to verify the company functional is opened. Then to close the company function.

4. To log out of Sage X3.

Example of a scenario outline

The following scenario is for a "scenario outline," which is a type of scenario used for running the same test multiple times with different datasets for each iteration.

The values defined between the delimited characters "<>" are the parameters.

Under the Examples tag, a data table delimited by pipe characters | declares the parameters and their corresponding values.

In this example, the scenario outline will be executed twice:
The first time to create a sales order for customer NA008 and the product BMS012, and the second time using a different customer, product and price according to the data in the parameter values in the data table.

Scenario and scenario outline examples

The first scenario consists of opening the required function or entity.

The second scenario is a scenario outline. This scenario consists of creating the necessary data with different values for each iteration.

The third scenario consists of closing the required function that was opened.

Writing automated tests

How to fine the x3name for a field

When you write automated test with (Undefined variable: Sage_X3.X3ATP), you can select the graphical components either by the field name (field description) or by the x3name (X3 field code) in the HTML code.

Let’s take the example of the companies’ functionality.

The company text field can be selected by its field description.

And the user selects the text field with name: "Company"

However, the long description doesn’t have field description, so you need to identify it by selecting X3 field name.

And the user selects the text field with the x3name: "CPY0_CPYNAM"

How to find the x3name for a text field

1. Using the screenshot above as a model, place your cursor over the UI graphical element (text field) containing the value NAProcess.

2. Right-click and select Inspect.

The HTML code for the page displays. The x3name for the text field is in the data-s-field-x3name tag.

How to find the x3name for a data table

1. Click the left top corner of the data table, right-click and select Inspect.

2. In the HTML code for the data table is in the data-s-field-x3name.

This data x3name value contains the keyword ARRAY.

Types of automated tests

Remember when you write automated tests:

• CRUD: You use this type to test simple object Create, Read, Update, Delete actions

• FLOW: You can create more complex tests for generating a business flow such as creating a sales order, creating a delivery by selecting the sales order, generating a sales invoice by selecting the delivery, etc… You can also validate the different controls developed for a specific functionality.
  

• GraphQL: type of tests aimed at executing GraphQL queries or mutations
  Refer to Executing GraphQL queries & mutations (Extensibility) in the Extensibility framework documentation.
  

Using stored values

In certain situations, Sage X3 generates documents and data with automatically and dynamically created reference numbers.

• Because the sequence number is generated dynamically, it is not possible for the Sage X3 Automated Test Platform to predict which value will be assigned to the new document created.

• It is important to create your automated tests to be generic, so they can be executed several times without having to modify the test each time you need to execute it again.

The Sage X3 Automated Test Platform makes it possible to store the value of a given text field or a given data table cell into a stored value that can be reused later in your test.

• A stored value is only usable in the feature file in which it has been created.

• It means the stored value can only be used inside the current feature file and can’t be passed to another feature file. The stored value saved in memory is deleted once the test execution is completed successfully or not.

Since Sage X3 Automated Test Platform version 2.3.0, the stored values capability has been enhanced. You can use stored values directly to write or verify dynamic values in:

• Text fields

• Text editor fields

• Text area fields

• Password fields

• Data table cells

In these cases, stored values need to be between square brackets.

In addition to the new stored values management, a new step definition has also been implemented. It allows you to generate a dynamic sequence and store the result into a stored value.

For more information on how to use stored values, consult the Sage X3 Automated Test Platform - Step definitions guide.

Using user stored values

Thanks to the user stored values functionality available with Sage X3 Automated Test Platform version 2.4.0, you can create your own set of parameters with their respective values.

Contrary to stored values which are instantiated during the executing of your test, user stored values are specific parameters defined with their respecting values in a user-stored-values file which is loaded in memory before each test is executed.

To use the user stored values capability:

• The userStoredValuesFile parameter can be defined in the parameters or parameter-cloud-chrome files.

  • This parameter contains the path of the specific user-stored-values file that contains the user parameters and their respective values.

  • This parameter is optional.

    Example: parameters file

    

  • At the beginning of the execution, if the userStoredValuesFile parameter contains an incorrect path, no feature will be executed. The execution will directly stop with the following error message:

    parameter userStoredValuesFile: cucumber-user-config/user/<FileName> file not found

  • If the parameter is not defined or left blank, no user-stored-values file is loaded.

• The user-stored-values file is located in the cucumber-user-config folder.

  

• Like for static parameters defined in the parameters or parameters-cloud-chrome files, the user-stored-values file contains the list of parameters with their respective values.

  

• As for the stored values, you can use the user stored values to write and verify dynamic values in text fields, text area fields, text editor fields, password fields, or data table cells by encapsulating them between square brackets.

  Example: Automated flow test using user stored values

  

  Recommendation
  When you create a user stored value, use a naming convention that makes it easy to search from in your tests or in VS code. In this sample, the user stored values are prefixed with USV_ so they can easily be found in all the tests and the user-stored-values file.
  

  

• It is possible to manage several user-stored-values files and to name them with your own naming convention. However, it is only possible to load one user-stored-values file per execution.

• The user stored values are instantiated prior to each feature (test) execution and are active for each feature executed. The user stored value is deleted once the feature execution is completed.

  • If you modify its value during the feature execution, this will impact only the current feature executed and not the other feature that can use the same user stored value.

  • Like for stored values, it is not possible to move a user stored value from one feature to another. The user stored value is active during the entire feature execution and across all the scenarios of the current test executed.

Executing tests in parallel

Sage X3 Automated Test Platform version 3.0.0 allows to run automated tests in parallel.

• The maximum number of executors depends on your configuration and can’t exceed the MAXATP value defined in your Sage X3.

  • This information can be found in Sage X3 / Warehousing in the following functionality Administration > Administration > Licenses > Licenses data

  • Click your license to display the license information.

  • Scroll to the parameters section and look for the parameter Number of ATP users parameters (MAXATP)

  • Default Sage X3 Automated Test License are sold with 5 users.

• Refer to Running simultaneous tests in VS Code

  • Setup MAX_INSTANCES parameters

    • It recommended not to set MAX_INSTANCES with a value greater than MAXATP -1 and depending on the number of concurrent access your Sage X3 web server can support.

  • Setup timeoutLocks parameters.

Why run tests in parallel

Running tests in parallel, allows to decrease the time of tests executions. It also Allows to test concurrent executions of functionalities.

• However, by running tests in parallel, tests can conflict with each other provoking lock in Sage X3.

• With Sage X3 Automated Test Platform 3.0.0 new step definitions have been included to prevent locks from happening in Sage X3 when the tests are executed in parallel.

the user adds the lock entry "(.*)"

the user removes the lock entry "(.*)"

How to write tests to avoid locks in your SageX3

Let’s take the example of the following sample features which try to modify the same document.

• In this example, two tests are trying to interact in the same function with the same record.

  • Note the tests should be independent of each other and not use the same document. However, this example has been done on purpose to explain how to avoid the tests to conflicts with each other.

  • The first thing to consider will be to make sure the tests are data independent: meaning the tests should use different data.

• However, even doing so, it can happen in certain cases that one test locks very briefly a record and so the other test that need to interact with this record will fail.

• To avoid this colliding issue, it is necessary to add additional step definitions to prevent the tests from conflicting with each other.

  • In the following example, the lock occurs on the GESBIC function. So, it is necessary to make sure the lock is activated before entering the function and remove it once all the interactions in this function is completed.

  • The same lock code is added and removed on both tests. (e.g. LCK_GESBIC) in a dedicated scenario.

• In addition to the lock scenario, you can also add a tag on your test using the same naming convention as the one you use for the lock code.

  This will allow you to execute again in parallel all the tests that are defined with the same tag (e.g. @LCK_GESBIC).

• Let's verify the execution of the two tests in parallel.

  • Refer to: Running simultaneous tests in VS Code

• Now, because of the lock mechanism is activated for those two tests and because they are sharing the same lock code, the second test has waited for the first one to remove the lock before continuing with the execution. Both tests passed without conflicting anymore with each other.

Recommendation to write test for parallel execution.

• Executing tests in parallel can be difficult to manage. This should be done in a secondary step when your tests are already working well with a single instance.

• All tests conflicting with each other’s, should share the same lock identifier.

  • The lock code given should be a unique identifier that is easy to look for.

  • For instance, in the example above all the lock codes starts with the prefix “LCK_” which makes it easy to find them using visual studio code.

• In the easiest case, adding a lock will consist in preventing several functions from being opened at the same time. Generally, you will create a specific scenario using the lock step definitions prior / after the business scenario that opens the specific business function. It is recommended to add the lock code to the scenario description, so you can quickly identify it in your test or in allure report.

• It is not prohibited to add / remove a lock into a business scenario (for instance in the GESBIC modification scenario),

  • However, it can make the maintenance more difficult.

  • But for performance reasons, it can be interested to remove the lock before closing the function, especially if the current business scenario takes too much time.

  • Locking a scenario for too long will increase the tests execution time.

  • The maximum time the Robot is waiting for the lock to be removed corresponds to the value defined in the parameters timeoutLocks.

    When you add a lock in memory, don't forget to remove the lock from memory, otherwise the lock will be removed only once the tests execution is completed. For performance reason, and to avoid increasing the duration of the test executions, the lock should be removed when it is not anymore necessary in the test.

• When you execute a test that pass with a single instance and fails when running multiple instances, it means you may have tests conflicting with each other’s.

  • use allure report timeline to identify, which tests were executing in parallel.

    • For instance with MAX_INSTANCE =4 , 4 tests executed in parallel.

    

  • Try to run those tests in parallel from vscode to identify from where the problem comes from.

    • Refer to Running simultaneous tests in VS Code

Executing GraphQL queries & mutations (Extensibility)

Since Sage X3 Automated Test Platform 3.2.0 it is possible to test GraphQL Queries and mutations. Samples can be found in Samples/04-GraphQL-Samples.

GraphQL tests structure

Tests are organized per type of tests: queries or mutations.

• For the queries, you have 3 files:

  • the cucumber feature.

  • the GraphQL query.

  • the GraphQL response (if you want to validate the entire response).

  

• For mutations, you have 2 files:

  • the cucumber test.

  • the GraphQL mutation.
    (compared to GraphQL query,there is no file to compare the expected response with the actual response returned by Sage X3, as the values of the actual response can differ at each execution (e.g. timestamp))

  

GraphQL tests codification

• It is recommended to differentiate cucumber GraphQL tests from regular cucumber tests (CRUD/FLOW).

• To easily differentiate queries and mutations, it is also suggested to separate them from each other and to add the type of GraphQL tests in the name of the test to facilitate the monitoring in allure report.

• Example of codification:

Parameters and Parameters-cloud-chrome

By default, browserLanguage parameter is delivered with language en-US.
If you don't define the parameter browserLanguage, the browser will be displayed by default in en-US.

• Refer to Setting up your parameters
• Refer to Running simultaneous tests in VS Code

Step definitions

	Type	step definitions	Examples
1	GraphQL - redirect the Robot to GraphQL query API page	the user opens the application on a desktop using the following link: \"(.*)\"	the user opens the application on a desktop using the following link: "/xtrem/explorer/"
2	GraphQL - write the query / mutation	the user writes\"(.*)\"GraphQL request	the user writes "./0402B-gql- query-read-customer-request.graphql"
3	GraphQL - query / mutation execution	the user clicks the \"(.*)\" button in the GraphQL page header	the user clicks the "Run" button in the GraphQL page header
4	GraphQL - response full validation	the "(.*)" GraphQL response is valid	the "./ 0402B-gql- query-read-customer-response.json" GraphQL response is valid
5	GraphQL - attach the expected / actual response to allure report	the user attaches the actual GraphQL response to allure report
6	GraphQL - response property selection	the user selects the\"(.*)\" GraphQL property	When the user selects the "data.xtremX3MasterData.customer.query.edges[1].node" GraphQL property
7	GraphQL - element of the response validation with exact match json format is requested	the selected GraphQL property value is '(.*)' the selected GraphQL property value is """ """	Then the selected GraphQL property value is '{"_id":"NA001","code":{"code":"NA001"},"companyName":"International Distributors","shortCompanyName":"IntlDist","serviceContract":null}' Then the selected GraphQL property value is """ { "_id": "NA001", "code": { "code": "NA001" }, "companyName": "International Distributors", "shortCompanyName": "IntlDist", "serviceContract": null } """
8	GraphQL - element of the response validation by contains text format is requested	the selected GraphQL property value contains'(.*)' the selected GraphQL property value contains""" """	Then the selected GraphQL property value contains '"_id": "NA001"' Then the selected GraphQL property value contains """ "_id": "NA001" """
9	GraphQL - element of the response - negative test case json format is requested	the selected GraphQL property value is not '(.*)' the selected GraphQL property value is not """ """	Then the selected GraphQL property value is not '{"_id":"NA002","code":{"code":"NA002"},"companyName":"International Distributors","shortCompanyName":"IntlDist","serviceContract":null}' Then the selected GraphQL property value is not """ { "_id": "NA002", "code": { "code": "NA002" }, "companyName": "International Distributors", "shortCompanyName": "IntlDist", "serviceContract": null } """

 

Additional information

Response property selection

When the property is returned several times in the GraphQL reponse, you can define the occurrence number, to select the required property or group of property.

Example:

data.x3MasterData.customer.query.edges[2].node

Element of the response validation

You can use the step definitions (7) to verify the content of the GraphQL response with exact match or step definition (9) to do a negative test.

• The information provided in parameter should be consistent with the json syntax.
• To format the json “in line” or “in block”, you can use notepad++ plugin (e.g JsonTools) , to pretty or compress the json.
  

• If you don't provide a correct json syntax, the following error will be returned at execution time:
  SyntaxError: Unexpected end of JSON input

• Example for formatting:
  

When using the step definition (8 with contain), simple text format is expected (json format is not allowed).

• Example:
  

• In case multiple verifications are required, you have to repeat the step definition for each verification needed.

Allure report

In allure report you can find the different GraphQL attachments attached to the specific step definitions.

Query / Mutation attachment

• Attachment corresponds to the query / mutation GraphQL file defined in the test.
  

Expected / Actual response attachment

• Expected response attachment corresponds to the GraphQL response file defined in the test.

• Actual attachment corresponds to the GraphQL response returned by SageX3 application.
  

Actual response attachment

• Attachment of the actual response returned by Sage X3 application.

• It is use for the mutation, where it is not possible to define a GraphQL response file in the test as some elements of the response are changing at each execution.
  
  

Allure pdf report

When printing allure report in pdf that contains GraphQL tests, it is also possible to print in the pdf file the allure attachment.

• Refer to Setting up Allure pdf functionality and make sure the file allure-config/allure-configuration.json contains the following properties.

• You can copy past them from allure-config/allure-configuration_template

Properties to add in "PageLayout"

• "DisplayTextAttachment": true

Properties to add in "TextFormat"

• "TextAttachmentFont": "Helvetica",

• "TextAttachmentSize": 7,

• "TextAttachmentFormatOptions": {"Bold": false, "Italic": false, "Underline": false, "Color": "black"}

Verifying XML files and controlling E-Invoice / E-credit-note conformance validation

XML File content verification - step definitions

Those step definitions allows to control XML files content.

Type	Step Definitions
Files - renaming	the user renames the file "(.*)" with name "(.*)" the value can be a concatenation of text & (user) stored value the file is searched in the path defined by the parameter downloadFolder
XML - read files	the user reads the "(.*)" XML file the value can be a concatenation of text & (user) stored value the file is searched in the path defined by the parameter downloadFolder
XML - select the XML path	the user selects the "(.*)" XML path and occurrence (.*)
XML - verifies value	the value of the XML path is "(.*)" the value can be a concatenation of text & (user) stored value
XML - count the number of occurences	the number of occurrences for XML path "(.*)" is (.*)
XML - verify the content (for less technical user)	the XML files contains """ """ the value can be a concatenation of text & (user) stored value

XML Conformance validation - step definitions

Those step definitions allow to control the conformance validation of the E-invoice / E-credit-note

The following step definition execute an API provided by the following site https://www.itb.ec.europa.eu/invoice/upload.

Type	Step definitions
E-Invoice / E-Credit note - run API	the user verifies the (e-invoice|e-credit-note) conformance
E-Invoice / E-Credit note - Control API response status	the (e-invoice|e-credit-note) conformance status is (SUCCESS|FAILURE)
E-Invoice / E-Credit note - Control the Error or warning in the response	the (e-invoice|e-credit-note) conformance validation returns the (warning|error) """ """
E-Invoice / E-Credit note - Control the response doesn't contain any error or warning or messages	the (e-invoice|e-credit-note) conformance validation returns no (ERROR|WARNING|MESSAGE)

API Status code

Status code	Status code description
200	Success (for successful validation)
404	Not found (for an invalid domain value)
500	Error (If a problem occurred with processing the request)

Allure report

In allure report you can find the different XML files attachments

XML read file attachment

When reading the XML file, the XML file is attached to the allure report

Expected / Actual response attachment

When controlling the content of the XML file

• In case of success
  

• In case of error
  

Conformance validation attachment

The XML information sent to API is attached to the allure report.

Conformance validation Actual API status attachment

The response provided by the API Is attached to the allure report.

Logging out of Sage X3

Version compatibility