Test Scenarios

Learn how to create and use Test Scenarios to test conversations with GenerativeAgent.

A Test Scenario simulates a customer profile to test conversations with GenerativeAgent.

These scenarios help you define both the mock API responses and the customer’s side of the interaction—goals, known information, and personality—so you can assess how GenerativeAgent handles different scenarios, edge cases, and potential issues.

Use Test Scenarios to test:

• Key happy-flows
• Edge cases
• Common problems or issues

Create a Test Scenario

To create a Test Scenario:

1. Navigate to Test Scenarios

   Go to the Scenario Testing page in GenerativeAgent and click ”+ New test scenario” to begin creating a new test scenario.

2. Define Basic Information

   Start by providing the essential details that identify and describe your test scenario:

   • Test name: A clear, descriptive name for your scenario
   • Description: (Optional) A brief explanation of what you’re testing and what you expect to achieve
   • Tags: (Optional) Add relevant tags to categorize and easily find your scenario later
   • Test suite: (Optional) Enable it if you want to add this scenario to a test suite - a group of related test scenarios that can be run together.
3. Configure API Profile

   Set up the User API Profile to define the mock responses for your test case.

4. (Optional) Set Input Context

   Add optional settings like Starting Task or Input Variables to define how the test conversation begins.

5. (Optional) Define Simulation Profile

   If you are using the automatic mode, you can configure the customer’s behavior and goals by filling in the Simulation Details section. Toggle evaluations on.

6. (Optional) Add Evaluations

   Toggle evaluations on and optionally add Applicability Criteria and Evaluation Criteria to assess the conversation outcomes.

7. Save your Test Scenario

   After completing these steps, save your Test Scenario and start running it in either manual or automatic simulation mode.

You can now preview the test scenario directly from the test file after saving your changes.

Configure User API Profile

The data that GenerativeAgent uses plays a critical role in how it behaves. The User API Profile defines the mock data that will be provided to GenerativeAgent when using this test scenario.

You have two options to mock data:

1. Auto-generate mock data.
2. Manually define functions

Auto-generate mock data

Auto-generate mock data creates function mocks using the scenario you describe and the schema of the functions in the main branch of the draft environment.

To auto-generate mock data:

1. Click Generate.
2. Describe your scenario in a few sentences.
3. Wait up to one minute for the profile to be created (you can close the dialog while you wait).

The system pulls mocked data schemas from the main branch of the draft environment. To test a new function, it needs to have been added to the main branch of the draft environment.

Adding details about what information the customer will have to share with GenerativeAgent and what GenerativeAgent will have to do will significantly improve the ability to auto-mock the appropriate APIs.

Manually define functions

If you know the specific data you want to mock for a scenario, you can manually define the functions and mock data.

1. Click + Select functions.
2. Choose the API calls you want to mock (e.g. getAccountInfo, confirmCode).
3. For each function, provide the mocked request and response in JSON.

Add variants to mocked data

To simulate different API responses under the same function:

1. Click + Add variants at the bottom of a function.
2. Define alternative request parameters and corresponding response schema.
3. Save to include the variant in your scenario.

Updating Mock data

You may need to update the mock data after creating a Test Scenario. After creating a Test Scenario, ensure your mock data aligns with your testing needs.

This can be done by updating your scenario when auto-generating the mock data or manually adding the functions and mock data.

Date & time override (optional)

Often APIs have an implicit or explicit date and time when the system performs them. By default, GenerativeAgent assumes the customer interaction is happening at the date and time the scenario is run and updates timestamps in the API profile accordingly.

You can override the assumed timestamp of the interaction.

Set Input Context

The Input Context defines the initial data and configuration that your system provides to GenerativeAgent when a conversation begins. This ensures your test scenarios accurately simulate real-world interactions.

The input context consists of two main components:

1. Input Variables: JSON data that provides context about the customer, conversation, or session
2. Starting Task: The specific task or flow that GenerativeAgent should begin with

Ensure your input context matches exactly how your production system passes data to GenerativeAgent. This guarantees your test scenarios will match real conversations identically.

Define Simulation Details

Simulation details allow you to configure the customer’s behavior. This is especially useful for testing how GenerativeAgent handles different customer personalities, levels of information, and goals.

The main components of the simulation details include:

• Start of conversation: Define who initiates the conversation: the customer or GenerativeAgent.
• Customer Goals: Define the motivations of the simulated customer. The customer will attempt each goal in sequence; if it fails, it may escalate or skip.
• Information the customer knows: Define the identity or account data the customer can provide. To test how GenerativeAgent will handle a customer who does not have their account number or a critical piece of information, just leave it off the list.
• Customer personality: Define the tone and style for your simulated customer (optional). eg. rude, insistent, confused, etc. Click Revert to default personality to restore.
• Max number of turns: Set the maximum number of conversation turns for the simulation. By default, it is set to 10 turns, but you can adjust it based on your testing needs. This is especially useful for testing longer conversations or ensuring that the simulation runs until a specific outcome is reached. Additionally you can set the max number of turns the simulation can run. By default, it is set to 15 turns.

Add Evaluations

When evaluations are toggled on, the system runs evaluation logic after the conversation ends.

Applicability Criteria (optional)

Define preconditions or milestones that must be met in the conversation for the evaluation to run. This is useful to disqualify cases where the conversation didn’t reach the part you’re testing.

If left blank, the evaluation runs by default.

Example:

“Customer asked for their bill amount”

Evaluation Criteria

Define one or more evaluation checks to apply to the conversation.

Examples:

“The Agent provided the current month’s bill amount of $144.72” “The Agent did NOT offer a discount”

Evaluation results appear in the Previewer, side-by-side with the conversation after clicking on the Results button.

If you update your evaluation or applicability criteria later, you can return to the same conversation in the Previewer and click Run Eval Again to re-run against the updated test.

Run Test Scenarios

Test scenarios are run in the Scenario Runner on the side-panel.

To run a Test Scenario:

1. Navigate to the Scenario Runner on the side-panel.
2. Click + New test run.
3. Create Test Run by defining:
   • Test name: A clear, descriptive name for your test run
   • Test selection: Select to either run a Test Suite (a group of related Test Scenarios) or an individual Test Scenario. To run an individual Test Scenario, enable Tags and select the tag of the scenario you want to run.
   • Number of runs: Define how many times you want to run the selected scenario(s). This is especially useful for testing the consistency of GenerativeAgent’s behavior across multiple runs.
   • Simulation channel: Select the channel you want to use for the simulation: chat or voice.
   • Environment: Select the environment you want to use for the simulation: draft, sandbox, or production.
   
4. Click Run to begin the test.

Once the conversation ends:

• Click Results to open detailed evaluation outcomes.
• Click Run again to re-run evaluations on the conversation in the previewer.

Next steps

Input Variables

Learn how to use Input Variables to provide context to GenerativeAgent in your test scenarios.

Enter specific task

Learn how to set a specific starting task for GenerativeAgent in your test scenarios.

System Transfer Function

Learn how to create and use System Transfer Functions to transfer control to an external system in your conversations.

Previewer

Learn how to use the Previewer to analyze conversation details and evaluation outcomes for your test scenarios.