- Task Instructions
- API Response Data
- Retrieved Knowledge Base Articles
Acceptance Testing Objectives
- Ensure GenerativeAgent does not make mistakes given expected inputs
- Focus on preventing potential hallucinations or bad inputs
- Ensure GenerativeAgent handles expected customer scenarios correctly
Testing Process
Pretesting
In the pretesting phase, keep in mind cases like the following use case scenarios: Reading a sample of production scenarios for this task:- Read summaries for 100 sample conversations to understand typical conversations within this use case across both the virtual agent and those that escalate to a live agent
- Have clear should/must-dos for each task
- Have a clear idea of the things that GenerativeAgent should do vs. must do within each task
- Keep in mind the common scenarios you expect users to go through based on the sample of real conversations
- Clear test users to do the testing
- Consider the permutations of test data that are important to cover. For example:
- Someone with a flight canceled a few minutes ago
- Someone with two flights, one which is canceled and one which is not
- Someone with elite status vs. someone with no status
Testing GenerativeAgent
Once you’ve completed the pretesting phase, you’re ready to start testing GenerativeAgent itself. This phase involves simulating real-world scenarios and interactions to ensure GenerativeAgent performs as expected. Here are some key points to keep in mind:- Aim to test approximately 100 conversations per use case
- Go through the expected conversation scenarios, as relevant, for each of the test users
- Make sure to operate in a manner that is consistent with the data in the test account you are using
- Formulate questions, based on the sample of conversations, that aim to test the knowledge articles available to GenerativeAgent
- Plan to repeat some scenarios with slight variation to ensure GenerativeAgent responses are consistent (though no response is likely to ever be exactly the same due to its generative nature)
Example Test
The following is an example scenario of Functional Testing for a task.Test Scenario
If a customer asks about their flight status, GenerativeAgent should provide the relevant details.Preconditions
A correct confirmation number and last nameTest Procedure
- IF a customer asks about their current flight status
- THEN GenerativeAgent will invoke the flight_status task
- AND GenerativeAgent will request the necessary criteria to look up the customer’s flight details
- AND if the customer provides a valid confirmation number and last name
- THEN GenerativeAgent will call the appropriate API
- AND GenerativeAgent will retrieve the required information
- AND GenerativeAgent will inform the customer of their current flight status based on the API response
Test Objectives
- Confirm that GenerativeAgent correctly invokes the flight_status task
- Verify that GenerativeAgent identifies the necessary information from the customer to verify the flight
- Ensure that GenerativeAgent requests the required information (confirmation number and last name)
- Check that the appropriate API is called
- Validate the information provided by the customer through the API
- Ensure GenerativeAgent gathers the necessary flight status information
- Confirm GenerativeAgent accurately communicates the flight status to the customer