Trial Mode

Learn about the upcoming Trial Mode feature for ASAPP.

Overview

ASAPP is adding the “Trial Mode” option to functions. This trial mode allows you to safely deploy GenerativeAgent use cases by trialing functions in production.

A function can be marked as being in trial mode, so that when GenerativeAgent calls that function, it instead will escalate to a human agent.

This can allow you to:

• Ensure GenerativeAgent called the function properly given the conversation context.
• Ensure GenerativeAgent interpreted the function response.
• Be protected from unknown API response variations that you might not have accounted for during development and testing.

After running a function in trial mode and confirming it responds as expected, you can disable trial mode, deploying the function into full production use.

Check out the Trial Mode guide for more information.

Use and Impact

One of the key challenges of configuring GenerativeAgent is ensuring it behaves as you expect. One of the biggest impacts on behavior is how GenerativeAgent calls functions, and interprets the results.

This is particularly important when interpreting API results. It is not unusual that some edge cases and other nuance of API responses were not accounted for, resulting in unexpected behavior.

Trial Mode allows you to experiment and try out new function configurations, seeing what GenerativeAgent would have done. While still escalating the issue to an agent, so that your customer’s experiences are not negatively impacted by your testing.

How it Works

1. Enable trial mode on functions that you want to observe.
2. Depending on your configuration, GenerativeAgent can either call the function before escalating to an agent, or escalate to an agent before calling the function. If escalating before, you will still be able to see the function that would have been called.
3. Observe GenerativeAgent’s use of the function during trial mode, in previewer.

FAQs

How does trial mode relate to A/B testing in terms of safe deployment? A/B tests and trial mode are two complementary functionalities that both enable safe deployment. A/B tests are configured at the GenerativeAgent level, where a customer is either seeing the GenerativeAgent treatment or the control treatment. Trial mode can be configured within an A/B test.

If I enable trial mode, will my containment rates be low? Enabling trial mode will temporarily reduce your containment rates because conversations are configured to escalate to a live agent instead of being fully handled by GenerativeAgent. However, this temporary reduction in containment is a trade-off for the added safety and reliability gained from observing and validating GenerativeAgent’s behavior in a controlled environment before full deployment.

How do I know when it’s safe to turn off trial mode and fully deploy the function? You can turn off trial mode and fully deploy the function when you have gathered sufficient data and confidence that GenerativeAgent is correctly calling the function and interpreting its responses. This can be determined by monitoring the escalations, reviewing how GenerativeAgent would have handled the interactions, and ensuring that there are no significant issues or undesired behaviors. Once you are satisfied with the performance, you can disable trial mode for that function.