Trial Mode

Trial mode allows admins to safely deploy new GenerativeAgent use cases to production.

A function can be marked as being in trial mode, so that when GenerativeAgent calls that function, it instead will escalate to a human agent.

This can allow you to:

• Ensure GenerativeAgent called the function properly given the conversation context.
• Ensure GenerativeAgent interpreted the function response and responded to the customer correctly.
• Be protected from unknown API response variations that you might not have accounted for during development and testing.

After running a function in trial mode and confirming it responds as expected, you can disable trial mode, deploying the function into full production use.

Trial mode is distinct from A/B testing. Trial mode is intended to ensure a function works correctly, not comparing outcomes between two functions or version.

Using Trial Mode

Enable trial mode on functions when you want to observe how GenerativeAgent would use that function in production by forcing escalation to a live agent immediately before or after the function is called.

Enabling trial mode will temporarily reduce your containment rates because conversations are configured to escalate to a live agent instead of being fully handled by GenerativeAgent.

However, this temporary reduction in containment is a trade-off for the added safety and reliability gained from observing and validating GenerativeAgent’s behavior in a controlled environment before full deployment.

For example, suppose a new use case allows GenerativeAgent to check a customer’s refund eligibility and then issue a refund if eligible. An admin may want to gate this new task based on two functions:

• Checking the refund eligibility; and then
• Issuing the refund.

An admin may decide to configure trial mode in two phases for this use case.

Phase 1

An admin can configure GenerativeAgent to call the first function (checking the refund eligibility), but then immediately escalate to a live agent to continue resolving the customer’s issue. This would allow admins to observe how GenerativeAgent calls the refund eligibility function and how it would have interpreted and communicated the response of the function back to the customer.

Phase 2

For the second function (issuing the refund), an admin may configure that GenerativeAgent should escalate to a live agent before the function is called, as this type of function actually performs an action in a backend system. Configuring trial mode to escalate before the function is called allows admins to observe how GenerativeAgent would have called the function.

In both scenarios, trial mode lets admins observe how GenerativeAgent would have performed on production data before actually letting GenerativeAgent use the function responses to interact with the customer.

Deciding to deploy a function

You can turn off trial mode and fully deploy the function when you have gathered sufficient data and confidence that GenerativeAgent is correctly calling the function and interpreting its responses.

This can be determined by monitoring the escalations, reviewing how GenerativeAgent would have handled the interactions, and ensuring that there are no significant issues or undesired behaviors.

Toggle Trial Mode

By default, trial mode is toggled off.

When you want to enable trial mode for a function, click the toggle.

When using trial mode, you need to specify:

• Escalation behavior:
  • Before Calling: GenerativeAgent will escalate to a live agent before calling the function. This allows you to see what GenerativeAgent would have called the function.
  • After Calling: GenerativeAgent will call the function, but then escalate to an agent before responding to the customer.
• Message to send to customer: The message that will be sent to customer before escalation. Can be left blank to not send a dedicated message.

Evaluate behavior in Previewer

When trail mode is activated, you can see the trial behavior in previewer.

Escalate Before Call

When trial mode is toggled on and configured to escalate before the function is called, you can see the example function request GenerativeAgent would have made before the “Transfered to agent” event.

Escalate After Call

When trial mode is toggled on and configured to escalate after the function is called, GenerativeAgent will call the function, and then escalate to an agent.

Trial Mode vs A/B Testing

A/B tests and trial mode are two complementary functionalities that both enable safe deployment.

A/B tests are configured at the GenerativeAgent level, where a customer is either seeing the GenerativeAgent treatment or the control treatment (where the control treatment might be the treatment customers saw prior to GenerativeAgent).

Trial mode can be configured within an A/B test. For example, it could be that 10% of the traffic in an A/B test is seeing GenerativeAgent, and within that 10% trial mode is on for one or more functions.