Set Prompt Cache Sharing for a Policy
Choose whether clients share the prompt cache or each keep their own. You set this on the policy that their workspace follows. For what each choice means and the risk that sharing brings, refer to Prompt Cache Sharing. For the labels on the Policies tab, refer to Access Management Reference. If a repeated prompt shows no cache reads, refer to Troubleshoot a Repeated Prompt with No Cache Reads.
Prerequisites
-
A running PaletteAI Inference Launchpad appliance, with the console reachable.
-
Console access as a person who holds the Platform administrator role.
Change Cache Sharing on a Policy
-
In the workspace drop-down menu in the console header, select All workspaces.
-
From the left main menu, select Access Management.
-
Select the Policies tab.
-
In the policy's row, check the Followed by column for the workspaces whose clients the change affects.
-
In the policy's row, open the three-dot menu, and then select Edit. The policy's page opens, titled with the policy name.
-
Select the Cache sharing section.
-
In the Prompt cache sharing drop-down menu, select Share within the workspace or Each client is isolated.
-
Select Save changes.
-
Repeat steps 4 through 8 for each policy you want to change.
Choose Cache Sharing on a New Policy
-
In the workspace drop-down menu in the console header, select All workspaces.
-
From the left main menu, select Access Management.
-
Select the Policies tab, and then select Create Policy. The Create policy wizard opens on the Overview step.
-
Complete the Overview, Quotas, and External access steps.
-
On the Cache sharing step, in the Prompt cache sharing drop-down menu, select Share within the workspace or Each client is isolated, and then select Next.
-
Complete the Routing step, and then select Create policy.
-
Select the Workspaces tab.
-
In the row of the workspace that should follow every setting of the new policy, open the three-dot menu, and then select Manage.
-
In the Policy field, select the new policy, and then select Save changes.
To give a team a new workspace that follows the policy instead, refer to Create a Workspace.
Validate
-
In the workspace drop-down menu in the console header, select All workspaces.
-
From the left main menu, select Access Management.
-
Select the Policies tab.
-
In the policy's row, confirm that the Cache sharing column reads Workspace for Share within the workspace, or Each client for Each client is isolated.
-
Confirm that the Followed by column lists the workspaces whose clients you meant to change.
Troubleshoot a Repeated Prompt with No Cache Reads
Use these steps when a second client in the same workspace sends the same opening to the same model and gets zero cache reads. For why this can happen, refer to Models on Two or More Nodes.
-
Check which choice the workspace's policy uses, as described in Validate.
-
If it uses Each client is isolated on purpose, no action is required.
-
If it uses Each client is isolated and you want the clients to share, change the choice, as described in Change Cache Sharing on a Policy.
-
If it uses Share within the workspace, go to the next step.
-
-
From the left main menu, select Cluster, and then select the Models tab.
-
Expand the model row, and then count the nodes that run the model.
-
If the model runs on two or more nodes, no action is required. For why, refer to Why a Repeated Prompt Shows No Cache Reads.
-
If the model runs on one node and the cache sharing choice changed between the two requests, no action is required. For why, refer to When a Change Takes Effect.
-
If the model runs on one node and the choice did not change, something is not working as expected. Contact Spectro Cloud support at support@spectrocloud.com.
-
Next Steps
-
Create a Workspace to give a team its own workspace that follows a policy you set.
-
View Token Usage to check how many prompt tokens came from the cache.