Tokens processed in Amazon Bedrock

IMO It's potentially misleading for the re:Post Agent to say conversation tokens are "typically cumulative", but it's correct that in this case your total tokens processed should be the sum of all the reported numbers.

Agents in Amazon Bedrock support session state which is used to persist information over multiple turns of conversation and remember current context when generating a reply. This contributes to the increasing inputTokens count you see in each turn, but you can refer to the linked doc for options to customize the session management.

In plain Converse or InvokeModel calls, you'd need to manage this session state on the client side: Storing the previous conversation history and passing it in each time, which makes it a bit clearer how & why the inputTokens counts would accumulate each time.

In either case, the inputTokens and outputTokens returned in each API response should be the counts processed for that particular request... So to monitor usage for billing tracking purposes, you'd add them all up i.e. 300+600+900+1200 input tokens, 200+230+300+400 output tokens.

In Amazon Bedrock, the input tokens for each interaction in a conversation are typically cumulative. This means that the model maintains context from previous interactions within the same session.

Based on the information provided in your example:

1. The first prompt processed 300 input tokens.
2. The second prompt processed 600 input tokens, which likely includes the previous 300 tokens plus the new input.
3. The third prompt processed 900 input tokens, again including all previous interactions.
4. The final prompt processed 1200 input tokens, which would include all previous interactions plus the new input.

This cumulative approach allows the model to maintain context throughout the conversation, enabling it to provide more coherent and contextually relevant responses. The model is not just processing the last 300 tokens, but rather the entire conversation history up to that point.

It's important to note that there are usually token limits for models, and if the cumulative input exceeds this limit, older parts of the conversation may be truncated to fit within the model's context window. The exact behavior can depend on the specific model and configuration being used in Amazon Bedrock.

When working with agents in Amazon Bedrock, it's good practice to be aware of these token counts and manage your conversation length accordingly to ensure you stay within the model's capabilities and avoid unexpected truncation of important context.

Sources
Build generative AI applications on Amazon Bedrock with the AWS SDK for Python (Boto3) | AWS Machine Learning Blog