Is the account being charged for timeout?

Anonymous
2023-06-18T21:14:35.0033333+00:00

If a request to GPT-4 hits the 600s limit resulting in a timeout error and no generation:

(1) Is the account still being charged for the prompt?

(2) Does this count to the tokens rate limit? (If so, does it count only the tokens in the prompt or prompt + max tok)

Azure OpenAI in Foundry Models

Answer accepted by question author
AshokPeddakotla-MSFT 35,346 Reputation points Moderator
2023-06-19T03:13:33.3966667+00:00

Marzena Karpinska Greetings & Welcome to Microsoft Q&A forum!

(1) Is the account still being charged for the prompt?

The account will not be charged for the prompt if the request times out and no generation occurs. Azure OpenAI Service only charges for successful API requests that generate a response.

(2) Does this count to the tokens rate limit? (If so, does it count only the tokens in the prompt or prompt + max tok)

If the request times out and no generation occurs, it will not count towards the token rate limit. The token rate limit is based on the number of successful API requests that generate a response, not on the number of requests that are made. However, if the request includes a prompt and a maximum number of tokens, both the prompt and the maximum number of tokens will be counted towards the token rate limit if the request is successful and generates a response.

Please see Monitoring Azure OpenAI Service on available metrics and options to enable logging.

Also, See How to request increases to the default quotas and limits

If you are seeing these errors consistently then you probably could also apply for a quota increase request through this form.

Do let us know if that helps or have any further queries.

Was this answer helpful?

1 person found this answer helpful.
0 comments No comments

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.