bedrock: How to tokenizer and count tokenizer?

[kerlion]

It seems that bedrock has lost a lot of features. such as:
How to do tokenization and count tokenizer?

And is there any api guide? I can find 2 simple code examples.

[salahaz]

Any updates on this ?

[zwluoqi]

wait

[bbhorrigan]

Any update on this coming? I am trying to find some examples.

[olu-an]

Hi @kerlion, A good approach you can consider is to conservatively estimate token count based on historical usage e.g. using byte counts and/or mapping from the usage field in the response.

You can find an example response with the usage field here: https://docs.anthropic.com/en/api/messages-examples#basic-request-and-response.

Please reach out if you have more questions or need further guidance.