Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature]: Cloudflare AI Gateway support for Bedrock #1040

Open
Manouchehri opened this issue Dec 7, 2023 · 4 comments · May be fixed by #3467
Open

[Feature]: Cloudflare AI Gateway support for Bedrock #1040

Manouchehri opened this issue Dec 7, 2023 · 4 comments · May be fixed by #3467
Assignees
@Manouchehri
Labels
enhancement New feature or request

Comments

@Manouchehri
Copy link
Collaborator

The Feature

The docs aren't added yet, but Cloudflare has added AI Gateway support for Amazon Bedrock.

https://developers.cloudflare.com/ai-gateway/providers/

The example request looks like:

curl -X POST https://gateway.ai.cloudflare.com/v1/account_id_here/gateway_name_here/aws-bedrock/bedrock-runtime/us-east-1/model/amazon.titan-embed-text-v1/invoke \
  -v --aws-sigv4 aws:amz:us-east-1:bedrock \
  -H 'Accept: */*' \
  -H 'Content-Type: application/json' \
  -u key:secret \
  -d '{
    "prompt": "Where did the phrase Hello World come from"
  }'

Motivation, pitch

Free caching :D

Twitter / LinkedIn details

https://www.linkedin.com/in/davidmanouchehri/
@Manouchehri Manouchehri added the enhancement New feature or request label Dec 7, 2023
@Manouchehri
Copy link
Collaborator Author

Lower priority since we aren't using Amazon Bedrock as much after discovering that our AWS Activate credits don't apply to Anthropic or Cohere models on Bedrock. =P

This was referenced Dec 26, 2023
Closed
Closed
@krrishdholakia krrishdholakia mentioned this issue Jan 3, 2024
Closed
@Manouchehri
Copy link
Collaborator Author

Small note: I'm not even 100% sure how to use Cloudflare AI Gateway with Bedrock (ignoring LiteLLM), so don't worry about this ticket at all in the meantime. Not a high priority at all imo.

@Manouchehri
Copy link
Collaborator Author

This is doable using boto3 hooks. I helped Cloudflare figure this one out, they will eventually be documenting it at https://developers.cloudflare.com/ai-gateway/providers/bedrock/

In the meantime, gonna save my notes here just in case.

https://stackoverflow.com/a/66097613/2079814

event_system.register_first('after-sign.*.*', _add_header)

or:

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html#before-send

before-send looks like it'd work too.

Unlike my example, you should only only boto3 requests going to bedrock invoke and invoke stream, since hooking others is just gonna break things if eventually the same boto3 client is used to do other API calls in the future.

@Manouchehri
Copy link
Collaborator Author

JavaScript/TypeScript solution here: https://gist.github.com/Manouchehri/d1fa65b6d842a76d836cb6b3f9e1bbff

@Manouchehri Manouchehri self-assigned this Apr 12, 2024
@Manouchehri Manouchehri linked a pull request May 5, 2024 that will close this issue
Open
4 tasks
This was referenced May 16, 2024
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Manouchehri Manouchehri

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(bedrock.py): Add Cloudflare AI Gateway support Manouchehri/litellm
1 participant
@Manouchehri