Sign up to save tools and stay up to date with the latest in AI
bg
bg
1

AWS brings prompt routing and caching to its Bedrock LLM service | TechCrunch

Dec 04, 2024 - techcrunch.com
At the re:invent conference, AWS announced new features for its Bedrock LLM hosting service aimed at reducing costs and improving efficiency. The first feature is a caching service that prevents the model from reprocessing the same queries repeatedly, potentially reducing costs by up to 90% and lowering latency by up to 85%. The second feature is intelligent prompt routing, which automatically routes prompts to different models in the same model family based on the predicted performance of each model for a given query.

Additionally, AWS is launching a new marketplace for Bedrock, offering about 100 emerging and specialized models. While AWS partners with many large model providers, the marketplace is designed to support hundreds of specialized models that may only have a few dedicated users. Users of these models will need to manage the capacity of their infrastructure themselves, a task typically handled automatically by Bedrock.

Key takeaways:

  • AWS announced new features for its Bedrock LLM hosting service at the re:invent conference, including caching and intelligent prompt routing.
  • Caching can reduce cost by up to 90% and lower the latency for getting an answer back from the model by up to 85%.
  • Intelligent prompt routing allows Bedrock to automatically route prompts to different models in the same model family, balancing performance and cost.
  • AWS is also launching a new marketplace for Bedrock, offering about 100 emerging and specialized models, with more to come.
View Full Article

Comments (0)

Be the first to comment!