Additionally, AWS is launching a new marketplace for Bedrock, offering about 100 emerging and specialized models. While AWS partners with many large model providers, the marketplace is designed to support hundreds of specialized models that may only have a few dedicated users. Users of these models will need to manage the capacity of their infrastructure themselves, a task typically handled automatically by Bedrock.
Key takeaways:
- AWS announced new features for its Bedrock LLM hosting service at the re:invent conference, including caching and intelligent prompt routing.
- Caching can reduce cost by up to 90% and lower the latency for getting an answer back from the model by up to 85%.
- Intelligent prompt routing allows Bedrock to automatically route prompts to different models in the same model family, balancing performance and cost.
- AWS is also launching a new marketplace for Bedrock, offering about 100 emerging and specialized models, with more to come.