Amazon has taken an important step in the world of artificial intelligence with the launch of S3 Vectors, the first cloud storage service with native support for large-scale vectors. This innovation promises to reduce costs by up to 90% for uploading, storing, and querying vector data.
What are vectors and why do we care?
Vectors are numerical representations of unstructured data (text, images, audio, video) generated by embedding models. They are the foundation of generative AI applications that need to find similarities between data using distance metrics.
If you’ve worked with RAG (Retrieval-Augmented Generation) applications or semantic searches, you already know that vector storage can be expensive and complex. Until now, we’ve had to manage specialized vector databases or use solutions that don’t scale economically well.
The S3 Vectors proposal
S3 Vectors introduces a new concept: vector buckets. These are specialized buckets with dedicated APIs for storing, accessing, and querying vector data without needing to provision infrastructure.
Key features:
- Vector indexes: Each bucket can have up to 10,000 vector indexes
- Scalability: Each index can store tens of millions of vectors
- Metadata: Each vector can have metadata as key-value pairs for filtering queries
- Automatic optimization: S3 automatically optimizes vector data for better price-performance
- Distance metrics: Support for Cosine and Euclidean
Practical example with Amazon Bedrock
The typical workflow would be to generate embeddings and store them in S3 Vectors. Here’s an example using Python:
import boto3
import json
# Client for Bedrock (embeddings)
bedrock = boto3.client("bedrock-runtime", region_name="us-west-2")
# Client for S3 Vectors
s3vectors = boto3.client('s3vectors', region_name='us-west-2')
# Generate embeddings for text
texts = [
"Docker containers tutorial",
"Nginx SSL configuration guide",
"GitHub Actions automation"
]
embeddings = []
for text in texts:
body = json.dumps({"inputText": text})
response = bedrock.invoke_model(
modelId='amazon.titan-embed-text-v2:0',
body=body
)
response_body = json.loads(response['body'].read())
embeddings.append(response_body['embedding'])
# Insert vectors into S3 Vectors
s3vectors.put_vectors(
vectorBucketName="my-blog-vectors",
indexName="articles-index",
vectors=[
{
"key": "docker-tutorial",
"data": {"float32": embeddings[0]},
"metadata": {"category": "DevOps", "topic": "containers"}
},
{
"key": "nginx-ssl",
"data": {"float32": embeddings[1]},
"metadata": {"category": "Systems", "topic": "web-server"}
},
{
"key": "github-actions",
"data": {"float32": embeddings[2]},
"metadata": {"category": "DevOps", "topic": "CI/CD"}
}
]
)
Interesting integrations
Amazon Bedrock Knowledge Bases: S3 Vectors integrates directly for more economical RAG applications.
Amazon OpenSearch: You can keep less queried vectors in S3 Vectors and move the most active ones to OpenSearch for real-time queries.
SageMaker Unified Studio: Knowledge base management directly from the unified environment.
My initial impression
As someone who has dealt with the cost of vector databases, this AWS proposal seems very interesting to me. The approach of separating storage (economical on S3) from queries (fast on OpenSearch when necessary) makes a lot of sense.
The fact that it’s a managed service eliminates the complexity of maintaining vector infrastructure, which is always welcome when working with large datasets.
Availability and next steps
S3 Vectors is in preview in US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Frankfurt), and Asia Pacific (Sydney).
If you’re working with AI applications that handle vectors, it’s worth checking out. The promised 90% cost reduction is significant, although as always, you’ll need to test with real data to confirm these numbers.
The native integration with the AWS ecosystem (Bedrock, OpenSearch, SageMaker) makes it a very attractive option if you’re already on that cloud.
Comments