AWS Acquires DuckLabs: What Changes and What Doesn't for DuckDB Users
5 min read

AWS Acquires DuckLabs: What Changes and What Doesn't for DuckDB Users

947 words

On August 26th, Mark Raasveldt and Hannes Mühleisen published a very short note on the DuckDB blog: DuckLabs, the company behind DuckDB, is becoming a subsidiary of Amazon Web Services. The deal closes in early September.

I read it more carefully than usual because at CARTO we’ve been using DuckDB for a relatively short time, and I’ve written here about file formats and performance and about the httpfs proxy mess. When you adopt a piece of software in production and months later a hyperscaler buys it, the least you can do is sit down and read the fine print.

What the announcement says

It’s short, so it’s worth carefully separating the two entities involved, because the headline blurs them easily.

DuckLabs is the company. That’s what AWS is buying.

DuckDB is the open source project. It’s held by the DuckDB Foundation, a non-profit, and it is not part of the deal.

On that separation, the announcement commits to four specific points:

Commitment
LicenseMIT, unchanged
GovernanceStays under the DuckDB Foundation
RoadmapUnchanged
ScopeDuckDB, DuckLake, Quack and the other extensions

And two items that are more than just continuity: the Foundation will set up a stakeholder advisory board able to influence the projects’ direction, and the limitations on community support are being lifted.

That second point is the one I find most interesting, and I’ll come back to it.

Why the structure matters here

The project living in a foundation separate from the company isn’t a bureaucratic detail. It’s exactly what separates this news from others we’ve lived through worse.

When a company owns the copyright to its own open source project, it can relicense it. We’ve seen it several times over the past decade: a switch to a non-OSI license, the community reacts, a fork appears, and everyone running it in production spends six months deciding which branch to follow. Redis, Elasticsearch, Terraform, HashiCorp in general. The pattern repeats.

Here, ownership was already outside the company before the acquisition. AWS acquires DuckLabs — the team, the business, the commercial support — but it doesn’t acquire the ability to relicense DuckDB, because that ability was never DuckLabs’ to begin with. The Foundation has existed for years precisely for this.

Put differently: the protection doesn’t come from the promise in the announcement, it comes from the legal structure that was already in place. That’s an important distinction, because continuity promises after an acquisition are cheap and we’ve all watched a few evaporate.

What does change

It would be naive to read this as “nothing changes”. Things do change, just not the ones the headline makes you worry about.

Support is no longer limited. This is AWS money funding something that until now had to be rationed. For anyone running DuckDB in production it’s probably the best news in the announcement, and it’s purely positive.

Integration with the AWS ecosystem will accelerate. That’s the obvious economic rationale for the purchase. Expect better S3 support, better Iceberg, better Glue catalog. None of that hurts anyone not on AWS, but it does mark where the effort will concentrate.

Roadmap bias. This is where I’d focus attention over the medium term. The roadmap doesn’t change today, but the team deciding it is now paid by AWS. When it comes to prioritizing between an improvement that benefits S3 and one that benefits Azure Blob, the pressure won’t be explicit: it’ll be a matter of who you sit across from every day. The stakeholder advisory board seems designed precisely to counterbalance that, and its composition will say considerably more than any press release.

How it looks from our case

At CARTO we use DuckDB in one specific area — data ingestion and processing — not as the platform’s core engine. That position makes this news far calmer than it would be in another scenario.

And I think that’s where the practical lesson lies, beyond this particular acquisition: what determines your exposure isn’t who buys whom, it’s how tightly you’ve coupled your architecture to that piece.

If DuckDB is a tool you use to read Parquet, transform and write, your replacement cost is real but bounded. If you’ve built on top of it with custom extensions, specific functions and dependencies on its exact behavior, then any change of direction in the project becomes your problem. The acquisition doesn’t alter that equation, it just makes it worth looking at.

It’s the same reflection I was making a few days ago about connecting to other people’s data warehouses: the part you control is small, and it pays to know exactly where it ends.

My take

I’m moderately optimistic, with one reservation.

The optimism comes from the structure. The Foundation, the MIT license, and the fact that ownership was already separate from the business are real guarantees, not statements of intent. Add that the founders continue leading the technical direction and that the project has a large community — it passed 40,000 GitHub stars earlier this month — and the catastrophic scenario looks unlikely.

The reservation is fuzzier and has nothing to do with clauses. A project funded by a hyperscaler tends, over time, to look like what that hyperscaler needs. Not out of bad faith, but out of gravity: that’s where the paying users are, the reported cases, the priorities that arrive every morning. DuckDB was born as an embedded analytical database that runs anywhere, and that “anywhere” is a good part of its appeal.

For now, there’s nothing to do beyond continuing to use it. Adoption at CARTO carries on unchanged, and these two announcements — the advisory board and who ends up sitting on it — are what’s worth watching over the coming months.

Latest Posts

7 min

1393 words

In April 2021 I made my first commit to the monorepo behind CARTO’s cloud-native platform. It was PR number 9, a docker-compose. Five years later I’ve looked back with some calm and found around 450 commits, around 447 pull requests, and a presence in virtually every service in the repository.

The numbers aren’t the interesting part. What’s interesting is that, going through that history, a thread shows up that I hadn’t fully identified myself: I’ve spent half a decade connecting the platform to other people’s data warehouses. Six different providers — BigQuery, Snowflake, Redshift, Databricks, Oracle and PostgreSQL — each with its own credential model, its pooling, its timeouts and its error messages.

4 min

767 words

The problem: httpfs ignores your environment variables

If you work with DuckDB and the httpfs extension to read remote Parquet files, CSVs from S3, or any HTTP resource, you probably assume that the HTTP_PROXY and HTTPS_PROXY environment variables work just like every other tool. Curl respects them. wget respects them. Python requests respects them. Node.js respects them.

DuckDB does not.

I ran into this while working in a corporate environment with a mandatory proxy. I had a script reading Parquet files from Google Cloud Storage using httpfs, and it simply would not work. No clear error, no descriptive timeout, just silence. Meanwhile, a curl to the same resource with the same environment variables returned data without issue.

3 min

555 words

Amazon has taken an important step in the world of artificial intelligence with the launch of S3 Vectors, the first cloud storage service with native support for large-scale vectors. This innovation promises to reduce costs by up to 90% for uploading, storing, and querying vector data.

What are vectors and why do we care?

Vectors are numerical representations of unstructured data (text, images, audio, video) generated by embedding models. They are the foundation of generative AI applications that need to find similarities between data using distance metrics.

11 min

2159 words

Every query starts with a plan. Every slow query probably starts with a bad one. And more often than not, the statistics are to blame. But how does it really work?

PostgreSQL doesn’t run the query to find out — it estimates the cost. It reads pre-computed data from pg_class and pg_statistic and does the maths to figure out the cheapest path to your data.

In the ideal scenario, the numbers read are accurate, and you get the plan you expect. But when they’re stale, the situation gets out of control. The planner estimates 500 rows, plans a nested loop, and hits 25,000. What seemed like an optimal plan turns into a cascading failure.

3 min

455 words

Lately I’ve been working quite a bit with DuckDB, and one of the things that interests me most is understanding how to optimize performance according to the file format we’re using.

It’s not the same working with Parquet, compressed CSV, or uncompressed CSV. And the performance differences can be dramatic.

Let’s review the key optimizations to keep in mind when working with different file formats in DuckDB.

Parquet: Direct Query or Load First?

DuckDB has advanced Parquet support, including the ability to query Parquet files directly without loading them into the database. But when should you do one or the other?