Blog

The problem: httpfs ignores your environment variables

If you work with DuckDB and the httpfs extension to read remote Parquet files, CSVs from S3, or any HTTP resource, you probably assume that the HTTP_PROXY and HTTPS_PROXY environment variables work just like every other tool. Curl respects them. wget respects them. Python requests respects them. Node.js respects them.

DuckDB does not.

I ran into this while working in a corporate environment with a mandatory proxy. I had a script reading Parquet files from Google Cloud Storage using httpfs, and it simply would not work. No clear error, no descriptive timeout, just silence. Meanwhile, a curl to the same resource with the same environment variables returned data without issue.

Every query starts with a plan. Every slow query probably starts with a bad one. And more often than not, the statistics are to blame. But how does it really work?

PostgreSQL doesn’t run the query to find out — it estimates the cost. It reads pre-computed data from pg_class and pg_statistic and does the maths to figure out the cheapest path to your data.

In the ideal scenario, the numbers read are accurate, and you get the plan you expect. But when they’re stale, the situation gets out of control. The planner estimates 500 rows, plans a nested loop, and hits 25,000. What seemed like an optimal plan turns into a cascading failure.

I recently came across an exceptionally dense technical analysis about container security that’s worth sharing. The author started with a simple hypothesis: container filesystem isolation should be sufficient for multi-tenant workloads without virtual machines, if you sufficiently understand what’s happening at the syscall level.

After thorough investigation, the conclusion is more uncomfortable than expected: the defaults protect you well, but the moment you reach for “advanced” features like bidirectional mount propagation or SELinux relabeling, you’re one misconfiguration away from handing an attacker the keys to your host.

Greg Brockman, President and Co-Founder of OpenAI, recently published a thread that perfectly describes the moment we’re living in software development. According to him, we’re witnessing a genuine renaissance in software development, driven by AI tools that have improved exponentially since December.

The qualitative leap

The most striking part of Brockman’s thread is how they describe the internal change at OpenAI: engineers who previously used Codex for unit tests now see the tool writing practically all code and handling a large portion of operations and debugging. This isn’t an incremental improvement, it’s a paradigm shift.

A few days ago I was reading a fascinating essay by David Bessis that made me think deeply about something we all take for granted: intelligence. Bessis, a mathematician and author of Mathematica: A Secret World of Intuition and Curiosity, presents a thesis that challenges many of our deeply held beliefs about the origin of cognitive differences between people.

What’s most interesting about his approach is not so much the conclusion -which I’ll get to- but the path he traces to get there, a path that mixes mathematics, neuroscience, and a good dose of personal experience.

A few days ago I discovered Agent Lightning, a Microsoft project that I believe marks a before and after in how we think about AI agent orchestration. It’s not just another library; it’s a serious attempt to standardize how we build multi-agent systems.

What is Agent Lightning?

Agent Lightning is a Microsoft framework for orchestrating AI agents. It enables composition, integration, and deployment of multi-agent systems in a modular and scalable way. The premise is simple but powerful: agents should be components that can be combined, connected, and reused.

A few days ago I watched a video that has given me a lot to think about. Jeffrey Way, founder of Laracasts and one of the most influential people in the Laravel/PHP community, shared a brutally honest reflection on how artificial intelligence is transforming his business and his profession.

The video starts with a phrase that leaves you cold: “I’m done”. It’s not a goodbye to programming, but an acceptance of the reality to come.

A few days ago I discovered Tangible Media Collection, a website that has resonated deeply with me. It’s a collection of some 1700 objects related to information storage: from vinyl records and magnetic tapes to punch cards and optical discs. But it’s not just a collection of technological objects; it’s a testament to human curiosity.

The Collection Born from a Lost Tape

The story of how this collection started is fascinating. In 2001, John Wallace (the collector) found a computer tape from the late 80s in a desk drawer. He knew what it contained: an archive of computer-generated images from his grad school days, the product of late nights and weekends in the computer lab. But the tape drive that could read it was long gone.

Vercel has announced the general availability of Vercel Sandbox, an execution layer designed specifically for AI agents. But beyond the AI agent hype, there’s an interesting question: can it be useful for running code safely in different languages like PHP, Node, or Go?

What is Vercel Sandbox?

Vercel Sandbox provides on-demand Linux microVMs. Each sandbox is isolated, with its own filesystem, network, and process space. You get sudo access, package managers, and the ability to run the same commands you’d run on a Linux machine.

If you’re using tools like Claude Code, GitHub Copilot Workspace, or similar, you’ve probably noticed there’s technical jargon that goes beyond simply “chatting with AI”. I’m talking about terms like rules, commands, skills, MCP, and hooks.

These concepts are the architecture that makes AI agents truly useful for software development. They’re not just fancy marketing words — each one serves a specific function in how the agent works.

Let’s break them down one by one in a clear way.

Lately I’ve been working quite a bit with DuckDB, and one of the things that interests me most is understanding how to optimize performance according to the file format we’re using.

It’s not the same working with Parquet, compressed CSV, or uncompressed CSV. And the performance differences can be dramatic.

Let’s review the key optimizations to keep in mind when working with different file formats in DuckDB.

Parquet: Direct Query or Load First?

DuckDB has advanced Parquet support, including the ability to query Parquet files directly without loading them into the database. But when should you do one or the other?

Lately I’m seeing more and more AI models calling themselves “open source”. Llama, Mistral, Falcon… they all claim to be “open”. But are they really? How open are they actually?

I recently discovered the European Open Source AI Index (OSAI), a European initiative doing excellent work systematically evaluating how open generative AI models really are.