Polars 2.0 Explained: Streaming Default, Spill-to-Disk and Breaking Changes

An upgrade to Polars 2.0 can make a job faster and leaner while it quietly returns different rows. Nothing crashes, so the first person to notice is often a customer.

Picture a common setup. A delivery business ran a nightly Polars job over 18 months of order lines, about 40 crore (400 million) rows. It fed a latest orders panel that support staff used to answer customer calls.

The job ran on a 32 GB server with 16 cores. Its requirements file said polars with no version limit.

On 7 October the server image was rebuilt, and the build picked up Polars 2.0.0, released the day before. The next run finished in 11 minutes instead of 26, and memory use halved.

Then a customer called about yesterday's Rs 640 order, and support could not find it. The latest orders panel was full of orders from March.

The tests had passed and nothing had crashed. The setup is made up, but the bug is real, and section 04 shows how to reproduce it.

Polars 2.0 runs lazy queries on the streaming engine by default, lets some steps spill to disk, and removes old shortcuts that could hide mistakes. It was released on 6 October 2026, and it is a release about defaults more than new features.

Two terms first. A LazyFrame is a query plan that has not run yet: you list the steps, and nothing happens until you call collect().

An engine is the part of Polars that actually runs the plan. Polars has two engines that run on the CPU. The in-memory engine loads the whole input of a step before working on it. The streaming engine works through the data in batches (slices of rows), using every CPU core at once.

Here is the picture for the whole guide: the kitchen at a big Indian wedding, cooking for 1,500 guests.

• The menu card written before cooking = the LazyFrame. Nothing is cooked yet.
• The head cook saying "start" = calling collect().
• One giant pot for the whole feast = the in-memory engine. The pot must hold everything at once.
• Ten stoves cooking small batches at the same time = the streaming engine.
• Counter space in the kitchen = the computer's memory (RAM).
• The cold store behind the kitchen = the disk, where spilled data waits until it is needed.
• Plates leaving as each batch is ready = rows arriving in any order.

Where the picture breaks: a real cold store takes any dish. Polars 2.0 can spill only some steps today: sorting, many expressions and window functions, which compute a value per row from a group of rows. Joins and group-bys are not on the supported list yet, so a very large one can still run out of memory.

You have met these ideas in apps already. Here are two moments many people have lived.

• A food app's order history shows the same orders in a new order after you pull to refresh. Behind the screen, several servers answered at once, and nobody sorted the merged list.
• A gallery app stays smooth while you scroll through 20,000 photos on a cheap phone. Behind the screen, it keeps a few photos in memory and reads the rest from storage when needed.

The streaming engine works through your data in batches on every CPU core, instead of loading each step's full input into memory first. That uses far less memory, and on most queries it is faster too.

Start with the naive version. An early version of the orders job called read_parquet() to load all 18 months of orders, then filtered for the last 90 days. On the 32 GB server it crashed, because the full table did not fit.

The fix was to describe the work first and run it once. With scan_parquet(), Polars builds a lazy plan, sees the 90-day filter, and skips the parts of the files it can rule out.

In Polars 1.x, that lazy plan still ran on the in-memory engine unless you asked for streaming. In Polars 2.0, collect() chooses the streaming engine for you.

Three details matter. Eager DataFrame code is unchanged and still runs in memory. Writing results with sink_parquet() already used streaming. And any step the streaming engine cannot run yet quietly falls back to the in-memory engine. That step must hold its whole input in memory. Run with POLARS_VERBOSE=1 and look for in-memory-map in the log to spot one.

Row order changes because Polars 2.0 no longer promises to keep input order after joins, group-bys and unpivots. Keeping order costs time and memory when many batches run at once, so Polars now does it only when you ask.

There is no fixed answer, and that is the bug. The sort happened before the join, and the join no longer promises to keep it. The panel showed 20 rows, just not the newest 20.

Here is a tiny version. Three orders arrive sorted newest first: O-103 at 9:40, O-102 at 9:15 and O-101 at 8:50. After the join, Polars is free to return them as O-101, O-103, O-102. Then head(1) gives O-101, the oldest order.

Order can also survive by luck. Eager DataFrames and engine="in-memory" still keep input order, and so did Polars 1.x. So tests that ran on 1.x or on eager code prove nothing about the streaming engine. Even a three-row join can come back in a different order between runs.

When a Polars 2.0 query reaches about 80% of the machine's memory, steps that support it move some data to disk. They read it back when needed, so the query finishes more slowly instead of crashing.

Polars calls this out-of-core processing: working on more data than fits in memory, by keeping part of it on disk. The Polars team says the 80% starting point may need tuning.

On the orders job's 32 GB server, spilling starts at about 25.6 GB. That figure counts only memory Polars itself has allocated, not other programs running on the same server. By default, Polars writes up to 64 GB of spilled data. Past that, the query stops with an error instead of filling the disk.

The release notes promise spilling for sorting, window functions and many expressions. Joins and group-bys are not on that list yet, so a very large one can still run out of memory.

The silent ones are row order after joins and group-bys, empty lists in explode(), some number types and a few output column names. Your code runs, and the numbers are different.

The same job's coupon report had a second surprise. Each order has a list of coupon codes, and many orders have an empty list. The job explodes the list into one row per code, then counts rows with no code as "no coupon".

Take three orders: [FRESH10, VEG5], [] and [FRESH10]. Polars 1.x turned the empty list into one null row, so the report counted one order without a coupon. Polars 2.0 turns it into no rows at all, so that count drops to zero.

Removed methods and lossy casts now fail straight away, and the error message names the replacement. Polars added two error types for this: AttributeRemovedError and ArgumentRemovedError.

The date rule is deliberate. str.to_date() accepts a format, so with format="%d/%m/%Y" the text 06/10/2026 is read as 6 October, exactly as you say, never as 10 June by guesswork.

Most removed features had warned for several releases first. The upgrade guide lists the version that deprecated each one, some as far back as 0.19.

Pin the old version, save today's outputs and upgrade in a branch. Then compare the new outputs with the saved ones, first ignoring row order and then with it. This part is for engineers who own pipelines.

1. 1

   Pin before anything else

   Write polars<2 in your requirements file, so a rebuilt server image cannot upgrade you by surprise.

2. 2

   Clear warnings on the last 1.x release

   Run your tests with deprecation warnings visible. Most features removed in 2.0 warned there first.

3. 3

   Save golden outputs

   Write the result of every important query on 1.x to Parquet. These files are your answer key.

4. 4

   Upgrade in a branch and fix loud errors

   Install Polars 2.0 and fix each typed error; its message names the replacement.

5. 5

   Compare outputs twice

   First ignore row order, to catch changed values. Then compare with order wherever order carries meaning, like rankings and invoice numbers.

6. 6

   Keep an escape hatch

   If you must ship before every fix lands, pl.Config.set_engine_affinity("in-memory") or POLARS_ENGINE_AFFINITY=in-memory restores the old engine. Treat it as temporary.

Here is the comparison step on a tiny example, together with the fix for the latest orders bug.

import polars as pl
from polars.testing import assert_frame_equal

orders = pl.LazyFrame({
    "order_id": ["O-103", "O-102", "O-101"],  # newest first
    "customer_id": [7, 3, 7],
})
customers = pl.LazyFrame({"customer_id": [3, 7], "city": ["Pune", "Nashik"]})

kept = orders.join(customers, on="customer_id", how="left", maintain_order="left")
print(kept.head(1).collect())  # always O-103, the newest order

free = orders.join(customers, on="customer_id", how="left").collect()
assert_frame_equal(free, kept.collect(), check_row_order=False)  # same rows
print("Same rows; only the order may differ.")

Check three things: the memory limit Polars believes it has, the disk space for spilled data, and which steps of your query can spill.

Polars sets its spill threshold at 80% of the memory it sees, and it reads container limits when it works this out. Three settings in 2.0.0 change this: POLARS_OOC_MEMORY_BUDGET_FRACTION sets the fraction, POLARS_OOC_MEMORY_BUDGET_MB sets a hard cap, and POLARS_OOC_SPILL_DIR moves spilled files.

Polars uses the lower of the fraction and the cap, so the cap suits a shared server where other programs need memory too. On Linux, spilled files go to /var/tmp/polars-USERNAME/spill by default, where USERNAME is your login name. The user guide does not document these settings as of October 2026, so pin your Polars version if you rely on them.

The disk budget has its own setting. When spilled data passes the budget, Polars stops the query with an error that names POLARS_OOC_DISK_BUDGET_MB, the variable to raise. Clearer wording for that error was merged the day after 2.0.0 shipped, so the exact message depends on your version.

Now the trap. As of October 2026, an open issue reports that Polars reads the memory limit only at the top level of Linux control groups. These cgroups are the kernel's way of capping how much memory a process may use.

Inside a SLURM job or a systemd service, the real limit sits one level lower. So Polars plans with the whole machine's RAM, and the kernel can kill the job before spilling ever starts. Ordinary containers are fine, because there the top-level limit is the container's own.

No maintainer had replied to the issue as of 19 October 2026, but it suggests a workaround: set POLARS_OVERRIDE_TOTAL_MEMORY_MB to the job's real limit before importing Polars. This job script does that, and also lowers the disk budget for a small disk.

import os

# Inside a SLURM job capped at 8 GB, tell Polars the real limit.
# Set both before importing polars, to be safe.
os.environ["POLARS_OVERRIDE_TOTAL_MEMORY_MB"] = "8000"
os.environ["POLARS_OOC_DISK_BUDGET_MB"] = "20000"  # spill at most 20 GB

import polars as pl

prices = pl.LazyFrame({"item": ["onion", "tomato", "potato"], "rs_per_kg": [42, 30, 28]})
print(prices.sort("rs_per_kg").collect())  # runs on the streaming engine by default

To check it worked, run the script with POLARS_VERBOSE=1. The first log line should read total memory: 7.451 GiB. That is 8000 decimal megabytes, so the number looks smaller but the setting worked. If it shows the node's full RAM, the setting came too late.

Joins and group-bys are not on the supported list yet, so shrink the data before it reaches them. Filter early, select only the columns you need, and aggregate before joining when you can.

Try it yourself: find the order-sensitive spots in one of your own pipelines. It takes about 20 minutes.

1. 1

   List the reshaping steps

   Write down every join, group_by and unpivot in one lazy pipeline.

2. 2

   Look downstream

   For each one, check whether a later step depends on position: head, tail, first, last, with_row_index or a horizontal concat.

3. 3

   Fix the real dependencies

   Add maintain_order or a sort only where order carries meaning, such as rankings or invoice numbers.

4. 4

   Run it on both engines

   Collect once with the default engine and once with engine="in-memory", then compare with assert_frame_equal(..., check_row_order=False).

5. 5

   Check your result

   You got it right if both runs have the same rows, and every output whose order matters is identical on both engines.

Explain it as one default change, one trade-off and one safety net, then show how you would upgrade. Walk through these five points, one line each.

1. 1

   Name the default

   collect() on a LazyFrame now runs the streaming engine: batches across all cores, far less memory.

2. 2

   Name the trade-off

   Row order after joins, group-bys and unpivots is no longer promised; sort or pass maintain_order where order matters.

3. 3

   Explain the safety net

   At about 80% of RAM, sorts, window functions and many expressions spill to disk, up to a 64 GB budget by default.

4. 4

   Split loud from silent

   Removed APIs raise typed errors that name the fix; silent changes need output comparisons.

5. 5

   Describe the rollout

   Pin, save golden outputs, upgrade in a branch, compare with and without order, and keep in-memory affinity as a temporary fallback.