From SQL Server to Polars 2.0 with ArpeMSSQL

Update Oct 9, 2026 - Added the arrow-odbc timing.

We recently released the Arpeio ADBC drivers, a set of database drivers that exchange data in a columnar format. This post shows an example of usage with SQL Server, in Python: we read a table into a Polars DataFrame with the ArpeMSSQL driver. We use Polars 2.0, released on October 6, 2026.

ODBC and JDBC are row-oriented APIs, while dataframe libraries store data column by column, and the conversion can take longer than the query itself. An ADBC driver builds Apache Arrow columns in native code, and Polars reads Arrow.

Data

We use the US Reporting Carrier On-Time Performance data from the Bureau of Transportation Statistics. The 2024 data has 7,079,061 domestic flights. Nineteen columns were loaded into a SQL Server 2025 table, airline.dbo.flights (with ArpeMSSQL's bulk ingest):

CREATE TABLE dbo.flights (
    FlightDate        DATE        NOT NULL,
    Reporting_Airline VARCHAR(2)  NOT NULL,
    Tail_Number       VARCHAR(10) NULL,
    Flight_Number     SMALLINT    NULL,
    Origin            VARCHAR(3)  NOT NULL,
    OriginCityName    VARCHAR(64) NOT NULL,
    Dest              VARCHAR(3)  NOT NULL,
    DestCityName      VARCHAR(64) NOT NULL,
    CRSDepTime        SMALLINT    NOT NULL,  -- scheduled departure, local hhmm
    DepTime           SMALLINT    NULL,
    DepDelay          SMALLINT    NULL,      -- minutes
    CRSArrTime        SMALLINT    NOT NULL,
    ArrTime           SMALLINT    NULL,
    ArrDelay          SMALLINT    NULL,
    ArrDel15          BIT         NULL,
    Cancelled         BIT         NOT NULL,
    Diverted          BIT         NOT NULL,
    CRSElapsedTime    SMALLINT    NULL,
    Distance          SMALLINT    NOT NULL
)

Setup

The driver is installed and registered with the ADBC driver manager by a one-line script, as described in the documentation. A trial licence, along with this install command, can be requested on the trial page:

curl -fsSL https://raw.githubusercontent.com/arpe-io/adbc-drivers/main/install.sh \
  | sh -s -- arpemssql --license /path/to/your.lic

On the Python side, we only need Polars and the ADBC driver manager:

pip install "polars>=2.0" adbc-driver-manager

Imports

import adbc_driver_manager.dbapi as dbapi
import polars as pl

Package versions

Used in this post:

Python               : 3.13.0
OS                   : Linux
adbc_driver_manager  : 1.12.0
polars               : 2.0.0
ArpeMSSQL            : 0.7.1
SQL Server           : 2025 (RTM-CU7) 17.0.4065

The following packages are not needed for the transfer itself, but they were used for results quoted below: pyodbc, arrow-odbc and the ODBC driver for the timing comparisons with ODBC, and PyArrow for the zero-copy check.

pyarrow              : 25.0.1
pyodbc               : 5.3.0
arrow-odbc           : 10.6.0
ODBC Driver 18       : 18.7

Loading the table

pl.read_database accepts an ADBC connection. The driver is loaded by name:

uri = (
    "sqlserver://sa:<password>@localhost:1435/"
    "?database=airline&encrypt=true&TrustServerCertificate=true"
)

with dbapi.connect(driver="arpemssql", db_kwargs={"uri": uri}) as conn:
    flights = pl.read_database("SELECT * FROM dbo.flights", conn)

7,079,061 rows x 19 columns in 3.64 s (1.94 M rows/s)

As a reference, the same pl.read_database call through pyodbc and the Microsoft ODBC Driver 18 takes 41.7 s, against 3.64 s with ArpeMSSQL, 11.4 times longer. Both use TLS (Transport Layer Security, the encryption of the connection between client and server), and both timings are the best of three runs. SQL Server runs in a Docker container on the same laptop, an Intel i9-12900H with 32 GB of RAM, so client and server share the CPU and the gap does not come from the network. With pyodbc, every value becomes a Python object before Polars rebuilds the columns. pyodbc is the most common way to reach SQL Server from Python, but not the fastest way to use ODBC. Polars can also read through arrow-odbc, which fetches columns in blocks and keeps the column types: the same read then takes 6.43 s, best of five runs, and ArpeMSSQL is still 1.8 times faster.

The column types survive the transfer:

flights.schema

Schema([('FlightDate', Date),
        ('Reporting_Airline', String),
        ('Tail_Number', String),
        ('Flight_Number', Int16),
        ('Origin', String),
        ...
        ('DepDelay', Int16),
        ...
        ('ArrDel15', Boolean),
        ('Cancelled', Boolean),
        ('Diverted', Boolean),
        ('CRSElapsedTime', Int16),
        ('Distance', Int16)])

SMALLINT stays a 16-bit integer, DATE stays a date and BIT becomes a boolean. Through pyodbc, every SMALLINT column arrives as Int64, because pyodbc describes it as a plain Python int, and takes four times the memory, unless the types are set by hand with schema_overrides.

If the result ever does not fit in memory, Polars can read it in batches: read_database(..., iter_batches=True) yields one DataFrame per batch. With an ADBC connection, these are the driver's batches, whose size is set with the option shown below, and this mode needs PyArrow.

Arrow in Polars

From the Polars documentation:

Polars can consume and produce Arrow data often with zero-copy operations. Note that Polars is not built on a Pyarrow/Arrow implementation. Instead, Polars has its own compute and buffer implementations.

Arrow is first of all a specification of how columns are laid out in memory. PyArrow is one implementation of that format, and Polars has its own, in Rust. When the driver hands its result over through the Arrow C stream interface, Polars can adopt the driver's buffers instead of copying them, as long as its internal type has the same layout.

The word "often" in the documentation excerpt above matters. For five columns, we fetched the result as a PyArrow table, converted it with pl.from_arrow(table, rechunk=False), and compared buffer addresses to see how many bytes of each Polars column still live in the driver's buffers:

Column	Arrow type (driver)	Polars type	Shared with the driver	Polars column
DepDelay	int16	Int16	15.0 MB	15.0 MB
FlightDate	date32	Date	28.3 MB	28.3 MB
Cancelled	bool	Boolean	0.9 MB	0.9 MB
Origin	string	String	0.0 MB	113.3 MB
OriginCityName	string	String	92.6 MB	205.9 MB

Numeric, date and boolean columns are fully zero-copy. Strings are different. The driver returns Arrow string (offsets and bytes), and Polars stores strings as string_view (German-style strings), with one 16-byte view per value. Strings of 12 bytes or less are stored inside the view, so the Origin column, with its 3-letter airport codes, is copied entirely. For such short strings, the views also take more memory than the driver's layout: 113.3 MB for Origin, against about 50 MB of offsets and bytes. Longer strings keep their bytes in the driver's buffer, and only the views are new: for OriginCityName (such as "Dallas/Fort Worth, TX"), 92.6 MB of the 205.9 MB are shared with the driver.

read_database keeps the driver's record batches as chunks. ArpeMSSQL streams results in batches of 32,000 rows by default, so the 7,079,061 rows arrive as 221 full batches plus one of 7,061 rows, hence 222 chunks. An explicit rechunk(), or an operation that needs one contiguous buffer, such as to_numpy(), copies the column into one buffer.

The batch size can be changed for the whole connection with the adbc.arpemssql.buffer_size option:

with dbapi.connect(
    driver="arpemssql",
    db_kwargs={"uri": uri, "adbc.arpemssql.buffer_size": "250000"},
) as conn:
    flights = pl.read_database("SELECT * FROM dbo.flights", conn)

flights.n_chunks()

29

The transfer time does not change measurably with the batch size.

Wrap-up

With ArpeMSSQL, the 7,079,061 rows go from SQL Server into Polars 2.0 in 3.6 s. Column types are kept, and the numeric, date and boolean columns are not copied between the driver and Polars. From there, flights is an ordinary Polars DataFrame, and you can do whatever you want with it.

Driver downloads are at github.com/arpe-io/adbc-drivers. The binaries download freely and run once a licence is present. A free trial licence for ArpeMSSQL can be requested here; for a full licence, contact us at sales@arpe.io or see arpe.io.

Appendix: reading the table through ODBC

The ODBC timings use the same pl.read_database call, through the Microsoft ODBC Driver 18 with TLS enabled. With pyodbc:

import pyodbc

odbc_conn_str = (
    "DRIVER={ODBC Driver 18 for SQL Server};SERVER=localhost,1435;DATABASE=airline;"
    "UID=sa;PWD=<password>;Encrypt=yes;TrustServerCertificate=yes"
)

with pyodbc.connect(odbc_conn_str) as conn:
    flights = pl.read_database("SELECT * FROM dbo.flights", conn)

With arrow-odbc (pip install arrow-odbc), there is no connection object: when read_database receives an ODBC connection string (one containing Driver={...}) instead of a connection, Polars reads through arrow-odbc:

flights = pl.read_database("SELECT * FROM dbo.flights", odbc_conn_str)