FastBCP 1.3: Netezza over ADBC and DBPartition exports

FastBCP 1.3 adds two things: Netezza / IBM NPS export without a vendor .NET provider, and a DBPartition parallel method that exports a partitioned table along its native partitions.

It also changes how the adbc_* connection types behave. If you run those in production, read the upgrade checks at the end of the article before deploying 1.3.

Netezza through ADBC: -C adbc_netezza

The new adbc_netezza connection type uses ArpeNetezza, an ADBC driver embedded in FastBCP, for both the metadata step and the data step. NZdotNET is no longer needed.

• Works with IBM Cloud NPS. NPS 7.x needs tls_min_version=1.0 for TLS 1.0/1.1.
• Output is Parquet only, as for the other adbc_* types.
• Parallel methods: None, NZDataSlice, Ntile, Random, RangeId, DataDriven.
• --sourceconnectstring accepts ADO.NET-style keys (Server, Port, User ID…) without the adbc.arpenz. prefix. Unsupported keys are ignored with one warning.

NZDataSlice needs no distribution key: FastBCP splits the query along the table's Netezza data slices. In 1.3, slices go through a shared work queue, so an idle worker picks up the next slice.

Windows

.\FastBCP.exe `
  --connectiontype adbc_netezza `
  --sourceconnectstring "Server=nps-host;Port=5480;Database=SALES;User ID=admin;Password=FastPassword" `
  --sourceschema "ADMIN" `
  --sourcetable "ORDERS" `
  --directory "D:\export\orders" `
  --fileoutput "orders.parquet" `
  --parallelmethod NZDataSlice `
  --paralleldegree 8

Linux

./FastBCP \
  --connectiontype adbc_netezza \
  --sourceconnectstring "Server=nps-host;Port=5480;Database=SALES;User ID=admin;Password=FastPassword" \
  --sourceschema "ADMIN" \
  --sourcetable "ORDERS" \
  --directory "/data/export/orders" \
  --fileoutput "orders.parquet" \
  --parallelmethod NZDataSlice \
  --paralleldegree 8

To adapt this command to your environment, use the Netezza to local Parquet pipeline builder.

Throughput. We exported a wide Netezza table with this setup (adbc_netezza, NZDataSlice, automatic partitioning):

Rows	Columns	Elapsed time	Throughput	FastBCP memory
423 million	160	636 s	106 MCells/sec (about 665,000 rows/sec)	under 3 GB

A cell is one column value in one row, so MCells/sec accounts for table width, which rows/sec alone hides. The export ran with --paralleldegree 30. Results in other environments will differ.

The source table occupies 200 GB in Netezza. The export wrote 40 GB of Parquet compressed with zstd, one fifth of that size:

Data size, 423 million rows × 160 columns

Netezza table

200 GB

Parquet (zstd)

40 GB

NPS 7 pre-releases

Files exported from NPS 7 with v1.3.0-netezza.1 or v1.3.0-netezza.2 may contain wrong NUMERIC values. Re-export them with FastBCP 1.3.

Native partitions: -m DBPartition

DBPartition runs one extraction per native partition or subpartition, on SQL Server, PostgreSQL, Oracle, MySQL and Teradata, and their adbc_* counterparts. It requires --sourceschema and --sourcetable.

CHUNK is the default. HIVE writes Hive-style directories that Spark, DuckDB, Trino or Athena can read as a partitioned dataset. SQL Server uses bounds=<lower>_<upper>/ and Teradata uses L1=<n>/. If the table is not partitioned, has a single partition, or the source is not supported, FastBCP falls back to None and logs a warning: check the log.

Windows

$env:FASTBCP_DBPARTITION_TARGET_MODE = "HIVE"

.\FastBCP.exe `
  --connectiontype oraodp `
  --server "dbserver:1521/ORCLPDB" `
  --user "FastUser" `
  --password "FastPassword" `
  --sourceschema "SALES" `
  --sourcetable "ORDERS" `
  --directory "D:\export\orders" `
  --fileoutput "orders.parquet" `
  --parallelmethod DBPartition `
  --paralleldegree 8 `
  --merge false

Linux

export FASTBCP_DBPARTITION_TARGET_MODE=HIVE

./FastBCP \
  --connectiontype oraodp \
  --server "dbserver:1521/ORCLPDB" \
  --user "FastUser" \
  --password "FastPassword" \
  --sourceschema "SALES" \
  --sourcetable "ORDERS" \
  --directory "/data/export/orders" \
  --fileoutput "orders.parquet" \
  --parallelmethod DBPartition \
  --paralleldegree 8 \
  --merge false

If one partition holds most of the rows, a key-based method such as RangeId or Ntile may balance the work better.

Azure Blob Storage and ADBC connectors

Azure Blob Storage. FastBCP now uses AZURE_STORAGE_CONNECTION_STRING or AZURE_STORAGE_SAS_URL when one of them is set. Otherwise it falls back to DefaultAzureCredential. The log shows which method was used. A missing create permission is tolerated when the container already exists.

ADBC connectors:

• adbc_mssql honours --applicationintent (default ReadOnly), so read-only routing on an Availability Group listener sends the export to a readable secondary.
• adbc_pgsql reads the libpq environment: PGUSER, PGPASSWORD, ~/.pgpass, PGSSLROOTCERT.
• adbc_oracle resolves TNS aliases through tnsnames.ora.

Before you upgrade

These changes can alter the result of an existing job:

1. adbc_mssql / adbc_pgsql / adbc_oracle run fully on ADBC. The metadata and planning step no longer uses SqlClient, Npgsql or ODP.NET, and one connection string serves both steps. Native keys with no driver equivalent (e.g. Pooling=false) are dropped with a warning. Run each job once and read the warnings.
2. adbc_pgsql: NaN in a NUMERIC column now fails the read instead of becoming NULL. To keep the old behaviour, add adbc.arpepgsql.numeric_special=null to -g.
3. adbc_oracle sends encryption=accepted and data_integrity=accepted, so the server decides. Override with FASTBCP_ADBC_ENCRYPTION / FASTBCP_ADBC_DATA_INTEGRITY if needed.
4. adbc_oracle + DataDriven: a scale-0 NUMBER(<=18) key is now an integer, as with oraodp. Check Hive paths if a downstream job depends on them.

Notable fixes

• Ntile could lose rows when the last bucket held a single key value. A midnight lower bound could also truncate the upper bound's time. If you used Ntile before 1.3, compare source and target row counts.
• Quoted passwords containing ; are now fully masked in logs and telemetry.
• adbc_mssql resolves named instances (host\INSTANCE) through the SQL Server Browser.
• oraodp / adbc_oracle with a TNS alias plus -U / -X no longer fail with ORA-50008.
• Netezza: fixes for nzoledb, nzcopy -m Timepartition, and NZDataSlice with --timestamped.

The full list is in the release notes.

Resources

• FastBCP 1.3 release notes
• Parallel parameters and FASTBCP_DBPARTITION_TARGET_MODE
• Azure authentication methods
• Netezza to local Parquet pipeline builder
• Request a FastBCP trial