Announcing FastBCP 0.30

We're excited to announce FastBCP 0.30, a significant release that brings powerful new capabilities for working with temporal data, fine-tuning Parquet exports, and managing complex configurations more efficiently.

This release focuses on three major enhancements: time-based parallel partitioning, Parquet row group configuration, and YAML configuration file support.

What's New in 0.30

1. Time-Based Parallel Partitioning (Timepartition Method)

The headline feature of this release is the new Timepartition parallel method, designed specifically for time-series data and large temporal datasets.

Why Time-Based Partitioning?

When dealing with tables that contain date or datetime columns, traditional partitioning methods (like RangeId or Ntile) may not align with how your data is naturally organized. Time-based partitioning allows FastBCP to:

• Split work naturally along temporal boundaries (year, month, week, day)
• Leverage time-based indexes for optimal query performance
• Enable each parallel thread to export a specific time period independently

How It Works

The Timepartition method uses a special format for the --distributekeycolumn parameter that specifies both the date column and the desired time granularity:

Syntax:

(datecolumn, year, month, day)    # Partition by year, month, and day
(datecolumn, year, month)         # Partition by year and month
(datecolumn, year, week)          # Partition by year and week
(datecolumn, year)                # Partition by year only

Example: Exporting Orders by Month

Let's export an orders table partitioned by year and month:

.\FastBCP.exe `
  --connectiontype "mssql" `
  --server "localhost" `
  --database "tpch_copy" `
  --user "FastUser" `
  --password "FastPassword" `
  --sourceschema "tpch_10" `
  --sourcetable "orders_date_sorted" `
  --directory "D:\temp\TestPartition" `
  --fileoutput "orders.csv" `
  --method "Timepartition" `
  --distributekeycolumn "(o_orderdate,year,month)" `
  --paralleldegree 16 `
  --merge "False"

For complete documentation, see Parallel Parameters.

---

2. Parquet Row Group Size Configuration

Available since version 0.30.1

This feature is available starting from FastBCP version 0.30.1.

For users exporting to Parquet format, version 0.30 introduces control over row group size through the FASTBCP_RGSIZE environment variable.

Why Does Row Group Size Matter?

Row groups are the fundamental unit of parallelization and compression in Parquet files. The size you choose impacts:

• Query performance - Smaller row groups enable finer-grained filtering
• Compression ratio - Larger row groups improve compression efficiency
• Memory usage - Larger row groups require more memory during read/write
• Parallel processing - Row group boundaries define parallelization opportunities

Default and Configuration

• Default value: 1,000,000 rows per row group
• Configuration: Set via FASTBCP_RGSIZE environment variable before running FastBCP

Example Usage

Windows

# Set the row group size to 500,000 rows
$env:FASTBCP_RGSIZE = "500000"

# Run FastBCP
.\FastBCP.exe `
  --connectiontype mssql `
  ...

Linux

# Set the row group size to 500,000 rows
export FASTBCP_RGSIZE=500000

# Run FastBCP
./FastBCP \
  --connectiontype mssql \
  ...

For complete documentation, see Parquet Formatting.

---

3. YAML Configuration File Support

Complex FastBCP commands with many parameters can become difficult to read and maintain. Version 0.30 introduces the --config parameter for YAML configuration files.

Why YAML Configuration?

Benefits:

• Human-readable format with comment support
• Version control friendly - track configuration changes over time
• Structured sections - organized by concern (connection, source, output, performance)
• Reusable - save different configurations for different scenarios
• Simplified command line - reduce a 15-parameter command to just --config myconfig.yaml

Generate YAML from Command Line

You can use the FastBCP Configuration Wizard to convert your existing command-line parameters into a YAML configuration file. Simply input your parameters and download the generated YAML file.

YAML Structure

A FastBCP YAML configuration file has five main sections:

# FastBCP Configuration Example

connection:          # Database connection details
  type: mssql
  server: localhost
  database: mydb
  trusted: true

source:              # Source table information
  schema: dbo
  table: orders

output:              # Output file settings
  file: "orders_{startdate}.csv"
  directory: 'D:\exports\{database}\{schema}\{table}\'
  delimiter: "|"
  decimal_separator: "."
  date_format: "yyyy-MM-dd HH:mm:ss"
  encoding: UTF-8

performance:         # Parallel execution settings
  method: RangeId
  degree: -2
  distribute_key_column: o_orderkey
  merge: false

logging:             # Logging and run identification
  run_id: mssql_to_csv_parallel

Complete Example

Here's a production-ready configuration for exporting orders using the RangeId parallel method:

# FastBCP – MSSQL to CSV using RangeId parallel method
connection:
  type: mssql
  server: localhost
  database: tpch10_collation_bin2
  trusted: true

source:
  schema: dbo
  table: orders

output:
  file: "mssql_orders_{startdate}.csv"
  directory: 'D:\temp\{database}\{schema}\{table}\full\'
  delimiter: "|"
  decimal_separator: "."
  date_format: "yyyy-MM-dd HH:mm:ss"
  encoding: UTF-8

performance:
  method: RangeId
  degree: -2
  distribute_key_column: o_orderkey
  merge: false

logging:
  run_id: mssql_to_csv_parallel-2_rangeid

Usage:

.\FastBCP.exe --config samples\sample_mssql_to_csv.yaml

That's it! No need to remember all the parameter names or manage long PowerShell command lines.

For complete documentation, see Advanced Parameters.

---

Upgrade Path

FastBCP 0.30 is fully backward compatible with previous versions. Your existing command-line scripts and JSON settings files will continue to work unchanged.

Want to try it on your own data? Download FastBCP and get a free 30-day trial.

The ARPEIO Team