parquetDatastore

Create a datastore for one or more Parquet files.

📝Syntax

pds = parquetDatastore(location)
pds = parquetDatastore(location, Name, Value)

📥Input Arguments

Parameter	Description
location	a filename, folder name, wildcard pattern, string array, or cell array of character vectors.
Name, Value	optional arguments specified as name-value pairs.

📤Output Arguments

Parameter	Description
pds	a nelson.io.datastore.ParquetDatastore object.

📄Description

pds = parquetDatastore(location) creates a datastore that reads local Parquet files from the specified location.

When location is a folder, files ending with .parquet in that folder are selected. Wildcard patterns can be used to select multiple files.

Supported name-value pairs are ReadSize, SelectedVariableNames, OutputType, RowTimes, RowFilter, and VariableNamingRule.

The datastore reads one file at a time. Use hasdata, read, readall, preview, and reset to navigate the data.

💡Examples

folder = tempdir();
file1 = [folder, 'doc_parquet_ds_1.parquet'];
file2 = [folder, 'doc_parquet_ds_2.parquet'];
parquetwrite(file1, table([1; 2], [10; 20], 'VariableNames', {'Id', 'Value'}));
parquetwrite(file2, table([3; 4], [30; 40], 'VariableNames', {'Id', 'Value'}));
pds = parquetDatastore([folder, 'doc_parquet_ds_*.parquet']);
pds.VariableNames
T = readall(pds)

folder = tempdir();
file1 = [folder, 'doc_parquet_ds_filter_1.parquet'];
file2 = [folder, 'doc_parquet_ds_filter_2.parquet'];
parquetwrite(file1, table([1; 2], [10; 20], 'VariableNames', {'Id', 'Value'}));
parquetwrite(file2, table([3; 4], [30; 40], 'VariableNames', {'Id', 'Value'}));
rf = rowfilter({'Id', 'Value'});
pds = parquetDatastore([folder, 'doc_parquet_ds_filter_*.parquet'], ...
  'SelectedVariableNames', {'Id', 'Value'}, 'RowFilter', rf.Value > 15);
T = readall(pds)

🕔Version History

Version	Description
2.0.0	initial version

Edit this page on GitHub