Sponsor
DSL References
DSL stands for Domain Specific Language. In Flow, the DSL is a set of small functions that wrap object construction so pipelines read top-to-bottom. See the examples for usage in context.
HELPER
filesystem_telemetry_config
/**
* Create a telemetry configuration for the filesystem.
*/
filesystem_telemetry_config(Telemetry $telemetry, ClockInterface $clock, ?FilesystemTelemetryOptions $options) : FilesystemTelemetryConfig/**
* Create options for filesystem telemetry.
*
* @param bool $traceStreams Create a single span per stream lifecycle (default: ON)
* @param bool $collectMetrics Collect metrics for bytes/operation counts (default: ON)
*/
filesystem_telemetry_options(bool $traceStreams, bool $collectMetrics) : FilesystemTelemetryOptions/**
* Copy a file from one path to another, across any filesystems mounted in the table.
* Always streams bytes; same-filesystem copies do not use server-side optimizations
* because `Filesystem::mv` is a move, not a copy.
*/
file_copy(FilesystemTable $table, ?OperationOptions $options) : Copy/**
* Move a file from one path to another, across any filesystems mounted in the table.
* Intra-filesystem moves delegate to `Filesystem::mv` for server-side optimizations;
* cross-filesystem moves stream-copy then remove the source (non-atomic).
*/
file_move(FilesystemTable $table, ?OperationOptions $options) : Move/**
* Create a new filesystem table with given filesystems.
* Filesystems can be also mounted later.
* If no filesystems are provided, local filesystem is mounted.
*/
fstab(Filesystem $filesystems) : FilesystemTable/**
* Create a new memory filesystem and writes data to it in memory.
*/
memory_filesystem(string $protocol) : MemoryFilesystemmount(string $protocol) : Mountnative_local_filesystem(string $protocol) : NativeLocalFilesystem/**
* Options shared by filesystem operations.
*
* @param int $chunkSize Number of bytes read/written per iteration when streaming across filesystems (default: 8192)
*/
operation_options(int $chunkSize) : OperationOptionspartition(string $name, string $value) : Partitionpartitions(Partition $partition) : Partitions/**
* Path supports glob patterns.
* Examples:
* - path('*.csv') - any csv file in current directory
* - path('/** / *.csv') - any csv file in any subdirectory (remove empty spaces)
* - path('/dir/partition=* /*.parquet') - any parquet file in given partition directory.
*
* Glob pattern is also supported by remote filesystems like Azure
*
* - path('azure-blob://directory/*.csv') - any csv file in given directory
*
* @param array<string, null|bool|float|int|string|\UnitEnum>|Path\Options $options
*/
path(string $path, Options|array $options) : Path/**
* Resolve real path from given path.
*
* @param array<string, null|bool|float|int|string|\UnitEnum> $options
*/
path_real(string $path, array $options) : Path/**
* Write-only filesystem useful when we just want to write the output to stdout.
* The main use case is for streaming datasets over http.
*/
stdout_filesystem(string $protocol) : StdOutFilesystem/**
* Wrap a filesystem with telemetry tracing support.
* All filesystem and stream operations will be traced according to the configuration.
*/
traceable_filesystem(Filesystem $filesystem, FilesystemTelemetryConfig $telemetryConfig) : TraceableFilesystem
