flow php

UNIFIED DATA PROCESSING FRAMEWORK

composer require flow-php/etl ^0.10.0

Changelog

elephant
extract

Extracts

Read from various data sources.

arrow
transform

Transforms

Shape and optimize for your needs.

arrow
load

Loads

Store and secure in one of many available data sinks.

DSL Functions


DSL stands for Domain Specific Language. In the case of Flow, the DSL is used to define simple functions that can be used to transform data. Most of those functions are initializing a new instance of a class under the hood since Flow is fully object-oriented. Please look at the examples below to get a better understanding of how to use the DSL functions.

EXTRACTOR


/**
 * Extractor will automatically try to iterate over whole index using one of the two iteration methods:.
 *
 * - from/size
 * - search_after
 *
 * Search after is selected when you provide define sort parameters in query, otherwise it will fallback to from/size.
 *
 * @param array{
 *  hosts?: array<string>,
 *  connectionParams?: array<mixed>,
 *  retries?: int,
 *  sniffOnStart?: boolean,
 *  sslCert?: array<string>,
 *  sslKey?: array<string>,
 *  sslVerification?: boolean|string,
 *  elasticMetaHeader?: boolean,
 *  includePortInHostHeader?: boolean
 * } $config
 * @param array<mixed> $parameters - https://www.elastic.co/guide/en/elasticsearch/reference/master/search-search.html
 * @param ?array<mixed> $pit_params - when used extractor will create point in time to stabilize search results. Point in time is automatically closed when last element is extracted. https://www.elastic.co/guide/en/elasticsearch/reference/master/point-in-time-api.html - @deprecated use withPointInTime method instead
 */
from_es(array $config, array $parameters, ?array $pit_params) : ElasticsearchExtractor

HELPER


/**
 * Transforms elasticsearch results into clear Flow Rows using ['hits']['hits'][x]['_source'].
 *
 * @return HitsIntoRowsTransformer
 */
es_hits_to_rows(DocumentDataSource $source) : HitsIntoRowsTransformer

LOADER


/**
 * https://www.elastic.co/guide/en/elasticsearch/reference/master/docs-bulk.html.
 *
 * In order to control the size of the single request, use DataFrame::chunkSize() method just before calling DataFrame::load().
 *
 * @param array{
 *  hosts?: array<string>,
 *  connectionParams?: array<mixed>,
 *  retries?: int,
 *  sniffOnStart?: boolean,
 *  sslCert?: array<string>,
 *  sslKey?: array<string>,
 *  sslVerification?: boolean|string,
 *  elasticMetaHeader?: boolean,
 *  includePortInHostHeader?: boolean
 * } $config
 * @param string $index
 * @param IdFactory $id_factory
 * @param array<mixed> $parameters - https://www.elastic.co/guide/en/elasticsearch/reference/master/docs-bulk.html - @deprecated use withParameters method instead
 */
to_es_bulk_index(array $config, string $index, IdFactory $id_factory, array $parameters) : ElasticsearchLoader
/**
 *  https://www.elastic.co/guide/en/elasticsearch/reference/master/docs-bulk.html.
 *
 * In order to control the size of the single request, use DataFrame::chunkSize() method just before calling DataFrame::load().
 *
 * @param array{
 *  hosts?: array<string>,
 *  connectionParams?: array<mixed>,
 *  retries?: int,
 *  sniffOnStart?: boolean,
 *  sslCert?: array<string>,
 *  sslKey?: array<string>,
 *  sslVerification?: boolean|string,
 *  elasticMetaHeader?: boolean,
 *  includePortInHostHeader?: boolean
 * } $config
 * @param string $index
 * @param IdFactory $id_factory
 * @param array<mixed> $parameters - https://www.elastic.co/guide/en/elasticsearch/reference/master/docs-bulk.html - @deprecated use withParameters method instead
 */
to_es_bulk_update(array $config, string $index, IdFactory $id_factory, array $parameters) : ElasticsearchLoader

Contributors

Join us on GitHub external resource
scroll back to top