Edit

#Project Documentation

The project includes several levels of documentation, including:

• Markdown files in the documentation directory
• API documentation generated by PHPDoc, built in the /web/landing/build/documentation/ directory
• Usage examples in the examples directory, divided into categories and topics

#Project structure

This project is developed as a monorepo, which means that it contains multiple packages in a single repository. The project is structured as follows:

├── bin
├── build
├── documentation
├── examples
├── phpdoc
├── src
│   ├── adapter
│   ├── bridge
│   ├── cli
│   ├── core
│   ├── functions.php
│   ├── lib
│   └── tools
├── tools
├── var
├── vendor
└── web
    └── landing

• bin contains the executable scripts used during development
• build contains the build artifact, currently only the PHAR file. To build the PHAR file, you can run composer build:phar.
• documentation contains the documentation files of the entire project but also of each package separately. This folder is used to generate the documentation website by converting markdown into html.
• examples contains example code that demonstrates how to use the project. Those examples are also displayed on the documentation website.
• phpdoc contains the API documentation generated by PHPDoc. It can be built by running composer build:docs:api.
• src contains the source code of the project, which is divided into several packages:
  • adapter adapters are packages that provides implementations for different building blocks of the DataFrame like Extractor, Loader, Transformer
  • bridge contains bridges to connect flow libs with other libraries and frameworks.
  • cli contains the command line interface application.
  • core contains the core functionality of the project, it holds the entire DataFrame.
  • lib contains standalone libraries that can be used independently of the project, like doctrine-dbal-bulk and parquet.
  • tools contains tools used during development.
• tools contains tools used during development, like the phpstan, phpunit and others, to not pollute project autoloader and to keep tools outside of project dependencies.
• var contains temporary files, like cache and logs.
• vendor contains the dependencies of the project, managed by Composer.
• web
  • landing contains the landing page of the project, which is a symfony application that is automatically dumped to static HTML files and served by GitHub Pages. It has it's own composer.json that defines the dependencies for the landing page and commands.

#Monorepo packages

Each package is structured in a very similar way, with the following structure:

src/core
└── etl
    ├── CONTRIBUTING.md
    ├── LICENSE
    ├── README.md
    ├── composer.json
    ├── src
    │   └── Flow
    └── tests
        └── Flow

Tests are usually divided into:

• Unit - src/core/etl/tests/Flow/ETL/Tests/Integration
• Integraiton - src/core/etl/tests/Flow/ETL/Tests/Unit

In some cases on that level, there might be some test helpers, like:

• Doubles - mocks / stubs / fakses used across tests
• Benchark - performance tests written using phpbench library

#Packages Dependencies

The rule is that we should keep as few dependencies as possible. But when we are adding dependency, it should be as wide as possible, so we don't block other projects by our constraints.

Packages from this monorepo can depend on each other, but ther are strict rules about that:

• lib - libraries can depend only on other lib packages, never on anything else.
• adapter - adapters can depend on lib / bridge and they always depend on core. Adpaters should not depend on cli
• bridge - bridges can depend only on lib
• cli - CLI can depend on core, lib and adapter and bridge, cli always depends on core
• core - core can depend on lib or bridge, but should should never depend on adapter, or cli

The above rules apply also on namespaces. So for example Adapter for CSV can't depend on Parquet adapter.

#PHP Versions

This project supports only the latest three PHP versions, for example: 8.2, 8.3, and 8.4. (assuming that 8.4 is the latest version). Development is done using the lowest supported version, which is currently PHP 8.2. The project is tested against all three versions, so you can use any of them for development.

#Tools

There are several tools used in this project to help with development, testing, and building the project. Most of them are available as Composer scripts, so you can run them using composer <script-name>.

• composer static:analyze runs static analysis tools like PHPStan to check the code for errors and potential issues.
• composer cs:php:fix runs the PHP CS Fixer and Rector to automatically fix coding standards issues in the code.
  • composer test runs all tests in the project, including unit tests, functional tests, and integration tests. It's a combination of all other test commands:
  • composer test:core
    • composer test:cli
    • composer test:lib:array-dot
    • ...
    • composer test:bridge:filesystem-azure
    • ...
    • composer test:adapter:xml
    • ...
    • composer test:benchmark runs the benchmark tests to measure the performance of the certain parts of the project.
    • composer test:website runs the tests for the website
    • composer test:examples runs all examples
    • composer test:mutation runs the mutation tests to check the quality of the tests.

#Coding Standards

The whole project is developed as an object oriented code, with a focus on clean code principles. The coding standards are defined and automatically enforced by the PHP CS Fixer and Rector.

Each package comes with a DSL (Domain Specific Language) that provides an easy, functional API. All functions defined in DSL (usually in functions.php files) are following the same codding standards as PHP code. snake_case is used for function names and arguments in DSL. The only other place where snake_case is used is in the tests, where it is used for test method names.

To make sure, that the whole project is aligned with the codding standards, run following commands:

composer cs:php:fix
composer static:analyze

Only when both commands pass, you should commit your changes.

#Testing

• Tests in the project are divided into
  • Unit - tests a single behavior in isolation, without any dependencies.
  • Integration - tests a single behavior with dependencies, like database or external services.
• Test cases of all packages should extends \Flow\ETL\Tests\FlowTestCase class, the only exceptions are lib and bridge packages, which can use their own test cases.
• Each test method should test only one scenario, when one behavior needs to be tested against multiple input data, use PHPUnit\Framework\Attributes\TestWith or PHPUnit\Framework\Attributes\DataProvider attributes.