Exporters description

What are exporters?

Exporters aims to provide a flexible and easy to extend infrastructure for exporting data to and from multiple sources, with support for filtering and transformation.

Architecture

Exporters architecture

Config file

You can define Exporters behaviour with a configuration object. This object has the following attributes:

  • reader (mandatory): defines what reader module the export should use and its options.
  • writer (mandatory): defines what writer module the export should use and its options.
  • exporter_options: it contains general export options, including output format.
  • filter_before: defines what filter module should be used before transforming and its options.
  • filter_after: defines what filter module should be used after transforming and its options.
  • transform: defines what transform module the export should use and its options.
  • persistence: defines what persistence module the export should use and its options.
  • stats_manager: defines what stats_manager module the export should use and its options.
  • grouper: defines what grouper module the export should use and its options.

This is an example of the simplest config file that can be used.

 {

     "reader": {
         "name": "exporters.readers.random_reader.RandomReader",
         "options": {
             "number_of_items": 1000,
             "batch_size": 10
         }
     },
     "writer":{
         "name": "exporters.writers.console_writer.ConsoleWriter",
         "options": {
         }
     }
}

Features

  • Multiple readers and writers.
  • Resume support using different technologies as a backend (MySQL, Postgres, Pickle...)
  • Support for bypass (direct copying) export pipeline for certain configurations to improve speed exports.
  • Easy to extend and flexible architecture.
  • Different output formats.
  • Notifications and webhooks support.
  • Export stats gathering.
  • Grouping support.
  • Filter and transform exported items.
  • Loading options from env.

Going further

Exporters functionality can be exteded by plugins. Please check https://github.com/exporters-plugins to find out about them.