Import Feeds

The "Import Feeds" module enables the use of import feeds in a user-friendly manner. In general, import feed is a data import template that depends on the user needs and entity data to be imported to your system.

With the help of the "Import Feeds" module, data import may be performed with the help of import feeds in two ways:

  • manually – via the preconfigured import templates, i.e. import feeds themselves;
  • automatically – via preconfigured import cron job.

Import feeds can be further configured and customized, as well as reused at different time intervals. Import feeds can also be used to automate the data import process via import cron jobs.

By default, import feed of the Simple type comes with the "Import Feeds" module installation. However, along with the installation of other modules, the list of import feed types may be extended, and these additional import feeds may be further customized according to your needs. At the same time, additional import feeds will operate using the logics of the "Import Feeds" module.

Installation

To install the "Import Feeds" module to your system, go to Administration > Module Manager, find this module in the "Store" list and click Install:

Import install

Select the desired version in the installation pop-up window that appears and click the Install button. The module background will turn green and it will be moved to the "Installed" section of the Module Manager. Click Run update to confirm its installation.

Please, note that running the system update will lead to the logout of all users.

To update/remove the "Import Feeds" module from the system, use the corresponding options from its single record actions menu in Administration > Module Manager.

Administrator Functions

The "Import Feeds" module significantly extends the functionality of the TreoPIM system, so further description of the module is given in the context of TreoPIM.

After the module installation, a new Import configuration group is added to the TreoPIM administration page. Also it is possible to add Import Cron Jobs, Import Feeds, and Import Results as separate navigation menu items on the Administration > User Interface page:

Import feeds adding

Access Rights

To enable import feed creation, editing, usage and deletion by other users, configure the corresponding access rights to the Import cron jobs, Import feeds, Import results entities for the desired user / team / portal user role on the Administration > Roles > 'Role name' page:

Import role cfg

Please, note that enabling at least Import feeds reading right for the user is a required minimum for him to be able to run import feeds.

User Functions

After the "Import Feeds" module is installed and configured by the administrator, user can work with import feeds in accordance with his role rights that are predefined by the administrator.

Import Feed Creation

To create a new import feed, click Import Feeds in the navigation menu and then click the Create Import Feed button.

If there is no Import Feeds option in the navigation menu, please, contact your administrator.

The common creation pop-up will appear:

Import feed creation

Here fill in the required fields and select the import feed type from the corresponding drop-down list. Currently the only import feed type supported is simple – designed for importing data into any entity existing in your system with the ability to configure the list of fields to be imported, their order and names.

Сlick the Save button to complete the operation; the new record will be added to the import feeds list. You can configure it right away on the detail view page that opens or return to it later.

Import Feed Configuration

To configure the import feed, click the desired record in the import feeds list; the following detail view page will open:

Import feed cfg

In-line editing is supported here. So prior to making any changes, you have to click the pencil icon located on the right of each editable field.

For details on in-line editing and other features of the TreoPIM system, refer to the Entity Records section of the Views and Panels article of our user guide.

The following settings are available on the OVERVIEW panel:

  • Active – select this checkbox to activate the import feed. If the import feed is not activated, the importing feature will be disabled for it.
  • Name – modify the import feed name, if needed.
  • Description – enter the description of the import feed usage, as a reminder for the future or as a hint for other users of the given import feed. This field is not required.
  • Action – define the action to be performed in the system during the data import:
    • Create Only – new data records will be created;
    • Update Only – data in the already existing records will be updated;
    • Create & Update – new data records will be created and the existing records will be updated.
  • Type – the import feed type defined on its creation only; it cannot be modified.
  • Limit – maximum number of records to be imported per import job via one import feed. Depending on the limit value entered, the import template will be split into separate parts accordingly, but data import will be performed in accordance with one import feed configuration. The default records limit is 1000.

By default, the import feed is not active and the Create Only action is assigned. Please, note that the choice of the action influences the content of the SIMPLE TYPE SETTINGS panel, described below.

File Properties

The import file parameters are configured on the FILE PROPERTIES panel:

Import feed cfg file

  • Example file (CSV) – load the template in the CSV format that will be used as an example file for data import. You can use either a full or shortened import file, or file with column headers as an example file. All the data in the example file should be UTF-8 encoded.
  • Header row – activate the checkbox to include the column names in the import file or leave it deselected to skip column names from importing.
  • Field delimiter – select the preferred field delimiter to be used in the import file: ,, ;,\t, |.
  • Decimal mark – select the preferred decimal mark to be used in the import file: . or ,.
  • Text qualifier – select the preferred separator of the values within a cell: single or double quotes.

Simple Type Settings

To enable editing of the parameters on the SIMPLE TYPE SETTINGS panel, click the Edit button on the detail view page of the current import feed and configure the following settings:

Simple type settings

  • Entity – from the drop-down list of entities available in the system select the desired entity type, for which this import feed will be used.
  • Field value delimiter – enter the preferred separator of the values within a field. The default symbol is ;.
  • ID – select the name of the data field that will be used as the identifier for update in the given import feed. The ID parameter is displayed here when either the Update only or Update & Create action is defined on the OVERVIEW panel. Moreover, it is a required field for the Update only action.

Please, note that the defined Field delimiter and Field value delimiter symbols must be different.

Configurator

The configuration of the entity fields is performed on the CONFIGURATOR panel on either the detail or edit view page of the import feed. By default, the required fields of the entity type defined on the SIMPLE TYPE SETTINGS panel are displayed there, and depending on this choice, the CONFIGURATOR panel contains different fields. For products, this panel looks as follows:

Configurator panel

To modify the entity field displayed on the CONFIGURATOR panel, use the Edit option from the single record actions menu and make the desired changes in the editing pop-up that appears for the given entity field:

Field editing

Please, note that import by ID and code is a recommended setting for the entity field to be used in the import feed.

Adding Entity Fields

It is possible to add more entity fields for import. For this select the Add entity field option from the adding drop-down menu; the following creation pop-up will appear:

Entity field adding

Here select the field from the drop-down list of all fields available in the system for the given entity, define its default value and/or select its file column. Also define whether data import should be made by ID, name or code (for those fields, where this option is available).

Please, note that either the File column or Default value field must be filled in. Otherwise, the current entity field cannot be created:

Field creation error

For the Product categories field there is also the ability to choose its scope level:

Product category scope

If the Channel scope level is defined, also select the required channel to be used for product categories in the corresponding field.

Please, note that entity fields can be added to the import feed record only once, except for the Product categories field, which may be added as many times as needed, but with different scope levels and different channels:

Product categories

Once the entity field is added to the configurator, it is added to the ID drop-down list on the SIMPLE TYPE SETTINGS panel.

Adding Product Attributes

The "Import Feeds" module also allows you to import product attribute values. They can be added to the import feed on the CONFIGURATOR panel via the Add product attribute option from the adding drop-down menu:

Product attribute adding

Please, note that this feature is available only when the TreoPIM module is installed together with the "Import Feeds" module.

In the creation pop-up that appears, select the attribute from the list of the existing attributes, define its default value and/or select its file column. Also define the attribute scope level – Global or Channel.

Make sure that the File column or Default value field is filled in; otherwise, the given product attribute will not be created.

Please, note that the same product attribute can be added to the import feed record more than once, but with different scope levels (Global or Channel) and different channels.

Adding Product Images

Product images can also be added for import via the "Import Feeds" module. For this select the Add product image option from the adding drop-down menu on the CONFIGURATOR panel:

Product image adding

Please, note that this feature is available only when the TreoPIM module is installed together with the "Import Feeds" module.

In the creation pop-up that appears, select the file column for the image import and/or attach the locally stored image file to be used as the default value for import. Also define the image scope level – Global or Channel.

Make sure that at least the File column field is filled in; otherwise, the given product image will not be created.

Having added the entity field, product attribute and product image records to the import feed, you can edit or remove them via the corresponding options of the single record actions menu:

Single record menu

Running Import Feed

To start importing the data via the active import feed, select the Run import option from the actions menu on its detail view page or from the single record actions menu on the "Import Feeds" list view page:

Run import option

In the pop-up that appears, attach the CSV file with data to be imported and click the Run import button to start the process:

Run import pop-up

Please, note that the data file must match the example file, defined for the given import feed. Otherwise, the error message will appear:

Wrong file error

When import is started, its details and current status are displayed in the Queue Manager pop-up that appears automatically:

Queue manager

Please, note that if the number of records in the data file exceeds the limit value defined on the OVERVIEW panel of the given import feed, the import job will be split into the corresponding parts in accordance with the limit value.

Import Results

Information about completed import jobs is displayed on the IMPORT RESULTS panel, which is empty on the import feed creation step, but gets filled in after the data import is performed via the given import feed.

Results of the data import operations can be viewed in two ways:

  • on the IMPORT RESULTS panel of the import feed – the details on the import operations performed via the currently open import feed:

Import results panel

  • on the import results list view page – the details on all import operations performed in the system via import feeds:

Import results list

The import results details contain the following information:

  • Name – the import result record name, which is generated automatically based on the date and time of the import operation start.
  • Import feed – the name of the import feed used for the import operation.
  • Imported file – the name of the data file (CSV) used for the import operation.
  • Status – the current status of the import operation.
  • Restored – the indication of whether the given import result record has been restored (the checkbox is selected) or not. Available only on the import results list view page.
  • Start – the date and time of the import operation start.
  • End – the date and time of the import operation end.
  • Created – the number of records created as a result of the performed import operation.
  • Updated – the number of records updated as a result of the performed import operation.
  • Errors - the number of errors, if any, that occurred during the import operation.
  • Error file – the name of the CSV file that contains only rows with errors. The error file name is generated automatically based on the imported file name.

The following status options are available:

  • Running – for the currently running import job.
  • Pending – for the import job, which is next in line for execution.
  • Success – for the successfully finished import job (no matter if it contains any errors in it).
  • Failed – for the import job that could be performed due to some technical issues.

To view the import result record details, click its name on the IMPORT RESULTS panel or in the import results list; the corresponding record detail view page opens:

Import result details

The error messages, if any, are displayed on the ERRORS LOG panel on this page:

Errors log

To correct the data in the defined rows, download the error file (CSV), which contains only these rows, make changes in them and run the import operation again using the corrected error file as the data file.

Alternatively, view the import result record details in the pop-up that appears when you use the View option from the single record actions menu for the desired record on the import results list view page or on the IMPORT RESULTS panel of the currently open import feed:

Import result pop-up

Please, note that you can download the imported and error files from any interface page, where their names are clickable.

Data Restoration

The "Import Feeds" module supports data restoration for separate import results record to the pre-import state. To do this, select the Restore option from the single record actions menu for the desired import result record on the import feed detail view page:

Restore option

Click the Restore button in the confirmation message that appears to start the process or Cancel to abort the process. The Queue Manager pop-up will automatically appear:

Restore process

As a consequence, the 'reverted' import result record will disappear from the IMPORT RESULTS panel, and on the import results list view page the Restored checkbox will become selected for the 'initial' import result record:

Restored checkbox

Import Feed Operations and Actions

Import feed records can be duplicated and removed whenever needed.

To duplicate the existing import feed record, use the corresponding option from the actions menu on the desired import feed record detail view page:

Duplicate feed

You will be redirected to the import feed creation page and get all the values of the last chosen import feed record copied in the empty fields of the new feed record to be created.

In order to remove the import feed record, use the corresponding option from the actions menu on the desired import feed record detail view page or from the single record actions menu on the import feeds list view page:

Remove feed

To complete the operation, click the Remove button in the confirmation message that appears.

The "Import Feeds" module also supports common TreoCore mass actions that can be applied to several selected import feed records, i.e. records with set checkboxes. These actions can be found in the corresponding menu on the import feeds list view page:

Mass actions

  • Remove – to remove the selected import feed records (multiple deletion).
  • Merge – to merge the selected import feed records.
  • Mass update – to update several selected import feed records at once. To have a longer list of fields available for mass updating, please, contact your administrator.
  • Export – to export the desired data fields of the selected import feed records in the XLSX or CSV format.
  • Add relation – to relate the selected import feed records with other import result record(s).
  • Remove relation – to remove the relations that have been added to the selected import feed records.

Import Cron Job Configuration

In order to run import automatically according to the schedule, you can configure import cron jobs. To create a new job record, click the Import Cron Jobs in the navigation menu to get to the import cron jobs list view, and then click the Create Import Cron Job button.

If there is no Import Cron Jobs option in the navigation menu, please, contact your administrator.

The common creation window will open:

Cron job creation

Here define the following parameters for the import cron job being created:

  • Active – select the checkbox to activate the job.
  • Name – enter the import cron job name.
  • Import feed – select the desired import feed, based on which data will be imported.
  • Link – enter a direct link to the example file (CSV) from which data will be imported.
  • Scheduling – enter a desired cron job schedule, i.e. frequency of job runs, using common syntax rules.

When an import cron job is started, its details are shown in the Queue Manager pop-up.

The results of the run import cron job are displayed on the LOGS panel of the corresponding job record:

Cron job logs

To view the import cron jobs queue and status details, please, go to Administration > Schedule Jobs and click the Jobs button:

Cron jobs queue

Get the "Import Feeds" module now to run import of your data manually or automatically via highly customizable and reusable import templates!

Do you have questions about this module?