> For the complete documentation index, see [llms.txt](https://fredhopper.gitbook.io/product-discovery/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://fredhopper.gitbook.io/product-discovery/sending-and-managing-product-data/flatfiles/incrementalprodfeed/incremental-feed-csv-data-format.md).

# Incremental Feed CSV Data Format

This page describes the data format to follow when implementing a CSV integration with Fredhopper.

### Files Required <a href="#incrementaldata-csv-standardinputformats" id="incrementaldata-csv-standardinputformats"></a>

The table below lists all the files that can be expected as part of the incremental update:

<table><thead><tr><th width="287">Filename</th><th>Description</th></tr></thead><tbody><tr><td><strong>categories.csv</strong></td><td>This file should be the exact same format as that provided in the most recent full re-index data file upload: it should contain the complete category tree rather than just the categories which are related to the catalogue items being updated as part of the incremental update.</td></tr><tr><td><strong>products.csv</strong></td><td>This file should only contain references to the product ids which are to be updated as part of the incremental update.</td></tr><tr><td><strong>variants.csv</strong></td><td>Similar in logic to products.csv, this file should only contain references to any variants which are to be updated.</td></tr><tr><td><strong>custom_attributes_meta.csv</strong></td><td>This file should contain attribute definitions for all the attributes in the catalogue, whether they feature within the incremental update or not.</td></tr><tr><td><strong>custom_attributes_values.csv</strong></td><td>This file should (ideally only) contain the product level attributes which are to be updated as part of the incremental update (optional).</td></tr><tr><td><strong>custom_variant_attributes.csv</strong></td><td>This file should (ideally only) contain the variant level attributes which are to be updated as part of the incremental update (optional).</td></tr></tbody></table>

{% hint style="info" %}
If your data model is such that there is no concept of variants, then the two variant CSV files can be omitted. Whilst these files have a .csv extension, the data columns are in fact separated by double tab characters rather than a comma.
{% endhint %}

#### Required Files <a href="#incrementaldata-csv-requiredfiles" id="incrementaldata-csv-requiredfiles"></a>

For incremental updates, only the necessary files are required. This is depends on the update scenario and on what is updated. Only variants, products or both and what type of attribute are updated. Please see *Fast Incremental* below for more information regarding the fast incremental file requirements.

* **categories.csv** - Ideally the same file as used in the most recent full data feed.
* **custom\_attributes\_meta.csv** - Minimally all relevant attributes, but is recommended to use the version from the most recent full data feed.
* **products.csv** - Only the updated products
* **custom\_attributes\_values.csv** - Recommended to export only the new attribute values. This depends on the operation type. When using 'replace' all new values need to be provided.
* **variants.csv** (optional) - Only necessary when variant data is updated. Only relevant variant id's need to be listed.
* **custom\_variant\_attributes.csv** (optional) - Only necessary when variant data is updated. Recommended to export only the new attribute values. This depends on the operation type. When using 'replace' all new values need to be provided.

{% hint style="warning" %}
All files always need to have at least one value line (apart from the header line). Optional files should not be included to the data batch when there are no values for those specific files.

If a file is created and sent but not fitting these criteria, the data-load trigger will fail. This is a fail-safe mechanism.
{% endhint %}

#### Operation Type <a href="#incrementaldata-csv-operationtype" id="incrementaldata-csv-operationtype"></a>

The full re-index process always expects that the entire product catalogue be provided within the data files provided. This differs in comparison to the incremental update process as here we may wish to only update an attribute value, add a new product or delete a variant from the catalogue entirely. The way that we inform Fredhopper which action to perform in an incremental update is defined by the 'operation\_type' attribute within the data. The locale for this attribute does not matter and if specified, it will be disregarded. For example, if a variant with two locales must be removed it is sufficient to include only one delete operation type in custom\_variant\_attributes.csv (see in the sample data below how this is done).

{% hint style="danger" %}
Please be aware that you must not specify an attribute\_value\_id for 'operation\_type' entries.
{% endhint %}

The 'operation\_type' attribute is defined within custom\_attributes\_values.csv and (if required) custom\_variant\_attributes.csv for each product / variant respectively and can be one of four values below. If no 'operation\_type' attribute is configured for an item, then the 'add' operation will be applied by default.

<table><thead><tr><th width="162">Operation Type</th><th>Description</th></tr></thead><tbody><tr><td><strong>update</strong></td><td>An existing product or variant item needs to have some of its data modified. By default, if the given item does not exist within the FAS index, no changes will be completed. This default behaviour can be overridden in the system.xml settings so that the item is processed as an add type of update instead. This operation_type cannot be used to nullify an existing attribute value or remove an attribute completely for an item - the <em>replace</em> operation must be used in this case.</td></tr><tr><td><strong>replace</strong></td><td>An existing product or variant item (and all its attributes) should be removed completely from the FAS index and will be replaced with the new data provided within the incremental update.</td></tr><tr><td><strong>add</strong></td><td>A new product or variant item has been defined within the incremental update and the attribute values need be loaded for the first time. By default, if the item exists already within the index, FAS will apply the replace logic to the existing item in this case. This default behaviour can be overridden in the system.xml so that the existing item does not get replaced and a warning will be displayed in the logs for the affected item(s) instead.</td></tr><tr><td><strong>delete</strong></td><td>An existing product or variant item needs to be removed completely from the FAS index.</td></tr></tbody></table>

*\*An item can be either a product or variant.*

{% hint style="warning" %}
FAS will execute all operations on the specified item so, if both a product and its variants need updating, all of those updates must be provided explicitly. The only exception to this rule is that, deleting a product that has variants attached, will remove both the product and all variants automatically without the need for each variant to be deleted explicitly.
{% endhint %}

#### Fast Incremental Updates <a href="#incrementaldata-csv-fastincrementalupdates" id="incrementaldata-csv-fastincrementalupdates"></a>

The speed that it takes to process and load an incremental update is already much faster than it takes a full data upload to complete. However, it is possible to have a faster, more expedited update process when sending an incremental update batch that contains only numerical and/or basic text values. In this situation, some of the standard data processing tasks and verification checks that are built into the process can be skipped because they are not necessary. This method should be used whenever possible to provide near real-time updates of numerical and/or text values, for example to apply stock and/or price updates. For this to work, the operation\_type can only be 'update' and the input files will only contain a subset of the product/variant attributes that are to be updated.

The data API will automatically complete a fast incremental update when all of the following conditions have been met:

* The custom\_attributes\_meta.csv file is present in the input data-incremental.zip file
* The custom\_attributes\_values.csv and/or the custom\_variant\_attributes.csv file(s) is(are) present in data-incremental.zip file
* The operation\_type values for all changes are set to 'update' (using any other value will result in a 'normal' incremental update)
* The data type of all the attribute values to be updated are either int, float or text (using any other data type will result in a 'normal' incremental update)

### Incremental Product Data - Input Format <a href="#incrementaldata-csv-incrementalproductdata-inputformat" id="incrementaldata-csv-incrementalproductdata-inputformat"></a>

The data model depicted in the examples below represents a single universe, multi locale deployment which has two categories (womens & shoes) that contain two products and three variants in total. The update scenario that we would like to achieve is as follows:

* p1 is updated with a new colour attribute value (blue)
* p2 is added as a new item to the product catalogue
* v1 is removed from the product catalogue
* v2 is updated with a new price value (30.00)

#### categories.csv <a href="#incrementaldata-csv-categories.csv" id="incrementaldata-csv-categories.csv"></a>

<table><thead><tr><th>category_id</th><th>parent_category_id</th><th width="134">locale</th><th>name</th></tr></thead><tbody><tr><td>catalog01</td><td>catalog01</td><td>en_US</td><td>Shop</td></tr><tr><td>catalog01</td><td>catalog01</td><td>de_DE</td><td>Geschaeft</td></tr><tr><td>women</td><td>catalog01</td><td>en_US</td><td>Women</td></tr><tr><td>women</td><td>catalog01</td><td>de_DE</td><td>Damen</td></tr><tr><td>shoes</td><td>catalog01</td><td>en_US</td><td>Shoes</td></tr><tr><td>shoes</td><td>catalog01</td><td>de_DE</td><td>Schuhe</td></tr></tbody></table>

#### products.csv <a href="#incrementaldata-csv-products.csv" id="incrementaldata-csv-products.csv"></a>

<table><thead><tr><th width="206">product_id</th><th width="211">locale</th><th>category_ids</th></tr></thead><tbody><tr><td>p1</td><td>de_DE</td><td>shoes</td></tr><tr><td>p1</td><td>en_US</td><td>shoes</td></tr><tr><td>p2</td><td>de_DE</td><td>shoes</td></tr><tr><td>p2</td><td>en_US</td><td>shoes</td></tr></tbody></table>

#### custom\_attributes\_meta.csv <a href="#incrementaldata-csv-custom_attributes_meta.csv" id="incrementaldata-csv-custom_attributes_meta.csv"></a>

<table><thead><tr><th width="195">attribute_id</th><th width="168">type</th><th width="158">locale</th><th>name</th></tr></thead><tbody><tr><td>stock</td><td>int</td><td><br></td><td>Stock</td></tr><tr><td>color</td><td>set</td><td>en_US</td><td>Color</td></tr><tr><td>color</td><td>set</td><td>de_DE</td><td>Kleur</td></tr><tr><td>price</td><td>float</td><td><br></td><td>Price</td></tr></tbody></table>

#### custom\_attributes\_values.csv <a href="#incrementaldata-csv-custom_attributes_values.csv" id="incrementaldata-csv-custom_attributes_values.csv"></a>

<table><thead><tr><th width="134">product_id</th><th width="135">locale</th><th>attribute_id</th><th width="174">attribute_value_id</th><th>attribute_value</th></tr></thead><tbody><tr><td>p1</td><td>en_US</td><td>color</td><td>blue</td><td>Blue</td></tr><tr><td>p1</td><td>de_DE</td><td>color</td><td>blue</td><td>Blauw</td></tr><tr><td>p1</td><td>en_US</td><td>operation_type</td><td><br></td><td>update</td></tr><tr><td>p2</td><td>en_US</td><td>color</td><td>red</td><td>Red</td></tr><tr><td>p2</td><td>de_DE</td><td>color</td><td>red</td><td>Rot</td></tr><tr><td>p2</td><td><br></td><td>stock</td><td><br></td><td>10</td></tr><tr><td>p2</td><td>en_US</td><td>operation_type</td><td><br></td><td>add</td></tr></tbody></table>

#### variants.csv <a href="#incrementaldata-csv-variants.csv" id="incrementaldata-csv-variants.csv"></a>

| variant\_id | product\_id | locale |
| ----------- | ----------- | ------ |
| v1          | p1          | en\_US |
| v1          | p1          | de\_DE |
| v2          | p1          | en\_US |
| v2          | p1          | de\_DE |
| v3          | p2          | en\_US |
| v3          | p2          | de\_DE |

#### custom\_variants\_attributes.csv <a href="#incrementaldata-csv-custom_variants_attributes.csv" id="incrementaldata-csv-custom_variants_attributes.csv"></a>

<table><thead><tr><th width="132">variant_id</th><th width="109">locale</th><th width="150">attribute_id</th><th>attribute_value_id</th><th>attribute_value</th></tr></thead><tbody><tr><td>v1</td><td><br></td><td>operation_type</td><td><br></td><td>delete</td></tr><tr><td>v2</td><td><br></td><td>price</td><td><br></td><td>30.00</td></tr><tr><td>v2</td><td>en_US</td><td>operation_type</td><td><br></td><td>update</td></tr></tbody></table>