Iterators

Introduction

Iterators are used to generate a list of argument sets that are used for iteration over the main block of the digger logic. Other words, the digger performs the basic logic for each element of the list, where each element is a set of arguments.

Typically, an iterator is used when you need to do search on the website using different sets of parameters as search arguments, or when you need to collect data for certain dates, using the date as the query argument.

Currently, iterators of three types are supported:
date - dates based iterators
csv - keeps single field and multiple values for this field in CSV format
fieldset - keeps fieldsets with values.

You can use multiple iterators of same or different types, in this case the system will combine all the iterators and create a single iterator with all possible sets of fields and values. This allows you to create shorter records for repeated values. For example, if we want to create an iterator that has several fields, the best option would be to use the fieldset. But if one of the fields takes values ​​from 1 to 10, then we will have to put this fieldset 10 times in configuration. So using combination of csv and fieldset iterators can shorten the iterator entry:

              iterator:
# DESCIBING BASIC FIELDSET
- type: fieldset
    fields:
    - season: fall
      category: athletics

# DESCRIBING FIELD WHICH HAS MULTIPLE VALUES
- type: csv
  name: age
  value: 1,2,3,4,5,6,7,8,9,10
              

As a result, the following list of sets of arguments will be formed, and digger will execute main logic block for each set from this list:

              [
    { "season": "fall", "category": "athletics", "age": 1 },
    { "season": "fall", "category": "athletics", "age": 2 },
    { "season": "fall", "category": "athletics", "age": 3 },
    { "season": "fall", "category": "athletics", "age": 4 },
    { "season": "fall", "category": "athletics", "age": 5 },
    { "season": "fall", "category": "athletics", "age": 6 },
    { "season": "fall", "category": "athletics", "age": 7 },
    { "season": "fall", "category": "athletics", "age": 8 },
    { "season": "fall", "category": "athletics", "age": 9 },
    { "season": "fall", "category": "athletics", "age": 10 }
]
                

As you can see, using a combination of iterators, you can quickly and conveniently describe different sets of arguments. Next, we'll see in details into each type of iterator and we will start with date iterators.