Other Commands

Clearing Cache

If you are loading pages in the unique mode, the digger saves the URL address of each processed page in a special cache and skips (does not load) already processed pages in the current run or in all subsequent digger runs.

However, sometimes there are situations when you need to delete the page from the cache. Since page gets into the cache automatically when digger tries to load it, we can not re-read it with the page_reload command in case if proxy was blocked. In order to work around such situations, we added the link_remove command. It deletes the URL of the current page from the cache and the digger can retrieve it again.

Usage example:

              - walk:
    to: http://somesite.com/page.html
    mode: unique
    do:
    - find:
        path: body
        do:
        # CHECK IF PROXY IS BLOCKED
        - parse
        - if:
            match: "request has been blocked"
            do:
            # SWITCH PROXY
            - proxy_switch
            # DELETE CURRENT URL FROM THE CACHE
            - link_remove
            # RELOAD PAGE
            - page_reload