Free web scraper for Cabela’s to extract data about products

With this free web scraper for the Cabela’s online store,, you can collect data for dozens of thousands of products for outdoor activities, hunting, fishing and tourism. Cabela’s company is one of the leading American retailers in the field of goods for various outdoor activities and sports. The company was founded in 1961 by Richard Cabela in the city of Sydney, Nebraska and is still managed by members of the Cabela family.

Approx number of goods: 70000
Approx number of page requests: 70000
Recommended subscription plan: Small

PLEASE NOTE! The number of requests can exceed the number of products, because data about variations, images, etc. can be scraped from other resources and will require additional requests. Also part of the product data can be delivered using XHR requests, which also increases the total number of required page requests.

How to use the web scraper to extract data about goods and prices for cabelas.com

To use the web scraper for Cabela’s store website, you must have an account with our Diggernaut service. You can just simply follow this comprehensive guide:

1. Go through this registration link to open free account with Diggernaut
2. After registering and confirming the email address, you will need to log in to your account
3. Create a project with any name and description, if you do not know how to do it, please refer to our documentation
4. Switch to the created project and create a digger with any name, if you do not know how to do it, please refer to our documentation
5. Copy the following digger configuration to the clipboard and paste it into the digger you created, if you do not know how to do it, refer to our documentation
6. Switch the mode of the digger from Debug to Active, if you do not know how to do it, please refer to our documentation
7. Run your digger and wait until the completion, if you do not know how to do it, please refer to our documentation
8. Download the scraped dataset in the format you need, if you do not know how to do it, please refer to our documentation

You can also setup a schedule for running your scraper and collect data regularly.

Scraping configuration for the digger

---
config:
    debug: 2
    agent: Chrome
do:
- walk:
    to: http://www.cabelas.com
    do:
    - find: 
        path: div.shopDropdown>a
        do: 
        - parse:
            attr: href
            filter: ^(.+\/\_\/N\-\d+)
        - space_dedupe
        - trim
        - if:
            match: \w+
            do:
            - normalize:
                routine: url
            - link_add:
                pool: catalog
- walk:
    to: links
    pool: catalog
    do:
    - sleep: 2
    - variable_clear: good
    - find:
        path: div.leftnav_content
        do:
        - variable_set:
            field: good
            value: "yes"
    - find:
        path: div.leftnav_content a
        do:
        - parse:
            attr: href
            filter: ^(.+\/\_\/N\-\d+)
        - space_dedupe
        - trim
        - if:
            match: \w+
            do:
            - normalize:
                routine: url
            - link_add:
                pool: catalog
    - find:
        path: a.entry:contains('Next')
        do:
        - parse:
            attr: href
        - space_dedupe
        - trim
        - if:
            match: \w+
            do:
            - normalize:
                routine: url
            - link_add:
                pool: catalog
    - find:
        path: div.productContentBlock>a
        do:
        - variable_set:
            field: good
            value: "yes"
        - parse:
            attr: href
        - space_dedupe
        - trim
        - if:
            match: \w+
            do:
            - normalize:
                routine: url
            - link_add:
                pool: pages
    - find:
        path: html
        do:
        - variable_get: good
        - if:
            match: "yes"
            else:
            - nocontent_category
- walk:
    to: links
    pool: pages
    do:
    - variable_clear: good
    - sleep: 2
    - find:
        path: 'div#productDetailsTemplate'
        do:
        - variable_set:
            field: good
            value: "yes"
        - variable_clear: list
        - variable_clear: desc
        - variable_clear: pid
        - variable_clear: cid
        - object_new: product
        - eval:
            routine: js
            body: '(function (){var d = new Date(); return d.toISOString()})();'
        - object_field_set:
            object: product
            field: date
        - static_get: url
        - object_field_set:
            object: product
            field: url
        - find:
            path: span.itemNumber
            do:
            - node_remove: b
            - parse
            - space_dedupe
            - trim
            - variable_set: pid
            - object_field_set:
                object: product
                field: sku
        - find:
            path: h1.label
            do:
            - parse
            - space_dedupe
            - trim
            - object_field_set:
                object: product
                field: name
        - register_set: Cabela's
        - variable_set: brand
        - find:
            in: doc
            path: script[type="application/ld+json"]
            do:
            - parse
            - normalize:
                routine: replace_substring
                args:
                - Luck\s+"E"\s+Strike: Luck E Strike
                - \"@type\"\:\s*\"Product\"\,\s+\"name\"\:.+: ''
                - \"description\"\:.+: ''
                - \n+: ''
                - \}\,\]: '}]'
            - normalize:
                routine: json2xml
            - to_block
            - find:
                path: brand>name
                do:
                - parse
                - space_dedupe
                - trim
                - variable_set: brand
            - find:
                path: offers:has(standardPrice:not(:contains("null")))
                slice: 0
                do:
                - find:
                    path: standardPrice
                    do:
                    - parse
                    - object_field_set:
                        object: product
                        type: float
                        field: price
                - find:
                    path: price
                    do:
                    - parse
                    - object_field_set:
                        object: product
                        type: float
                        field: price
                - register_set: USD
                - object_field_set:
                    object: product
                    field: currency
        - variable_get: brand
        - object_field_set:
            object: product
            field: brand
        - find:
            in: doc
            path: meta[name="description"]
            do:
            - parse:
                attr: content
            - space_dedupe
            - trim
            - variable_set: desc
        - find:
            path: div.pdp-desc-long
            do:
            - parse
            - space_dedupe
            - trim
            - variable_set: desc
        - find:
            path: 'div#description'
            do:
            - node_replace:
                path: br
                with: "\n"
            - split:
                context: text
                delimiter: \n+
            - find:
                path: div.splitted
                slice: 0
                do:
                - parse
                - space_dedupe
                - trim
                - variable_set: desc
        - variable_get: desc
        - object_field_set:
            object: product
            field: description
        - find:
            path: select.js-dropdown:has(option:contains("Select COLOR"))
            do:
            - find:
                path: option
                slice: 1:-1
                do:
                - parse
                - space_dedupe
                - trim
                - if:
                    match: \w+
                    do:
                    - object_field_set:
                        object: product
                        joinby: "|"
                        field: variations
        - find:
            path: script:contains('params_viewlarger.push("asset",')
            do:
            - parse:
                filter: \s+params_viewlarger\.push\(\"asset\"\,\s+"([^"]+)"\)\;
            - to_block
            - split:
                context: text
                delimiter: \s*[\;\,]
            - find:
                path: div.splitted
                do:
                - parse:
                    filter: ^([^\?]+)
                - space_dedupe
                - trim
                - if:
                    match: \w+
                    do:
                    - if:
                        match: _sw_
                        else:
                        - register_set: http://images.cabelas.com/is/image/<%register%>?wid=1000
                        - object_field_set:
                            object: product
                            joinby: "|"
                            field: images
        - find:
            path: script:contains('params_viewlarger.push("asset",')
            do:
            - parse:
                filter: \s+altviewparams_viewlarger\.push\(\"asset\"\,\s+"([^"]+)"\)\;
            - to_block
            - split:
                context: text
                delimiter: \s*[\;\,]
            - find:
                path: div.splitted
                do:
                - parse:
                    filter: ^([^\?]+)
                - space_dedupe
                - trim
                - if:
                    match: \w+
                    do:
                    - if:
                        match: _sw_
                        else:
                        - register_set: http://images.cabelas.com/is/image/<%register%>?wid=1000
                        - object_field_set:
                            object: product
                            joinby: "|"
                            field: images
        - find:
            path: ul.breadcrumb>li>a
            do:
            - parse
            - space_dedupe
            - trim
            - if:
                match: \w+
                do:
                - object_field_set:
                    object: product
                    joinby: "|"
                    field: categories
        - object_save:
            name: product
    - find:
        path: html
        do:
        - variable_get: good
        - if:
            match: "yes"
            else:
            - nocontent_product

Sample of scraped data

Below is a sample of a dataset with several products in JSON format (so you can easily review it and see data structure). The dataset can be downloaded as CSV, XLSX, XML, or any other text format using the templates.

[{
    "product": {
        "brand": "Cabela's",
        "categories": "Hunting|Hunting Bags & Packs",
        "currency": "USD",
        "date": "2017-12-27T13:24:43.531Z",
        "description": "For total concealment and game-stalking quiet construction, look no further than our Traditional Hydration Pack. Incorporating low-nap poly tricot with PVC backing, this 70-oz.-capacity pack features comfort shoulder straps and a low-profile design. Taste-free hydration-tube system. External zippered pocket and shock-cord cargo panel. Imported.",
        "images": "http://images.cabelas.com/is/image/Cabelas/s7_518535_510_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_518535_510_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_518535_510_alt01_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_518535_510_alt02_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_518535_510_alt03_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_518535_510_alt04_01?wid=1000",
        "name": "Cabela's Traditional Hydration Pack",
        "price": 23.88,
        "sku": "IK-518535",
        "url": "http://www.cabelas.com/product/hunting/hunting-bags-packs/pc/104791680/c/104392080/cabelas-traditional-hydration-pack/728032.uts"
    }
}
,{
    "product": {
        "brand": "Badlands",
        "categories": "Hunting|Hunting Bags & Packs",
        "currency": "USD",
        "date": "2017-12-27T13:24:46.525Z",
        "description": "Long, challenging hunting trips require you to be prepared for any obstacle you might encounter. With Cabela's-exclusive Badlands' Release Day Pack, you'll have the pocket space, support and carry capabilities you need to be just that – prepared. The legendary Air-Track™ Suspension system combines with the technical, ultralight construction to deliver easy mobility for long treks over the most rugged terrain. Made of super-quiet KXO-32 fabric with rugged Hypalon- and Kevlar-reinforced stress points for durability that stands up to years of use in the field, it also offers a multitude of pockets - five to be exact - that team up to keep all of your must-have hunting gear and accessories secure while you're on the move. It even boasts bow-carrying capabilities for added hands-free convenience. External bedroll and compression straps. Accepts up to a 110-oz. reservoir (sold separately). Comes with Badlands' Unconditional Lifetime Warranty. Imported.",
        "images": "http://images.cabelas.com/is/image/Cabelas/s7_463111_999_04?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_463111_999_04?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_463111_999_alt01_04?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_463111_999_alt02_04?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_463111_999_alt03_04?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_463111_999_alt04_04?wid=1000",
        "name": "Badlands Release Day Pack – Cabela's Exclusive",
        "price": 99.99,
        "sku": "IK-463111",
        "url": "http://www.cabelas.com/product/hunting/hunting-bags-packs/pc/104791680/c/104392080/badlands-release-day-pack/2000208.uts"
    }
}
,{
    "product": {
        "brand": "Herter's",
        "categories": "Hunting|Hunting Bags & Packs",
        "currency": "USD",
        "date": "2017-12-27T13:24:48.912Z",
        "description": "Transport your calls, ammo and other duck-hunting accessories safely and securely in Herter's Waterfowl Field Bag. A large zippered opening delivers access to the main compartment, which has an adjustable divider to separate your gear. Exterior pockets on the front, back and sides keep essentials at the ready. Adjustable shoulder strap and neoprene-wrapped handle for easy carry. Rugged 600-denier polyester fabric resists snags and tears. Imported.",
        "images": "http://images.cabelas.com/is/image/Cabelas/s7_421520_560_02?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_421520_560_02?wid=1000",
        "name": "Herter's® Waterfowl Field Bag",
        "price": 24.99,
        "sku": "IK-421520",
        "url": "http://www.cabelas.com/product/hunting/hunting-bags-packs/pc/104791680/c/104392080/herters-waterfowl-field-bag/1643686.uts"
    }
}
,{
    "product": {
        "brand": "Cabela's",
        "categories": "Hunting|Hunting Bags & Packs",
        "currency": "USD",
        "date": "2017-12-27T13:24:51.318Z",
        "description": "You won’t find a better gear bag for a lower price. Its durable, weather-resistant 600-denier polyester construction makes it ideal for toting everything from packable rain gear to extra odds and ends. Sturdy 1-1⁄2\" nylon web carry straps can be joined by a hand-friendly wrap handle. Six exterior pockets, including zippered mesh pockets on top and side, provide multiple storage and organization options. Embroidered Cabela’s logo on front pocket. Imported.",
        "images": "http://images.cabelas.com/is/image/Cabelas/s7_580032_999_03?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_999_03?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_014_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_014_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_022_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_022_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_027_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_027_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_148_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_148_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_928_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_928_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_896_05?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_896_05?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_014_alt01_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_148_alt01_01?wid=1000",
        "name": "Cabela's Catch-All Gear Bags",
        "price": 14.99,
        "sku": "IK-580032",
        "url": "http://www.cabelas.com/product/hunting/hunting-bags-packs/pc/104791680/c/104392080/cabelas-catch-all-gear-bags/753422.uts",
        "variations": "BLUE|TAN|PINK|GRAY|CAMO|O2 OCTANE"
    }
}]