If for your work you need to collect data from Instagram business profiles, you probably used a mobile application for it. You were forced to do it because there were no some business data in the web version. In particular, it was impossible to determine if you are looking at the business profile or personal. Now it’s possible to process them automatically with a web scraper using the mobile API. We found this solution on the Internet, one of our users wrote it and shared it with the community on one of the popular Internet marketing resources. Let’s examine how the web scraper works.
Article updated on 11.17.2019 due to changes to the API
To use the web scraper, you must specify the login and password for your Instagram account, and the list of accounts you want to collect business information about. Bear in mind that Instagram can block your account if using this web scraper may violate the TOS, so use it at your own risk, we are publishing it just for educational purposes. Below is the actual web scraper code:
---
config:
agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36
debug: 2
do:
- variable_set:
field: username
value: YOUR INSTAGRAM ACCOUNT USERNAME
- variable_set:
field: password
value: YOUR INSTAGRAM ACCOUNT PASSWORD
- variable_set:
field: accounts
value: LIST OF INSTAGRAM ACCOUNTS, COMMA SEPARATED, YOUR WANT TO EXTRACT BUSINESS DATA ABOUT
- walk:
to: https://www.instagram.com/
do:
- find:
path: body
do:
- parse:
filter: window\._sharedData\s+\=\s+([^;]+);
- normalize:
routine: json2xml
- to_block
- find:
path: config>csrf_token
do:
- parse
- variable_set: token
- walk:
to:
post: https://www.instagram.com/accounts/login/ajax/
headers:
x-csrftoken:
x-instagram-ajax: 1
x-requested-with: XMLHttpRequest
data:
username:
password:
do:
- find:
path: status
do:
- parse
- if:
match: "fail"
do:
- cannot_login_probably_checkpoint_is_required
- exit
- find:
path: authenticated
do:
- parse
- if:
match: "true"
else:
- wrong_login_or_password
- exit
- cookie_get: mid
- variable_set: mid
- cookie_get: rur
- variable_set: rur
- cookie_get: ds_user_id
- variable_set: dsuserid
- cookie_get: sessionid
- variable_set: sessionid
- variable_get: accounts
- to_block
- split:
context: text
delimiter: ','
- find:
path: div.splitted
do:
- parse
- space_dedupe
- trim
- variable_set: account
- agent_set: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36
- walk:
to: https://www.instagram.com//
do:
- find:
path: script:contains("window._sharedData =")
do:
- parse
- space_dedupe
- trim
- filter:
args:
- window\._sharedData\s+\=\s+(.+)\s*;\s*$
- normalize:
routine: json2xml
- to_block
- find:
path: body_safe
do:
- find:
path: entry_data > profilepage > graphql > user > id
do:
- parse
- variable_set: id
- agent_set: Mozilla/5.0 (iPhone; CPU iPhone OS 12_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 Instagram 105.0.0.11.118 (iPhone11,8; iOS 12_3_1; en_US; en-US; scale=2.00; 828x1792; 165586599)
- walk:
to: https://i.instagram.com/api/v1/users//info/
headers:
X-IG-App-ID: 567067343352427
X-IG-Capabilities: 3brDAw==
X-IG-Connection-Type: WIFI
X-IG-Connection-Speed: 3400
X-IG-Bandwidth-Speed-KBPS: -1.000
X-IG-Bandwidth-TotalBytes-B: 0
X-IG-Bandwidth-TotalTime-MS: 0
Cookie: mid=; csrftoken=; rur=; ds_user_id=; sessionid=; ig_or=;
X-FB-HTTP-Engine: Liger
Accept: '*/*'
Accept-Language: en-US
do:
- find:
path: body_safe > user
do:
- object_new: item
- find:
path: address_street
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: address_street
- find:
path: category
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: category
- find:
path: city_name
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: city_name
- find:
path: contact_phone_number
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: contact_phone_number
- find:
path: external_url
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: external_url
- find:
path: full_name
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: full_name
- find:
path: is_business
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: is_business
- find:
path: latitude
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: latitude
- find:
path: longitude
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: longitude
- find:
path: pk
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: id
- find:
path: public_email
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: public_email
- find:
path: public_phone_country_code
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: public_phone_country_code
- find:
path: public_phone_number
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: public_phone_number
- find:
path: username
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: username
- find:
path: zip
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: zip
- object_save:
name: item
- sleep: 5As you probably already know, the config section is intended for presetting the scraper. In this case, to set the debug mode level (which is only required for development and could be omitted) and the browser name on behalf of which the web scraper sends requests to the server. Technically, there could be Chrome or Safari, but the author decided that there should be Firefox. By the way, sometimes the server can give different data, depending on the name of the browser. Also, sometimes it may be necessary to use a complete User-Agent string instead of a preset, they can be found here.
config:
agent: Firefox
debug: 2The main logic block of the scraper is located in the section do. At the very beginning, the variables are initialized with your login, the password for the Instagram and the account list you want to extract:
- variable_set:
field: username
value: YOUR INSTAGRAM ACCOUNT USERNAME
- variable_set:
field: password
value: YOUR INSTAGRAM ACCOUNT PASSWORD
- variable_set:
field: accounts
value: LIST OF INSTAGRAM ACCOUNTS, COMMA SEPARATED, YOUR WANT TO EXTRACT BUSINESS DATA ABOUTNext, the web scraper loads the Instagram homepage and goes into the body tag.
- walk:
to: https://www.instagram.com/
do:
- find:
path: body
do:It parses all the text and extracts the Javascript object, translates it into XML and turns it into a DOM block, and then switches to this context.
- parse:
filter: window\._sharedData\s+\=\s+([^;]+);
- normalize:
routine: json2xml
- to_blockNow in our context, there is an extracted Javascript object (JSON) as DOM, and we can walk through its elements as if it was a standard HTML page. So we find the config node and inside of it the csrf_token node, parse the content and extract the token that we need for the login to the Instagram. We save it to the token variable. Then we log in to Instagram using the token, username and password, which we are already keeping in variables:
- find:
path: config>csrf_token
do:
- parse
- variable_set: token
- walk:
to:
post: https://www.instagram.com/accounts/login/ajax/
headers:
x-csrftoken:
x-instagram-ajax: 1
x-requested-with: XMLHttpRequest
data:
username:
password: Next, the scraper checks whether Instagram has authorized us.
- find:
path: status
do:
- parse
- if:
match: "fail"
do:
- cannot_login_probably_checkpoint_is_required
- exitSo if not, you will see an error and the scraper finishes the work. If you see this error in the log, try logging in through your browser and manually resolve the challenge. After that, you’ll be able to sign in to your account from web scraper. If the authorization is successful, the scraper will continue to work and transfer the necessary cookies to the variables to be able to use them in requests:
- find:
path: authenticated
do:
- parse
- if:
match: "true"
else:
- wrong_login_or_password
- exit
- cookie_get: mid
- variable_set: mid
- cookie_get: rur
- variable_set: rur
- cookie_get: ds_user_id
- variable_set: dsuserid
- cookie_get: sessionid
- variable_set: sessionidThen the scraper reads the variable with the list of accounts into the register and converts the text in the register to the block and switches to this context. It is done to use the command split since the command works with the contents of the block, not the register. After splitting, the scraper iterates through each account and executes commands in the do block:
- split:
context: text
delimiter: ','
- find:
path: div.splitted
do:All that happens next applies to every account listed in the CSV string you passed. The scraper parses the block which contains the account name, clears it of extra spaces and writes it to the variable so it can be used in requests.
- parse
- space_dedupe
- trim
- variable_set: accountThe scraper takes the page of the channel to extract the channel ID because we need channel ID to call the mobile API. The ID is stored in the variable.
- walk:
to: https://www.instagram.com//
do:
- find:
path: script:contains("window._sharedData")
do:
- parse
- space_dedupe
- trim
- filter:
args:
- window\._sharedData\s+\=\s+(.+)\s*;\s*$
- normalize:
routine: json2xml
- to_block
- find:
path: body_safe
do:
- find:
path: entry_data > profilepage > graphql > user > id
do:
- parse
- variable_set: idAfter that, a request is made to the Instagram mobile API. As we can see, the web scraper is masking for mobile application, using specific request headers.
- walk:
to: https://i.instagram.com/api/v1/users//info/
headers:
X-IG-App-ID: 567067343352427
X-IG-Capabilities: 3brDAw==
X-IG-Connection-Type: WIFI
X-IG-Connection-Speed: 3400
X-IG-Bandwidth-Speed-KBPS: -1.000
X-IG-Bandwidth-TotalBytes-B: 0
X-IG-Bandwidth-TotalTime-MS: 0
Cookie: mid=; csrftoken=; rur=; ds_user_id=; sessionid=; ig_or=;
X-FB-HTTP-Engine: Liger
Accept: '*/*'
Accept-Language: en-US
do:The mobile API returns a response in JSON format. Diggernaut automatically converts it to XML and lets you work with the DOM structure using the standard find command. So all further code picks up the data using specific CSS selectors and saves them to the data object.
- object_new: item
- find:
path: biography
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: biography
- find:
path: follower_count
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: follower_count
- find:
path: address_street
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: address_street
- find:
path: category
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: category
- find:
path: city_name
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: city_name
- find:
path: contact_phone_number
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: contact_phone_number
- find:
path: external_url
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: external_url
- find:
path: full_name
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: full_name
- find:
path: is_business
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: is_business
- find:
path: latitude
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: latitude
- find:
path: longitude
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: longitude
- find:
path: pk
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: id
- find:
path: public_email
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: public_email
- find:
path: public_phone_country_code
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: public_phone_country_code
- find:
path: public_phone_number
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: public_phone_number
- find:
path: username
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: username
- find:
path: zip
do:
- parse
- space_dedupe
- trim
- object_field_set:
object: item
field: zip
- object_save:
name: itemIn general, we think, the logic of the web scraper is simple, there is the only complicated point in the process of masking for a mobile application. An example of the data obtained is given below:
{
item : {
category : "Product/Service",
username : "adidas",
is_business : "true",
contact_phone_number : "",
zip : "91074",
public_phone_number : "",
longitude : "10.9094251",
latitude : "49.5831932",
public_phone_country_code : "",
full_name : "adidas",
city_name : "Herzogenaurach",
address_street : "Adi-Dassler-Str. 1",
id : "20269764",
public_email : "",
external_url : "http://a.did.as/BuiltToDefy"
}
}