Diggernaut makes it easy to work with JSON format by converting it to XML. So now I’ll show you how you can work with it.
I found one of the public JSON Rest API we use as a sample. Link.
It’s a sample of their server response:
[
{
"id": 1,
"name": "Leanne Graham",
"username": "Bret",
"email": "Sincere@april.biz",
"address": {
"street": "Kulas Light",
"suite": "Apt. 556",
"city": "Gwenborough",
"zipcode": "92998-3874",
"geo": {
"lat": "-37.3159",
"lng": "81.1496"
}
},
"phone": "1-770-736-8031 x56442",
"website": "hildegard.org",
"company": {
"name": "Romaguera-Crona",
"catchPhrase": "Multi-layered client-server neural-net",
"bs": "harness real-time e-markets"
}
},
...
...
...
...
]
First, we create a digger, switch it to debug mode, and set Firefox as the name of the browser.
---
config:
debug: 2
agent: Firefox
Let’s add do block and use walk command with the address of the test RestAPI service inside. Also, I am going to show you a little trick that should be very useful in the future. The essence of the trick is to stop the execution of digger at some point you need. To do this, add the command stop to the do block.
---
config:
debug: 2
agent: Firefox
do:
- walk:
to: http://jsonplaceholder.typicode.com/users
do:
- stopRun digger, and wait for the completion. As expected, the digger finished with an error.
Let us move to the digger’s log and see why it happened.
Unsupported operator used: stop
The system does not support this command, it raised an exception, and stopped the execution of digger, as a digger is in debug mode, and the number of errors cannot be more than 1.
Now let’s see how server response was handled by the digger.
Its how it looks in text editor.
<html>
<head></head>
<body>
<body_safe>
<element>
<address>
<city>Gwenborough</city>
<geo>
<lat>-37.3159</lat>
<lng>81.1496</lng>
</geo>
<street>Kulas Light</street>
<suite>Apt. 556</suite>
<zipcode>92998-3874</zipcode>
</address>
<company>
<bs>harness real-time e-markets</bs>
<catchphrase>Multi-layered client-server neural-net</catchphrase>
<name>Romaguera-Crona</name>
</company>
<email>Sincere@april.biz</email>
<id>1</id>
<name>Leanne Graham</name>
<phone>1-770-736-8031 x56442</phone>
<username>Bret</username>
<website>hildegard.org</website>
</element>
<element>
<address>
<city>Wisokyburgh</city>
<geo>
<lat>-43.9509</lat>
<lng>-34.4618</lng>
</geo>
<street>Victor Plains</street>
<suite>Suite 879</suite>
<zipcode>90566-7771</zipcode>
</address>
<company>
<bs>synergize scalable supply-chains</bs>
<catchphrase>Proactive didactic contingency</catchphrase>
<name>Deckow-Crist</name>
</company>
<email>Shanna@melissa.tv</email>
<id>2</id>
<name>Ervin Howell</name>
<phone>010-692-6593 x09125</phone>
<username>Antonette</username>
<website>anastasia.net</website>
</element>
...
...
...
...
...
...
...
...
...
</body_safe>
</body>
</html>
Let’s delete the command stop from out config and go to each element tag and extract and collect data from it. It’s simple. You can do it the same way as you do it with standard HTML, so I’ll describe it very briefly using comments inside config. Comments are marked with a # character.
---
config:
debug: 2
agent: Firefox
do:
- walk:
to: http://jsonplaceholder.typicode.com/users
do:
# Lets find all element tags and traverse through them
- find:
path: element
do:
Let's create an object to store data on each iteration for the element tag
- object_new: user
#Collecting data
#We will extract not all data, just few fields
- find:
path: id
do:
- parse
#Saving value of the register to the field "id" of the object "user" (futher we omit such comments as all operations are similar to this)
- object_field_set:
object: user
field: id
- find:
path: name
do:
- parse
- object_field_set:
object: user
field: name
- find:
path: username
do:
- parse
- object_field_set:
object: user
field: username
#Lets make full address merged to single field (we will use joinby option for it)
- find:
path: address
do:
- find:
path: street
do:
- parse
- object_field_set:
object: user
field: address
- find:
path: suite
do:
- parse
- object_field_set:
object: user
field: address
joinby: ", "
- find:
path: city
do:
- parse
- object_field_set:
object: user
field: address
joinby: ", "
- find:
path: zipcode
do:
- parse
- object_field_set:
object: user
field: address
joinby: ", "
#Save object "user"
- object_save:
name: userSo we built our config and can now switch digger to active mode and run it.
When it’s done, we can download data and check if all worked well. I downloaded data as JSON, and it looks this way in the text editor:
{
"user": [{
"username": "Bret",
"id": "1",
"name": "Leanne Graham",
"address": "Kulas Light, Apt. 556, Gwenborough, 92998-3874"
},
...
...
...
{
"username": "Moriah.Stanton",
"address": "Kattie Turnpike, Suite 198, Lebsackbury, 31428-2261",
"id": "10",
"name": "Clementina DuBuque"
}],
"session": "session_595",
"digger": "digger_120"
}
That’s all, if you have any questions, feel free to ask in comments, I will answer.
