Revisions to Combine multipage API results in Mathematica

Bumped by Community user

occurred Feb 16, 2023 at 19:08

Bumped by Community user

occurred Oct 19, 2022 at 19:01

Code correction

Source Link

edited Sep 19, 2022 at 18:38

Steven Buehler

63
4

Updated to show more code

Can't seem to find a good post/answer to this so I'm asking the question.

I'm pulling data from the Swarm/Foursquare API, and it's paged (max 250 results per pull):

url = StringTemplate[ "https://api.foursquare.com/v2/users/self/checkins?startTime=`a`&\ endTime=`b`&oauth_token=`c`&limit=250&offset=`d`&v=20220406"][<| "a" -> startTime, "b" -> endTime, "c" -> oauthToken, "d" -> offset|>]; response = Import[url, "RawJSON"]"RawJSON"]; checkins = response[["response"]][["checkins"]][["items"]]; Until[Length[response[["response"]][["checkins"]][["items"]]] == 0, offset += 250; url = StringTemplate[ "https://api.foursquare.com/v2/users/self/checkins?startTime=`a`&\ endTime=`b`&oauth_token=`c`&limit=250&offset=`d`&v=20220406"][<| "a" -> startTime, "b" -> endTime, "c" -> oauthToken, "d" -> offset|>]; response = Import[url, "RawJSON"]; AppendTo[checkins, response[["response"]][["checkins"]][["items"]]] ]

I should have around 1,200 results. Instead, I have 256. What am I doing wrong here?

Updated to show more code

Can't seem to find a good post/answer to this so I'm asking the question.

I'm pulling data from the Swarm/Foursquare API, and it's paged (max 250 results per pull):

url = StringTemplate[ "https://api.foursquare.com/v2/users/self/checkins?startTime=`a`&\ endTime=`b`&oauth_token=`c`&limit=250&offset=`d`&v=20220406"][<| "a" -> startTime, "b" -> endTime, "c" -> oauthToken, "d" -> offset|>]; response = Import[url, "RawJSON"] checkins = response[["response"]][["checkins"]][["items"]]; Until[Length[response[["response"]][["checkins"]][["items"]]] == 0, offset += 250; url = StringTemplate[ "https://api.foursquare.com/v2/users/self/checkins?startTime=`a`&\ endTime=`b`&oauth_token=`c`&limit=250&offset=`d`&v=20220406"][<| "a" -> startTime, "b" -> endTime, "c" -> oauthToken, "d" -> offset|>]; response = Import[url, "RawJSON"]; AppendTo[checkins, response[["response"]][["checkins"]][["items"]]] ]

I should have around 1,200 results. Instead, I have 256. What am I doing wrong here?

Updated to show more code

Can't seem to find a good post/answer to this so I'm asking the question.

I'm pulling data from the Swarm/Foursquare API, and it's paged (max 250 results per pull):

url = StringTemplate[ "https://api.foursquare.com/v2/users/self/checkins?startTime=`a`&\ endTime=`b`&oauth_token=`c`&limit=250&offset=`d`&v=20220406"][<| "a" -> startTime, "b" -> endTime, "c" -> oauthToken, "d" -> offset|>]; response = Import[url, "RawJSON"]; checkins = response[["response"]][["checkins"]][["items"]]; Until[Length[response[["response"]][["checkins"]][["items"]]] == 0, offset += 250; url = StringTemplate[ "https://api.foursquare.com/v2/users/self/checkins?startTime=`a`&\ endTime=`b`&oauth_token=`c`&limit=250&offset=`d`&v=20220406"][<| "a" -> startTime, "b" -> endTime, "c" -> oauthToken, "d" -> offset|>]; response = Import[url, "RawJSON"]; AppendTo[checkins, response[["response"]][["checkins"]][["items"]]] ]

I should have around 1,200 results. Instead, I have 256. What am I doing wrong here?

Posted the full logic that I'm trying to use.

Source Link

edited Sep 19, 2022 at 18:35

Steven Buehler

63
4

Updated to show more code

Can't seem to find a good post/answer to this so I'm asking the question.

I'm pulling data from the Swarm/Foursquare API, and it's paged (max 250 results per pull):

url = StringTemplate[ "https://api.foursquare.com/v2/users/self/checkins?startTime=`a`&\ endTime=`b`&oauth_token=`c`&limit=250&offset=`d`&v=20220406"][<| "a" -> startTime, "b" -> endTime, "c" -> oauthToken, "d" -> offset|>]; response = Import[url, "RawJSON"] checkins = response[["response"]][["checkins"]][["items"]]; Until[Length[response[["response"]][["checkins"]][["items"]]] == 0, offset += 250; url = StringTemplate[ "https://api.foursquare.com/v2/users/self/checkins?startTime=`a`&\ endTime=`b`&oauth_token=`c`&limit=250&offset=`d`&v=20220406"][<| "a" -> startTime, "b" -> endTime, "c" -> oauthToken, "d" -> offset|>]; response = Import[url, "RawJSON"]; AppendTo[checkins, response[["response"]][["checkins"]][["items"]]] ]

I should have around 1,200 results. Instead, I have 256. What am I doing wrong here?

Updated to show more code

Can't seem to find a good post/answer to this so I'm asking the question.

I'm pulling data from the Swarm/Foursquare API, and it's paged (max 250 results per pull):

url = StringTemplate[ "https://api.foursquare.com/v2/users/self/checkins?startTime=`a`&\ endTime=`b`&oauth_token=`c`&limit=250&offset=`d`&v=20220406"][<| "a" -> startTime, "b" -> endTime, "c" -> oauthToken, "d" -> offset|>]; response = Import[url, "RawJSON"] checkins = response[["response"]][["checkins"]][["items"]]; Until[Length[response[["response"]][["checkins"]][["items"]]] == 0, offset += 250; url = StringTemplate[ "https://api.foursquare.com/v2/users/self/checkins?startTime=`a`&\ endTime=`b`&oauth_token=`c`&limit=250&offset=`d`&v=20220406"][<| "a" -> startTime, "b" -> endTime, "c" -> oauthToken, "d" -> offset|>]; response = Import[url, "RawJSON"]; AppendTo[checkins, response[["response"]][["checkins"]][["items"]]] ]

I should have around 1,200 results. Instead, I have 256.

Updated to show more code

Can't seem to find a good post/answer to this so I'm asking the question.

I'm pulling data from the Swarm/Foursquare API, and it's paged (max 250 results per pull):

url = StringTemplate[ "https://api.foursquare.com/v2/users/self/checkins?startTime=`a`&\ endTime=`b`&oauth_token=`c`&limit=250&offset=`d`&v=20220406"][<| "a" -> startTime, "b" -> endTime, "c" -> oauthToken, "d" -> offset|>]; response = Import[url, "RawJSON"] checkins = response[["response"]][["checkins"]][["items"]]; Until[Length[response[["response"]][["checkins"]][["items"]]] == 0, offset += 250; url = StringTemplate[ "https://api.foursquare.com/v2/users/self/checkins?startTime=`a`&\ endTime=`b`&oauth_token=`c`&limit=250&offset=`d`&v=20220406"][<| "a" -> startTime, "b" -> endTime, "c" -> oauthToken, "d" -> offset|>]; response = Import[url, "RawJSON"]; AppendTo[checkins, response[["response"]][["checkins"]][["items"]]] ]

I should have around 1,200 results. Instead, I have 256. What am I doing wrong here?

added 237 characters in body

Source Link

edited Sep 19, 2022 at 18:35

Steven Buehler

63
4

Updated to show more code

Can't seem to find a good post/answer to this so I'm asking the question.

I'm pulling data from the Swarm/Foursquare API, and it's paged (max 250 results per pull):

url = StringTemplate["httpsStringTemplate[ "https://api.foursquare.com/v2/users/self/checkins?auth_token=`c`&limit=250&v=20220406"][<|"c"startTime=`a`&\ endTime=`b`&oauth_token=`c`&limit=250&offset=`d`&v=20220406"][<| "a" -> oauthToken|>];startTime, "b" -> endTime, "c" -> oauthToken,  checkins "d" -> offset|>]; response = Import[url, "RawJSON"] checkins = response[["response"]][["checkins"]][["items"]]; Until[Length[response[["response"]][["checkins"]][["items"]]] == 0, offset += 250; url = StringTemplate[ "https://api.foursquare.com/v2/users/self/checkins?startTime=`a`&\ endTime=`b`&oauth_token=`c`&limit=250&offset=`d`&v=20220406"][<| "a" -> startTime, "b" -> endTime, "c" -> oauthToken, "d" -> offset|>]; response = Import[url, "RawJSON"]; AppendTo[checkins, response[["response"]][["checkins"]][["items"]]] ]

(in the above code, c is a variable that I populate outside the script)

In order to get additional data, I should have to pull with the additional offset=250 parameter (or offset=500around 1, &c.). The loop stops when the API query returns no items.

It's simple enough to compile multiple200 results in, say, PowerShell, Python, &c. My question is how would you combine the multiple pages in Wolfram to create a single data set? I'm having trouble getting this doneInstead, I have 256.

Can't seem to find a good post/answer to this so I'm asking the question.

I'm pulling data from the Swarm/Foursquare API, and it's paged (max 250 results per pull):

url = StringTemplate["https://api.foursquare.com/v2/users/self/checkins?auth_token=`c`&limit=250&v=20220406"][<|"c" -> oauthToken|>]; checkins = Import[url, "RawJSON"]

(in the above code, c is a variable that I populate outside the script)

In order to get additional data, I have to pull with the additional offset=250 parameter (or offset=500, &c.). The loop stops when the API query returns no items.

It's simple enough to compile multiple results in, say, PowerShell, Python, &c. My question is how would you combine the multiple pages in Wolfram to create a single data set? I'm having trouble getting this done.

Updated to show more code

Can't seem to find a good post/answer to this so I'm asking the question.

I'm pulling data from the Swarm/Foursquare API, and it's paged (max 250 results per pull):

url = StringTemplate[ "https://api.foursquare.com/v2/users/self/checkins?startTime=`a`&\ endTime=`b`&oauth_token=`c`&limit=250&offset=`d`&v=20220406"][<| "a" -> startTime, "b" -> endTime, "c" -> oauthToken,   "d" -> offset|>]; response = Import[url, "RawJSON"] checkins = response[["response"]][["checkins"]][["items"]]; Until[Length[response[["response"]][["checkins"]][["items"]]] == 0, offset += 250; url = StringTemplate[ "https://api.foursquare.com/v2/users/self/checkins?startTime=`a`&\ endTime=`b`&oauth_token=`c`&limit=250&offset=`d`&v=20220406"][<| "a" -> startTime, "b" -> endTime, "c" -> oauthToken, "d" -> offset|>]; response = Import[url, "RawJSON"]; AppendTo[checkins, response[["response"]][["checkins"]][["items"]]] ]

I should have around 1,200 results. Instead, I have 256.