This article was originally posted on my personal blog
In this tutorial, we'll go over how to search through each of Twitter, Instagram, Tumblr, and Pinterest. We'll be setting up a server with express with a simple setup just to demonstrate the functionalities.
You can find the code for this tutorial in this GitHub repository.
Project Setup
Create a directory for the project then switch to it:
mkdir social-search
cd social-search
Next, let's initialize it with NPM:
npm init
You can enter any information that's relevant for you, or just leave the default values.
Once done, we'll install some initial packages we'll need:
npm i express axios dotenv
Where express is for our server, axios is to send requests which we'll use with some of the social media platforms, and dotenv which we'll use to store necessary tokens in .env.
Then, create a file called index.js in the root with the following content:
require('dotenv').config()
const app = require('express')()
const axios = require('axios')
app.listen(3000, function () {
console.log("started server for social search!")
})
This will create a simple server that listens on port 3000. Last thing we'll need to do to start our server is add a start script in package.json:
"scripts": {
"start": "node index.js"
},
That's it! To test our server, let's run:
npm start
If everything worked correctly, you'll see a message saying "started server for social search!"
Now, we're ready to start searching through social media.
Searching Twitter
To search Twitter, you first need to create a new app on twitter's developer portal. Depending on whether this is your first app or not, you might need to enter some extra information, but generally, you just need to enter the name of the app.
Then, you will be given the API key and the API secret. Make sure to copy them and save them somewhere.
Once the app is created, go to "Keys and Tokens" tab on the app's page. Under "Authentication Tokens", click on "Generate" for Access Token and Secret. You will be given two extra keys that you need to copy and save somewhere.
Back to our project, create .env file which will hold the keys:
TWITTER_API_KEY=
TWITTER_API_SECRET=
TWITTER_ACCESS_TOKEN=
TWITTER_ACCESS_TOKEN_SECRET=
where the first 2 keys are the first 2 you received, and the second 2 are the ones you generated.
To access Twitter's API easily, we'll use the library twitter-api-client. It provides an easy to use interface to access Twitter's APIs. We need to install it first with NPM:
npm i twitter-api-client
Now, go to index.js, require the library and initialize the Twitter client:
const TwitterClient = require('twitter-api-client').TwitterClient
const twitterClient = new TwitterClient({
apiKey: process.env.TWITTER_API_KEY,
apiSecret: process.env.TWITTER_API_SECRET,
accessToken: process.env.TWITTER_ACCESS_TOKEN,
accessTokenSecret: process.env.TWITTER_ACCESS_TOKEN_SECRET,
});
Since we've added at the beginning of the file require('dotenv').config(), we can now use the keys we added in .env through process.env.
Next, we'll add a new GET route that takes query parameter, which we'll use to search through Twitter:
app.get('/twitter/:query', function (req, res) {
});
Inside the callback function for this route, we need to search through twitter for tweets including this given query. To search tweets, we'll use the twitterClient.tweets.search which takes as a parameter an object that includes various properties. The one we'll use is q which we'll pass the query:
const query = req.params.query
twitterClient.tweets.search({
q: query
}).then((response) => {
res.send(response)
}).catch((err) => {
console.error(err)
res.status(500).send('An error occurred, please try again later.')
})
When we get a response, we're just sending it back. If an error occurs, we're printing it to the console and sending a status 500 response.
That's it! To test it out, start the server:
npm start
Then go to the route we specified, passing it any query you want. For example, localhost:3000/twitter/cats. If everything is done correctly, you should see a JSON object that holds an array of tweets.
You can look at the documentation for twitter-api-client to understand more the options you can pass, the response, etc...
Searching Instagram
Instagram APIs are a mess. Long story short, now there's Hashtag Search that can be accessed through Instagram Graph API. However, this only allows you to search through posts of a certain user, not search in general. Not to mention that this API needs you to access a bunch of endpoints in order to finally get the posts with the hashtag you're querying.
There's another unofficial way to search through Instagram. There's an endpoint you can find at https://www.instagram.com/explore/tags/KEYWORD/?__a=1, where when you replace KEYWORD with the query, you'll get a JSON response with all the posts using the hashtag.
We'll create a new endpoint that allows us to search through Instagram:
app.get('/instagram/:query', function (req, res) {
});
Inside the callback, we'll use axios to send a GET request to the endpoint we mentioned above, then send back the response once received:
const query = req.params.query
axios.get('https://www.instagram.com/explore/tags/' + query + '/?__a=1')
.then((response) => {
res.send(response.data)
})
.catch((err) => {
console.error(err)
res.status(500).send('An error occurred, please try again later.')
})
That's it. To test it out, start the server:
npm start
Then, go to the route we just created, passing it the query you want. For example, localhost:3000/instagram/cats. If everything was done correctly, the response will be a JSON object with the posts using the hashtag cats.
Searching Tumblr
To search through Tumblr, first register a new application. You'll need a tumblr account to do that. You'll have to enter information like Application name, Application URL (you can just enter example.com), etc...
Once done, you'll be redirected to the Applications page and it will show you the newly created app. You can see the OAuth Consumer Key. Copy it and add it to .env:
TUMBLR_CONSUMER_KEY=
To search posts by tags, we can use the /tagged endpoint. We'll pass it the API key, which is the Consumer key we just added to .env, and the tag which is the query we're searching.
First, we'll create the new GET route:
app.get('/tumblr/:query', function (req, res) {
const query = req.params.query
});
As usual, we're creating a route that takes a query as a parameter.
Then, we'll send a GET request to Tumblr's /tagged endpoint using axios passing it the parameter as explained above:
axios.get('https://api.tumblr.com/v2/tagged?tag=' + query + '&api_key=' + process.env.TUMBLR_CONSUMER_KEY)
.then((response) => {
res.send(response.data)
})
.catch((err) => {
console.error(err)
res.status(500).send('An error occurred, please try again later.')
})
As before, we're just sending back the response once received.
To test it out, start the server:
npm start
Then, go to the route we created passing it the query you want. For example, localhost:3000/tumblr/cats. If everything is working correctly, the response will be a JSON response with an array of posts using the tag cats.
Searching Pinterest
Pinterest doesn't have an API that allows you to search through their pins. To search through Pinterest, we'll have to use Google's Custom Search JSON API. This API allows you to retreive search results in JSON format. We'll use it to search through Pinterest. It's not optimal, but at the moment of writing this, it's the easiest way. You should note that this API provides 100 queries per day for free, then you'll be billed 5$ for every additional 1000 query.
First, you'll need to get an API key to use. Just click the "Get a Key" button. It will ask you to create a project or use an existing one, then you'll be provided with an API key. Copy the key and add it to .env:
CUSTOM_SEARCH_API_KEY=
Next, you'll need to create a "Programmable Search". You can do that here. You'll need to enter the URL of the site you'll be searching. In our case it will be pinterest.com. You can also change the language and Name of the Programmable Search. Once done, click Create. It will take you to a page that will show you a bunch of options. Choose "Control Panel" button.
Once you're on the Control Panel, copy the "Search Engine ID" and add it to .env:
CUSTOM_SEARCH_CX=
We'll need to pass this as a parameter when we send our requests to the API.
Let's start by creating the GET route to query Pinterest:
app.get('/pinterest/:query', function (req, res) {
const query = req.params.query
});
Next, we'll send a request to the Custom Search API endpoint passing it the API key, CX which is the Search Engine ID, and the query we're searching:
axios.get('https://www.googleapis.com/customsearch/v1?key=' + process.env.CUSTOM_SEARCH_API_KEY +
'&cx=' + process.env.CUSTOM_SEARCH_CX + '&q=' + query)
.then((response) => {
res.send(response.data.items)
})
.catch((err) => {
console.error(err)
res.status(500).send('An error occurred, please try again later.')
})
As usual, we're just sending back the response received.
To test it out, start the server:
npm start
Then, go to the route we just created, passing it a query. For example, localhost:3000/pinterest/cats. You'll see a JSON array, each object having the title of the pin, the link to it, and other information.
Conclusion
In this tutorial, we went over searching Twitter, Instagram, Tumblr, and Pinterest. In the next one, we'll go through searching more social media platforms!