Keyword Research: How I Generated 15,000 Keywords and Found the 20 Best
What do you do when you’re not very skilled at keyword research? Without quality keywords, you can’t create relevant content that people will search for. You can’t set up Adwords campaigns that convert. You can’t optimize landing pages or set up a site’s architecture to lead people to the information they want. I […]
What do you do when you’re not very skilled at keyword research?
Without quality keywords, you can’t create relevant content that people will search for. You can’t set up Adwords campaigns that convert. You can’t optimize landing pages or set up a site’s architecture to lead people to the information they want. I wasn’t very good, but I wanted to be. So to get some tips, I chatted with Rand Fishkin, founder of Moz and a speaker at the Growth Marketing Conference.
After talking to Rand and reading a bunch of blog posts on the subject, I was able to generate about 15,000 keywords, narrow them down to a useful group of about 300, and then assign a priority score to help me narrow down the top 20 keywords to target first.
Here’s how I did it.
Start by talking to the people who fit your customer personas. In this case, I talked to people who love beer.
Step 1 (Do This if You Don’t Know Much About Your Customers)
Set up interviews with the type of people who you think will buy your product:
How much do you know about the people who are going to use your product? The startup I did keyword research for (let’s call them Startup X) is a craft beer delivery startup with three customer personas:
Gift Givers – Urban/suburban. Scours the internet for the best, most special, unique, unusual, and luxurious gifts.
Foodies – Urban. Instagrams their food. Pairs beer and food. Looks for high quality food and drink.
Beer Aficionados – Loves trying all types of beers. Goes to beer tasting parties and beer tours. Passionate about deepening their beer knowledge.
But while Startup X had a good idea about their customers, I personally didn’t. So I called people I knew who fit these personas and asked them some questions.
As a seasoned product designer once told me, people are terrible at imagining what they would do in a hypothetical scenario, but they’re great at telling you what they’ve already done. So I asked questions such as:
Can you tell me about a positive beer drinking experience you had recently?
Can you tell me about the last time you drank homebrewed beer?
What was the last gift you bought? How did you find it?
Then I asked plenty of follow ups:
When/where was this?
What type of beer was it?
What did the beer taste like?
Who were you with?
For general tips on customer development and creating interview guides, check out the incredible (FREE!) book Talking to Humans. In terms of generating a large quantity of keywords, this wasn’t the most effective technique, but it was crucial to understand these personas for some of the more powerful keyword generation methods described later.
Read Other Content/Digital Media About Your Vertical
As I Googled beer-related keywords, I noticed a general trend in which the larger the publication, the less the writing resembled everyday speech. Serious Eats is one of the top food media sites around and takes on a pretty casual tone, but you’ll notice that their Beer Bucket List sounds writer-y, with words and phrases like “inextricably,” “unified balance,” and “dangerously drinkable.” But some other sites generated new keywords. This niche beer site called More Beer didn’t sound very conversational either, but it opened up all types of ways people can discuss beer. This led to new keywords related to taste, history, ingredients, and brewing process.
As seasoned SEO’s know, forums are loaded with great keywords, and unlike digital media publications, they much more closely resemble how people talk – and thus, how they search. BeerAdvocate is a forum that was frequently mentioned in my research and was reported on SEMrush to have over 1.3 million clicks coming from organic search, so I dove in. Here I looked for themes in the forum users’ conversations with each other. I noticed lots of talks about trades, specific yeast strains, and home brewing methods. I found smaller details like the fact that they frequently discussed growlers, a type of large bottle frequently used by microbreweries. There were 100 new keywords from these forums. BeerAdvocate also has a list of the top 250 beers as reviewed by members, so I used Import.io to scrape the list in about 10 seconds, adding the names of beers, breweries, and beer styles to the list of potential keywords.
Look It Up on Wikipedia
Wikipedia is filled with the kinds of keywords and topics that you won’t find in Keyword Planner. One of the best resources in my keyword research journey was Backlinko’s Definitive Guide to Keyword Research In it, Brian Dean advocates using Wikipedia for keyword research. Keyword Planner is great at spitting out keywords that include your search terms. So when I inputted “beer” in Keyword Planner, it suggested keywords like “beer clubs,” “beer halls,” and “rare beers.” What it couldn’t do is suggest related keywords like “Belgian yeast,” “clarifying agent,” or “commercial hops,” none of which include the word “beer.”
That’s where Wikipedia comes in, especially the Table of Contents which lists each section.
I found the Wikipedia entries for subjects like “beer,” “microbrewery,” and “beer in the United States,” and then I used Kimono to scrape the Table of Contents for each of those entries.
Go Through Table of Contents for Books on Amazon
Also courtesy of Backlinko is this trick. Similar to Wikipedia, by finding books related to your topic on Amazon, you can often find their Table of Contents through the Look Inside/Preview feature, showing you images of the book pages. By adapting chapter titles to keywords, I found all kinds of words that didn’t include the word beer but were highly relevant to Startup X. In looking at the book How to Brew, for example, I found keywords like “malt extract,” “malt sugar,” “dry-hopping,” and “how to measure hops.”
Step 2 (Do This if You Want to Learn More About Your Competitors):
Do Competitive Analysis on SEMrush:
As startup marketers know, SEMrush is a critically valuable tool for SEO, Paid Search, and Content because it tells you which keywords are driving the most traffic to your site and your competitors’ sites. In addition, you can see what keywords your competitors are bidding on, what their Adwords ads look like, and where they’re getting all their backlinks. I looked through the SEMrush pages for related beer sites like BeerAdvcoate, beer of the month sites, and also subscription box sites (even ones not related to beer) to see what keywords were bringing people to their sites. My client hypothesized that wine subscription companies’ sites would be especially relevant because they often have the budgets to hire experienced marketers to properly optimize sites and run profitable Adwords campaigns. Looking at ads was very helpful as well to help differentiate between top, middle, and bottom of the funnel keywords. In competitive analysis there’s always a level of inference and hypothesis, but the guess was that keywords that were heavily bid on were more likely to be valuable to them and more likely to be valuable to Startup X.
Step 3 (Do This Once You’ve Established Customer Personas):
Generate Keywords On Your Own
There’s nothing like rolling up your sleeves and brainstorming potential keywords. If you’re worried about being influenced by your research, you can do this in the very beginning. I found I was better at thinking of broader topics (“homebrew recipes”, “beer pairings”) than long tail keywords, and after about 50 keywords, I was ready to move on.
Analyze the Keywords on Your Own Site
I wanted to find which keywords Startup X currently had across its domain. This became especially valuable as I compared these keywords’ volume to the keywords I was generating with my other methods. In doing so, I discovered that Startup X was (perhaps unknowingly) optimizing its title for two keywords that were relatively low volume in comparison to some similar keywords that were higher volume and no more difficult.
Get into your customers’ heads and brainstorm niche topics related to your product.
Create Keyword Niches for Each of Your Personas
This was maybe the most valuable of the techniques I used (and another learned from Backlinko). Here I tried to get into the heads of each of my personas to think of topics that surrounded the product being sold, in this case craft beer. For foodies, for example, that meant topics like beer pairings, local beers, seasonal beer, beer education, and beer tours. For gifters, I thought of all the ways they could land on Startup X as a perfect gift, such as gifts for dads, gifts for beer lovers, gifts for urbanites, and limited edition gifts. I then put these niche topics into Keyword Planner to get suggested related keywords.
Use Ubersuggest or KeywordTool.io
You know how as you’re typing a google search term, Google starts to make suggestions for what you might want to search for? Ubersuggest and KeywordTool.io give you the ability the input any number of words and get all the possibilities that Google would suggest for you. So when I input “beer” into Ubersuggest, it returned suggestions like “beer can chicken,” “beer cheese soup,” “beer cheese dip,” and “beer can chicken oven.” I did the same with all my niche topics, and for terms like “craft beer” and “beer club.” It’s pretty easy to end up with some obscure long tail keywords if you search for very narrow-focused terms, but when I kept them broad enough, these search results provided some of my most valuable keywords.
Step 4 (Do This if You’ve Been Around a Little While)
Check Out Your Traffic on Google Search Console
Google Search Console (previously called Google Webmaster Tools) allows you to see all the search terms that led people to your site. If you start to notice people arriving to your ski cap site with the keyword “ski cap made in USA,” you know there’s potential there for you to include the keyword “made in the USA” on your pages, in your content, and in your sales copy. For an early stage startup like Startup X, Google doesn’t currently show it on many search engine results pages (SERPs) so almost everyone who comes through Google did so by searching for some variation of the keyword “Startup X.”
The more established you are, the more data you will have to play with, and the more keywords you’ll find.
Step 5 (Everyone Should Do This)
Use Keyword Planner
Not surprisingly, Google’s own tool is the most powerful. There are lots of options for generating keywords, from analyzing your landing page (which I did for startupx.com), exploring a product category (wine and beer collecting and brewing), to finding related keywords for any keyword you input (I used the niche topics I brainstormed). Monthly search volume doesn’t properly account for seasonal variation, so if you have a seasonal product, try using Google Trends to get that data.
Don’t get overwhelmed! Now that you have a giant list, a few simple tactics will eliminate thousands of keywords.
How to Start When You Have a Giant List of Keywords
Now that I had my master list, I wanted to look at estimated monthly search volume and suggested bid. To do so, I had to run them through Keyword Planner. As SEO’s are well aware, the monthly search volume as generated by Keyword Planner is often incorrect. But Keyword Planner should provide rough ballparks, and it allows you to compare volume between keywords which I needed to rank my keywords. Comparing the search volume numbers generated by different platforms would be unreliable.
Make Sure You Get All Your Keywords from Keyword Planner
First, I removed duplicates. Then I did a search for all special characters and removed them because Keyword Planner won’t show search volume for any keywords that include them. No matter how many rows there are on the CSV you upload to Keyword Planner, it will only return a maximum of 800 keywords, so I broke them up into individual files and uploaded them one by one. By downloading the results into individual CSV’s and consolidating the results, I was left with 8,549 keywords. And after sorting by search volume, I removed everything with less than a search volume of 100. That left me 1,880 keywords with plenty of long tail keywords.
How to Evaluate Keywords
Evaluating keywords is the secret sauce to good keyword research, but no one says it’s easy. “It’s very time-consuming and unfortunately today, it’s pretty manual,” Rand said. At the time of our conversation, he had just been whiteboarding a new product that could evaluate which keywords are most valuable. As Rand explains, effectively evaluating keywords is especially difficult when your startup is early stage because you don’t know how long your customer journey will be and you don’t have data on how particular terms or phrases will perform. “Do people convert after reading two or three posts on your blog? Do they convert after ten? Do they almost never convert but they amplify to the people who eventually do? It’s pretty challenging to know those things ahead of time,” Rand said.
According to Rand, the most common mistake made by people new to keyword research is assuming that people will convert directly from a keyword to a transaction and then ignoring keywords that don’t immediately lead to conversions.
“The path to conversion is a long and winding journey,” Rand said. “Ranking for a keyword and earning a visit is only a part of that long-winding journey. If you ignore that fact and ignore keywords that aren’t bringing you directly converting traffic, your competitors are probably going to take big advantage of that. And many times the long-term value of ranking for a keyword can be greater for keywords that don’t bring you directly converting traffic.” To help you make informed guesses ahead of time, Rand has noticed that most sophisticated SEO’s judge keywords by four criteria:
Based on these four, they then assign an aggregate priority score to ultimately rank the keywords and figure out which they should target first. Here’s how I created my own version of his model:
As rough and inaccurate it can be, Rand advocates using Keyword Planner to find keyword search volume. “We’re in the process of evaluating other sources of data including SimilarWeb, which we think may have some really good data,” Rand said. “For now, Adwords is where I point people,”
This estimates how difficult it would be for your page to rank for a given keyword. “For that, you have to look at the top 10 results,” Rand said. “How powerful are the pages? How many links and how powerful are the links that point to these pages?” While a more experienced SEO might actually dig through these pages to create a difficulty score themselves, I went straight to the Moz Keyword Difficulty Tool. By accounting for the page and domain authority of the top twenty search results for any given keyword, Moz can give me a numeric value to tell me how likely it is that I could rank for that keyword.
All SERPs are not created equal. For example, searching for “homebrew podcast” reveals a page with ten blue links and nothing else. As Rand explained, on pages such as these, the first result might get 45% of searchers clicking through to the page. Compare that to the results for “growler,” which includes the Product Listing Ads at the top, a definition underneath, image results below the first result (a Wikipedia article), and Adwords ads at the bottom of the page (not shown in the picture below). Even if your page ranks on the first page of a SERP like this one, your click-through rate (CTR) will be far lower than on a SERP like the one for “homebrew podcast.” There’s no large scale tool to quantify this, but Rand did tip me off to a tool that Moz has created called Keyword Opportunity. Described by Rand as an alpha product, there’s no CSV upload or download, so I ended up spending over an hour copy and pasting my keywords and transcribing the results in an Excel doc. While in the future I’d likely outsource the task to Fiverr or Upwork, going through them manually taught me a lot about how Moz is calculating the opportunity score. For the keyword “beer of the month,” the SERP includes nine Adwords ads, but the score remains a very high 81% (the higher the score, the larger the opportunity). Anyone who has advertised on Adwords knows how easily and often the ads are ignored. So if Startup X ranks as the first result, they’d still likely enjoy a high CTR on a SERP filled with Adwords ads like this. Sitelinks, on the other hand, immediately sinks the Keyword Opportunity score.
Searching for “beer association” returns a first result with six sitelinks (“Insights & Analysis,” “State Craft Beer Stats,” etc.) Though you’re not bombarded with Adwords ads like with “beer of the month,” the keyword has a much lower opportunity score of 24% because those six sitelinks take up so much space and likely return an extremely high CTR for brewersassociation.org. With the combination of those six site links and the Twitter section, this SERP only has room for six results. Not much of a chance for Startup X to rank, and even if it did, its CTR would likely be very low.
“That’s the company’s internal estimate, from 1 to 10 or 1 to 100 for how important it is for that company to rank for that keyword,” Rand explained.
Some factors to consider, according to Rand:
How directly someone will convert to becoming a customer by searching for that keyword
The value that keyword provides for a company
If it’s a branded keyword (like your company’s name), how important it is to your company to rank for that keyword
I went through each keyword and assigned my estimate. Because I don’t have as much context as Startup X themselves to assign a score, I ended up scoring 1 through 3. If it was a keyword that indicated that the searcher had a high level of interest in investigating or purchasing product like Startup X (such as “craft beer of the month club”), I assigned it a 3. If the path to conversion looked less clear but it seemed to be a keyword that one of the three target personas would search for (such as “how to brew beer at home”), I assigned it a 2. If the keyword had little to do with the target personas or was a more general beer term (such as “beer ingredients”), I assigned it a 1. The next time around, I’ll likely do as Rand suggested and include a larger range like 1-10 because 1-3 didn’t quite give me enough variation.
From these four scores, you create a fifth priority score. From this Whiteboard Friday video in which Rand explores this topic further, it seems like experienced SEO’s consider each case individually and assign the score manually. To create a provisional aggregate score, I threw the four criteria together in a quick and dirty equation:
(Search Volume) x (Business Value) x (Opportunity Score) x (1 – (Keyword Difficulty))
I subtracted the Keyword Difficulty from 1 because the lower the Keyword Difficulty, the better. The idea here was that roughly each component would be weighted equally and the higher the score, the better. This gave me a gigantic number, so I divided by 100 and rounded to whole numbers to make it a little easier to read. Here’s a link to the spreadsheet format I used to evaluate my keywords. (Note: Most keywords were removed on behalf of Startup X.)
How did it work?
It certainly isn’t a perfect score, as I noticed that volume ended up having the biggest impact on the score. So “IPA” has the highest priority score because it gets 90,500 searches when perhaps a more compelling keyword like “beer of the month club” ranks third despite also having a very high search volume of 22,200. I ended up sorting by the various criteria and looking for the outliers to see what keywords looked like low hanging fruit. For example, I sorted by lowest keyword difficulty or lowest search volume and looked for keywords with an unusually high priority score. After sorting and resorting, I found about 20 keywords I thought were most relevant to Startup X.
The process of selecting keywords is a lot like the keywords themselves. Much like there’s no one perfect keyword, there isn’t a cookie cutter method for everyone to follow. And the only way to know what works is to test out your educated guesses and see what works. What I did learn was how to integrate keyword research into a broader goal of understanding customers by brainstorming niche topics related to a product and talking to customers directly. I saw that following conversations about related topics on forums and digital media can yield new topics and keywords. Next time, I’ll likely look for these conversations on social media channels as well. I found that the Table of Contents on Wikipedia and individual books can be valuable to better understand how others think about the product and related topics, and it seemed the most relevant for topic modeling and deciding site architecture. I also learned to use the many ready-made tools that are available, such as SEMrush, Ubersuggest and Google Search Console. And perhaps most importantly, I learned that the process of evaluating keywords is simultaneously art and science. Properly evaluating keywords is clearly something that an SEO gets more effective at with time, though the suite of Moz tools certainly helped me get started. After going through this process, I found plenty of the type of keywords Startup X was looking for, those with just high enough search volume but low enough competition to be interesting. The next time you do keyword research, with these tips and tricks, hopefully you can too.
Head of Marketing - Gorgias.io
Head of Marketing - Gorgias.io - #1 Rated Shopify Helpdesk. He helps build contagious brands and passionate fans. He’s regularly seen speaking and blogging about growth and digital marketing in San Diego, San Francisco, and across the universe.