One of the many problems facing SEO’s these days is how to carry out effective keyword research.
With [not provided] now accounting for about 80-90% of organic keyword data in Google Analytics, we’re effectively working blind when it comes to trustable data on what search terms are being used.
There are lots of ways to shed light on keyword data – from studying the landing page data in Analytics to relying on data from Bing/Yahoo and using the search query reports in Google Webmaster Tools (which are ridiculously rubbish if you ask me).
The most popular way to research different keywords, get new keyword ideas and compare search volumes is to use the Keyword Planner in Google AdWords.
However, I’m increasingly becoming more and more uncomfortable with using this tool as the data it provides doesn’t quite seem right.
Because I work in an agency that handles a lot of PPC accounts, I have a lot of paid-for data at my disposal so I decided to put together a quick comparison together to see if there are any big discrepancies between actual AdWords data and the figures in the Keyword Planner.
For the record, I picked the test keyphrases at random and got the AdWords data from the ‘See Search Terms’ reports of each campaign. All the keywords were receiving ‘first page’ impressions throughout the month, so the ‘Impressions’ data is probably the best representation of search volumes for these keyphrases I can get.
Test 1 – musical instruments
Here’s the data from AdWords for two reasonably popular musical instrument-based search terms for the month of October 2013 (I’ve blocked out some irrelevant data so I don’t get in trouble with the PPC team!):
As you can see, impressions for the month for [Yamaha flute] were 4,559, and for [buffet clarinet] they were 3,909.
Now let’s see if these impression figures correspond with the ‘Avg. Monthly Searches’ in the UK in Google’s Keyword Planner:
Errm, no. That’s a pretty huge discrepancy. I know this is an ‘average’ figure taken from 12 months, but even accounting for that a difference of about 670% (for [Yamaha flute]) is ridiculous!
Let’s try another one…
Test 2 – [magic whiteboard]
Again for the month of October, here’s the impressions data for the search term [magic whiteboard]:
And here’s what the Keyword Planner says:
Only a difference of about 200% this time!
Let’s do one final test:
Test 3 – product term
For this I’ve picked the fairly long-tail term [Samsung ue40f6400] – which is a TV.
Here’s what the AdWords data gives us for October 2013:
And here’s the Keyword Planner data:
That’s a difference of about 187%.
I get the feeling that I could go on all day with this and get the same results!
So what’s going on here? Is the Keyword Planner simply not to be trusted, or is my non-scientific study ignoring some obvious fact that is skewing the data?
I’d love to get your opinions on this so please join the discussion in the comments below, also – let me know what tools/processes you use to determine search volumes and pick your target keywords because as of now I officially don’t know the best way to research keywords for an on-page optimisation plan!
One last thing, let me know if you have any cool ideas for studies and insights you might want to see from all the AdWords data we have (obviously with no brand terms included!) – I’m thinking of putting together a big research study together based on ‘what types of keywords convert’, what do you think?
Luke Glassford 
Are these broad matched?
Hi Dave,
The Keyword Planner shows the exact match data. I used the ‘See Search Terms’ report in AdWords to get exact match impression data for the month.
Hi Luke, quick question. Are you positive the Adwords impression data is for exact match keywords? I know keyword planner only gives exact match numbers, but when you add keywords to an Adwords campaign, they are added as broad match by default. I could be way off here, but I THINK that may be why you are seeing such a big discrepancy.
I’ve used data from the See Search Terms report in AdWords, rather than looking at the performance of actual keywords in different adgroups – here’s how Google explains the See Search Terms report: https://support.google.com/adwords/answer/2472708?hl=en “A search term is the exact word or set of words a customer enters when searching on Google.com or one of our Search Network sites”
Hi Luke,
I think that’s the problem. You didn’t include the Search Parner searches in Keyword Planner, I did it for Yamaha flute and it changed the volume from 590 to 5,400 which changed the margin of error from 670% to just 19%.
Considering 5,400 is a 12-months average and October has been the lowest month according to Google Trends, that brings the margin of error to almost 0 and the keyword planner is now accurate.
Another factor that I see might be affecting your impressions is “Exact Match Close Variant”. Close Variant includes variations of the keywords such as plurals.
Let me know what you think.
Hi Al, I think you’re right to some degree – changing the Keyword Planner to include Search Partner data gets the figures closer to the PPC impressions on some keyphrases, but not for all. Looking at the [samsung ue40f6400] keyphrase, the Keyword Planner including Search Partners estimates 2,900 monthly searches – not much difference to the non-Search Partner data and still a long way off the PPC impressions of 6,902.
So we get back to square one again – how can we trust the tool when there’s just so many different figures coming out of it?
You’re right Luke. Would love to see another blogpost covering an even larger group of keywords. IMHO, there’ll always be margin of errors but the question is, what’s acceptable?
Mine is 10% but I’m actually working on a solution that will help get this value to around 3-5%. It’s based on Rand Fishkin’s method described on http://moz.com/blog/100-percent-keyword-not-provided-whiteboard-tuesday
You enter a URL and the software will do the following:
1-Crawl your website for all your landing pages.
2-Analyze your pages content and come up with a list of currently ranking terms, and keywords you can possibly rank for.
2.1 Assign a Rankability and KEI value to each keyword.
2.2 Actually track the ranking of each keyword in real time(daily).
2.3 Keep generating new keywords from already existing keywords to find ranking opportunities.
3. Besides the Volume data from Keyword Planner, it’ll also use the data from Google Analytics and Google Webmaster Tools, correlate and extrapolate to come up with what I’m calling “True Traffic”.
4. The value for “True Traffic”(trademarked, lol) will only get accurate over time as it tracks and analyze historical Analytics and GWT data.
5. Would-be-nice feature: Add a tracking pixel so you can track actual conversion and measure actual keyword ROI (again).
Thoughts?
p/s: Is it OK if I add you to my personal list of people that might be interested to give this app a whirl once it’s done mid-Jan 2014? It’s totally cool if you say no, I’ll still come to your blog 🙂
Sounds interesting, sign me up!
Hi Luke,
I’ve done this same type of comparison between Keyword Planner stats and real impressions seen in existing AdWords campaigns. I think there might be a problem with your comparison numbers for the keywords you have here and it might be more related to the versions you have listed.
For instance, the following Exact keywords should be listed in your AdWords UI screenshots with the brackets around them []. However your images do not show them with the brackets, which indicated they are set to Broad match.
Exact match (with brackets): [Yamaha flute]
Broad match (without brackets): Yamaha flute
Exact version (with brackets): [buffet clarinet]
Broad match (without brackets): buffet clarinet
Since the images have the match type column blacked out, it is not transparent which type is being displayed, but if they were Exact match, the brackets would be there.
Please note that this is very important for accurately correlating search volume between Keyword Planner data and AdWords Impressions for the same keywords because Broad match keywords can map to such a wide keyword set and can even map to session based searches.
In conclusion the research that I’ve also done does suggest that Keyword Planner data is not completely accurate as well. And to that your post “can you trust Google’s Keyword Planner?” can be answered with: Not really.
Cheers!
Hi James, thanks for the comment.
The reason the [exact match] brackets aren’t showing is because I took the screen drops from the See Search Term report, which doesn’t show them. I’ve just re-checked and they are all definitely exact match.
I’m just going to re-do the images to show this, so give me 10 minutes and you’ll see 🙂
Hi Luke, Just read the comments from the others that were writing theirs as I was writing mine! There are also a few other things you probably should also look into:
1. Does the search volume for those Exact match keywords also include Search Partner impressions or Google only?
2. Does your Geographic targeting match between the 2 sources?
3. Does your language settings match between the 2?
4. How about specific monthly matches and not just the average?
Hi James, thanks for the input. I think the Search Partner impressions could skew the data a bit, and the fact the Keyword Planner data is as an average could also affect the results so they probably need to be looked into more.
The targeting is UK and language is English for both the Keyword Planner and AdWords data.
Just changed the AdWords images to show the Match Type info.
Stupid question, but I assume that the keywords shown are set to show in the UK only and the Keyword Planner was set to UK only?
Hi James, yes I set Keyword Planner to UK and the AdWords accounts in question are only targeting the UK
Hey Luke,
I couldn’t trust the data displayed in Keywords Planner hence this study seems to come off right. Comparing actual impressions data makes sense and I believe if you can perform a research on a large no. of keywords and compare the data in Keywords Planner with that of PPC impressions and do some correlation study, we can get SOME formula which we can rely on.
I did same sort of comparison of data showed by Keywords Planner and what Adwords Keywords Tool (RIP), which by the way was much better, used to show. This was based on about 1500 keywords and while I’m not really convinced with the result, it is something to take into consideration while finalizing keywords for a campaign. I’m sure with this value of data at your disposal, we could really get something.
Wonderful post though, and sorry for the rant.
Whilst I can’t comment on the reason for the huge impression discrepancy, I’ve always found the keyword planner to be difficult to do research for non-transactional type searches (since the data comes from SEM).
I’ve found myself more & more using a mixture of this, SEMRush & also something like UberSuggest to grab variants from search suggestions.
The keyword planner can’t be trusted, that’s for sure. But that difference between average monthly searches and impressions can be explained (at least to some point) with people who look through several pages in Gooogle SERP – it’s still same search, but different impressions of the Adwords block.
Also working on a website, which is targeting quite specific area, some of the keywords have only 10 searches, according to Keyword Planner (yes, i know it is not the real 10). But what I see in Webmaster tools is that impressions of the website on that keyword, are much much more than 10.
James S. mentions researching specific monthly matches vs. averages. Do you do this using the Keyword Planner, or another way? This is a metric I would like to measure, but when I use the Keyword Planner to run this search (for example, number of searches for March 2014 and number of searches for April 2014) the results I get don’t seem right.
Luke,
I’ve noticed some troubling results when using G’s new keyword Planner this month.
Specifically, I found a keyword phrase that meets my client’s business offering perfectly that was also trending upward according to the GKP.
I took a screenshot of the monthly performance stats on June 2 that reported 33,100 searches for April. But I then did another querie on the same term on the 23rd and this time the GKP reported NO searches at all!
I then tried it again 2 hours later and this time is reported 5400 searches for April.
Which of these results are most accurate?
Thanks for your advice.
Doug
I would like to know if Google’s Keyword Planner has a minimum search threshold below which it does not show search traffic even if there is some search traffic. For example, I never see traffic numbers such as an average of 2 searches per month. Or an average of 5 searches per month. Or 7 searches per month. Why is that? If there is a minimum average monthly search traffic threshold that Google doesn’t consider worth reporting, what is it? Because for some products (especially very expensive products where sale of just a single product is worth a lot of money), search traffic even is the average number of searches is small, is worth know. Does anyone have any insights or direct knowledge about this? — Burnet