Lately, thousands of Google Analytics properties had been hit by spam affecting the quality of its reports and in some cases leaving them useless for any analysis. Making some people wonder about the effectiveness of GA.

But let’s be fair Google Analytics is still the most powerful web analytics service on the Internet and on top of that it’s free (unless you are paying the 150,000 dollars the premium service costs). Plus it also provides us with the necessary features to stop and clean the mess the spam is making on your reports.

So no need freak out, there is still a way to get back your precious data. I will show you how to prevent the spam in Google Analytics by creating ONLY 2 filters, one for ghost spam, and one for crawler spam.

There are many ways to do it, but these solutions have been proved to work by me and over 100 thousand readers that used this guide over the last months.

On this article, I’ll focus on preventing future hits, to remove spam from historical/past data this article will help you. I recommend you to follow this guide first.

If you don’t have the time or need help to apply what is explained in both articles, I can set and configure everything for you  Show me

Some signs that you might have spam:

• The clearest is the presence of suspicious referrals on your reports. Below you will find a list of the currently active spammers.
• Drastic changes in your metrics like very High Bounce Rate or Low Avg. Session time.
• Another common characteristic is that the spam leaves (not set) in various dimensions like Country, City and especially in the Hostname

– List of Recent/Active Spammers

Short on time use the Quick Navigation (here ↓ or at the corner ↘)

What is Referrer Spam?

It is called referrer spam because initially it was showing as a fake referral, but the spam has evolved and it can also show up as a search term (organic keyword), a page, an event or even as a direct visit.

Referrer VS Referrer Spam

Let’s start by explaining what is a legit Referrer. A referrer is a name that is passed along when a browser goes from one page to another page (through an HTTP header), and usually it is used to indicate where the user is coming from.

Now, this referrer can be changed for anything, and some people will set it to be a page they want to promote. Then they will make repeated requests (probably with an automated script) with the intention of showing in the reports and logs of this WebSite, this is called Referrer Spam.

Why are they messing with my reports?

You may wonder how they get benefit from this. The most obvious reason is traffic; people are curious by nature, and they want to know what is going on their websites, so they go to the referral URL to check it. The surprise comes when you get there, and there is no mention to your site at all.

These spammers hit thousands of Google Analytics properties so you can imagine the amount of traffic they are getting with this method. Sometimes they want to promote a page, sell you some SEO service and sometimes they will redirect you to aliexpress.com a known online store where they will get a cut if someone buys anything through an affiliate program.

How do they do it? (Types of Spam)

The most common types of spam are: Ghost Spam (floating-share-buttons.com) and Crawler Referrer Spam (success-seo.com).

Ghost Spam

For example: floating-share-buttons.com \ referral | satellite.maps.ilovevitaly.com / referral

Ghost referrals on Google analytics are the vast majority, and they are called Ghost because they never access your site, so there is no interaction with any of your pages. It hits directly your Google Analytics.

To do this, and without going much into technicalities, they use the Measurement Protocol, which is intended to allow developers to send data directly to Google Analytics Servers. This has been possible for a long time, but it wasn’t well documented, so not everyone knew about it.

Measurement Protocol is a good thing, it allows more ways to gather and measures data, but people found how to use it in a bad way. The only thing they need is a GA tracking ID, and the rest is injected by them, referrals, pages, hostnames, keywords, etc.

How they get the IDs, well most likely they generate random codes in the form of UA-XXXXXX-1 and then using an automated script they send fake data to our Reports.

No matter if you use WordPress, Joomla, Shopify or any other Content Manager System (CMS), the only way to stop ghost spam in Google Analytics is with filters.

A common mistake is to try to block them using server solutions like the .htaccess or plugins. The .htaccess is a configuration file that can control who access your WebSite, but as we know now, this type of spam never reaches your site.

People get confused and think they successfully blocked Ghost Referrals from the .htacces file because this ghost spam usually shows up only for a few days and then disappear, so it is just a coincidence.

Below in the post you can find a demonstration

Crawler Referrer Spam

For example semaltmedia.com /referral | sitevaluation.org / referral | videos-for-your-business.com / referral

A Web crawler is an Internet bot that browsers WebSites, typically for the purpose of Web indexing. Like Google bots that find pages and indexed them so you can find them when you search, these are good crawlers.

A Crawler Referrer Spam also browse sites but with a different purpose like getting traffic to their site. This crawler usually ignores all rules like robots.txt which are supposed to stop the spiders from “crawling” specific pages.

The main difference between a Ghost and a Crawler is that the second actually visit your site, therefore it will use a valid hostname. Since this type of spam has an interaction with your site, you can block it using server solutions like the htaccess file or web.config, but filtering in GA will be enough for most of the cases.

---

How to detect Referrer Spam?

The simplest way is by looking for unusual and suspicious referrals; their sites won’t have any real reference to yours. But you want to avoid visiting them so to be sure you can use the data they leave in your reports.

Checking Referral Hostname

This method is the most reliable way at the moment to detect Ghost Referrer Spam. This type spam will have this field either as (not set) or as a fake hostname. To find the hostname they are using:

1 Go to Reporting on the to tabs and Select a wide Timeframe the bigger, the better

2 In the lateral bar select Aquisition

3 Expand All Traffic and Select All Channels

4 Click on Secondary Dimension type Hostname and select it

As you can see Ghost Referrer Spam (in Red) have an Invalid Hostname since they don’t know who are they targeting. The Crawlers Referrer Spam(Orange) on the other hand use valid Hostnames.

Other ways to detect spam (Expand)

Ghost Spam is easier to spot since it uses false data, Crawlers, on the other hand, are harder to detect. If you are still unsure and you suspect of a referral find a list below in the post with all the spam (It is updated frequently).

Avoid using the referral exclusion list and the htaccess

The referral exclusion list is made for other porpuses and shouldn’t be used to block spam in Google Analytics. If you add spam here you will get rid of the referral part of the hit and it will show as a direct visit instead, leaving you with a bigger problem since it will be harder to detect.

Another thing you should avoid is trying to block ghost spam with server-side solutions like the htaccess file or the web.config file. The best case scenario it won’t have any effect, the worst since these files are very sensitive just a misplaced character could shut down your entire site.

Here is a demonstration of why server solutions won’t work on ghost spam. I took a segment of a Google Analytics Report with all Referrer Spam (crawler and ghost) that hit me on March, 2015.

I used AWStats to analyze the access log of my site (same month) and looked for all the Spam on the previous list. As you can see only semalt.com and buttons-for-website.com, marked in red, are logged, these 2 are Crawler Referrer Spam. The rest marked in blue are all Ghost Referrer Spam.

Of course if you are comfortable working with the htaccess file or you have someone to do it for you, you can use it ONLY for crawler spam, below in the post you can find a guide to do it.

Referrer Spam List.

I update the list frequently so you can keep it as a reference.

The only way to stop Ghost Spam like floating-share-buttons.com, is by creating filters. The Valid Hostname Filter. will take care of all of them

For the Crawler Spam you can use a Campaign Source filter with a expression with all crawler spam.

How to stop the Spam in Google Analytics with 2 filters

A combination of 2 filters to stop all the spam in google analytics:

• Valid Hostname Filter that will take care of all ghost spam in any form (referral, organic or fake direct visit)
• Campaign Source Filter with an expression built with all the known Crawlers Spam.

As an extra you can enable the built-in feature “Exclude all hits from known bots and spiders“, this option will exclude all the traffic from bots/crawlers that are not spam but they aren’t human visits either.

Below I explain with detail how to keep your Google Analytics clean using these methods. If you don’t have the time or still need help configuring the necessary filters and settings I can give you a hand Read More…

Valid Hostname Filter (Recommended for Ghost Spam).

This is by far the most efficient way to stop the spam in Google Analytics, I have been using it for 5 months now in my principal view and I haven’t received a single hit from ghost spam since I applied it. There are 3 huge advantages of this method over others.

+ You will stop the spam before it hits you, adding a filter for the referral after you see it will stop it but by the time you apply it you will have already records of the spam.

+ You will need only ONE filter to stop ghost spam, instead of creating various sets of filters with all the spam, one filter will do the work.

+ Lately some of the spammers have been hitting GA accounts with fake direct visits along with the referral, the filter for the referral won’t stop the direct visit, on the other hand. The Valid hostname filter will stop ALL ghost spam in any form whether it shows as a referral (e.g. floating-share-buttons.com), keyword (e.g. erot.co) or direct visit.

How does this filter works? All ghost spam uses an Invalid Hostname since as I explain before the Spammer doesn’t know who the target is, if you check the spam you will see either a fake hostname or is “(not set)”.

The most important part of this solution is to get a list of all VALID HOSTNAMES to not exclude any legit traffic.

As a best practice, you should always have a view without filters no matter what are the filters for, so if you don’t have one yet I recommend you to create it.

To create this filter first you need to get a list of your hostnames

1 Go to the Reporting tab on GA  and Select a wide Timeframe on the calendar.

2 In the lateral bar select Audience.

3 Expand Technology and Select Network.

4 At the top of the report make sure you select Hostname because by default you Service Provider is selected.

5 Once there, you will see a table like this. Find and Copy all the valid hostnames.

Valid hostnames

Mainly your hostnames will include all the places where you put the tracking code (UA-XXXXXX-1) of Google Analytics www.yourdomain.com, yourdomain.com, blog.yourdomain.com

You may have services in which you also add your tracking id like your shopping cart these are also valid hostnames yourshopingcart.com. If your site  has visitors from many countries and they use a translation service on your pages then those will appear as a valid hostname translateservice.com.

Not valid hostnames

Essentially, any other hostname that you don’t recognize even known names like google.com or amazon.com (spammers use this names to mislead people). A (not set) hostname it is also invalid.

6 Once you gather all your valid hostnames, you should create an Expression that match all of them.
yourdomain\.com|otherdomain\.com|translateservice\.com|yourshoppingcart\.com|otherservice\.net
The expression could be more simple or more complex depending on the website age and size. If you have subdomains it is not necessary to add all of them if you add yourdomain.com; it will match any subdomain like blog.yourdomain.com or help.yourdomain.com

All custom filters in Google Analytics use Regular Expressions(REGEX) here are some things that may help you build your expression.

Once you have the REGEX build, you should add it to an INCLUDE Hostname filter.

7 Go to the Admin tab and select the View where you want to apply the filter.

8 Select Filters

If you haven’t created an “Unfiltered” View, create one now it will help you to check later if this or any other filter is working correctly

9 Select New Filter 

10 Select Create New Filter and enter Valid hostname filter as a name.

11 In Filter Type select Custom

12 Make sure you choose Include and select Hostname from the dropdown.

13 Finally, paste the REGEX that you build with your valid hostnames in Filter Pattern.

I strongly recommend you to verify this filter before applying it. You can save time on testings and protect your data. You can find more information about filter verification and how to use it here Save time and protect your data, use filter verification

So before saving select Verify this filter. It will show a table showing you sample data of before and after applying the filter.

After you make sure no valid data is excluded you can save the filter. Sometimes the verification is limited by the sample data it takes. As an alternative, you can create a segment first to test your expression.

This filter doesn’t require much maintenance but is VERY IMPORTANT that every time you add a Hostname, or you include your tracking-ID in any service like PayPal or youtube, you include it at the end of the REGEX.

The filter will take care of future hits of ghost spam, to see clean reports with previously affected data, you can follow this article: Remove Referrer Spam from historical data with Segments

Excluding good Bots and Spiders

This kind of Crawlers are not bad, on the contrary, they keep running the internet and helping us get the better results for our searches. But they also add records when visiting your site, records that are not useful.

You shouldn’t block these bots / spiders because that can make you less visible for the web, but you can exclude the safely from your Analytics.

Google recently add a function to do this a lot easier. You can see full details here Google Analytics Bots and Spider Filtering.

1 Go to the admin tab  

2 Select the View you want to apply it

3 Click on View Settings

4 Check Exclude all hits from known bots and spiders, almost at the bottom of the settings screen.

5 Save and its DONE!

Selecting this option will exclude all hits that come from bots and spiders on the IAB know bots and spiders list. The backend will exclude hits matching the User Agents named in the list as though they were subject to a profile filter. This will allow you to identify the real number of visitors that are coming to your site. -Google Analytics

Using Campaign Source (Crawler Spam)

Crawler Spam uses a valid hostname so you will need a different a different filter. You can do it by creating an Exclude Campaign Source filter with an expression that matches all known Crawler Spam.

Note: You can also use it if for Ghost Spam, but the Valid Hostname is by far a more efficient solution for this type of spam.

1 Go to your Google Analytics account and select Admin tab.

2 Under View Column Select  Filters    

3 Click on New Filter    

4 Enter Crawler Spam Filter as a name.

5 Select Filter Type Custom. In Filter Field, find and select Campaign Source. Enter the Crawler Spam expression in the Filter Pattern text box.

This is the expression that I’m currently using for Crawlers:

(best|100dollars|success)\-seo|(videos|buttons)\-for|anticrawler|musica\-gratis|semalt|forum69|7makemoney|sharebutton|ranksonic|sitevaluation|dailyrank

If you use the filter verification tool, you can actually see the filter working. It will show you the data before and after applying the filter.

This tool takes a sample of data of 7 days(not including the present day) so if in those days there is no appearance of the Spam the verification won’t work. If you followed the instructions carefully the filter will work.

You can find more information about filter verification and how to use it here Save time and protect your data, use filter verification

 

6 After you set everything Save. You can repeat this process for all the other spam.

It’s important to consider that it may take up to 24 hours before filter effects become visible in your data. So don’t worry if you still see them after you apply it, they will disappear eventually.

Using the .htaccess file (ONLY Crawler Spam)

This another way to block the spam in google analytics, in specific Crawler Spam.

This file controls who access your site, between many other things, and it’s useful to block Crawler Referrer Spam like semaltmedia.com, sitevaluation.org, videos-for-your-business.com / referral.

Don’t try to use it for Ghost Spam because it won’t have any effect since as I mentioned before this kind of referrer never accesses your site.

Remember always to make a backup before changing anything and be careful when you modify it because even one misplaced character can leave your site inaccessible.

One of the most common Crawler Referrer Spam is semalt.com  if you don’t want to use .htaccess file you can use one of the solutions here 4 ways to STOP semalt.com referral spam

If you feel comfortable working with the .htaccess file, you can use any of these mods

With mod_rewrite

## STOP CRAWLER SPAM
RewriteCond %{HTTP_REFERER} semalt\.com [NC,OR]
RewriteCond %{HTTP_REFERER} 100dollars\-seo\.com [NC,OR]
RewriteCond %{HTTP_REFERER} best\-seo\-solution\.com [NC,OR]
RewriteCond %{HTTP_REFERER} buttons\-for\-your\-website\.com [NC,OR]
RewriteCond %{HTTP_REFERER} forum69\.info [NC,OR]
RewriteCond %{HTTP_REFERER} buttons\-for\-website\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ranksonic\.com [NC,OR]
RewriteCond %{HTTP_REFERER} 7makemoneyonline\.com [NC,OR]
RewriteCond %{HTTP_REFERER} sharebutton\.net [NC,OR]
RewriteCond %{HTTP_REFERER} sitevaluation\.org [NC,OR]
RewriteCond %{HTTP_REFERER} dailyrank\.net [NC,OR]
RewriteCond %{HTTP_REFERER} videos\-for\-your\-business\.com [NC,OR]
RewriteCond %{HTTP_REFERER} semaltmedia\.com [NC,OR]
RewriteCond %{HTTP_REFERER} best\-seo\-offer\.com [NC]
RewriteRule .* - [F]

With mod_setenvif
# Set Referrer Spam as spambot
SetEnvIfNoCase Referer semalt.com spambot=yes
SetEnvIfNoCase Referer semaltmedia.com spambot=yes
SetEnvIfNoCase Referer sitevaluation.org spambot=yes
SetEnvIfNoCase Referer dailyrank.net spambot=yes
SetEnvIfNoCase Referer videos-for-your-business.com spambot=yes
SetEnvIfNoCase Referer buttons-for-website.com spambot=yes
## add all the SPAM sites you wantOrder allow,deny
Allow from all
Deny from env=spambot

 

Changing your tracking ID

(This solution is no as effective as it used to be, you will still be less vulnerable from many of the spammers but now there are some that target higher ids)

This method doesn’t exactly block Referrer Spam, but it makes your Google Analytics less visible to them. Is a good option for fresh Websites.

Since this kind of Spam usually targets UA-XXXXXXX-1 ID’s, if you change your Google Analytics tracking ID for one that doesn’t end in 1 like UA-XXXXXXX-12 most of the Referrer won’t reach you.

Although there are some cases where Referrer Spam hits higher IDs, you will be still less susceptible to attacks.

I tested this solution with good results for Ghost Referral Spam. Here is a Screenshot of a test Inactive Google Analytics account with 3 trackings ID. As you can see the only one that got hit is UA-XXXXXXXX-1, the other 2 as expected, are untouched.

 

This is another proof that Ghost Referrer Spam can’t be stopped by .htaccess rules since this site is not active the “visits” didn’t visit anything.

To change your tracking-ID

1 First go to the admin tab in GA.

2 Click the drop-down below the PROPERTY column and select Create new property.

3 Choose a new Name and enter your website address

4 Press Get tracking ID button

5 You will get a new tracking ID, add it to your website.

---

Conclusion

The Spam affects many of us to a greater or lesser extent and it shouldn’t be taken lightly.

Even on high volume websites were data spamming would be marginal, you still have to explain why there’s such a discrepancy. As an analyst you can’t dismiss it simply by saying “nah… we’re not too sure what it is, but I heard about that spamming thing…”

–Stéphane Hamel

It should be clear by now that you can’t block Ghosts Spam (the vast majority) with .htaccess rules or any server side solution.

You can stop the spam in google analytics  with a combination of 2 filters, the Valid hostname filter that will take care or all ghost spam and a Campaign Source filter containing the current Crawler Spam.

What to do after you stop the spam?

Filters will work from the moment you apply them. To remove the spam in google analytics and get clean reports from your historical data you can create a segment following this guide  Remove Referrer Spam from historical data with Segments

I hope the article helped you to keep your analytics data clean and accurate. Spam hits thousands of people, If the article was useful for you, please consider sharing it or leave a comment with your experience, it may help other people too 🙂

I’m updating the article with new information frequently, so keep an eye on it.

 

---

Active Spammers:

Resources

These are other great posts about Referrer Spam:

Thanks to Ben and Nick for the ideas and help to build this article.

All the Google Analytics screenshots are taken from my reports. I readjusted some of them by removing blank spaces between data, so it might not match exactly Google Analytics screens, but all the data is all real.

Updates

• July 7th – Remove the spam in Google Analytict with 2 filters
• May 5th – Spam hits with fake direct visits, valid hostname filter stops them
• April 10th – Verification of Changing the tracking Id solution
• March 29th – Types of Referrer Spam and how to identify it
• March 28th – Excluding good bots and spiders  > Method
• March 26th – Updated single filter method  > Method
• March 22th – Valid Hostname Filter > Method
• March 16th – How to detect Referrer Spam
• February 25th – Changing your tracking ID > Method