Magento 2.0 SEO Setup & Settings

The Guide

Introduction

Setting up Magento 1.9.x was relatively routine. Magento 2.0 keeps it simple too – its basic setup is as follows.

Note: Click on the icons to get extended information and the rationale behind each change or click here to show them all.

Canonical Catalogue Settings

Modern ecommerce platforms can create a significant quantity of duplicate content – and without proper configuration, Magento 2.0 is no exception.

Consequently, canonical URLs must be used to fix this.

Click For More Details About Ecommerce Canonical Issues
For example:

Category Duplication

  • http://www.example.com/chocolates
  • http://www.example.com/chocolates/
  • http://www.example.com/chocolates/?product_list_limit=30
  • http://www.example.com/chocolates/?product_list_order=name
  • http://www.example.com/chocolates?ingredients=5

Near duplicates of category pages are primarily created in Magento 2.0 by faceted navigation from attribute filters, trailing slashes and sorting options.

Product Duplication

For products, it’s slightly more simple.

  • http://www.example.com/chocolates/mars-bar-84g
  • http://www.example.com/chocolates/bars/mars-bar-84g/
  • http://www.example.com/mars-bar-84g
  • http://www.example.com/mars-bar-84g/
  • http://www.example.com/brands/mars/mars-bar-84g

The various categories products are associated with (and trailing slashes) can contribute to the same product appearing at different URLs.

Checking for this

A quick way to check for category or product duplication is to use Google itself. Using search operators such as site:, inurl: and intitle: together will usually find this, e.g. site:.co.uk inurl:air-wick intitle:”Air Wick Mrs Claus Apple Pie” – using titles from a random product, category or CMS page/blog post.

product-duplication

Note: You will not need to test every page like this, just 2 or 3 of each page type.

The Solution

This may be considered duplicate content. Adding canonical link for products and categories indicates to Google which of the multiple URLs that the categories and products can be accessed at is the correct URL. This can be resolved quite easily with some configuration settings (as follows).

Setting this up is nearly identical to Magento 1.9.x.

  • Search Engine Optimization settings for categories and products can be accessed via Stores > Settings / Configuration > Catalog > Catalog

magento2-canonical-settings

  • Actions;
    • Canonical links enabled on both products and categories.
    • Product URLs not using category path.
    • And Product URL / Category URL suffix set as / (unless something like .html is preferred).
      • If this is changed, this will not be updated unless the index is rebuilt (System > Tools / Index Management) and the cache is cleared (System > Tools / Cache Management).
      • A trailing slash is recommended because redirects can be implemented to force non-files to have a trailing slash with a small modification to the server. This is described below.
  • There are some small disadvantages to using some of these settings (specifically losing the category from the URL and how Magento handles breadcrumbs) but they are outweighed by the benefits.
Click For More Details About Forcing a Trailing Slash

If the product URL and Category URL Suffix is set as a trailing slash, a good modification is to force your site to redirect from example.com/category to example.com/category/. In Magento 1.9.x, this could be actioned as a setting in the Creare SEO extension but (at time of writing) this is not yet available for Magento 2.0.x, so a small modification will need to be made to your webserver.

Apache Forced Trailing Slash

This requires modification of the htaccess file. A solution from StackOverflow can be made to work for Magento 2.0.x.

RewriteCond %{REQUEST_URI} !(.*)/$ RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_URI} !^index.php RewriteCond %{REQUEST_URI} !\.(html|jpg|png|gif)$ RewriteRule ^(.*)$ $1/ [L,R=301]

Placed after any RewriteBase (if neccessary) but before the RewriteRule citing index.php.

nginx Forced Trailing Slash

An update to nginx configuration file will be needed.

location / { rewrite ^([^.]*[^/])$ $1/ permanent; }

Note: Any redirect changes should be thoroughly tested after implementation – including a check for retention of query string. Existing rules and correct order of rules can affect this.

Popular Search Terms

On the same page, it’s fine to allow popular search terms – but these pages will typically be blocked in the robots.txt file or noindexed so won’t serve a SEO benefit or handicap.

Canonical CMS Pages

Enabling canonical CMS pages needs a third party extension and is described in the next chapter.

XML Sitemap

XML sitemaps allow Google to easily identify and crawl the URLs on your site and they are still a core feature in Magento 2.0. The second step in a basic setup is enabling their generation.

Click To See What Has Changed
  • One notable improvement is the ability to split your sitemap into multiple files – sites with over 50,000 products are typically recommended to do this. A second minor improvement is the ability to add the URLs for the sitemaps to the robots.txt file (Search Engine Submission Settings) – a small quality of life improvement and often neglected by site owners.
  • There is still no out of the box support for Hreflang XML sitemaps for those running international/multi-language sites – a real shame considering how effective Magento is at implementing these via multi-store front installations. This is discussed in the next chapter.
  • HTML Sitemaps are no longer part of Magento Core but can easily be reimplemented via a template or third party extension
  • This can be accessed via Stores > Settings / Configuration > Catalog > XML Sitemap.
  • Enabling daily generation is best practice but this will vary – in practice this puts a small load on the server when there are thousands of products.

magento2-sitemap

  • Default sitemap settings for categories, products and CMS page options are fine – Google tends to determine crawl priority itself – any settings in the sitemap will only be used as a guide.

magento2-robots-sitemap

Note: Any sitemaps generated should be immediately submitted to Google Search Console to aid indexation of all your pages.

Robots.txt file

The robots.txt file is a “text file webmasters create to instruct […] search engine robots […] how to crawl and index pages on their website “.

As mentioned above, Magento 2.0 adds the ability to edit the robots.txt file from the admin backend (Stores > Configuration > Design / Search Engine Robots).

magento2-robots-txt

Click For More Details About Google and the robots.txt file
In October 2014, Google updated their guidelines to tell web developers to ensure that their robots.txt file doesn’t block images, fonts, Javascript, CSS or any other files that influence how the page looks when it’s rendered.

magento2-mobile-friendly

If you’d like to check whether your existing robots.txt file does this, there’s a robots.txt checker built into Google Search Console but a quick test is Google’s Mobile Friendly checker .

Since this change, configuration of robots.txt file became a lot less important on Magento sites – with the exception of blocking query strings and other junk URLs (add to wishlist, send to friend). This situation changes substantially when certain extensions are used – specifically those that create (so-called) “SEO pages” for attributes. When this happens, your robots.txt file becomes one of the main things stopping your site from having more pages than Amazon.

Robots.txt for Staging Sites

Our recommended setting is;

Disallow: /

This blocks every URL on the site. This is useful on staging sites but should never be used on live sites – although it’s a reasonable way of dealing with duplicate “Trade” sites.

Robots.txt for Live Sites

If you’re looking for an off-the-shelf robots.txt file, Creare’s Magento 1.x robot.txt file is pretty good but;

  • Lines 45-60 should be dealt with in other methods (noindexing, nofollowing) or just aren’t needed any more (although shouldn’t cause any harm).
  • Line 79 (URLs with query strings) will need to be reviewed. This is discussed below (“Extra information about robots.txts”).
Extra information about robots.txts

Blocking URLs with query strings from Google

On a Magento site, URLs with query strings are often duplicates of page categories or just non-indexable pages that do not need to be crawled. The above robots.txt file doesn’t block URLs with query strings (line 79) because it is “commented out” – there is a # in front of it. To enable this, line 79 must have the hash removed.

However whether or not you should depends on several factors.

  • If the site has already allowed URLs with parameters to be indexed previously, blocking them will prevent GoogleBot from recrawling them and these pages will stay indexed.
    • If these pages have noindex added to them (discussed next chapter), these pages will be eventually removed (typically 1-2 months later).
    • After this has happened, it’s perfectly fine to block URLs with query strings from Google.
    • However, if there are no plans to noindex URLs with query strings, line 79 can be uncommented (remove the #) to stop unnecessary pages from being crawled. This simply prevents new variations from being indexed, and prevents unnecessary pages from being crawled.
  • If the site is fresh, line 79 can be uncommented without issue.

Verifying Your Robots.txt

Our recommendation is to use a tool such as Screaming Frog to crawl your site to see which pages are being picked up that shouldn’t be – for example, hundreds of URLs containing something like “wishlist”.

These URLs can safely to added to the robots.txt file. Failing that, our Magento 2.0 Robots.txt fork is a good building block.

Note: If your site is setup on github or similar version control, it’s probably a bad idea to update the robots.txt file via the Magento backend since this can be overwritten.

Search Engine Robots

Whilst you’re on this screen (Stores > Settings / Configuration > General / Design), it’s also a good idea to check Search Engine Robots (to verify that your store front is set as INDEX, FOLLOW to ensure that your site isn’t blocking Google!

Click For More Details
Search Engine Robots relates to a tag that can be added to a page that prevents Google from indexing the site. This is useful for staging sites, less useful for sites that you want to be indexed. It’s good to check that this setting is correct.

Removing index.php

By default, Magento 2.0 and 1.9.x URLs look something like http://magento2.pushonltd.co.uk/index.php/viktor-lumatech-trade-pant.html – removing the “index.php” from the URL should also be on your checklist.

magento2-rewrite

  • Navigate to Stores > Settings / Configuration > General / Web > Search Engine Optimization
  • Enable Use Web Server Rewrites.

As long as the htaccess file is setup (Apache) and conf file (nginx), index.php will be stripped from the URLs. To check they’re in place, look for the following lines.

Apache

RewriteRule .* index.php [L]

nginx

location / {
try_files $uri $uri/ /index.php?$args;
}

Non-WWW/WWW Redirects

Whilst you’re on this screen, change the Auto-redirect to Base URL to 301 Redirect in Url Options.

Click For Information About Why Non-WWW/WWW Redirects are necessary
Your site usually exists at example.com and www.example.com. Usually the Base URL will be set to one or the other and will redirect to the preferred site.

By default, this is set to 302 redirect and historically this was a bad thing because 302 redirects were not believed to pass link equity/authority. This is considered outdated information but this should still be enabled.

HTTP Compression and Caching

Users demand fast loading sites – and so does Google – it’s believed to be beneficial to ensure that your website loads in under 3 seconds.

  • Gzip compression of static files (HTML, JS, CSS) reduces the download time for your website and can improve page load time.
  • Caching of static resources (template files, images, JS, CSS) ensures that the same files aren’t downloaded twice – reducing strain on your webserver and improving page load time for users.

You can’t set this up via the Magento backend, you’ll to update this via your web server settings.

Magento 2 Apache Settings

If your web server is running Apache, you’ll need to update your htaccess file.

Most of the Creare Magento 1.x htaccess file is fine to build from but we’ve forked the file for Magento 2.0. This is a partial file so will need integrating into your main htaccess file – individual sections can be copied and pasted into it.

Useful sections include;

  • Lines 28-40 and 70-92 setup gzip compression via mod_deflate.
    • This typically outperforms mod_gzip (lines 97-106).
    • Line 35 (php_flag zlib.output_compression on) can sometimes generate server errors if added to a server without gzip output enabled in PHP. If this occurs, this line can be safely removed – although this can be fixed by following the instructions on this blog post (Error 500) to enable it.
  • Lines 112-122 setup caching of certain filetypes via mod_expires – and does it slightly better than the default htaccess supplied with Magento 2.0.

The rest of the file is optional. For example – it implements ETags (lines 49-52) – a different and complimentary way to handle caching – but this can cause issues with some configurations.

Magento 2 nginx Settings

PushON’s recommendation for Magento hosting is nginx – it offers some improvements with respect to latency and dealing with concurrent connections. HTTP Compression and Caching can be setup for ninx too and this involves updating the conf file for that site. The one supplied by Magento works fine and has most things properly setup.

Testing Caching/Compression

Be sure to test your site using Google’s PageSpeed Insights Tool to ensure everything is working correctly. This also checks whether or not minify is working.

magento2-pagespeed

Note: You will not be able to enable caching or compression for third party scripts hosted off site – e.g. Google Analytics, Facebook share buttons.

Minify Javascript and CSS/HTML

Minify is a great way of improving pageload speeds simply by combining scripts.

Click For More Details
  • Modern browsers can download 6-8 files simultaneously from a server – including images, scripts and JS, a page may have between 120 and 200 files. This is why PushON recommend use of sub-domains or external providers for a CDN – using a media, theme and scripts sub-domain (e.g. media.ryman.co.uk, skin.ryman.co.uk, js.ryman.co.uk) will quadruple the amount of files that are downloaded simultaneously – increasing page load speed substantially. This topic is covered in more detail in the upcoming advanced configuration chapter.
  • Minify picks up on this by combining script/CSS files, reducing both the download time and the render time – each individual JS/CSS file loaded triggers or delays a new page render.
As mentioned in the previous chapter, this functionality has been improved in Magento 2.0. We now have three options plus bundling.

  • Merge JS and CSS files without minifying. Slightly more likely to cause issues. Reduces pageload times by reducing the amount of render passes and file downloads.
  • Minify JS and CSS files. Simply removes excess carriage returns and whitespaces from the files. Reduces the file size by a small amount (small relative to the total webpage size) but does not really have the same benefits as the above.
  • Both of the above.

Enabling Minification

magento2-minify

  • To set this this up, navigate to Stores > Settings / Configuration > Advanced / Developer and;
    • To enable HTML minification, enable Minify HTML in Template Settings.
    • To enable Javascript minification or merging, navigate to JavaScript Settings.
    • And for CSS merge and minify, navigate to CSS Settings

Warning: JavaScript Minify is also a great way to break a website. Minify should never be tested in a live environment and certainly not without testing all forms, add to basket and a complete transaction. It should also be tested using more than one browser.

Bundling

The last option is bundling. Another new feature in Magento 2.0 core is RequireJS which programmatically loads JavaScript files. Since Magento 2.0’s file merging and minification only applies to assets included via layouts, files loaded by RequireJS won’t won’t be included in this process. That’s where bundling comes in . It needs additional configuration which is outside the scope of the basic setup – but will result in improved page load and render times if setup.

Page Titles and Meta Description

Setting page titles in Magento 2.0 comes in 2 stages.

Individual pages can easily have their meta data updated by selecting the appropriate page for that store front. For example, to update the homepage meta description and title, you must edit those fields on the CMS page associated with the homepage.

magento2-homepage-title

But you also need to specify boilerplate brand title text.

magento2-metadata

  • Navigate to Stores > Settings / Configuration > General / Design > HTML Head
  • Title Suffix or Title Prefix allows you add the name of your store to every page.

Do you need to add your brand name to your website title? Yes. Google will mostly add it to the displayed title in the Search Engine results page anyway so you should add a version of it yourself. This at least gives you control over how it’s displayed.

Setting Product Default Templates

A good new feature in Magento 2.0 is being able to use product attributes in the meta data – in fact, a default template can easily be defined for products.

  • The default meta description is only used on products and only when a specific meta description is not set. It defaults to product name and product description.
  • Unfortunately there’s currently a bug in Magento 2.0 where any HTML tags from the description are pulled into this instead of being stripped. It’s relatively easy to fix at a developer level and we don’t anticipate this being an issue much longer.
  • For now it’s a good idea to override the default, navigate to Stores > Settings / Configuration > Catalog / Catalog > Product Field Auto-Generation.
  • Using a “mask for meta description” (default) such as “Buy {{name}} from Example.com. Available now for {{price}} with free delivery for orders over £20” will populate with the appropriate product name and price. Any defined attribute associated with that product can be used and will be automatically populated.
  • It’s not necessary to use the “meta keywords” tag but it can be helpful to populate this with non-visible attributes if you use Google Tag Manager, e.g. SKU, colour, manufacturer. This data can then be pulled by GTM into Google Analytics for custom dimensions. For example, on a children’s clothing website, age group – this would allow segments to be built to analyse conversion rate for all baby clothes vs infant clothing.

magento2-default-metas

Note: Meta descriptions aren’t believed to be a ranking factor for Google – they’re just persuasive text to encourage people to click through to your site from the search results.

Organization/Website Schema

Organization/LocalBusiness Schema is especially helpful for businesses with a physical location. Adding this markup to the site can help get data pulled into the Google Knowledge Graph in search.

google-knowledge-graph

WebSite Schema can also help Google identify and use your site’s internal search engine.

Click For More Details
Creating the snippets is relatively easy using a JSON-LD Structured Data Generator . An example snippet is below.

<script type="application/ld+json">
  [{
    "@context": "http://schema.org",
    "@type": "WebSite",
    "about": "Travelling Man",
    "url": "http://www.travellingman.co.uk",
    "potentialAction": {
      "@type": "SearchAction",
      "target": "http://www.travellingman.co.uk/catalogsearch/result?q={query}&x=0&y=0",
      "query-input": "required name=query"
    }
  }, {
    "@context": "http://schema.org",
    "@type": "Organization",
    "url": "http://www.travellingman.co.uk",
    "name": "Travelling Man",
    "logo": "http://www.travellingman.co.uk/skin/frontend/default/travellingman/images/logo.png",
    "subOrganization": [{
      "@context": "http://schema.org",
      "@type": "LocalBusiness",
      "address": {
        "@type": "PostalAddress",
        "postalCode": "M1 1JW",
        "streetAddress": "4 Dale Street",
        "addressLocality": "Manchester",
        "addressCountry": "GB"
      },
      "email": "manchester@travellingman.com",
      "name": "Travelling Man - Manchester",
      "parentOrganization": "Travelling Man",
      "telephone": "0161 237 1877",
      "url": "http://www.travellingman.co.uk/our-stores/manchester",
      "openingHours": ["Mo-Sa 10:00-18:00", "Su 11:00-16:30"]
    }, {
      "@context": "http://schema.org",
      "@type": "LocalBusiness",
      "address": {
        "@type": "PostalAddress",
        "streetAddress": "32 Central Road",
        "addressLocality": "Leeds",
        "postalCode": "LS1 6DE",
        "addressCountry": "GB"
      },
      "email": "info@travellingman.com",
      "name": "Travelling Man - Leeds",
      "parentOrganization": "Travelling Man",
      "telephone": "0113 243 6461",
      "url": "http://www.travellingman.co.uk/our-stores/leeds",
      "openingHours": ["Mo-Sa 10:00-18:00", "Su 11:00-17:00"]
    }, {
      "@context": "http://schema.org",
      "@type": "LocalBusiness",
      "address": {
        "@type": "PostalAddress",
        "addressLocality": "York",
        "postalCode": "YO1 7LF",
        "streetAddress": "54 Goodramgate",
        "addressCountry": "GB"
      },
      "email": "info@travellingman.com",
      "name": "Travelling Man - York",
      "parentOrganization": "Travelling Man",
      "telephone": "01904 628 787",
      "url": "http://www.travellingman.co.uk/our-stores/york",
      "openingHours": ["Mo-Sa 10:00-18:00", "Su 11:00-17:00"]
    }, {
      "@context": "http://schema.org",
      "@type": "LocalBusiness",
      "address": {
        "@type": "PostalAddress",
        "streetAddress": "43 Grainger Street",
        "addressLocality": "Newcastle-upon-Tyne",
        "postalCode": "NE1 5JE",
        "addressCountry": "GB"
      },
      "email": "info@travellingman.com",
      "name": "Travelling Man - Newcastle",
      "parentOrganization": "Travelling Man",
      "telephone": "0191 261 4993",
      "url": "http://www.travellingman.co.uk/our-stores/newcastle",
      "openingHours": ["Mo-Sa 10:00-18:00", "Su 11:00-16:30"]
    }],
    "sameAs": ["https://www.facebook.com/TravellingManShops", "https://twitter.com/TravellingManUK", "https://instagram.com/travellingmanuk/"]
   }] 
</script>

Defines a website with multiple locations, their opening times, phone numbers and the internal search engine used by the site.

It’s a good idea to test any code via Google’s Structured Data Testing Tool .

Adding Organization/WebSite Schema

The code for your website can easily be pasted into the HTML head of the website via Stores > Settings / Configuration > General / Design > HTML Head.

magento2-schema

Google Analytics

As a last step, Google Analytics will need to be setup. This is covered in greater detail in chapter 4.

Copyright 2015 PushON | All Rights Reserved | PushON Ltd