The Goal-Based Approach to Domain Selection – Whiteboard Friday

22 Jun

Posted by KameronJenkins

Choosing a domain is a big deal, and there’s a lot that goes into it. Even with everything that goes into determining your URL, there are two essential questions to ask that ought to guide your decision-making: what are my goals, and what’s best for my users? In today’s edition of Whiteboard Friday, we’re beyond delighted to welcome Kameron Jenkins, our SEO Wordsmith, to the show to teach us all about how to select a domain that aligns with and supports your business goals.

https://fast.wistia.net/embed/iframe/l3thq6mf6d?seo=false&videoFoam=true

https://fast.wistia.net/assets/external/E-v1.js

Goal-based Approach to Domain Selection

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hey, everyone. Welcome to this week’s edition of Whiteboard Friday. My name is Kameron Jenkins, and I am the SEO Wordsmith here at Moz. Today we’re going to be talking about a goals-based approach to choosing a domain type or a domain selection.

There are a lot of questions in the SEO industry right now, and as an agency, I used to work at an agency, and a lot of times our clients would ask us, “Should I do a microsite? Should I do a subdomain? Should I consolidate all my sites?” There is a lot of confusion about the SEO impact of all of these different types of domain choices, and there certainly are SEO ramifications for each type, but today we’re going to be taking a slightly different approach and focusing on goals first. What are your business goals? What are your goals for your website? What are your goals for your users? And then choosing a domain that matches those goals. By the end, instead of what’s better for SEO, we’re going to hopefully have answered, “What best suits my unique goals?”

Before we start…define!

Before we start, let’s launch into some quick definitions just so we all kind of know what we’re talking about and why all the different terminology we’re going to be using.

Main domain

Main domain, this is often called a root domain in some cases. That’s anything that precedes your dot com or other TLD. So YourSite.com, it lives right before that.

Subdomain

A subdomain is a third-level domain name for your domain. So example, Blog.YourSite.com, that would be a subdomain.

Subfolder

A subfolder, or some people call this subdirectory, those are folders trailing the dot com. An example would be YourSite.com/blog. That /blog is the folder. That’s a subfolder.

Microsite

A microsite, there’s a lot of different terminology around this type of domain selection, but it’s just a completely separate domain from your main domain. The focus is usually a little bit more niche than the topic of your main website.

That would be YourSite1.com and YourSite2.com. They’re two totally, completely separate domains.

Business goals that can impact domain structure

Next we’re going to start talking about business goals that can impact domain structure. There are a lot of different business goals. You want to grow revenue. You want more customers. But we’re specifically here going to be talking about the types of business goals that can impact domain selection.

1. Expand locations/products/services

The first one here that we’re going to talk about is the business wants to expand their locations, their products, or their services. They want to grow. They want to expand in some way. An example I like to use is say this clothing store has two locations. They have two storefronts. They have one in Dallas and one in Fort Worth.

So they launch two websites — CoolClothesDallas.com and CoolClothesFortWorth.com. But the problem with that is if you want to grow, you’re going to open stores in Austin, Houston, etc. You’ve set the precedent that you’re going to have a different domain for every single location, which is not really future-proof. It’s hard to scale. Every time you launch a brand-new website, that’s a lot of work to launch it, a lot of work to maintain it.

So if you plan on growing and getting into new locations or products or services or whatever it might be, just make sure you select a domain structure that’s going to accommodate that. In particular, I would say a main root domain with subfolders for the different products or services you have is probably the best bet for that situation. So you have YourSite.com/Product1, /Product2, and you talk about it in that sense because it’s all related. It’s all the same topic. It’s more future-proof. It’s easier to add a page than it is to launch a whole new domain.

2. Set apart distinct facets of business

So another business goal that can affect your domain structure would be that the business wants to set apart distinct facets within their business. An example I found that was actually kind of helpful is Apple.com has a subdomain for Trailers.Apple.com.

Now, I’m not Apple. I don’t really know exactly why they do this, but I have to imagine that it was because there are very different intents and uses for those different types of content that live on the subdomain versus the main site. So Trailers has movie trailers, lots of different content, and Apple.com is talking more about their consumer products, more about that type of thing.

So the audiences are slightly different. The intents are very different. In that situation, if you have a situation like that and that matches what your business is encountering, you want to set it apart, it has a different audience, you might want to consider a subdomain or maybe even a microsite. Just keep in mind that it takes effort to maintain each domain that you launch.

So make sure you have the resources to do this. You could, if you didn’t have the resources, put it all on the main domain. But if you want a little bit more separation, the different aspects of your business are very disparate and you don’t want them really associated on the same domain, you could separate it out with a subdomain or a microsite. Just, again, make sure that you have the resources to maintain it, because while both have equal ability to rank, it’s the effort that increases with each new website you launch.

3. Differentiate uniquely branded sub-departments

Three, another goal is to differentiate uniquely branded sub-departments. There is a lot of this I’ve noticed in the healthcare space. So the sites that I’ve worked on, say they have Joe Smith Health, and this is the health system, the umbrella health system. Then within that you have Joe Smith Endocrinology.

Usually those types of situations they have completely different branding. They’re in a different location. They reach a different audience, a different community. So in those situations I’ve seen that, especially healthcare, they usually have the resources to launch and maintain a completely different domain for that uniquely branded sub-department, and that might make sense.

Again, make sure you have the resources. But if it’s very, very different, whether in branding or audience or intent, than the content that’s on your main website, then I might consider separating them. Another example of this is sometimes you have a parent company and they own a lot of different companies, but that’s about where the similarities stop.

They’re just only owned by the parent company. All the different subcompanies don’t have anything to do with each other. I would probably say it’s wisest to separate those into their own unique domains. They probably definitely have unique branding. They’re totally different companies. They’re just owned by the same company. In those situations it might make sense, again, to separate them, but just know that they’re not going to have any ranking benefit for each other because they’re just completely separate domains.

4. Temporary or seasonal campaigns

The fourth business goal we’re going to talk about is a temporary or a seasonal campaign. This one is not as common, but I figured I would just mention it. Sometimes a business will want to run a conference or sponsor an event or get a lot of media attention around some initiative that’s separate from what their business does or offers, and it’s just more of an events-based, seasonal type of thing.

In those situations it might make sense to do a microsite that’s completely branded for that event. It’s not necessary. For example, Moz has MozCon, and that’s located on subfolder Moz.com/MozCon. You don’t have to do that, but it certainly is an option for you if you want to uniquely brand it.

It can also be really good for press. I’ve noticed just in my experience, I don’t know if this is widely common, but sometimes the press tends to just link to the homepage because that’s what they know. They don’t link to a specific page on your site. They don’t know always where it’s located. It’s just easier to link to the main domain. If you want to build links specifically for this event that are really relevant, you might want to do a microsite or something like that.

Just make sure that when the event is over, don’t just let it float out there and die. Especially if you build links and attention around it, make sure you 301 that back to your main website as long as that makes sense. So temporary or seasonal campaigns, that could be the way to go — microsite, subfolder. You have some options there.

5. Test out a new agency or consultant

Then finally the last goal we’re going to be talking about that could impact domain structure is testing out a new agency or consultant.

Now this one holds a special place in my heart having worked for an agency prior to this for almost seven years. It’s actually really common, and I can empathize with businesses who are in this situation. They are about to hand over their keys to their domain to a brand-new company. They don’t quite know if they trust them yet.

Especially this is concerning if a business has a really strong domain that they’ve built up over time. It can be really scary to just let someone take over your domain. In some cases I have encountered, the business goes, “Hey, we’d love to test you out. We think you’re great.However, you can’t touch the main domain.You have to do your SEO somewhere else.” That’s okay, but we’re kind of handcuffed in that situation.

You would have to, at that point, use a subdomain or a microsite, just a completely different website. If you can’t touch the main domain, you don’t really have many other options than that. You just have to launch on a brand-new thing. In that situation, it’s a little frustrating, actually quite frustrating for SEOs because they’re starting from nothing.

They have no authority inherited from that main domain. They’re starting from square one. They have to build that up over time. While that’s possible, just know that it kind of sets you back. You’re way behind the starting line in that situation with using a subdomain or a microsite, not being able to touch that main domain.

If you find yourself in this situation and you can negotiate this, just make sure that the company that’s hiring you is giving you enough time to prove the value of SEO. This is tried-and-true for a reason, but SEO is a marathon. It’s not a sprint. It’s not pay to play like paid advertising is. In that situation, just make sure that whoever is hiring you is giving you enough time.

Enough time is kind of dependent on how competitive the goals are. If they’re asking you, “Hey, I’m going to test you out for this really, really competitive, high-volume keyword or group of keywords and you only have one month to do it,” you’re kind of set up to fail in that situation. Either ask them for more time, or I probably wouldn’t take that job. So testing out a new agency or consultant is definitely something that can impact your ability to launch on one domain type versus another.

Pitfalls!

Now that we’ve talked about all of those, I’m just going to wrap up with some pitfalls. A lot of these are going to be repeat, but just as a way of review just watch out for these things.

Failing to future-proof

Like I said earlier, if you’re planning on growing in the future, just make sure that your domain matches your future plans.

Exact-match domains

There’s nothing inherently wrong with exact-match domains. It’s just that you can’t expect to launch a microsite with a bunch of keywords that are relevant to your business in your domain and just set it and forget it and hope that the keywords in the domain alone are what’s going to get it to rank. That doesn’t work anymore. It’s not worked for a while. You have to actually proactively be adding value to that microsite.

Maybe you’ve decided that that makes sense for your business. That’s great. Just make sure that you put in the resources to make it valuable outside of just the keywords in the domain.

Over-fragmenting

One thing I like to say is, “Would you rather have 3 websites with 10 backlinks each, or 1 website with 30 backlinks?” That’s just a way to illustrate that if you don’t have the resources to equally dedicate to each of those domains or subdomains or microsites or whatever you decided to launch, it’s not going to be as strong.

Usually what I see when I evaluate a customer or a client’s domain structure, usually there is one standout domain that has all of the content, all of the authority, all of the backlinks, and then the other ones just kind of suffer and they’re usually stronger together than they are apart. So while it is totally possible to do separate websites, just make sure that you don’t fragment so much that you’re spread too thin to actually do anything effective on the SEO front.

Ignoring user experience

Look at your websites from the eyes of your users. If someone is going to go to the search results page and Google search your business name, are they going to see five websites there? That’s kind of confusing unless they’re very differently branded, different intents. They’ll probably be confused.

Like, “Is this where I go to contact your business? How about this? Is it this?” There are just a lot of different ways that can cause confusion, so just keep that in mind. Also if you have a website where you’re addressing two completely different audiences within your website — if a consumer, for example, can be browsing blouses and then somehow end up accidentally on a section that’s only for employees — that’s a little confusing for user experience.

Make sure you either gate that or make it a subdomain or a microsite. Just separate them if that would be confusing for your main user base.

Set it and forget it

Like I said, I keep repeating this just because it’s so, so important. Every type of domain has equal ability to rank. It really does.

It’s just the effort that gets harder and harder with each new website. Just make sure that you don’t just decide to do microsites and subdomains and then don’t do anything with them. That can be a totally fine choice. Just make sure that you don’t set it and forget it, that you actually have the resources and you have the ability to keep building those up.

Intent overlap between domains

The last one I’ll talk about in the pitfall department is intent overlap between domains.

I see this one actually kind of a lot. It can be like a winery. So they have tastings.winery.com or something like that. In that situation, their Tasting subdomain talks all about their wine tasting, their tasting room. It’s very focused on that niche of their business. But then on Winery.com they also have extensive content about tastings. Well, you’ve got overlap there, and you’re kind of making yourself do more work than you have to.

I would choose one or the other and not both. Just make sure that there’s no overlap there if you do choose to do separate domains, subdomains, microsites, that kind of thing. Make sure that there’s no overlap and each of them has a distinct purpose.

Two important questions to focus on:

Now that we’re to the end of this, I really want the takeaway to be these two questions. I think this will make domain selection a lot easier when you focus on these two questions.

What am I trying to accomplish? What are the goals? What am I trying to do? Just focus on that first. Then second of all, and probably most important, what is best for my users? So focus on your goals, focus on your users, and I think the domain selection process will be a lot easier. It’s not easy by any means.

There are some very complicated situations, but I think, in the end, it’s going to be a lot easier if you focus on your goals and your users. If you have any comments regarding domain selection that you think would be helpful for others to know, please share it in the comments below. That’s it for this week’s Whiteboard Friday, and come back next week for another one. Thanks everybody.

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

from Blogger https://ift.tt/2K8ipKh

Advertisements

The Goal-Based Approach to Domain Selection – Whiteboard Friday

22 Jun

Posted by KameronJenkins

Choosing a domain is a big deal, and there’s a lot that goes into it. Even with everything that goes into determining your URL, there are two essential questions to ask that ought to guide your decision-making: what are my goals, and what’s best for my users? In today’s edition of Whiteboard Friday, we’re beyond delighted to welcome Kameron Jenkins, our SEO Wordsmith, to the show to teach us all about how to select a domain that aligns with and supports your business goals.

https://fast.wistia.net/embed/iframe/l3thq6mf6d?seo=false&videoFoam=true

https://fast.wistia.net/assets/external/E-v1.js

Goal-based Approach to Domain Selection

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hey, everyone. Welcome to this week’s edition of Whiteboard Friday. My name is Kameron Jenkins, and I am the SEO Wordsmith here at Moz. Today we’re going to be talking about a goals-based approach to choosing a domain type or a domain selection.

There are a lot of questions in the SEO industry right now, and as an agency, I used to work at an agency, and a lot of times our clients would ask us, “Should I do a microsite? Should I do a subdomain? Should I consolidate all my sites?” There is a lot of confusion about the SEO impact of all of these different types of domain choices, and there certainly are SEO ramifications for each type, but today we’re going to be taking a slightly different approach and focusing on goals first. What are your business goals? What are your goals for your website? What are your goals for your users? And then choosing a domain that matches those goals. By the end, instead of what’s better for SEO, we’re going to hopefully have answered, “What best suits my unique goals?”

Before we start…define!

Before we start, let’s launch into some quick definitions just so we all kind of know what we’re talking about and why all the different terminology we’re going to be using.

Main domain

Main domain, this is often called a root domain in some cases. That’s anything that precedes your dot com or other TLD. So YourSite.com, it lives right before that.

Subdomain

A subdomain is a third-level domain name for your domain. So example, Blog.YourSite.com, that would be a subdomain.

Subfolder

A subfolder, or some people call this subdirectory, those are folders trailing the dot com. An example would be YourSite.com/blog. That /blog is the folder. That’s a subfolder.


Microsite

A microsite, there’s a lot of different terminology around this type of domain selection, but it’s just a completely separate domain from your main domain. The focus is usually a little bit more niche than the topic of your main website.

That would be YourSite1.com and YourSite2.com. They’re two totally, completely separate domains.

Business goals that can impact domain structure

Next we’re going to start talking about business goals that can impact domain structure. There are a lot of different business goals. You want to grow revenue. You want more customers. But we’re specifically here going to be talking about the types of business goals that can impact domain selection.

1. Expand locations/products/services

The first one here that we’re going to talk about is the business wants to expand their locations, their products, or their services. They want to grow. They want to expand in some way. An example I like to use is say this clothing store has two locations. They have two storefronts. They have one in Dallas and one in Fort Worth.

So they launch two websites — CoolClothesDallas.com and CoolClothesFortWorth.com. But the problem with that is if you want to grow, you’re going to open stores in Austin, Houston, etc. You’ve set the precedent that you’re going to have a different domain for every single location, which is not really future-proof. It’s hard to scale. Every time you launch a brand-new website, that’s a lot of work to launch it, a lot of work to maintain it.

So if you plan on growing and getting into new locations or products or services or whatever it might be, just make sure you select a domain structure that’s going to accommodate that. In particular, I would say a main root domain with subfolders for the different products or services you have is probably the best bet for that situation. So you have YourSite.com/Product1, /Product2, and you talk about it in that sense because it’s all related. It’s all the same topic. It’s more future-proof. It’s easier to add a page than it is to launch a whole new domain.

2. Set apart distinct facets of business

So another business goal that can affect your domain structure would be that the business wants to set apart distinct facets within their business. An example I found that was actually kind of helpful is Apple.com has a subdomain for Trailers.Apple.com.

Now, I’m not Apple. I don’t really know exactly why they do this, but I have to imagine that it was because there are very different intents and uses for those different types of content that live on the subdomain versus the main site. So Trailers has movie trailers, lots of different content, and Apple.com is talking more about their consumer products, more about that type of thing.

So the audiences are slightly different. The intents are very different. In that situation, if you have a situation like that and that matches what your business is encountering, you want to set it apart, it has a different audience, you might want to consider a subdomain or maybe even a microsite. Just keep in mind that it takes effort to maintain each domain that you launch.

So make sure you have the resources to do this. You could, if you didn’t have the resources, put it all on the main domain. But if you want a little bit more separation, the different aspects of your business are very disparate and you don’t want them really associated on the same domain, you could separate it out with a subdomain or a microsite. Just, again, make sure that you have the resources to maintain it, because while both have equal ability to rank, it’s the effort that increases with each new website you launch.

3. Differentiate uniquely branded sub-departments

Three, another goal is to differentiate uniquely branded sub-departments. There is a lot of this I’ve noticed in the healthcare space. So the sites that I’ve worked on, say they have Joe Smith Health, and this is the health system, the umbrella health system. Then within that you have Joe Smith Endocrinology.

Usually those types of situations they have completely different branding. They’re in a different location. They reach a different audience, a different community. So in those situations I’ve seen that, especially healthcare, they usually have the resources to launch and maintain a completely different domain for that uniquely branded sub-department, and that might make sense.

Again, make sure you have the resources. But if it’s very, very different, whether in branding or audience or intent, than the content that’s on your main website, then I might consider separating them. Another example of this is sometimes you have a parent company and they own a lot of different companies, but that’s about where the similarities stop.

They’re just only owned by the parent company. All the different subcompanies don’t have anything to do with each other. I would probably say it’s wisest to separate those into their own unique domains. They probably definitely have unique branding. They’re totally different companies. They’re just owned by the same company. In those situations it might make sense, again, to separate them, but just know that they’re not going to have any ranking benefit for each other because they’re just completely separate domains.

4. Temporary or seasonal campaigns

The fourth business goal we’re going to talk about is a temporary or a seasonal campaign. This one is not as common, but I figured I would just mention it. Sometimes a business will want to run a conference or sponsor an event or get a lot of media attention around some initiative that’s separate from what their business does or offers, and it’s just more of an events-based, seasonal type of thing.

In those situations it might make sense to do a microsite that’s completely branded for that event. It’s not necessary. For example, Moz has MozCon, and that’s located on subfolder Moz.com/MozCon. You don’t have to do that, but it certainly is an option for you if you want to uniquely brand it.

It can also be really good for press. I’ve noticed just in my experience, I don’t know if this is widely common, but sometimes the press tends to just link to the homepage because that’s what they know. They don’t link to a specific page on your site. They don’t know always where it’s located. It’s just easier to link to the main domain. If you want to build links specifically for this event that are really relevant, you might want to do a microsite or something like that.

Just make sure that when the event is over, don’t just let it float out there and die. Especially if you build links and attention around it, make sure you 301 that back to your main website as long as that makes sense. So temporary or seasonal campaigns, that could be the way to go — microsite, subfolder. You have some options there.

5. Test out a new agency or consultant

Then finally the last goal we’re going to be talking about that could impact domain structure is testing out a new agency or consultant.

Now this one holds a special place in my heart having worked for an agency prior to this for almost seven years. It’s actually really common, and I can empathize with businesses who are in this situation. They are about to hand over their keys to their domain to a brand-new company. They don’t quite know if they trust them yet.

Especially this is concerning if a business has a really strong domain that they’ve built up over time. It can be really scary to just let someone take over your domain. In some cases I have encountered, the business goes, “Hey, we’d love to test you out. We think you’re great.However, you can’t touch the main domain.You have to do your SEO somewhere else.” That’s okay, but we’re kind of handcuffed in that situation.

You would have to, at that point, use a subdomain or a microsite, just a completely different website. If you can’t touch the main domain, you don’t really have many other options than that. You just have to launch on a brand-new thing. In that situation, it’s a little frustrating, actually quite frustrating for SEOs because they’re starting from nothing.

They have no authority inherited from that main domain. They’re starting from square one. They have to build that up over time. While that’s possible, just know that it kind of sets you back. You’re way behind the starting line in that situation with using a subdomain or a microsite, not being able to touch that main domain.

If you find yourself in this situation and you can negotiate this, just make sure that the company that’s hiring you is giving you enough time to prove the value of SEO. This is tried-and-true for a reason, but SEO is a marathon. It’s not a sprint. It’s not pay to play like paid advertising is. In that situation, just make sure that whoever is hiring you is giving you enough time.

Enough time is kind of dependent on how competitive the goals are. If they’re asking you, “Hey, I’m going to test you out for this really, really competitive, high-volume keyword or group of keywords and you only have one month to do it,” you’re kind of set up to fail in that situation. Either ask them for more time, or I probably wouldn’t take that job. So testing out a new agency or consultant is definitely something that can impact your ability to launch on one domain type versus another.

Pitfalls!

Now that we’ve talked about all of those, I’m just going to wrap up with some pitfalls. A lot of these are going to be repeat, but just as a way of review just watch out for these things.

Failing to future-proof

Like I said earlier, if you’re planning on growing in the future, just make sure that your domain matches your future plans.

Exact-match domains

There’s nothing inherently wrong with exact-match domains. It’s just that you can’t expect to launch a microsite with a bunch of keywords that are relevant to your business in your domain and just set it and forget it and hope that the keywords in the domain alone are what’s going to get it to rank. That doesn’t work anymore. It’s not worked for a while. You have to actually proactively be adding value to that microsite.

Maybe you’ve decided that that makes sense for your business. That’s great. Just make sure that you put in the resources to make it valuable outside of just the keywords in the domain.

Over-fragmenting

One thing I like to say is, “Would you rather have 3 websites with 10 backlinks each, or 1 website with 30 backlinks?” That’s just a way to illustrate that if you don’t have the resources to equally dedicate to each of those domains or subdomains or microsites or whatever you decided to launch, it’s not going to be as strong.

Usually what I see when I evaluate a customer or a client’s domain structure, usually there is one standout domain that has all of the content, all of the authority, all of the backlinks, and then the other ones just kind of suffer and they’re usually stronger together than they are apart. So while it is totally possible to do separate websites, just make sure that you don’t fragment so much that you’re spread too thin to actually do anything effective on the SEO front.

Ignoring user experience

Look at your websites from the eyes of your users. If someone is going to go to the search results page and Google search your business name, are they going to see five websites there? That’s kind of confusing unless they’re very differently branded, different intents. They’ll probably be confused.

Like, “Is this where I go to contact your business? How about this? Is it this?” There are just a lot of different ways that can cause confusion, so just keep that in mind. Also if you have a website where you’re addressing two completely different audiences within your website — if a consumer, for example, can be browsing blouses and then somehow end up accidentally on a section that’s only for employees — that’s a little confusing for user experience.

Make sure you either gate that or make it a subdomain or a microsite. Just separate them if that would be confusing for your main user base.

Set it and forget it

Like I said, I keep repeating this just because it’s so, so important. Every type of domain has equal ability to rank. It really does.

It’s just the effort that gets harder and harder with each new website. Just make sure that you don’t just decide to do microsites and subdomains and then don’t do anything with them. That can be a totally fine choice. Just make sure that you don’t set it and forget it, that you actually have the resources and you have the ability to keep building those up.

Intent overlap between domains

The last one I’ll talk about in the pitfall department is intent overlap between domains.

I see this one actually kind of a lot. It can be like a winery. So they have tastings.winery.com or something like that. In that situation, their Tasting subdomain talks all about their wine tasting, their tasting room. It’s very focused on that niche of their business. But then on Winery.com they also have extensive content about tastings. Well, you’ve got overlap there, and you’re kind of making yourself do more work than you have to.

I would choose one or the other and not both. Just make sure that there’s no overlap there if you do choose to do separate domains, subdomains, microsites, that kind of thing. Make sure that there’s no overlap and each of them has a distinct purpose.

Two important questions to focus on:

Now that we’re to the end of this, I really want the takeaway to be these two questions. I think this will make domain selection a lot easier when you focus on these two questions.

What am I trying to accomplish? What are the goals? What am I trying to do? Just focus on that first. Then second of all, and probably most important, what is best for my users? So focus on your goals, focus on your users, and I think the domain selection process will be a lot easier. It’s not easy by any means.

There are some very complicated situations, but I think, in the end, it’s going to be a lot easier if you focus on your goals and your users. If you have any comments regarding domain selection that you think would be helpful for others to know, please share it in the comments below. That’s it for this week’s Whiteboard Friday, and come back next week for another one. Thanks everybody.

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Q&A: Lost Your Anonymous Google Reviews? The Scoop on Removal and Moving Forward

21 Jun

Posted by MiriamEllis

Did you recently notice a minor or major drop in your Google review count, and then realize that some of your actual reviews had gone missing, too? Read on to see if your experience of removal review was part of the action Google took in late May surrounding anonymous reviews.

Q: What happened?

A: As nearly as I can pinpoint it, Google began discounting reviews left by “A Google User” from total review counts around May 23, 2018. For a brief period, these anonymous reviews were still visible, but were then removed from display. I haven’t seen any official announcement about this, to date, and it remains unclear as to whether all reviews designated as being from “A Google User” have been removed, or whether some still remain. I haven’t been able to discover a single one since the update.

Q: How do I know if I was affected by this action?

A: If, prior to my estimated date, you had reviews that had been left by profiles marked “A Google User,” and these reviews are now gone, that’s the diagnostic of why your total review count has dropped.

Q: The reviews I’ve lost weren’t from “A Google User” profiles. What happened?

A: If you’ve lost reviews from non-anonymous profiles, it’s time to investigate other causes of removal. These could include:

  • Having paid for or incentivized reviews, either directly or via an unethical marketer
  • Reviews stemming from a review station/kiosk at your business
  • Getting too many reviews at once
  • URLs, prohibited language, or other objectionable content in the body of reviews
  • Reviewing yourself, or having employees (past or present) do so
  • Reviews were left on your same IP (as in the case of free on-site Wi-Fi)
  • The use of review strategies/software that prohibit negative reviews or selectively solicit positive reviews
  • Any other violation of Google’s review guidelines
  • A Google bug, in which case, check the GMB forum for reports of similar review loss, and wait a few days to see if your reviews return; if not, you can take the time to post about your issue in the GMB forum, but chances are not good that removed reviews will be reinstated

Q: Is anonymous review removal a bug or a test?

A: One month later, these reviews remain absent. This is not a bug, and seems unlikely to be a test.

Q: Could my missing anonymous reviews come back?

A: Never say “never” with Google. From their inception, Google review counts have been wonky, and have been afflicted by various bugs. There have been cases in which reviews have vanished and reappeared. But, in this case, I don’t believe these types of reviews will return. This is most likely an action on Google’s part with the intention of improving their review corpus, which is, unfortunately, plagued with spam.

Q: What were the origins of “A Google User” reviews?

A: Reviews designated by this language came from a variety of scenarios, but are chiefly fallout from Google’s rollout of Google+ and then its subsequent detachment from local. As Mike Blumenthal explains:

As recently as 2016, Google required users to log in as G+ users to leave a review. When they transitioned away from + they allowed users several choices as to whether to delete their reviews or to create a name. Many users did not make that transition. For the users that chose not to give their name and make that transition Google identified them as ” A Google User”…. also certain devices like the old Blackberry’s could leave a review but not a name. Also users left + and may have changed profiles at Google abandoning their old profiles. Needless to say there were many ways that these reviews became from “A Google User.”

Q: Is the removal of anonymous reviews a positive or negative thing? What’s Google trying to do here?

A: Whether this action has worked out well or poorly for you likely depends on the quality of the reviews you’ve lost. In some cases, the loss may have suddenly put you behind competitors, in terms of review count or rating. In others, the loss of anonymous negative reviews may have just resulted in your star rating improving — which would be great news!

As to Google’s intent with this action, my assumption is that it’s a step toward increasing transparency. Not their own transparency, but the accountability of the reviewing public. Google doesn’t really like to acknowledge it, but their review corpus is inundated with spam, some of it the product of global networks of bad actors who have made a business of leaving fake reviews. Personally, I welcome Google making any attempts to cope with this, but the removal of this specific type of anonymous review is definitely not an adequate solution to review spam when the livelihoods of real people are on the line.

Q: Does this Google update mean my business is now safe from anonymous reviews?

A: Unfortunately, no. While it does mean you’re unlikely to see reviews marked as being from “A Google User”, it does not in any way deter people from creating as many Google identities as they’d like to review your business. Consider:

  • Google’s review product has yet to reach a level of sophistication which could automatically flag reviews left by “Rocky Balboa” or “Whatever Whatever” as, perhaps, somewhat lacking in legitimacy.
  • Google’s product also doesn’t appear to suspect profiles created solely to leave one-time reviews, though this is a clear hallmark of many instances of spam
  • Google won’t remove text-less negative star ratings, despite owner requests
  • Google hasn’t been historically swayed to remove reviews on the basis of the owner claiming no records show that a negative reviewer was ever a customer

Q: Should Google’s removal of anonymous reviews alter my review strategy?

A: No, not really. I empathize with the business owners expressing frustration over the loss of reviews they were proud of and had worked hard to earn. I see actions like this as important signals to all local businesses to remember that you don’t own your Google reviews, you don’t own your Google My Business listing/Knowledge Panel. Google owns those assets, and manages them in any way they deem best for Google.

In the local SEO industry, we are increasingly seeing the transformation of businesses from the status of empowered “website owner” to the shakier “Google tenant,” with more and more consumer actions taking place within Google’s interface. The May removal of reviews should be one more nudge to your local brand to:

  • Be sure you have an ongoing, guideline-compliant Google review acquisition campaign in place so that reviews that become filtered out can be replaced with fresh reviews
  • Take an active approach to monitoring your GMB reviews so that you become aware of changes quickly. Software like Moz Local can help with this, especially if you own or market large, multi-location enterprises. Even when no action can be taken in response to a new Google policy, awareness is always a competitive advantage.
  • Diversify your presence on review platforms beyond Google
  • Collect reviews and testimonials directly from your customers to be placed on your own website; don’t forget the Schema markup while you’re at it
  • Diversify the ways in which you are cultivating positive consumer sentiment offline; word-of-mouth marketing, loyalty programs, and the development of real-world relationships with your customers is something you directly control
  • Keep collecting those email addresses and, following the laws of your country, cultivate non-Google-dependent lines of communication with your customers
  • Invest heavily in hiring and training practices that empower staff to offer the finest possible experience to customers at the time of service — this is the very best way to ensure you are building a strong reputation both on and offline

Q: So, what should Google do next about review spam?

A: A Google rep once famously stated,

The wiki nature of Google Maps expands upon Google’s steadfast commitment to open community.”

I’d welcome your opinions as to how Google should deal with review spam, as I find this a very hard question to answer. It may well be a case of trying to lock the barn door after the horse has bolted, and Google’s wiki mentality applied to real-world businesses is one with which our industry has contended for years.

You see, the trouble with Google’s local product is that it was never opt-in. Whether you list your business or not, it can end up in Google’s local business index, and that means you are open to reviews (positive, negative, and fallacious) on the most visible possible platform, like it or not. As I’m not seeing a way to walk this back, review spam should be Google’s problem to fix, and they are obliged to fix it if:

  • They are committed to their own earnings, based on the trust the public feels in their review corpus
  • They are committed to user experience, implementing necessary technology and human intervention to protect consumers from fake reviews
  • They want to stop treating the very businesses on whom their whole product is structured as unimportant in the scheme of things; companies going out of business due to review spam attacks really shouldn’t be viewed as acceptable collateral damage

Knowing that Alphabet has an estimated operating income of $7 billion for 2018, I believe Google could fund these safeguards:

  1. Take a bold step and resource human review mediators. Make this a new department within the local department. Google sends out lots of emails to businesses now. Let them all include clear contact options for reaching the review mediation department if the business experiences spam reviews. Put the department behind a wizard that walks the business owner through guidelines to determine if a review is truly spam, and if this process signals a “yes,” open a ticket and fix the issue. Don’t depend on volunteers in the GMB forum. Invest money in paid staff to maintain the quality of Google’s own product.
  2. If Google is committed to the review flagging process (which is iffy, at best), offer every business owner clear guidelines for flagging reviews within their own GMB dashboard, and then communicate about what is happening to the flagged reviews.
  3. Improve algorithmic detection of suspicious signals, like profiles with one-off reviews, the sudden influx of negative reviews and text-less ratings, global reviews within a single profile, and companies or profiles with a history of guideline violations. Hold the first few reviews left by any profile in a “sandbox,” à la Yelp.

Now it’s your turn! Let’s look at Google’s removal of “A Google User” reviews as a first step in the right direction. If you had Google’s ear, what would you suggest they do next to combat review spam? I’d really like to know.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Moz’s Mid-Year Retrospective: Exciting Upgrades from the First Half of 2018

19 Jun

Posted by NeilCrist

Every year, we publish an overview of all the upgrades we’ve made to our tools and how those changes benefit our customers and Moz Community members. So far, 2018 has been a whirlwind of activity here at Moz — not only did we release a massive, long-awaited update to our link building tool, we’ve also been improving and updating systems and tools across the board to make your Moz experience even better. To that end, we’re sharing a mid-year retrospective to keep up with the incredible amount of progress we’ve made.

We receive a lot of amazing feedback from our customers on pain points they experience and improvements they’d like to see. Folks, we hear you.

We not only massively restructured some of our internal systems to provide you with better data, we also innovated new ways to display and report on that data, making the tools more accurate and more useful than ever before.

If you’ve been tasked with achieving organic success, we know your job isn’t easy. You need tools that get the job done, and done well. We think Moz delivered.

Check out our 2018 improvements so far:

Our new link index: Bigger, fresher, better than ever

Our link index underwent a major overhaul: it’s now 20x larger and 30x fresher than it was previously. This new link index data has been made available via our Mozscape API, as well as integrated into many Moz Pro tools, including Campaigns, Keyword Explorer, the MozBar, and Fresh Web Explorer. But undoubtedly the largest and most-anticipated improvement the new link index allowed us to make was the launch of Link Explorer, which we rolled out at the end of April as a replacement for Open Site Explorer.

Link Explorer addresses and improves upon its predecessor by providing more data, fresher data, and better ways to visualize that data. Answering a long-asked-for feature in OSE, Link Explorer includes historical metrics, and it also surfaces newly discovered and lost links:

Below are just a few of the many ways Link Explorer is providing some of the best link data available:

  • Link Explorer’s link index contains approximately 4.8 trillion URLs — that’s 20x larger than OSE and surpasses Ahrefs’ index (~3 trillion pages) and Majestic’s fresh index (~1 trillion pages).
  • Link Explorer is 30x fresher than OSE. All data updates every 24 hours.
  • We believe Link Explorer is unique in how accurately our link index represents the web, resulting in data quality you can trust.
  • Link Explorer has the closest robots.txt profile to Google among the three major link indexes, which means we get more of the links Google gets.
  • We also improved Domain Authority, Page Authority, and Spam Score. The size and freshness of our index has allowed us to offer a more stable DA and PA score. Though it will still fluctuate as the index fluctuates (which has always been by design), it will not be as dramatic as it was in Open Site Explorer.

Explore your link profile

You can learn more about Link Explorer by reading Sarah Bird’s announcement, watching Rand’s Whiteboard Friday, or visiting our Link Explorer Help Guide. Even though it’s still in beta, Link Explorer already blows away OSE’s data quality, freshness, and capabilities. Look for steady improvements to Link Explorer as we continue to iterate on it and add more key features.

New-and-improved On-Page Grader

Moz’s On-Page Grader got a thorough and much-needed overhaul! Not only did we freshen up the interface with a new look and feel, but we also added new features and improved upon our data.

Inside the new On-Page Grader, you’ll find:

  • An updated metrics bar to show you Page Title, Meta Description, and the number of Keywords Found. No need to dig through source code!
  • An updated Optimization Score to align with the Page Optimization feature that’s inside Campaigns and in the MozBar. Instead of a letter grade (A–F), you now have Page Score (0–100) for a more precise measurement of page optimization performance.
  • On-page factors are now categorized so you can see what is hurting or helping your Page Score.
  • On-page factors are organized by importance so you can prioritize your efforts. Red indicates high importance, yellow indicates moderate importance, and blue indicates low importance.

On-Page Grader is a great way to take a quick look at how well a page is optimized for a specified keyword. Here’s how it works.

Input your page and the keyword you want that page to rank for…

… and On-Page Grader will return a list of suggestions for improving your on-site optimization.

Check it out!

Keyword ranking data now available for Canada, UK, and Australia

We’re very excited to announce that, as of just last week, international data has been added to the Keywords by Site feature of Keyword Explorer! This will now allow Moz Pro customers to see which keywords they rank for and assess their visibility across millions of SERPs, now encompassing the US, Canada, the United Kingdom, and Australia! Keywords by Site is a newer feature within Keyword Explorer, added just last October to show which and how many keywords any domain, subdomain, or page ranks for.

Want to see which keywords your site ranks for in the US, UK, Canada, or Australia?

See what you rank for

It’s easy to use — just select a country from the dropdown menu to the right. This will show you which keywords a domain or page is ranking for from a particular country.

On-Demand Crawl now available

We know it can be important to track your site changes in real time. That’s why, on June 29th, we’re replacing our legacy site audit tool, Crawl Test, with the new and improved On-Demand Crawl:

Whether you need to double-check a change you’ve made or need a one-off report, the new On-Demand Crawl offers an updated experience for Moz Pro customers:

  • Crawl reports are now faster and available sooner, allowing you to quickly assess your site, a new client or prospect’s, or the competition.
  • Your site issues are now categorized by issue type and quantity, making it easier to identify what to work on and how to prioritize:

  • Recommendations are now provided for how to fix each issue, along with resources detailing why it matters:

  • Site audit reports are now easier than ever to package and present with PDF exports.
  • An updated, fresh design and UX!

On-Demand Crawl is already available now in Moz Pro. If you’re curious how it works, check it out:

Try On-Demand Crawl

Improvements to tool notifications & visuals

Moz’s email notification system and tools dashboard didn’t always sync up perfectly with the actual data update times. Sometimes, customers would receive an email or see updated dates on their dashboard before the data had rolled out, resulting in confusion. We’ve streamlined the process, and now customers no longer have to wonder where their data is — you can rest assured that your data is waiting for you in Moz Pro as soon as you’re notified.

Rank Tracker is sticking around

While we had originally planned to retire Rank Tracker at the beginning of June, we’ve decided to hold off in light of the feedback we received from our customers. Our goal in retiring Rank Tracker was to make Moz Pro easier to navigate by eliminating the redundancy of having two options for tracking keyword rankings (Rank Tracker and Campaigns), but after hearing how many people use and value Rank Tracker, and after weighing our options, we decided to postpone its retirement until we had a better solution than simply shutting it down.

Right now, we’re focused on learning more from our community on what makes this tool so valuable, so if you have feedback regarding Rank Tracker, we’d love it if you would take our survey. The information we gather from this survey will help us create a better solution for you!

Updates from Moz Academy

New advanced SEO courses

In response to the growing interest in advanced and niche-specific training, Moz is now offering ongoing classes and seminars on topics such as e-commerce SEO and technical site audits. If there’s an advanced topic you’d like training on, let us know by visiting https://moz.com/training and navigating to the “Custom” tab to tell us exactly what type of training you’re looking for.

On-demand coursework

We love the fact that we have Moz customers from around the globe, so we’re always looking for new ways to accommodate those in different timezones and those with sporadic schedules. One new way we’re doing this is by offering on-demand coursework. Get training from Moz when it works best for you. With this added scheduling flexibility (and with added instructors to boot), we hope to be able to reach more people than ever before.

To view Moz’s on-demand coursework, visit moz.com/training and click on the “On-Demand” tab.

Certificate development

There’s been a growing demand for a meaningful certification program in SEO, and we’re proud to say that Moz is here to deliver. This coursework will include a certificate and a badge for your LinkedIn profile. We’re planning on launching the program later this year, so stay tuned by signing up for Moz Training Alerts!

Tell us what you think!

Have feedback for us on any of our 2018 improvements? Any ideas on new ways we can improve our tools and training resources? Let us know in the comments! We love hearing from marketers like you. Your input helps us develop the best tools possible for ensuring your content gets found online.

If you’re not a Moz Pro subscriber and haven’t gotten a chance to check out these new features yet, sign up for a free trial!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

An 8-Point Checklist for Debugging Strange Technical SEO Problems

19 Jun

Posted by Dom-Woodman

Occasionally, a problem will land on your desk that’s a little out of the ordinary. Something where you don’t have an easy answer. You go to your brain and your brain returns nothing.

These problems can’t be solved with a little bit of keyword research and basic technical configuration. These are the types of technical SEO problems where the rabbit hole goes deep.

The very nature of these situations defies a checklist, but it’s useful to have one for the same reason we have them on planes: even the best of us can and will forget things, and a checklist will provvide you with places to dig.


Fancy some examples of strange SEO problems? Here are four examples to mull over while you read. We’ll answer them at the end.

1. Why wasn’t Google showing 5-star markup on product pages?

  • The pages had server-rendered product markup and they also had Feefo product markup, including ratings being attached client-side.
  • The Feefo ratings snippet was successfully rendered in Fetch & Render, plus the mobile-friendly tool.
  • When you put the rendered DOM into the structured data testing tool, both pieces of structured data appeared without errors.

2. Why wouldn’t Bing display 5-star markup on review pages, when Google would?

  • The review pages of client & competitors all had rating rich snippets on Google.
  • All the competitors had rating rich snippets on Bing; however, the client did not.
  • The review pages had correctly validating ratings schema on Google’s structured data testing tool, but did not on Bing.

3. Why were pages getting indexed with a no-index tag?

  • Pages with a server-side-rendered no-index tag in the head were being indexed by Google across a large template for a client.

4. Why did any page on a website return a 302 about 20–50% of the time, but only for crawlers?

  • A website was randomly throwing 302 errors.
  • This never happened in the browser and only in crawlers.
  • User agent made no difference; location or cookies also made no difference.

Finally, a quick note. It’s entirely possible that some of this checklist won’t apply to every scenario. That’s totally fine. It’s meant to be a process for everything you could check, not everything you should check.

The pre-checklist check

Does it actually matter?

Does this problem only affect a tiny amount of traffic? Is it only on a handful of pages and you already have a big list of other actions that will help the website? You probably need to just drop it.

I know, I hate it too. I also want to be right and dig these things out. But in six months’ time, when you’ve solved twenty complex SEO rabbit holes and your website has stayed flat because you didn’t re-write the title tags, you’re still going to get fired.

But hopefully that’s not the case, in which case, onwards!

Where are you seeing the problem?

We don’t want to waste a lot of time. Have you heard this wonderful saying?: “If you hear hooves, it’s probably not a zebra.”

The process we’re about to go through is fairly involved and it’s entirely up to your discretion if you want to go ahead. Just make sure you’re not overlooking something obvious that would solve your problem. Here are some common problems I’ve come across that were mostly horses.

  1. You’re underperforming from where you should be.
    1. When a site is under-performing, people love looking for excuses. Weird Google nonsense can be quite a handy thing to blame. In reality, it’s typically some combination of a poor site, higher competition, and a failing brand. Horse.
  2. You’ve suffered a sudden traffic drop.
    1. Something has certainly happened, but this is probably not the checklist for you. There are plenty of common-sense checklists for this. I’ve written about diagnosing traffic drops recently — check that out first.
  3. The wrong page is ranking for the wrong query.
    1. In my experience (which should probably preface this entire post), this is usually a basic problem where a site has poor targeting or a lot of cannibalization. Probably a horse.

Factors which make it more likely that you’ve got a more complex problem which require you to don your debugging shoes:

  • A website that has a lot of client-side JavaScript.
  • Bigger, older websites with more legacy.
  • Your problem is related to a new Google property or feature where there is less community knowledge.

1. Start by picking some example pages.

Pick a couple of example pages to work with — ones that exhibit whatever problem you’re seeing. No, this won’t be representative, but we’ll come back to that in a bit.

Of course, if it only affects a tiny number of pages then it might actually be representative, in which case we’re good. It definitely matters, right? You didn’t just skip the step above? OK, cool, let’s move on.

2. Can Google crawl the page once?

First we’re checking whether Googlebot has access to the page, which we’ll define as a 200 status code.

We’ll check in four different ways to expose any common issues:

  1. Robots.txt: Open up Search Console and check in the robots.txt validator.
  2. User agent: Open Dev Tools and verify that you can open the URL with both Googlebot and Googlebot Mobile.
    1. To get the user agent switcher, open Dev Tools.
    2. Check the console drawer is open (the toggle is the Escape key)
    3. Hit the … and open “Network conditions”
    4. Here, select your user agent!

  1. IP Address: Verify that you can access the page with the mobile testing tool. (This will come from one of the IPs used by Google; any checks you do from your computer won’t.)
  2. Country: The mobile testing tool will visit from US IPs, from what I’ve seen, so we get two birds with one stone. But Googlebot will occasionally crawl from non-American IPs, so it’s also worth using a VPN to double-check whether you can access the site from any other relevant countries.
    1. I’ve used HideMyAss for this before, but whatever VPN you have will work fine.

We should now have an idea whether or not Googlebot is struggling to fetch the page once.

Have we found any problems yet?

If we can re-create a failed crawl with a simple check above, then it’s likely Googlebot is probably failing consistently to fetch our page and it’s typically one of those basic reasons.

But it might not be. Many problems are inconsistent because of the nature of technology. 😉

3. Are we telling Google two different things?

Next up: Google can find the page, but are we confusing it by telling it two different things?

This is most commonly seen, in my experience, because someone has messed up the indexing directives.

By “indexing directives,” I’m referring to any tag that defines the correct index status or page in the index which should rank. Here’s a non-exhaustive list:

  • No-index
  • Canonical
  • Mobile alternate tags
  • AMP alternate tags

An example of providing mixed messages would be:

  • No-indexing page A
  • Page B canonicals to page A

Or:

  • Page A has a canonical in a header to A with a parameter
  • Page A has a canonical in the body to A without a parameter

If we’re providing mixed messages, then it’s not clear how Google will respond. It’s a great way to start seeing strange results.

Good places to check for the indexing directives listed above are:

  • Sitemap
    • Example: Mobile alternate tags can sit in a sitemap
  • HTTP headers
    • Example: Canonical and meta robots can be set in headers.
  • HTML head
    • This is where you’re probably looking, you’ll need this one for a comparison.
  • JavaScript-rendered vs hard-coded directives
    • You might be setting one thing in the page source and then rendering another with JavaScript, i.e. you would see something different in the HTML source from the rendered DOM.
  • Google Search Console settings
    • There are Search Console settings for ignoring parameters and country localization that can clash with indexing tags on the page.

A quick aside on rendered DOM

This page has a lot of mentions of the rendered DOM on it (18, if you’re curious). Since we’ve just had our first, here’s a quick recap about what that is.

When you load a webpage, the first request is the HTML. This is what you see in the HTML source (right-click on a webpage and click View Source).

This is before JavaScript has done anything to the page. This didn’t use to be such a big deal, but now so many websites rely heavily on JavaScript that the most people quite reasonably won’t trust the the initial HTML.

Rendered DOM is the technical term for a page, when all the JavaScript has been rendered and all the page alterations made. You can see this in Dev Tools.

In Chrome you can get that by right clicking and hitting inspect element (or Ctrl + Shift + I). The Elements tab will show the DOM as it’s being rendered. When it stops flickering and changing, then you’ve got the rendered DOM!

4. Can Google crawl the page consistently?

To see what Google is seeing, we’re going to need to get log files. At this point, we can check to see how it is accessing the page.

Aside: Working with logs is an entire post in and of itself. I’ve written a guide to log analysis with BigQuery, I’d also really recommend trying out Screaming Frog Log Analyzer, which has done a great job of handling a lot of the complexity around logs.

When we’re looking at crawling there are three useful checks we can do:

  1. Status codes: Plot the status codes over time. Is Google seeing different status codes than you when you check URLs?
  2. Resources: Is Google downloading all the resources of the page?
    1. Is it downloading all your site-specific JavaScript and CSS files that it would need to generate the page?
  3. Page size follow-up: Take the max and min of all your pages and resources and diff them. If you see a difference, then Google might be failing to fully download all the resources or pages. (Hat tip to @ohgm, where I first heard this neat tip).

Have we found any problems yet?

If Google isn’t getting 200s consistently in our log files, but we can access the page fine when we try, then there is clearly still some differences between Googlebot and ourselves. What might those differences be?

  1. It will crawl more than us
  2. It is obviously a bot, rather than a human pretending to be a bot
  3. It will crawl at different times of day

This means that:

  • If our website is doing clever bot blocking, it might be able to differentiate between us and Googlebot.
  • Because Googlebot will put more stress on our web servers, it might behave differently. When websites have a lot of bots or visitors visiting at once, they might take certain actions to help keep the website online. They might turn on more computers to power the website (this is called scaling), they might also attempt to rate-limit users who are requesting lots of pages, or serve reduced versions of pages.
  • Servers run tasks periodically; for example, a listings website might run a daily task at 01:00 to clean up all it’s old listings, which might affect server performance.

Working out what’s happening with these periodic effects is going to be fiddly; you’re probably going to need to talk to a back-end developer.

Depending on your skill level, you might not know exactly where to lead the discussion. A useful structure for a discussion is often to talk about how a request passes through your technology stack and then look at the edge cases we discussed above.

  • What happens to the servers under heavy load?
  • When do important scheduled tasks happen?

Two useful pieces of information to enter this conversation with:

  1. Depending on the regularity of the problem in the logs, it is often worth trying to re-create the problem by attempting to crawl the website with a crawler at the same speed/intensity that Google is using to see if you can find/cause the same issues. This won’t always be possible depending on the size of the site, but for some sites it will be. Being able to consistently re-create a problem is the best way to get it solved.
  2. If you can’t, however, then try to provide the exact periods of time where Googlebot was seeing the problems. This will give the developer the best chance of tying the issue to other logs to let them debug what was happening.

If Google can crawl the page consistently, then we move onto our next step.

5. Does Google see what I can see on a one-off basis?

We know Google is crawling the page correctly. The next step is to try and work out what Google is seeing on the page. If you’ve got a JavaScript-heavy website you’ve probably banged your head against this problem before, but even if you don’t this can still sometimes be an issue.

We follow the same pattern as before. First, we try to re-create it once. The following tools will let us do that:

  • Fetch & Render
    • Shows: Rendered DOM in an image, but only returns the page source HTML for you to read.
  • Mobile-friendly test
    • Shows: Rendered DOM and returns rendered DOM for you to read.
    • Not only does this show you rendered DOM, but it will also track any console errors.

Is there a difference between Fetch & Render, the mobile-friendly testing tool, and Googlebot? Not really, with the exception of timeouts (which is why we have our later steps!). Here’s the full analysis of the difference between them, if you’re interested.

Once we have the output from these, we compare them to what we ordinarily see in our browser. I’d recommend using a tool like Diff Checker to compare the two.

Have we found any problems yet?

If we encounter meaningful differences at this point, then in my experience it’s typically either from JavaScript or cookies

Why?

We can isolate each of these by:

  • Loading the page with no cookies. This can be done simply by loading the page with a fresh incognito session and comparing the rendered DOM here against the rendered DOM in our ordinary browser.
  • Use the mobile testing tool to see the page with Chrome 41 and compare against the rendered DOM we normally see with Inspect Element.

Yet again we can compare them using something like Diff Checker, which will allow us to spot any differences. You might want to use an HTML formatter to help line them up better.

We can also see the JavaScript errors thrown using the Mobile-Friendly Testing Tool, which may prove particularly useful if you’re confident in your JavaScript.

If, using this knowledge and these tools, we can recreate the bug, then we have something that can be replicated and it’s easier for us to hand off to a developer as a bug that will get fixed.

If we’re seeing everything is correct here, we move on to the next step.

6. What is Google actually seeing?

It’s possible that what Google is seeing is different from what we recreate using the tools in the previous step. Why? A couple main reasons:

  • Overloaded servers can have all sorts of strange behaviors. For example, they might be returning 200 codes, but perhaps with a default page.
  • JavaScript is rendered separately from pages being crawled and Googlebot may spend less time rendering JavaScript than a testing tool.
  • There is often a lot of caching in the creation of web pages and this can cause issues.

We’ve gotten this far without talking about time! Pages don’t get crawled instantly, and crawled pages don’t get indexed instantly.

Quick sidebar: What is caching?

Caching is often a problem if you get to this stage. Unlike JS, it’s not talked about as much in our community, so it’s worth some more explanation in case you’re not familiar. Caching is storing something so it’s available more quickly next time.

When you request a webpage, a lot of calculations happen to generate that page. If you then refreshed the page when it was done, it would be incredibly wasteful to just re-run all those same calculations. Instead, servers will often save the output and serve you the output without re-running them. Saving the output is called caching.

Why do we need to know this? Well, we’re already well out into the weeds at this point and so it’s possible that a cache is misconfigured and the wrong information is being returned to users.

There aren’t many good beginner resources on caching which go into more depth. However, I found this article on caching basics to be one of the more friendly ones. It covers some of the basic types of caching quite well.

How can we see what Google is actually working with?

  • Google’s cache
    • Shows: Source code
    • While this won’t show you the rendered DOM, it is showing you the raw HTML Googlebot actually saw when visiting the page. You’ll need to check this with JS disabled; otherwise, on opening it, your browser will run all the JS on the cached version.
  • Site searches for specific content
    • Shows: A tiny snippet of rendered content.
    • By searching for a specific phrase on a page, e.g. inurl:example.com/url “only JS rendered text”, you can see if Google has manage to index a specific snippet of content. Of course, it only works for visible text and misses a lot of the content, but it’s better than nothing!
    • Better yet, do the same thing with a rank tracker, to see if it changes over time.
  • Storing the actual rendered DOM
    • Shows: Rendered DOM
    • Alex from DeepCrawl has written about saving the rendered DOM from Googlebot. The TL;DR version: Google will render JS and post to endpoints, so we can get it to submit the JS-rendered version of a page that it sees. We can then save that, examine it, and see what went wrong.

Have we found any problems yet?

Again, once we’ve found the problem, it’s time to go and talk to a developer. The advice for this conversation is identical to the last one — everything I said there still applies.

The other knowledge you should go into this conversation armed with: how Google works and where it can struggle. While your developer will know the technical ins and outs of your website and how it’s built, they might not know much about how Google works. Together, this can help you reach the answer more quickly.

The obvious source for this are resources or presentations given by Google themselves. Of the various resources that have come out, I’ve found these two to be some of the more useful ones for giving insight into first principles:

But there is often a difference between statements Google will make and what the SEO community sees in practice. All the SEO experiments people tirelessly perform in our industry can also help shed some insight. There are far too many list here, but here are two good examples:

7. Could Google be aggregating your website across others?

If we’ve reached this point, we’re pretty happy that our website is running smoothly. But not all problems can be solved just on your website; sometimes you’ve got to look to the wider landscape and the SERPs around it.

Most commonly, what I’m looking for here is:

  • Similar/duplicate content to the pages that have the problem.
    • This could be intentional duplicate content (e.g. syndicating content) or unintentional (competitors’ scraping or accidentally indexed sites).

Either way, they’re nearly always found by doing exact searches in Google. I.e. taking a relatively specific piece of content from your page and searching for it in quotes.

Have you found any problems yet?

If you find a number of other exact copies, then it’s possible they might be causing issues.

The best description I’ve come up with for “have you found a problem here?” is: do you think Google is aggregating together similar pages and only showing one? And if it is, is it picking the wrong page?

This doesn’t just have to be on traditional Google search. You might find a version of it on Google Jobs, Google News, etc.

To give an example, if you are a reseller, you might find content isn’t ranking because there’s another, more authoritative reseller who consistently posts the same listings first.

Sometimes you’ll see this consistently and straightaway, while other times the aggregation might be changing over time. In that case, you’ll need a rank tracker for whatever Google property you’re working on to see it.

Jon Earnshaw from Pi Datametrics gave an excellent talk on the latter (around suspicious SERP flux) which is well worth watching.

Once you’ve found the problem, you’ll probably need to experiment to find out how to get around it, but the easiest factors to play with are usually:

  • De-duplication of content
  • Speed of discovery (you can often improve by putting up a 24-hour RSS feed of all the new content that appears)
  • Lowering syndication

8. A roundup of some other likely suspects

If you’ve gotten this far, then we’re sure that:

  • Google can consistently crawl our pages as intended.
  • We’re sending Google consistent signals about the status of our page.
  • Google is consistently rendering our pages as we expect.
  • Google is picking the correct page out of any duplicates that might exist on the web.

And your problem still isn’t solved?

And it is important?

Well, shoot.

Feel free to hire us…?

As much as I’d love for this article to list every SEO problem ever, that’s not really practical, so to finish off this article let’s go through two more common gotchas and principles that didn’t really fit in elsewhere before the answers to those four problems we listed at the beginning.

Invalid/poorly constructed HTML

You and Googlebot might be seeing the same HTML, but it might be invalid or wrong. Googlebot (and any crawler, for that matter) has to provide workarounds when the HTML specification isn’t followed, and those can sometimes cause strange behavior.

The easiest way to spot it is either by eye-balling the rendered DOM tools or using an HTML validator.

The W3C validator is very useful, but will throw up a lot of errors/warnings you won’t care about. The closest I can give to a one-line of summary of which ones are useful is to:

  • Look for errors
  • Ignore anything to do with attributes (won’t always apply, but is often true).

The classic example of this is breaking the head.

An iframe isn’t allowed in the head code, so Chrome will end the head and start the body. Unfortunately, it takes the title and canonical with it, because they fall after it — so Google can’t read them. The head code should have ended in a different place.

Oliver Mason wrote a good post that explains an even more subtle version of this in breaking the head quietly.

When in doubt, diff

Never underestimate the power of trying to compare two things line by line with a diff from something like Diff Checker. It won’t apply to everything, but when it does it’s powerful.

For example, if Google has suddenly stopped showing your featured markup, try to diff your page against a historical version either in your QA environment or from the Wayback Machine.


Answers to our original 4 questions

Time to answer those questions. These are all problems we’ve had clients bring to us at Distilled.

1. Why wasn’t Google showing 5-star markup on product pages?

Google was seeing both the server-rendered markup and the client-side-rendered markup; however, the server-rendered side was taking precedence.

Removing the server-rendered markup meant the 5-star markup began appearing.

2. Why wouldn’t Bing display 5-star markup on review pages, when Google would?

The problem came from the references to schema.org.

        
<p> <h1 itemprop="name">Avatar</h1> </p> <p> <span>Director: <span itemprop="director">James Cameron</span> (born August 16, 1954)</span> </p> <p> <span itemprop="genre">Science fiction</span> </p> <p> <a href="../movies/avatar-theatrical-trailer.html" itemprop="trailer">Trailer</a> </p> <p></div> </p>

We diffed our markup against our competitors and the only difference was we’d referenced the HTTPS version of schema.org in our itemtype, which caused Bing to not support it.

C’mon, Bing.

3. Why were pages getting indexed with a no-index tag?

The answer for this was in this post. This was a case of breaking the head.

The developers had installed some ad-tech in the head and inserted an non-standard tag, i.e. not:

  • <title>
  • <style>
  • <base>
  • <link>
  • <meta>
  • <script>
  • <noscript>

This caused the head to end prematurely and the no-index tag was left in the body where it wasn’t read.

4. Why did any page on a website return a 302 about 20–50% of the time, but only for crawlers?

This took some time to figure out. The client had an old legacy website that has two servers, one for the blog and one for the rest of the site. This issue started occurring shortly after a migration of the blog from a subdomain (blog.client.com) to a subdirectory (client.com/blog/…).

At surface level everything was fine; if a user requested any individual page, it all looked good. A crawl of all the blog URLs to check they’d redirected was fine.

But we noticed a sharp increase of errors being flagged in Search Console, and during a routine site-wide crawl, many pages that were fine when checked manually were causing redirect loops.

We checked using Fetch and Render, but once again, the pages were fine.

Eventually, it turned out that when a non-blog page was requested very quickly after a blog page (which, realistically, only a crawler is fast enough to achieve), the request for the non-blog page would be sent to the blog server.

These would then be caught by a long-forgotten redirect rule, which 302-redirected deleted blog posts (or other duff URLs) to the root. This, in turn, was caught by a blanket HTTP to HTTPS 301 redirect rule, which would be requested from the blog server again, perpetuating the loop.

For example, requesting https://www.client.com/blog/ followed quickly enough by https://www.client.com/category/ would result in:

  • 302 to http://www.client.com – This was the rule that redirected deleted blog posts to the root
  • 301 to https://www.client.com – This was the blanket HTTPS redirect
  • 302 to http://www.client.com – The blog server doesn’t know about the HTTPS non-blog homepage and it redirects back to the HTTP version. Rinse and repeat.

This caused the periodic 302 errors and it meant we could work with their devs to fix the problem.

What are the best brainteasers you’ve had?

Let’s hear them, people. What problems have you run into? Let us know in the comments.

Also credit to @RobinLord8, @TomAnthonySEO, @THCapper, @samnemzer, and @sergeystefoglo_ for help with this piece.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

from Blogger https://ift.tt/2I1fn5i

An 8-Point Checklist for Debugging Strange Technical SEO Problems

19 Jun

Posted by Dom-Woodman

Occasionally, a problem will land on your desk that’s a little out of the ordinary. Something where you don’t have an easy answer. You go to your brain and your brain returns nothing.

These problems can’t be solved with a little bit of keyword research and basic technical configuration. These are the types of technical SEO problems where the rabbit hole goes deep.

The very nature of these situations defies a checklist, but it’s useful to have one for the same reason we have them on planes: even the best of us can and will forget things, and a checklist will provvide you with places to dig.


Fancy some examples of strange SEO problems? Here are four examples to mull over while you read. We’ll answer them at the end.

1. Why wasn’t Google showing 5-star markup on product pages?

  • The pages had server-rendered product markup and they also had Feefo product markup, including ratings being attached client-side.
  • The Feefo ratings snippet was successfully rendered in Fetch & Render, plus the mobile-friendly tool.
  • When you put the rendered DOM into the structured data testing tool, both pieces of structured data appeared without errors.

2. Why wouldn’t Bing display 5-star markup on review pages, when Google would?

  • The review pages of client & competitors all had rating rich snippets on Google.
  • All the competitors had rating rich snippets on Bing; however, the client did not.
  • The review pages had correctly validating ratings schema on Google’s structured data testing tool, but did not on Bing.

3. Why were pages getting indexed with a no-index tag?

  • Pages with a server-side-rendered no-index tag in the head were being indexed by Google across a large template for a client.

4. Why did any page on a website return a 302 about 20–50% of the time, but only for crawlers?

  • A website was randomly throwing 302 errors.
  • This never happened in the browser and only in crawlers.
  • User agent made no difference; location or cookies also made no difference.

Finally, a quick note. It’s entirely possible that some of this checklist won’t apply to every scenario. That’s totally fine. It’s meant to be a process for everything you could check, not everything you should check.

The pre-checklist check

Does it actually matter?

Does this problem only affect a tiny amount of traffic? Is it only on a handful of pages and you already have a big list of other actions that will help the website? You probably need to just drop it.

I know, I hate it too. I also want to be right and dig these things out. But in six months’ time, when you’ve solved twenty complex SEO rabbit holes and your website has stayed flat because you didn’t re-write the title tags, you’re still going to get fired.

But hopefully that’s not the case, in which case, onwards!

Where are you seeing the problem?

We don’t want to waste a lot of time. Have you heard this wonderful saying?: “If you hear hooves, it’s probably not a zebra.”

The process we’re about to go through is fairly involved and it’s entirely up to your discretion if you want to go ahead. Just make sure you’re not overlooking something obvious that would solve your problem. Here are some common problems I’ve come across that were mostly horses.

  1. You’re underperforming from where you should be.
    1. When a site is under-performing, people love looking for excuses. Weird Google nonsense can be quite a handy thing to blame. In reality, it’s typically some combination of a poor site, higher competition, and a failing brand. Horse.
  2. You’ve suffered a sudden traffic drop.
    1. Something has certainly happened, but this is probably not the checklist for you. There are plenty of common-sense checklists for this. I’ve written about diagnosing traffic drops recently — check that out first.
  3. The wrong page is ranking for the wrong query.
    1. In my experience (which should probably preface this entire post), this is usually a basic problem where a site has poor targeting or a lot of cannibalization. Probably a horse.

Factors which make it more likely that you’ve got a more complex problem which require you to don your debugging shoes:

  • A website that has a lot of client-side JavaScript.
  • Bigger, older websites with more legacy.
  • Your problem is related to a new Google property or feature where there is less community knowledge.

1. Start by picking some example pages.

Pick a couple of example pages to work with — ones that exhibit whatever problem you’re seeing. No, this won’t be representative, but we’ll come back to that in a bit.

Of course, if it only affects a tiny number of pages then it might actually be representative, in which case we’re good. It definitely matters, right? You didn’t just skip the step above? OK, cool, let’s move on.

2. Can Google crawl the page once?

First we’re checking whether Googlebot has access to the page, which we’ll define as a 200 status code.

We’ll check in four different ways to expose any common issues:

  1. Robots.txt: Open up Search Console and check in the robots.txt validator.
  2. User agent: Open Dev Tools and verify that you can open the URL with both Googlebot and Googlebot Mobile.
    1. To get the user agent switcher, open Dev Tools.
    2. Check the console drawer is open (the toggle is the Escape key)
    3. Hit the … and open “Network conditions”
    4. Here, select your user agent!

  1. IP Address: Verify that you can access the page with the mobile testing tool. (This will come from one of the IPs used by Google; any checks you do from your computer won’t.)
  2. Country: The mobile testing tool will visit from US IPs, from what I’ve seen, so we get two birds with one stone. But Googlebot will occasionally crawl from non-American IPs, so it’s also worth using a VPN to double-check whether you can access the site from any other relevant countries.
    1. I’ve used HideMyAss for this before, but whatever VPN you have will work fine.

We should now have an idea whether or not Googlebot is struggling to fetch the page once.

Have we found any problems yet?

If we can re-create a failed crawl with a simple check above, then it’s likely Googlebot is probably failing consistently to fetch our page and it’s typically one of those basic reasons.

But it might not be. Many problems are inconsistent because of the nature of technology. 😉

3. Are we telling Google two different things?

Next up: Google can find the page, but are we confusing it by telling it two different things?

This is most commonly seen, in my experience, because someone has messed up the indexing directives.

By “indexing directives,” I’m referring to any tag that defines the correct index status or page in the index which should rank. Here’s a non-exhaustive list:

  • No-index
  • Canonical
  • Mobile alternate tags
  • AMP alternate tags

An example of providing mixed messages would be:

  • No-indexing page A
  • Page B canonicals to page A

Or:

  • Page A has a canonical in a header to A with a parameter
  • Page A has a canonical in the body to A without a parameter

If we’re providing mixed messages, then it’s not clear how Google will respond. It’s a great way to start seeing strange results.

Good places to check for the indexing directives listed above are:

  • Sitemap
    • Example: Mobile alternate tags can sit in a sitemap
  • HTTP headers
    • Example: Canonical and meta robots can be set in headers.
  • HTML head
    • This is where you’re probably looking, you’ll need this one for a comparison.
  • JavaScript-rendered vs hard-coded directives
    • You might be setting one thing in the page source and then rendering another with JavaScript, i.e. you would see something different in the HTML source from the rendered DOM.
  • Google Search Console settings
    • There are Search Console settings for ignoring parameters and country localization that can clash with indexing tags on the page.

A quick aside on rendered DOM

This page has a lot of mentions of the rendered DOM on it (18, if you’re curious). Since we’ve just had our first, here’s a quick recap about what that is.

When you load a webpage, the first request is the HTML. This is what you see in the HTML source (right-click on a webpage and click View Source).

This is before JavaScript has done anything to the page. This didn’t use to be such a big deal, but now so many websites rely heavily on JavaScript that the most people quite reasonably won’t trust the the initial HTML.

Rendered DOM is the technical term for a page, when all the JavaScript has been rendered and all the page alterations made. You can see this in Dev Tools.

In Chrome you can get that by right clicking and hitting inspect element (or Ctrl + Shift + I). The Elements tab will show the DOM as it’s being rendered. When it stops flickering and changing, then you’ve got the rendered DOM!

4. Can Google crawl the page consistently?

To see what Google is seeing, we’re going to need to get log files. At this point, we can check to see how it is accessing the page.

Aside: Working with logs is an entire post in and of itself. I’ve written a guide to log analysis with BigQuery, I’d also really recommend trying out Screaming Frog Log Analyzer, which has done a great job of handling a lot of the complexity around logs.

When we’re looking at crawling there are three useful checks we can do:

  1. Status codes: Plot the status codes over time. Is Google seeing different status codes than you when you check URLs?
  2. Resources: Is Google downloading all the resources of the page?
    1. Is it downloading all your site-specific JavaScript and CSS files that it would need to generate the page?
  3. Page size follow-up: Take the max and min of all your pages and resources and diff them. If you see a difference, then Google might be failing to fully download all the resources or pages. (Hat tip to @ohgm, where I first heard this neat tip).

Have we found any problems yet?

If Google isn’t getting 200s consistently in our log files, but we can access the page fine when we try, then there is clearly still some differences between Googlebot and ourselves. What might those differences be?

  1. It will crawl more than us
  2. It is obviously a bot, rather than a human pretending to be a bot
  3. It will crawl at different times of day

This means that:

  • If our website is doing clever bot blocking, it might be able to differentiate between us and Googlebot.
  • Because Googlebot will put more stress on our web servers, it might behave differently. When websites have a lot of bots or visitors visiting at once, they might take certain actions to help keep the website online. They might turn on more computers to power the website (this is called scaling), they might also attempt to rate-limit users who are requesting lots of pages, or serve reduced versions of pages.
  • Servers run tasks periodically; for example, a listings website might run a daily task at 01:00 to clean up all it’s old listings, which might affect server performance.

Working out what’s happening with these periodic effects is going to be fiddly; you’re probably going to need to talk to a back-end developer.

Depending on your skill level, you might not know exactly where to lead the discussion. A useful structure for a discussion is often to talk about how a request passes through your technology stack and then look at the edge cases we discussed above.

  • What happens to the servers under heavy load?
  • When do important scheduled tasks happen?

Two useful pieces of information to enter this conversation with:

  1. Depending on the regularity of the problem in the logs, it is often worth trying to re-create the problem by attempting to crawl the website with a crawler at the same speed/intensity that Google is using to see if you can find/cause the same issues. This won’t always be possible depending on the size of the site, but for some sites it will be. Being able to consistently re-create a problem is the best way to get it solved.
  2. If you can’t, however, then try to provide the exact periods of time where Googlebot was seeing the problems. This will give the developer the best chance of tying the issue to other logs to let them debug what was happening.

If Google can crawl the page consistently, then we move onto our next step.

5. Does Google see what I can see on a one-off basis?

We know Google is crawling the page correctly. The next step is to try and work out what Google is seeing on the page. If you’ve got a JavaScript-heavy website you’ve probably banged your head against this problem before, but even if you don’t this can still sometimes be an issue.

We follow the same pattern as before. First, we try to re-create it once. The following tools will let us do that:

  • Fetch & Render
    • Shows: Rendered DOM in an image, but only returns the page source HTML for you to read.
  • Mobile-friendly test
    • Shows: Rendered DOM and returns rendered DOM for you to read.
    • Not only does this show you rendered DOM, but it will also track any console errors.

Is there a difference between Fetch & Render, the mobile-friendly testing tool, and Googlebot? Not really, with the exception of timeouts (which is why we have our later steps!). Here’s the full analysis of the difference between them, if you’re interested.

Once we have the output from these, we compare them to what we ordinarily see in our browser. I’d recommend using a tool like Diff Checker to compare the two.

Have we found any problems yet?

If we encounter meaningful differences at this point, then in my experience it’s typically either from JavaScript or cookies

Why?

We can isolate each of these by:

  • Loading the page with no cookies. This can be done simply by loading the page with a fresh incognito session and comparing the rendered DOM here against the rendered DOM in our ordinary browser.
  • Use the mobile testing tool to see the page with Chrome 41 and compare against the rendered DOM we normally see with Inspect Element.

Yet again we can compare them using something like Diff Checker, which will allow us to spot any differences. You might want to use an HTML formatter to help line them up better.

We can also see the JavaScript errors thrown using the Mobile-Friendly Testing Tool, which may prove particularly useful if you’re confident in your JavaScript.

If, using this knowledge and these tools, we can recreate the bug, then we have something that can be replicated and it’s easier for us to hand off to a developer as a bug that will get fixed.

If we’re seeing everything is correct here, we move on to the next step.

6. What is Google actually seeing?

It’s possible that what Google is seeing is different from what we recreate using the tools in the previous step. Why? A couple main reasons:

  • Overloaded servers can have all sorts of strange behaviors. For example, they might be returning 200 codes, but perhaps with a default page.
  • JavaScript is rendered separately from pages being crawled and Googlebot may spend less time rendering JavaScript than a testing tool.
  • There is often a lot of caching in the creation of web pages and this can cause issues.

We’ve gotten this far without talking about time! Pages don’t get crawled instantly, and crawled pages don’t get indexed instantly.

Quick sidebar: What is caching?

Caching is often a problem if you get to this stage. Unlike JS, it’s not talked about as much in our community, so it’s worth some more explanation in case you’re not familiar. Caching is storing something so it’s available more quickly next time.

When you request a webpage, a lot of calculations happen to generate that page. If you then refreshed the page when it was done, it would be incredibly wasteful to just re-run all those same calculations. Instead, servers will often save the output and serve you the output without re-running them. Saving the output is called caching.

Why do we need to know this? Well, we’re already well out into the weeds at this point and so it’s possible that a cache is misconfigured and the wrong information is being returned to users.

There aren’t many good beginner resources on caching which go into more depth. However, I found this article on caching basics to be one of the more friendly ones. It covers some of the basic types of caching quite well.

How can we see what Google is actually working with?

  • Google’s cache
    • Shows: Source code
    • While this won’t show you the rendered DOM, it is showing you the raw HTML Googlebot actually saw when visiting the page. You’ll need to check this with JS disabled; otherwise, on opening it, your browser will run all the JS on the cached version.
  • Site searches for specific content
    • Shows: A tiny snippet of rendered content.
    • By searching for a specific phrase on a page, e.g. inurl:example.com/url “only JS rendered text”, you can see if Google has manage to index a specific snippet of content. Of course, it only works for visible text and misses a lot of the content, but it’s better than nothing!
    • Better yet, do the same thing with a rank tracker, to see if it changes over time.
  • Storing the actual rendered DOM
    • Shows: Rendered DOM
    • Alex from DeepCrawl has written about saving the rendered DOM from Googlebot. The TL;DR version: Google will render JS and post to endpoints, so we can get it to submit the JS-rendered version of a page that it sees. We can then save that, examine it, and see what went wrong.

Have we found any problems yet?

Again, once we’ve found the problem, it’s time to go and talk to a developer. The advice for this conversation is identical to the last one — everything I said there still applies.

The other knowledge you should go into this conversation armed with: how Google works and where it can struggle. While your developer will know the technical ins and outs of your website and how it’s built, they might not know much about how Google works. Together, this can help you reach the answer more quickly.

The obvious source for this are resources or presentations given by Google themselves. Of the various resources that have come out, I’ve found these two to be some of the more useful ones for giving insight into first principles:

But there is often a difference between statements Google will make and what the SEO community sees in practice. All the SEO experiments people tirelessly perform in our industry can also help shed some insight. There are far too many list here, but here are two good examples:

7. Could Google be aggregating your website across others?

If we’ve reached this point, we’re pretty happy that our website is running smoothly. But not all problems can be solved just on your website; sometimes you’ve got to look to the wider landscape and the SERPs around it.

Most commonly, what I’m looking for here is:

  • Similar/duplicate content to the pages that have the problem.
    • This could be intentional duplicate content (e.g. syndicating content) or unintentional (competitors’ scraping or accidentally indexed sites).

Either way, they’re nearly always found by doing exact searches in Google. I.e. taking a relatively specific piece of content from your page and searching for it in quotes.

Have you found any problems yet?

If you find a number of other exact copies, then it’s possible they might be causing issues.

The best description I’ve come up with for “have you found a problem here?” is: do you think Google is aggregating together similar pages and only showing one? And if it is, is it picking the wrong page?

This doesn’t just have to be on traditional Google search. You might find a version of it on Google Jobs, Google News, etc.

To give an example, if you are a reseller, you might find content isn’t ranking because there’s another, more authoritative reseller who consistently posts the same listings first.

Sometimes you’ll see this consistently and straightaway, while other times the aggregation might be changing over time. In that case, you’ll need a rank tracker for whatever Google property you’re working on to see it.

Jon Earnshaw from Pi Datametrics gave an excellent talk on the latter (around suspicious SERP flux) which is well worth watching.

Once you’ve found the problem, you’ll probably need to experiment to find out how to get around it, but the easiest factors to play with are usually:

  • De-duplication of content
  • Speed of discovery (you can often improve by putting up a 24-hour RSS feed of all the new content that appears)
  • Lowering syndication

8. A roundup of some other likely suspects

If you’ve gotten this far, then we’re sure that:

  • Google can consistently crawl our pages as intended.
  • We’re sending Google consistent signals about the status of our page.
  • Google is consistently rendering our pages as we expect.
  • Google is picking the correct page out of any duplicates that might exist on the web.

And your problem still isn’t solved?

And it is important?

Well, shoot.

Feel free to hire us…?

As much as I’d love for this article to list every SEO problem ever, that’s not really practical, so to finish off this article let’s go through two more common gotchas and principles that didn’t really fit in elsewhere before the answers to those four problems we listed at the beginning.

Invalid/poorly constructed HTML

You and Googlebot might be seeing the same HTML, but it might be invalid or wrong. Googlebot (and any crawler, for that matter) has to provide workarounds when the HTML specification isn’t followed, and those can sometimes cause strange behavior.

The easiest way to spot it is either by eye-balling the rendered DOM tools or using an HTML validator.

The W3C validator is very useful, but will throw up a lot of errors/warnings you won’t care about. The closest I can give to a one-line of summary of which ones are useful is to:

  • Look for errors
  • Ignore anything to do with attributes (won’t always apply, but is often true).

The classic example of this is breaking the head.

An iframe isn’t allowed in the head code, so Chrome will end the head and start the body. Unfortunately, it takes the title and canonical with it, because they fall after it — so Google can’t read them. The head code should have ended in a different place.

Oliver Mason wrote a good post that explains an even more subtle version of this in breaking the head quietly.

When in doubt, diff

Never underestimate the power of trying to compare two things line by line with a diff from something like Diff Checker. It won’t apply to everything, but when it does it’s powerful.

For example, if Google has suddenly stopped showing your featured markup, try to diff your page against a historical version either in your QA environment or from the Wayback Machine.


Answers to our original 4 questions

Time to answer those questions. These are all problems we’ve had clients bring to us at Distilled.

1. Why wasn’t Google showing 5-star markup on product pages?

Google was seeing both the server-rendered markup and the client-side-rendered markup; however, the server-rendered side was taking precedence.

Removing the server-rendered markup meant the 5-star markup began appearing.

2. Why wouldn’t Bing display 5-star markup on review pages, when Google would?

The problem came from the references to schema.org.

        
<p> <h1 itemprop="name">Avatar</h1> </p> <p> <span>Director: <span itemprop="director">James Cameron</span> (born August 16, 1954)</span> </p> <p> <span itemprop="genre">Science fiction</span> </p> <p> <a href="../movies/avatar-theatrical-trailer.html" itemprop="trailer">Trailer</a> </p> <p></div> </p>

We diffed our markup against our competitors and the only difference was we’d referenced the HTTPS version of schema.org in our itemtype, which caused Bing to not support it.

C’mon, Bing.

3. Why were pages getting indexed with a no-index tag?

The answer for this was in this post. This was a case of breaking the head.

The developers had installed some ad-tech in the head and inserted an non-standard tag, i.e. not:

  • <title>
  • <style>
  • <base>
  • <link>
  • <meta>
  • <script>
  • <noscript>

This caused the head to end prematurely and the no-index tag was left in the body where it wasn’t read.

4. Why did any page on a website return a 302 about 20–50% of the time, but only for crawlers?

This took some time to figure out. The client had an old legacy website that has two servers, one for the blog and one for the rest of the site. This issue started occurring shortly after a migration of the blog from a subdomain (blog.client.com) to a subdirectory (client.com/blog/…).

At surface level everything was fine; if a user requested any individual page, it all looked good. A crawl of all the blog URLs to check they’d redirected was fine.

But we noticed a sharp increase of errors being flagged in Search Console, and during a routine site-wide crawl, many pages that were fine when checked manually were causing redirect loops.

We checked using Fetch and Render, but once again, the pages were fine.

Eventually, it turned out that when a non-blog page was requested very quickly after a blog page (which, realistically, only a crawler is fast enough to achieve), the request for the non-blog page would be sent to the blog server.

These would then be caught by a long-forgotten redirect rule, which 302-redirected deleted blog posts (or other duff URLs) to the root. This, in turn, was caught by a blanket HTTP to HTTPS 301 redirect rule, which would be requested from the blog server again, perpetuating the loop.

For example, requesting https://www.client.com/blog/ followed quickly enough by https://www.client.com/category/ would result in:

  • 302 to http://www.client.com – This was the rule that redirected deleted blog posts to the root
  • 301 to https://www.client.com – This was the blanket HTTPS redirect
  • 302 to http://www.client.com – The blog server doesn’t know about the HTTPS non-blog homepage and it redirects back to the HTTP version. Rinse and repeat.

This caused the periodic 302 errors and it meant we could work with their devs to fix the problem.

What are the best brainteasers you’ve had?

Let’s hear them, people. What problems have you run into? Let us know in the comments.

Also credit to @RobinLord8, @TomAnthonySEO, @THCapper, @samnemzer, and @sergeystefoglo_ for help with this piece.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

When Bounce Rate, Browse Rate (PPV), and Time-on-Site Are Useful Metrics… and When They Aren’t – Whiteboard Friday

15 Jun

Posted by randfish

When is it right to use metrics like bounce rate, pages per visit, and time on site? When are you better off ignoring them? There are endless opinions on whether these kinds of metrics are valuable or not, and as you might suspect, the answer is found in the shades of grey. Learn what Rand has to say about the great metrics debate in today’s episode of Whiteboard Friday.

https://fast.wistia.net/embed/iframe/sh1auopisi?seo=false&videoFoam=true

https://fast.wistia.net/assets/external/E-v1.js

When bounce rate browse rate and ppc are useful metrics and when they suck

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Howdy, Moz fans, and welcome to another edition of Whiteboard Friday. This week we’re chatting about times at which bounce rate, browse rate, which is pages per visit, and time on site are terrible metrics and when they’re actually quite useful metrics.

This happens quite a bit. I see in the digital marketing world people talking about these metrics as though they are either dirty-scum, bottom-of-the-barrel metrics that no one should pay any attention to, or that they are these lofty, perfect metrics that are what we should be optimizing for. Neither of those is really accurate. As is often the case, the truth usually lies somewhere in between.

So, first off, some credit to Wil Reynolds, who brought this up during a discussion that I had with him at Siege Media’s offices, an interview that Ross Hudgens put together with us, and Sayf Sharif from Seer Interactive, their Director of Analytics, who left an awesome comment about this discussion on the LinkedIn post of that video. We’ll link to those in this Whiteboard Friday.

So Sayf and Wil were both basically arguing that these are kind of crap metrics. We don’t trust them. We don’t use them a lot. I think, a lot of the time, that makes sense.

Instances when these metrics aren’t useful

Here’s when these metrics, that bounce rate, pages per visit, and time on site kind of suck.

1. When they’re used instead of conversion actions to represent “success”

So they suck when you use them instead of conversion actions. So a conversion is someone took an action that I wanted on my website. They filled in a form. They purchased a product. They put in their credit card. Whatever it is, they got to a page that I wanted them to get to.

Bounce rate is basically the average percent of people who landed on a page and then left your website, not to continue on any other page on that site after visiting that page.

Pages per visit is essentially exactly what it sounds like, the average number of pages per visit for people who landed on that particular page. So people who came in through one of these pages, how many pages did they visit on my site.

Then time on site is essentially a very raw and rough metric. If I leave my computer to use the restroom or I basically switch to another tab or close my browser, it’s not necessarily the case that time on site ends right then. So this metric has a lot of imperfections. Now, averaged over time, it can still be directionally interesting.

But when you use these instead of conversion actions, which is what we all should be optimizing for ultimately, you can definitely get into some suckage with these metrics.

2. When they’re compared against non-relevant “competitors” and other sites

When you compare them against non-relevant competitors, so when you compare, for example, a product-focused, purchase-focused site against a media-focused site, you’re going to get big differences. First off, if your pages per visit look like a media site’s pages per visit and you’re product-focused, that is crazy. Either the media site is terrible or you’re doing something absolutely amazing in terms of keeping people’s attention and energy.

Time on site is a little bit misleading in this case too, because if you look at the time on site, again, of a media property or a news-focused, content-focused site versus one that’s very e-commerce focused, you’re going to get vastly different things. Amazon probably wants your time on site to be pretty small. Dell wants your time on site to be pretty small. Get through the purchase process, find the computer you want, buy it, get out of here. If you’re taking 10 minutes to do that or 20 minutes to do that instead of 5, we’ve failed. We haven’t provided a good enough experience to get you quickly through the purchase funnel. That can certainly be the case. So there can be warring priorities inside even one of these metrics.

3. When they’re not considered over time or with traffic sources factored in

Third, you get some suckage when they are not considered over time or against the traffic sources that brought them in. For example, if someone visits a web page via a Twitter link, chances are really good, really, really good, especially on mobile, that they’re going to have a high bounce rate, a low number of pages per visit, and a low time on site. That’s just how Twitter behavior is. Facebook is quite similar.

Now, if they’ve come via a Google search, an informational Google search and they’ve clicked on an organic listing, you should see just the reverse. You should see a relatively good bounce rate. You should see a relatively good pages per visit, well, a relatively higher pages per visit, a relatively higher time on site.

Instances when these metrics are useful

1. When they’re used as diagnostics for the conversion funnel

So there’s complexity inside these metrics for sure. What we should be using them for, when these metrics are truly useful is when they are used as a diagnostic. So when you look at a conversion funnel and you see, okay, our conversion funnel looks like this, people come in through the homepage or through our blog or news sections, they eventually, we hope, make it to our product page, our pricing page, and our conversion page.

We have these metrics for all of these. When we make changes to some of these, significant changes, minor changes, we don’t just look at how conversion performs. We also look at whether things like time on site shrank or whether people had fewer pages per visit or whether they had a higher bounce rate from some of these sections.

So perhaps, for example, we changed our pricing and we actually saw that people spent less time on the pricing page and had about the same number of pages per visit and about the same bounce rate from the pricing page. At the same time, we saw conversions dip a little bit.

Should we intuit that pricing negatively affected our conversion rate? Well, perhaps not. Perhaps we should look and see if there were other changes made or if our traffic sources were in there, because it looks like, given that bounce rate didn’t increase, given that pages per visit didn’t really change, given that time on site actually went down a little bit, it seems like people are making it just fine through the pricing page. They’re making it just fine from this pricing page to the conversion page, so let’s look at something else.

This is the type of diagnostics that you can do when you have metrics at these levels. If you’ve seen a dip in conversions or a rise, this is exactly the kind of dig into the data that smart, savvy digital marketers should and can be doing, and I think it’s a powerful, useful tool to be able to form hypotheses based on what happens.

So again, another example, did we change this product page? We saw pages per visit shrink and time on site shrink. Did it affect conversion rate? If it didn’t, but then we see that we’re getting fewer engaged visitors, and so now we can’t do as much retargeting and we’re losing email signups, maybe this did have a negative effect and we should go back to the other one, even if conversion rate itself didn’t seem to take a particular hit in this case.

2. When they’re compared over time to see if internal changes or external forces shifted behavior

Second useful way to apply these metrics is compared over time to see if your internal changes or some external forces shifted behavior. For example, we can look at the engagement rate on the blog. The blog is tough to generate as a conversion event. We could maybe look at subscriptions, but in general, pages per visit is a nice one for the blog. It tells us whether people make it past the page they landed on and into deeper sections, stick around our site, check out what we do.

So if we see that it had a dramatic fall down here in April and that was when we installed a new author and now they’re sort of recovering, we can say, “Oh, yeah, you know what? That takes a little while for a new blog author to kind of come up to speed. We’re going to give them time,” or, “Hey, we should interject here. We need to jump in and try and fix whatever is going on.”

3. When they’re benchmarked versus relevant industry competitors

Third and final useful case is when you benchmark versus truly relevant industry competitors. So if you have a direct competitor, very similar focus to you, product-focused in this case with a homepage and then some content sections and then a very focused product checkout, you could look at you versus them and their homepage and your homepage.

If you could get the data from a source like SimilarWeb or Jumpshot, if there’s enough clickstream level data, or some savvy industry surveys that collect this information, and you see that you’re significantly higher, you might then take a look at what are they doing that we’re not doing. Maybe we should use them when we do our user research and say, “Hey, what’s compelling to you about this that maybe is missing here?”

Otherwise, a lot of the time people will take direct competitors and say, “Hey, let’s look at what our competition is doing and we’ll consider that best practice.” But if you haven’t looked at how they’re performing, how people are getting through, whether they’re engaging, whether they’re spending time on that site, whether they’re making it through their different pages, you don’t know if they actually are best practices or whether you’re about to follow a laggard’s example and potentially hurt yourself.

So definitely a complex topic, definitely many, many different things that go into the uses of these metrics, and there are some bad and good ways to use them. I agree with Sayf and with Wil, but I think there are also some great ways to apply them. I would love to hear from you if you’ve got examples of those down in the comments. We’ll see you again next week for another edition of Whiteboard Friday. Take care.

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

from Blogger https://ift.tt/2taHidF