And even the best of us don’t get everything right all the time. Last year, I looked at how Twitter’s technical infrastructure was hurting their ability to be visible in search (they’ve since fixed a lot of it). For instance, they were 301 redirecting from the www version of the site to the non-www version. Great! Except that they also had blocked the entire www subdomain from search engine crawling with robot.xt. So search engines couldn’t follow the redirects or consolidate the value from all of those links to the www versions of the pages to the corresponding non-www versions (which were the one’s being indexed).

As I put together the slides, I was reminded that it would be impossible to fully cover this topic in an hour. I picked some top issues to talk through, but even then, I really only have to go over what the issues are and how to find them, not the intricacies of how to solve them (which of course, may vary based on the environment). So rather than ask those participating in the webinar to scramble to write down a bunch of links, I figured I’d post a few resources here for a more relaxed perusal. Get a glass of wine! Click at your leisure!

If you can’t attend the webinar live, O’Reilly will make the archive available. Update: the video is now live at the original location.

Search Engine tools

• Google webmaster central
• Bing webmaster center

Blocking Pages From Indexing

• Removing Pages From Google
• Meta Robots Example
• Use of nofollow

JavaScript and AJAX

• Google’s crawlable AJAX standard
• Google’s crawlable AJAX documentation
• JavaScript and Flash crawlability
• Hijax

Canonicalization

• Search engine support for the canonical attribute
• Google’s parameter handling funcctionality
• Google’s documentation
• Bing’s documentation

 Pagination

• Google’s support of pagination attributes
• Pagination implementation details
• Google’s post on pagination
• Bing doesn’t support pagination attributes
• View all option

Status Codes

• Server maintenance
• Google blog post

XML Sitemaps

• sitemaps.org
• Wikipedia

Microsoft-Specific Solutions

Rewriters

• URLRewrite (IIS 7)
• URLRewriter (IIS 6 and 7)
• ISAPI_Rewrite 2 (IIS 6 and 7)

Implementing 301 Redirects

• ASP.NET 4.0
• ASP.NET 4.0
• Configuring IIS and ASP.NET

Normalizing URL Casing

• URL Rewrite for IIS
• ASP.NET MVC

Custom Error Pages

• .NET 2.0 SP2
• Custom error redirects
• ASP.NET customer code

Apache-Specific Solutions

Redirects

• mod_rewrite
• htaccess

Page Speed

• Google and Bing on page speed
• Page speed and usability stats
• Page speed and Google ranking
• mod_pagespeed plugin for Apache

Structured Markup

• schema.org support
• Breadcrumbs
• Product markup
• Rich snippets testing tool (Google)

Search Engine Crawl Issues

• Google’s crawl issues reports
• Google’s crawl issues API options

“The idea is basically to try and level the playing ground a little bit. So all those people who have sort of been doing, for lack of a better word, “over optimization” or “overly” doing their SEO, compared to the people who are just making great content and trying to make a fantastic site, we want to sort of make that playing field a little bit more level.

So that’s the sort of thing where we try to make the Google Bot smarter, we try to make our relevance more adaptive so that people don’t do SEO—we handle that—and then we also start to look at the people who sort of abuse it, whether they throw too many keywords on the page, or whatever they exchange way too many links, or whatever they are doing to sort of go beyond what a normal person would expect in a particular area. So that is something where we continue to pay attention and we continue to work on it, and it is an active area where we’ve got several engineers on my team working on that right now…”

 [And later after talking about the positives of SEO] “Absolutely there are some people who take it too far. What we’re mindful of is when someone says, “We’re White Hat. We continue to do the right thing, and we see the Black Hats who are over optimizing or going too far, and they seem to be doing too well.” So we’ve been working on changes to try to make sure that if you are a White Hat or if you’ve been doing very little SEO that you are going to not be affected by this change. But if you’ve been going way far beyond the pale, then that’s the sort of thing where your site might not rank as highly as it did before.”

A lot of people have asked me what this means for those who include search engine optimization as part of their marketing mix. Some are worried that Google will begin to penalize sites that have implemented search engine optimization techniques. My thoughts? I think that some site owners should worry. But whether or not you should depends on what you mean by search engine optimization.

AS I’ve talked about and written about over and over (notably in my book and most recently in my article about Clay Johnson‘s talk about SEO killing America), SEO means lots of different things to lots of different people. When I talk and write about SEO and when we work with clients here at Nine By Blue, I mean:

• Using search data to better understand your audience and solve their problems (by creating compelling, high-quality content about relevant topics to your business)
• Understanding how search engine crawl and index sites and ensuring that your site’s technical infrastructure can be comprehensively crawled and indexed

But the definition of SEO is a continuum. Some of it is clearly spam. But there’s a gray area of SEO that’s not exactly spam, but it’s really not those two bullets above either.

For instance, I’ll look at a page and see a bunch of keyword-rich links in the footer. “Does anyone click on those?” I might ask. “Nah, those are just there for search engines”. I go to conferences and hear people debating keyword density percentages, how many times a keyword should be repeated in a title tag, how to get links that “appear” natural. At some point, search engine optimization goes beyond making sure pages are as useful as possible for the target audience and that the site is crawlable and becomes a game of guess the algorithms.

Anyone who’s read or heard me before knows that I’m not an advocate for algorithm chasing. Historically, I’ve had this view because I don’t find it productive. Algorithms change hundreds of times a year. Signals differ for individual queries. The goal is always to extract all of the data on the web and show the very best page for searchers. So why not just invest time in making sure all of your content is extractable and are in fact the very best pages?

Now, there’s another reason to follow this strategy.

The type of algorithm changes Matt talked about in this SXSW session remind me a bit of how Google described the Panda algorithm. Panda wasn’t about spam. It was about separating high-quality, useful pages from pages that were just a collection of words about a particular topic. This seems similar, like yet another way of discerning that. At one point in the session, Matt said:

“We’re always trying to best approximate if a user lands on a page, are they going to be really, really happy instead of really, really annoyed? And if it’s the sort of thing where they land on a page and they are going to be annoyed, then that is the sort of thing that we’ll take action on.”

Matt talked about finding ways to surface smaller sites that may be poorly optimized, if, in fact, those sites have the very best content. This is not anything new from Google. They’ve always had a goal to rank the very best content, regardless of how well optimized or not it may be. And I think that’s the key. If a page is the very best result for a searcher, Google wants to rank it even if the site owner has never heard of title tags. And Google wants to rank it if the site owner has crafted the very best title tag possible. The importance there is that it’s the very best result.

Matt talked about this later:

“We tell people over and over again, “Make a compelling site. Make a site that’s useful. Make a site that’s interesting. Make a site that’s relevant to people’s interests… all of the changes we make, over 500 a year, are designed to try to approximate if a user lands on that page, just how happy are they going to be with what they get? So if you keep that in mind, then you should be in good shape no matter what.”

He also mentioned making Googlebot smarter, which is more an evolution of what they’ve been working on for years: being able to extract content from JavaScript, AJAX, Flash, images, forms… We’ve seen this in the last year with smarter handling of paginated content, for instance. (I wrote about the pagination tags Google supports here, but my post was based on a Google video and blog post where Maile Ohye mentions that if you don’t implement the tags, Google will use patterns from your site to try and create paginated clusters for you.)

Another thing to keep in mind about how Matt described this upcoming change is that he wasn’t speaking at a search conference. The audience was at least in part non-SEOs. He introduced himself as the person in charge of catching those who try to cheat Google. He was talking to people who (based especially on the question that triggered Matt’s  comments) were coming from the perspective of thinking of the type of SEO that’s really about reverse engineering algorithms.

Matt first talked about the benefits of SEO. He said to think of SEO like a coach who helps to present yourself better. He said that Google wants to level the playing field so that all content has a chance to compete equally. And when he talked about the kinds of techniques that this algorithm would look for he said they were looking for abuse: too many keywords, too many link exchanges. He contrasted what the algorithm was looking to flag to “great content”.

In particular, Matt said the following in support of SEO:

“The way that I often think about SEO is that it’s like a coach. It’s someone who helps you figure out how to present yourself better. In an ideal world, though, you wouldn’t have to think about presenting yourself and whether search engines can crawl your website. Because they’d just be so good that they can figure out how to call through the Flash, how to crawl through the forums, how to crawl through the JavaScript, how to crawl through whatever it is…

A lot of people seem to think that Google hates SEO. That’s definitely not the case…

 We even made a video about this. If you do a search for webmaster videos, we’ve made something like 400 videos. And we made one specifically to say Google does not hate SEO, because SEO can often be very helpful. It can make a site more crawlable. It can make a site more accessible. It can think about the words that users are going to type whenever they come to a search engine and make sure that those words are on the page, which just makes the site more user-friendly.

So the same sorts of things you do to optimize your return on investment and how well something spreads virally or socially is the exact same sort of stuff that often works well from a search engine perspective. So there is a ton of stuff that is fantastic to do as an SEO, it just makes your content more crawlable and more accessible.“

This isn’t the oft-heralded death of SEO. But it may be the first nail in the coffin of those who go beyond SEO and lose track of creating the best possible content for their audiences.

Check out the article here.

On Monday, the U.S. Federal government kicked off a two week discussion about how they can best improve .gov web sites.

Here’s the thing. .gov web sites are an integral part of how Americans access government information and services. And the visibility of these sites in search engines such as Google and Bing is key to this access. A PEW Internet study from last year found that 82% of online American adults have accessed a government web site in the previous 12 months and 44% start at a major search engine when looking for government information. As I wrote about this before:

As they note in the report:

“As we found in our previous studies on this topic, search engines are frequently the first option when Americans need to find government information or services online. Fully 44% of those who could remember the last government website they visited found that site by conducting an online search. This is much higher than the percentage who visited a site they had used before (16% did this) or who relied on a friend or family member (14%), a government publication or notice (11%) or a general government website such as usa.gov (4%).”

Aaron Smith of Pew Internet & American Life, in an interview with O’Reilly’s Alex Howard said:

“When we asked this question previously in 2003, we found that search engines were far and away the number one way that folks were reaching their destinations,” said Smith. “We found almost exactly the same thing this time around. Sometimes they’ll go to a place that they saw in a notice or an email or a friend or a family member told them about. But, by and large, if they need to find information on their tax bill, they’ll search tax information or IRS in their search engine of choice and get to their destination that way. So that’s very much inline with what we’ve found in previous years and in other areas of online life as well, which is that the search is generally the default entry point to all sorts of information and other types of activities that people take part in.”

This behavior pattern underlines the need for government sites and others who provide government-related data to ensure they can be found in search engines to fully engage with the American public.

But the truth is that some .gov sites are cumbersome and hard to navigate. And there’s a lot of duplication and confusion. Also, there are a lot of sites. In June, President Obama signed an executive order on a Campaign to Cut Government Waste:

“As one of the campaign’s first steps, the Administration will be targeting duplication and waste among federal websites. There are almost 2,000 separate websites across the Federal Government. With so many separate sites, Americans often do not know where to turn for information. The Administration will immediately put a halt to the creation of new websites. The Administration will also shutdown or consolidate 25% of the 2000 sites over the next few months and set a goal of cutting the number of separate, stand alone sites in half over the next year.”

This is a monumental effort. Seriously. But the people involved are highly motivated.  A cross-functional task force has been working on ways to tackle the issue in a number of ways and has been tapping into the resources of experts across the country in usability, customer service, search, and the like. You can read more about this on the White House blog. I’ve been talking to them about things like:

• How can you improve the overall technical infrastructure of .gov web sites so that they can be more easily crawled and indexed?
• How can you look at content strategy from a task-focus to help identify areas to consolidate and expand?
• How can you built best practices/guidelines so that anyone working on a .gov can easily access the same learnings?

A foundation for this already exists at howto.gov, but it needs to be updated and expanded. In addition, the task force is building a specific plan of attack to both consolidate and improve .gov sites and develop long term best practices.

How You Can Help

I’m hosting a one hour “dialogue-a-thon” specifically about search tomorrow, Friday September 23rd at noon Eastern. In advance of that, I’ll be gathering together all of the ideas discussed so far so we can expand on those ideas and start talking about a plan of attack. I’d love for search experts to volunteer to help put together suggested best practices and guidelines around crawlable technical architecture, content strategy and consolidation, and organic search visibility. I’ll help lead that effort and make sure none of the ideas are overlooked.

We’re at SMX East this week and we’re (yes, I’m speaking mostly in the royal “we”, but Todd is on a panel too!) speaking on a BUNCH of panels. If you have a question about anything search related, even if you’re not going to be at the conference, please ask in the comments below, as I’m gathering up questions as fast as I can for panelists. I’ll cover what I can and I’ll ask the speakers and search engine reps on the panels to fill in the rest. In particular, if there’s anything you’d like to ask Google or Bing that’s related to the panels I’m moderating, please let me know!

I’ll post a recap after the conference and talk through the details on the next Office Hours.

Using Searcher Personas (Today, Tuesday 9-13-11, 3:45pm)

We base all of the search acquisition and content strategy work we do on searcher personas. Rather than optimize for keywords, why not actually learn about your customers and solve their problems? I wrote about using searcher personas to better connect with searchers and meet your business goals in my book, Marketing in the Age of Google and will be walking through the process I use at the panel today. A couple of other panelists will be walking through how they use searcher personas as well, which I’m looking forward to seeing!

Duplication, Aggregation, Syndication, Affiliates, Scraping, and Information Architecture (Tomorrow, Wednesday 9-14-11, 10:45am)

Sites can suffer from all kinds of duplication: exact duplication from syndication and aggregation, approximate duplication from overlapping topics… Audiences are confused, search engines are confused, and in some cases, sites with a lot of exact or approximate duplication can be hurt by Google’s Panda algorithm. We’ll dive into all the details and provide actionable advice for differentiating content.

Schema.org, Rel=Author, Rich Snippets (Tomorrow, Wednesday 9-14-11,  1:30pm)

Despite the title of this session, I want to really dive into Google’s search results display. We’ll have Jack Menzel from Google to talk through the details of the new Sitelinks, the kinds of rich snippets available, and anything else you want to know.

Real Answers for Technical SEO Problems (Tomorrow, Wednesday 9-14-11, 3:30pm)

This is always my favorite session of the conference because we really dive deep into real problems sites have and how to solve them. Get your questions in now! (Todd will be on this panel with me!)

Making Data From Google Webmaster Central & Bing Webmaster Tools Actionable (Tomorrow, Wednesday 9-14-11, 5pm)

This topic is near and dear to my heart. Obviously. You can do AWESOME things with this data. We’ll have reps on hand from both Google and Bing.

Pagination and SEO (Thursday 9-15-11,  9am)

Pagination and canonicalization are really complicated topics. Maile from Google will be on hand to help us sort this out.

Beyond Ranking Reports (Thursday 9-15-11, 12:45pm)

I am very excited for this session. These are so many actionable SEO metrics beyond rankings reports. Awesome fun stuff.

Ask the SEOs (Thursday 9-15-11, 2pm)

If you’ve ever been to one of these, you know that it’s a bunch of us arguing about everything. Well, except Greg Boser and me. Lately, he mostly agrees with me.

Hope to see you this week and even if you’re not here, post your questions now!

When I was at Google, I realized that organizations needed education and data about how to operate within our searching culture, and that’s why I was so passionate about building Webmaster Central. Post-Google, I’ve remained passionate about that (it’s why I wrote Marketing the Age of Google — you don’t write books to get rich!) and Nine By Blue has been all about helping organizations improve their sites’ technical architecture and overall content and audience engagement strategy to better take advantage of unpaid search.

But it’s not enough. Organizations need a methodology: a clear process. You need comprehensive education and resources, in-depth training, and you need data – both to measure your search efforts and to pinpoint what issues to tackle first. I talked to and looked at specific issues for literally thousands of site owners while I was at Google, but post-Google, I’ve had the opportunity to dive in much deeper with companies and begin to understand the obstacles that go well beyond 302 redirects where 301s should be and keyword research that shows searchers are looking for “cute cat videos” when the site is full of “cute kitten videos”.

Companies have silos and have to find ways to work together well. Technology departments need to know not only what issues exist, but how to prioritize them within the larger needs of the organization. Is fixing that 302 redirect more important than fixing the broken checkout button on the shopping cart? (Well, probably not.) What is the impact? How do we change the process so that we don’t have to keep fixing the same problem every launch?

The idea behind the name Nine By Blue is that organizations can take numbers (the amazing amount of data available online) and mix in a bit of color (you are looking to connect with real people after all) and build something real and long-lasting that both your audiences and search engines love. Our new logo (created by our haiku-writing graphic designer, Ally) reflects that.

We’ll continue to help organizations improve site architecture and content strategy, but we’re now focusing more time on providing comprehensive educational resources and training, helping organizations improve process, and on providing data. We’ve developed custom reporting that incorporates data from server logs, search engine webmaster tools, and analytics to both monitor progress and identify key obstacles to search success. You need a search blueprint and to that end, we’ve developed a methodology for incorporating search into an organization and for helping you determine ROI and prioritization.

Our first step is launching this site. Watch the learning center as we begin building up resources there. And if you’re interested in training or checking out our reporting, please let me know! Here at Nine By Blue, we’re excited about the changes, and we hope that you join us change the world of search engine optimization so that it becomes integrated in development, marketing, and product strategy.

This session was already SUPER AWESOME because it features Maile Ohye (of Google), Todd Nemet (who formerly worked with me at Google and now works with me here at Nine By Blue), and Jonathon Colman (of REI) who will answer the craziest, most complicated technical questions you’ve got. I’ve peeked at their slides and they’ve got answers to page speed problems, redirect problems, unintentional cloaking, IIS, .net, and browsercap.ini issues, load balancing, performance optimization, JavaScript refactoring, reducing HTTP calls, CSS sprites, Apache modules, crawl efficiency, pagination, sorting, and filtering, URL parameters, multi-select navigation, duplicate content, and domain, region, and language issues with international sites.

I’d make sure to bring a way to take notes.

Bonus Searcher Persona Details

At 3:30, Dennis Goedegebuure from eBay and I will be talking searcher personas. You would think the session couldn’t get any better than that, but you would be wrong! We’ve just added Tamara Adlin to the lineup. She quite literally wrote the book on personas (and the abridged version, if you’re short on time) and she and I worked together when I was developing the idea of searcher personas for my book. (In fact, you’ll find a sidebar in chapter 4 that Tamara wrote.)

(I’ll even have copies of said book on hand if you want to buy one!)

So if you’re add SMX, please come by!

• FailCon, San Francisco, Monday October 25th - I’ll be speaking with Tony Adam on “how to fail at SEO”. Our session is at 9:50 and I’ll be signing books at 4pm.  If you are in the area and want to come out, use the discount code 10Fox for 10% off the already low price.
• Dublin Web Summit, Dublin, Friday October 29th - I’ll be speaking about “7 ways to use search data to build audience engagement”. My session is at 10am and I’ll be signing books at 11:15. Also, I’ll likely be drinking beer and eating potatoes. OK, maybe not at 10am. Although I’ll probably still be jet lagged and it’ll be 2am in Seattle, so why not!
• User Interface Engineering (UIE) 15, Boston, Wednesday, November 10th - I’ll be speaking with Tamara Adlin on “getting them and keeping them there”. Search is how you get people to a web site these days, but once you get searchers there, you likely want them to do something! Not to mention that search data provides great info about your audience. We’ll be talking about how search and user experience go hand and hand.
• Dreamforce, San Francisco, December 6-9 - I’ll be speaking about “advanced online marketing” with Marshall Simmonds and Lauren Vaccarello. I’m not sure the date and time yet, so stay tuned.
• Webinar, Thursday, November 18th - I’m also doing a webinar with Compendium at 2pm eastern on Thursday, November 18th. Not only is it a free webinar, but the first 25 registrants get a free copy of my book! I’ll be taking all of your search-related questions.

I’ll also be doing a book signing at lunch on Wednesday, so you can buy a book there too. And maybe you could bring me a muffin. I’ll probably get hungry too.

Where I’ll be:

Solving the Big URL Issues (Tuesday 10:45 – 12)

Even for those who have been doing SEO for a long time, URL structure can still be an issue. It’s one thing to know that you should implement 301 redirects when you change URLs and that you should use hyphens to separate keywords in URLs, but what really is the best option for pagination and canonicalization? What about faceted navigation? Breadcrumbs? Come ask your toughest questions and we’ll try to get all the URL issues sorted out.

Actionable Metrics and Diagnostics (Tuesday 1:30 – 2:45)

What data is really actionable? How can you best track ROI with SEO? Tons of data is available to us now and we’ll take a look at how best to take advantage of it all to better measure effectiveness and engage with your audience.

Industrial Strength SEO (Tuesday 3:15 – 4:30)

When you’re dealing with SEO at the enterprise level, so much more goes into it than just figuring out URL structure. Multiple teams can impact SEO and it can be difficult to ensure everyone keeps this in mind when they’re making development, marketing, and content changes. We’ll get advice from those who have been there and dig into issues that audience is facing.

Build it Better: Site Architecture for the Advanced SEO (Tuesday 4:45 – 6)

In this session, we’ll talk through advanced issues we didn’t hit in the earlier session. What about site speed? Internal link structure? Crawl efficiency? Bring your questions and we’ll dive in.

Credit Where Credit is Due: Demystifying Attribution (Wednesday 9 – 10:15)

I can’t promise we’ll solve the attribution question, but the panelists will talk through the issues and provide some case studies of what’s working.

Lunch Book Signing (11:45 – 12:45)

Come say hi! Or buy a book! Or bring me a muffin!

Ask the SEOs (Wednesday 2-3)

If you’ve been to an SMX conference before, you know this session is a free for all. Ask questions from those who have been around for a long time. Note: we don’t always agree so we tend to mock each other a lot.

If you’re not at SMX, I’ll do a round up post next week with all the details! Hope to see you this week!