By now, you must be wondering - “If not sitemap.xml, then what’s the cure?” - I believe instead of just spoon feeding the Search Engines with pages with the help of a sitemap.xml file in a hope that it will pick all the listed pages one day, you must rather focus on making the un-indexed pages better and worthy in the eyes of Search Engines by improving the content, acquiring better/more links to them, and so forth. When Google, doesn’t indexes some pages on your website, it doesn’t necessarily mean that the Search Engines had trouble finding the page, but actually the Search Engines never found those pages worthy of keeping them in its main index. Just because a website doesn’t has a sitemap.xml doesn’t mean search engines wont crawl it unless the website has weak internal linking and information architecture.

Please don’t get me wrong here. I am not against the use of sitemap.xml file but what I want to say is that if your website has indexing problem, don’t just rely on the sitemap.xml as the sole indexing factor. Consolidate the internal linking of your website, use nofollow to control the link juice flow and most important – add quality content on the pages and get quality back links for those pages.

• Make it a thumb rule to avoid repeating yourself. If it’s a commonly used object property, place it in CSS. If it’s a repeated Javascript, store it in an external file and call where ever required.

• Don’t mess up the code with unused or unclosed DIV’s. If a DIV opens, it should close as well and if it doesn’t, it shouldn’t open. It’s as simple as that.

• The most common web developer mistake - every page has it’s own unique title and description, please do not put the title and description tag in the include files.

• Please make all links and references to images, CSS and JavaScript root relative by starting them with a slash, “/”. Dreamweaver users, please set the “Links relative” option to “Site root” in the Site Definition wizard. This is handy because root relative links don’t break when files are moved from one directory to another.

• Don’t put too many files in a single directory. Keep the heirachy going and manage the files in different directories.

• Use Validator to keep the code and CSS clean. Clean code is easier to manage then something that’s messed up.

• Don’t use <br> for <p> tag as it’s unprofessional and results in inconsistent layout in different browsers.

• Avoid spacer graphics and nested tables. Use heading tags, unordered lists and numbered lists to organize content instead.

• Make it a habit to put a forward back slash ‘/’ at the end of every URL.

Image credit:Flickr

• Create some rocking Wordpress themes and showcase them on your website as free ware with back link to your website.
• This one is too good. Create a contact form widget that uses Google Docs as database.

If you are creative and have good understanding about your industry, you can also come up with hundreds of such widget bait ideas. But remember, for the success of your widget, it’s important that an average non-techie user should be able to understand and use the widget application and it must be really easy to implement/install.

There are many powerful widgets that can help you to control the web around you. Here are few of them:

1. Bookmark This
2. Email Subscription Widget
3. Picture Badge
4. Recent Visitor
5. Search Blog
6. Snapshot widget
7. Twitter Update
8. Playlist Video Widget
9. Web Poll Widget
10. What I am Listening

1. Google proxy hacking
2. Automated Comment spamming your website’s URL on explicit websites.
3. Automated directory submission of your website in crap directories (your ranking on Yahoo may jump but Google will surely bury you for links from such directories).
4. Your competitor might generate hundreds of links with same anchor text from single IP address.
5. Your competitors might use content scrapping scripts to replicate your entire website. So many times, I have seen Google penalizing the original website and ranking the content scrappers.
6. 301/302/Javascript redirect hijacking.

The consequence of such attacks are not always fatal but even a drop of 5 places can hurt your website’s traffic badly. I know that if you have strong immunity (Google trust rank) you can survive such competitor attacks but it takes time and efforts to build such a strong immunity and not every websites is blessed with it. I don’t think Google should allow website’s competitor destroy its ranking just because it has a weak immunity.

Solutions

• Block the IP address of the websites scrapping your content.
• There are many ways to fight Google proxy hacking like mod rewrite from .htacceess file, reverse cloaking & other PHP scrips.
• Use honeypots.

Additionally, what I feel Google can do here is to let the webmasters drop the the links manually from their webmaster central account which they think might harm their website’s ranking and immunity. Search Engines need to evolve…big time!!!
Image credit: flickr

In the end, I think it was a much needed party for all of us as it helped us rejuvenate and restore our energy levels back to optimum. We all had a terrific time. Cheers!

If we have a look at the origin of a nofollow attribute, we find that nofollow attribute was made with the primary motto to combat comment spam (which it has failed miserably as comment spammers are still employed). Further Google found that nofollow can also help Google bots to firstly determine the most important pages out of huge websites with complex blog categories in little time and secondly the webmasters could use nofollow while linking to some website content which they don’t want to get associated with and vote to. Here is a recent precise statement made by Matt Cutts on use of nofollow attribute:

“The nofollow attribute is just a mechanism that gives webmasters the ability to modify PageRank flow at link-level granularity. Plenty of other mechanisms would also work (e.g. a link through a page that is robot.txt’ed out), but nofollow on individual links is simpler for some folks to use. There’s no stigma to using nofollow, even on your own internal links; for Google, nofollow’ed links are dropped out of our link graph; we don’t even use such links for discovery. By the way, the nofollow meta tag does that same thing, but at a page level.”

Now getting back to my anti-nofollow visionary, the nofollow tags also facilitate a fake information architecture. As we know Google ranks the sites with solid internal navigational architecture higher on SERP as these are the sites which provide rich user experience to the Google users as the important content is just 3 clicks away, nofollow isn’t helping to their cause. Coz what most webmasters are doing at the moment is hoarding the page rank and shooting it on the targeted pages even when the target page is nothing but pure crap and weakly linked from other pages of the site.

Having said that, please don’t take me wrong, as I am not questioning the effectiveness of page rank sculpting, because it’s working great at the moment, but I am not too sure how long will this nofollow rampage last: I am afraid not too long. So I have a simple advise to give – use nofollow but only after you have crystallized your internal navigation and don’t rely 100% on this nofollow ploy as you might soon see Google derank the websites with weak internal structure indulging in page rank sculpting. I am looking forward to SMX Advanced and hopeful that it might clear up the mounted clouds over rank sculpting dilemma.

Are you still wondering if all the pages of your website, indexed in Search Engines or not? Leave your worries to us as now we are offering a unique SEO service - Deep Indexing Solution. Our robots will crawl your website, will find the un-indexed pages and will force the pages from the supplemental index to the main index. Yes, our robots can determine which of your pages are lying in the Supplemental Index. Our deep-indexing campaign not only improves your site’s deep-indexing but also improves search engine ranking for your keyword phrases. Please click here for more information about our premiere service.

301 Redirects
301 redirect is undoubtedly the safest way to redirect a website as all Search Engines treat a ‘301 redirect’ in same way ie. They simply pass all the link value, juice, ranking etc to the redirected page. A 301 redirect tells a search engine that the requested page has permanently been shifted to a different location so search engines simply ignore the original URL and index the destination URL foreg. If we 301 redirect a website that is about watches (let’s say - awesome-watches.com) which is ranking for keyword ‘wrist watches’ to another website – cool-watches.com, the search engines will simply ignore awesome-watches.com and will index and rank cool-watches.com for the keyword ‘wrist watches’. 301 redirect is also the most handy tool to handle canonical URLs. The most common cases when we use 301 redirects are:

• If the page is deleted or permanently moved.

• You want a new top level domain for your website without damaging your link value and rankings. To serve either of the version of www vs non-www. This is done with a combination of 301 redirect and mod-rewrite.

• Content duplication. For example you may 301 redirect www.yoursite.com/index.htm to www.yourwebsite.com

  If you have many topically relevant but outdated websites which your are not willing to maintain anymore, you may slowly redirecting the outdated websites to the most current website. But beware, if there are many websites, redirect them one by one slowly as otherwise you might end up getting flagged for being a spammer. Please take a note that At SMX, all the engineers from all the search engines made a statement that 301 redirect will not carry the full effect if the content of the redirecting website is topically not the same. We should avoid redirecting our website to a site which is topically irrelevant to our website. For e.g. if you are planning to redirect (301) your website about cars, then you should not redirect it to a website selling watches. An inappropriate redirect will not lead to any benefit to your website.

302 Redirects
302 redirect, also known as temporary redirect tell search engines that the content of the requested URL is temporarily available at a different URL location but will be soon restore to the original URL. So in case of a 301 redirect, the search engines will index the original URL, but they will extract the content content from the 301 redirected URL. This is the the most dicey redirect that you can use on your website as all the search engines tend to treat a 302 redirect differently with lots of exceptions to what they claim. Here is a link for any budding 302 hijacker
Mattcutts has explained here with simple examples, how Google, MSN and yahoo handle a 302 redirect. You may consider using a 302 redirect within your website pages (ie. Onsite redirect), when you want to serve search engines a simple version of a URL and the content from a different page with a complex URL, since simple and short URL’s look more click-enticing in the Search Engine result pages foreg www.yourwebsite.com can be 302 redirected to www.yourwebsite.com/userdata?user=12xc2?id=crap
However you must never try a cross domain 302-redirect.
MSN treats a 302 redirect exactly how it treats a 301 permanent redirect, that is, it will always ignore the original URL and instead index the destination URL. Same is the case with Yahoo, but yahoo reserves the right to make exceptions to this declaration (which they do at many occasions). You must be very careful as 302 redirects are often the default redirect in host control panels and JavaScript. Many meta redirects produce the same 302 redirect effect.

Meta-refresh Redirects
Actually a meta-refresh is not a redirect but it is s simple instruction to the browser to refresh the page after a certain period of time (content in seconds, with or with out a new supplied URL instead of the current one). It is situated in the head of the HTML page and looks something like this: <meta http-equiv=”refresh” content=”0;url=http://www.anotherdomain.com/category/shoes.htm”>
All search engines understand a meta-redirect but again, tend to react differently depending on the content figure. If the content time is 0 or 1 second, most search engines take it as a permanent 301 redierct; anything more than 1 is normally considered a 302 redirect. Use meta-refresh only if you your current hosting provider doesn’t allow a 301, primarily because the W3C’s Web Content Accessibility Guidelines (7.4) discourage the creation of auto-refreshing pages, since most web browsers do not allow the user to disable or control the refresh rate and secondly Spammers use a meta-refresh to refresh the page after every 5 seconds to save themselves from any type of ranking punishment. I am sure you don’t want to look like a spammer. If you want get into details here are some really useful threads on meta-refresh tags

Javascript Redirects
Don’t use Javascript redirect as search engines simply don’t understand it or tend to confuse it as a 302 redirect.

Have you ever wondered why an Australian website (yourdomain.com.au) physically hosted in Australia ranks higher when searched on Google.com.au than an evilcompetitor.com hosted in Europe? Or Does a country-specific domain name (TLD) (e.g., .co.uk) actually helps if someone is targeting a specific country market? These are some of the most common issues which many webmasters across the globe face when they are looking to rank a website at different country specific search engines (ie. Google.com/.co.uk/.com.au etc).

I recently wrote this guest post at seomoz.org, featuring the major issues that you must consider while Geographically Targeting a website for country specific search engines (without getting penalized for content duplication). You must read it, I know that it’s not all that comprehensive but I am quite sure that it will give you a good insight of ranking discrepancies in different geographies. Cheers !