Dear David
Despite over 0.5 mio products, search is generally performing very well on my dedicated server. This is unfortunately (but obviously) not the case when a specific category or merchant is searched that has 50,000 or 100,000 products. In my opinion, there's no real benefit in giving my site visitors access to these kind of searches and therefore removing the links to A-Z Merchants and A-Z Categories is probably the easiest and best solution to resolve this issue.
But what about the search engines? Do they need access to A-Z Merchants or A-Z Categories to properly index the website or for SEO reasons? If not, how do I stop them to access these pages/make these searches?
I really hope you can shed some light on this topic.
Best regards,
Ivo
Hello David
Thank you very much for your detailed explanations.
How do I block both category and brand in the wordpress Price Tapestry distribution? Do I need to block the actual or the virtual base hrefs or both?
My robots.txt currently looks as following, but I'am not sure if this is all correct for the Wordpress Price Tapestry distribution?
User-agent: *
Disallow: /categories.php
Disallow: /brands.php
Disallow: /reviews.php
Disallow: /category/
Disallow: /brand/
Disallow: /review/
Disallow: /admin/
Disallow: /search.php
Disallow: /jump.php
Best regards,
Ivo
Hello Ivo,
Yes, that's the standalone Price Tapestry recommended robots.txt so doesn't apply to a WordPress site using the plugin; it's best to continue to disallow jump.php etc but you would need to include the folder of the Price Tapestry installation; so if Price Tapestry is installed in /pt/ use something like:
User-Agent: *
Disallow: /category
Disallow: /brand
Disallow: /pt/jump.php(where /category and /brand are what you have configured as the "Virtual" base hrefs in wp-admin > Settings > PriceTapestry.org)
Cheers,
David.
--
PriceTapestry.com
Hello Ivo,
I normally recommend giving search engines access to the Merchant index as this guarantees a path to every product page on your site; in fact in the recommended robots.txt in the standalone Price Tapestry distribution I suggest blocking /category/ and /brand/, however some users do actually like having the category and brand index pages open to search engines as they do provide very good keyword variety.
However as you suggest you can easily remove the prominent links from the site by removing the line commencing
<br />Search or browse by...from the Search Form HTML template in wp-admin > Settings > PriceTapestry.org, and simply leave a discreet "Merchant A-Z" link, linking directly to /merchant (or whatever you have used as your Merchant A-Z virtual base HREF) for search engines to pick-up and that should do the trick...Don't forget of course that the plugin generates an XML sitemap through the /shopping?pto_sitemap=xml URL (where /shopping is the permalink of the page containing the [pto] shortcode)...
Cheers,
David.
--
PriceTapestry.com