What is required to make XMOD-stored data visible to search engine indexers? Is it even possible?

I know this is an insanely novice question but I am only now getting to the point where it matters to me whether the data on my several dnn portals is accesible to search engines.

Good question. I'd like to know the answer too!

My experience is that anything that's not displayed on the page won't get indexed.

My xmod lists seem to get indexed pretty well but I guess as soon as there is data not shown on the page and only visible after search it won't turn up on google.

If your data sets are small then just make sure everything is listed. But if you have thousands of records I don't know what the answer is.

We're not the only ones struggling.

There are hundreds of e-commerce sites out there with products that don't show up in google.

And others that put their entire catalogue in one huge flash object. Just look at Vetus.com and try and find their products in google. Pitiful really..

( thinking....thinking...)

I think that XMOD data that presents directly on an aspx page (which is visible by bots) can get indexed. What is not indexed in anything in a so-called "detail" record. That stuff is in the xmod database until requested by a user. So, good user interface demands sometimes concealing data until summoned by the user but, until summoned, that data is in the dark web.

One could create a XMOD list view that exposed all the information for a given XMOD record, hide that page from normal view, tell googlebot that it is there through a reference in robots.txt. But then what? that content is indexed (I think) but still won't be accessible to a human viewer.

Depending on the data one is storing, this is can be no trivial question.

yes, that's what I was saying. The invisible web.

I did a bit of research on this and we are talking about what Google calls the
'deep web' or the 'hidden web'.

Apparently they are working on methods to search online databases for the huge resource of data that's hidden from normal search engine indexing.

Read the stuff on this site, it's encouraging

http://googlesystem.blogspot.com/2008/04/google-starts-to-index-invisible-web.html

Angus

Surely there has got to be a creative way to solve this. The human interface power of showing summary information with drill-down to the details is absolutely necessary but so is the ability to have important detail content indexed by indexers outside of the DNN environment.

So....my current experiment is to:

1. Create my XMOD data in list / detail structure.

2. Expose to the human user the list information with the normal xmod click-for-details interface.

3. Create another page using the repeater template to display the detail information. Put this on a page that is "visible" to a bot but not in the public menu structure of DNN and place a reference to that page in robots.txt to ensure it gets indexed.

Input and other ideas most welcome.

Ken

Check the DNN forums - there was just a thread about a free site map module that will craw dynamic data.

Greg

Greg,

Thanks for the response. I searched the forums but I'm not finding the thread you mentioned. Do you have another phrase I can use to try to locate it?

Ken

The thread is pinned to the top of Configure It! here:

http://www.dotnetnuke.com/Community/Forums/tabid/795/forumid/108/threadid/207139/scope/posts/Default.aspx

The solutions mentioned here may not do exactly what you are looking for but it may give you some added info.

Greg

This may be what you are looking for.

http://www.ifinity.com.au/Products/Google_Sitemap_For_DNN/

I have been doing some digging take a look at this:

http://www.fatgeorge.co.uk/Home/articles/tabid/373/articleType/ArticleView/articleId/44/Trawling-the-Deep-Web-The-XMod-Way.aspx

fatgeorge,

Excellent stuff. I'm glad somebody of your calibre is looking into this. As I said above, this is no small matter and an excellent answer has been wanting. I haven't thought a lot about what your current experiment but it sure looks promising! Of course, it is not only xmod but any data structure / presentation tool that wants to provide summary -> detail structure. For my purposes, having an excellent XMOD-based solution will be superior!

Ken

I would be wary of 'hiding' text without thoroughly researching and testing the implications. In some cases 'hiding' text constitutes Black Hat SEO, which in some cases may cause Google to ban your website.

[quote]Posted By danopdx on 2/24/2009 12:16 PM

I would be wary of 'hiding' text without thoroughly researching and testing the implications. In some cases 'hiding' text constitutes Black Hat SEO, which in some cases may cause Google to ban your website.[/quote]

Hi Danopdx,

we are not talking about 'hiding' text; because the display of XMod data is dynamic Google and other bots will not necessarily know of its existence or where to find it.

This is the 'hidden' web, we are looking at methods to expose this part of the web to these bots.

fatgeorge,

I now have time to return to this issue and to drive for some form of a solution. I'd like to understand how you built the stuff you referenced in your 11.08.08 post above. This will help me determine how I can achieve my goal...to whit....create an xmod-based list of "articles" using a Repeater Template for full visual control of the display....clicking "Detail" displays the article itself as normal and the article gets indexed by googlebot.

Sooo...
In your example above you used the following parts...

1. Your tag cloud
2. straight XMOD
3. Your XMOD add-in syndication module to get the stuff visible to at least googlebot.

Do I understand correctly?

Thanks,

Ken

Hi Ken,

To get Google and other bots to see your 'hidden' data all you need to do is have an RSS 2.0 feed from your XMod data (see Syndication for XMod) submitted to the search engines as a sitemap.
You do not need the Tag Cloud, that is for user navigation/filtering of XMod data.

If you wish to setup a test for you xmod data please contact me at admin AT fatgeorgeDOTcoDOTuk

It seems that a method has been forgotten here. Please review my notes in this thread.

http://dnndev.com/Forums/tabid/215/forumid/1/tpage/1/view/topic/postid/26593/Default.aspx#27245

Hi Drew,

for this to work wouldn't you have to list all of your records i.e. have all records that you wanted to be indexed in one long list on a page that the searchbot could find and index?

The beauty of the RSS sitemap method is that you just need your standard paged list for users to view the indexing is done on the RSS content which can be all records or a filtered set of data.

Searchbots cannot index urls that they cannot 'see' hence the use of sitemaps and in this case the RSS sitemap.

Take a look at this article on the deep/hidden web:
http://www.fatgeorge.co.uk/Home/Articles/tabid/373/articleType/ArticleView/articleId/44/Trawling-the-Deep-Web--The-XMod-Way.aspx

This an unsolicited confirmation / endorsement of the RSS module from fatgeorge. I've been using it for months. It did exactly what I needed it to do. It remains the perfect answer to getting XMOD detail date indexed while maintaining a crip user-interface.

Though I have XMod pro installed in a different dnn instance, I have not tried the module there.

Hi, is there a way for us to create an xmod provider to be attached to iFinity Google Sitemap so that all articles will get indexed?

Posted By albin on 06 Oct 2009 03:12 AM
Hi, is there a way for us to create an xmod provider to be attached to iFinity Google Sitemap so that all articles will get indexed?

You could download the iFinity Google Sitemap Providers Source Code and create one for XMod.
But this would only give you a sitemap for Google, using the RSS SiteMap method you can submit your sitemap to Google, Yahoo and Bing/Live search.

I hope this isn't a dumb question but will the search bots "walk" the pages of a grid view with >>Next Page links in the footer?

Are we only talking about hidden detail records here or do we need to worry about pages 2 through n of the master list too?

Thanks,

Tom

Posted By tmhudg on 01 Feb 2010 06:03 PM
I hope this isn't a dumb question but will the search bots "walk" the pages of a grid view with >>Next Page links in the footer?

Are we only talking about hidden detail records here or do we need to worry about pages 2 through n of the master list too?

Thanks,

Tom

Good question, something that an SEO expert might know.

Ahhh the mysteries of the Google searchbot.

Recently, I had the need to move an application to a new page location. This was a paged application and I was concerned that I was not going to be indexed by the major search engines.

Since Google searched my outdoors site, http://justnorth.com, several times a day, it was a good test to track which pages were now getting indexed.

I soon discovered that the records on page 1 were immediately getting re-indexed since the links were exposed. But, the deeper links were not getting picked up.

Since Xmod paging is JavaScript generated, I could not rely on my paged deep content being accessible to the search bots.

I needed a fallback option.

That is when I purchased the Syndication add-on for Xmod by Fatgeorge.

I began generating RSS feeds using FatGeorge Syndication and used the generated RSS feed as a site map that I submitted to Google Webmaster Tools.

Problem solved. My paged content is now indexed on a scheduled basis and submitted to Google.

Reference:

How to Get XMod Detail Pages Indexed On Search Engines

http://dnnprofessor.com/Articles/tabid/223/articleType/ArticleView/articleId/243/PageID/290/How_to_Get_XMod_Detail_Pages_Indexed_On_Search_Engines.aspx


Buck