Rod,

the data model I'm currently designing and the keyword system are two different things.

My assumption is that current database systems and search engines offer a lot of functionality for finding close matches. So I think we should rely on that (why reinvent the wheel).

Handling synonyms is a different thing. Would be interesting to see how you solve that.

Regards
Martin

Martin;

You're right. The data model and the keyword system are different, but they are not independent. They must work together to create a successful and useful system.

After 40 years in data processing as employee and consultant, on both the monster mainframes and the micro computers, as well a quite a few developing and refining my "index", I have learned a few things about interaction between the various 'off the shelf' parts of systems. They are not necessarily 'plug and play', even when the are purported to be. The data model must support the search engine you choose and vice versa, and the user interface must support both.

That said, and in view of the fact that Joe seems to have already rejected my 'data model', I seriously doubt that you would find my method of handling spelling variations and synonyms would fit into your "index", and I wouldn't want to cloud the issue as you build from the ground up.

Good luck with your endeavours.

Rod

Rod, you misinterpreted my post.

I'm saying your UI needs work, not necessarily your underlying data model or search parser.

I'm open to hearing more about your back-end architecture and how you believe you've solved various problems.

However, your back-end needs a better front-end - the one that's there now is utilitarian and somewhat unfriendly. And the search results display is not especially easy to interpret.

There's a great book on the web user interface: Don't make me think! - it covers how to design a web app UI so it's super obvious how to use it. Mainframe-style interfaces don't translate particularly well to the web, unfortunately.

One of the best things you can do to "test" your web interface is to get 5 people from various walks of life, set them down in front of your app, and tell them to "go for it" with absolutely no instruction. Make sure you have the video cameras running - you will find all kinds of assumptions you made about your interface and its ease of use were totally wrong.

Web users can't "read the manual" for your web site, so a good web site UI needs a lot more thought than your ordinary application.

I've been designing web apps since 1993 and have managed web developers since the late 90s. I lost count of the number of times I sent a web developer back to their office with the response: "you need to rethink your interface" ...

Hi,

a few moments ago I uploaded the PDF version of my first data model. Look at

http://home.vrweb.de/martin_fischer/articles.pdf

Regards

Martin

I see two ways of handling synonyms and spelling variations, pre-processing and post-processing.

Pre-processing is generating all the possible synonyms and spelling variations for the search argument at search time, and then searching for all the variations.

I see the advantage of pre-processing is that a newly discovered synonym or spelling variation is immediately applied to all items. The disadvantage is the extra load on the processor for the generation of the "keywords" on every search.

Post-processing is generating all the possible synonyms and spelling variations on a one-shot basis when the item is entered or edited, and then stored.

I see the advantage of post-processing is that there is a smaller load on the processor for each search. The disadvantage is that a new synonym or spelling variation is not applied to an existing item until is is edited.

A thought that occurs to me is that when I use Google to do a search, it frequently asks if I mean an alternate spelling.  Sometimes I use a two words together to for a single word when Google thinks it ought to be two separate words.  Sometimes I'm looking for a company name that deliberately mis-spelled a word in the name or combined 2 or more words to form a single name.  Google also puts alternative spellings and synonims at the bottom of a search page.  I'm wondering if some sort of similar "spell check" system would be appropriate to help navigate through the index instead of trying to list every single entry by however many ways someone might spell it?

Russ;

Do you remember any examples of when you used two words together for a single (compound)  word and Google thought it ought to be separate words? Or perhaps the opposite, when you used separate words and Google though it should be a compound word?

However it is done, all the words and variations must be in the system somewhere. So we must determine where, and whether we are going to use our own very specific RR words, or some dictionary off the shelf.

Rod

on Google will look for the incorrect speling and give you alternates that it thinks are more correct.  I don't remember what words I used that resulted in that sort of thing.  I retired from a Carrier-Transicold dealer in 2006, so I put transicold into the search to see what came up and I was given references to Carrier-Transicold with a couple of alternative suggestions at the bottom of the page.

Clint Hyde came over yesterday and we did an "alpha test" on his software.  It now supports both simple single keyword look-ups and a query language (e.g. "(koester or sperandeo) and operations" )  In entering sample data, we found a bunch more things to consider (e.g. quotes in article titles?, special characters such as the ø in Pelle Søeborg's name, adding volume and issue number, better help, better defaults to make typing easier, etc.)

We're doing some more "alpha testing" to get the behavior and data content/data model down as solid as we can before doing anything more public.  

Stay tuned!

dave

An interesting little twist to Google's "did you mean ---?".

If you search for "at&sf", the old reporting mark, it asks if you mean "atsf", the new one. If you ask for the new one, it says nothing.

If you search for "drgw", the new reporting mark, it asks if you mean "at&sf", the old one. If you ask for the old one, it asks nothing.

In both cases the new and old give differing counts for items found.

Rod.

p.s. "Don't Make Me Think" arrived yesterday.