Most ESP users have already taken action and completed or at least started a migration project. Those who want to stay within the Microsoft world, must move to Sharepoint. For others that is an awkward solution, and for several of these, we have facilitated migrations to Open Source solutions such as Apache Solr™ or ElasticSearch™.

Still not decided what to do? Call us for a chat. We know ESP inside/out and also Lucene based solutions and may guide you through a successful migration whether you like to do 10% or 90% of it yourself. See our migration page for more.

This is mostly a bugfix release over 4.2, including such things as Collections API fixes, SolrCloud replication fixes and more.

If you missed the 4.2 release, the major news was that stored fields are now compressed by default, meaning you could get a significantly smaller index footprint if you store a lot of fields.

Scaling search with Apache Solr 4.0 and SolrCloud from JavaZone on Vimeo.

The most important changes is perhaps better maturity for SolrCloud along with a lot of bug fixes. If you need the ease of scalability that SolrCloud brings, this beta is a safe version to get started with – even in production. We (the developers) promise that we’ll not break backwards compat unless absolutely necessary due to a bug or something.

Some of the contributions from Cominvent include SOLR-3377, SOLR-1856, SOLR-1929, SOLR-3672, SOLR-3467, SOLR-3660, SOLR-3648, SOLR-3439, SOLR-2802, SOLR-3255, SOLR-3351 and more.

So what do you get and how does this compare to Solr? Well, the engine powering CloudSearch is Amazon’s own A9 technology. It features many of the same basic features as Lucene and Solr but there are also differences. You get plain field search, boolean search, range search, facets, stemming, stopwords, synonyms and a relevancy search. The relevancy search can be tuned using a JavaScript like syntax in which you tell it what to boost. The concept reminds me of Solr’s FunctionQueries which lets you “program” your own boosts.

If you want to run Solr but be able to scale it into the cloud, CloudSearch won’t do that for you. Then you should consider using SolrCloud on Amazon EC2 or another cloud platform instead. There are even providers of hosted plans out there such as IndexDepot and WebSolr which take care of the hosting part for you.

I look forward to playing with the service the next time I find myself wondering what to do

Here’s the release statement:

In addition to these news, Jan Høydahl has also contributed these features and fixes to 3.6:

• SOLR-1052: Deprecate/Remove and in favor of in solrconfig.xml
• SOLR-2202: Money/Currency FieldType (Handling multiple currencies in the same field See http://wiki.apache.org/solr/CurrencyField)
• SOLR-2970: CSV ResponseWriter returns fields defined as stored=false in schema
• SOLR-3026: eDismax: Locking down which fields can be explicitly queried (user fields aka uf - control what fields users may query)
• SOLR-2764: Create a NorwegianLightStemmer and NorwegianMinimalStemmer – simpler and sometimes better than the Snowball
• SOLR-2826: URLClassify Update Processor (good for web search)
• SOLR-3140: Make omitNorms default for all numeric field types (simplifies schema, better default)
• SOLR-2901: Upgrade Solr to Tika 1.0
• SOLR-3009: hitGrouped.vm isn’t shipped with 3.x (fixes grouping in the /browse GUI)
• SOLR-3059: Provide example for XSLTUpdateRequest (allows you to POST a Solr search result into another Solr index as XML)
• SOLR-2487: New Ant build target to not include slf4j-jdk14 jar in WAR (nice if you want to use another log framework than JUL)

This version is backwards compatible with the other 3.x versions, so upgrading is as simple as replacing the WAR file.

Even if it’s some work setting up, the benefits to this approach are really compelling:

1. Suggest on multi word or sentences
2. Suggests on prefix of whole line and/or individual words
3. Full relevancy tuning capabilities of Solr (in contrast to a single frequency sorting from TermsComponent or Faceting)
4. Increased recall using Phonetics, Fuzzy, Character normalization, or any other Solr feature
5. Rich suggestions including thumbnails, extra texts etc
6. Easy to mix different “categories” of suggestsions, e.g. “Book titles”, “Authors”, “Genres”, and group these together
7. Much more

So let’s get to it. To make it easy, we’ve shared with you a ready to run example setup at https://github.com/cominvent/autocomplete which provides a template for you to build upon for your own needs. First we’ll guide you through downloading and running the example, indexing some example data with names of all countries and all major cities in the world. Then we’ll look more behind the scenes to explain how it’s all setup.

Running the example

The example contains a complete solr-home configured with an “ac” core, and some example data in CSV format which we’ll feed to our core using HTTP POST (curl). Solr itself is not included in the example, so you’ll need to download Solr 3.5or newer first. We’ll then start the Solr example app, pointing to our autocomplete solr-home.

1. Download and unpack Solr3.5 if you have not already http://www.apache.org/dyn/closer.cgi/lucene/solr/
2. Download and unpack autocomplete example from GitHub: https://github.com/cominvent/autocomplete/zipball/master (alternatively check out the code using git)
3. Cd to the autocomplete folder, open README.TXT and follow the instructions. When done you will have Solr up and running with the example data indexed into the “ac” core.
4. When you browse to http://localhost:8983/solr/ac/browse and start typing, you’ll see countries and cities suggested

Behind the scenes

1. Querying Solr from the page

This Javascript makes prepares the URL to query. Here we ask Solr directly, but normally you’ll want to call some (PHP/JSP/ASPX) servlet of yours which in turn queries the private Solr server. So put the URL to your servlet. Your servlet will then call the autocomplete core similar to what we do here. We call the normal “/select” handler, but with qt=ac we select the handler we want, and instead of XML back we ask for a velocity template called “suggest” (&wt=velocity&v.template=suggest), which formats one suggestion per line. Finally we ask for 20 results.

2. Format the response to suit your Ajax component

In our example we use suggest.vm to format the results to make it suitable for our AutoComplete component. You would probably want to do this formatting in your client code instead. Our simple jQuery plugin expects one suggestion text per line. A more advanced Ajax component may be able to take in multiple pieces of information, such as thumbnail URLs etc.

3. The Autocomplete component

We use jQuery autocomplete component, and it eats the response from suggest.vm one line at a time and displays it in a nice (ehem) dropdown list.

4. Solr schema and config

Now the juice of our solution lies in the AC core itself. It has a schema which matches the input data, and this should be a good starting point for you too. Apart from the fields we return, we also have some special fields which are only searched. These are textnge, textng, textphon, extrasearch and phonetic. They are copied from textsuggest, but have different processing, which can be seen from the fieldTypes:

textnge uses fieldType autocomplete_edge which will match only from the left edge of the suggestion text. For this we use KeywordTokenizerFactory and EdgeNGramFilterFactory along with some regex cleansing.

textng uses fieldType autocomplete_ngram which matches from the start of every word, so that you can get right-truncated suggestions for any word in the text, not only from the first word. The main difference from textnge is the tokenizer which is StandardTokenizerFactory, thus the EdgeNGram filter will produce N-grams for every single token.

textphon and phonetic uses fieldType text_phonetic_do which applies the DoubleMetaphoneFilterFactory phonetic filter. This lets a query for “muhammed” also match “mohammad” etc. Note that it does not make sense to combine N-gram and phonetics, so we only compute phonetic normalization per whole word. The difference between the two fields is how they get populated – textphon gets populated with a copyField from textsuggest, while “phonetic” must be explicitly fed in the content. This is to let you choose whether phonetics should apply to all suggestions or just some.

Finally, the field extrasearch is simply queried in addition to textsuggest, and is a way to match hidden text that is not displayed. In our example we add country codes to this field so you can get a match for “Sweden” by typing “SE”. This field could also contain your hand-edited list of synonyms.

solrconfig.xml

In our solrconfig we have setup two request handler configs, one for our “/browse” interface, for testing, and one “ac” without the Velocity stuff. In real life we would not have a “/browse” section because your main search index would reside in another core and probably contain other data than the autoComplete data. In our search GUI, if you click the search button you will get a normal search result directly from the “ac” core, so don’t be confused by that Let’s look at the config:

We use edismax query parser, and set it up with “qf=textsuggest^30 extrasearch^30.0 textng^50.0 phonetic^10″ and pf=textnge^50.0. So the most important field is textng, and we weigh down the phonetic results to avoid noise. If you want to always search phonetic, add textphon to qf. The textnge field is searched through pf parameter (PhraseField), to give a boost to exact suggestions matching from beginning of query.

In addition to textual relevance we boost (bf) by popularity (in our case population), and we also do a multiplicative boost (boost) formula to be able to say that countries type entries should be more frequently suggested than the ones with type=Cities. This lets you tune the balance between the various types in a dynamic way without reindexing. You can even change a weight with a URL parameter instead of editing solrconfig.xml.

5. The example data

The example data file ac-example.csv is a comma separated file (which you may open in your spreadsheet application if you wish to view it more readable). Its columns match our schema, and you can easily use it as a template for entering you own autosuggest data. To feed the file to Solr you may use any HTTP client you choose, like curl as being done by feed-ac.sh referenced in README.TXT. This concludes the walkthrough. Comments welcome!

References

[1]: Lucid’s blog post about AutoComplete using edgeNgram

Contributions from Cominvent include LanguageIdentifier, Plugging in Hunspell stemmer in Solr and SOLR-2742 which makes commitWithin more accessible through the SolrJ APIs. Also, Apache Tika is upgraded to version 0.10, fixing several bugs in parsing PDFs and Office documents.