So what do you get and how does this compare to Solr? Well, the engine powering CloudSearch is Amazon’s own A9 technology. It features many of the same basic features as Lucene and Solr but there are also differences. You get plain field search, boolean search, range search, facets, stemming, stopwords, synonyms and a relevancy search. The relevancy search can be tuned using a JavaScript like syntax in which you tell it what to boost. The concept reminds me of Solr’s FunctionQueries which lets you “program” your own boosts.

If you want to run Solr but be able to scale it into the cloud, CloudSearch won’t do that for you. Then you should consider using SolrCloud on Amazon EC2 or another cloud platform instead. There are even providers of hosted plans out there such as IndexDepot and WebSolr which take care of the hosting part for you.

I look forward to playing with the service the next time I find myself wondering what to do

Here’s the release statement:

In addition to these news, Jan Høydahl has also contributed these features and fixes to 3.6:

• SOLR-1052: Deprecate/Remove and in favor of in solrconfig.xml
• SOLR-2202: Money/Currency FieldType (Handling multiple currencies in the same field See http://wiki.apache.org/solr/CurrencyField)
• SOLR-2970: CSV ResponseWriter returns fields defined as stored=false in schema
• SOLR-3026: eDismax: Locking down which fields can be explicitly queried (user fields aka uf - control what fields users may query)
• SOLR-2764: Create a NorwegianLightStemmer and NorwegianMinimalStemmer – simpler and sometimes better than the Snowball
• SOLR-2826: URLClassify Update Processor (good for web search)
• SOLR-3140: Make omitNorms default for all numeric field types (simplifies schema, better default)
• SOLR-2901: Upgrade Solr to Tika 1.0
• SOLR-3009: hitGrouped.vm isn’t shipped with 3.x (fixes grouping in the /browse GUI)
• SOLR-3059: Provide example for XSLTUpdateRequest (allows you to POST a Solr search result into another Solr index as XML)
• SOLR-2487: New Ant build target to not include slf4j-jdk14 jar in WAR (nice if you want to use another log framework than JUL)

This version is backwards compatible with the other 3.x versions, so upgrading is as simple as replacing the WAR file.

Even if it’s some work setting up, the benefits to this approach are really compelling:

1. Suggest on multi word or sentences
2. Suggests on prefix of whole line and/or individual words
3. Full relevancy tuning capabilities of Solr (in contrast to a single frequency sorting from TermsComponent or Faceting)
4. Increased recall using Phonetics, Fuzzy, Character normalization, or any other Solr feature
5. Rich suggestions including thumbnails, extra texts etc
6. Easy to mix different “categories” of suggestsions, e.g. “Book titles”, “Authors”, “Genres”, and group these together
7. Much more

So let’s get to it. To make it easy, we’ve shared with you a ready to run example setup at https://github.com/cominvent/autocomplete which provides a template for you to build upon for your own needs. First we’ll guide you through downloading and running the example, indexing some example data with names of all countries and all major cities in the world. Then we’ll look more behind the scenes to explain how it’s all setup.

Running the example

The example contains a complete solr-home configured with an “ac” core, and some example data in CSV format which we’ll feed to our core using HTTP POST (curl). Solr itself is not included in the example, so you’ll need to download Solr 3.5or newer first. We’ll then start the Solr example app, pointing to our autocomplete solr-home.

1. Download and unpack Solr3.5 if you have not already http://www.apache.org/dyn/closer.cgi/lucene/solr/
2. Download and unpack autocomplete example from GitHub: https://github.com/cominvent/autocomplete/zipball/master (alternatively check out the code using git)
3. Cd to the autocomplete folder, open README.TXT and follow the instructions. When done you will have Solr up and running with the example data indexed into the “ac” core.
4. When you browse to http://localhost:8983/solr/ac/browse and start typing, you’ll see countries and cities suggested

Behind the scenes

1. Querying Solr from the page

This Javascript makes prepares the URL to query. Here we ask Solr directly, but normally you’ll want to call some (PHP/JSP/ASPX) servlet of yours which in turn queries the private Solr server. So put the URL to your servlet. Your servlet will then call the autocomplete core similar to what we do here. We call the normal “/select” handler, but with qt=ac we select the handler we want, and instead of XML back we ask for a velocity template called “suggest” (&wt=velocity&v.template=suggest), which formats one suggestion per line. Finally we ask for 20 results.

2. Format the response to suit your Ajax component

In our example we use suggest.vm to format the results to make it suitable for our AutoComplete component. You would probably want to do this formatting in your client code instead. Our simple jQuery plugin expects one suggestion text per line. A more advanced Ajax component may be able to take in multiple pieces of information, such as thumbnail URLs etc.

3. The Autocomplete component

We use jQuery autocomplete component, and it eats the response from suggest.vm one line at a time and displays it in a nice (ehem) dropdown list.

4. Solr schema and config

Now the juice of our solution lies in the AC core itself. It has a schema which matches the input data, and this should be a good starting point for you too. Apart from the fields we return, we also have some special fields which are only searched. These are textnge, textng, textphon, extrasearch and phonetic. They are copied from textsuggest, but have different processing, which can be seen from the fieldTypes:

textnge uses fieldType autocomplete_edge which will match only from the left edge of the suggestion text. For this we use KeywordTokenizerFactory and EdgeNGramFilterFactory along with some regex cleansing.

textng uses fieldType autocomplete_ngram which matches from the start of every word, so that you can get right-truncated suggestions for any word in the text, not only from the first word. The main difference from textnge is the tokenizer which is StandardTokenizerFactory, thus the EdgeNGram filter will produce N-grams for every single token.

textphon and phonetic uses fieldType text_phonetic_do which applies the DoubleMetaphoneFilterFactory phonetic filter. This lets a query for “muhammed” also match “mohammad” etc. Note that it does not make sense to combine N-gram and phonetics, so we only compute phonetic normalization per whole word. The difference between the two fields is how they get populated – textphon gets populated with a copyField from textsuggest, while “phonetic” must be explicitly fed in the content. This is to let you choose whether phonetics should apply to all suggestions or just some.

Finally, the field extrasearch is simply queried in addition to textsuggest, and is a way to match hidden text that is not displayed. In our example we add country codes to this field so you can get a match for “Sweden” by typing “SE”. This field could also contain your hand-edited list of synonyms.

solrconfig.xml

In our solrconfig we have setup two request handler configs, one for our “/browse” interface, for testing, and one “ac” without the Velocity stuff. In real life we would not have a “/browse” section because your main search index would reside in another core and probably contain other data than the autoComplete data. In our search GUI, if you click the search button you will get a normal search result directly from the “ac” core, so don’t be confused by that Let’s look at the config:

We use edismax query parser, and set it up with “qf=textsuggest^30 extrasearch^30.0 textng^50.0 phonetic^10″ and pf=textnge^50.0. So the most important field is textng, and we weigh down the phonetic results to avoid noise. If you want to always search phonetic, add textphon to qf. The textnge field is searched through pf parameter (PhraseField), to give a boost to exact suggestions matching from beginning of query.

In addition to textual relevance we boost (bf) by popularity (in our case population), and we also do a multiplicative boost (boost) formula to be able to say that countries type entries should be more frequently suggested than the ones with type=Cities. This lets you tune the balance between the various types in a dynamic way without reindexing. You can even change a weight with a URL parameter instead of editing solrconfig.xml.

5. The example data

The example data file ac-example.csv is a comma separated file (which you may open in your spreadsheet application if you wish to view it more readable). Its columns match our schema, and you can easily use it as a template for entering you own autosuggest data. To feed the file to Solr you may use any HTTP client you choose, like curl as being done by feed-ac.sh referenced in README.TXT. This concludes the walkthrough. Comments welcome!

References

[1]: Lucid’s blog post about AutoComplete using edgeNgram

Contributions from Cominvent include LanguageIdentifier, Plugging in Hunspell stemmer in Solr and SOLR-2742 which makes commitWithin more accessible through the SolrJ APIs. Also, Apache Tika is upgraded to version 0.10, fixing several bugs in parsing PDFs and Office documents.

Sure, we have AutoCommit, where clients don’t need to think about committing, but then it gets less flexible; What if you sometimes want to index in larger batches, while other times you need low latency?

Discover CommitWithin! CommitWithin is a commit strategy introduced in Solr 1.4, which lets the client ask Solr to make sure this <add> request gets committed within a certain time. This leaves the control of when to do the commit to Solr itself, optimizing number of commits to a minimum while still fulfilling the update latency requirements. If I send an <add commitWithin=10000> (in an XML update), that tells Solr to make sure the document gets committed within 10000ms, i.e. 10s. You can then continue to add other documents, and Solr will automatically do a <commit> when the oldest <add> is due.

First, CommitWithin was only possible to specify when using XML updates, on the <add> parameter. Later we got the same support in the JSON updates, and as HTTP parameter for Binary and JSON handlers. You can also specify commitWithin through SolrJ, but you have to write four lines of code instead of the one line required to do a plain add():

    UpdateRequest req = new UpdateRequest();
    req.add(mySolrInputDocument);
    req.setCommitWithin(10000);
    req.process(server);

vs:

    server.add(mySolrInputDocument);

I decided to try fixing the situation, making this highly desired feature more visible and easier to use from all kind of clients and Request Handlers. This effort resulted in a new Wiki page describing the feature, and several JIRA issues. One already made it into Solr 3.4 which is on its way out the door these days:

SOLR-2540: CommitWithin as an Update Request parameter. (Solr3.4) This patch introduces the commitWithin request parameter in XML, CSV and Extracting request handlers, letting you specify on the URL itself whether this content is time critical or not. This enables clients like ManifoldCF to prioritize certain data sources or crawls over other, simply by adding a request parameter on the updates (CONNECTORS-202).

The others are scheduled for 3.5 – whenever that will be released. If you need any of these today, add them as patches and build your own version of Solr:

SOLR-2742: Add commitWithin to SolrServer’s add methods. This patch adds another optional commitWithinMs parameter to all the add() methods of SolrJ’s SolrServer, making it fast and much more intuitive for developers to use this feature.

SOLR-2280: commitWithin ignored for a delete query. This patch makes it possible to also specify commitWithin for <delete>s, which makes sense since it is just as important for many applications to quickly commit deletes as it is for adds

Spinning beachball at login screen

I bought a new SSD disk and performed a clean install, just to start from scratch. But even before restoring any of my old settings, I got an issue with spinning beach-ball on the login screen before I could log in. Sometimes it also went straight to “bluescreen” telling me to restart.

The solution was found here, in short  you need to login quickly before the lockup, then open Energy Saving preferences and disable automatic graphics switching. It solved the issue for me..

Java applets not working

This other issue affected my ability to use my internet banking. First of all, Java is not installed by default in Lion, but you can do that by starting “Java Preferences” from the “Utilities” sub folder of your “Applications” folder. It will then request a Java install from Mac App Store. Once done, enable the check box “Enable applet plug-in and Web Start applications”. Re-start your browser and you might be ok.

But for me it wasn’t that simple. In Chrome, nothing showed up, and Firefox simply locked completely trying to load an applet. So next solution is to repair home folder permissions. You may think that it sounds familiar, but do not mix this with the “Repair Permissions” in Disk Utility – which does not fix your home folder. This may be very useful anyway, for me it also fixed issues with two other applications. Here’s how to do it (inspired by many posts, mainly this):

1. Reboot your Mac, and hold cmd+R when rebooting (or hold “option/alt” and then select recovery partition)
2. Once in recovery, on the Utilities menu, select “Terminal”
3. In the Terminal window, write “resetpassword” to start the Password Reset utility (which was on the utilities menu in Leopard)
4. Select your user in the drop-down, and then click the little “Reset” button on the bottom.
5. Reboot and try that page with the applet agina. Voila!

Good luck with your new Lion!

I’ve been actively participating in the Lucene/Solr community for a few years: Answering questions on the solr-user and dev mailing lists, reporting bugs, uploading patches etc. This week I was invited by the PMC as a committer on the Lucene/Solr project, joining the twenty-something other existing committers. I’m honored of this invite, which also shows that the ASF works as a true meritocracy, those who show persistent contribution over time will be given more responsibility.

Looking forward to my first commit and many more to come, improving an already great code base. If you have a JIRA issue you’d like me to work on, please suggest one in the comment field!

The drawing is created using Google draw, and the original is shared on Google Docs. We have licensed the diagram under the very permissive CC-by license which lets you use, modify and re-distribute the diagram, even commercially, as long as you give attribution with link to Cominvent. To get started, simply:

1. Click the thumbnail (link) which will open Google Docs (log in to your Google Account)
2. To make your own copy, select “Make a copy…” in the File menu
3. Make sure to keep the CC license logo and a link to www.cominvent.com
4. Please share your edits with the community if you think it is useful to others

Comments are welcome!

So what’s new? For me, the single most important features are the Extended Dismax parser (SOLR-1553) and Geospatial search. The full list of improvements is found in CHANGES.TXT, but here are my favorites:

eDisMax query parser

This new parser builds on disMax, and allows full Lucene query syntax, including support for wildcard and fuzzy searches. It also improves on stopword handling and gives you new pf2 and pf3 parameters for bi- and tri-gram phrase boosts. Your new favourite query parser. Enable with defType=edismax

GEO-spatial querying

Have lat/lon info for your data? Want to sort or boost results based on distance from a point? Want to filter results within a certain radius? You can do this with the new built-in GEO support. Filtering for all results within 5km is done as simple as: &fq={!geofilt pt=45.15,-93.85 sfield=store d=5}. More info at the Solr Wiki.

Better prototyping GUI

The built-in Velocity-based prototyping GUI is improved, and will now work with faceting, GEO, did you mean etc. It’s my favourite place to start mocking up a search UI. It now lives at /solr/browse instead of /solr/itas, and has a nicer color scheme. It’s still not perfect, and my patches to make range faceting dynamic (SOLR-2383) and to add ability to see all fields (SOLR-2384) did not make it in time for the code freeze, but you may add them yourself.

Sort by function

The sort feature now accepts a Function Query as input, meaning you can do stuff like &sort=sum(a,b) asc. This generic feature is also what is used to sort on distance, since GEO distance can be expressed as a function.

Numeric range facets

Earlier we had date range facets, but no numeric range facet. With the new range facet you can do both numeric and date range faceting with the same feature, specifying the gap. E.g. &facet.range=price&facet.range.gap=50 will bring you back a price facet automatically split in ranges of 0-50, 50-100, 100-150… Plans are under way to make range facets even more flexible by allowing unequally spaced gaps, but that will be in a later release (see SOLR-2366)

Take action

So what are you waiting for? Go and download a fresh copy and check it out!

Also, grab a copy of Cominvent’s Solr Product Sheet, giving a quick functional overview of Solr which even you boss understands

Cominvent have updated our training offering to cover all the new features. Check it out over at www.solrtraining.com