Contributions from Cominvent include LanguageIdentifier, Plugging in Hunspell stemmer in Solr and SOLR-2742 which makes commitWithin more accessible through the SolrJ APIs. Also, Apache Tika is upgraded to version 0.10, fixing several bugs in parsing PDFs and Office documents.

I’ve been actively participating in the Lucene/Solr community for a few years: Answering questions on the solr-user and dev mailing lists, reporting bugs, uploading patches etc. This week I was invited by the PMC as a committer on the Lucene/Solr project, joining the twenty-something other existing committers. I’m honored of this invite, which also shows that the ASF works as a true meritocracy, those who show persistent contribution over time will be given more responsibility.

Looking forward to my first commit and many more to come, improving an already great code base. If you have a JIRA issue you’d like me to work on, please suggest one in the comment field!

So what’s new? For me, the single most important features are the Extended Dismax parser (SOLR-1553) and Geospatial search. The full list of improvements is found in CHANGES.TXT, but here are my favorites:

eDisMax query parser

This new parser builds on disMax, and allows full Lucene query syntax, including support for wildcard and fuzzy searches. It also improves on stopword handling and gives you new pf2 and pf3 parameters for bi- and tri-gram phrase boosts. Your new favourite query parser. Enable with defType=edismax

GEO-spatial querying

Have lat/lon info for your data? Want to sort or boost results based on distance from a point? Want to filter results within a certain radius? You can do this with the new built-in GEO support. Filtering for all results within 5km is done as simple as: &fq={!geofilt pt=45.15,-93.85 sfield=store d=5}. More info at the Solr Wiki.

Better prototyping GUI

The built-in Velocity-based prototyping GUI is improved, and will now work with faceting, GEO, did you mean etc. It’s my favourite place to start mocking up a search UI. It now lives at /solr/browse instead of /solr/itas, and has a nicer color scheme. It’s still not perfect, and my patches to make range faceting dynamic (SOLR-2383) and to add ability to see all fields (SOLR-2384) did not make it in time for the code freeze, but you may add them yourself.

Sort by function

The sort feature now accepts a Function Query as input, meaning you can do stuff like &sort=sum(a,b) asc. This generic feature is also what is used to sort on distance, since GEO distance can be expressed as a function.

Numeric range facets

Earlier we had date range facets, but no numeric range facet. With the new range facet you can do both numeric and date range faceting with the same feature, specifying the gap. E.g. &facet.range=price&facet.range.gap=50 will bring you back a price facet automatically split in ranges of 0-50, 50-100, 100-150… Plans are under way to make range facets even more flexible by allowing unequally spaced gaps, but that will be in a later release (see SOLR-2366)

Take action

So what are you waiting for? Go and download a fresh copy and check it out!

Also, grab a copy of Cominvent’s Solr Product Sheet, giving a quick functional overview of Solr which even you boss understands

Cominvent have updated our training offering to cover all the new features. Check it out over at www.solrtraining.com

We have created a 2-page Apache Solr product sheet, which very briefly (and beautifully) describes the high-level features of the popular search engine, including links for downloading and getting started.

Use it to explain to business persons and decision makers what open source search can do. This is the missing “glossy” merchandise piece of the puzzle if you like.

You are free to re-use the product sheet in your commercial business, as it is licensed under Creative Commons BY-SA, meaning you can even change it as long as you leave the credit and link to Cominvent in place and also share your changes under the same license in the ODF source form.

Download the Solr 3.1 product sheet (PDF).

And here is the Solr 3.1 product sheet master (ODT).

The presentation is in Norwegian.

English transcript

NHST Media Group publishes many online newspapers including DN.no (financial), Tradewinds.no (shipping), ReCharge (renewable energy) etc. This presentation was held by NHST and Cominvent.

Agenda:

• Project background
• Architecture
• Search ABC
• The project
• Summary

Project Background

Large amount of news articles on paper and online.

FAST ESP as search platform since 2006, Solr for tax report search since 2009.

Open source, Linux and Java is heavily used in the organization.

FAST was acquired by Microsoft in 2008 and Linux support discontinued. This prompted a new evaluation of the search architecture, and Solr was chosen for the future, with Cominvent as technology partner.

Architecture before

FAST uses one monolithic index, so all sources shared the same data schema (index-profile). Escenic is the main source of content. A plugin existed to push content to FAST based on triggers.

On the search side each publication were either using the FAST search API directly or some flavor of a home-grown search middleware. However, each publication had their own result presentation logic and innovations in one publication would not benefit the others.

Search ABC

Search is NOT database. Optimized for free text, but also handles boolean logic well.

Commercial engines: FAST/Microsoft, Google Search Appliance (GSA), Autonomy IDOL

Open source engines: Apache Solr/Lucene, Xapian, Elastic Search

Usage areas: Intranet, shopping, social media, news etc

Solr is an open source Java based search server with Lucene in the core. It is released by the Apache Foundation under the permissive open source license Apache Software License 2.0, meaning you can do almost anything you like with the software, including sharing it or closing it and charging for it.

The project

We introduced a new, common search and indexing middleware which all publications use. The role of the middleware  is to isolate the clients from details and changes in the search engine. There is also a presentation layer in the middleware which provide JSP taglibs for delivering a standard result page with pagination, facets, did-you-mean etc. This makes it very rapid to plug in search in a new publication.

All the data sources also now use the same middleware for indexing, taking care of indexing the content to the right search core .

Challenges

Some features of FAST did not exist in Solr. FAST is more a search platform while Solr is a search server. The major difference was linguistic support which is strong in FAST. This was solved in Solr.

We were using entity extraction in FAST, but did not include that in Solr in this project, as it does not come out of the box, but need integration with 3rd party solutions.

Differences

While FAST uses a monolithic index, Solr can be split in cores, each having its own data schema and configuration. This means that if you need to reconfigure or re-index one data source such as tax-list, you do not affect the rest of the articles. It also allows for easy staging of new content to a new core, and then swapping it into production when ready, without the need for another physical staging server as was needed with FAST.

FAST ships with Lemmatization, while in Solr we use stemming, which is inferior and causes some problems. These are mitigated by tuning the stemming dictinoaries.

To give Solr language support, we implemented some language abstractions in the middleware, adding a language field to each document, and choosing separate fields title_no for Norwegian content and title_en for english content, and then making this implementation detail transparent from the search clients.

Tuning

News is fresh meat. You need immediate indexing as things change (push instead of pull). We also implemented date boost through Solr’s Function Query formula. There are tons of formulas available, and there is almost no limit to what you can tune and boost.

Summary

Solr is a lot less resource demanding than FAST. Can easily run virtualized or in the cloud. NHST scaled into the Amazon EC2 cloud during the peak period of the tax list search last year.

Each developer may run a local copy of Solr on his laptop, this was very hard with FAST.

Cleaner architecture than before, more flexible with multiple cores.

A big win to gather all search related business logic into a common search middleware, including a JSP presentation layer.

Superb tuning possibilities, easier to tune than the old engine.

Although there were challenges and we had to sacrifice entity extraction in the first phase, we’re very happy with the decision to migrate to Solr

We have now expanded our comprehensive training offering, as shown in the training modules illustration, covering the whole range from short half-day introduction for anyone to full certification track for developers.

Go visit our training site www.solrtraining.com and sign up for the training which fits you best.

Mike also introduces a cool new merge policy called TieredMergePolicy (LUCENE-854) which is much smarter and slightly more efficient than the default one. Hope this becomes the new default merge policy in Solr.

Open Source Search is gaining more and more traction. First you had Lucene (2001), giving great search for programmers. Then we got Solr (2006) making search accessible for non programmers, but a certain level of expertise is still needed. And then came Constellio, an open source (GPL) enterprise search distribution (distro) built on Solr, adding a slick GUI, connector and crawling support and more.

Say again. A Solr distro?

I call it “distro” because I like to compare the evolution to what we have seen in GNU/Linux. First there was the Linux core. Then there was the GNU tools that made Linux so much more usable but still only for engineers comfortable with the command line. And last, companies like RedHat and Suse built complete distros including modern GUI, ready-to use tools such as OpenOffice, Thunderbird and more. Without these distros, Linux would just have been a “core” leaving to the user to add the extra sugar.

I dare state that the same is about to happen with Open Source Search. There are many companies out there already with their own proprietary Apache Solr/Lucene based ”distro”, but Constellio is the first open-source one I have seen so far.

As a Solr user, you’ll feel at home within ./constellio/tomcat/webapps/constellio/WEB-INF/solrcores/<your_core> where you’ll find the regular schema, solrconfig etc. But I do suspect that any manual edits here will be overwritten by the GUI…

Tapping into Google Search Appliance

The creators of Constellio have done a pretty good job in this first 1.0 release. Easy installation, nice administration GUI, easy to get started crawling, etc. And they have been bold enough to tap into Google’s open-sourced GSA connectors available at Google Code as opposed to using ManifoldCF from Apache or another connector framework. They also hook in to Google OneBox APIs, thus enabling users to plug in to all the smart search “widgets” that can for instance intercept the query, and if it detects a stock ticker, deliver a stock price graph on top of search results. Nifty! I bet Google didn’t anticipate their connector framework being used outside of the GSA…

So what’s the catch?

Well, for one, it is GPL (v3), meaning that it excludes some potential users right away (unless they are able to dual license?). You have to register on the site in order to download, meaning you’ll probably be contacted at some point in time by sales – no big deal. It is open source and the source code is available, but it is not developed by a community in an open way. You can download the source as a zip, but if you change it, who’s gonna maintain your changes? Probably yourself…

Luckily there is no limits on number of documents you can index or the QPS rate. Thus it is a true free (as in free beer) solution, which cannot be said about the weak MS search server Express or the old and maxdoc-limited Omnifind Yahoo! edition. Being free™ may be enough reason to give value to many users who would otherwise have to pay consultants to bring up a solution from scratch based on the individual components.

Constellio’s business model is to live from support and consulting fees, and that may very well work. But I cannot see how they will be able to create a true open community around their product, and for that reason I believe it will be a distro without very large adoption.

Quirks

It is obviously an early version 1.0. If it was an ASF project it would probably have version number 0.x. A few quirks: The logo upload did not work. It identified my Norwegian web pages as Danish, and it crashed on me (see screen shots). But good luck to the creators with making this into a mature Solr distro.

Screen shots

Constellio search page

Constellio collection admin

Constellio - edit collection

Constellio - server management tab

Constellio - connectors management

Constellio - edit field types in GUI

Constellio - configuring a field type with analysis

Constellio - hey, it's only version 1.0

Book cover © Amazon & Wrox

Over due by several years, Wrox just published a book about Microsoft Enterprise Search, including the different FAST flavours. Bravo!

You can ask how all the users of FAST technology could have managed for so many years without some public source of learning the products. Up until now FAST/MS and their partners have been the sole source of learning FAST Search [1]. Now, we’re part of that eco-system and may have profited on the lack of material available, but that’s another story.

The book is written by Jeff Fried (Ex-FAST), Mark Bennett, Natalya Voskresenskaya and Miles Kehoe and covers the chapters

1. What is Enterprise Search
2. Developing a strategy – the business process of search
3. Overview of Microsoft Enterprise search products
4. Search within Sharepoint 2010
5. FAST Search within Sharepoint 2010
6. Customizing search with Sharepoint 2010
7. Introduction to FAST ESP
8. Customnization and deployment of FAST ESP 5.x
9. Advanced topics
10. Enterprise search is social search
11. Search and business intelligence
12. The future of search

I have not read the book yet – but I have bought it to my Kindle and will flip though it on my iPad if time allows. I could of course, humble as I am, have authored much of this book myself, but maybe I can learn a thing or two as well

Click the book cover to go to Amazon.com’s book page.

[1]: Except from http://www.fastforum.info/ (owner) and http://fastesphelp.com/ (by Anand Kumar Pandey)

But what about non Sharepoint users? MS has an offering to them as well, called FAST Search for Internet Sites. Read Comperio’s excellent blog article about it. A bit disappointing that the core is still the more than three year old ESP5.3 wrapped in new MS APIs, but cool that you can still hack the ESP internals..

This figure from the book “Professional Microsoft Search” shows an overview of where the ESP  based offerings fit into the overall product offerings from MS:

20©10 Wrox publishing

Question is; what is the long term outlook for FSIS? It can obviously not stay on the aging ESP5.3 core forever. Will Microsoft upgrade this product or just put it on the last page of the price sheet to have an answer if someone asks? Can the SP version be adapted to work standalone? Perhaps. But it too is in desperate need of a new modern search core. I guess we can assume that since a new book pops up mentioning FAST ESP explicitly, we can expect it to stick around for some while… Give your thoughts about the future of non-sharepoint search from Microsoft.