The presentation is in Norwegian.

English transcript

NHST Media Group publishes many online newspapers including DN.no (financial), Tradewinds.no (shipping), ReCharge (renewable energy) etc. This presentation was held by NHST and Cominvent.

Agenda:

• Project background
• Architecture
• Search ABC
• The project
• Summary

Project Background

Large amount of news articles on paper and online.

FAST ESP as search platform since 2006, Solr for tax report search since 2009.

Open source, Linux and Java is heavily used in the organization.

FAST was acquired by Microsoft in 2008 and Linux support discontinued. This prompted a new evaluation of the search architecture, and Solr was chosen for the future, with Cominvent as technology partner.

Architecture before

FAST uses one monolithic index, so all sources shared the same data schema (index-profile). Escenic is the main source of content. A plugin existed to push content to FAST based on triggers.

On the search side each publication were either using the FAST search API directly or some flavor of a home-grown search middleware. However, each publication had their own result presentation logic and innovations in one publication would not benefit the others.

Search ABC

Search is NOT database. Optimized for free text, but also handles boolean logic well.

Commercial engines: FAST/Microsoft, Google Search Appliance (GSA), Autonomy IDOL

Open source engines: Apache Solr/Lucene, Xapian, Elastic Search

Usage areas: Intranet, shopping, social media, news etc

Solr is an open source Java based search server with Lucene in the core. It is released by the Apache Foundation under the permissive open source license Apache Software License 2.0, meaning you can do almost anything you like with the software, including sharing it or closing it and charging for it.

The project

We introduced a new, common search and indexing middleware which all publications use. The role of the middleware  is to isolate the clients from details and changes in the search engine. There is also a presentation layer in the middleware which provide JSP taglibs for delivering a standard result page with pagination, facets, did-you-mean etc. This makes it very rapid to plug in search in a new publication.

All the data sources also now use the same middleware for indexing, taking care of indexing the content to the right search core .

Challenges

Some features of FAST did not exist in Solr. FAST is more a search platform while Solr is a search server. The major difference was linguistic support which is strong in FAST. This was solved in Solr.

We were using entity extraction in FAST, but did not include that in Solr in this project, as it does not come out of the box, but need integration with 3rd party solutions.

Differences

While FAST uses a monolithic index, Solr can be split in cores, each having its own data schema and configuration. This means that if you need to reconfigure or re-index one data source such as tax-list, you do not affect the rest of the articles. It also allows for easy staging of new content to a new core, and then swapping it into production when ready, without the need for another physical staging server as was needed with FAST.

FAST ships with Lemmatization, while in Solr we use stemming, which is inferior and causes some problems. These are mitigated by tuning the stemming dictinoaries.

To give Solr language support, we implemented some language abstractions in the middleware, adding a language field to each document, and choosing separate fields title_no for Norwegian content and title_en for english content, and then making this implementation detail transparent from the search clients.

Tuning

News is fresh meat. You need immediate indexing as things change (push instead of pull). We also implemented date boost through Solr’s Function Query formula. There are tons of formulas available, and there is almost no limit to what you can tune and boost.

Summary

Solr is a lot less resource demanding than FAST. Can easily run virtualized or in the cloud. NHST scaled into the Amazon EC2 cloud during the peak period of the tax list search last year.

Each developer may run a local copy of Solr on his laptop, this was very hard with FAST.

Cleaner architecture than before, more flexible with multiple cores.

A big win to gather all search related business logic into a common search middleware, including a JSP presentation layer.

Superb tuning possibilities, easier to tune than the old engine.

Although there were challenges and we had to sacrifice entity extraction in the first phase, we’re very happy with the decision to migrate to Solr

Open Source Search is gaining more and more traction. First you had Lucene (2001), giving great search for programmers. Then we got Solr (2006) making search accessible for non programmers, but a certain level of expertise is still needed. And then came Constellio, an open source (GPL) enterprise search distribution (distro) built on Solr, adding a slick GUI, connector and crawling support and more.

Say again. A Solr distro?

I call it “distro” because I like to compare the evolution to what we have seen in GNU/Linux. First there was the Linux core. Then there was the GNU tools that made Linux so much more usable but still only for engineers comfortable with the command line. And last, companies like RedHat and Suse built complete distros including modern GUI, ready-to use tools such as OpenOffice, Thunderbird and more. Without these distros, Linux would just have been a “core” leaving to the user to add the extra sugar.

I dare state that the same is about to happen with Open Source Search. There are many companies out there already with their own proprietary Apache Solr/Lucene based ”distro”, but Constellio is the first open-source one I have seen so far.

As a Solr user, you’ll feel at home within ./constellio/tomcat/webapps/constellio/WEB-INF/solrcores/<your_core> where you’ll find the regular schema, solrconfig etc. But I do suspect that any manual edits here will be overwritten by the GUI…

Tapping into Google Search Appliance

The creators of Constellio have done a pretty good job in this first 1.0 release. Easy installation, nice administration GUI, easy to get started crawling, etc. And they have been bold enough to tap into Google’s open-sourced GSA connectors available at Google Code as opposed to using ManifoldCF from Apache or another connector framework. They also hook in to Google OneBox APIs, thus enabling users to plug in to all the smart search “widgets” that can for instance intercept the query, and if it detects a stock ticker, deliver a stock price graph on top of search results. Nifty! I bet Google didn’t anticipate their connector framework being used outside of the GSA…

So what’s the catch?

Well, for one, it is GPL (v3), meaning that it excludes some potential users right away (unless they are able to dual license?). You have to register on the site in order to download, meaning you’ll probably be contacted at some point in time by sales – no big deal. It is open source and the source code is available, but it is not developed by a community in an open way. You can download the source as a zip, but if you change it, who’s gonna maintain your changes? Probably yourself…

Luckily there is no limits on number of documents you can index or the QPS rate. Thus it is a true free (as in free beer) solution, which cannot be said about the weak MS search server Express or the old and maxdoc-limited Omnifind Yahoo! edition. Being free™ may be enough reason to give value to many users who would otherwise have to pay consultants to bring up a solution from scratch based on the individual components.

Constellio’s business model is to live from support and consulting fees, and that may very well work. But I cannot see how they will be able to create a true open community around their product, and for that reason I believe it will be a distro without very large adoption.

Quirks

It is obviously an early version 1.0. If it was an ASF project it would probably have version number 0.x. A few quirks: The logo upload did not work. It identified my Norwegian web pages as Danish, and it crashed on me (see screen shots). But good luck to the creators with making this into a mature Solr distro.

Screen shots

Constellio search page

Constellio collection admin

Constellio - edit collection

Constellio - server management tab

Constellio - connectors management

Constellio - edit field types in GUI

Constellio - configuring a field type with analysis

Constellio - hey, it's only version 1.0

The latest move is by the Apache Software Foundation, whose Apache licensed “Harmony” implementation of the JVM is being banned by Oracle because they want OpenJDK to be the only open implementation of Java (obviously to make sure there is a reason to purchase JRockit from Oracle and to keep control). Apache this week threatens to leave the JCP (Java Community Process) if Oracle does not grant the Harmony project it’s legal right to the TCK.

Personally I cheer for Apache and hope the other JCP members will back the claim, and with the help of Google (and hopefully IBM) eventually see a true open model for the stewardship of Java, including an Apache licensed JVM for anyone to use freely. At the end of the day that will give Java a huge boost and attract more developers.

Jan will be speaking about “Key topics when migrating from FAST to Solr”, pitched towards a Solr audience. There will be a short overview of FAST ESP, and a walkthrough of the migration process including what pain-points you could expect and how to handle them.

The seminar is at Radisson Blu Plaza Hotel in Oslo, between 8-10am wednesday May 5th.

Join us for a discussion around these topics:

• What is open source and how does it affect search technology?
• How does Solr/Lucene really work?
• What alternatives are there within Solr/Lucene?

We’ll also present a few customer cases to highlight successful implementation of open source enterprise search.

See the attached invitation (PDF, Norwegian) for more.

Sign up now

Perhaps it’s time for a faster, bigger, more feature rich, more extensible or more affordable search engine? A migration requires good and structured planning and deep knowledge of the existing solution as well as the target technology.

With almost 10 years of full-time, hands-on consulting experience within all kinds of enterprise search solutions, Jan Høydahl at Cominvent has what it takes. See our new migration page for more info.

The training material, developed by Lucid Imagination, is recently updated to cover Solr 1.4, plus some additional contrib features like Carot2 clustering. The handout comprises more than 400 slides, but fear not – there are plenty of labs and room for Q&A and discussions, which makes the pace comfortable.

Soon to come is an advanced training class covering more complex issues such as performance tuning, scaling, plugin development etc. Stay tuned!

Read more at www.solrtraining.com and also follow us on Twitter.

Wrong. Maybe you did not know it, but the official Solr 1.4 release can easily be extended with loads of so called “Patches”. A “patch” is normally referred to as something fixing a problem and is perhaps not a good name for extra functionality, but that’s the way it works in the Apache world.

Patch SOLR-773 adds local search to Solr, enabling stuff like

• Only consider results closer than 50km from me (filter)
• Sort the resturant hits by distance from my location
• Rank restaurant hits by user ratings, but boost those in walking distance a bit

To learn more, view a recording of Grant Ingersoll’s excellent webcast on TheServerSide, where he explains more and also talks to YP.com and a GIS company about their experiences with GEO search in Solr.

I could have compiled the source from scratch, but a quick search gave this page from Gdgt providing a VMWare image of a complete install of Chrome OS. So I created a new virtual machine in VMWare Fusion on my MacBook, selected “Other Linux 2.6.x kernel” as OS type and poited it to the .vmdk disk image. See Figure 1 for how that looks.

Figure 1: VMWare Fusion with Chrome OS VM

After powering on the VM, it virtually took only 8 seconds for it to boot into the login screen. Here you fill in a valid Google login, e.g. a GMail address:

Then wait another 9 seconds for the Chrome browser to show up, already logged in to your GMail account. Nice. Everything in Chrome OS are web apps, and the Chrome browser shows App tabs differently than normal web page tabs. This is a nice and clean idea. Signing in to an OS with a web-based user name and password feels strange, but hey, it just works! And if you’re without internet connection, you can still sign in with an account that you’ve used before!

From here, browsing works like normal. You can create tabs, windows, switch between windows with Alt-TAB etc. In the upper righthand corner are some icons which provides you with some menus for battery status, WiFi and network connection and a general menu with access to options etc.

I was unable to test WiFi since my VMWare only provides ethernet to the local OS. I also tried to start the leftmost tab which is supposed to be a quick-launch for Web Apps, but it requires a <user>@google.com account which I don’t have, so I don’t know how to do that:

Glitches

This being an early pre-release of the OS one year before the launch of the first public version, no wonder there are some way to go still. Here are some of the minor issues I found

1. Not possible to change locale in login screen. U.S. keyboard layout is default., which means Norwegians need to search for the @ key and others.
2. Loggin in to a “Google Apps for Domains” account, auto-login to mail and calendar gets lost.
3. The quick-launch tab to the left does not let me login with a GMail account…
4. The whole thing just freezes sometimes Could be due to VMWare?

Read more

Here are a few articles I found useful

• GDGT: Download Google Chrome OS
• Live Blog: The Google Chrome OS Press Event
• PCWorld: Google Chrome OS: Visual Tour

Apache Solr 1.4 has been released and is now available for
public download! http://www.apache.org/dyn/closer.cgi/lucene/solr/

Solr is the popular, blazing fast open source enterprise search
platform from the Apache Lucene project. Its major features include
powerful full-text search, hit highlighting, faceted search,
dynamic clustering, database integration, and rich document (e.g.,
Word, PDF) handling. Solr is highly scalable, providing distributed
search and index replication, and it powers the search and
navigation features of many of the world’s largest internet sites.

Solr is written in Java and runs as a standalone full-text search
server within a servlet container such as Tomcat. Solr uses the
Lucene Java search library at its core for full-text indexing and
search, and has REST-like HTTP/XML and JSON APIs that make it easy
to use from virtually any programming language. Solr’s powerful
external configuration allows it to be tailored to almost any type
of application without Java coding, and it has an extensive plugin
architecture when more advanced customization is required.

New Solr 1.4 features include
- Major performance enhancements in indexing, searching, and
  faceting
- Revamped all-Java index replication that’s simple to configure
  and can replicate config files
- Greatly improved database integration via the DataImportHandler
- Rich document processing (Word, PDF, HTML) via Apache Tika
- Dynamic search results clustering via Carrot2
- Multi-select faceting (support for multiple items in a single
  category to be selected)
- Many powerful query enhancements, including ranges over arbitrary
  functions, and nested queries of different syntaxes
- Many other plugins including Terms for auto-suggest, Statistics,
  TermVectors, Deduplication

Getting Started
---------------
New to Solr? Follow the steps below to get up and running ASAP.

1. Download Solr at http://www.apache.org/dyn/closer.cgi/lucene/solr/
2. Check out the tutorial at http://lucene.apache.org/solr/tutorial.html
3. Read the Solr wiki (http://wiki.apache.org/solr) to learn more
4. Join the community by subscribing to solr-user@lucene.apache.org
5. Give Back (Optional, but encouraged!)
   See http://wiki.apache.org/solr/HowToContribute

For more information on Apache Solr, see http://lucene.apache.org/solr

Release-declaration sourced from Grant Ingersoll’s blog.

Cominvent delivers training on Solr 1.4.