The presentation is in Norwegian.

English transcript

NHST Media Group publishes many online newspapers including DN.no (financial), Tradewinds.no (shipping), ReCharge (renewable energy) etc. This presentation was held by NHST and Cominvent.

Agenda:

• Project background
• Architecture
• Search ABC
• The project
• Summary

Project Background

Large amount of news articles on paper and online.

FAST ESP as search platform since 2006, Solr for tax report search since 2009.

Open source, Linux and Java is heavily used in the organization.

FAST was acquired by Microsoft in 2008 and Linux support discontinued. This prompted a new evaluation of the search architecture, and Solr was chosen for the future, with Cominvent as technology partner.

Architecture before

FAST uses one monolithic index, so all sources shared the same data schema (index-profile). Escenic is the main source of content. A plugin existed to push content to FAST based on triggers.

On the search side each publication were either using the FAST search API directly or some flavor of a home-grown search middleware. However, each publication had their own result presentation logic and innovations in one publication would not benefit the others.

Search ABC

Search is NOT database. Optimized for free text, but also handles boolean logic well.

Commercial engines: FAST/Microsoft, Google Search Appliance (GSA), Autonomy IDOL

Open source engines: Apache Solr/Lucene, Xapian, Elastic Search

Usage areas: Intranet, shopping, social media, news etc

Solr is an open source Java based search server with Lucene in the core. It is released by the Apache Foundation under the permissive open source license Apache Software License 2.0, meaning you can do almost anything you like with the software, including sharing it or closing it and charging for it.

The project

We introduced a new, common search and indexing middleware which all publications use. The role of the middleware  is to isolate the clients from details and changes in the search engine. There is also a presentation layer in the middleware which provide JSP taglibs for delivering a standard result page with pagination, facets, did-you-mean etc. This makes it very rapid to plug in search in a new publication.

All the data sources also now use the same middleware for indexing, taking care of indexing the content to the right search core .

Challenges

Some features of FAST did not exist in Solr. FAST is more a search platform while Solr is a search server. The major difference was linguistic support which is strong in FAST. This was solved in Solr.

We were using entity extraction in FAST, but did not include that in Solr in this project, as it does not come out of the box, but need integration with 3rd party solutions.

Differences

While FAST uses a monolithic index, Solr can be split in cores, each having its own data schema and configuration. This means that if you need to reconfigure or re-index one data source such as tax-list, you do not affect the rest of the articles. It also allows for easy staging of new content to a new core, and then swapping it into production when ready, without the need for another physical staging server as was needed with FAST.

FAST ships with Lemmatization, while in Solr we use stemming, which is inferior and causes some problems. These are mitigated by tuning the stemming dictinoaries.

To give Solr language support, we implemented some language abstractions in the middleware, adding a language field to each document, and choosing separate fields title_no for Norwegian content and title_en for english content, and then making this implementation detail transparent from the search clients.

Tuning

News is fresh meat. You need immediate indexing as things change (push instead of pull). We also implemented date boost through Solr’s Function Query formula. There are tons of formulas available, and there is almost no limit to what you can tune and boost.

Summary

Solr is a lot less resource demanding than FAST. Can easily run virtualized or in the cloud. NHST scaled into the Amazon EC2 cloud during the peak period of the tax list search last year.

Each developer may run a local copy of Solr on his laptop, this was very hard with FAST.

Cleaner architecture than before, more flexible with multiple cores.

A big win to gather all search related business logic into a common search middleware, including a JSP presentation layer.

Superb tuning possibilities, easier to tune than the old engine.

Although there were challenges and we had to sacrifice entity extraction in the first phase, we’re very happy with the decision to migrate to Solr

Book cover © Amazon & Wrox

Over due by several years, Wrox just published a book about Microsoft Enterprise Search, including the different FAST flavours. Bravo!

You can ask how all the users of FAST technology could have managed for so many years without some public source of learning the products. Up until now FAST/MS and their partners have been the sole source of learning FAST Search [1]. Now, we’re part of that eco-system and may have profited on the lack of material available, but that’s another story.

The book is written by Jeff Fried (Ex-FAST), Mark Bennett, Natalya Voskresenskaya and Miles Kehoe and covers the chapters

1. What is Enterprise Search
2. Developing a strategy – the business process of search
3. Overview of Microsoft Enterprise search products
4. Search within Sharepoint 2010
5. FAST Search within Sharepoint 2010
6. Customizing search with Sharepoint 2010
7. Introduction to FAST ESP
8. Customnization and deployment of FAST ESP 5.x
9. Advanced topics
10. Enterprise search is social search
11. Search and business intelligence
12. The future of search

I have not read the book yet – but I have bought it to my Kindle and will flip though it on my iPad if time allows. I could of course, humble as I am, have authored much of this book myself, but maybe I can learn a thing or two as well

Click the book cover to go to Amazon.com’s book page.

[1]: Except from http://www.fastforum.info/ (owner) and http://fastesphelp.com/ (by Anand Kumar Pandey)

But what about non Sharepoint users? MS has an offering to them as well, called FAST Search for Internet Sites. Read Comperio’s excellent blog article about it. A bit disappointing that the core is still the more than three year old ESP5.3 wrapped in new MS APIs, but cool that you can still hack the ESP internals..

This figure from the book “Professional Microsoft Search” shows an overview of where the ESP  based offerings fit into the overall product offerings from MS:

20©10 Wrox publishing

Question is; what is the long term outlook for FSIS? It can obviously not stay on the aging ESP5.3 core forever. Will Microsoft upgrade this product or just put it on the last page of the price sheet to have an answer if someone asks? Can the SP version be adapted to work standalone? Perhaps. But it too is in desperate need of a new modern search core. I guess we can assume that since a new book pops up mentioning FAST ESP explicitly, we can expect it to stick around for some while… Give your thoughts about the future of non-sharepoint search from Microsoft.

As an example, we recently migrated the Norwegian financial newspaper Dagens Næringsliv and several other online newspapers from FAST to Solr, including Norwegian linguistics and Escenic integration.

Our friends in Lucid Imagination will host a free webinar on this topic on November 18th, don’t miss it if you want to hear more success stories.

Also see our migration slides.

Jan will be speaking about “Key topics when migrating from FAST to Solr”, pitched towards a Solr audience. There will be a short overview of FAST ESP, and a walkthrough of the migration process including what pain-points you could expect and how to handle them.

Perhaps it’s time for a faster, bigger, more feature rich, more extensible or more affordable search engine? A migration requires good and structured planning and deep knowledge of the existing solution as well as the target technology.

With almost 10 years of full-time, hands-on consulting experience within all kinds of enterprise search solutions, Jan Høydahl at Cominvent has what it takes. See our new migration page for more info.

As a part of that planning process, we have decided that in order to deliver more innovation per release in the future, the 2010 products will be the last to include a search core that runs on Linux and UNIX.

This should not be a shock to anyone, and I hinted to this danger in my blog post about the acquisition:

Let us not hope that the Linux, AIX and Solaris versions will be discontinued. I don’t expect that to happen in the short term, as the press release clearly states that they will be supported…

I guessed back then that Linux support would continue for another five years, but it turned out to be less than two While it is an expected and understandable, I think we all hoped that the press release following the acquisition hinted for a new, more open policy from the software giant, but that turned out to be too optimistic. But existing customers are not left completely alone; Olstad continues:

We will support ESP 5.3—the cross-platform search core in the 2010 products—for 10 years as per our standard support policy (5 years mainstream support and 5 years extended support).  Non-Windows customers on ESP 5.3 can continue running their core on Linux and UNIX and add Windows-only innovations or cloud-based services by using a mixed-platform architecture.

Many search consultant with experience from FAST ESP have experienced that the product is more capable under Linux than under Windows when you start pushing the boundaries. ESP is also easier to manage on a Linux platform due to SSH and friends. But that is not the major selling point anyway.

What this will mean for the existing and potential future customers is hard to predict. Obviously some organizations, especially smaller ones, demand Linux only in their data centers – these will start looking for another vendor.  Some of the bigger companies already have mixed environments and will easily be able to migrate, and may be happy.

Last but not least, the hint of cloud-support from the blog post opens another opportunity for existing customers running on Linux: Outsource the whole search hosting to Microsoft and don’t worry about the servers or OS at all. This may be the best option for many? Expect to see more references to hosted search going forward!

Office 14

The main news is that ESP will be included in Office 14, to serve as the advanced/extended search server of Sharepoint. It has long been a problem that the Sharepoint Search Server can only handle about 50 million documents, and by offering the FAST-based extension, this will be extended to almost infinite number of docs, as well as enabling some features which Sharepoint did not have as well (faceted navigation being one of them). The product will be more limited than the full-fledged ESP in that you cannot tune as many parameters. Much of the middleware and administrative components of ESP are stipped out and replaced with Windows/Sharepoint/MSSQL components, for a tighter integration.

But Office14 will not ship before 2010, so as a gap-filler Microsoft will sell ESP for Sharepoint for as little as USD 25.000 per server, which is a fraction of the typical license cost for such a system. This product is basically the same as today’s ESP, but if you intend to upgrade to Office 14 Extended Search, you better not use all the features of ESP, but stick to the recommended customization options which are compatible with the coming Office 14 search.

FAST Search for Internet Business

The second product announced is “FAST Search for Internet Business”, intended to fill the needs of the typical existing FAST customer within site search or e-commerce. Even the Linux versions of this product will be developed and maintained alongside the Windows versions.

A good question is how MS/FAST is going to maintain all these code bases going forward. I’d expect a consolidation sooner or later, and perhaps also an end-of-life announcement for the Linux platform support within the next 5 years. But that will only be speculations anyway

Today, the deal where Microsoft buys FAST, was completed. That means that the Norwegian search engine vendor Fast Search & Transfer is now a fully owned subsidiary of Microsoft.

The FAST ESP product will continue to be offered on all current platforms, and the FAST sales and tech organization continues to operate almost as before, so customers and users will not experience any noise around this transaction.

FAST, when under the MS umbrella, will of course increase focus within the MS Office Sharepoint segment, and will together with MS engineers make an even smoother packaging of the technologies to new and existing customers of high-end Sharepoint sites with large data volumes.

Expect to see continued innovation from FAST in the years to come, and expect also to see a shift towards stronger support for the Windows platform. It is a known fact that the Linux platform has been the most stable up until now for ESP, but now this might shift as Windows versions will get the major focus in QA and patching.

Let us not hope that the Linux, AIX and Solaris versions will be discontinued. I don’t expect that to happen in the short term, as the press release clearly states that they will be supported, and also this blog post by MS’s Kirk Koenigsbauer in the Sharepoint division states that “We’re making a pragmatic decision to continue to delight a core part of FAST’s customer base that has chosen the Linux/UNIX OS. You can bet that we’ll innovate on Windows, too, and over time we hope customers will see .NET as a preferred platform choice“. Let’s hope that lasts for many many years to come, so that history can be re-written in this area.

Congratulations, Microsoft, with an excellent new member organization

Congratulations, John Marcus Lervik with the new role of leading MS’s Enterprise Search Business!

See also official press release and FAST’s customer FAQ