Book cover © Amazon & Wrox

Over due by several years, Wrox just published a book about Microsoft Enterprise Search, including the different FAST flavours. Bravo!

You can ask how all the users of FAST technology could have managed for so many years without some public source of learning the products. Up until now FAST/MS and their partners have been the sole source of learning FAST Search [1]. Now, we’re part of that eco-system and may have profited on the lack of material available, but that’s another story.

The book is written by Jeff Fried (Ex-FAST), Mark Bennett, Natalya Voskresenskaya and Miles Kehoe and covers the chapters

1. What is Enterprise Search
2. Developing a strategy – the business process of search
3. Overview of Microsoft Enterprise search products
4. Search within Sharepoint 2010
5. FAST Search within Sharepoint 2010
6. Customizing search with Sharepoint 2010
7. Introduction to FAST ESP
8. Customnization and deployment of FAST ESP 5.x
9. Advanced topics
10. Enterprise search is social search
11. Search and business intelligence
12. The future of search

I have not read the book yet – but I have bought it to my Kindle and will flip though it on my iPad if time allows. I could of course, humble as I am, have authored much of this book myself, but maybe I can learn a thing or two as well

Click the book cover to go to Amazon.com’s book page.

[1]: Except from http://www.fastforum.info/ (owner) and http://fastesphelp.com/ (by Anand Kumar Pandey)

As a part of that planning process, we have decided that in order to deliver more innovation per release in the future, the 2010 products will be the last to include a search core that runs on Linux and UNIX.

This should not be a shock to anyone, and I hinted to this danger in my blog post about the acquisition:

Let us not hope that the Linux, AIX and Solaris versions will be discontinued. I don’t expect that to happen in the short term, as the press release clearly states that they will be supported…

I guessed back then that Linux support would continue for another five years, but it turned out to be less than two While it is an expected and understandable, I think we all hoped that the press release following the acquisition hinted for a new, more open policy from the software giant, but that turned out to be too optimistic. But existing customers are not left completely alone; Olstad continues:

We will support ESP 5.3—the cross-platform search core in the 2010 products—for 10 years as per our standard support policy (5 years mainstream support and 5 years extended support).  Non-Windows customers on ESP 5.3 can continue running their core on Linux and UNIX and add Windows-only innovations or cloud-based services by using a mixed-platform architecture.

Many search consultant with experience from FAST ESP have experienced that the product is more capable under Linux than under Windows when you start pushing the boundaries. ESP is also easier to manage on a Linux platform due to SSH and friends. But that is not the major selling point anyway.

What this will mean for the existing and potential future customers is hard to predict. Obviously some organizations, especially smaller ones, demand Linux only in their data centers – these will start looking for another vendor.  Some of the bigger companies already have mixed environments and will easily be able to migrate, and may be happy.

Last but not least, the hint of cloud-support from the blog post opens another opportunity for existing customers running on Linux: Outsource the whole search hosting to Microsoft and don’t worry about the servers or OS at all. This may be the best option for many? Expect to see more references to hosted search going forward!

Let’s compare:

From a decision engine I’d expect to get an overview of possible travel options, such as Air, Bus, Train. This however is just the normal bla-bla. Let’s try the same on Google:

At least the top two results on Google are about flights between the two cities, which is better.

Let’s try to get decision help in purchasing an iPhone. From some more searching, it seems as if it is unfair to do the comparison on the Norwegian version of Bing, as it does not contain ads or related searches, so let’s go for the English. Search is “buy iPhone”:

That’s a bit better:

Here, both Bing and Google at least show both relevant ads and iPhone links. But decision engine????

The shopping support does not show up until I try to find it, searching for “casio exilim” – the first link is “Shop for Casio Exilim”, and clicking that takes me to a nice result set with photos of the models, reviews and price. Not very impressive, as Google has had this with Froogle for many years.

Bing seems not to be the revolusion Microsoft promised, so time will show if it becomes a hit or not.

Office 14

The main news is that ESP will be included in Office 14, to serve as the advanced/extended search server of Sharepoint. It has long been a problem that the Sharepoint Search Server can only handle about 50 million documents, and by offering the FAST-based extension, this will be extended to almost infinite number of docs, as well as enabling some features which Sharepoint did not have as well (faceted navigation being one of them). The product will be more limited than the full-fledged ESP in that you cannot tune as many parameters. Much of the middleware and administrative components of ESP are stipped out and replaced with Windows/Sharepoint/MSSQL components, for a tighter integration.

But Office14 will not ship before 2010, so as a gap-filler Microsoft will sell ESP for Sharepoint for as little as USD 25.000 per server, which is a fraction of the typical license cost for such a system. This product is basically the same as today’s ESP, but if you intend to upgrade to Office 14 Extended Search, you better not use all the features of ESP, but stick to the recommended customization options which are compatible with the coming Office 14 search.

FAST Search for Internet Business

The second product announced is “FAST Search for Internet Business”, intended to fill the needs of the typical existing FAST customer within site search or e-commerce. Even the Linux versions of this product will be developed and maintained alongside the Windows versions.

A good question is how MS/FAST is going to maintain all these code bases going forward. I’d expect a consolidation sooner or later, and perhaps also an end-of-life announcement for the Linux platform support within the next 5 years. But that will only be speculations anyway

Today, the deal where Microsoft buys FAST, was completed. That means that the Norwegian search engine vendor Fast Search & Transfer is now a fully owned subsidiary of Microsoft.

The FAST ESP product will continue to be offered on all current platforms, and the FAST sales and tech organization continues to operate almost as before, so customers and users will not experience any noise around this transaction.

FAST, when under the MS umbrella, will of course increase focus within the MS Office Sharepoint segment, and will together with MS engineers make an even smoother packaging of the technologies to new and existing customers of high-end Sharepoint sites with large data volumes.

Expect to see continued innovation from FAST in the years to come, and expect also to see a shift towards stronger support for the Windows platform. It is a known fact that the Linux platform has been the most stable up until now for ESP, but now this might shift as Windows versions will get the major focus in QA and patching.

Let us not hope that the Linux, AIX and Solaris versions will be discontinued. I don’t expect that to happen in the short term, as the press release clearly states that they will be supported, and also this blog post by MS’s Kirk Koenigsbauer in the Sharepoint division states that “We’re making a pragmatic decision to continue to delight a core part of FAST’s customer base that has chosen the Linux/UNIX OS. You can bet that we’ll innovate on Windows, too, and over time we hope customers will see .NET as a preferred platform choice“. Let’s hope that lasts for many many years to come, so that history can be re-written in this area.

Congratulations, Microsoft, with an excellent new member organization

Congratulations, John Marcus Lervik with the new role of leading MS’s Enterprise Search Business!

See also official press release and FAST’s customer FAQ

As of feb 8th it is official, Microsoft has aquired 97,2% of FAST, and can now legally force the buyout of the rest. Microsoft have been very active on the acquisition front lately, also trying a hostile takeover of Yahoo to gain a share of the global online advertising market which Google currently dominates. There is definitely a cosolidation in the search market towards fewer, bigger players.

With the FAST ESP technology, Microsoft will be able to compete better in the Enterprise search space. It is strange how come the world’s largest software firm does not manage to be innovative themselves, but have to buy other companies all the time to be able to really compete. It will be interesting to observe MS’s long-term roadmap for the FAST technology.

For Search, there are a few players picking up speed that you should be aware of:

OSS players a threat?

You probably know the search library Lucene from Apache Foundation. It provides a low-level library for simple indexing needs in your application. Then you have Apache Nutch, which adds a crawler, document converters, search UI etc to Lucene to make it an intranet/internet search engine. A pretty new project is Apache Solr, which is focusing more on the enterprise search side, allowing large-scale search indexes and faceted search. See links at end of article for more examples.

Now, what about enterprise search? Will commercial players like Google, Fast Search & Transfer (now Microsoft), Autonomy, and IBM face any competition by the Open Source products? The short answer is no. Not for the top-end market where search really is your business.

However, the products mentioned above can prove to be very attractive alternatives for the lowe-end market, for companies that need a simple quick search feature for their application, or a site search for their web page(s), a simple product search for their product catalogue or similar. I believe that already now many of those needs can be fulfilled by Open Source engines, and in the future we’ll also see more companies dedicated to installing and supporting such installations, getting their revenues from consulting, not licenses.

Note that we have not mentioned Free “down-scaled” versions of commercial products, such as the IBM Omnifind Yahoo! edition. This could be an excellent alternative for some needs, but it is not open source, so it is not mentioned here.

So keep your eyes wide open! Ask us if you need advice on how to handle your search needs.

Links

• http://infoaccess.wiki.zoho.com/Enterprise-Search-Engine-Vendors-in-2007.html (Impressive one-page overview of Enterprise Search players including OSS)
• http://lucene.apache.org/nutch/ (Apache Nutch)
• http://lucene.apache.org/solr/ (Apache Solr)
• http://www.opensourcesearch.org/ (Another list of OSS search)
• http://omnifind.ibm.yahoo.net/ IBM Omnifind Yoahoo! edition