One of the first widely used search engines was Lexis, launched in 1973. A “searchable database” of legal content, it remains valuable and popular. So too is Nexis, a searchable database of news, which followed a few years later. For many who started their careers before the rise of the Internet, these information services were likely their first exposure to the power of search.
It wasn’t long before search appeared in the enterprise, particularly in publishing. As the Internet emerged and changed everything, it became almost ubiquitous. The “intranet” emerged.
People quickly discovered an unfortunate reality: web search turns out to be a very different challenge than enterprise search. They are just completely different.
On the web:
- Most content is the same – HTML pages
- Limited set of query operations supported
- Great deal of interlinking
- More content all the time, much of it topical
In the enterprise:
- Content is highly variable – huge range of formats/types
- Huge range of query operations
- There is minimal interlinking of data
- Content is actually added relatively slowly, and often in waves
There is also a critical difference in data type. Enterprises depend very heavily on structured data, typically stored in a database in normalized form. Such data is not instantly consumable by a search engine, and preparing structured data for search can require significant work.
One thing both worlds have in common is … change. Inside or outside the enterprise, what people are looking for doesn’t stay in one place. Maintaining search takes a sustained effort. This is well known to technology vendors, but it can be hard for buyers to solve. Thus, search in particular tends to follow a value-destroying five-year cycle of under-investment, disillusionment and finally … replacement.
This whitepaper explores this unfortunate cycle in detail – defining the various phases, what to expect in each, and how to justify the needed investments to avoid it, along guidelines and best practices to ensure your choice of search solution provides value for a decade or more.
Reasons for Deploying Enterprise Search
Enterprises invest in search for a number of reasons. At the business level, the objectives are typically either general and broad or specific to a particular function or business. The intranet, intended for general use by a plurality of employees, is an example of the former. Typical goals for the intranet include:
- Rapid, “up-to-date,” “Google-like” search
- Access to commonly needed information, such as employee handbooks, forms, corporate procedures, training material, etc.
- Access to specific topics, projects or organizational unit information, as authorized
- A rich user interface that allows rating, tagging, collection, packaging, sharing and collaboration on search results
Metrics used to manage intranets may include number of searches per day, number of users or sessions per day, number of queries per session, number of pages viewed per query, use of search features (like facets), or number of queries that returned zero results. To the degree an intranet solution does not satisfy some or all users, there may also be specific goals around improvement.
The goals for an enterprise search application, such as a “Customer 360,” are typically more detailed, focused and specific:
- Show auto-complete on customer last name, as soon as possible after the user starts typing
- Present related information, like address, city, and state, along with the names
- Once a name is selected, display everything known about this customer from the following sources: <lengthy list of internal and external, structured and unstructured sources>
- Customer service representatives (CSRs) will be able to find customers within five seconds or less, 99% of the time
The goals for an e-Commerce application driven by search – such as a storefront, or self-service support system – would likely focus on conversion rates, average order value, average revenue per user, or number of exceptions.
Ultimately, all of these goals add up to one thing: user satisfaction. If the relevant community of information consumers can find what they need, across the sources they need, and make use of the information – in both a timely fashion, and over a sustained period of time – then the system is working and delivering value. When this is not the case, users will complain, productivity will lag behind leaders in the same industry – and presumably the company will seek to close the gaps.
Typically, that gap-closing effort begins with an evaluation.
Evaluating and Implementing Search
Search evaluations typically start with a “Request For Information” (RFI) or other, similar requirements document. It is ultimately shared with one or more vendors, who respond in various levels of detail. The buyer team then reviews the RFI and scores it. From there, a “short list” of viable vendors is identified, and more detailed analysis begins.
To the degree the requirements are really a single application scope, and a vendor has that application available, the “short list” may be the end of the evaluation. This is not uncommon in very specific niches like eDiscovery or Customer Self-Service. For broader use cases like Intranet, however, it is usually impossible to complete the evaluation because the potential solutions are “platforms” that require some degree of implementation.
The final part of such evaluations is typically a “Proof of Concept” (POC) or “Value” (POV). One or more vendors are selected and given specific tasks and evaluation criteria, along with data to load. They construct a solution and demonstrate it; some even leave it behind so the users can evaluate it over a period of time.
From this process emerges a set of scores and perceptions – and hopefully at least one viable “winner” – a vendor the company is comfortable working with to implement the solution. Some period of time later, that solution will most likely be deployed – with much fanfare.
Figure 1: The Five-Year Replacement Cycle
Entering The Cycle
The solution’s first year of post-evaluation/deployment operation is the so-called “honeymoon” phase. The implementation has hit the target. The metrics used to track the system look excellent, especially those identified as the business goals early on. The new search engine will garner positive reception for at least a year. The effect will be particularly pronounced if the solution evolves over time – adding new sources or search features, for example.
Success in the honeymoon period is actually a big driver of the cycle. The initial good results, and especially good metrics, lead to complacency. Key resources and attention needed to keep search working is typically re-assigned or de-prioritized in favor of other activities. The search manager, who was instrumental in building and moving on the search roadmap, often gets moved on to a newer, more important project. The roadmap for functionality is forgotten. Issues begin to linger. New sources take longer, and building new front-ends becomes slow – at best.
After operating without sufficient investment for a year or two, metrics, to the degree they are still analyzed, begin to reflect frustrated users and a need for maintenance.
It is quite common for the company to engage the vendor at this point. They are invariably interested in helping, but also nervous – vendors know that such an engagement can be challenging. They may want to be paid for the engagement. Buyers are likely to object to this, feeling that the product is not working as it should. The vendor may then feel obliged to point out that the training, staffing, and ongoing maintenance they recommended years ago have not been done – and that they cannot be expected to instantly rectify a situation that has taken years to develop.
From here, the solution typically limps along for another year – at most. It may continue to please some users, but there will also be a steady decline in the number of searches as some give up. New search solutions, with newer software, will soon begin to appear. The implementers of the new solutions will report success, and this message will spread throughout the company.
At this point the buyer is likely to engage the vendor in a final attempt to rectify the situation. The vendor will continue to be interested in fixing the problem but, after 3-4 years, may well suggest upgrading to their newer version – a significant project – instead of trying to make the old version work better.
This may well be the last straw for some buyers.
The cycle concludes with the ultimate determination by senior management that the search solution has broken and can’t be fixed. A new one is required. A new evaluation process will be authorized. It will yield a new search solution after approximately one year of work. It will then be deployed, and ultimately go through the same cycle… in approximately five years.
Avoiding the Cycle
The reason for this non-optimal cycle is an underlying reality: search is in the business of understanding and interpreting human expression. That reality turns out to be a challenging problem, mainly because it is a constantly evolving target!
Language defies the conformity found in most enterprise data. Highly structured data systems know what to expect and how to model that data, and therefore, they have a tendency to keep working as long as conforming data continues to arrive. This is not the case with search. In a five-year cycle, those authoring content – inside or outside the enterprise – will adopt new terminology.
Keeping track of these types of shifts and accounting for them is critical to keeping search “working” and to avoiding the five-year replacement cycle. Failing to do so is the main driver of the five-year cycle.
To maximize the value of your search investment, Knowledgent recommends observing a range of best practices, including:
- Quantifying the value of search
- Identify the technology strategy
- Obtain necessary skills during the evaluation
- Make required investments
- Track metrics that guide priorities
- Follow a process for managing search
The following sections go into more detail about each of these.
The Value of Search
What is the dollar value that search creates in your enterprise? This is an important question because it helps determine the level of investment to achieve success.
Domain-specific search solutions should be evaluated using different metrics. E-commerce search, for example, should be valued in terms of conversion rate, average order value and/or average revenue per user. Self-service systems should the customer retention percentage, mean-time-to-resolve, or percentage not handled. These metrics can be translated into timesavings and associated costs, and again, compared against the cost of the solution.
There are also important qualitative measures you can use to determine the value and ROI of search in your organization. Surveys can quickly help identify fundamental gaps in content or capability. (Be sure to collect enterprise demographics, too. It is important to understand the needs of specific teams.)
An even better approach is to ask users to rate the results produced by the search engine. Simply capturing a basic “thumbs up” or “thumbs down” rating can quickly identify weak spots.
Ultimately, some combination of qualitative and quantitative methods will yield an estimate of search, and the value it has to the company.
The next question to answer is equally important: Should the company build or buy the search solution?
Ten years ago, it would have been easy to answer: buy it. That was the only realistic option. Today, however, we live in the age of search. The core problems of indexing and query resolution have been commoditized. Open source engines SOLR and Elastic, built on Apache Lucene, dominate the landscape. Large ecosystems of companies have sprung up around open-source search, providing the needed components and expertise that make the difference between basic search and a great application.
Free/Open Source Software (FOSS) is now completely viable for search. It is already embedded in a large range of applications – from salesforce.com to foundational new platforms like Cloudera’s Distribution of Hadoop (CDH).
That leaves the enterprise itself as the question. Does the enterprise develop software? Does develop with open source? In particular, does it track and knit together changes across multiple projects?
Figure 2: Flow Chart: Build or Buy?
If your organization answers these questions affirmatively, and search is valuable,
Knowledgent recommends building the solution. Over time, the organization will save money and have more satisfied users, with potentially extreme ROI.
If search is valuable but the organization does not build back-end applications, consider working with a visionary company. Visionary companies are typically the most innovative, most responsive and easy to work with. It is usually possible to achieve a very high ROI with visionaries.
Otherwise, consider buying a leading solution. Leaders are rarely at the cutting-edge of capability, but their product maturity, support eco-system, and relatively low price points make them excellent choices. A solid ROI can absolutely be obtained without burdensome overhead.
The key in selecting build or buy as a search strategy is to ignore trends and focus on your own internal culture and capabilities. Just because another company is building search, or buying it, is no reason to do the same!
Regardless of the technology strategy, it is critical to have at least one person with search and related technology expertise involved while evaluating solution options.
If the approach chosen is to build the solution, expertise is critical for designing the overall solution so that it fits in the existing environment and for making choices about which capabilities to integrate (and the order in which to do so). If the approach is to buy a solution, expertise will be endlessly useful in working with the vendors. Understanding the difference between stemming and lemmatization, for example, can be very important.
Keeping expertise engaged throughout the deployment of the solution, and thereafter to help steer and evolve the solution is also highly recommended.
Even the perfect solution – carefully evaluated by experts and determined to be in alignment with the company’s goals and capabilities – will not automatically sustain itself.
Indeed, if building when one should buy, or the reverse, is responsible for half of the five-year replacement cycle, then failure to invest over time and to sustain the solution is the other half. To ensure that the solution delivers value over time, it is essential that the following minimum capabilities are present within the organization at all times:
- Adding/removing content
- Securing content
- Validating content security
- Validating content access
- Adding/removing front-ends
How these things are done is less important than their availability. They simply need to be available via a visible, reasonable process, and completed in a timely fashion to make the solution a success.
Another critical part of sustaining the search solution is related to human capital. The search team should include at minimum a manager who can analyze the system and help prioritize demand. More precisely, for companies with 1,000 or more users of search, Knowledgent recommends allocating two full-time employees: one to manage the solution, one to lead implementations. Additional developers, analysts, QA engineers, and other roles may be added to the team on a per-project basis.
Another key to sustaining the search solution is being able to manage it using metrics. Without metrics, users will fall back on anecdotes about good or bad search results for a specific query. Knowledgent recommends the formal, regular collection of metrics, including the following:
- Number of front-ends/UIs
- Number of data sources
- Time since last update to search software
- Time to last update of taxonomy, ontology or other text analytic modules
- Number of users or sessions per day
- Users/sessions per day, per front-end/UI or application
- Queries per day
- Queries per day, per user/front-end/app
- Queries that returned zero results
- Queries that return an error
- Query performance (median, percentiles)
- Queries that were redirected via business logic
- Time spent viewing result page
- Number of result pages viewed, per query
- Surveys (traditional or on result page)
- Number of search enhancement requests
- Time to deliver search enhancement requests
- Number of unfulfilled search enhancement requests
- Number of security incidents
These metrics are useful because they can be used to identify issues and drive changes in priorities to address them. For example, if departments are complaining about not being able to access an important source of data, the trend in number of data sources is flat, and the time to deliver search enhancement requests is trending towards increasing, a case can be made that more development resources are required to keep pace.
If the number of user sessions or queries per session drops significantly after a major update, the odds are high that something has changed – for the worse, and people are giving up on the system, or for the better, and they are querying less often because they are finding what they need more rapidly. Surveys can be very critical in pinpointing the good or bad reality that is sometimes hard to tease out of usage reports.
Almost as important as the people who manage the solution is the process by which they manage it.
Knowledgent recommends creating an operating model for search that defines:
- RACI matrix
- Regular tempo for meeting
- Information retention policy for log files and other search artifacts
- Process for onboarding and terminating users
- Process for securely onboarding new information sources
- Process for collecting, validating, and analyzing search-related metrics
- Process for collecting, sorting, and prioritizing change requests
- Process for prioritizing, scheduling, testing, and rolling out search updates
- Process for communicating decisions
- Service level agreement (SLA) around adding new sources and front-ends
- Protocol for handling search emergencies
- Protocol for handling security breaches, eDiscovery requests
A well-developed and battle-tested operating model will help to ensure that search is properly managed and maintained.
The five-year replacement cycle for search can be frustrating and cost intensive for enterprises. From the implementation of search to the need for a replacement, the cycle undermines both the enterprise’s initial investment of time and resources as well as the buy-in leveraged to implement the solution originally.
In Knowledgent’s experience, there are methods by which enterprises can limit the damage caused by this cycle. By quantifying the value of search, obtaining the necessary skills during the search evaluation, tracking metrics to guide priorities, following a process for managing search, and other best practices, enterprises can break out of the cycle and even avoid it entirely.