1 Introduction to Oracle Ultra Search

1.2.1 Oracle Ultra Search Crawler

The Oracle Ultra Search crawler is a Java process activated by your Oracle server according to a set schedule. When activated, the crawler spawns a configurable number of processor threads that fetch documents from various data sources and index them using Oracle Text. This index is used for querying. Data sources can be Web sites, database tables, files, mailing lists, Oracle Application Server Portal page groups, or user-defined data sources.

The crawler maps links and analyzes relationships. The crawler schedule is integrated with and driven from the DBMS_JOB queue mechanism. Whenever the crawler encounters embedded, non-HTML documents during the crawling, it uses Oracle Text filters to automatically detect the document type and filter and index the document.

1.2.2 Oracle Ultra Search Backend

The Oracle Ultra Search backend consists of an Oracle Ultra Search repository and Oracle Text. Oracle Text provides the text indexing and search capabilities required to index and query data retrieved from your data sources. The backend is not visible to users; it indexes information from the crawler and serves up the query results.

1.2.3 Oracle Ultra Search Administration Tool

The administration tool is a J2EE-compliant Web application. You can use it to manage Oracle Ultra Search instances, and you can access it from any browser in your intranet. The administration tool is independent from the Oracle Ultra Search query application. Therefore, the administration tool and query application can be hosted on different computers to enhance security and scalability.

1.2.4 Oracle Ultra Search APIs and Sample Applications

Oracle Ultra Search provides the following APIs:

• The query API works with indexed data. The Java API does not impose any HTML rendering elements. The application can completely customize the HTML interface.

• The crawler agent API crawls and indexes proprietary document repositories.

• The email Java API accesses archived emails and is used by the query application to display emails. It can also be used when building your own custom query application.

• The URL rewriter API is used by the crawler to filter and rewrite extracted URL links before they are inserted into the URL queue.

Oracle Ultra Search includes highly functional query applications to query and display search results. The query applications are based on JSP and work with any JSP1.1 compliant engine.

1.3.1 Integration with Oracle Application Server

Although Oracle Ultra Search in the Oracle Application Server is the same product as Oracle Ultra Search in Oracle Collaboration Suite and Oracle Ultra Search in the Oracle Database, there are a couple differences:

• The Oracle Database is not integrated with Oracle Application Server Portal. With Oracle Application Server and Oracle Collaboration Suite, Portal users add powerful multi-repository search to their Portal pages. Oracle Application Server and Oracle Collaboration Suite also have the capability to crawl and make searchable Portal's own repository.

• Oracle Application Server includes a Single Sign-On (SSO) server. SSO users can log on once for all components of the Oracle Application Server product, and the Oracle Ultra Search administrative interface allows user management operations on either database users or SSO users. Authenticated SSO users never see the Oracle Ultra Search logon screen. Instead, they can immediately choose an instance. If the SSO user does not have permissions to manage Oracle Ultra Search (set in the Users Page), then the SSO user receives an error. SSO is available only with the Oracle Identity Management infrastructure.

  
  

  See Also: http://portalstudio.oracle.com

  
  

1.3.2 Extensible Crawler and Crawler Agents

You can define, edit, or delete your own data sources and types in addition to the ones provided. You might implement your own crawler agent to crawl and index a proprietary document repository, such as Lotus Notes or Documentum, which contain their own databases and interfaces. The proprietary repository is called a user-defined data source. The module that enables the crawler to access the data source is called a crawler agent.

1.3.3 Federated Search

Traditionally, Oracle Ultra Search used centralized search to gather data on a regular basis and update one index that cataloged all searchable data. This provided fast searching, but it required that the data source to be crawlable before it could be searched. Oracle Ultra Search now also provides federated search, which allows multiple indexes to perform a single search. Each index can be maintained separately. By querying the data source at search-time, search results are always the latest results. User credentials can be passed to the data source and authenticated by the data source itself. Queries can be processed efficiently using the data's native format. To use federated search, you must deploy an Oracle Ultra Search search adapter, or searchlet, and create an Oracle source. A searchlet is a Java module deployed in the middle tier (inside OC4J) that searches the data in an enterprise information system on behalf of a user. When a user's query is delegated to the searchlet, the searchlet runs the query on behalf of the user. Every searchlet is a JCA 1.0 compliant resource adapter.

1.3.4 Secure Search

Oracle Ultra Search supports secure searches, which return only documents satisfying the search criteria that the search user is allowed to view. For secure searches, each indexed document should be protected by an access control list (ACL). During searches, the ACL is evaluated. If the user performing the search has permission to read the protected document, then the document is returned by the query API. Otherwise, it is not returned.

There are two ways to secure a data source:

• Specify a single ACL for protecting all documents of a data source.

  The administrator specifies the permissions of the single ACL in the Oracle Ultra Search administration tool. The resulting ACL is used to protect all documents belonging to that data source.

• Crawl ACLs from the data source.

  The data source is expected to provide the ACL together with the document. This lets each document be protected by its own unique ACL.

Oracle Ultra Search performs ACL duplicate detection. This means that if a crawled document's ACL already exists in the Oracle Ultra Search system, then the existing ACL is used to protect the document, instead of creating a new ACL within Oracle Ultra Search. This policy reduces storage space and increases performance.

Oracle Ultra Search supports only a single LDAP domain. The LDAP users and groups specified in the ACL must belong to the same LDAP domain.

Searches run against a secure-search enabled Oracle Ultra Search instance are slower than those run against a non secure-search enabled instance. This is because each candidate hit could require an ACL evaluation. ACLs are evaluated natively by the Oracle server for optimum performance. Nevertheless, this is a finite time. Therefore, the time taken to return hits in a secure search varies depending on the number ACL evaluations that must be made.

1.3.5 Sample Query Applications

Oracle Ultra Search includes fully functional sample query applications to query and display search results. The sample query applications include a sample search portlet. The sample Oracle Ultra Search portlet demonstrates how to write a search portlet for use in Oracle Application Server Portal. This same portlet is installed as a feature of the Oracle Application Server Portal product.

1.3.6 Sample Search Portlet

Oracle Ultra Search provides a search portlet that can be embedded in Oracle Application Server Portal pages. It is implemented as a JavaServer Page application.

The Oracle Ultra Search search portlet supports most of the functionality provided by the Query API Complete Sample application.

1.3.7 Query API

Oracle Ultra Search offers a flexible query API to incorporate search functionality to your sites. The query API includes the following functionality:

• Three attribute types: string, number, and date

• Multivalued attributes

• Display name support for attributes, attribute list of values (LOV), and data groups

• Document relevancy boosting

• Arbitrary grouping of attribute query operator using operators (AND, OR), with control over attribute operator evaluation order

• Selection of metadata returned in query result

  
  

  See Also: "Oracle Ultra Search Query API" Oracle Ultra Search Java API Reference

  
  

1.3.8 URL Rewrite

The URL rewriter is a user-supplied Java module for implementing the Oracle Ultra Search UrlRewriter interface. It is used by the crawler to filter or rewrite extracted URL links before they are put into the URL queue. URL filtering removes unwanted links, and ULR rewriting transforms the URL link. This transformation is necessary when access URLs are used.

1.3.9 Robots Exclusions

Robots exclusion lets you control which parts of your sites can be visited by robots. If robots exclusion is enabled (default), then the Web crawler traverses the pages based on the access policy specified in the Web server robots.txt file. For example, when a robot visits http://www.foobar.com/, it checks for http://www.foobar.com/robots.txt. If it finds it, the crawler analyzes its contents to see if it is allowed to retrieve the document. If you own the Web sites, then you can disable robots exclusions. However, when crawling other Web sites, you should always comply with robots.txt by enabling robots exclusion.

1.3.10 Display URL Support

When gathering information from a database-based Web application, Oracle Ultra Search lets you specify a URL to display the data retrieved on a browser, rendered by a screen of a Web application corresponding to the data in the database tables. The URL points to a screen in the Web application corresponding to the data in the database. This is available for table data sources, file data sources, and user-defined data sources.

1.3.11 Document and Search Attributes

Document attributes, or metadata, describe the properties of a document. Each data source has its own set of document attributes. The value is retrieved during the crawling process and then mapped to one of the search attributes and stored and indexed in the database. This lets you query documents based on their attributes. Document attributes in different data sources can be mapped to the same search attribute. Therefore, you can query documents from multiple data sources based on the same search attribute.

The list of values (LOV) for a search attribute can help you specify a search query. If attribute LOV is available, then the crawler registers the LOV definition, which includes attribute value, attribute value display name, and its translation.

1.3.12 Metadata Loader

Oracle Ultra Search provides a command-line tool to load metadata into an Oracle Ultra Search database. If you have a large amount of data, this is probably faster than using the HTML-based administration tool. The loader tool supports the following types of metadata:

• Search attribute list of values (LOVs) and display names

• Document relevance boosting and document loading

  
  

  See Also: Appendix A, " Loading Metadata into Oracle Ultra Search"

  
  

1.3.13 Document Relevancy Boosting

You can override the search results and influence the order that documents are ranked in the query result list with document relevancy boosting. This can promote important documents to higher scores and make them easier to find.

Relevancy boosting assigns a score to a document for specific queries entered by the search user.

Relevancy boosting has the following limitations:

• Comparison of the user's query against the boosted queries uses exact string match. This means that the comparison is case-sensitive and space-aware. Therefore, a document with a boosted score for "Ultra Search" is not boosted when you enter "ultrasearch".

• Relevancy boosting requires that the query application pass in the search term in the Query API getResult() method call. The sample applications are designed to pass the basic search terms as the boost term. Advanced search criteria based on search attributes are ignored.

  
  

  See Also: "Queries Page"

  
  

1.3.14 Data Harvesting Mode

For initial planning purposes, you might want the crawler to collect URLs without indexing. After crawling is done, you can examine document URLs and status, remove unwanted documents, and start indexing. You can update the crawling mode to the following:

• Automatically accept all URLs for indexing

• Examine URLs before indexing

• Index only

  
  

  See Also: "Schedules Page"

  
  

1.3.15 Instance Snapshot Support

You can create a read-only snapshot of a master Oracle Ultra Search instance. This is useful for query processing or for a backup. You can also make a snapshot instance updatable. This is useful when the master instance is corrupted and you want to use a snapshot as a new master instance.

1.3.16 Integration with Oracle Internet Directory

Oracle Internet Directory (OID) is Oracle's native LDAP v3-compliant directory service, built as an application on top of the Oracle Database. Oracle Internet Directory hosts the Oracle common identity. All Oracle Web-based products integrate with the SSO server for single sign-on support.

1.3.17 Single Sign-On Authentication

The Oracle Ultra Search administration tool supports three modes of logging on, depending on the type of user. You can log on as:

• A single sign-on (SSO) user managed in the Oracle Internet Directory (OID) and authenticated with the SSO server

• A local database schema user in the Oracle Ultra Search database (non-SS0 mode)

• A Portal user

• An Enterprise Manager user

  
  

  Note: Single Sign-On (SSO) is available only with the Oracle Identity Management infrastructure.

  
  
  
  

  See Also: "Logging On to Oracle Ultra Search"

  
  

1.3.18 Query Syntax Expansion

Oracle Ultra Search translates each user query into a database query. This process is called query syntax expansion. The expansion logic determines relevancy, recall of the search results. The Oracle Ultra Search default expansion boosts the relevancy of those documents that matches the user's query as a part of their title. The query syntax expansion can be customized with the query API.