7 Understanding the Oracle Ultra Search Crawler and Data Sources

7.3.1 Using Crawler Agents

If you are defining a user-defined data source to crawl and index a proprietary document repository or management system, such as Lotus Notes or Documentum, then you must implement a crawler agent as a Java class. The agent collects document URLs and associated metadata from the proprietary document source and returns the information to the Oracle Ultra Search crawler, which enqueues it for later crawling. For more information on defining a new data source type, see the User-Defined sub-tab in Sources page in the administration tool.

7.3.2 Synchronizing Data Sources

You can create synchronization schedules with one or more data sources attached to it. Synchronization schedules define the frequency at which the Oracle Ultra Search index is kept up to date with existing information in the associated data sources. To define a synchronization schedule, use the Sources page in the administration tool.

7.3.3 Display URL and Access URL

For some applications, for security reasons, the URL crawled is different from the one seen by the end user. For example, crawling on an internal Web site inside a firewall might be done without security checking, but when queried by the end user, a corresponding mirror URL outside the firewall must be used. This mirror URL is called the display URL.

By default, the display URL is treated as the access URL unless a separate access URL is provided. The display URL must be unique in a data source; so two different access URLs cannot have the same display URL.

7.5.1 Queuing and Caching Documents

Figure 7-1 and Figure 7-2 illustrate an instance of the crawling cycle in a sequence of nine steps. The example uses a Web data source, although the crawler can also crawl other data source types.

Figure 7-1 illustrates how the crawler and its crawling threads are activated. It also shows how the crawler queues hypertext links to control its navigation. This figure corresponds to Steps 1 to 5.

Figure 7-2 illustrates how the crawler caches Web pages. This figure correspond to Steps 6 to 8.

The steps are the following:

1. Oracle spawns the crawler according to the schedule you specify with the administration tool. When crawling is initiated for the first time, the URL queue is populated with the seed URLs. Figure 7-1.

2. Crawler initiates multiple crawling threads.

3. Crawler thread removes the next URL in the queue.

4. Crawler thread fetches the document from the Web. The document is usually an HTML file containing text and hypertext links.

5. Crawler thread scans the HTML file for hypertext links and inserts new links into the URL queue. Duplicate links already in the document table are discarded.

6. Crawler caches the HTML file in the local file system. Figure 7-2.

7. Crawler registers URL in the document table.

8. Crawler thread starts over by repeating Step 3.

Fetching a document, as shown in Step 4, can be time-consuming because of network traffic or slow Web sites. For maximum throughput, multiple threads fetch pages at any given time.

Figure 7-1 Queuing URLs

Description of the illustration isrch005.gif

Figure 7-2 Caching URLs

Description of the illustration isrch006.gif

7.5.2 Indexing Documents

When the file system cache is full (default maximum size is 20 megabytes), document caching stops and indexing begins. In this phase, Oracle Ultra Search augments the Oracle Text index using the cached files referred to by the document table. See Figure 7-3.

Figure 7-3 Indexing Documents

Description of the illustration isrch004.gif

7.7.1 URL Boundary Rule

The URL boundary rule consists of domain rules and path rules. A domain rule specifies the set of Web sites allowed using a host name prefix or suffix. A path rule specifies the URL file path allowed or disallowed for a particular host. You can specify an inclusion or exclusion rule for both a domain rule and a path rule. Exclusion rules always override inclusion rules. Path rules are always host-specific.

For example, an inclusion domain ending with oracle.com limits the Oracle Oracle Ultra Search crawler to hosts belonging to Oracle worldwide. Anything ending with oracle.com is crawled, but http://www.oracle.com.tw is not crawled. If you change the inclusion domain to someurl.com with a new seed http://www.someurl.com, then all oracle.com URLs are dropped by the crawler.

An exclusion domain uk.oracle.com prevents the crawler from crawling Oracle hosts in the United Kingdom. You can also include or exclude Web sites with a specific port. (By default, all ports are crawled.) You can have port inclusion or port exclusion rules for a specific host, but not both.

All URLs must pass domain rules before being checked for path rules. Path rules let you further restrict the crawling space. Path rules are host-specific, but you can specify more than one path rule for each host. For example, on the same host, you can include the path /host/doc and exclude the path /host/doc/private. Note that path rules are prefix-based.

Regular expression-based domain and path rules are not supported in the current release.

The following rules restrict the crawler to only crawl www.oracle.com and otn.oracle.com. Furthermore, only URLs under /products/database/ and /products/ias/ but not under /products/ias/web_cache/ will be crawled.

Domain inclusion: www.oracle.com
Domain inclusion: otn.oracle.com
Path inclusion for otn.oracle.com:         /products/database/
                                           /products/ias/
Path exclusion for otn.oracle.com:         /products/ias/web_cache/

7.7.2 robots.txt Protocol and robots META Tag

The robots.txt protocol is the webmaster's path rule for any spider or crawler that visits his or her Web site. (It is described in the document "A Standard for Robot Exclusion" at http://www.robotstxt.org/wc/norobots.html.) The following example /robots.txt file specifies that no robots should visit any URL starting with /cyberworld/map/ or /tmp/, or /foo.html:

# robots.txt for http://www.acme.com/
 
User-agent: *
Disallow: /cyberworld/map/ 
Disallow: /tmp/ 
Disallow: /foo.html

By default, the Oracle Ultra Search crawler observes the robots.txt protocol, but it also allow the user to override it. If the Web site is under the user's control, a specific robots rule can be tailored for the crawler by specifying the Ultra Search crawler agent name "User-agent: Oracle Ultra Search." For example,

User-agent: Oracle Ultra Search
 
Disallow: /tmp/

The robots META tag instructs the crawler whether to index a Web page or follow the links within it. It is described in "HTML Author's Guide to the Robots META tag" (http://www.robotstxt.org/wc/meta-user.html).

7.7.3 Crawling Depth

Crawling depth controls how deep the crawler follows a link starting from the given seed URL. Since crawling is multi-threaded, this is not a deterministic control, as there may be different routes to a particular page.

The crawling depth limit applies to all Web sites in a given Web data source.

7.7.4 URL Rewriter

You implement the URL rewriter API as a Java class to perform link filtering or rewriting. Extracted links within a crawled Web page are passed to this module for checking. This enables ultimate control over which links extracted from a Web page are allowed and which ones should be discard. See "Oracle Ultra Search URL Rewriter API" for details.

7.7.5 URL Redirection and Boundary Rule Enforcement

With regard to HTTP redirection, earlier Oracle Ultra Search releases (9.0.2, 9.0.3, and 9.2.0.4) applied the same boundary checking to a redirected URL. Thus, a redirected URL would be rejected if it was outside the boundary rule. If the redirected URL was to be crawled, you had to make sure it was covered by the boundary rule.

In 9.2.0.5, iAS 10g, and Oracle Database 10g the redirected URL is always allowed if it is a temporary redirection (HTTP status 302, 307). For permanent redirection (status 301), the redirected URL is still subject to boundary rules.

HTTP meta tag redirection is always checked against boundary rules.