2.1 Introduction

Because of its inherent complexity, the analysis of natural language documents is usually not carried out in a single, large, monolithic step. Rather, (computational) linguists deal with natural languages at several layers of description (processing). A common decomposition distinguishes between the consideration of words (the lexical level), the organization of groups of words in sentences as phrases or clauses (the syntactic level), and the meaning that can be ascribed to these entities at the content layer (the semantic level). Given such a division into various descriptions and processing levels, a need for integration arises. Hence, the architecture of natural language systems becomes a crucial issue. It provides the organizational platform to integrate the outcome of many decomposed subprocesses, all being part of the grand picture, which is the content-oriented analysis of natural language. In this chapter, we look at the processing of natural language documents for text mining from this holistic architectural perspective, as well as from the perspective of its decomposition into the major components at the lexical, syntactic, and semantic level. We are only concerned with the English language, since it serves as the international lingua franca for the field of biology.

We start at the lexical level in Section 2.2. First, we deal with the problem of identifying words from a given document stream, which is the tokenization problem (see Section 2.2.1). Once words are determined, lexical variants (which are due to inflection and derivation) are unified through morphological analysis, by...