Marc Kupietz, Nils Diewald, Eliza Margaretha, Helge Stallkamp & Franck Bodmer
Leibniz-Institute for the German Language (IDS)

The Corpus Analysis Platform KorAP

its Philosophy, Features, Pros & Cons

Journée d'étude sur les outils d’exploration de corpus numériques, Paris 2022-06-17

Overview

1. Background, History & Philosophy

2. KorAP Goals & Approaches

3. Using KorAP

4. Using KorAP via Client Libraries

5. KorAP in contrastive Linguistics

6. Installing your own KorAP

7. Summary & Outlook

Leibniz-Institut für Deutsche Sprache (IDS)

• central scientific institution for the documentation and research of the German language in the present and recent history

• founded in 1964

• is one of the 96 institutes of the Leibniz Association

  • financed jointly by the federal government and the state of Baden-Württemberg

• 227 employees (105 researchers + students + administration)

  • (10 of which in the Corpus Linguistics programme area)

• closely cooperates with Mannheim and Heidelberg University

Leibniz-Institut für Deutsche Sprache (IDS)

R5-building

Corpora and analysis tools at the IDS

long tradition (Teubert & Belica 2014)

1964

start of first corpus building projects

1969

1st first corpus published: Mannheimer Korpus I

• 2.2M words on punch cards

1983

1st query software REFER (Brückner 1989)

1992

1st analysis software online: COSMAS I

• (al-Wadi 1994) IDS-Corpora: ca. 20M. words

German Reference Corpus DeReKo

IDS corpora of contemporary written German (Kupietz et al. 2010, 2018)

• construction started in 1964

• aims to serve as an empirical basis for German linguistics

• samples the current use of written German

  • since ~1956

  • is continually expanded

• covers a broad range of text types

• legally compliant through > 200 licence agreements

Primordial Sample Design

Most distinctive feature of DeReKo (Kupietz et al. 2010)

• unlike other reference corpora, DeReKo does not strive for »balance«

  • because »balance« like »representativeness« depends on the research question and the targeted language domain

• researchers themselves should be able to draw stratified sub-samples (»virtual corpora«) from DeReKo

  • that are as representative as possible wrt. to their targeted language domain and research question

Distinction between Observation and Interpretation

Second most distinctive design principle (Belica et al. 2011)

• e.g.: linguistic annotations are just interpretations

• in case of interpretations, allow multiple ›opinions‹

DeReKo-growth since 2000

One more distinctive feature: Size

Big pile of corpus

on its own, not very useful

not readily interpretable linguistically

legally restricted, high-dimensional, opaque structured, ...

If the data cannot move …

… pave ways to put the computation near the data (Gray 2003, Kupietz et. al 2010)

Research Tools that make DeReKo accessible

different tools for different purposes

Main general purpose platform: COSMAS II

http://cosmas2.ids-mannheim.de/ (Bodmer 1994)

• Corpus Search, Management and Analysis System
  

• designed in 1994

  • EU-project MECOLB (with John Sinclair, …)

• has > 40,000 users

• very stable and lots of features, e.g. …

Morphological Assistant

Recursive Collocation Analysis

Which provides multi-unit syntagmatic patterns

Different result aggregations

By text type, topic domain, decade, place of publication, …

However

• COSMAS II already designed in 1994

• underlying database managing gigabytes out of maintenance

• limited to < 17G words (and less with annotations)

• further developments increasingly expensive

What to do?

• market survey in 2009: there is no corpus platform that …

  • supports > 8G words

  • supports multiple, potentially concurrent annotation layers

  • is open source

• conclusion:

  • build a new corpus platform

• funding opportunity:

  • Risk R&D section of the “Leibniz Competition”

Background: Summary

• huge, fast growing corpora

• > 10k users from all areas of linguistics

• methodological fundamentals:

  • user definable virtual corpora

  • theory neutrality and distinction between observation & interpretation

• already working but hardly extensible query platform

• absolute necessity to act IPR- and licence-compliant in order not to lose good reputation and text donors

The KorAP project

• project proposal was successful in the Risk-R&D-section of the Leibniz Competition

• 10 PY knock-on funding 2011 – 2015

• some additional funding (CLARIN, KobRA)

• support by an increasing portion of the COSMAS-II project members

Main project aims

• minimize the risk of total failure:

• start with complementing COSMAS II

  • there will be no single jack of all trades corpus tool!

  • rather many of them complementing each other

  • also: corpus-driven background, but focus on corpus-based features

• core-sustainability for >20 years

• realistic extensibility with important features

• always meet the requirements of a scientific tool

Sub-Goals and approaches

as of the project start (Bański et al 2012)

• support for in principle unlimited amount of primary data and annotation levels

  • by horizontal scalability
    (if the system gets too slow, just add another machine)

• good support of user defined virtual corpora

• easy integration of external developments

  • it's impossible to develop all desired features allone

  • ideally join forces with other corpus hosting institutions

General Architecture

Microservice Approach

Some more Central Ideas

• support multiple query languages to reach out for different user communities

• use “query rewriting” to handle fine-grained authorization

  • more efficient than filtering query hits

  • more transparent (traceable, replicable)

  • backends and frontends can be developed without paying attention to authorization

Open source Project

BSD licensed and available from https://github.com/KorAP and via our gerrit code review

• All contributions and issue reports are extremely welcome!

Low-threshold entry UI

Information-on-Demand- approach (Diewald et al. 2020)

In principle, unlimited corpus size

Unbounded number of annotation layers

For DeReKo currently …

Definition of virtual corpora

➞ stratified sub-sampling based on text metadata (Bański et al 2013)

»Corpus by Match«

Definition of virtual corpora based on query hits (Kupietz et al. 2020)

Multiple Query Languages

(Bingel & Diewald 2015)

Poliqarp+

CQP dialect, developed for Polish National Corpus NKJP (Janus/Przepiórkowski 2007)

• currently best supported QL, very similar to QLs of:

  • IMS Open Corpus Workbench (CWB)

  • (No)SketchEngine

• differences (a.o.):

  • syntax for regular expressions and verbatim strings

  • span handling: CQP: "laufen" </base/s=s>;
    vs. Poliqarp+: endsWith(<base/s=s>, laufen)

• plain CQP in development

• QL discussion spin-off:

  • ISO 24623 Corpus Query Lingua Franca CQLF (Bański et al. 2016, Evert et al. 2020)

Annotation queries supported by search assistant

»Query by Example/Match«

Learning complex queries without prior knowledge (Diewald et al. 2019)

Queries involving text-structural annotations

Example: Search ›Optimierung‹ only in sub-headings

Maximize Recall using concurrent annotations

Search for ›das‹ annotated as relative pronoun only by CoreNLP tools

Maximize Recall using concurrent annotations

Search for ›das‹ annotated as relative pronoun by CoreNLP **OR** tree-tagger

Search on morphological Annotations

Superlative adjective + superlative adjective + noun

• using Query by Example

• starting from »die ehrlichste, anständigste Anlageform« ??

  • incorrectly annotated!

• starting from »der höchste Preis«

• [marmot/m=degree:sup & marmot/p=ADJA]{2} [tt/p=NN]

  • apparently a larger amount of false negatives are to be expected!

Regular Expressions

Verb that starts with ›ver‹ or ›zer‹ and contains an ›ö‹ Umlaut (➞ Cheat-Sheet)

Search on Constituency Annotations

using »spans«: NP that ends with postposded attributive adjective ›pur‹

Sentence Annotation, Negation, Fokus

trying to find a postposed attributive adjective ›satt‹

Search on Dependency Annotations

Verb with „Satz“ as direct object (with Annis QL)

Goal: Make corpora as accessible as possible

by providing Multiple Levels of Access (Kupietz et al. 2022)

0

User Interface

• the web user interface

1

Web Service API

• accessible directly or via client libraries

2

Plugin

• user interface plugins

3

Instance

• independent access by fully customized installations

4

Open Source

• new features by source code contributions

5

Corpus

• direct access to corpus data (without KorAP)

Intended Properties of the levels

the higher the, the cheaper and the more application instances (Kupietz et al. forthcoming)

Level 1: Web Service API

• makes all backend functionality available

  • all query languages

  • virtual corpus definitions

  • complex query expressions

  • UI itself uses the API only

• provides OAuth2 for authorized access to restricted data

• offers unauthorised access to not copyrighted numerical data

• accessible directly (➞ documentation)

• and via client libraries for R and Python (Kupietz et al. 2020b)

API client libraries for R and Python

Application areas

• complex, multipart queries

• applications where reproducibility and replicability is required (with varied query or corpus base)

• providing features that are not yet supported by KorAP's backend or UI, currently e.g.

  • collocation analysis

  • aggregation of search results

• aim: make programmatic use as easy as possible

  • in order to also pick up linguists who never coded before

  • and to support close links between quantitative analysis and qualitative interpretation

Installation of Korap’s R client Library

in RStudio: Tools ➞ Install Packages ➞ RKorAPClient

Installation of Korap’s Python client Library

• first install R client library, then

• pip3 install KorAPClient
  

Basic Example: Frequency query

for ›Shop‹ and ›Boutique‹ in DeReKo newspapers' year slices ➞ data frame

Helper Functions for interactive Plots included

with CIs, and all data points linked to corresponding queries (Kupietz et al. 2017)

Example: Comparison between different VC

relative frequency of ›sozusagen‹ (=so to say) in spoken FOLK corpus vs. DeReKo

Build you own specialised UI with R & Shiny

Contrasting of orthographic variants with ›KoViT‹ (Fischer & Lang 2022)

Good starting point for other applications:

Run and modify one of the demos provided with the R-package

Collocation analysis

recursively applicable, with many options – but still slow

collocationAnalysis(
  "head",
  vc = "corpusSigle=WPE15",      # virtual corpus definition
  lemmatizeNodeQuery = FALSE,
  minOccur = 5,                  # minimum absolute number of observed co-occurrences
  leftContextSize = 3,           # size of the left context window
  rightContextSize = 3,          # size of the right context window
  topCollocatesLimit = 40,       # limit analysis to the n most frequent collocates in the sample
  searchHitsSampleLimit = 10000, # limit the size of the search hits sample
  ignoreCollocateCase = TRUE,
  withinSpan = "base/s=s",       # KorAP span specification for collocations to be searched within
  exactFrequencies = TRUE,       # retrieve exact co-occurrence frequencies
  stopwords = STOPWORDS,         # words not to be considered as collocates
  maxRecurse = 1,                # apply collocation analysis recursively maxRecurse times
  addExamples = T,               # add found instances of collocations
  ...
)
 
 

Collocation analysis of ›head‹

based on English Wikipedia corpus WPE15

Diachronic Collocation Analysis

of ›Umwelt‹ (›environment‹)

Contrastive applications are Very relevant for KorAP

and vice-versa

• to support contrastive corpus linguistics

• to test the idea of dynamically definable, virtual comparable corpora

• to join forces in the development of linguistic research software

• ongoing initiatives:

  • European Reference Corpus EuReCo

  • International Comparable Corpus ICC (Kirk et al. 2017, Čermáková et al. 2021).

  • WikiCorp (Poudat et al. i.p.)

European Reference Corpus EuReCo

(Kupietz et al. 2020, Kupietz et al. 2017b)

• open initiative founded in 2013 by IDS and the academies in Poland, Romania and Hungary.

• goal and idea:

  • many dynamically definable comparable corpora based on existing large corpora, with:

• 

• 

• 

Reference Corpus of Contemporary Romanian CoRoLa

https://korap.racai.ro/ (Cristea et al. 2019)

Comparable DE-RO-Corpus

➞ KorAP auf DRuKoLa-VC ➞ KorAP auf CoRoLa: https://korap.racai.ro/

Hungarian National Corpus HNC

https://korap.nlp.nytud.hu/ (Váradi 2002)

WIP: Polish National Corpus NKJP

currently only internally available http://10.0.10.58:8651/

French (WikiCorp)

only small Wikipedia sample: https://korap.ids-mannheim.de/instance/wikidemo

English

English Wikipedia Corpus WPE15: https://korap.ids-mannheim.de/instance/english

Czech

in development: Czech part of ICC

Current experiments: LVC comparison Romanian-German

using contrastive collocation analysis via RKorAPClient (Kupietz & Trawiński forthcoming)

Preliminary experiences

with contrastive DE-RO analyses using KorAP

• when comparing syntagmatic patterns, the corpus composition itself can play a greater role than comparability

  • dynamically definable corpora are key

• corpus access via scripts seems essential

  • for reproducible results which are essential not only in the iterative corpus refinements

  • but for all analyses consisting of multiple steps

  • for custom-tailored visualizations

• practical comparability through a common tool seems more essential than ›theoretical‹ comparability

Now easy with KorAP-Docker

• simply follow the instructions at
  https://github.com/KorAP/KorAP-Docker

• supports web UI, API, TEI conversion, indexing

• not yet included: authorization-components

KorAP Strengths

• support for very large corpora

• user definable virtual corpora

• arbitrary number of morphosyntax, constituency and dependency annotations

• arbitrary metadata

• multiple, extensible query languages

• query by example

• client libraries for R and Python

• sustainable standing FOSS project

  • with 3.5 RSE FTEs

Not yet completed important Features

• horizontal scalability (Krawfish component)

  • sorting and aggregation of query results

  • sampling function for virtual corpus construction

  • collocation analysis

  • export/analysis of token frequency lists

General weaknesses

• very limited support for spoken or multi-modal corpora

  • no time axis, just one token axis

• not primarily intended for personal installations

  • now very easy using KorAP-Docker

• statistical functions not as easily optimizable as in other corpus tools

  • because of flexible virtual corpora, frequencies and association measures cannot be pre-calculated

Most important current issues

• punctuation, emoticons, emojis are not queryable

• no query/display of possible metadata values

• virtual corpora:

  • no overview over contents

  • no UI for management, export, persistence, …

• import filters for other corpus formats not well supported

  • TEI P5, NoSketchEngine, CWB, TXM
    (in descending order of support)

Next steps

• implement all missing features and fix all issues :)

• focus: horizontal scaling component

  • not so important for corpora with < 10G tokens

  • but essential for DeReKo

  • and code base for all missing quantitative functionalities

References I

References II

References III