Marc Kupietz¹ · Peter Leinen² · Nils Diewald¹
¹Leibniz-Institute for the German Language (IDS)
²German National Library (DNB)

Towards a Very Large German Academic Corpus

Step 1: Building and Making Available a Corpus of 10,000 Doctoral Dissertations

Workshop on Comparable and Interoperable Corpora
of Academic Texts @CLARIN2024, Barcelona 2024-10-17

Overview

1. Introduction: Background

2. Legal Challenges

3. Data and Methodology

4. Interoperability

5. Comparable Corpora & Cross-Linguistic Research

6. Summary & Conclusions

Work in Progress Timeline

German Reference Corpus DeReKo

IDS corpora of contemporary written German (Kupietz et al. 2010, 2018)

• established in 1964

• aims to serve as an empirical basis for German linguistics

  • for a very broad range of applications

• samples the current use of written German

  • since ~1956

  • is continuously expanded (now ~60 billion words)

• takes a primordial sample approach (Kupietz et al. 2010)

  • invites users to define their own sub-corpora

Stratum sizes in DeReKo

• relative sizes don’t matter

• but sparsely populated strata can hurt

• many of such, but most notably:

  • fiction texts

  • academic texts

Linguistics is in a special situation

• our research data is usually affected by third parties' rights

  • authors, speakers, publishers …

  • all not part of the scientific community (!)

• affected rights:

  • intellectual property rights

  • general personality rights

This will not change fundamentally

Ultimately: collision of fundamental rights

Text and Data Mining Exception

EU directive 2019/790 on copyright and related rights in the Digital Single Market

• overrides licence contracts (unlike other exceptions)

• German transposition allows for:

  • sharing corpora within research projects

  • storing them as long as necessary for the purposes of scientific research or the verification of scientific findings

  • remuneration to collecting societies no longer required

• however:

  • corpus data still can only be shared within a »specifically delimited circle of persons« and not within the whole linguistic community

➞ Open-Data-Models still not applicable

• models like:

  • I make my corpus available to you and in turn, you make your corpus available to me

  • we make all our corpora available since the taxpayers financed them

• lead to nothing:

  • my corpora = your corpora = our corpora = our corpora, which the taxpayer paid = {}

How to Share / Provide Corpora Legally and practically?

1. use copyright exceptions

2. wait for expiration of copyright

3. try to stay below the threshold of originality

4. (conclude and) use licences

Special challenges for academic texts

• (technical challenges)

• relatively high proportion of CC licenses for PhD theses

• apart from that licensing very costly:
  individual agreements with each author required

• licensing with publishers costly

How to Share / Provide Corpora Legally and practically?

1. use copyright exceptions

2. wait for expiration of copyright

3. try to stay below the threshold of originality

4. (conclude and) use licences

5. solve legal problems by infrastructural means

If the data is not allowed to move …

(Gray 2003)

… build some technology

to make it usable from where it is (Kupietz et al. 2010; 2021)

»National Library as Corpus« – »KorAP@DNB«

latest cooperation (IDS+DNB) under this motto

• § 14 DNBG (German National Library Act): Mandatory Digital Deposit

  • publishers in Germany are required by law to submit copies of their published electronic books to DNB

  • in addition the DNB must provide access to the books

  • comparable laws in some other EU countries,
    but digital deposit often not mandatory (see Roudik et al. 2017)

• run a corpus query system at the DNB that provides access to the deposited e-books for linguists and related parties

1ˢᵗ project: German Fiction Corpus DeLiKo@DNB

sample of 26.000 books via KorAP@DNB (last week publicly launch)

• publicly accessible under https://korap.dnb.de/

• 

• no login required

• restriction: only ~40 words per KWIC

Why is a National Library interested in such a cooperation?

(Or: How can I convince my National Library)

• in Germany the DNB is partner in the National Research Data Infrastructure (NFDI) Text+

• making the data hosted at the DNB available to the scientific community as useful as possible is in their own strong interest

• no need to convince them

• in addition:

  • very good connection between DNB and the German Publishers and Booksellers Association

  • top legal expertise available at the DNB, to make things possible that are legally possible

Motivations of the DNB

• offering users an attractive, expanding and broad range of services for their research

• strengthening the role as a partner to the scientific community

• reaching new groups of users through expanded services

• improving access to data especially for machine analyses

German Academic Corpus DeFoKo@DNB

today's focus

• aim: provide a corresponding academic corpus to the fiction corpus

• start with 10,000 doctoral dissertations

  • advantages:

    • no multi-column layout, article breaks, …

    • homogeneous

Data

• 350,000 dissertations collected by the DNB as part of the DissOnline project

• planned stratified sampling by:

  • domain (DDC top level)

  • year of publication

• to arrive at 10,000 dissertations

• what domains would make sense?

Current Test Sample Composition

• ~100 PhD theses as PDF

• year of publication 2000-2020

Conversion PDF ➞ TEI I5

TEI I5 ➞ KorAP

including tokenization and annotation

Benchmark for converting the 99 diss. test set

1.5 million tokens

Test Corpus accessible via KorAP

http://korap.dnb.de/defako

Expanded annotations

http://korap.dnb.de/defako

Expanded metadata

http://korap.dnb.de/defako

Evaluation & Quality Control

• currently: battery of CI tests

• What are realistic approaches?

  • if you want to do this regularly

  • without founding or staff

  • for ci and unit tests with XSLT

Interoperability on different levels

(Kupietz et al 2022)

Interoperable Domain Classification: DDC

Dewey Decimal Classification system – the library standard

KorAP supports extensible set of Query Languages

(Bingel & Diewald 2015)

European Reference Corpus EuReCo

(Kupietz et al. 2024, 2020, 2017)

• open initiative established in 2012

• re-use existing national and reference corpora

  • (no corpus data is moved)

• define pairwise comparable virtual corpora dynamically

  • based on metadata properties

• integrated corpora:

  • DeReKo , CoRoLa (Cosma & Kupietz 2019), HNC (Váradi 2002)

  • WIP: BNRC (Spassova 2023), NKJP, kielipankki

• CLARIN project proposal planned for 2025

  • decided at EuReCo -workshop@CaC 2023

Aim: Comparable Academic Corpora

• integrate other academic corpora into EuReCo

• maybe following the national library cooperation route

  • however: digital deposit in many countries not mandatory

• also via Conference of European National Librarians (CENL) Dialogue Forum National Libraries as Data

• possible starting point:
  https://www.clarin.eu/resource-families/corpora-academic-texts

Objective: Very Large German Academic Corpus

»National Library as Corpus«

• step 1: build and provide access to a 10,000-doctoral-dissertation corpus

• approach: infrastructural solution to legal obstacles:

  • run KorAP instance at German National Library (DNB) to serve all data from there

• current state:

  • ~100 test PhD theses available via KorAP in 1st version

• next steps:

  • stratified sampling of 10,000 dissertations

  • refine metadata extraction & improve evaluation

  • add other kinds of academic texts

  • integrate multiple backend results in one KorAP UI

References I

References II