The lexicographer's toolkit: building a corpus from scratch

A corpus begins where intuition ends. For any serious dictionary-maker, the leap from collecting interesting words to building a defensible record of how a language actually behaves requires a structured, searchable, citable collection of texts, transcripts, and metadata. In Australia, where communities from Parramatta to Perth juggle dozens of heritage languages alongside English, the demand for transparent linguistic evidence has never been stronger. Researchers at the Australian Institute of Aboriginal and Torres Strait Islander Studies in Canberra have shown that machine-readable corpora make it possible to track rare vocabulary that would otherwise vanish from print.

Building such a resource from the ground up looks intimidating, yet the workflow is approachable once broken into stages. The same principles that guide a lexicographer working on Setswana in southern Africa apply equally to a researcher documenting Murri or Noongar speech in Brisbane or Adelaide. What follows is a practical toolkit for assembling a small, focused corpus without needing a supercomputer or a grant the size of the Melbourne tram network.

Defining the scope and purpose

Before downloading a single webpage, the lexicographer needs a sharp answer to the question of what the corpus is actually for. A dictionary of colloquial Brisbane slang will demand different texts than a historical dictionary of nineteenth-century correspondence held in the State Library of New South Wales. Scope decisions shape everything downstream, from file formats to metadata fields and even the spelling conventions used in transcription.

A useful first step is to write a one-page brief listing the target audience, the time period covered, and the genres to be included. Will the corpus hold newspaper editorials, parliamentary debates, school readers, or transcribed oral histories? Each genre captures a different slice of the lexicon, and a balanced collection blends at least three of them. A corpus aimed at etymological research will lean on archived letters and journals, while one designed to support language preservation in a regional community will prioritise field recordings.

Documenting exclusions matters as much as documenting inclusions. If spam, machine-translated text, and very short social media posts are filtered out, that filter should be recorded openly. Future users, including the version of yourself who returns to the project after a long weekend watching the cricket, will thank you for the clarity.

Sourcing raw material

The hunt for authentic text begins in libraries and archives, but in Australia it often extends into the National Archives in Canberra, the Australian Broadcasting Corporation's transcript archive, and the digital collections maintained by the National Library of Australia in Parkes. Each repository offers different terms of access, and a careful compiler reads the licence agreements before scraping or downloading anything substantial.

For spoken material, fieldwork remains irreplaceable. Consent forms, reliable recording equipment, and a clear protocol for handling culturally sensitive vocabulary are essential. Many Indigenous language projects follow the guidelines published by AIATSIS, which emphasise community ownership of recordings and the right of speakers to withdraw material. When fieldwork is impossible, public-domain radio interviews and podcast transcripts with clear licences can stand in, provided they are tagged as secondary sources.

Written sources include books out of copyright, newspaper archives, government publications, and online fora where users post at length. A useful trick is to start from a seed word and follow hyperlinks outward, building a web of texts that share vocabulary. This snowball method works just as well for tracking borrowed words between languages as it does for hunting slang in an online forum.

Cleaning, normalising, and tokenising

Raw text arrives messy. Scanned pages contain OCR errors, PDFs split words across lines, and CSV exports from survey tools mix data fields unpredictably. The cleaning stage strips out headers, footers, page numbers, and duplicate entries. Regular expressions make short work of repeated patterns such as email addresses, running titles, or navigation menus left in scraped HTML.

Normalisation comes next. Should "colour" and "color" be treated as the same token, or kept distinct? Should abbreviations be expanded, or preserved in their abbreviated form? The answers depend on the corpus purpose, but the choices must be applied consistently across the whole collection. A corpus designed for historical linguistics will keep historical spellings intact, while one aimed at pedagogical use may regularise them to a modern standard.

Tokenisation, the splitting of the cleaned stream into individual words and punctuation marks, is the bridge between raw text and searchable data. Most modern tools handle English tokenisation reasonably well, but for a language like Setswana, with its agglutinative tendencies and concordial morphology, custom rules often outperform off-the-shelf packages. A solid tokeniser recognises prefixes, suffixes, and the joining behaviour of noun classes, so that searching for a stem still surfaces inflected forms.

Annotation, metadata, and searchability

A corpus without annotation is a haystack without labels. Each text needs metadata: author, year, genre, region, and any relevant sociolinguistic markers. In an Australian project, "region" might distinguish between material gathered in Hobart and material gathered in Darwin, since vocabulary, climate references, and even borrowed words shift across such distances. Capturing this metadata up front costs little and saves hours later.

Linguistic annotation goes further, marking parts of speech, lemmas, and grammatical relationships. Even a modest part-of-speech tagger earns its keep quickly, because it allows the lexicographer to ask precise questions about collocations and grammatical patterns. A corpus that answers these questions transforms from a reading list into a research instrument.

Searchability depends on file format. Plain text with one sentence per line is portable and human-readable. XML, structured under the TEI guidelines, adds rigour that survives decades of software change. Many African language projects favour TEI because it accommodates multiple scripts, interlinear glosses, and audio references within a single file. Beginners often start with plain text and migrate later, once the design has settled.

Validation, maintenance, and ethical care

Once assembled, the corpus must be tested. Random samples should be read aloud, or by a second researcher, to catch systematic errors. Frequency lists generated from the corpus can be compared against independent sources, such as the Macquarie Dictionary's word lists or the Australian National Dictionary Centre's published frequencies, to flag obvious gaps or over-representations. A corpus with too many cricket reports and no parliamentary speeches needs adjusting, just as one with only formal prose needs a wash of colloquial material.

Maintenance is ongoing. New texts arrive, software updates change how files are read, and storage formats evolve. A simple version-control system, even a dated folder structure, prevents the loss of earlier work. Many projects keep a release snapshot each year and store it separately from the working files, the same way a librarian archives a finished catalogue before the next edition begins.

Ethical care deserves the final word here. A corpus built on community speech carries obligations that software cannot discharge. Speakers must be credited where they wish to be, sensitive material must be restricted, and access policies must reflect the wishes of those whose language is being documented. The lexicographer who traces the word morse back through its history soon learns that the answer depends as much on the community that uses it as on any printed citation.

Choosing a workflow that fits the project

Not every corpus needs the same pipeline. A small dialect dictionary compiled over a single weekend in a regional town can be built with a laptop, a spreadsheet, and a handful of audio files. A national historical dictionary, by contrast, demands years of curation, a team of researchers, and an institutional home. Honest matching of ambition to resources is the most underrated skill in the field.

A practical way to decide is to draft a one-paragraph description of the finished corpus, then list the steps required to produce it. If the list runs to more than twenty items, the scope is probably too wide for a first attempt. Trim, build a minimum viable corpus, and only then expand. This sequencing prevents the common failure mode of a project that never escapes its planning phase.

Budget matters too. Software licences, cloud storage, and transcription services all add up, and many Australian projects pair modest university grants with in-kind support from cultural institutions. The cheapest components are careful planning and clear documentation; the most expensive are usually rushed cleaning and lost source files.

Approach	Best suited for	Setup cost	Licence risk	Maintenance load
Manual fieldwork collection	Indigenous and endangered languages	High in time, low in cash	Low when consent is recorded	Steady, predictable
Web scraping pipeline	Contemporary online discourse	Medium developer hours	Moderate, needs site checks	Frequent technical updates
Institutional API harvest	Newspapers, parliamentary records	Low once configured	Low, governed by agreement	Minimal after setup
Crowdsourced submission	Community-driven dictionaries	Low but slow	Depends on submission terms	Ongoing community liaison
Hybrid pipeline	Most serious projects	Highest overall	Lowest when combined	Shared across methods

A corpus earns its value the moment a stranger can pick it up, follow its documentation, and reach a useful conclusion without phoning the original compiler. That standard is reachable, and reaching it is mostly a matter of patient sequencing: define, source, clean, annotate, validate, and document. Treat the lexicographer's toolkit as a living set of habits rather than a fixed checklist, and the work will hold up to scrutiny years down the track.

A few practices travel well across all six stages. Keep raw files untouched and work on copies, so that any failed experiment can be undone. Date every script and every output, even when the date feels obvious. Write notes for the next researcher, because in corpus work the next researcher is almost always you, six months later, trying to remember why a particular filter was applied.

The deepest lesson is that a corpus is a snapshot of language at a moment in time. Languages change, speakers move, new vocabulary appears in a Melbourne cafe one month and an Adelaide football club the next. A well-built corpus does not freeze language forever; it captures it honestly, so that the next generation of lexicographers can see clearly what their predecessors heard, read, and recorded.