Building a Setswana Twitter corpus for lexical analysis

Twitter, now operated as X, offers a large but selective record of everyday language. For Setswana lexicography, its value lies less in representing the whole language than in exposing how words circulate in short, time-stamped and socially situated posts. A corpus of Setswana Twitter data can reveal emerging expressions, changes in spelling, local topics and patterns of meaning that may be under-described in printed dictionaries. It cannot substitute for oral evidence or careful historical lexicography, but it can provide a timely layer of usage data.

The practical task is to turn a stream of posts into a bounded collection whose contents can be explained and reproduced. Researchers must decide which API fields to retain, how far back to sample, which language-identification rules apply, and what counts as a Setswana tweet rather than an English post containing one Setswana word. Because the platform changed its access model after rebranding, older Twitter archives and newer X API results may not offer identical coverage. The corpus design should therefore state its collection dates, endpoint, query logic and known blind spots.

For an Australian audience, several local conditions shape that design. Researchers in Melbourne, Sydney and Brisbane regularly work with southern African diaspora communities; bilingual networks can produce rapid English–Setswana code-switching. Australian English also supplies familiar informal features, including shortened forms such as “info” and “org,” understatement and conversational fillers like “ta” or “nah.” Local place hashtags such as #Melbourne, #Perth and #Adelaide can anchor otherwise mobile conversations. Timestamps must be interpreted against Australian Eastern Standard Time, with daylight-saving changes affecting seasonal comparisons.

Designing the corpus around a clear research question

Every corpus answers a particular question more clearly when its boundaries are explicit. The ongoing lexical research page provides the wider setting for this work, but the Twitter collection should have its own research rationale: track candidate words, test usage hypotheses or document language contact. The central unit should be the post, not the hashtag or individual token, so repeated copying and replies can later be measured without allowing one prolific account to dominate frequency counts.

Collection should preserve the information needed for linguistic interpretation: post text, creation time, language label, user identifiers where permitted, conversation identifiers and public engagement counts. Personally identifying details should be minimised or pseudonymised, while a stable internal identifier can preserve relationships among a reply, retweet and quoted post. Public visibility does not remove ethical responsibilities, especially where posts mention health, work, religion or political activity.

Twitter’s paid API tiers and restricted historical access also affect what an Australian project can obtain. Universities and research teams may negotiate institutional access, purchase metered data or combine a newly sampled collection with licensed archive extracts. These arrangements belong in the technical record, because an apparently similar corpus can differ sharply if one includes full-history search and another receives only recent matching posts. Corpus size alone is therefore a poor measure of quality.

Sampling posts without freezing a moving language

Keyword sampling can miss words whose spelling changes, while broad language labels can include posts containing only a token of Setswana. The search strategy should combine Setswana terms, character patterns, frequent hashtags, names and manually identified phrases with an automated language score. Queries should be tested against known examples in formal Setswana, colloquial Setswana and code-switched English. This reduces both false positives and the exclusion of posts that use an innovative spelling absent from the query list.

Representativeness also requires attention to accounts and automated activity. Frequency should not be inferred from raw posts without checking whether a small group of users, commercial accounts or automated services generated large shares of the data. A corpus intended for dictionary work may retain ordinary posts while flagging suspected spam, duplicated advertising and coordinated campaigns. These categories should be documented rather than silently removed, since their presence may itself explain unusual distributions.

The sampling frame can use Australian time and topical conditions to improve coverage. Day and night, weekday patterns and daylight-saving boundaries may affect when people post. Election campaigns, sporting events, university examinations and public health announcements may also create temporary bursts of vocabulary. Comparing regional hashtags with national discussions can show whether an expression belongs to a broad community or to a localised network in Brisbane, Melbourne or another city.

Normalising text without erasing meaningful variation

Normalisation must distinguish computational cleanup from linguistic alteration. Unicode should be unified so that visually identical characters can be counted together, while original spelling should remain available in a separate field. Repeated whitespace can be collapsed, and obvious artefacts such as truncated URLs or platform-generated labels can be tagged. Case folding may help with searching, but it should not erase distinctions that matter in Setswana orthography or in hashtags. Users sometimes exploit capitalisation for emphasis, community identity or playful word formation.

Tokenisation requires Setswana-specific decisions. A tokeniser trained only on English may split clitics, punctuation or multiword terms incorrectly, especially in code-switched passages. Hashtags need separate treatment because they can function simultaneously as topics, lexical items and boundaries marking a change of register. Mentions, emoji and punctuation can influence meaning, but not every marker should receive equal analytical weight. The corpus should preserve enough of the original post to allow researchers to revisit those choices.

Orthographic variation deserves particular care. Standard Setswana spelling may coexist with informal forms, phonological spellings, English loanwords and deliberate stylisation. Language identification should therefore combine script, dictionary coverage, function words and contextual cues rather than treating every non-English character sequence as Setswana. Work on Afrikaans loanwords in Setswana illustrates why imported forms may be difficult to classify from spelling alone.

Annotating words, senses and emerging usage

The first analytical pass can identify candidate headwords, lemmas, parts of speech and collocations. A word appearing thousands of times may be lexically stable, while a rare expression may reveal an emerging sense or a new compound. Counts should therefore be interpreted alongside semantic annotation. Context windows, topic labels and user behaviour help distinguish a genuine lexical item from a typo, a temporary campaign slogan or a quotation copied from another source.

Domain tagging can connect social-media usage with specialised lexicography. A term discussed by football supporters, public servants or university students may have a narrower meaning than its general dictionary entry suggests. The project on Setswana legal terminology demonstrates the value of placing terms in defined contexts and recording their institutional meanings. Twitter evidence can complement such a termbase by showing how official or borrowed terms are adapted in public conversation.

Human validation remains essential. Automated tools can propose segmentations, candidate senses and frequency estimates, but trained Setswana speakers should assess ambiguous cases. Independent annotators can double-sample the data to measure agreement, after which disputed decisions should be retained as examples in the codebook. Context screenshots should accompany particularly important records because platform interfaces can reveal replies, media and thread structure that a plain-text export omits.

Placing Twitter usage beside dictionaries and etymology

Dictionary evidence provides the historical baseline against which Twitter usage can be interpreted. A post can show that a word has become more frequent, acquired a humorous tone or developed a new collocational preference, but those observations need comparison with older dictionaries and archival texts. The corpus should record when an item first appeared within the collection, although that date marks visibility in the sample rather than the actual invention of the word.

Etymology requires a similarly cautious method. A platform search can retrieve an old quotation or a recent meme, but neither proves linguistic ancestry. Historical dictionaries, specialist studies and traceable examples should support claims about origin. The history of Sic semper tyrannis is a useful model: familiar sayings travel through books, political language and public circulation, acquiring new associations as they travel. Setswana expressions similarly change through contact, quotation and adaptation.

Comparative analysis can also test whether online frequency reflects broader usage. Candidates should be checked against general corpora, newspapers, fiction, legal texts and interviews where available. A term common on Twitter but absent from these sources may be innovative, socially restricted or simply inaccessible to standard collection methods. These possibilities should remain separate until further evidence distinguishes them.

Governing, documenting and releasing the corpus

Australian privacy expectations make governance part of the corpus itself. Collection and storage procedures should be reviewed through the relevant institutional ethics process and assessed against the Privacy Act 1988 and Australian Information Commissioner’s guidance. Access controls should reflect the sensitivity of the platform data and the terms imposed by the API. The published corpus may need to exclude live identifiers while retaining stable research identifiers and enough metadata for reproducible analysis.

Documentation should include the corpus rationale, sampling dates, API queries, exclusion rules, language filters, annotation codebook and processing software. Licensing conditions must be respected, and researchers should record whether derived files can be redistributed. A useful release may combine a text corpus with frequency lists, concordance tools, sample contexts and a small set of fully documented research cases. Transparent limitations are essential because platform archives remain incomplete and susceptible to deletion, bot activity and changes in access policy.

A pilot should first test the search expressions, language filter and annotation scheme on a manageable sample. The next concrete step is to publish the sampling frame and codebook for review by Setswana-speaking lexicographers and corpus specialists.