Words

What are Words?

Exactly what counts as a word is kind of hard to pin down. It’s typically understood to be a unit of meaning that can stand alone in an utterance. There’s a lot of variation among languages, especially in the orthographic representations—think of agglutinative languages like Finnish, Hungarian, Turkish, Japanese, Swahili, Korean, Quecha, and Tamil.

In language processing systems, we typically use tokens (units of recognition) that can be anything from single characters (letters, digits, punctuation) to whole words of arbitrary length, using a modified Byte-Pair Encoding (BPE) technique.

Credits:

Much of the material here is adapted from Steven Mithen’s book The Language Puzzle, specifically Chapters 3, 6, and 13.

Basic Terminology

Remember that language is often spoken, signed, or written.

Here are some linguistic terms useful in the study of spoken words, just to get started:

• Spoken Word: a unit of meaning that can stand alone in an utterance, made up of one or more phonemes
• Phoneme: a single speech sound
• Vowel: a phoneme produced without significant constriction of the vocal tract
• Consonant: a phoneme produced with some constriction of the vocal tract
• Syllable: a larger sound unit
• Phonology: the study of the functional patterns of speech sounds, specifically how phonemes create meaning
• Morpheme: a part of a word that contribute meaning
• Morphology: How words (tokens) are made up of smaller pieces
• Syntax: How words are structured to form phrases and sentences
• Semantics: What sentences mean
• Pragmatics: How language is used to communicate ideas and concepts

The word chereme was used for a unit of sign language, but today the term has largely been replaced by phoneme in the context of sign language. So phonemes are now used to describe the smallest meaningful units in both spoken and signed languages. Similarly cherology was replaced by phonology in the study of sign language.

In written language, the basic unit of writing is the grapheme.

There are a lot of -eme words in lingusitics, apparently.

Lexical and Grammatical Words

Words are often divided into lexical and grammatical words:

To be honest, the boundary between these categories is not always clear-cut, for example, auxiliary verbs in English feel like they can be both.

Lexical words can range from the concrete (e.g., apple, fly, blue) to the abstract (e.g., freedom, oppression, justice).

Iconicity

Some words are iconic, meaning their form resembles (in sound, shape, texture, movement, size, or other perceptual qualities) their meaning (e.g., especially the onomatopoeias like buzz, bang, hiss, plop, honk), and others are arbitrary (e.g., dog, cat). Sometimes you’ll see hybrids.

Iconic words are also called sound-symbolic or imagistic words. Recent research suggests that they may have been common in the early stages of language evolution, and that they play a big role in the way infants acquire language. A degree of iconicity has been found in thousands of languages, so it can be considered universal.

Oddly enough, some linguists marginalized iconic words. The most influential voice here is Ferdinand de Saussure, whose principle of the arbitrariness of the sign (l’arbitraire du signe)—laid out in his posthumously published Course in General Linguistics (1916)—held that the link between a word’s form (the signifier) and its meaning (the signified) is fundamentally unmotivated and a matter of social convention. Saussure did carve out onomatopoeia and interjections as exceptions, but treated them as marginal, and this arbitrariness-first view became the dominant assumption in 20th-century structuralist and generativist linguistics. Working in that tradition, many linguists went on to treat iconic and sound-symbolic vocabulary (especially ideophones) as peripheral, “expressive,” or otherwise outside the core, rule-governed grammar of a language, rather than as a fully productive part of the lexicon. As late as 1994, Steven Pinker dismissed sound symbolism in The Language Instinct as “a quaint curiosity.”

But the importance of iconic words came back, with experiments asking participants about:

• Which is small and which is large: mil vs. mal?
• Which is round and which is spiky: maluma vs. takete?
• Which is round and which is spiky: bouba vs. kiki?

People gave consistent answers no matter their native language. Humans are prone to cross-modal perception.

The linguist Roger Wescott noticed and recorded some interesting observations in Linguistic Iconism, many of which were found across many languages.

• High, front vowels for small, fast, and light things: tiny, little, mini, bee, bit.
• Low, back vowels for large, slow, and heavy things: mammoth, vast, huge, cow, huge.
• Lateral consonants (l) for fluid, smooth, or flowing motion: flow, glide, lull, slide.
• Labial consonants (p, b, m) for roundness, fullness, or softness: balloon, bubble, plump, puffy.
• Stop consonants (p, t, k, b, d, g) for abrupt, sudden, or discrete actions and sounds: pop, bang, click, tap.
• Dental obstructions (t, d) for small, sharp, or precise things and actions: tick, dot, tiny, dart.

English also has plenty of its own onset clusters that reliably signal a shared meaning, called phonesthemes—these are more of a language-specific pattern than a cross-linguistic universal like the ones above, but they show the same principle at work:

• sl-: slipperiness, sliding, or (often derogatory) looseness—slide, slip, slick, sludge, slime, sloppy, sly.
• fl-: movement through air or light, often quick or fluttery—fly, flow, float, flutter, flicker, flap.

Note that there can only be so many iconic words, so the vast majority of words in any language are arbitrary. Iconic words tend to be used for general categories of things, which is often enough for a child learning a language.

Lexicons

The lexicon of a language is its inventory of words (and their meanings).

The average adult English speaker knows ~50,000–100,000 words (their passive vocabulary), but frequently uses ~20,000–30,000. An unlimited number can be made in theory which can be figured out.

Children acquire words at a remarkable pace, about 10 new words per day between ages 2–8, many of which are inferred from a single exposure in context. LLMs, on the other hand, are trained on trillions of tokens but don’t seem to generalize as easily.

Morphemes

A word is made up of one or more morphemes, the smallest units of meaning.

They can be lexical (anchored to a concept) or grammatical (inflection, affix), and can be either free or bound.

English examples:

• Free lexical morpheme (root), e.g. dog, happy, walk
• Bound lexical morpheme, e.g. bio (as in biology), tele (as in telephone)
• Free grammatical morpheme, e.g. and, but, the, as, of
• Bound grammatical morpheme, e.g. -s (plural), -ed (past tense), -ing (progressive), -ment (noun-forming)

An affix is a bound morpheme that attaches to a root or stem to modify its meaning or create a new word. Common types of affixes include prefixes (added to the beginning), suffixes (added to the end), infixes (inserted within), and circumfixes (added around). Two types:

• Derivational Affix – changes the meaning or category of a word, e.g. happy → unhappy, teach → teacher, move → movement, beauty → beautiful, kind → kindness, simple → simplify.
• Inflectional Affix – indicates grammatical features such as tense, number, or case, e.g. -s (plural), -ed (past tense), -ing (progressive), -s (third-person singular present, e.g. walks), -en (past participle, e.g. eaten), -’s (possessive), -er (comparative, e.g. faster), -est (superlative, e.g. fastest)

Word Meanings

How do words get their meanings? It is a huge question. There are some theories, like Rosch’s Prototype Theory. The idea is that word categories aren’t defined by certain conditions, but rather organized around prototypes (or best examples), such as a robin for the category “bird” (and not a penguin). Other members of the category are included based on their similarity to the prototype, leading to graded membership rather than a strict boundary.

The important question is whether a hot dog, or even a pop tart, is a sandwich.

Prototype Theory in LLMs

Prototype theory works really well in LLMs! You don’t have strict category boundaries. Similarity between vectors is good enough.

You might also check out Putnam’s Twin Earth Thought Experiment, which gives evidence for meaning not being entirely in one’s head (intensionally) but also (extensionally) within one’s environmental context.

Another theory is the Causal Theory of Reference, which suggests that words get their meaning anchored historically, through chains of use.

Lexical Relations

Words are definitely related to each other! Learn these terms about these relationships to impress your friends:

Synonymy

Words with very similar meanings, close to being interchangeable but not necessarily.

• big / large
• happy / joyful
• begin / start / initiate
• watch / observe
• smart / intelligent

Antonymy

Words that have opposite meanings. Gradable antonyms are at opposite ends of a spectrum.

• hot / cold
• big / small

Complementary antonyms are absolute binary opposites, such as:

• on / off
• alive / dead
• pass / fail

Converse antonyms are relational opposites, such as:

• buy / sell
• parent / child
• teacher / student

Hyponymy

Words with a subcategory relationship.

• dog is a hyponym of animal (the hypernym)
• rose / flower
• car / vehicle
• poodle / dog / canine / animal

Polysemy

When a word has many related meanings (can be metaphorical or metonymic or systematic). Think: one lexeme with multiple senses.

• foot (body part / mountain base)
• head (body part / leader / top of something)
• run (move fast on foot / operate a business)
• get (obtain / understand / become)
• paper (material / academic article / publication / newspaper company)
• right (correct / direction, opposite of left)—both senses trace back to the same root, tied to the “right” hand historically being favored as the “correct” one

Homonymy

When words coincidentally look or sound the same, despite having completely different origins and distinct, unrelated meanings. Think: distinct lexemes sharing a surface form.

• bat (flying mammal / baseball equipment)
• bark (tree skin / dog sound)
• lie (recline / falsehood)
• bank (financial institution / river bank)
• stalk (plant stem / follow stealthily)

Word Vectors Encode These Relations

Embedding spaces capture many lexical relations geometrically. The famous example: king − man + woman ≈ queen. Synonymy, hyponymy, and analogy all emerge during training. They were not intentionally programmed in.

Evolution of Words

Language, we know, is incredibly fluid. So it makes sense that we can look at how words change over time. Chapter 13 of Mithen’s The Language Puzzle looks at four dimensions of word change:

• How word meanings change
• The invention of new words
• How pronunciation shifts
• How lexical words become grammaticalized

Like Mithen, we’ll look at these four dimensions from the point of view of recent changes in English. English is a good case study because it is widely spoken, diverged faster than the Romance languages from its common ancestor, has massive influences from other languages, and has a long written history. Written “English” goes back over a thousand years, and we can see how Old English, Middle English, and Modern English are considered completely different languages.

Semantic Change

Individual words change their meanings over time. Mithen identifies several types of semantic change.

Broadening

Becoming more general

• holiday (once specifically a “holy day” → now any day off)
• dog (once a specific breed → now any canine)

Narrowing

Becoming more specific

• meat (once any food → now specifically animal flesh)
• girl (once a young person of either sex → now specifically female)

Amelioration

Becoming more positive

• nice (once “foolish, ignorant” → now “pleasant”)
• knight (once “boy, servant” → now a noble title)

Pejoration

Becoming more negative

• silly (once “blessed, happy” → now “foolish”)
• villain (once “farmhand” → now “evildoer”)

Weakening

Losing emphasis

• awesome (once “inspiring awe or dread” → now mild, casual praise)
• terribly (once “in a way that causes terror” → now just an intensifier, as in “terribly nice”)

Strengthening

Gaining emphasis

• kill (once “to hit or strike” generally → now specifically “to cause death”)

Word conversion

Change of word class without morphological change

• Verbification
  • to rain, to snow, to pocket, to captain, to email, to google
• Verbs becoming nouns
  • to appear → appearance, to appoint → appointment, to inform → information

Metonymy

Shift in meaning based on association

• Washington (the capital, standing for the U.S. government)
• the crown (standing for the monarchy)
• Hollywood (standing for the American film industry)
• the White House (standing for the U.S. presidential administration)
• suits (standing for business executives)

Metonymy vs. Polysemy

Metonymy is association-based: the new sense refers to something contiguous with, or part of, the original (a capital standing for the government housed there). Polysemy (see Lexical Relations, above) is often similarity-based: the new sense is a metaphorical extension (head of a company resembling the head atop a body). In practice the line between the two can blur, and many polysemous senses trace back to a metonymic or metaphorical shift.

But how do these changes come about? Sometimes from misunderstanding, sometimes from metaphorical extension, and sometimes to shift a meaning as a social tactic (for identity and relationship building), and sometimes people found a way to make expression more efficient.

New Words

Here are a few ways new words are created in English:

Coinage

The straight up creation of a new word, e.g. googol, kleenex, quark (coined by physicist Murray Gell-Mann), nylon (invented as a brand name by DuPont)

Conversion

Changing the word class of an existing word, e.g. to google, to text, to friend (someone on social media), to microwave (food)

Borrowing

Adopting a word from another language, e.g. kindergarten (German), sushi (Japanese), café (French), safari (Swahili), robot (Czech)

Compounding

Combining two existing words to create a new one, e.g. notebook, sunflower, toothbrush, airport

Derivation

Adding prefixes or suffixes to create new words, e.g. happiness, unhappy, careless, readable

Backformation

Creating a new word by removing an affix, e.g. edit from editor, burgle from burglar, babysit from babysitter

Reduplication

Forming new words by repeating a whole or part of a word, e.g. tick-tock, super-duper, flip-flop, hocus-pocus

Blending

Combining parts of two words to create a new one, e.g. brunch (breakfast + lunch), smog (smoke + fog), motel (motor + hotel), spork (spoon + fork)

Clipping

Shortening a longer word, e.g. ad (advertisement), lab (laboratory), gym (gymnasium), exam (examination)

Acronym

Forming a word from the initial letters of a phrase, e.g. NASA, scuba, laser (light amplification by stimulated emission of radiation), radar (radio detection and ranging)

Sound Change

Sound changes (vowel shifts, consonant changes, stress shifts, etc.) also occur for reasons of identity marking and efficiency. We’ll cover this in our notes on phonology.

Grammaticalization

Grammaticalization refers to the process by which lexical words become grammatical words or even just morphemes. A popular example from English: will originally meant "want" or "wish," but over time it has become a grammatical marker for future tense.

A few more examples:

• going to → gonna: a verb of physical motion toward a destination became a marker of future tense, losing its sense of spatial movement (I’m going to the store vs. it’s gonna rain).
• fixing to → finna: in African American Vernacular English (AAVE), the phrase fixing to (originally meaning "preparing to" or "getting ready to") has grammaticalized into finna, a marker of immediate future or intention (I’m finna leave), functionally parallel to gonna but following its own independent path of reduction and fusion.
• have: the possession verb (I have a car) was grammaticalized into the auxiliary marking perfect aspect (I have eaten).
• let us → let’s: an imperative verb phrase (literally "allow us") was reduced to a hortative marker inviting joint action (let’s go), losing its literal sense of permission.
• because: from the full prepositional phrase by cause (that), fused over time into a single subordinating conjunction.

Recall Practice

Here are some questions useful for your spaced repetition learning. Many of the answers are not found on this page. Some will have popped up in lecture. Others will require you to do your own research.

1. What is a word, roughly?
   A unit of meaning that can stand alone in an utterance (though the exact boundary is hard to pin down).
2. What is a token, in language processing?
   A unit of recognition—anything from a single character up to a whole word.
3. What is a phoneme?
   A single speech sound.
4. What's the difference between a vowel and a consonant?
   Vowels have no significant constriction of the vocal tract; consonants have some.
5. What is phonology?
   The study of the functional patterns of speech sounds.
6. What is a morpheme?
   The smallest unit of meaning in a word.
7. What is morphology?
   How words are built from smaller pieces (morphemes).
8. What is syntax?
   How words combine into phrases and sentences.
9. What is semantics?
   What sentences (or utterances) mean.
10. What is pragmatics?
    How language is used to communicate in context.
11. What terms replaced chereme and cherology in the study of sign language?
    Phoneme and phonology, respectively.
12. What do we call the basic unit of written language?
    A grapheme.
13. What is the difference between open-class and closed-class words? Give examples of each.
    Open-class words (nouns, verbs, adjectives) accept new members freely like selfie and to google. Closed-class words (prepositions, conjunctions, determiners) form a small, stable set and are the structural glue of syntax.
14. Give an example of a concrete lexical word and an abstract one.
    Concrete: apple. Abstract: freedom.
15. What does it mean for a word to be iconic?
    Its form resembles its meaning, as in onomatopoeia (buzz, hiss).
16. What are two other names for iconic words?
    Sound-symbolic, imagistic.
17. What is Saussure's principle of the arbitrariness of the sign?
    The link between a word's form (signifier) and meaning (signified) is unmotivated and a matter of convention.
18. What did Pinker dismissively call sound symbolism, as late as 1994?
    “A quaint curiosity.”
19. In the bouba/kiki effect, which word do people consistently match to the round shape, and which to the spiky one?
    Bouba (also maluma) → round. Kiki (also takete) → spiky.
20. According to Wescott, what kind of vowels tend to show up in words for small, fast, light things?
    High, front vowels.
21. What is a phonestheme? Give an English example.
    A recurring sound cluster tied to a shared meaning, e.g., sl- for slipperiness (slide, slick).
22. Why is only a small fraction of any language's vocabulary iconic, even though a degree of iconicity seems universal?
    Sound alone can only iconically convey so much; iconic words tend to cover broad, general categories.
23. What is a lexicon?
    A language's inventory of words (and their meanings).
24. Roughly how many words does an adult English speaker actively use vs. passively know?
    ~20,000–30,000 active; ~50,000–100,000 passive.
25. What's the difference between a free and a bound morpheme?
    Free morphemes can stand alone (dog); bound morphemes cannot (-s, bio-).
26. What's the difference between a lexical and a grammatical morpheme?
    Lexical morphemes carry content meaning; grammatical morphemes serve a grammatical function.
27. What are the four types of affixes, by position?
    Prefix, suffix, infix, circumfix.
28. What's the difference between a derivational and an inflectional affix?
    Derivational: changes a word's meaning or category (teach → teacher). Inflectional: marks grammatical features like tense or number (walk → walks).
29. What is Rosch's Prototype Theory?
    Word categories are organized around best examples (prototypes), with graded membership rather than strict boundaries.
30. What does Putnam's Twin Earth Thought Experiment argue about meaning?
    Meaning isn't purely in the head (intensional); it also depends on environmental context (extensional).
31. What does the Causal Theory of Reference say about how words get their meaning?
    Meaning is anchored historically, through chains of use.
32. What's the difference between gradable, complementary, and converse antonyms?
    Gradable: opposite ends of a spectrum (hot/cold). Complementary: absolute binary opposites (on/off). Converse: relational opposites (buy/sell).
33. What is hyponymy?
    A subcategory relationship, e.g., dog is a hyponym of animal (the hypernym).
34. Distinguish polysemy from homonymy, and give an example of each.
    Polysemy: one word with multiple related meanings (bank: riverbank / financial institution). Homonymy: one word with unrelated meanings (bat: animal / baseball bat).
35. What famous vector equation illustrates that word embeddings encode lexical relations?
    king − man + woman ≈ queen
36. What are the four dimensions of word change that Mithen discusses?
    Change in meaning, invention of new words, shifts in pronunciation, and grammaticalization.
37. What's the difference between semantic broadening and narrowing?
    Broadening: meaning becomes more general (dog). Narrowing: meaning becomes more specific (meat).
38. What's the difference between amelioration and pejoration?
    Amelioration: meaning becomes more positive (nice). Pejoration: meaning becomes more negative (silly).
39. What's the difference between semantic weakening and strengthening?
    Weakening: loses emphasis (awesome). Strengthening: gains emphasis (kill).
40. What is metonymy? Give an example.
    A shift in meaning based on association, e.g., Washington standing for the U.S. government.
41. How does metonymy differ from polysemy?
    Metonymy is association-based (contiguity); polysemy is often similarity-based (metaphor).
42. Name at least five ways new words are formed in English.
    Coinage, conversion, borrowing, compounding, derivation, backformation, reduplication, blending, clipping, acronym.
43. What's the difference between blending and compounding?
    Blending combines parts of two words (brunch); compounding combines two whole words (notebook).
44. What's the difference between clipping and backformation?
    Clipping shortens a word (lab); backformation removes an affix to create a new word (edit from editor).
45. What is grammaticalization? Give the classic English example.
    Lexical words becoming grammatical words or morphemes, e.g., will (once "want/wish") becoming a future-tense marker.
46. Give an AAVE example of grammaticalization parallel to going to → gonna.
    fixing to → finna

Summary