Rule-based & Regex Tokenization

How do you consistently split 'New York-based' into three tokens or separate 'U.S.A.' into individual letters, without writing custom logic for every edge case? This challenge requires rule-based and regular expression (regex) tokenization, methods that apply predefined patterns to accurately segment text.

The Limits of Whitespace Tokenization

pythonBasic String Splitting and Its Shortcomings

Crafting Custom Rules for Tokenization

Rule-based Tokenization

A method of text segmentation that applies a set of predefined linguistic rules to identify and separate tokens, often refining the output of a simpler initial split.

Example: A rule might state: 'If a token ends with a period, comma, or exclamation mark, separate the punctuation into its own token.'

A General Process for Rule-Based Tokenization

Initial Segmentation

Begin by performing a basic split, often using whitespace, to get an initial set of potential tokens. This provides a starting point for refinement.

Apply Merge Rules

Examine adjacent tokens or specific patterns to determine if they should be combined. For instance, merge 'U.S.' and 'A.' into 'U.S.A.', or ensure hyphenated words like 'New-York' remain a single token.

Apply Split Rules

Identify tokens that contain internal delimiters or attached punctuation that should be separated. This includes splitting contractions (e.g., 'don't' into 'do' and 'n't') or detaching punctuation (e.g., 'word.' into 'word' and '.').

pythonSimple Rule-Based Tokenizer Example

Check Your Understanding

Given the rule 'split any token containing an apostrophe into two parts at the apostrophe', how would 'they're' be tokenized?

Unlocking Patterns with Regular Expressions

While explicit rule sets can work, they quickly become cumbersome for complex patterns. Regular expressions (regex) offer a powerful, concise language for defining intricate text patterns. Instead of writing conditional logic for each case, you construct a single pattern that describes what a valid token looks like or where a split should occur. This allows for highly flexible and precise tokenization, handling many edge cases within a single, expressive string.

✨ Essential Regex Metacharacters

. : Matches any character (except newline)
$\w$ : Matches any word character (alphanumeric + underscore)
$\s$ : Matches any whitespace character
$\d$ : Matches any digit
+ : Matches one or more occurrences of the preceding character/group
: Matches zero or more occurrences
? : Matches zero or one occurrence
[] : Matches any single character within the brackets (e.g., [aeiou])
| : Acts as an OR operator (e.g., cat|dog)
() : Groups expressions, creating a sub-pattern
^ : Matches the start of a string
$ : Matches the end of a string

pythonBasic Regex Tokenization with `re.findall`

Mastering Complex Tokenization with Regex

The real power of regular expressions emerges when handling tricky cases that simple splitting or basic rules miss. Consider phrases like 'U.S.A.', 'New York-based', or contractions such as 'can't'. A robust tokenizer needs to treat 'U.S.A.' as a single token, split 'New York-based' into three distinct tokens ('New', 'York', 'based'), and correctly separate 'can't' into 'can' and 'n't'. Crafting a single regex pattern that gracefully handles these diverse requirements is a hallmark of advanced tokenization.

pythonComprehensive Regex for Advanced Tokenization

Check Your Understanding

Which part of the regex r" $\b\w+($ ?:' $\w+)$ ? $\b$ |[.,!?;:]" is responsible for matching contractions like 'won't'?

Choosing Your Tokenization Strategy

Comparing Tokenization Methods

Feature	`str.split()` (Whitespace)	Simple Rule-Based	Regex-Based
Flexibility	Low (fixed delimiter)	Medium (customizable rules)	High (pattern matching)
Complexity	Very Low	Medium (rule logic)	High (regex syntax)
Performance	Very High (native)	Medium (iterative rules)	Medium to High (optimized C engine)
Use Cases	Simple word counts, clean text	Refining basic splits, specific edge cases	Complex linguistic patterns, robust NLP
Maintenance	Low	Medium (adding/modifying rules)	Medium (complex patterns can be hard to read)

A comparison of common tokenization approaches and their suitability.

Try It Yourself

The current regex handles basic words and punctuation. Modify the pattern variable to also correctly tokenize email addresses (e.g., user@example.com) as single tokens. Ensure the email pattern is prioritized over general word or punctuation patterns.

python

Key Takeaways

Simple whitespace splitting is often insufficient for real-world text due to contractions, punctuation, and compound terms.
Rule-based tokenization applies explicit linguistic rules to refine initial token splits, handling specific patterns like contractions or trailing punctuation.
Regular expressions provide a powerful, concise language for defining complex tokenization patterns, offering high flexibility.
Regex patterns use metacharacters (e.g., $\w$ , +, ?, |) to describe what constitutes a token or where splits should occur.
Advanced regex can handle intricate cases such as abbreviations (e.g., 'U.S.A.'), hyphenated words (e.g., 'New York-based'), and contractions within a single pattern.
Choosing the right tokenization strategy depends on the text's complexity and the required precision, balancing flexibility with performance.
Rule-based and regex methods provide the necessary precision to handle complex linguistic patterns, ensuring accurate text analysis where simple splitting fails, directly addressing the challenge of consistently tokenizing phrases like 'New York-based' or 'U.S.A.'.

Try it yourself

Tokenization

A real byte-pair tokenizer, trained live on a small corpus. Type “strawberry” and see why a model that writes essays cannot count the r’s in it.

Open the lab

← All lessons in Tokenization