Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- 1. Lex keywords as identifiers. Compute simple hashsum while recognizing (acc = acc * N + x).
- 2. Make pass over tokens to resolve hashes
- consult ehash = EXACT_HASH[(hash >> (i * P)) & ((1 << P) - 1)] for i in 0..Q
- if ehash.key matches your hash, record ehash.value as the hash instead
- otherwise, record 0
- N, P and Q make PHF
- N is choosen to minimize P and Q,
- P is the size of the EXACT_HASH table
- Q is the max collision size (usually 2)
- 3. When new token begins:
- 3.a) add TOK[REC] to the retired token's value
- 3.b) set REC to REC_HASH[hash].
- 4. If CHECK[REC] == current character, advance to REC+1,
- otherwise, set REC=BADREC
- Generation:
- TOK[X] = 0 if X does not recognize a keyword
- TOK[X] = TOKEN(Keyword) - TOKEN(Identifier) if X recognizes a keyword
- REC_HASH[hash(X)] = X
- CHECK[X + 0], CHECK[x + 1] etc equal to characters 0, 1, ... in X keyword
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement