Tokenization is the process by which the MDEX Engine breaks up compound phrases in the search query string into their constituent words or characters, producing a sequence of distinct query terms.