tokens module¶
Utitiles for parsing and create nxt token strings.
-
register_token(prefix, detect, resolve)[source]¶ Define a token to be expandable by resolve. Requires 3 components, a prefix, a detect callable, and a resolve callable.
For an example token see the example plugin in the nxt plugins directory.
Parameters: - prefix (str) – prefix of token
- detect (callable) – callable that takes a single argument and returns True if given input can be resolved by this token type.
- resolve (callable) – callable that resolves token content
-
get_token_content(token_str)[source]¶ Removes token prefix and suffix from given token_str
Parameters: token_str (str) – string to remove token syntax from Returns: given token_str with token syntax removed Return type: str
-
make_token_str(token_content)[source]¶ Build a token around given token_content. The opposite of get_token_content
Parameters: token_content (str) – token content to wrap Returns: token content wrapped in token syntax Return type: str
-
get_standalone_tokens(raw_value, token_types=None)[source]¶ Get tokens in the given value that are not nested within another token. If none are found, an empty list is returned. If the token syntax is malformed(extra starts or ends), return empty list. Returns list including outer token syntax
Parameters: - raw_value (str) – value to find standalone tokens in.
- token_types (list, tuple, or None) – Optionally a list of specific token types can be provided, only standalone tokens that look like those token type(s) will be returned.
Returns: list of standalone tokens, if found.
Return type: list
-
atomic_token_partition(value)[source]¶ Partition given value on a token that appears resolvable(contains no sub tokens). Returns in a tuple: (before_token, token, after_token). Returned token includes token syntax. If no tokens are found, returned tuple contains None in all values.
Parameters: value (str) – text to find a token from, and partition Returns: before_token, token, after_token Return type: tuple(str, str, str)