strtok, strtok_r, strtok_s, strsep - separate strings into tokens
#include <string.h> char *strtok(char *restrict s1, const char *restrict s2);
char *strtok_r(char *restrict s1, const char *restrict s2,
char **restrict lasts);
char *strsep(char **stringp, const char *delim);
#define __STDC_WANT_LIB_EXT1__ 1
#include <string.h>
char *strtok_s(char *restrict s1, rsize_t *restrict s1max,
const char *restrict s2, char **restrict ptr);
These functions split a null-terminated string into a sequence of tokens delimited by single byte separator characters. The strtok() family of functions treats a sequence of one or more delimiter characters as a single break between tokens, while strsep() treats each as a separate delimiter and returns zero-length tokens between them.
They are designed for use with encodings representing each character as a single byte, and all character counts are measured in individual bytes, even if a string with multibyte characters is used. They do not check for null pointers, and programs may crash if passing null or otherwise invalid pointers to these functions.
A sequence of calls to strtok() breaks the string pointed to by s1 into a sequence of tokens, each of which is delimited by a byte from the string pointed to by s2. The first call in the sequence has s1 as its first argument, and is followed by calls with a null pointer as their first argument. The separator string pointed to by s2 can be different from call to call.
The first call in the sequence searches the string pointed to by s1 for the first byte that is not contained in the current separator string pointed to by s2. If no such byte is found, then there are no tokens in the string pointed to by s1 and strtok() returns a null pointer. If such a byte is found, it is the start of the first token.
The strtok() function then searches from there for a byte that is contained in the current separator string. If no such byte is found, the current token extends to the end of the string pointed to by s1, and subsequent searches for a token return a null pointer. If such a byte is found, it is overwritten by a null byte that terminates the current token. The strtok() function saves a pointer to the following byte in thread-specific data, from which the next search for a token starts.
Each subsequent call, with a null pointer as the value of the first argument, starts searching from the saved pointer and behaves as described above.
The strtok_r() function considers the null-terminated string s1 as a sequence of zero or more text tokens separated by spans of one or more characters from the separator string s2. The argument lasts points to a user-provided pointer which points to stored information necessary for strtok_r() to continue scanning the same string.
In the first call to strtok_r(), s1 points to a null-terminated string, s2 to a null-terminated string of separator characters, and the value pointed to by lasts is ignored. The strtok_r() function returns a pointer to the first character of the first token, writes a null character into s1 immediately following the returned token, and updates the pointer to which lasts points.
In subsequent calls, s1 is a null pointer and lasts is unchanged from the previous call so that subsequent calls move through the string s1, returning successive tokens until no tokens remain. The separator string s2 can be different from call to call. When no token remains in s1, a null pointer is returned.
The strsep() function locates, in the null-terminated string referenced by stringp, the first occurrence of any character in the string delim (or the terminating ‘\0’ character) and replaces it with a ‘\0’. The location of the next character after the delimiter character (or NULL, if the end of the string was reached) is stored in stringp. The original value of stringp is returned.
An “empty” field (one caused by two adjacent delimiter characters) can be detected by comparing the location referenced by the pointer returned by strsep() to ‘\0’.
If stringp is initially NULL, strsep() returns NULL.
The strtok_s() function is part of the C11 bounds checking interfaces specified in the C11 standard, Annex K. It provides similar functionality to the strtok_r() function, but with additional safety checks in the form of explicit runtime constraints as defined in the C11 standard. See runtime_constraint_handler(3C) and INCITS/ISO/IEC 9899:2011.
If a runtime constraint violation is detected and the handler returns, or there is no token, the strtok_s() function returns a null pointer, otherwise a pointer to the first character of the token is returned.
strtok_s() will fail if:
Null pointer is passed or source and destination overlap
s1max argument is not valid value
Destination array is too small
The following example searches for tokens separated by space characters.
#include <string.h> ... char *token; char line[] = "LINE TO BE SEPARATED"; char *search = " "; /* Token will point to "LINE". */ token = strtok(line, search); /* Token will point to "TO". */ token = strtok(NULL, search);Example 2 Break a Line.
The following example uses strtok() to break a line into two character strings separated by any combination of SPACEs, TABs, or NEWLINEs.
#include <string.h>
...
struct element {
char *key;
char *data;
};
...
char line[LINE_MAX];
char *key, *data;
...
key = strtok(line, " \t\n");
data = strtok(NULL, " \t\n");
Example 3 Search for tokens
The following example uses strtok(), strtok_r(), and strsep() to search for tokens separated by characters from the string pointed to by the second argument, “/”.
#define __EXTENSIONS__
#include <stdio.h>
#include <string.h>
int main(void) {
char buf[] = "5/90/45";
char buf1[] = "//5//90//45//";
char buf2[] = "//5//90//45//";
char *token;
char *lasts;
printf("tokenizing \"%s\" with strtok:\n", buf);
if ((token = strtok(buf, "/")) != NULL) {
printf("token = \"%s\"\n", token);
while ((token = strtok(NULL, "/")) != NULL) {
printf("token = \"%s\"\n", token);
}
}
printf("\ntokenizing \"%s\" with strtok_r:\n", buf1);
if ((token = strtok_r(buf1, "/", &lasts)) != NULL) {
printf("token = \"%s\"\n", token);
while ((token = strtok_r(NULL, "/", &lasts)) != NULL) {
printf("token = \"%s\"\n", token);
}
}
printf("\ntokenizing \"%s\" with strsep:\n", buf2);
nexts = buf2;
if ((token = strsep(&nexts, "/")) != NULL) {
printf("token = \"%s\"\n", token);
while (nexts != NULL) {
token = strsep(&nexts, "/");
printf("token = \"%s\"\n", token);
}
}
}
When compiled and run, this example produces the following output:
tokenizing "5/90/45" with strtok: token = "5" token = "90" token = "45" tokenizing "//5//90//45//" with strtok_r: token = "5" token = "90" token = "45" tokenizing "//5//90//45//" with strsep: token = "" token = "" token = "5" token = "" token = "90" token = "" token = "45" token = "" token = ""
See attributes(7) for descriptions of the following attributes:
|
The strtok() function is MT-Safe.
The strsep() and strtok_r() functions are Async-Signal-Safe.
The strtok_s() function cannot be used safely in a multithreaded application due to the runtime constraint handler. For more information, see the runtime_constraint_handler(3C) man page.
See standards(7) for descriptions of the following standards:
|
runtime_constraint_handler(3C), strpbrk(3C), strspn(3C), wcstok(3C), attributes(7), standards(7)
A standard conforming application can gain access to strtok_r() only by defining __EXTENSIONS__ or by defining _POSIX_C_SOURCE to a value greater than or equal to 199506L.
The strtok() function is safe to use in multithreaded applications on Oracle Solaris because it saves its internal state in a thread-specific data area. (This is not true on some other platforms though, so should not be relied upon in portable software.) However, its use is discouraged, even for single-threaded applications. The strtok_r() function should be used instead.
Do not pass the address of a character string literal as the argument s1 to strtok(), strtok_r(), or strtok_s(). Similarly, do not pass a pointer to the address of a character string literal as the argument stringp to strsep(). These functions can modify the storage pointed to by these parameters. The C99 standard specifies that attempting to modify the storage occupied by a string literal results in undefined behavior. This allows compilers (including gcc, clang, and the Oracle Developer Studio compilers) to place string literals in read-only memory. Note that in Example 1 above, this problem is avoided because the variable line is declared as a writable array of type char that is initialized by a string literal rather than a pointer to char that points to a string literal.
Support for the following functions is available in Oracle Solaris starting with the listed release:
|