Parsing data with strtok in C

Some programs can just process an entire file at once, and other programs need to examine the file line-by-line. In the latter case, you likely need to parse data in each line. Fortunately, the C programming language has a standard C library function to do just that.

The strtok function breaks up a line of data according to "delimiters" that divide each field. It provides a streamlined way to parse data from an input string.

Reading the first token

Suppose your program needs to read a data file, where each line is separated into different fields with a semicolon. For example, one line from the data file might look like this:

102*103;K1.2;K0.5

In this example, store that in a string variable. You might have read this string into memory using any number of methods. Here's the line of code:

char string[] = "102*103;K1.2;K0.5";

Once you have the line in a string, you can use strtok to pull out "tokens." Each token is part of the string, up to the next delimiter. The basic call to strtok looks like this:

#include <string.h>
char *strtok(char *string, const char *delim);

The first call to strtok reads the string, adds a null (\0) character at the first delimiter, then returns a pointer to the first token. If the string is already empty, strtok returns NULL.

#include <stdio.h>
#include <string.h>

int
main()
{
  char string[] = "102*103;K1.2;K0.5";
  char *token;

  token = strtok(string, ";");

  if (token == NULL) {
    puts("empty string!");
    return 1;
  }

  puts(token);

  return 0;
}

This sample program pulls off the first token in the string, prints it, and exits. If you compile this program and run it, you should see this output:

102*103

102*103 is the first part of the input string, up to the first semicolon. That's the first token in the string.

Note that calling strtok modifies the string you are examining. If you want the original string preserved, make a copy before using strtok.

Reading the rest of the string as tokens

Separating the rest of the string into tokens requires calling strtok multiple times until all tokens are read. After parsing the first token with strtok, any further calls to strtok must use NULL in place of the string variable. The NULL allows strtok to use an internal pointer to the next position in the string.

Modify the sample program to read the rest of the string as tokens. Use a while loop to call strtok multiple times until you get NULL.

#include <stdio.h>
#include <string.h>

int
main()
{
  char string[] = "102*103;K1.2;K0.5";
  char *token;

  token = strtok(string, ";");

  if (token == NULL) {
    puts("empty string!");
    return 1;
  }

  while (token) {
    /* print the token */
    puts(token);

    /* parse the same string again */
    token = strtok(NULL, ";");
  }

  return 0;
}

By adding the while loop, you can parse the rest of the string, one token at a time. If you compile and run this sample program, you should see each token printed on a separate line, like this:

102*103
K1.2
K0.5

Multiple delimiters in the input string

Using strtok provides a quick and easy way to break up a string into just the parts you're looking for. You can use strtok to parse all kinds of data, from plain text files to complex data. However, be careful that multiple delimiters next to each other are the same as one delimiter.

For example, if you were reading CSV data (comma-separated values, such as data from a spreadsheet), you might expect a list of four numbers to look like this:

1,2,3,4

But if the third "column" in the data was empty, the CSV might instead look like this:

1,2,,4

This is where you need to be careful with strtok. With strtok, multiple delimiters next to each other are the same as a single delimiter. You can see this by modifying the sample program to call strtok with a comma delimiter:

#include <stdio.h>
#include <string.h>

int
main()
{
  char string[] = "1,2,,4";
  char *token;

  token = strtok(string, ",");

  if (token == NULL) {
    puts("empty string!");
    return 1;
  }

  while (token) {
    puts(token);
    token = strtok(NULL, ",");
  }

  return 0;
}

If you compile and run this new program, you'll see strtok interprets the ,, as a single comma and parses the data as three numbers:

1
2
4

Knowing this limitation in strtok can save you hours of debugging.

Using multiple delimiters in strtok

You might wonder why the strtok function uses a string for the delimiter instead of a single character. That's because strtok can look for different delimiters in the string. For example, a string of text might have spaces and tabs between each word. In this case, you would use each of those "whitespace" characters as delimiters:

#include <stdio.h>
#include <string.h>

int
main()
{
  char string[] = "  hello \t world";
  char *token;

  token = strtok(string, " \t");

  if (token == NULL) {
    puts("empty string");
    return 1;
  }

  while (token) {
    puts(token);
    token = strtok(NULL, " \t");
  }

  return 0;
}

Each call to strtok uses both a space and tab character as the delimiter string, allowing strtok to parse the line correctly into two tokens.

Wrap up

The strtok function is a handy way to read and interpret data from strings. Use it in your next project to simplify how you read data into your program.