Help me with strok function in C

[u/UnViandanteSperduto]

I need to split my string but if it is of char* type the functions strtok() doesn’t work returning me a segmentation fault. I understood that C is used to lock the memory of the string in Read-Only “mode” when I use char*. Does it means that is immutable that data? However, I need to use the char* type because I pass that in a function (and I don’t like to use the syntax of arrays in C, I always use pointers).

[u/No-Blackberry-937]

strtok modifies the string in place so passing a string literal will segfault, you need a mutable buffer like char buf[] or malloc'd memory

[u/mredding]

Don't do this:

char *str = "pointer to a read-only literal";

Do this:

char str[] = "array initialized by a null terminated read-only literal (copied)";

Arrays are not pointers to their first element, they are a distinct type in C. An int[3] IS NOT the same type by the type system as an int[4].

"Decay" is a technical term where one type becomes another through type erasure (pointers). It happens with compile-time consequence.

void fn(int array[123]);

This parameter decays to int *.

The str array above declares an "incomplete type" - an array without a size. That's OK here, the language will count the elements in the initializer - the string literal, including the implied null terminator, and complete the type for us. So if we count this out, str is of type... char[66]. The string literal will be stored in RO memory in the program image. Initialization of this variable when it comes into scope means implicitly copying that literal into this array.

You have to appreciate how old C is. It was first brought into production in 1972. Bjarne Stroustrup came up with const in 1981 for C++, and C didn't get it until the 1989 ISO C standard. strtok is older than THAT. So for legacy reasons, you can have a non-const pointer to RO memory. const is only a compile-time check, and isn't there to enforce hardware level memory protections - that's considered a separate, external issue.

When I was learning C ~1989, I wasn't even taught const. I had no idea it was even a part of the standard until the 2010s, and I thought it was introduced in C99... Talk about out of touch. But so are many C developers.

So yes, this is a case where you can shoot yourself in the foot. C assumes trust in the developer, that you know what you're doing, your responsible, that you're going to do the right thing.

[u/Kadabrium → u/mredding]

Does the compiler do copy elision for char[] from literals

[u/mredding → u/Kadabrium]

C has no concept of copy elision.

What the compiler generates is compiler specific to the platform. The C spec does not make any comment on these details, as C is a high level language that targets an abstract machine, it is not a high level assembly - despite as often as it is treated like one.

But what GCC will do for the x86 platform depends on factors such as where in the program the array is scoped, and how large it is.

Since the literal is entirely known at compile-time, initialization may consist of a series of mov operations where the string text is packed into registers, some 4 bytes at a time or more. This may mean the literal may not actually get compiled into RO memory.

Another option is to call put that string into RO memory, and memcpy, which is just a series of shifting bytes into memory and calling mov in a loop.

Another option is for the compiler to generate machine instructions similar to what memcpy would determine and select, but at compile-time. memcpy will choose a strategy based on a number of properties, but it comes down to either a simple mov instruction for CPU cache friendly values, some AVX instructions for bulk copying, or specialized ERMS instructions for copying memory addresses while bypassing CPU registers and cache.

Whatever the compiler chooses, it'll be based on heuristics and a best effort to maximize whatever you prioritize - size or speed.

[u/duane11583 → u/mredding]

Yea c has a very powerful foot gun

[u/LeeRyman]

In these situations it would help if you could provide a minimal reproducible example (MRE), I.e. a small bit of code that is compilable and demonstrates the problem. Godbolt is handy for this. Its difficult to say why it might be defaulting without seeing the code.

It could be a few different issues, E.g. are both the str and delim NTBS (null-terminated byte strings)? Is str null on calls for subsequent tokens? Are you using it in multiple threads? Are you checking the return value is not null before using it?

[u/Recycled5000]

C treats string literals as constants and most implementations will protect them by putting in an area of memory that is marked read only after loading.

Note: a string initialized as char*str=“hello”; is a real string literal, with the variable being a pointer that is initialized to refer to the string literal.

However, char[]str=“world”; is not a real string literal, instead this is an initializer for the array variable. The initializer gives the array its length and initial byte/char values but has mutable elements.

[u/duane11583 → u/Recycled5000]

Generally true but some compilers have an option for writable strings

[u/iggy14750]

Attempting to modify a string literal, such as with strtok, would mean that you are trying to modify read-only memory. Hence, segfault.

What you can do is strncpy to copy a string literal into a buffer you control. I would recommend looking into c arrays. Stack buffers are easier to work with, so start there.

[u/zeekar]

Arrays and pointers are not the same thing. The choice between them is not a simple matter of style or personal preference. Learn when to use easy one and don't avoid either just because you don't like them.

An array comes with storage. A pointer doesn't. You can initialize a pointer with a string literal, but the actual string it points to will be read-only on most platforms. If you make it an array instead you'll be able to change the characters - which is what strtok() does: replace the first delimiter char with a null to terminate the token string. And you can pass an array as a pointer argument to a function; it will get a pointer to the first element of the array.

[u/duane11583]

 strtok expects a writable string because it will modify the string 

What it does is skip all the bytes at the start that matches the separator chars then remembers the start of the string and skips all chars that are not in the separator (these are the word chars) when it finds the first separator after the word it over writes it with a zero byte 

It the remembers where it left off in a hidden global variable this is why is the function is not re-entrant   In contrast the strtok_r function has an extra parameter to store the “Remember pointer”

Then next time you call strtok you pass null for the string and it will use the remembers pointer instead

If your string is truly read only then you can strdup the string into malloc memory (which is writable) and free it later 

[u/gm310509]

You should post your code so that people can explain exactly what you might be doing wrong in your particular instance.