The impossible EFAULT
I have written a program; suppose it's called
(While the program is written in Haskell, I don't think that's particularly relevant to this post.)
(EDIT: Reproducer can be found here.)
(EDIT 2: Diagnosis by
int-e on irc here.)
worker starts a bunch of copies of a script.
Under normal circumstances this script sets up a container using Linux cgroups and Linux user namespaces, but none of that is relevant because the strange behaviour in question occurs just fine without all of that -- in fact, we'll let it start the following script, say
#!/bin/bash sleep 10
Clearly, there is no weird behaviour here, assuming that the system has
/bin, and mine does.
The copies of
sleep.sh are started by passing
process library does this for me.)
The thing is, occasionally (once every 5 to 10 invocations of
EFAULT ("Bad Address").
The manpage for
posix_spawnp says that:
The posix_spawn() and posix_spawnp() functions fail only in the case where the underlying fork(2), vfork(2) or clone(2) call fails; in these cases, these functions return an error number, which will be one of the errors described for fork(2), vfork(2) or clone(2).
In addition, these functions fail if:
ENOSYS Function not supported on this system.
Okay, so I should look for
clone(2) to figure out what goes wrong, right?
Or, in any case, none of those manpages mention
I've looked through the source code of
posix_spawnp in glibc and it at least doesn't throw
EFAULT directly; presumably, one of the subroutines it calls does.
glibc is large and I don't think looking through the entire call tree will be very productive, so I tried to diagnose the issue from the outside instead.
And this is where the weirdness starts.
Whenever my program encounters
posix_spawnp, it prints
Oops EFAULT; hence grepping for
EFAULT gives output precisely if the error occurred in this run.
I get the following observations:
./worker 2>&1 | grep EFAULT: errors occur.
./worker 2>&1 | grep EFAULT | cat: errors DO NOT occur.
./worker 2>&1 | grep --line-buffered EFAULT | cat: errors occur.
./worker 2>&1 | grep --line-buffered EFAULT: errors occur.
("errors occur" means that once every few executions I get output indicating that
EFAULT occurred; in the negative case I've run it for >20x the number of invocations that are necessary to produce
EFAULT in the other cases, without any
The only situation in which
posix_spawnp seems to always succeed, is when
stdout of the process that
worker's output is piped to, is block-buffered.
But this makes no sense: there shouldn't even be a reasonable way in which
worker can even determine whether this is the case!
Surely it can distinguish between
./worker | cat and
isatty(3) -- this is precisely what
grep does when not passed
--line-buffered), but in all of the above cases the output is piped to another process anyway.
This is already spooky, but it gets even spookier: if I replace the invocation of
./sleep.sh by an invocation of
sleep (i.e. removing the indirection of the shell script), errors occur in none of the above setups.
Somehow, starting a script is different from starting a native process (and changing
sleep.sh doesn't change anything).
posix_spawnp shouldn't care what it is starting!
That's the job of the loader, as far as I know.
So what gives?
I'll try to reduce my own program to a minimal reproducer, and if I find anything I'll post an update to this post.
In the meantime, spookiness.
snap-server modifies the environment to set the locale, and
setenv(3) is not atomic.
In particular, it breaks
execve(2) when they race, and this is what happens.
All possible solutions to this problem are hacks.