When a pipeline's output is just a single value, there's usually no need or benefit in using xargs. A simple command substitution will do.
I said "usually" above because xargs treats quotes specially and that could be useful in some scenarios (though not this one, which only produces numeric strings).
expr $(ls -l | wc -l) - 3
Why you're using ls' long format just to count lines is puzzling, but not germane to my point.