" "I would prefer not to."
"You will not?"
"I prefer not." " - Herman Melville, Bartleby, the Scrivener
I love reading stories about people that don't put forth the true effort to do their work. It kind of reminds me of students in high school that would mouth off to the teachers just to be the class clown. It certainly seems like the narrator is reacting the same way that an annoyed teacher would, by simply reprimanding the cause of the annoyance.

Do you really think Bartleby is the one with the problem? After all, the narrator is the boss, but he's an absolute pushover.
Also, I was talking to Mr. Esaias about the story, and he said it was written around and partly in response to "Civil Disobedience" and the idea that there are some people, even with the threat if violence, that you just cannot sway.