The word 'we' is not beyond his notice. It's probably another subtle way of roping him into the ordeal.

This time, he doesn't make a fuss about mincing words when he peers in his friend's direction. "Yes. We don't have time to dawdle, do we? It'll be grown soon enough." It's about as close to a complete concession as he will dare go.