Creativity vs. Toil

On knowing ourselves

A significant part of being a good software engineer, as well as a good engineering manager, involves understanding the psychology that plays out for an individual developer. As engineers, improvement comes not just from the acquisition of new skills and knowledge, but also from building a meta-game of how we strategize and navigate through difficulty. This extends to understanding ourselves.

So it’s valuable to know what may trigger distraction or frustration in the course of work. In some cases, these can be wholly negative experiences. Things like being interrupted out of focus mode or hitting a significant unanticipated problem always feels bad in a way that can disrupt progress on a project. We learn to recognize such experiences so that we can consider how to prevent or navigate around them.

But there are more complex cases: those experiences that can be welcome in one context and unpleasant in another. There are many of these, but one such dynamic I’ve been pondering lately is the relationship between toil and creativity. I believe there is fundamental tension between two sorts of work, where each can be satisfying and rewarding in their own rite, yet they feel horrible to do at the same time.

Toil, for the purposes of this line of thinking, is work which just needs to get done. When you have a task, you work away at it, and progress is made. Other work involves thinking creatively about what to do and how to do it. This work is more creatively demanding and more unpredictable in its progress. Most white collar jobs have aspects of both, and engineering is no exception: there is a side of our work that feels like inventing and architecting, and there is a side of it that feels like digging holes.

Everyone knows that engineers enjoy the creative challenges of the job, whereas to toil almost implies a loss of agency or humanity. But I was painting shelves recently and it struck me how enjoyable the work actually was. There was a feeling of steady progress, a feeling of industriousness. It’s not that we don’t enjoy toil. As long as you have conviction you’re working on the right thing, toil can feel like the incarnation of getting shit done. As a programmer, these may even be the periods when the most lines of code are written. I like this feeling of executing, this sense of being automatic.

I was on a project and working in this mode when I encountered an important question about product behavior which I hadn’t considered ahead of time, and seeing it created a moment of dread. But why did I experience that dread? Considering product behavior, making strategic and user-facing decisions, and then planning how to build is the very creative work which I enjoy and take pride in. Nonetheless, my first feeling was frustration.

I realized later that the problem was not in one type of work or the other, but in their intermixing. I thought I was in the get-shit-done mode of toiling, and I got yanked right out of it. This is the fundamental tension, and it led me to wonder:

• Do other jobs deal with this internal conflict less frequently? If so, why? How?

• Does the nature of programming necessitate dealing with this frustration?

• What can a developer do about it?

Do other jobs deal with this?

In considering the other roles at our company, I don’t think they deal with the same degree of intermixing of toil and creativity that developers do, even though most knowledge work surely involves both. The sales team, for instance, has a lot of execution to do, and a lot of repetitions to sustain. They execute on a plan and maintain energy to work across many deals a day. Their creative work seems to come in the exceptions like making judgment calls about how to plan steps for a tricky deal, or in the systematic thinking like revisiting their demo playbook. In general, they do seem to not mix the work for the most part. When they’re on calls and sending follow-up notes, they’re very much not looking to rethink the plan and the playbook.

Recruiting has a similar dynamic. You hope to square away the creative judgment calls prior to the toil: What are the stages of our process? What are the formats of our interview? What are the criteria for evaluation? What should the content of this rejection email be? To the extent possible, we try to answer these things ahead of time so that we can efficiently execute within the system. Then, periodically, we take the observations from doing so and zoom back out on the creative aspect. Critically, much like sales, we save cognitive energy so that we are ready and able to home in on tricky cases and exceptions as they come up.

Why don’t software engineers do this more effectively? Why can’t we simply execute within a plan or system and save the refactors, the product planning, and more for later in the day or another day entirely? Maybe our work is not categorically different, but the degree of the challenge does seem to be distinct.

Is it fundamental to software?

A fundamental trait of software engineering as a practice is the unpredictability that comes as a consequence of novelty. Software is free to copy and to reuse. In most cases, when something we need is perfectly solved by an existing library, tool, or product, we simply install it–we don’t write it anew. If software engineers are involved, there usually must be something particular about the goals or the context that isn’t already solved. Moreover, the challenge of software is rarely simply one of implementing the answer to a question–it’s usually figuring out what the answer is, what the precise intention is. We are almost always, in some way or another, working under conditions of novelty.

Contrast this with recruiting. Even if your process were exactly identical to another company’s, you’d still have to do the work. You can’t just import a library and call a function, even if it’s been perfectly solved. You have to send the email, you have to get on the call, you have to book the hotel.¹

So I do think there is a fundamental trait of our work which leads to unpredictability. But what is the connection between unpredictability and the creativity/toil association?

Put simply, the ability to separate the two (and thus avoid the tension) comes from the ability to toil consistently–to have a plan and to execute against it without disruption. This only works as long as the plan stays intact. For developers, it’s a fool’s errand to expect plans to stay intact. As a consequence, we don’t get to settle into a rhythm of getting shit done; we stay vigilant, paranoid even, for unforeseen problems and risks. It’s cognitively demanding in a way that can be very rewarding, but it’s also draining and leaves you occasionally wishing to have a simpler job.

So what do I do about it?

To be honest, I don’t think there is a silver bullet here. The best ideas I can offer are these:

Avoid it where you can

If you do know you have some high-precision focus work to execute on, and some creative planning or tricky architecture design to do, separate them. My point in this writing is that we simply cannot always do that, but in cases where you have the ability to separate the toil you should do so. Don’t make things harder than they have to be.

Reduce the demands of the toil

Another defining trait of software engineering is the level of depth and precision required in order to do the job. While we take pride in our ability and willingness to think precisely at low levels of abstraction, I also believe it is massive cognitive drain which makes it challenging to take on other items.

I recommend, wherever possible, to reduce the amount of brainpower required to toil. Break things down into smaller pieces, use reliable technologies and abstractions, and use any tooling that can help. Some tools, like programming languages, help to ease the challenge by raising the lowest level of abstraction you have to deal with. Even if you’re an expert with C, using Python can still free up your brain and your energy to be more prepared for the unexpected by raising the floor of abstraction at which you have to think. Just because you can do it all doesn’t mean you should. I believe AI copilots suit this purpose as well–one of their great benefits is not that they can write code that you wouldn’t be able to, but rather that you simply do not have to do it and can thus save your cognitive energy. For me personally, there is an amount of gas that stays in the tank by getting to offload some of the precise details of coding even in systems where I am perfectly capable of executing on the details.

Be mindful

Lastly, there is power in knowing something is coming even if you’re unable to prevent it. There will be moments of frustration and disruption. If you can be mindful of the situation and the impact it has on you, just knowing this can help you to recover and make progress. Perhaps it’s unsatisfying, but knowing one’s self is itself an important tool in dealing with such challenges, and the challenges will continue to come. In these cases, I recommend taking a moment to acknowledge the disruption and even planning to go for a walk or take a break to help smooth over the disruption.

1. Maybe booking hotels is something that soon won’t require a human to execute, but I resisted the tiny urge to turn this into an essay on AI agents or how they might impact other fields. I leave that as an exercise to the bored commenter.