After years of starting the day with a tall morning coffee, followed by several glasses of green tea at intervals, and the occasional cappuccino after lunch, I quit caffeine, cold turkey. It was not something that I particularly wanted to do, but I had come to the reluctant conclusion that the story I was writing demanded it. Several of the experts I was interviewing had suggested that I really couldn’t understand the role of caffeine in my life – its invisible yet pervasive power – without getting off it and then, presumably, getting back on. Roland Griffiths, one of the world’s leading researchers of mood-altering drugs, and the man most responsible for getting the diagnosis of “caffeine withdrawal” included in the Diagnostic and Statistical Manual of Mental Disorders (DSM-5), the bible of psychiatric diagnoses, told me he hadn’t begun to understand his own relationship with caffeine until he stopped using it and conducted a series of self-experiments. He urged me to do the same.

For most of us, to be caffeinated to one degree or another has simply become baseline human consciousness. Something like 90% of humans ingest caffeine regularly, making it the most widely used psychoactive drug in the world, and the only one we routinely give to children (commonly in the form of fizzy drinks). Few of us even think of it as a drug, much less our daily use of it as an addiction. It’s so pervasive that it’s easy to overlook the fact that to be caffeinated is not baseline consciousness but, in fact, an altered state. It just happens to be a state that virtually all of us share, rendering it invisible.

The scientists have spelled out, and I had duly noted, the predictable symptoms of caffeine withdrawal: headache, fatigue, lethargy, difficulty concentrating, decreased motivation, irritability, intense distress, loss of confidence and dysphoria. But beneath that deceptively mild rubric of “difficulty concentrating” hides nothing short of an existential threat to the work of the writer. How can you possibly expect to write anything when you can’t concentrate?

I postponed it as long as I could, but finally the dark day arrived. According to the researchers I’d interviewed, the process of withdrawal had actually begun overnight, while I was sleeping, during the “trough” in the graph of caffeine’s diurnal effects. The day’s first cup of tea or coffee acquires most of its power – its joy! – not so much from its euphoric and stimulating properties than from the fact that it is suppressing the emerging symptoms of withdrawal. This is part of the insidiousness of caffeine. Its mode of action, or “pharmacodynamics”, mesh so perfectly with the rhythms of the human body that the morning cup of coffee arrives just in time to head off the looming mental distress set in motion by yesterday’s cup of coffee. Daily, caffeine proposes itself as the optimal solution to the problem caffeine creates.

Sign up to the long read weekly email

At the coffee shop, instead of my usual “half caff”, I ordered a cup of mint tea. And on this morning, that lovely dispersal of the mental fog that the first hit of caffeine ushers into consciousness never arrived. The fog settled over me and would not budge. It’s not that I felt terrible – I never got a serious headache – but all day long I felt a certain muzziness, as if a veil had descended in the space between me and reality, a kind of filter that absorbed certain wavelengths of light and sound.

I was able to do some work, but distractedly. “I feel like an unsharpened pencil,” I wrote in my notebook. “Things on the periphery intrude, and won’t be ignored. I can’t focus for more than a minute.”

Over the course of the next few days, I began to feel better, the veil lifted, yet I was still not quite myself, and neither, quite, was the world. In this new normal, the world seemed duller to me. I seemed duller, too. Mornings were the worst. I came to see how integral caffeine is to the daily work of knitting ourselves back together after the fraying of consciousness during sleep. That reconsolidation of self took much longer than usual, and never quite felt complete.

Humanity’s acquaintance with caffeine is surprisingly recent. But it is hardly an exaggeration to say that this molecule remade the world. The changes wrought by coffee and tea occurred at a fundamental level – the level of the human mind. Coffee and tea ushered in a shift in the mental weather, sharpening minds that had been fogged by alcohol, freeing people from the natural rhythms of the body and the sun, thus making possible whole new kinds of work and, arguably, new kinds of thought, too.

By the 15th century, coffee was being cultivated in east Africa and traded across the Arabian peninsula. Initially, the new drink was regarded as an aide to concentration and used by Sufis in Yemen to keep them from dozing off during their religious observances. (Tea, too, started out as a little helper for Buddhist monks striving to stay awake through long stretches of meditation.) Within a century, coffeehouses had sprung up in cities across the Arab world. In 1570 there were more than 600 of them in Constantinople alone, and they spread north and west with the Ottoman empire.

The Islamic world at this time was in many respects more advanced than Europe, in science and technology, and in learning. Whether this mental flourishing had anything to do with the prevalence of coffee (and prohibition of alcohol) is difficult to prove, but as the German historian Wolfgang Schivelbusch has argued, the beverage “seemed to be tailor-​made for a culture that forbade alcohol consumption and gave birth to modern mathematics”.

In 1629 the first coffeehouses in Europe, styled on the Arab model, popped up in Venice, and the first such establishment in England was opened in Oxford in 1650 by a Jewish immigrant. They arrived in London shortly thereafter, and proliferated: within a few decades there were thousands of coffeehouses in London; at their peak, one for every 200 Londoners.

To call the English coffeehouse a new kind of public space doesn’t quite do it justice. You paid a penny for the coffee, but the information – in the form of newspapers, books, magazines and conversation – was free. (Coffeehouses were often referred to as “penny universities”.) After visiting London coffeehouses, a French writer named Maximilien Misson wrote, “You have all Manner of News there; You have a good fire, which you may sit by as long as you please: You have a Dish of Coffee; you meet your Friends for the Transaction of Business, and all for a Penny, if you don’t care to spend more.”

London’s coffeehouses were distinguished one from another by the professional or intellectual interests of their patrons, which eventually gave them specific institutional identities. So, for example, merchants and men with interests in shipping gathered at Lloyd’s Coffee House. Here you could learn what ships were arriving and departing, and buy an insurance policy on your cargo. Lloyd’s Coffee House eventually became the insurance brokerage Lloyd’s of London. Learned types and scientists – known then as “natural philosophers” – gathered at the Grecian, which became closely associated with the Royal Society; Isaac Newton and Edmond Halley debated physics and mathematics here, and supposedly once dissected a dolphin on the premises.

The conversation in London’s coffee houses frequently turned to politics, in vigorous exercises of free speech that drew the ire of the government, especially after the monarchy was restored in 1660. Charles II, worried that plots were being hatched in coffeehouses, decided that the places were dangerous fomenters of rebellion that the crown needed to suppress. In 1675 the king moved to close down the coffeehouses, on the grounds that the “false, malicious and scandalous Reports” emanating therefrom were a “Disturbance of the Quiet and Peace of the Realm”. Like so many other compounds that change the qualities of consciousness in individuals, caffeine was regarded as a threat to institutional power, which moved to suppress it, in a foreshadowing of the wars against drugs to come.

But the king’s war against coffee lasted only 11 days. Charles discovered that it was too late to turn back the tide of caffeine. By then the coffeehouse was such a fixture of English culture and daily life – and so many eminent Londoners had become addicted to caffeine – that everyone simply ignored the king’s order and blithely went on drinking coffee. Afraid to test his authority and find it lacking, the king quietly backed down, issuing a second proclamation rolling back the first “out of princely consideration and royal compassion”.

It’s hard to imagine that the sort of political, cultural and intellectual ferment that bubbled up in the coffeehouses of both France and England in the 17th century would ever have developed in a tavern. The kind of magical thinking that alcohol sponsored in the medieval mind began to yield to a new spirit of rationalism and, a bit later, Enlightenment thinking. French historian Jules Michelet wrote: “Coffee, the sober drink, the mighty nourishment of the brain, which unlike other spirits, heightens purity and lucidity; coffee, which clears the clouds of the imagination and their gloomy weight; which illumines the reality of things suddenly with the flash of truth.”

To see, lucidly, “the reality of things”: this was, in a nutshell, the rationalist project. Coffee became, along with the microscope, telescope and the pen, one of its indispensable tools.

After a few weeks, the mental impairments of withdrawal had subsided, and I could once again think in a straight line, hold an abstraction in my head for more than two minutes, and shut peripheral thoughts out of my field of attention. Yet I continued to feel as though I was mentally just slightly behind the curve, especially when in the company of drinkers of coffee and tea, which, of course, was all the time and everywhere.

Here’s what I was missing: I missed the way caffeine and its rituals used to order my day, especially in the morning. Herbal teas – which are barely, if at all, psychoactive – lack the power of coffee and tea to organise the day into a rhythm of energetic peaks and valleys, as the mental tide of caffeine ebbs and flows. The morning surge is a blessing, obviously, but there is also something comforting in the ebb tide of afternoon, which a cup of tea can gently reverse.

At some point I began to wonder if perhaps it was all in my head, this sense that I had lost a mental step since getting off coffee and tea. So I decided to look at the science, to learn what, if any, cognitive enhancement can actually be attributed to caffeine. I found numerous studies conducted over the years reporting that caffeine improves performance on a range of cognitive measures – of memory, focus, alertness, vigilance, attention and learning. An experiment done in the 1930s found that chess players on caffeine performed significantly better than players who abstained. In another study, caffeine users completed a variety of mental tasks more quickly, though they made more errors; as one paper put it in its title, people on caffeine are “faster, but not smarter”. In a 2014 experiment, subjects given caffeine immediately after learning new material remembered it better than subjects who received a placebo. Tests of psychomotor abilities also suggest that caffeine gives us an edge: in simulated driving exercises, caffeine improves performance, especially when the subject is tired. It also enhances physical performance on such metrics as time trials, muscle strength and endurance.

True, there is reason to take these findings with a pinch of salt, if only because this kind of research is difficult to do well. The problem is finding a good control group in a society in which virtually everyone is addicted to caffeine. But the consensus seems to be that caffeine does improve mental (and physical) performance to some degree.

Whether caffeine also enhances creativity is a different question, however, and there’s some reason to doubt that it does. Caffeine improves our focus and ability to concentrate, which surely enhances linear and abstract thinking, but creativity works very differently. It may depend on the loss of a certain kind of focus, and the freedom to let the mind off the leash of linear thought.

Cognitive psychologists sometimes talk in terms of two distinct types of consciousness: spotlight consciousness, which illuminates a single focal point of attention, making it very good for reasoning, and lantern consciousness, in which attention is less focused yet illuminates a broader field of attention. Young children tend to exhibit lantern consciousness; so do many people on psychedelics. This more diffuse form of attention lends itself to mind wandering, free association, and the making of novel connections – all of which can nourish creativity. By comparison, caffeine’s big contribution to human progress has been to intensify spotlight consciousness – the focused, linear, abstract and efficient cognitive processing more closely associated with mental work than play. This, more than anything else, is what made caffeine the perfect drug not only for the age of reason and the Enlightenment, but for the rise of capitalism, too.

The power of caffeine to keep us awake and alert, to stem the natural tide of exhaustion, freed us from the circadian rhythms of our biology and so, along with the advent of artificial light, opened the frontier of night to the possibilities of work.

What coffee did for clerks and intellectuals, tea would soon do for the English working class. Indeed, it was tea from the East Indies – heavily sweetened with sugar from the West Indies – that fuelled the Industrial Revolution. We think of England as a tea culture, but coffee, initially the cheaper beverage by far, dominated at first.

Soon after the British East India Company began trading with China, cheap tea flooded England. A beverage that only the well-to-do could afford to drink in 1700 was by 1800 consumed by virtually everyone, from the society matron to the factory worker.

To supply this demand required an imperialist enterprise of enormous scale and brutality, especially after the British decided it would be more profitable to turn India, its colony, into a tea producer, than to buy tea from the Chinese. This required first stealing the secrets of tea production from the Chinese (a mission accomplished by the renowned Scots botanist and plant explorer Robert Fortune, disguised as a mandarin); seizing land from peasant farmers in Assam (where tea grew wild), and then forcing the farmers into servitude, picking tea leaves from dawn to dusk. The introduction of tea to the west was all about exploitation – the extraction of surplus value from labour, not only in its production in India, but in its consumption by the British as well.

Tea allowed the British working class to endure long shifts, brutal working conditions and more or less constant hunger; the caffeine helped quiet the hunger pangs, and the sugar in it became a crucial source of calories. (From a strictly nutritional standpoint, workers would have been better off sticking with beer.) The caffeine in tea helped create a new kind of worker, one better adapted to the rule of the machine. It is difficult to imagine an Industrial Revolution without it.

So how exactly does coffee, and caffeine more generally, make us more energetic, efficient and faster? How could this little molecule possibly supply the human body energy without calories? Could caffeine be the proverbial free lunch, or do we pay a price for the mental and physical energy – the alertness, focus and stamina – that caffeine gives us?

Alas, there is no free lunch. It turns out that caffeine only appears to give us energy. Caffeine works by blocking the action of adenosine, a molecule that gradually accumulates in the brain over the course of the day, preparing the body to rest. Caffeine molecules interfere with this process, keeping adenosine from doing its job – and keeping us feeling alert. But adenosine levels continue to rise, so that when the caffeine is eventually metabolised, the adenosine floods the body’s receptors and tiredness returns. So the energy that caffeine gives us is borrowed, in effect, and eventually the debt must be paid back.

For as long as people have been drinking coffee and tea, medical authorities have warned about the dangers of caffeine. But until now, caffeine has been cleared of the most serious charges against it. The current scientific consensus is more than reassuring – in fact, the research suggests that coffee and tea, far from being deleterious to our health, may offer some important benefits, as long as they aren’t consumed to excess. Regular coffee consumption is associated with a decreased risk of several cancers (including breast, prostate, colorectal and endometrial), cardiovascular disease, type 2 diabetes, Parkinson’s disease, dementia and possibly depression and suicide. (Though high doses can produce nervousness and anxiety, and rates of suicide climb among those who drink eight or more cups a day.)

My review of the medical literature on coffee and tea made me wonder if my abstention might be compromising not only my mental function but my physical health, as well. However, that was before I spoke to Matt Walker.

An English neuroscientist on the faculty at University of California, Berkeley, Walker, author of Why We Sleep, is single-minded in his mission: to alert the world to an invisible public-health crisis, which is that we are not getting nearly enough sleep, the sleep we are getting is of poor quality, and a principal culprit in this crime against body and mind is caffeine. Caffeine itself might not be bad for you, but the sleep it’s stealing from you may have a price. According to Walker, research suggests that insufficient sleep may be a key factor in the development of Alzheimer’s disease, arteriosclerosis, stroke, heart failure, depression, anxiety, suicide and obesity. “The shorter you sleep,” he bluntly concludes, “the shorter your lifespan.”

Walker grew up in England drinking copious amounts of black tea, morning, noon and night. He no longer consumes caffeine, save for the small amounts in his occasional cup of decaf. In fact, none of the sleep researchers or experts on circadian rhythms I interviewed for this story use caffeine.

Walker explained that, for most people, the “quarter life” of caffeine is usually about 12 hours, meaning that 25% of the caffeine in a cup of coffee consumed at noon is still circulating in your brain when you go to bed at midnight. That could well be enough to completely wreck your deep sleep.

I thought of myself as a pretty good sleeper before I met Walker. At lunch he probed me about my sleep habits. I told him I usually get a solid seven hours, fall asleep easily, dream most nights.

“How many times a night do you wake up?” he asked. I’m up three or four times a night (usually to pee), but I almost always fall right back to sleep.

He nodded gravely. “That’s really not good, all those interruptions. Sleep quality is just as important as sleep quantity.” The interruptions were undermining the amount of “deep” or “slow wave” sleep I was getting, something above and beyond the REM sleep I had always thought was the measure of a good night’s rest. But it seems that deep sleep is just as important to our health, and the amount we get tends to decline with age.

Caffeine is not the sole cause of our sleep crisis; screens, alcohol (which is as hard on REM sleep as caffeine is on deep sleep), pharmaceuticals, work schedules, noise and light pollution, and anxiety can all play a role in undermining both the duration and quality of our sleep. But here’s what’s uniquely insidious about caffeine: the drug is not only a leading cause of our sleep deprivation; it is also the principal tool we rely on to remedy the problem. Most of the caffeine consumed today is being used to compensate for the lousy sleep that caffeine causes – which means that caffeine is helping to hide from our awareness the very problem that caffeine creates.

The time came to wrap up my experiment in caffeine deprivation. I was eager to see what a body that had been innocent of caffeine for three months would experience when subjected to a couple of shots of espresso. I had thought long and hard about what kind of coffee I would get, and where. I opted for a “special”, my local coffee shop’s term for a double-​shot espresso made with less steamed milk than a typical cappuccino; it’s more commonly known as a flat white.

My special was unbelievably good, a ringing reminder of what a poor counterfeit decaf is; here were whole dimensions and depths of flavour that I had completely forgotten about. Everything in my visual field seemed pleasantly italicised, filmic, and I wondered if all these people with their cardboard-sleeve-swaddled cups had any idea what a powerful drug they were sipping. But how could they?

They had long ago become habituated to caffeine, and were now using it for another purpose entirely. Baseline maintenance, that is, plus a welcome little lift. I felt lucky that this more powerful experience was available to me. This – along with the stellar sleeps – was the wonderful dividend of my investment in abstention.

And yet in a few days’ time I would be them, caffeine-tolerant and addicted all over again. I wondered: was there any way to preserve the power of this drug? Could I devise a new relationship with caffeine? Maybe treat it more like a psychedelic – say, something to be taken only on occasion, and with a greater degree of ceremony and intention. Maybe just drink coffee on Saturdays? Just the one.

How Nespresso’s coffee revolution got ground down

Read more

When I got home I tackled my to-do list with unaccustomed fervour, harnessing the surge of energy – of focus! – coursing through me, and put it to good use. I compulsively cleared and decluttered – on the computer, in my closet, in the garden and the shed. I raked, I weeded, I put things in order, as if I were possessed. Whatever I focused on, I focused on zealously and single-mindedly.

Around noon, my compulsiveness began to subside, and I felt ready for a change of scene. I had yanked a few plants out of the vegetable garden that were not pulling their weight, and decided to go to the garden centre to buy some replacements. It was during the drive that I realised the true reason I was heading to this particular garden centre: it had this Airstream trailer parked out front that served really good espresso.

This is an edited extract from This Is Your Mind on Plants: Opium-Caffeine-Mescaline by Michael Pollan, published by Allen Lane on 8 July and available at guardianbookshop.co.uk