I always knew we humans have a rather tenuous grip on the concept of time, but I never realized quite how tenuous it was until a couple of weeks ago, when I attended a conference on the nature of time organized by the Foundational Questions Institute. This meeting, even more than FQXi’s previous efforts, was a mashup of different disciplines: fundamental physics, philosophy, neuroscience, complexity theory. Crossing academic disciplines may be overrated, as physicist-blogger Sabine Hossenfelder has pointed out, but it sure is fun. Like Sabine, I spend my days thinking about planets, dark matter, black holes—they have become mundane to me. But brains—now there’s something exotic. So I sat rapt during the neuroscientists’ talks as they described how our minds perceive the past, present, and future. “Perceive” maybe isn’t strong enough a word: our minds construct the past, present, and future, and sometimes get it badly wrong.

Neuroscientist Kathleen McDermott of Washington University began by quoting famous memory researcher Endel Tulving, who called our ability to remember the past and to anticipate the future “mental time travel.” You don’t use the phrase “time travel” lightly in front of a group of physicists for whom the concept is not a convenient metaphor but a very real possibility. But when you hear about how our minds glide through time—and how our memory provides a link not only to the past but also to the future—you see Tulving’s point.

McDermott outlined the case of Patient K.C., who has even worse amnesia than the better-known H.M. on whom the film Memento was based. K.C. developed both retrograde and anterograde amnesia from a motorcycle crash in 1981. (The literature doesn’t say whether he was wearing a helmet, but let this be a lesson.) He can’t remember anything that happened more than a few minutes ago. He retains facts and skills, but can’t remember actually doing anything or being anywhere.

Tellingly, not only can he not recall the past, he can’t envision the future. When researchers ask him to picture himself somewhere he might go, he says that all he sees is “a big blankness.” Another patient McDermott has worked with can explain the future in the abstract, but says he can’t imagine himself in it.

To investigate the perception of past and future in people without brain injuries, McDermott did fMRI brain scans of 21 college students, asking them to recall a specific incident in their past and then envision themselves in a specific future scenario. Subjectively, the two feel very different. Yet the scans showed the same patterns of activity. Areas scattered all over the brain lit up; our temporal perception is distributed. As a control, McDermott also asked the students to remember events involving Bill Clinton (presumably, ones they were not personally involved in), and the patterns were very different. In a follow-up study, McDermott asked 27 students to anticipate an event in both a familiar and an unfamiliar place. The brain scan for the familiar one resembled the one for the act of remembering; the unfamiliar one was the odd man out.

The bottom line is that memory is essential to constructing scenarios for ourselves in the future. Anecdotal evidence backs this up. Our ability to project forward and to recollect the past both develop around age 5, and people who are good at remembering also report having vivid thoughts about the future.

McDermott’s colleague Henry Roediger studies metacognition—thinking about thinking. We express varying degrees of confidence in our memories. How we do this is clearly an issue for the court system. The N.J. Supreme Court recently tightened standards on the consideration of eyewitness testimony, citing the risk of false positives. Roediger pointed out that false negatives get less attention, but are equally bad. The worst eyewitnesses are full of passionate intensity, and the best lack all conviction. In both cases, innocent people can be sent to death row while the guilty walk.

Cognitive psychologists find that confidence sometimes correlates with accuracy, sometimes not. Roediger gave volunteers a memory word test. They had to study a list of words; afterwards, they were presented with a series of words and had to indicate whether each had been on the original list. They also had to say how confident they felt about their answer.

Whenever I hear about such tests, I brace myself for bad news. But Roediger said people actually did pretty well, and their confidence scores tracked the accuracy of their recall. Their blind spots were predictable. They systematically messed up, both in recall accuracy and self-assessment, when presented words that weren’t on the list but were synonyms of ones that were. The findings match what happens with eyewitnesses. We get things broadly right, but are easily confused by similar situations and faces.

It’s not that our memory is a glitchy wetware version of computer flash memory; it’s that the computer metaphor just doesn’t apply. Roediger said we store only bits and pieces of what happened—a smattering of impressions we weave together into feels like a seamless narrative. When we retrieve a memory, we also rewrite it, so that the time next we go to remember it, we don’t retrieve the original memory but the last one we recollected. So, each time we tell a story, we embellish it, while remaining genuinely convinced of the veracity of our memories.

So go easy on your friend who caught the 150-pound catfish. He wasn’t consciously lying, which is why he spoke with conviction, but that still doesn’t mean you should swallow his tale. To confuse is human; to accept we confuse, divine.

Speaking of fish, as neuroscientist Malcolm MacIver of Northwestern once put it to me, electric fish are the fruit flies of neuroscience—model organisms for studying how we sense the world. MacIver told the FQXi conference about his astoundingly comprehensive, leave-no-stone-unturned study of a species of Amazonian electric fish, using everything from supercomputer fluid simulations to an working model of the fish (captured in this video) and even an art installation.

The fish generates an electric field of about 1 millivolt per centimeter at a frequency that ranges from 50 to 2000 hertz. Water fleas, its prey, give themselves away by disrupting the field. (You can build a proximity sensor based on this concept. I use one to control the lights in my study.) What gets ichthyologists flapping is that, when this fish is out hunting, it doesn’t swim straight ahead, but at a 30-degree angle to the axis of its body—a seemingly cuckoo behavior that nearly triples the water drag force.

But MacIver demonstrated that the orientation also increases the effective volume of water sensed by the electric field. The fish strikes a balance between mechanical and sensory efficiency. Generalizing this insight, he distinguished between two distinct volumes around an organism: its sensory volume (the region it can scan for prey) and its motor volume (the region it can directly reach). For this fish and most other aquatic animals, the two are comparable in size—there’d be no point in looking out any farther. A fish’s reach does not exceed its grasp.

For land animals, though, things are quite different: their sensory volume is much bigger than their motor volume, since light travels much farther in air than in seawater. So when our ancestors crawled out of the sea, they gained the opportunity to plan their behavior in advance. No longer restricted to reacting to immediate stimuli, they had time to take in the scene and deliberate before moving. Animals that could arbitrage the difference in sensory and motor volumes gained an evolutionary advantage.

MacIver speculated that this set the stage for the evolution of consciousness. After all, what is consciousness, but the ability to make plans and gain some advantage over our environment, rather than lurching from crisis to crisis? Psychologist Bruce Bridgeman proposed this view of consciousness in the early 1990s. MacIver elaborated in a post on his blog, Science Not Fiction, earlier this year.

The fun thing about neuroscience is that you can do the experiments on yourself. David Eagleman of the Baylor College of Medicine proceeded to treat us as his test subjects. By means of several visual illusions, he demonstrated that we are all living in the past: Our consciousness lags 80 milliseconds behind actual events. “When you think an event occurs it has already happened,” Eagleman said.

In one of these illusions, the flash-lag effect, a light flashes when an object moves past it, but we don’t see the two as coincident; there appears to be a slight offset between them. By varying the parameters of the experiment, Eagleman showed that this occurs because the brain tries to reconstruct events retroactively and occasionally gets it wrong. The reason, he suggested, is that our brains seek to create a cohesive picture of the world from stimuli that arrive at a range of times. If you touch your toe and nose at the same time, you feel them at the same time, even though the signal from your nose reaches your brain first. You hear and see a hand clap at the same time, even though auditory processing is faster than visual processing. Our brains also paper over gaps in information, such as eyeblinks. “Your consciousness goes through all the trouble to synchronize things,” Eagleman said. But that means the slowest signal sets the pace.

The cost of hiding the logistical details of perception is that we are always a beat behind. The brain must strike a balance. Cognitive psychologist Alex Holcombe at Sydney has some clever demonstrations showing that certain forms of motion perception take a second or longer to register, and our brains clearly can’t wait that long. Our view of the world takes shape as we watch it.

The 80-millisecond rule plays all sorts of perceptual tricks on us. As long as a hand-clapper is less than 30 meters away, you hear and see the clap happen together. But beyond this distance, the sound arrives more than 80 milliseconds later than the light, and the brain no longer matches sight and sound. What is weird is that the transition is abrupt: by taking a single step away from you, the hand-clapper goes from in sync to out of sync. Similarly, as long as a TV or film soundtrack is synchronized within 80 milliseconds, you won’t notice any lag, but if the delay gets any longer, the two abruptly and maddeningly become disjointed. Events that take place faster than 80 milliseconds fly under the radar of consciousness. A batter swings at a ball before being aware that the pitcher has even throw it.

The cohesiveness of consciousness is essential to our judgments about cause and effect—and, therefore, to our sense of self. In one particularly sneaky experiment, Eagleman and his team asked volunteers to press a button to make a light blink—with a slight delay. After 10 or so presses, people cottoned onto the delay and began to see the blink happen as soon as they pressed the button. Then the experimenters reduced the delay, and people reported that the blink happened before they pressed the button.

Eagleman conjectured that such causal reversals would explain schizophrenia. All of us have an internal monologue, which we safely attribute to ourselves; if we didn’t, we might think of it as an external voice. So Eagleman has begun to run the same button-blink experiment on people diagnosed with schizophrenia. He reported that changing the delay time did not cause them to change their assessment of cause and effect. “They just don’t adjust,” Eagleman said. “They don’t see the illusion. They’re temporally inflexible.” He ventured: “Maybe schizophrenia is fundamentally a disorder of time perception.” If so, it suggests new therapies to cajole the brains of schizophrenic patients into recalibrating their sense of timing.

In the experiment for which Eagleman is best known, he sought to find out why time passes more slowly when we’re scared. Does something really happen in the brain—for instance, the time resolution of perception speeds up—or do we just think it does, in hindsight? After brainstorming scare tactics that probably wouldn’t have passed muster with a university ethics committee, he hit upon asking volunteers to take one of those Freefall or Demon Drop rides you find in amusement parks. They wore a special watch whose digits counted up too quickly for people to register them under normal conditions—thinking that, if perception really did speed up, people would be able to read the digits.

Alas, they couldn’t. Although they consistently reported that the ride took about a third longer than it really did, this must have been a trick of memory; their hyperacuity was a mirage.

Our memory becomes distorted because our brains react more strongly to novelty than to repetition. Eagleman investigated this effect by asking volunteers to estimate the duration of flashes of light; those flashes that were the first in a series, or broke an established pattern, seemed to last longer. This feature of consciousness, like the 80-millisecond rule, explain so much about our daily experience. When we’re sitting through a boring event, it seems to take forever. But when we look back on it, it went by in a flash. Conversely, when you’re doing something exciting, time seems to race by, but when you look back on it, it stretched out. In the first case, there was little to remember, so your brain collapsed the feeling of duration. In the second, there was so much to remember, so the event seemed to expand. Time flies when you’re having fun, but crawls when you recollect in tranquility.

I suspect that this inverse relation in our perception of time also explains how our experiences shift as we age. When you’re a kid, you wake up and say to yourself: “I’ve got a whole day ahead of me. How will I possibly fill it all?” But when you’re an adult, it’s more like: “I’ve got a day ahead of me. How will I possibly get it all done?” And don’t get me started on how people swear that the first year of their baby’s life went by so fast. (A second child is usually enough to disabuse them.)

You can probably tell from my lengthy description of Eagleman’s talk that it seemed to zip by at the time. The physicists in attendance found it one of the highlights of the conference. Not only was it engrossing in its own right, it had some professional interest for them. All theories of physics begin with sense-data. As Eagleman said, “We build our physics on top of our intuitions.”

We also build our physics on a recognition of the limits of perception. The whole point of theories such as relativity is to separate objective features of the world from artifacts of our perspective. One of the most important books of the past two decades on the physics and philosophy of time, Huw Price’s Time’s Arrow and Archimedes’ Point, argues that concepts of cause and effect derive from our experience as agents in the world and may not be a fundamental feature of reality.

Time plays a variety of roles in physics, from defining causal sequences to giving a direction to the unfolding of the universe. How many of these roles are rooted in the contingent ways our brains perceive time? How might an alien being, who perceives time in a radically different way, formulate physics?

Brain image courtesy of Washington University of St. Louis. Fish image courtesy of Malcolm MacIver lab.