Dream a Little Dream of…Deep Learning?

United thematically by seemingly infinite dimensions, formless creatures, and figures melting at the seams, the works of Salvador Dali are perhaps best described as hallucinatory snapshots. Dali, enigmatic in death as in life, possessed a distinct creative genius and painting style that captured the thin line between reality and fantasy

Examples of Salvador Dali’s Surrealist Works (From left to right: Galatea of the Spheres, Soft Construction with Boiled Beans, & The Burning Giraffe)

While the sources of Dali’s artistic visions have been the subject of a great deal of scholarship and debate, researchers at the Massachusetts Institute of Technology have turned their attention to Dali’s peculiar sleep practices. Like Thomas Edison, Edgar Allen Poe, and Ludwig van Beethoven, Dali believed strongly in the creative impulses unleashed during what is now referred to as hypnagogia – the “semi-lucid sleep stage” that occurs between wakefulness and the Rapid Eye Movement (REM) stage. Each night, Dali would go to bed with a steel ball in his hand. As his muscles relaxed in the transition from hypnagogia to REM, the ball would drop to the floor and wake him – thereby allowing Dali to recall and record his hypnagogic ideas, dreams, and visions. Hypnagogia has typically been studied within the context of narcolepsy and other sleep disorders, but it is experienced by every human being as he or she falls asleep. Cambridge University researcher Valdas Noreika describes the hypnagogic state as a “natural fragmentation of consciousness,” in which the brain produces vivid imagery, abstract scenes, audio experiences, and rapid thoughts without any particular connection to one another. Given that the hypnagogic state was of such importance to many of the world’s intellectual and cultural polymaths, it is perhaps no surprise that unlocking the potential of this unconstrained state of being is of tremendous interest to modern neuroscientists. In the past decade, tangible and previously unthinkable strides in the field have been facilitated by advanced brain activity mapping tools and deep-learning technologies.

In 2018, the aforementioned team of researchers at MIT successfully developed and tested an automated version of Dali’s dream capture system. The device, known as Dormio, is a piece of wearable technology that uses biosignal detection (muscle relaxation, heart rate, skin conductivity, etc) to determine when the wearer is transitioning from hypnagogia to REM. At the transitional point, the Dormio signals to either an accompanying “social robot,” known as Jibo, or a Jibo-based smartphone app to emit audio intended to rouse the wearer back to a semi-sleeping state. Jibo prompts the wearer to recall as much as possible about their dreams, recording them as they do so. Jibo can also voice basic categorical and item words – “fork,” for example, or “rabbit” – to test if the wearer’s next iteration of dreams will have such cues integrated into them. Though seemingly abstract, Dormio and its underlying neuroscience have been featured at the National Academy of Sciences, at the 2018 CHI Conference, and even on ABC’s 60 Minutes as a potential revolution in sleep-tracking technology and a critical step toward understanding the human mind’s unconscious processes.

MIT’s Dormio

Rather than risk the influence of so-called “hypnagogic amnesia” on collected data, other hypnagogia researchers have chosen to bypass human recall entirely and focus instead on measurable brain waves. At the University of Texas at Austin’s Cognitive Neuroscience Lab, for example, Daniel Oldis and David M. Schnyer are leveraging electromygram (EMG) and nerve conduction diagnostic tools to discover parallels in brain activity during states of sleep and of wakefulness. Oldis and Schnyer have been able to match electrical nerve impulses sent to various limbs or to the mouth to trigger speech that are consistent across hypnagogic semi-consciousness and waking consciousness. Identified impulses are then modeled on a digital avatar to “mimic the individual’s dream movement.”

A test subject’s dream movement avatar from the University of Texas at Austin’s Cognitive Neuroscience Lab

In a similar vein, neuroscientists and computer scientists at the University of California-Berkeley, under the leadership of Professor Jack Gallant, have combined functional Magnetic Resonance Imaging (fMRI) and deep-learning computing to reconstruct dynamic images from brain activity, specifically blood flow. The underlying artificial intelligence learns and iterates on the patterns of the brain’s reaction as it “sees” certain images or image types, thereby enabling it to categorize and attempt to recreate the reaction in pixel-image form.

In 2018, Kyoto University Professor Yukiyasu Kamitani built on Gallant’s work to produce an optimized deep-learning technique for independent computer construction of “internal images” of the mind. Kamitani  believes his “approach can be extended to reconstruct diverse types of subjective states, such as illusions, hallucinations, and dreams, thereby providing a new window into internal contents of the brain.” Five years prior, Kamitani received recognigion for developing a technique for scanning the brains of napping humans for prior-exposure image recognition patterns and algorithmically translating the basic content of the related dreams into short videos. Kamitani’s process achieved a 70% accuracy rate in comparison to test subjects’ content recall.

Alphabetic image reconstruction examples from Kamitani’s neural network thought-to-image process (results from three subjects)

As with any new technology or new application of existing technologies (in this case, medical diagnostics), there is a tendency to develop an unrealistic view of the near-term horizon. While some researchers claim that a consumer-targeted device for tracking and decoding dream activity will become available in less than five years, any device produced with even the most advanced science currently available will not be able to record or replay any imagery. Instead, it would more likely be able to tell a user the general content category of their dreams – people, for example, or birds.

“We are opening a window into the movies in our minds.”

Professor Jack Gallant, University of California-Berkeley

Even in light of these more limited parameters, however, the current direction of dream-focused technological development is exciting not only for its apparent ability to turn science fiction into reality, but also for its potentially far-reaching implications. For instance, so-called “dream decoding” could be used to better understand, or even communicate, with non-verbal individuals, coma patients, and those who are otherwise unable to express the products of their internal machinations. A window into the activity of the unconscious brain would also provide unprecedented insights into the human psyche, imagination, learning abilities, and memory formation processes. By opening up the “mind’s eye,” we will be opening ourselves to an entirely new conception of the mind itself.


  1. I find this topic fascinating and a little scary. Dreams are our subconscious states. Our memories can sometimes be faulty, but our brain waves don’t lie. To me, it sounds like these companies are using technology to execute Freud’s goal of unlocking people’s subconscious. I think it would be interesting if Dormio was used on people who keep the same sleep schedule as Tim Cook where people wake up at 3:45am then take 30 minute naps throughout the day. I also believe this could be a double edged sword. While it would be neat, it also has the potential be creepy and invasive because you would have access to Tim Cook’s mind’s eye. The Japanese research takes it one step further and is basically a scientifically based hypnosis. The tech resembles a sci-fi movie where people are able to hack other people’s dreams and control images in your mind.

  2. jimhanrahan7 · ·

    This is insane, and until the last paragraph I was uncertain how this would ever get funding. The idea of communicating with coma patients could represent a huge boon to the medical field. This post also reinforces for me the idea of investing in “moonshots.” MIT and other institutions set up these well-funded incubators and give their participants free reign. What they develop, at face value, could appear to have zero utility, no market opportunity, but its underlying technology could have life-changing ramifications. It’s the ultimate brainstorm exercise. I wish more of our invested capital went towards programs like this.

  3. This was a truly fascinating post! Sleeping and dreaming is definitely not an area which I imagined held a lot of innovation. However, your post completely changed my mind! I can see so many opportunities for people to use this technology to better understand their subconscious and how it processes the data it received during the day. In addition, the medical application of being able to more accurately determine brain death could be enormously beneficial to the medical community. I recently saw a video where people used technology similar to the alphabetic image reconstruction to demonstrate that when they ask people to watch a video of an elephant walking across the screen they were able to recreate that video (albeit in very blurry resolution) through FRMI readings of the watchers’ brain. This really makes me wonder how many years until we’re able to accurately visualize people’s thoughts?

  4. adurney1 · ·

    Interesting post! I think this is toeing the line of creepy or cool. When reflecting on my own dreams, I am not entirely sure I would want that level of access to my subconscious. I agree with Jim though. If this technology can be used for coma patients, this would be ground breaking. Giving loved ones the ability to communicate with a coma patient would be priceless

  5. masonpeterman · ·

    This was an incredibly interesting topic, and one that I’ve thought about often. Dreams are such an interesting part of the human experience, and up until this point they seem to be shadowed in uncertainty. The brain is immensely complex and its really cool to see advanced technologies beginning to unlock some of the secrets of how our mind works. I do think that starting to analyze our consciousness with computers is beginning to trend towards a creepy application given the personal and subconscious nature of dreams. However, if this type of research can be used to help those with narcolepsy or sleeping troubles, I think it’s important work that should be continued. I don’t often remember my dreams and knowing that it happens every night, I think it would be super interesting to be able to know what’s on my mind while I’m sleeping. I don’t know how I feel about having to wear a device when going to sleep, but this research is very interesting. Thanks for bringing some of this work to light, great post!

  6. shannonbenoit5 · ·

    This was so cool to read! I always have wild dreams but forget them so fast, and wished that there was a way to record my dreams. Sometimes when I wake up I write down the general gist of what I was dreaming about on a note on my phone before I can forget. They have always been a subject of fascination for me. I do agree, though, that this does toe the creepy line, especially if this information was somehow leaked and other people would be able to have access to my subconscious (and we thought the Facebook data was bad). The application for communicating with patients in a coma would be absolutely amazing though!

%d bloggers like this: