Deep Learning for EEG: CNNs, RNNs, and Transformers

A Neural Network Looking at Neural Networks

Here's something that should make you pause for a second. When you train a convolutional neural network on EEG data, you are building an artificial neural network whose entire job is to find patterns in the output of a biological neural network. One network of neurons, made of silicon and matrix multiplications, learning to read another network of neurons, made of lipid membranes and electrochemical gradients.

The artificial one has maybe 10,000 parameters. The biological one has 86 billion neurons with roughly 100 trillion synaptic connections.

And yet, somehow, the artificial one is getting pretty good at this.

For most of the history of brain-computer interfaces, decoding EEG signals required a human expert to sit between the brain and the computer. That expert would decide which features to extract. Band powers. Spatial filters. Time-domain statistics. Connectivity metrics. The expert's neuroscience knowledge determined what the algorithm could even see. If the expert didn't know to look for a particular pattern, the algorithm was blind to it.

Deep learning flipped that entire paradigm. Instead of telling the algorithm what to look for, you give it labeled examples and let it figure out the patterns on its own. And the patterns it finds are often things no human neuroscientist would have thought to extract.

This guide is about the three families of deep learning architectures that are reshaping EEG analysis: convolutional neural networks, recurrent neural networks, and transformers. What each one brings to the table, where each one struggles, and why training any of them on brain data is fundamentally harder than training on images, text, or audio.

Why EEG Needed Deep Learning (And Why It Took So Long)

To understand why deep learning matters for EEG, you need to understand what came before it.

The classical approach to EEG decoding is a two-stage pipeline. Stage one: a neuroscientist designs features. You compute the Fast Fourier Transform, extract power in the alpha band (8-13 Hz), beta band (13-30 Hz), theta band (4-8 Hz). You apply Common Spatial Patterns to find the linear combinations of channels that best separate your classes. You compute Hjorth parameters, coherence between electrode pairs, maybe some wavelet coefficients. Stage two: you feed those features into a classifier. Support vector machine. Linear discriminant analysis. Random forest.

This worked. It still works. For well-studied paradigms like motor imagery or P300 detection, a carefully engineered classical pipeline can achieve 75-85% accuracy.

But it has a ceiling. And that ceiling is defined by the limits of human neuroscience knowledge.

Think about it this way. The EEG signal contains all the information about the brain's electrical state that makes it through the skull and scalp. Some of that information lives in frequency bands we've named and studied for a century. Some of it lives in spatial patterns that Common Spatial Patterns can capture. But some of it, maybe a lot of it, lives in transient, nonlinear, cross-frequency, cross-channel interactions that don't have names yet. Patterns we haven't discovered because we haven't had the mathematical tools to find them.

Hand-crafted features can only capture what you already know about the signal. Deep learning can capture what you don't know yet.

So why did it take until roughly 2016 for deep learning to gain traction in EEG research, when it had already transformed computer vision (2012) and natural language processing (2013)?

Three reasons. First, EEG datasets are tiny by deep learning standards. ImageNet has 14 million labeled images. The largest public EEG datasets have maybe a few hundred subjects with a few sessions each. Deep learning is hungry, and EEG data is scarce.

Second, EEG has a miserable signal-to-noise ratio. An eye blink produces voltage fluctuations 10 to 50 times larger than the neural signals you actually care about. Training a neural network on data that is mostly noise requires architectural choices that are very different from what works on clean images.

Third, the structure of EEG data is unusual. It's not an image (2D spatial grid). It's not a sentence (1D token sequence). It's a multivariate time series with spatial structure (electrode positions on the scalp), temporal structure (the evolution of the signal over time), and spectral structure (information encoded in different frequency bands simultaneously). No off-the-shelf deep learning architecture was designed for this kind of data.

Between 2016 and now, the field solved all three problems. And the results have rewritten the accuracy benchmarks on almost every EEG task.

CNNs: Teaching Convolutions to Read Brainwaves

Convolutional neural networks were the first deep learning architecture to seriously outperform classical methods on EEG. But not just any CNN. The architectures that work for brain data look very different from the ones that classify cats and dogs.

The EEGNet Architecture (And Why It Matters So Much)

EEGNet, published by Lawhern and colleagues in 2018, is the most important single architecture in deep learning for EEG. Not because it's the most accurate model ever built, but because it showed that a single, compact architecture could work across multiple BCI paradigms without modification. Motor imagery, P300, steady-state visual evoked potentials. Same network. Same hyperparameters. Different data.

Here's how it works, and this is genuinely elegant.

Step one: temporal convolution. The first layer applies a set of 1D convolution filters across the time dimension of each channel independently. Each filter is a small sliding window (typically 64 samples at 128 Hz, or about 500 ms of data) that learns to detect a particular temporal pattern. After training, these filters often look almost identical to bandpass filters in the frequency ranges neuroscientists have been studying for decades. The network independently rediscovers alpha, beta, and theta.

Step two: depthwise spatial convolution. The second layer takes each temporal feature map and convolves it across channels. This is the spatial step. It learns which combinations of electrodes are most informative for each temporal feature. If you've worked with Common Spatial Patterns, this is doing something conceptually similar, but learned from data rather than computed analytically.

Step three: separable convolution. A final separable convolution combines the spatial-temporal features and feeds them into a classification layer.

The whole network has roughly 2,000 to 10,000 parameters depending on the configuration. For context, ResNet-50 has 25.6 million parameters. EEGNet is tiny. And that tininess is a feature, not a limitation, because EEG datasets are tiny too. A model with millions of parameters would memorize the training data and learn nothing generalizable.

Beyond EEGNet: Deeper Architectures

EEGNet opened the door. Other architectures walked through it.

ShallowConvNet and DeepConvNet, from Schirrmeister et al. (2017), explored the depth axis. ShallowConvNet uses a single temporal-spatial convolution followed by squaring and log transformation, explicitly mimicking the computation of log-bandpower features. DeepConvNet stacks multiple convolution blocks to learn hierarchical features. The interesting finding: ShallowConvNet matched or beat DeepConvNet on most motor imagery tasks. Depth helps only when you have enough data to train the deeper layers, and most EEG datasets are too small.

TSception (Ding et al., 2022) introduced multi-scale temporal convolutions, applying filters of different lengths in parallel to capture patterns at multiple time scales simultaneously. This is particularly effective for tasks like emotion recognition, where the relevant neural dynamics span different durations.

FBCNet (Mane et al., 2021) combined filter-bank preprocessing with a CNN, explicitly decomposing the signal into multiple frequency bands before the convolutional layers. This blends the neuroscience-informed structure of classical approaches with the end-to-end learning of deep models.

The common thread across all successful EEG CNNs: they encode assumptions about the structure of EEG data into their architecture. Temporal convolutions for time patterns. Spatial convolutions for electrode combinations. The architectures that ignore the spatial-temporal structure of EEG and treat it like a generic 2D array consistently underperform.

RNNs and LSTMs: When the Brain's History Matters

Convolutional networks see the world through a fixed-size window. They can learn patterns within that window, but they don't have an inherent mechanism for reasoning about sequences that span much longer than their receptive field.

Brains, on the other hand, are deeply sequential. The mental state at this moment depends on what happened a second ago, ten seconds ago, sometimes minutes ago. A focus state doesn't appear as a snapshot. It builds, fluctuates, sometimes recovers, and sometimes collapses. Capturing these dynamics requires a model that has something CNNs lack: memory.

How LSTMs Process Brain Signals

Long Short-Term Memory networks maintain a cell state, a running memory that the network learns to write to, read from, and erase through a set of learned gates. At each time step, the LSTM sees the current EEG sample (or a feature vector derived from it) and updates its internal state based on three decisions:

1. What to forget from the previous state (forget gate)
2. What new information to store from the current input (input gate)
3. What to output based on the updated state (output gate)

This gating mechanism lets LSTMs selectively remember patterns from hundreds of time steps ago while ignoring irrelevant fluctuations in between.

For EEG, this means an LSTM can learn that a gradual buildup of theta power over 15 seconds, combined with declining beta power, predicts an attention lapse 5 seconds before it actually manifests in behavior. That kind of temporal reasoning is extremely difficult to encode as hand-crafted features. You'd need to know exactly what temporal pattern to look for, over what time window, with what dynamics. The LSTM discovers these relationships from data.

Where RNNs Shine (And Where They Don't)

RNNs and LSTMs have clear advantages for specific EEG tasks.

Continuous state tracking is the sweet spot. If you need to follow the trajectory of a cognitive state over time, not just classify snapshots, LSTMs outperform CNNs consistently. Tracking attention drift, monitoring fatigue accumulation over a work session, detecting the gradual onset of drowsiness in a driver. These are inherently sequential problems, and sequential models handle them better.

Variable-length inputs are natural for RNNs. An LSTM can process an EEG segment of any duration without architectural changes. CNNs require fixed-size inputs (or pooling tricks to handle variable lengths).

Temporal context aggregation lets LSTMs make more confident predictions by accumulating evidence over time. A single 1-second window of EEG might be ambiguous. But 10 seconds of context, processed sequentially, usually isn't.

The downsides are real, though. LSTMs are notoriously harder to train than CNNs. Vanishing gradients on long sequences, sensitivity to learning rate, slower convergence. They're also sequential by nature, which means you can't parallelize training across time steps the way you can parallelize across spatial locations in a CNN.

Gated Recurrent Units (GRUs) simplify the LSTM architecture by merging the forget and input gates. They train faster and often match LSTM performance on EEG tasks while using fewer parameters. If you're starting an RNN-based EEG project, GRUs are a reasonable default.

Transformers: Attention Is All Your EEG Needs

And then came the architecture that reshaped the entire field of machine learning. Self-attention.

The core idea behind transformers is simple to state and profound in its implications. Instead of processing a sequence step by step (like an RNN) or through a fixed-size window (like a CNN), a transformer computes relationships between every pair of positions in the input simultaneously. Every time point can attend to every other time point. Every channel can attend to every other channel. The model learns which relationships matter.

For EEG, this capability is almost suspiciously well-suited.

Why Attention Mechanisms Match Brain Signal Structure

Consider what makes EEG analysis hard. A gamma burst in your frontal channels at one moment might be deeply related to an alpha desynchronization in your parietal channels 300 milliseconds later. A brief pattern in one electrode might be the key that unlocks the meaning of a pattern in another electrode 2 seconds earlier. These long-range, cross-channel dependencies are exactly what self-attention was built to capture.

RNNs can theoretically model long-range dependencies, but in practice they struggle with sequences longer than a few hundred time steps because of gradient decay. Self-attention has no such limitation. A transformer with 1,000 time steps can directly compute the relationship between time step 1 and time step 1,000 with the same computational weight as between time step 1 and time step 2.

Here's the "I had no idea" moment. When researchers visualized the attention maps of transformer models trained on EEG data, they found something remarkable. The models learned to attend to physiologically meaningful time-frequency relationships that matched known neuroscience. Alpha desynchronization patterns during motor planning. Theta-gamma coupling during memory encoding. But they also attended to cross-frequency, cross-channel interactions that had never been described in the neuroscience literature. The transformers were finding structure in brain signals that human experts hadn't identified yet.

This isn't just an engineering result. It's a scientific tool. Attention maps from well-trained transformers can generate hypotheses about how the brain processes information. The artificial network might be telling us things about the biological network.

Key Transformer Architectures for EEG

EEG-Conformer (Song et al., 2022) combines a CNN front-end for local feature extraction with a transformer encoder for global dependency modeling. The CNN captures spatial-temporal patterns within short windows. The transformer captures relationships across windows. On the BCI Competition IV motor imagery dataset, EEG-Conformer pushed accuracy above 90%, compared to roughly 85% for CNN-only models and 80% for classical ML approaches.

BrainBERT and similar pre-trained models apply the masked pre-training paradigm from NLP to EEG. Train a transformer to predict randomly masked segments of EEG data from a large unlabeled corpus. The model learns general representations of brain electrical activity. Then fine-tune on your specific task with minimal labeled data. This is the foundation model approach, and it addresses the biggest bottleneck in EEG deep learning: labeled data scarcity.

ViT-inspired architectures treat EEG as a sequence of patches (short time-frequency windows) and apply the Vision Transformer paradigm. This works surprisingly well for classification tasks, though the optimal patch size varies significantly across tasks and datasets.

The Crown captures brainwave data at 256Hz across 8 channels. All processing happens on-device. Build with JavaScript or Python SDKs.

Explore the Crown

The Architecture Comparison: When to Use What

Choosing the right architecture for your EEG deep learning project is not about picking the "best" one. It's about matching the architecture's strengths to your task, your data, and your deployment constraints.

The practical takeaway: if you're starting a new EEG classification project and don't have a strong reason to choose otherwise, start with EEGNet. It's the best validated, most stable, and easiest to train CNN architecture for EEG. Treat it as your baseline. If your task involves continuous monitoring over time, add an LSTM on top. If you have a large dataset (or access to a pre-trained model) and need maximum accuracy, explore transformers.

The Training Challenges Nobody Warns You About

Here is where the real difficulty lives. Not in the architecture. In the data.

Building a deep learning model for images is hard. Building one for EEG is a different kind of hard. The challenges are so specific to brain data that solutions from computer vision and NLP often don't transfer at all.

The Small Dataset Problem

The largest public EEG datasets contain a few hundred subjects with a few recording sessions each. Compare that to the hundreds of millions of images and billions of text tokens that power modern computer vision and NLP models. This gap isn't just quantitative. It changes what architectures and training strategies are viable.

With 100 training subjects, a model with 50 million parameters will memorize every artifact, every idiosyncratic noise pattern, every accidental correlation in the training data. It will look brilliant during training and fail completely on a new subject. This is why EEGNet, with its 2,000 to 10,000 parameters, consistently outperforms much larger models on EEG benchmarks. It's not that bigger models can't learn EEG. It's that we don't have enough data to teach them.

Transfer learning is the most important mitigation strategy. Pre-train on a large public dataset. Fine-tune on your specific task. The Temple University EEG Corpus (over 10,000 clinical EEG recordings), PhysioNet's motor imagery datasets, and the MOABB benchmark collection provide viable pre-training sources. A transformer pre-trained on these corpora and fine-tuned on 50 sessions of your target task can outperform a CNN trained from scratch on 500 sessions.

Data augmentation helps but requires care. Common augmentations include adding Gaussian noise, randomly scaling amplitudes, cropping and shifting windows in time, and mixing channels. But unlike image augmentation (where flipping and rotating preserve semantic content), EEG augmentation can easily destroy the signal you're trying to learn. Flipping the time axis, for instance, reverses the temporal dynamics that carry most of the cognitive state information. Only augmentations that preserve the physiological structure of the signal are useful.

The Subject Variability Problem

This is the problem that makes EEG deep learning genuinely different from almost every other domain.

Every brain is physically unique. Cortical folding patterns, skull thickness, conductivity of scalp tissue, baseline oscillatory power, all of it varies substantially between people. An alpha rhythm that peaks at 10 Hz in one person might peak at 11.5 Hz in another. The spatial pattern of a motor imagery response that's strongest at C3 in one person might be strongest at CP3 in another.

This means a model trained on 50 subjects might perform beautifully on those 50 subjects and fail on subject 51. Not because the model is bad, but because subject 51's brain literally organizes its electrical activity differently.

Subject-adaptive training addresses this directly. The standard approach: train a base model on a large multi-subject dataset, then fine-tune the final layers on a small amount of calibration data from the new subject. Even 2 to 5 minutes of calibration data can dramatically improve cross-subject accuracy.

Domain adaptation techniques like Maximum Mean Discrepancy (MMD) regularization or adversarial domain adaptation can also help by explicitly encouraging the model to learn features that are invariant across subjects. The model learns to extract patterns that generalize, rather than patterns that are specific to the training population's brain anatomy.

Nonstationarity: The Signal That Won't Sit Still

EEG signals are nonstationary. Their statistical properties change over time, even within a single session. Your alpha power at minute 1 of a recording might have a completely different mean and variance than at minute 30, even if your cognitive state hasn't changed.

This breaks a fundamental assumption of most machine learning algorithms: that the training data and the test data come from the same distribution. In EEG, they often don't, even when they come from the same person on the same day.

Batch normalization, the workhorse of deep learning, can actually hurt on nonstationary data because it learns normalization statistics from the training distribution that don't match the test distribution. Instance normalization or group normalization, which normalize each sample independently, often work better for EEG.

Online adaptation, where the model continues to update its parameters (or at least its normalization statistics) during inference, is an active area of research that addresses nonstationarity directly.

Artifacts: The Noise That Looks Like Signal

Eye blinks, muscle tension, head movements, jaw clenches. These artifacts produce electrical signals that are often much larger than the neural signals you're trying to decode. And here's the insidious part: some artifacts are correlated with the very cognitive states you're trying to classify.

People blink more when they're unfocused. They tense their jaw when they're concentrating. If your deep learning model achieves 95% accuracy classifying focus states but it's actually detecting jaw tension, you haven't built a [brain-computer interface](/guides/what-is-bci-brain-computer-interface). You've built a jaw-computer interface.

Rigorous artifact handling is not optional. Use independent component analysis (independent component analysis) or artifact subspace reconstruction (ASR) to clean your training data. Then deliberately test on data with controlled artifact levels. If your accuracy drops by 20% when you remove muscle artifacts from the signal, your model was probably cheating.

From Model to Pipeline: What the Crown Makes Possible

For developers building deep learning EEG systems, the practical challenge isn't just the model. It's everything around the model. Signal acquisition. Preprocessing. Artifact handling. Real-time streaming. On-device inference.

The Neurosity Crown handles the parts of this pipeline that are hardest to build yourself.

Signal acquisition and conditioning happen on-device through the N3 chipset. Eight channels at CP3, C3, F5, PO3, PO4, F6, C4, and CP4, sampling at 256 Hz. The chipset handles filtering and basic artifact management at the hardware level, so the data that reaches your deep learning pipeline through the SDK is already cleaner than what you'd get from a raw EEG amplifier.

Data streaming through the JavaScript and Python SDKs gives you real-time access to raw EEG, power spectral density, and frequency band data. For training, you can record sessions to disk and build datasets. For inference, you can pipe live data directly into your model:

import { Neurosity } from "@neurosity/sdk";

const neurosity = new Neurosity({ deviceId: "YOUR_DEVICE_ID" });
await neurosity.login({ email, password });

// Stream raw EEG into your deep learning pipeline
neurosity.brainwaves("raw").subscribe((brainwaves) => {
  const { data, timestamp } = brainwaves;
  // data: array of 8 channel readings at 256Hz
  // Feed to your PyTorch/TensorFlow model
  myTransformerModel.predict(preprocessBatch(data));
});

BrainFlow and Lab Streaming Layer integration means you can use the Crown with the standard tools in the neuroscience Python ecosystem. Record data with BrainFlow, preprocess with MNE-Python, train with PyTorch, deploy with ONNX. The Crown fits into existing workflows rather than requiring a new one.

On-device ML through the N3 chipset already runs inference for focus scores, calm scores, and kinesis (mental commands). If your task maps to one of these built-in decoders, you don't need to train anything. You get production-grade neural decoding from day one. For custom tasks, stream the raw data and bring your own model.

The Near Future: Foundation Models for Brain Data

Here's where this gets genuinely exciting.

The same paradigm that produced GPT for language and DALL-E for images is being applied to brain data. Train a massive model on thousands of hours of EEG from hundreds of subjects. Don't give it any labels. Just teach it to understand the structure of brain electrical activity. Then fine-tune on your specific task with a tiny amount of labeled data.

The early results from models like LaBraM (Large Brain Model), BrainBERT, and NeuroGPT suggest this approach can dramatically reduce the labeled data requirements that have been the bottleneck for EEG deep learning. A foundation model pre-trained on 2,500 hours of clinical EEG, fine-tuned on 20 labeled sessions for your specific task, can outperform a model trained from scratch on 500 labeled sessions.

We're still in the early days. The largest EEG foundation models today have tens of millions of parameters, compared to billions for language models. The training corpora are growing but still orders of magnitude smaller than what's available for text and images. And the fundamental challenge of subject variability means that brain data may never achieve the same kind of universal representations that make language models so powerful.

But the trajectory is unmistakable. And if you're a developer building EEG applications today, the architectures and training strategies in this guide are the foundation you'll need when those pre-trained models become widely available. The interface between your application and a foundation model will still be the same: raw EEG in, predictions out. The Crown's SDK and streaming capabilities will work just as well with a fine-tuned foundation model as they do with a custom EEGNet you trained this afternoon.

The human brain has been running its own neural networks for 500 million years. We've been building artificial ones for about 70. The fact that the artificial ones are already learning to read the biological ones, finding patterns that neuroscientists hadn't even named yet, is one of the most remarkable developments in the history of either kind of network.

Your next model might discover something about the brain that no human has ever noticed. And that thought alone is worth staying up late for.