Additive PEFT: Adapters

Adapters are small trainable blocks inserted into the network. The big pretrained model stays mostly frozen. Each task gets a tiny side path that learns how to tweak the hidden states for that task.

Intuition

Think of the base model as a shared engine. An adapter is a small bolt-on box for one job (support tickets, summarization, QA). You keep one engine and swap boxes.

Two practical requirements for good adapters:

Few new parameters — so total size grows slowly as you add tasks.
Near-identity start — the adapter begins almost as “do nothing,” so training starts gently from the pretrained behavior.

How it works

Sequential adapters

The adapter sits in the main path (in the residual stream). Information flows through the adapter as part of the layer.

Can reach similar accuracy while updating only about 1% of original parameters.
Trade-offs to remember: some inference latency, cost that scales with depth, harder to parallelize with existing compute, and multi-task serving can get expensive if you are not careful.

Residual (parallel) adapters

The adapter sits beside the main layer, then adds its output back to the original hidden state.

Plain picture of a common bottleneck adapter:

Shrink the hidden vector (down-project)
Apply a simple non-linearity (for example ReLU)
Expand back up
Add that correction to the original hidden state

In words: new_hidden = old_hidden + up(relu(down(old_hidden)))

Why residual adapters are appealing for generation: they keep the original stream intact and inject task behavior more gently.

The squeeze is where the savings come from:

hidden size       = 4096
bottleneck size   = 64

adapter params ≈ (4096 x 64) + (64 x 4096) ≈ 524,000 per layer
full layer params                          ≈ 16,800,000 per layer

About 3% of the weights of that layer — and the base layer stays frozen. The "near-identity start" matters here too: the up-projection is initialised near zero, so at step 1 the adapter adds almost nothing and the model behaves exactly as before. Training then gradually teaches it what correction to make.

Task structure helps

Adapters often work better when the input clearly marks the task — for example special tokens for QA segments (document, question, answer), or similar markers for dialogue and summarization. That structure gives the small module clearer patterns to learn.

Other ideas in the same family

Idea	Plain-English idea
AdapterFusion	Combine several trained adapters so one task can borrow skills from others
Tiny-Attention Adapter	Keep the add-on small, but let it use a little attention to steer the base model
Multi-task / routing setups	Share or route among adapters instead of always training one giant fully tuned model

Why not just tune the last few layers?

Tuning only the top layers can work, but adapters are more modular:

One frozen backbone for many tasks
Swap only the tiny task module
Cleaner maintenance and often better parameter efficiency

What goes wrong

Expecting zero latency cost with sequential adapters in every layer.
Forgetting near-identity initialization and shocking the model at step one.
Building one adapter per task with no serving plan — modular is good only if you can load/swap them cleanly.

One-line summary

Adapters add a small trainable module to a frozen backbone so each task gets its own cheap specialty without rewriting the whole model.

Key terms

Adapter — Small trainable module that adjusts hidden states.
Sequential adapter — Adapter placed in the main computation path.
Residual adapter — Adapter in parallel that adds a correction back to the stream.
AdapterFusion — Combining multiple adapters for reuse.