Selective Steering | Norm-Preserving LLM Control

Abstract

Selective Steering: Norm-preserving rotation on discriminative layers.

Despite significant progress in alignment, large language models (LLMs) remain vulnerable to adversarial attacks that elicit harmful behaviors. Activation steering techniques offer a promising inference-time intervention approach, but existing methods suffer from critical limitations: activation addition requires careful coefficient tuning and is sensitive to layer-specific norm variations, while directional ablation provides only binary control.

Recent work on Angular Steering introduces continuous control via rotation in a 2D subspace, but its practical implementation violates norm preservation, causing distribution shift and generation collapse, particularly in models below 7B parameters.

We propose Selective Steering, which addresses these limitations through two key innovations: (1) a mathematically rigorous norm-preserving rotation formulation that maintains activation distribution integrity, and (2) discriminative layer selection that applies steering only where feature representations exhibit opposite-signed class alignment.

Experiments across nine models demonstrate that Selective Steering achieves 5.5× higher attack success rates than prior methods while maintaining zero perplexity violations and approximately 100% capability retention on standard benchmarks.

Key Contributions

Layer-Wise Analysis

First systematic analysis of layer-wise activation geometry for steering, identifying non-uniform norm growth and progressive discriminability emergence.

Norm-Preserving Rotation

Mathematically rigorous formulation guaranteeing ||h'|| = ||h|| for all activations, eliminating distribution shift and generation collapse.

Superior Performance

5.5× improvement on challenging small models, zero perplexity violations across all models and angles, ~100% baseline capability retention.

Interactive Demo

Watch Selective Steering in action, modifying LLM responses through angular control.

Try it yourself: bash run_ui.sh to launch the Gradio interface

Steering Examples

Explore how Selective Steering modifies model responses at different angles. Select a steering degree to view examples from gemma-2-2b-it.

Steering Degree:

Loading examples...

1 / 104

Method Overview

Selective Steering combines norm-preserving rotation with discriminative layer selection for robust LLM behavior control.

The Problem: Layer-Wise Heterogeneity

Prior methods apply uniform steering across all layers, ignoring heterogeneous layer roles. Our analysis reveals:

Non-uniform norm profiles: Activation norms vary substantially across depth
Progressive discriminability: Opposite-signed class separation emerges in middle layers
Layer-specific vulnerability: Some layers are more amenable to steering than others

Activation norms vary substantially across depth in Qwen2.5-7B-Instruct

Discriminative Layer Selection

We identify discriminative layers where classes exhibit opposite-signed projections:

𝓛_disc = { k : μ_pos^(k) · μ_neg^(k) < 0 }

This criterion ensures steering is applied only where:

Features are strongly represented
Steering effect is predictable
Separation is robust across samples

Feature alignment showing discriminability

Feature alignment analysis reveals discriminative layers

Algorithm: Selective Steering

Input: Activation h^(k), basis {b₁, b₂}, angle θ, means μ_pos^(k), μ_neg^(k)
Output: Steered activation h'^(k)

// Check if layer is discriminative
if μ_pos^(k) · μ_neg^(k) ≥ 0:
    return h^(k)  // Skip non-discriminative layers

// Apply norm-preserving rotation
R_θ ← [[cos(θ), -sin(θ)], [sin(θ), cos(θ)]]
R^P_θ ← I - (b₁b₁ᵀ + b₂b₂ᵀ) + [b₁ b₂] R_θ [b₁ b₂]ᵀ
h'^(k) ← R^P_θ h^(k)  // ||h'|| = ||h|| guaranteed

return h'^(k)

Experimental Results

Evaluated across 8 models spanning 3 families (Llama, Qwen, Gemma) on coherence, controllability, and robustness.

Attack Success Rate

Selective Steering achieves highest or second-highest ASR in 8/8 models — 5.5× improvement over prior methods

Benchmark Accuracy at Best Steering Angle

SS preserves ~100% of baseline performance while achieving high ASR

Key Finding: SS decouples steering effectiveness from capability preservation — high ASR with full benchmark retention.

Coherence Metrics

N-gram repetition, language consistency, and compression ratio across models

Perplexity Stability Across Steering Angles

Red markers indicate perplexity exceeding threshold (>2.0) — signaling generation instability

Key Finding: Selective Steering achieves zero perplexity violations across all 8 models and all steering angles (0°-360°, 10° intervals).

Citation

If you find our work useful, please consider citing:

@misc{dang2026selective,
  title     = {Selective Steering: Norm-Preserving Control Through Discriminative Layer Selection},
  author    = {Quy-Anh Dang and Chris Ngo},
  year      = {2026},
  url       = {https://github.com/knoveleng/steering}
}