Exact Tangent-Kernel Transport under Joint Training
Kevin L. Brown
August 2026
This paper develops an exact mathematical framework for tracking how the task-relevant quality of a learned internal representation changes while a neural network is being trained.
A central difficulty in representation learning is that improvement in the network’s training objective does not necessarily imply that its hidden representation has become more useful. A jointly trained output head may adapt successfully even while information available to an independently optimized readout temporarily becomes harder to extract. Conventional training-loss curves can therefore conceal important internal dynamics.
The paper formalizes this distinction using a profiled representation risk. For a nonlinear encoder with parameters (\theta), an independent diagnostic readout is optimized over the current representation, producing a reduced objective (F_E(\theta)) that measures the best task performance accessible from those features under the declared probe class. The actual network head used for training is kept separate from this profiled diagnostic head.
The first main result derives an exact transport law for (F_E) under encoder gradient flow. Representation quality changes according to the alignment between the encoder gradient of the profiled diagnostic objective and the encoder gradient generated by the network’s actual training head. Negative alignment therefore permits training loss to decrease while profiled representation risk increases.
For squared-loss ridge probes, the theory is lifted to a positive-semidefinite residual-risk matrix, preserving class- and output-resolved structure that is lost under scalar trace reduction. The finite-sample theory then derives an explicit training-to-diagnostic feature-tangent-kernel contraction, showing how training residuals, diagnostic residuals, the actual head, the profiled head, and cross-dataset encoder geometry jointly determine instantaneous representation change.
The framework also includes a second-order finite-step expansion and a stochastic-minibatch extension. Under an unbiased minibatch encoder-gradient estimator, the deterministic alignment law remains unchanged at first order, while the leading stochastic correction appears at second order through a curvature–gradient-covariance term. This separates ordinary gradient alignment from noise-induced local drift.
The theory is tested on the sklearn handwritten-digits dataset using a nonlinear (64\rightarrow32\rightarrow16\rightarrow10) network. Across random initializations, ordinary training frequently decreased training loss while simultaneously increasing profiled diagnostic risk. At small step sizes, predicted and observed representation-risk changes agreed almost perfectly, with correlation approximately (0.999995). Controlled scaling of the initial output head systematically changed the frequency of adverse representation motion, while zero-head initialization suppressed the immediate encoder update.
Most importantly, the phenomenon generalized to an untouched test set. During early training, diagnostic representation risk increased while independently evaluated test-probe loss worsened and test-probe accuracy declined, even as training loss fell sharply. Later, the alignment reversed and representation quality improved.
The contribution is deliberately narrower than prior work on linear probing, feature distortion, neural tangent kernels, optimized last-layer training, and representation forgetting. The claimed advance is the exact observer-versus-driver profiled-risk transport theory, its matrix-valued specialization, and its explicit cross-dataset feature-kernel form for nonlinear representation learning