# Mechanistic Evidence for Spectral Structures in Prior-Data Fitted Networks Source: [Mechanistic Evidence for Spectral Structures in Prior-Data Fitted Networks]() Collection index: [llms.txt]() Data updated: 2026-05-15T05:00:04.680Z Authors: Kaustubh Sharma, Srijan Tiwari, Ojasva Nema, Parikshit Pareek Published: 2026-01-29T13:51:26.000Z ## Abstract Prior-Data Fitted Networks (PFNs) enable amortized Bayesian inference in a single forward pass, yet their internal representations remain opaque. It is unknown whether PFNs encode identifiable Bayesian structure or merely memorize input-output mappings. We provide mechanistic evidence that PFNs learn structured spectral representations and that these can be extracted as explicit kernels. First, probing experiments across three architectures, including the publicly released TabPFN, show that spectral information is linearly decodable from the latent attention score and organized along a dominant principal axis. Activation patching and targeted subspace interventions establish that this information is causally used for prediction and concentrated in a low-dimensional subspace, with spectral directions an order of magnitude more effective than random ones. Crucially, these properties hold on TabPFN with both synthetic out-of-distribution inputs and real-world time series (Airline Passengers, Milk Production), indicating they are emergent features of PFN-style amortization over continuous regression tasks rather than artifacts of training prior. Second, we introduce a Filter Bank Decoder that maps frozen PFN latents to explicit spectral densities, reconstructing stationary kernels via Bochner's theorem. The resulting kernels support GP regression competitive with iterative baselines while requiring only a single forward pass, demonstrating that PFN priors are not merely implicit but are explicitly recoverable as portable Bayesian objects. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [GP regression on Kernel Cookbook 1.0 (test)](): GP regression; Kernel Cookbook 1.0 (test); MSE: 0.000573. ## Original sources and implementation [Original paper]() [PDF]()