The efficient coding hypothesis has accounted for many aspects of neural responses in sensory areas. However, most previous work focused on information maximization about static stimuli in networks with feedforward architectures and thus lacking internal dynamics. It remains unclear in which dynamical regime neurons should collectively operate to best represent time-varying stimuli. Here, we identify a class of network dynamics ideally suited for encoding the past history of a dynamically changing stimulus in a recurrent neural circuit. We demonstrate in simulations and prove analytically that information transmission is maximized in recurrent networks that have time-reversible dynamics when stimulus statistics are themselves time-reversible. Are such dynamics compatible with neurobiological constrains? Surprisingly, we show that recurrent circuits naturally self-organize for time-reversibility under a biological form of spike timing-dependent plasticity (STDP), and that the resulting circuit connectivity obeys the well known principles of Hebb and Dale: synaptic weights become proportional to correlations between pre- and postsynaptic activity, and neurons eventually fall into two distinct classes of excitatory and inhibitory cells. Finally, we identify signatures of adaptive time reversible circuit dynamics in experimental data. In the primary visual cortex of awake ferrets, we find that neural activity has a very small irreversible component; Furthermore, activity is initially irreversible when stimulus statistics change, but time reversibility increases with continued stimulus exposure on a timescale of a few minutes. Our work suggests a central role for time reversibility as a novel signature of efficient coding in recurrent circuits for dynamic stimuli, and provides the first joint normative account of Hebb and Dale’s principles.