Controlling Language and Diffusion Models by Transporting Activations