A diverse set of functional materials can be fabricated using dispersions of colloids and nanoparticles. If the dispersion is responsive to an external field, like dielectric and charged particles in an electric field or paramagnetic particles in a magnetic field, the field can be used to facilitate self-assembly and control particle transport. One promising feature of field-responsive materials is the ability to drive them out of equilibrium by varying the external field in time. Without the constraints of equilibrium thermodynamics, out-of-equilibrium dispersions display a rich array of self-assembled states with useful material and transport properties. To leverage their unique behaviors in real applications, a predictive, theoretical framework is needed to guide experimental design. In this thesis, I carry out a systematic investigation of the self-assembly and dynamics of colloidal dispersions in time-varying external fields using computer simulations, equilibrium and nonequilibrium thermodynamics, and electro-/magnetokinetic theory. I first develop efficient computational models for simulating suspensions of polarizable colloids in external fields. The simulations are accurate enough to quantitatively reproduce experiments but fast enough to reach the large length and time scales relevant for self-assembly. I use this simulation method to construct the complete equilibrium phase diagram for polarizable particles in steady external fields and find that many-bodied, mutual polarization has a remarkably strong influence on the nature of the self-assembled states. Correctly accounting for mutual polarization enables a thermodynamic theory to compute the phase diagram that agrees well with simulations and experiments. Though the equilibrium structures are crystalline, in practice, dispersions typically arrest in kinetically-trapped, disordered or defective metastable states due to strong interparticle forces. This is a key difficulty preventing scalable fabrication of colloidal crystals. I show that cyclically toggling the external field on and off over time leads to growth of colloidal crystals at significantly faster rates and with many fewer defects than for assembly in a steady field. The toggling protocol stabilizes phases that are only metastable in steady fields, including complex, transmutable crystal structures. I use nonequilibrium thermodynamics to predict the out-of-equilibrium states in terms of the toggle parameters. I also investigate the transport properties of dispersions of paramagnetic particles in rotating magnetic fields. Like toggled fields, rotating fields also drive dispersions out of equilibrium, and their dynamics can be tuned with the rotation frequency. I find that the rotating field greatly increases particle self-diffusivity compared to steady fields. The diffusivity attains a maximum value several times larger than the Stokes- Einstein diffusivity at intermediate rotation frequencies. I develop a simple phenomenological model for magnetophoresis through porous media in rotating fields that predicts enhanced mobility over steady fields, consistent with experiments. Lastly, I study the nonlinear dynamics of polarizable colloids in electrolytes and report a new mode of electrokinetic transport. Above a critical external field strength, an instabilty occurs and particles spontaneously rotate about an axis orthogonal to the field, a phenomenon called Quincke rotation. If the particle is also charged, its electrophoretic motion couples to Quincke rotation and propels the particle orthogonally to the driving field, an electrohydrodynamic analogue to the Magnus effect. Typically, motion orthogonal to a field requires anisotropy in particle shape, dielectric properties, or boundaries. Here, the electrohydrodynamic Magnus (EHM) effect occurs for bulk, isotropic spheres, with the Quincke rotation instability providing broken symmetry driving orthogonal motion. In alternating-current (AC) fields, electrophoresis is suppressed, but the Magnus velocity persists over many cycles. The Magnus motion is decoupled from the field and acts as a self-propulsion, so I propose the EHM effect in AC fields as a mechanism for generating a new type of active matter. The EHM "swimmers" behave as active Brownian particles, and their long-time dynamics are diffusive, with a field-dependent effective diffusivity that is orders of magnitude larger than the Stokes-Einstein diffusivity. I also develop a continuum electrokinetic theory to describe the electrohydrodynamic Magnus effect that is in good agreement with my simulations.