How do different levels of government debt affect the optimal conduct of monetary and fiscal policies? And what do these optimal policies imply for the evolution of government debt over time? To provide an answer, this paper studies a standard monetary policy model with nominal rigidities and monopolistic competition and adds to it a fiscal authority that issues nominal non-state contingent debt, levies distortionary labor income taxes and determines the level of public goods provision. Higher government debt levels make it optimal to reduce public spending, so as to dampen the adverse incentive effects of distortionary taxes, but also strongly influence the optimal stabilization response following technology shocks. In particular, higher debt levels give rise to larger risks to the fiscal budget and to tax rates. This makes it optimal to reduce government debt over time. The optimal speed of debt reduction is missed when using first order approximations to optimal policies, but is shown to be quantitatively significant in a second order approximation, especially when technology movements are largely unpredictable in nature.