Rapid technology developments in neuroscience are enabling us to record from an ever increasing number of neurons from the brain. However, with the deluge of experimental data, our ability to extract simple yet fundamental understandings of the neural mechanisms underlying behavior and cognition is hampered by a lack of theoretically principled data analytics procedures. In the present work, we outline a set of theoretical frameworks that begins to address this challenge. First, we focus on the analysis of trial-averaged data obtained over experimental repetitions of tightly controlled behaviors. We start by developing a theory of neural dimensionality, which explains the prevalence of low-dimensional dynamic portraits observed in system neuroscience. We then connect the experimental act of recording a random subset of neurons to the mathematical theories of random projection, and illustrate how we might understand anything about the brain given the infinitesimal fractional of behaviorally relevant neurons observed. The second part of the thesis addresses the analyses of single-trial neural data collected during potentially more complex or naturalistic behaviors that may not be repeatable. We explore the effects of trial-to-trial variability and neuronal noise in the context of several analytically tractable generative data models covering linear and nonlinear stimulus-response mappings as well as static and dynamic latent states. We derive exhaustively the functional dependencies of commonly applied analytics procedures' performances on the number of recorded neurons, the number of trials and other model specific parameters. For each of the theoretical puzzles addressed in this thesis, we formulate the question with mathematical precision, derive quantitative predictions testable against simulations and/or neural data, and provide guidelines for the interpretation of past results as well as the design of future experiments.