The objective of this research is to achieve accurate and scalable bridge health monitoring (BHM) by learning, integrating, and generalizing the monitoring models derived from drive-by vehicle vibrations. Early diagnosis of bridge damage through BHM is crucial for preventing more severe damage and collapses that could lead to significant economic and human losses. Conventional BHM approaches require installing sensors directly on bridges, which are expensive, inefficient, and difficult to scale up. To address these limitations, this research uses vehicle vibration data when the vehicle passes over the bridge to infer bridge conditions. This drive-by BHM approach builds on the intuition that the recorded vehicle vibrations carry information about the vehicle-bridge interaction (VBI) and thus can indirectly inform us of the dynamic characteristics of the bridge. Advantages of this approach include the ability for each vehicle to monitor multiple bridges economically and eliminating the need for on-site maintenance of sensors and equipment on bridges. Though the drive-by BHM approach has the above benefits, implementing it in practice presents challenges due to its indirect measurement nature. In particular, this research tackles three key challenges: 1) Complex vehicle-bridge interaction. The VBI system is a complex interaction system, making mathematical modeling difficult. The analysis of vehicle vibration data to extract the desired bridge information is challenging because the data have complex noise conditions as well as many uncertainties involved. 2) Limited temporal information. The drive-by vehicle vibration data contains limited temporal information at each coordinate on the bridge, which consequently restricts the drive-by BHM's capacity to deliver fine-grained spatiotemporal assessments of the bridge's condition. 3) Heterogeneous bridge properties. The damage diagnostic model learned from vehicle vibration data collected from one bridge is hard to generalize to other bridges because bridge properties are heterogeneous. Moreover, the multi-task nature of damage diagnosis, such as detection, localization, and quantification, exacerbates the system heterogeneity issue. To address the complex vehicle-bridge interaction challenge, this research learns the BHM model through non-linear dimensionality reduction based on the insights we gained by formulating the VBI system. Many existing physics-based formulations make assumptions (e.g., ignoring non-linear dynamic terms) to simplify the drive-by BHM problem, which is inaccurate for damage diagnosis in practice. Data-driven approaches are recently introduced, but they use black-box models, which lack physical interpretation and require lots of labeled data for model training. To this end, I first characterize the non-linear relationship between bridge damage and vehicle vibrations through a new VBI formulation. This new formulation provides us with key insights to model the vehicle vibration features in a non-linear way and consider the high-frequency interactions between the bridge and vehicle dynamics. Moreover, analyzing the high-dimensional vehicle vibration response is difficult and computationally expensive because of the curse of dimensionality. Hence, I develop an algorithm to learn the low-dimensional feature embedding, also referred to as manifold, of vehicle vibration data through a non-linear and non-convex dimensionality reduction technique called stacked autoencoders. This approach provides informative features for achieving damage estimation with limited labeled data. To address the limited temporal information challenge, this research integrates multiple sensing modalities to provide complementary information about bridge health. The approach utilizes vibrations collected from both drive-by vehicles and pre-existing telecommunication (telecom) fiber-optic cables running through the bridge. In particular, my approach uses telecom fiber-optic cables as distributed acoustic sensors to continuously collect bridge dynamic strain responses at fixed locations. In addition, drive-by vehicle vibrations capture the input loading information to the bridge with a high spatial resolution. Due to extensively installed telecom fiber cables on bridges, the telecom cable-based approach also does not require on-site sensor installation and maintenance. A physics-informed system identification method is developed to estimate the bridge's natural frequencies, strain and displacement mode shapes using telecom cable responses. This method models strain mode shapes based on parametric mode shape functions derived from theoretical bridge dynamics. Moreover, I am developing a sensor fusion approach that reconstructs the dynamic responses of the bridge by modeling the vehicle-bridge-fiber interaction system that considers the drive-by vehicle and telecommunication fiber vibrations as the system input and output, respectively. To address the heterogeneous bridge properties challenge, this research generalizes the monitoring model for one bridge to monitor other bridges through a hierarchical model transfer approach. This approach learns a new manifold (or feature space) of vehicle vibration data collected from multiple bridges so that the features transferred to such manifold are sensitive to damage and invariant across multiple bridges. Specifically, the feature is modeled through domain adversarial learning that simultaneously maximizes the damage diagnosis performance for the bridge with available labeled data while minimizing the performance of classifying which bridge (including those with and without labeled data) the data came from. Moreover, to learn multiple diagnostic tasks (including damage detection, localization, and quantification) that have distinct learning difficulties, the framework formulates a feature hierarchy that allocates more learning resources to learn tasks that are hard to learn, in order to improve learning performance with limited data. A new generalization risk bound is derived to provide the theoretical foundation and insights for developing the learning algorithm and efficient optimization strategy. This approach allows a multi-task damage diagnosis model developed using labeled data from one bridge to be used for other bridges without requiring training data labels from those bridges. Overall, this research offers a new approach that can achieve accurate and scalable BHM by learning, integrating, and generalizing monitoring models learned from drive-by vehicle vibrations. The approach enables low-cost and efficient diagnosis of bridge damage before it poses a threat to the public, which could avoid the enormous loss of human lives and property.