Uncertainty quantification (UQ) is the science of quantifying and characterizing response variation due to the presence of uncertainties in the input parameters and governing models. Among the prevalent methods for UQ, non-intrusive probabilistic techniques such as the perturbation method and Polynomial Chaos Expansion (PCE) are more popular due to their ability to integrate existing deterministic solvers as a black box. However, with the increase in the number of inputs, the number of basis terms in the expansion increases exponentially, also known as the 'curse of dimensionality', thereby requiring a large number of function realizations. Therefore, this dissertation is focused on exploring a new robust algorithms for the perturbation method as well as PCEand their application to composite structures while maintaining a balance between accuracy and computational efficiency. At first, an efficient approach for UQ using a higher-order Taylor series expansion is developed. Then, the local sensitivities in the Taylor series are evaluated using a high-accuracy and computationally efficient approach called modified forward finite difference (ModFFD). The number of function evaluations required for the sensitivity estimation equals the number of expansion terms in the series. Once the sensitivities are evaluated with ModFFD, the stochastic response is obtained for different realizations of the random inputs without additional function evaluations. This approach's main advantage is that it applies to any probability distribution of the inputs and is unrestricted by the nature of random input variables (correlated and uncorrelated). Several analytical and engineering problems were considered with up to twenty-two random variables to test the presented approach. A ten-bar truss problem with twenty-two random variables and buckling of a composite laminate with twenty random variables are considered as engineering problems. The comparison of the results with the reference solution obtained using many Monte Carlo Simulations (MCS) demonstrated its high accuracy and computational efficiency for random inputs with non-standard random inputs and varying correlation. Secondly, to further reduce the number of samples required to build a surrogate model to carry out uncertainty analysis, least-squares Polynomial Chaos Expansion (PCE) with L2 regularization for an under-determined system is presented. Moreover, a new method for selecting the regularization parameter is proposed and compared with the traditional L-curve method with Tikhonov regularization. This work aims to find the best solution from the limited number of function realizations (fewer response samples), thereby directly reducing the computational time required to build the surrogate while maintaining the desired accuracy. The proposed method is applied to several analytical problems and an engineering problem - the stochastic study of Mode-I delamination of a composite structure using Cohesive Zone Element (CZM). The results demonstrated the applicability and computational superiority of the proposed algorithm. Finally, stochastic buckling analysis of an unstiffened composite cylinder with geometric imperfection under axial compression is explored. The effect of random initial geometric imperfections, material properties, ply orientation, and ply thickness on the buckling limit load of thin-walled, composite cylindrical shells is studied. The initial geometric imperfections are modeled using the mode shapes of linear buckling analysis (LBA). In addition, to reduce the number of function evaluations required during the PCE building process for UQ, adaptive-sparse polynomial chaos expansion with L1-norm minimization is utilized based on orthogonal matching pursuit (OMP). Global sensitivity analysis (GSA) based on Sobol indices is used to identify the important parameters of the system. The results showed the uncertainties' significant effect on the buckling eigenvalues of the structures, thereby emphasizing the need to account for geometric imperfections and other sources of uncertainty during the design phase to obtain a robust design.