Mobile Edge Computing (MEC) is proving to be a very successful alternative to cloud computing (CC) for executing computationally intensive tasks that cannot be handled by end devices such as laptops and smart phones. The MEC paradigm facilitates task computation at the edge of the network rather than the cloud. It has been shown that MEC decreases delay and energy consumption while simultaneously reducing the load on backbone networks and the clouds. The hierarchical-MEC (H-MEC) paradigm adds on to MEC by involving idle end devices in the task computation process. H-MEC can offer the advantages of MEC, while further reducing the load on edge networks and edge servers. A large proportion of edge computing tasks comprise some form of data analytics where there is an increasing trend to use machine learning (ML) techniques. Therefore, in order to share the burden of complex ML algorithms and to preserve data privacy, employing the ML training in a distributed manner on end devices or learners, is becoming more common. This is especially important with the advent of the deployment of 5th generation (5G) networks which may offer the feature of fast device-to-device (D2D) communication. This dissertation proposes a method for optimal task offloading in H-MEC while jointly minimizing delay and energy consumption. Analysis on H-MEC task allocation with the joint time and energy minimization showed that the problem is NP-hard and hence, a heuristic solution was proposed. Results indicate that allowing one idle user to act as a server can provide up-to a 13% reduction in completion time and up-to 17% reduction in energy consumption. The focus of the dissertation then shifts to the major component of this work, which is enabling and optimizing the execution of machine learning tasks in a distributed manner, referred to as distributed Learning (DL), on H-MEC systems. Consequently, we design the novel paradigm of Mobile Edge Learning (MEL), where the goal is to allocate tasks optimally such that the ML model accuracy is maximized while taking into consideration the heterogeneous communication and computational capabilities of individual learners in a wireless MEC system. More specifically, our approach is heterogeneity aware (HA) compared to previous works which were heterogeneity unaware (HU). This part of the research investigates MEL optimal task allocation for synchronous (HA-Sync) and asynchronous (HA-Asyn) settings, with limits on the global completion time and local energy consumption. In the last part of the work, we provide recommendations on best scheme selection and how to apply the MEL in context of H-MEC. The problem of optimal task allocation in MEL is divided into four sub-problems consisting of HA-Sync and HA-Asyn with only time constraints and the HA-Sync/Asyn with dual time and energy constraints. All sub-problems are shown to be non-polynomial (NP) hard and hence, solutions based on relaxations are proposed. For the HA-Sync with time constraints, analytical upper bounds are proposed and shown to perform similar to the numerical approaches. For the HA-Asyn with time constraints and the HA-Sync/Asyn with dual time and energy constraints, solutions based on the suggest-and-improve (SAI) framework are proposed. Simulation results on MEL show that our proposed HA approaches achieve a superior validation accuracy and provide significant reductions in time for reaching a certain accuracy threshold compared to HU schemes when there is a limit on the global completion time. For example, the HA-Sync schemes reduce training time by up-to 25% compared to HU, whereas the HA-Asyn can provide further gains of up-to 10% in some settings. When there are joint global time and local energy consumption constraints, the HA-Sync/Asyn approaches can provide gains of up-to 25% compared to the HU schemes. Furthermore, because of different settings, where the HA-Asyn and HA-Sync outperform each other, this dissertation concludes by providing recommendations on how to select the appropriate scheme with the correct parameters, describing different application scenarios and identifying areas for future research.