Adaptive routing in data communication networks through reinforcement learning

Adaptive routing in data communication networks through reinforcement learning
Author:
Publisher:
Total Pages:
Release: 2000
Genre:
ISBN:

Esta dissertação investiga a aplicação dos métodos de Reinforcement Learning na descoberta de rotas ótimas em uma rede de comunicação. Uma rede de comunicação real possui um comportamento dinâmico, mudando seu estado com o tempo. Os algoritmos de roteamento devem, portanto, oferecer rapidez na resposta às mudanças do estado da rede. O objetivo do trabalho é avaliar a aplicação de técnicas de Reinforcement Learning (RL) como base de algoritmos adaptativos de roteamento de pacotes. O problema de roteamento de pacotes sob a visão de RL consiste na definição de cada nó na rede como um agente RL, sendo que este agente deve definir ações de forma a minimizar uma função objetivo que pode ser o tempo de roteamento dos pacotes. Um dos objetivos do RL é precisamente aprender a tomar as ações que minimizem uma função. O trabalho consistiu de 4 etapas principais: um estudo sobre a área de Reinforcement Learning (RL); um estudo sobre a área de redes de comunicação e roteamento de pacotes; a modelagem do problema de roteamento como um sistema RL e implementação de diferentes métodos de RL para obter algoritmos de roteamento; e o estudo de casos. O estudo na área de Reinforcement Learning abrangeu desde as definições mais fundamentais: suas características, os elementos de um sistema RL e modelagem do ambiente como um Processo de Decisão de Markov, até os métodos básicos de solução: Programação Dinâmica, método de Monte Carlo, e o método de Diferenças Temporais. Neste último método, foram considerados dois algoritmos específicos: TD e Q-Learning. Em seguida, foi avaliado o parâmetro Eligibility Traces como uma alternativa para apressar o processo de aprendizado, obtendo o TD(lambda) e o Q(lambda) respectivamente. O estudo sobre Redes de Comunicação e Roteamento de pacotes envolveu os conceitos básicos de redes de comunicações, comutação por pacotes, a questão do roteamento de pacotes e os algoritmos existentes adaptativos e não adaptativos, que são utilizados na atualidade. Nas redes de comunicação, definidas como um conjunto de nós ligados através de enlaces de comunicação, para se enviar uma mensagem de um nó a outro, geralmente, a mensagem é quebrada em pedaços, chamados pacotes, e enviados através de outros nós, até chegar ao destino. Deste modo surge o problema de escolher os nós que levem o pacote o mais rápido possível até o nó destino. Os algoritmos analisados foram: Shortest Path Routing que procura os caminhos com menor número de nós intermediários, não sendo sensível às mudanças na carga nem na topologia da rede; Weighted Shortest Path Routing, que oferece um melhor desempenho a partir de uma visão global do estado da rede, que nem sempre é fácil de obter em redes reais e o algoritmo de Bellman-Ford, baseado em decisões de roteamento locais e atualizações periódicas, com algumas limitações para obter políticas em altas cargas. Este último é um dos algoritmos mais utilizados na atualidade, sendo base de muitos protocolos de roteamento existentes. A modelagem do problema de roteamento como um sistema RL foi inspirada por uma característica na definição de um sistema RL: um agente que interage com o ambiente e aprende a atingir um objetivo. Assim, a modelagem dos algoritmos tem como objetivo aprender a descobrir as rotas que minimizem o tempo de roteamento de pacotes desde uma origem até um dado destino. A avaliação de uma rota escolhida não pode ser obtida antes que o pacote alcance o seu destino final. Este fato faz com que os processos de aprendizado supervisionado tenham dificuldade de se aplicar a esse problema. Por outro lado, o Reinforcement Learning não necessita de um par entrada-resposta para fazer o aprendizado, permitindo-lhe abordar o problema com relativa facilidade. Na modelagem efetuada, cada nó na rede se comporta como um agente de RL que age na própria rede, a qual é o ambiente. A informação das rotas é armazenada nas funções de valor existentes em todos os nós da rede para cada outro nó destino. Esta informação contém um valor estimado do tempo requerido para um pacote chegar até o nó destino. A atualização desses valores é feita durante a transição do pacote até o vizinho selecionado, isto é, aquele nó vizinho que apresente o menor valor estimado para alcançar o destino. Neste trabalho foram desenvolvidos vários algoritmos de roteamento, todos estes fazendo decisões de roteamento locais. Cada um dos algoritmos aplica características das técnicas no Reinforcement Learning: o Q(lambda)-Routing, baseado no Q(lambda); e o TD-Routing, baseado no TD(lambda). No estudo de casos, os algoritmos propostos foram avaliados utilizando-se uma ferramenta de simulação existente feita em linguagem C, após efetuar certas melhorias. Os algoritmos propostos foram inseridos na ferramenta, avaliados e comparados com as técnicas existentes, fazendo-se testes de adaptabilidade em carga baixa e alta, testes de mudança da carga e testes de mudança na topologia da rede. Como conclusões obtidas dos testes feitos no estudo de casos, os algoritmos propostos, em geral, forneceram respostas melhores no tempo de adaptabilidade às mudanças na rede.

A Reinforcement Learning Network based Novel Adaptive Routing Algorithm for Wireless Ad-Hoc Network

A Reinforcement Learning Network based Novel Adaptive Routing Algorithm for Wireless Ad-Hoc Network
Author: Jagrut Solanki
Publisher: GRIN Verlag
Total Pages: 11
Release: 2015-02-24
Genre: Technology & Engineering
ISBN: 3656904944

Scientific Essay from the year 2015 in the subject Engineering - Communication Technology, , language: English, abstract: Mobile communication has enjoyed an incredible rise in quality throughout the last decade. Network dependability is most important concern in wireless Ad-hoc network. a serious challenge that lies in MANET (Mobile Ad-hoc network) is that the unlimited mobility and lots of frequent failure because of link breakage. Standard routing algorithms are insufficient for Ad-hoc networks. as a results of major drawback in MANET is limited power provide, dynamic networking. In MANET each node works as a router and autonomously performs mobile practicality. The link connectivity changes ceaselessly because of mobility to reflect this routing information additionally needs to get changed ceaselessly. AODV protocol is projected for this extraordinarily mobile network. In ancient AODV if any node fails in middle of transmission the method starts from the source node but in our propose scheme the transmission starts from the closest neighbor node therefore shows very important reduction in delay and improvement in packet delivery ratio are achieved. It also reduces the routing overhead by reducing the frequency of route discovery process.

Mobile Ad Hoc Networks

Mobile Ad Hoc Networks
Author: G Ram Mohana Reddy
Publisher: CRC Press
Total Pages: 139
Release: 2016-08-19
Genre: Computers
ISBN: 1315351633

In recent years, a lot of work has been done in an effort to incorporate Swarm Intelligence (SI) techniques in building an adaptive routing protocol for Mobile Ad Hoc Networks (MANETs). Since centralized approach for routing in MANETs generally lacks in scalability and fault-tolerance, SI techniques provide a natural solution through a distributed approach for the adaptive routing for MANETs. In SI techniques, the captivating features of insects or mammals are correlated with the real world problems to find solutions. Recently, several applications of bio-inspired and nature-inspired algorithms in telecommunications and computer networks have achieved remarkable success. The main aims/objectives of this book, "Mobile Ad Hoc Networks: Bio-Inspired Quality of Service Aware Routing Protocols", are twofold; firstly it clearly distinguishes between principles of traditional routing protocols and SI based routing protocols, while explaining in detail the analogy between MANETs and SI principles. Secondly, it presents the readers with important Quality of Service (QoS) parameters and explains how SI based routing protocols achieves QoS demands of the applications. This book also gives quantitative and qualitative analysis of some of the SI based routing protocols for MANETs.

Applications of Reinforcement Learning to Routing and Virtualization in Computer Networks

Applications of Reinforcement Learning to Routing and Virtualization in Computer Networks
Author: Soroush Haeri
Publisher:
Total Pages: 157
Release: 2016
Genre:
ISBN:

Computer networks and reinforcement learning algorithms have substantially advanced over the past decade. The Internet is a complex collection of inter-connected networks with a numerous of inter-operable technologies and protocols. Current trend to decouple the network intelligence from the network devices enabled by Software-Defined Networking (SDN) provides a centralized implementation of network intelligence. This offers great computational power and memory to network logic processing units where the network intelligence is implemented. Hence, reinforcement learning algorithms viable options for addressing a variety of computer networking challenges. In this dissertation, we propose two applications of reinforcement learning algorithms in computer networks. We first investigate the applications of reinforcement learning for deflection routing in buffer-less networks. Deflection routing is employed to ameliorate packet loss caused by contention in buffer-less architectures such as optical burst-switched (OBS) networks. We present a framework that introduces intelligence to deflection routing (iDef). The iDef framework decouples design of the signaling infrastructure from the underlying learning algorithm. It is implemented in the ns-3 network simulator and is made publicly available. We propose the predictive Q-learning deflection routing (PQDR) algorithm that enables path recovery and reselection, which improves the decision making ability of the node in high load conditions. We also introduce the Node Degree Dependent (NDD) signaling algorithm. The complexity of the algorithm only depends on the degree of the node that is NDD compliant while the complexity of the currently available reinforcement learning-based deflection routing algorithms depends on the size of the network. Therefore, NDD is better suited for larger networks. Simulation results show that NDD-based deflection routing algorithms scale well with the size of the network and outperform the existing algorithms. We also propose a feed-forward neural network (NN) and a feed-forward neural network with episodic updates (ENN). They employ a single hidden layer and update their weights using an associative learning algorithm. Current reinforcement learning-based deflection routing algorithms employ Q-learning, which does not efficiently utilize the received feedback signals. We introduce the NN and ENN decision-making algorithms to address the deficiency of Q-learning. The NN-based deflection routing algorithms achieve better results than Q-learning-based algorithms in networks with low to moderate loads.The second application of reinforcement learning that we consider in this dissertation is for modeling the Virtual Network Embedding (VNE) problem. We develop a VNE simulator (VNE-Sim) that is also made publicly available. We define a novel VNE objective function and prove its upper bound. We then formulate the VNE as a reinforcement learning problem using the Markov Decision Process (MDP) framework and then propose two algorithms (MaVEn-M and MaVEn-S) that employ Monte Carlo Tree Search (MCTS) for solving the VNE problem. In order to further improve the performance, we parallelize the algorithms by employing MCTS root parallelization. The advantage of the proposed algorithms is that, time permitting, they search for more profitable embeddings compared to the available algorithms that find only a single embedding solution. The simulation results show that proposed algorithms achieve superior performance.

Interconnection Networks

Interconnection Networks
Author: Jose Duato
Publisher: Morgan Kaufmann
Total Pages: 626
Release: 2003
Genre: Computers
ISBN: 1558608524

Foreword -- Foreword to the First Printing -- Preface -- Chapter 1 -- Introduction -- Chapter 2 -- Message Switching Layer -- Chapter 3 -- Deadlock, Livelock, and Starvation -- Chapter 4 -- Routing Algorithms -- Chapter 5 -- CollectiveCommunicationSupport -- Chapter 6 -- Fault-Tolerant Routing -- Chapter 7 -- Network Architectures -- Chapter 8 -- Messaging Layer Software -- Chapter 9 -- Performance Evaluation -- Appendix A -- Formal Definitions for Deadlock Avoidance -- Appendix B -- Acronyms -- References -- Index.

Deep Reinforcement Learning for Wireless Communications and Networking

Deep Reinforcement Learning for Wireless Communications and Networking
Author: Dinh Thai Hoang
Publisher: John Wiley & Sons
Total Pages: 293
Release: 2023-06-30
Genre: Technology & Engineering
ISBN: 1119873738

Deep Reinforcement Learning for Wireless Communications and Networking Comprehensive guide to Deep Reinforcement Learning (DRL) as applied to wireless communication systems Deep Reinforcement Learning for Wireless Communications and Networking presents an overview of the development of DRL while providing fundamental knowledge about theories, formulation, design, learning models, algorithms and implementation of DRL together with a particular case study to practice. The book also covers diverse applications of DRL to address various problems in wireless networks, such as caching, offloading, resource sharing, and security. The authors discuss open issues by introducing some advanced DRL approaches to address emerging issues in wireless communications and networking. Covering new advanced models of DRL, e.g., deep dueling architecture and generative adversarial networks, as well as emerging problems considered in wireless networks, e.g., ambient backscatter communication, intelligent reflecting surfaces and edge intelligence, this is the first comprehensive book studying applications of DRL for wireless networks that presents the state-of-the-art research in architecture, protocol, and application design. Deep Reinforcement Learning for Wireless Communications and Networking covers specific topics such as: Deep reinforcement learning models, covering deep learning, deep reinforcement learning, and models of deep reinforcement learning Physical layer applications covering signal detection, decoding, and beamforming, power and rate control, and physical-layer security Medium access control (MAC) layer applications, covering resource allocation, channel access, and user/cell association Network layer applications, covering traffic routing, network classification, and network slicing With comprehensive coverage of an exciting and noteworthy new technology, Deep Reinforcement Learning for Wireless Communications and Networking is an essential learning resource for researchers and communications engineers, along with developers and entrepreneurs in autonomous systems, who wish to harness this technology in practical applications.

Distributed Routing for Very Large Networks Based on Link Vectors

Distributed Routing for Very Large Networks Based on Link Vectors
Author: Jochen Behrens
Publisher:
Total Pages: 238
Release: 1997
Genre: Algorithms
ISBN:

Routing is the network-layer function that selects the paths that data packets travel from a source to a destination in a computer communication network. This thesis is on distributed adaptive routing algorithms for large packet-switched networks. A new type of routing algorithms for computer networks, the link-vector algorithm (LVA) is introduced. LVAs use selective dissemination of topology information. Each router running an maintains a subset of the topology that corresponds to adjacent links and those links used by its neighbor routers in their preferred paths to known destinations. Based on that subset of topology information, the router derives its own preferred paths and communicates the corresponding link-state information to its neighbors. An update message contains a vector of updates; each such update specifies a link and its parameters. LVAs can be used for different types of routing policies. LVAs are shown to have better performance than the ideal link-state algorithm based on flooding and the distributed Bellman-Ford algorithm.

Vehicular Ad Hoc Networks

Vehicular Ad Hoc Networks
Author: Claudia Campolo
Publisher:
Total Pages:
Release: 2015
Genre:
ISBN: 9783319154985

This book presents vehicular ad-hoc networks (VANETs) from the their onset, gradually going into technical details, providing a clear understanding of both theoretical foundations and more practical investigation. The editors gathered top-ranking authors to provide comprehensiveness and timely content; the invited authors were carefully selected from a list of who's who in the respective field of interest: there are as many from Academia as from Standardization and Industry sectors from around the world. The covered topics are organized around five Parts starting from an historical overview of vehicular communications and standardization/harmonization activities (Part I), then progressing to the theoretical foundations of VANETs and a description of the day-one standard-compliant solutions (Part II), hence going into details of vehicular networking and security (Part III) and to the tools to study VANETs, from mobility and channel models, to network simulators and field trial methodologies (Part IV), and finally looking into the future of VANETs by investigating alternative, complementary communication technologies, innovative networking paradigms and visionary applications (Part V). The way the content is organized, with a differentiated level of technical details, makes the book a valuable reference for a large pool of target readers ranging from undergraduate, graduate and PhD students, to wireless scientists and engineers, to service providers and stakeholders in the automotive, ITS, ICT sectors.

The Adaptive Routing Problem for Large Store-and-Forward Computer-Communication Networks

The Adaptive Routing Problem for Large Store-and-Forward Computer-Communication Networks
Author: C. E. Houstis
Publisher:
Total Pages: 174
Release: 1977
Genre:
ISBN:

This research investigates and evaluates routing procedures for large store-and-forward message switched computer communication networks. Existing network topologies are examined. An adaptive routing algorithm is developed dealing with message routing when the full length message is transmitted in its entirety. A thresholding technique is used to work with variations in message length. The algorithm's performance is evaluated by comparison with the performance of the exact mathematical deterministic routing problem. In order to apply the routing strategy to large networks, a partitioning approach is used. A partitioning algorithm is developed for communications network topologies. This algorithm uses a generalized labeling technique to find the appropriate partitions of the original network. The adaptive routing algorithm is modified in order to take into account large network considerations. Then it is applied to the partitioned network in a two-level hierarchical structure. The exact two-level modified adaptive routing algorithm is described. Some new network topologies found in the literature are examined from the routing problem point of view. All the algorithms are tested by computer simulation. The adaptive routing algorithms have been implemented using two different simulation languages. (Author).