This work addresses the environmental recognition of autonomous off-road vehicles. Algorithms, like deep learning, offer impressive performance regarding the classification and segmentation of a scene. However, context changes, scene variabilities, or disturbances pose significant challenges to these approaches and cause perception failures. A challenge is achieving the universal applicability of perception algorithms. Usually, an algorithm fails in particular situations due to unconsidered circumstances in the design phase, and complexity prevents fully considering all details. Accordingly, this thesis aims to increase the perception’s robustness through context and data incorporation. Furthermore, it derives concepts for transferring methods to other robots and scenes. A hint that such a task is achievable provides human cognition, which is remarkably skillful and adjusts to arbitrary situations. Biologically motivated perception and cognitive research indicate how an achievable perception design might function, leading to guidelines for artificial perception conception. The paradigm of behavior-based systems suits these criteria due to modularity, reactivity, and robustness. It allows realizing robust and transferable perception and control systems. Consequently, the thesis proposes a novel and reconfigurable behavior-based top-down and bottom-up perception approach. Quality assessment for data filtering and deviation control is a central aspect, resulting in improved perception and data fusion results. Attentional processing allows for selecting data based on attractiveness, task, environmental context, and history. Further, context assessment of classification results enables reasoning according to the robot’s memories and knowledge. Validation uses five demonstrator vehicles operating in diverse environments and fulfilling distinct tasks. Here, a robust performance was achievable, and perception adjusted well to the tested scenes and hardware layouts.