This phase of the USCG Training and Licensing project established a methodology for determining performance standards to be used in measuring simulator training effectiveness. The Computer Aided Operations Research Facility (CAORF) Shiphandling Simulator was utilized to measure the shiphandling performance of experts, i.e., pilots. Special scenarios were designed to cover a variety of training objectives relevant to shiphandling. These objectives included: approaching a harbor, responding to a rudder failure in confined waters, negotiating a 51 turn with passing ship effects, negotiating a 129 turn around a shoal with incoming traffic, and responding to a propulsion failure in the vicinity of a bridge and shoal. Fourteen pilots were run through these scenarios in order to assess the responses of shiphandling experts to the test situations. Fourteen chief mates, upgrading to the master level, were then run through the same scenarios. The chief mates were considered novice shiphandlers. This report includes a detailed description of the test scenarios, performance measures, expert and novice data, and a method for deriving performance criteria to be used in assessing the effectiveness of simulator training. (Author).