Methods for Open Agent Systems Evaluation Initiative (MOASEI) 2026 Competition

Track #2: Rideshare (Task Openness Only)

Agents operating autonomous cars within a ridesharing application decide how to prioritize dynamically appearing passengers as tasks.

Rideshare Configurations

• Grid: 15x15
• Agent Start Positions: (13,11), (8,3), (2,14)
• Passenger Schedule: 70 passengers, appearing at various times and locations with diverse destinations.
• Pool Limit: 4
• Travel: No diagonal or fast travel

• Grid: 15x15
• Agent Start Positions: (12,9), (4,1), (0,8)
• Passenger Schedule: 65 passengers, staggered arrivals and varied destinations across the grid.
• Pool Limit: 4
• Travel: No diagonal or fast travel

• Grid: 15x15
• Agent Start Positions: (0,0), (2,14), (7,2)
• Passenger Schedule: 65 passengers, requests distributed throughout the episode with a range of trip lengths.
• Pool Limit: 4
• Travel: No diagonal or fast travel

Track #3: Wildfire (Both Agent and Task Openness)

Agents decide how to use limited suppressant resources to collaboratively put out wildfire tasks that appear both spontaneously and due to realistic fire-spread mechanics. Agents must temporarily disengage when they run out of limited suppressant to recharge before rejoining the firefighting efforts.

Wildfire Configurations

• Grid: 3x3
• Initial Fires: Lit at (0,0), (1,0), (1,2), (2,0)
• Agent Positions: (0,1), (1,1), (2,2)

• Grid: 3x3
• Initial Fires: Lit at (1,0), (1,1), (1,2), (2,1)
• Agent Positions: (0,0), (2,0), (2,2)

• Grid: 3x3
• Initial Fires: Lit at (0,0), (0,1)
• Agent Positions: (0,2), (1,1), (2,2)

Track #3A: Wildfire Bonus (Agent, Task and Frame Openness)

• Grid: 3x3
• Initial Fires: Lit at (0,0), (1,0), (1,2), (2,0)
• Agent Positions: (0,1), (1,1), (2,2)
• Equipment: This scenario introduces "frame openness"—agents can transition between equipment states, which increases or decreases their maximum suppressant capacity.

Evaluation

For each competition track, participants must submit a single set of policies (one for each agent) that will be applied across all scenarios within that track. The sole exception is the optional bonus track (3A) in the Wildfire domain, participants may elect to submit a distinct, dedicated set of policies for this specific challenge. Performance on the bonus track will not influence the primary competition rankings and will only be utilized as a tie-breaker if necessary. Participants are encouraged to attempt the bonus track to improve the overall applicability of their solution.

The scenarios described for each track represent publicly released training configurations. The testing configurations used for final evaluation are drawn from the same distribution as the training configurations but are not guaranteed to be identical to the training set. However, the format of observations provided by the environment will remain unchanged.

Solutions will be evaluated based on their performance on these testing configurations. Solutions that fail to execute to completion will be evaluated with respect to the tasks successfully completed prior to failure.

Domain-independent MARL Performance Measures: All solutions in all tracks will be evaluated according to the cumulative rewards earned by agents making decisions while operating in test scenarios of the domain utilizing the submitted MARL solution. By using the same performance measures across tracks/domains, we can also try to establish general trends that could generalize to other OASYS domains and characterize the solutions submitted by competitors.

Domain-specific Task Performance Measures: Ultimately , the success of MAS solutions for real-world applications and our ability as researchers to convince the general public of the usefulness of AI is measured by the ability of agents to accomplish domain-specific tasks. Thus, we will also measure the performance of solutions based on task-specific measures. In the Cybersecurity Defense domain, this includes the amount of time that different resources in the network remained free from infiltration by attackers, as well as the average amount of infiltration in each resource. In the Ridesharing domain, this includes the number of passengers successfully transported to their destination, the average wait time before passengers were picked up, and the average time a passenger rode in a car before arriving at their destination. In the Wildfire Suppression domain, this includes the total number of fires extinguished vs. burned out, the average duration of each fire and the efficiency of limited suppressant usage by agents.

Determining winners: The winner of each track will be determined based on a comprehensive evaluation of each of the above performance measures in each of the scenarios within the track. In particular, the highest scoring submission in each performance measure per scenario will receive points (where n ins the number of submissions to that track), the second highest scoring will receive n-1 points, etc. The points in each performance measure will be summed up, and the highest scoring team in each track will be determined to be the winner of that track.

Registration Guidelines

Participants must register their teams using the following form: Register Here.

After the registration, the MOASEI commitee will confirm your participation and provide directions for your submission.

Rules

To ensure a fair and competitive environment, we have established the following rules:

Both individuals and teams are allowed to register for the competition and submit solutions to one, two, or all three tracks of their choice. Participants must register for the competition. Submitted solutions are expected to run without errors on our computing platform before the submission deadline. Participants will have access to the simulators in advance to thoroughly test that their solutions operate without error.

We will provide a shared set of 2-3 scenarios within each track (e.g., different configurations of attackers and defender agents in the Cybersecurity Defense track, different numbers and challenges of fires in the Wildfire Suppression track) that participants will use for developing and training their solutions. Participants will be able to randomly generate episodes for each scenario in each track using different random seeds (or repeat the same episode using the same seed).

We are encouraging MARL solutions within the competition; however, solutions utilizing other decision-making paradigms (heuristics, planning, game theory) are permissible to ensure the broadest participation and reflect the full range of decision making possible in OASYS. At the same time, our simulation environments provide information most compatible with MARL solutions, such as observations of current states and rewards after taking each action. The APIs of the simulation environments will not provide access to the models of underlying state transition, observation, and reward dynamics. Communication between agents will not be supported nor permitted.

Solution submissions will consist of both (1) the source code needed to operate an agent within the simulators of the three domains (following a provided API structure for standardization), as well as (2) any serialized data files needed to operate the solutions (e.g., serialized neural networks for deep MARL solutions). We will impose a reasonable storage size for solutions (to be determined very soon) for the sake of competition management. Trained models must be submitted; we will not have computational capacity to retrain models that require extensive computation time for training. Timing of decision making will be enforced so that agents do not have unlimited time to make each decision, allowing a reasonable amount of time per decision, reflecting real-world operations.

Use of LLMs

Teams are welcome to use Large Language Models (LLMs) during the training and development phase of the competition, for example, for code generation, documentation, or offline policy design. However, during official evaluation runs, agents must operate without the use of any LLMs, regardless of whether they are accessed via external APIs or run locally. The inclusion of model weights, self-contained LLM inference engines, or local model files within the submission is strictly prohibited. Any LLM-derived features must be distilled into static code or standard non-LLM policy structures prior to submission, ensuring the evaluation is performed without active LLM inference.

Simulation Platform

The competition will be conducted within the Python programming language. We will provide simulation environments for the three tracks, implemented within our Free-Range Zoo MARL testbed – an OASYS variant of the popular PettingZoo MARL testbed– with a consistent API for all environments, documentation can be found here. Participants can use the following link to download the training configurations. Solutions can utilize popular frameworks for implementing reinforcement learning solutions, including the Tensorflow and PyTorch libraries for deep learning. In addition to user supplied code, we will permit any external libraries, so long as they can be downloaded through PyPI with pip using a requirements.txt document.

Solutions submitted by participants will be evaluated on research infrastructure hosted by the organizers's institutions.

Contact

For any queries or further information, please reach out to us at:

Email: Tbillings4@huskers.unl.edu

Acknowledgements

This research was supported by a collaborative NSF Grant #IIS-2312657 (to P.D.), #IIS-2312658 (to L.K.S.), and #IIS-2312659 (to A.E.). Additionally, this work was completed utilizing the Holland Computing Center of the University of Nebraska, which receives support from the UNL Office of Research and Economic Development, and the Nebraska Research Initiative. Finally, we thank the graduate and undergraduate students who have contributed to the development and testing of MOASEI: Ceferino Patino, Daniel Redder, Alireza Saleh Abadi, and Tyler Billings.