pkmn.ai

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque ultrices, tortor sed iaculismollis, odio ex porta ante, ac malesuada est elit eu elit. Etiam dapibus quam id lorem condimentummollis. Donec non dolor ipsum. Praesent vel ultrices ex. Donec nibh dui, hendrerit a erat et,tincidunt semper orci. Cras in lectus sapien. Proin vehicula eros varius, eleifend neque laoreet,iaculis leo. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse in quamcondimentum, eleifend felis vitae, congue lectus. Maecenas arcu urna, dictum et auctor quis,vestibulum sit amet odio. Aliquam felis neque, fringilla et porttitor a, posuere at magna.

An innovative and successful Generation VII Random Battle agent, Athena’s core thesis was that competitive Pokémon artificial intelligence requires an approach uniquely tailored to the domain as opposed to being something that can be achieved through the naive repurposing of off-the-shelf solutions used for computer chess or Go. The paper introducing Athena dismissed the machine learning approaches attempted by other projects for the following reasons:

• Pokémon’s complicated game mechanics and massive state space make it hard for neural networks to infer patterns.
• Pokémon’s high variance means reinforcement learning agents are suboptimal as they can get rewards while playing poorly.
• Every Pokémon battle is almost a unique environment likely to have zero samples in the training set, necessitating an approach that properly accounts for this by training on past turns of ongoing battles.

Furthermore, the paper claimed that traditional search methods are insufficient in Pokémon as game tree pathology is present in all game trees due to hidden information and stochasticity, yet strong play in Pokémon requires the ability to look ahead meaning determining the correct amount of look-ahead is crucial. Finally, the paper contended that “keeping a balanced team” is of central importance — at every point, players need to be sure that they have at least one way of beating all of an opponent’s Pokémon without losing all of their own Pokémon — and that look-ahead with minimax and Monte Carlo tree search are insufficient to achieve that.

Maintaining team balance was at the core of Athena’s algorithm — its process for making decisions involved using two turns of look-ahead before calculating the probability of it winning from each state, relying on a distributed transposition table and hand-crafted forward pruning heuristics to greatly reduce the number of states explored. To compute its probability of winning, Athena performed pairwise Pokémon matchup comparisons using a further 1-2 turns look-ahead in each of the individual 1v1s after which it computed a score based on the difference in HP and a bonus for not fainting. These scores were then combined into a single value that could be added to a payoff matrix by looking at the average of scores weighted by an estimated probability of each individual matchup actually occurring.

Athena fed the computed payoff matrix along with a history of its opponent’s moves and their corresponding payoff matrices into an unspecified machine learning algorithm that returned the probability of their opponent making each of their available moves, multiplied these probabilities by their respective payoffs, assumed their opponent would greedily choose the move with the lowest payoff for the player, and then chose whatever move was the best response to their opponent’s choice.

In cases where an opponent’s Pokémon’s species were known Athena adopted the usual practice of relying on usage statistics to deduce the most likely values of its other unknown attributes. Given that Athena’s domain was a random format it could not rely on teammate statistics to fill in unknown species and it instead took a more involved but novel approach: Athena used the IBM® ILOG® CPLEX® Optimizer and a precomputed table of 1v1 matchup scores for each species with every other species to solve a mixed integer linear programming problem that determined which species of Pokémon would best approximate the battle between the agent’s team and any particular possible values for the opponent’s unrevealed Pokémon.

Finally, Athena heavily abstracted the random number generator to pare down the state space explosion that would otherwise result from attempting to account for stochasticity. Eschewing determinizations due to their high branching factor and Monte Carlo sampling due to the Pokémon Showdown engine’s slow simulation speed, Athena instead modified Pokémon Showdown’s PRNG to return one of 8 different preconfigured values constantly for each side when called during the first turn of look-ahead followed by a constant value in the middle of the range for subsequent turns, dramatically reducing the number of possible states that the agent would need to explore.

The pmariglia/showdown project features a bespoke engine, a sophisticated client, and several agents that can achieve a high level of play across numerous generations. The project primarily supports both random and non-random Generation III-VIII singles formats, though also contains partial support for Generation IX — data and mechanics save for those added by The Indigo Disk DLC are present but support for Terastallization is still in progress.

The project’s engine relies on move and species data generated from Pokémon Showdown, modified to then work in the custom engine and patched for past generations. The engine is simplified by only reflecting generational differences via its data files and sharing a single control flow across generations though this reduced fidelity is unlikely to matter for its agents. Performance is a concern (to the point where the engine is undergoing a rewrite in Rust), but this is mitigated by the engine’s decision to migrate away from a traditional copy-based approach to an approach that is currently unique among Pokémon engines: the engine’s StateMutator allows for applying and reversing “instructions” that get generated by handlers. This architecture allows the project’s agents to very efficiently update the battle during search based on the actions generated by its $transitions$ function, reversing each of them before exploring other branches.

The project’s client contains the usual support for Pokémon Showdown protocol and team parsing, though goes far beyond others by making use of log ordering to bound speed ranges and by deducing not just the presence but also the absence of Choice Scarf, Choice Band or Choice Specs, Heavy Duty Boots, Assault Vest, and Life Orb. Choice Band and Choice Specs inference also relies on the engine’s damage calculator which uniquely supports several different approaches for bucketing the damage rolls.

The project includes a baseline $MaxDamagePlayer$ which relies on this same damage calculator, though the project’s primary agents are based on depth 2 simultaneous move expectiminimax search with pruning and move ordering utilizing a hand-crafted evaluation function and differ primarily in how they choose to select an action based on the results of the search — one deterministically elects to pick the ‘safest’ option whereas the other explicitly attempts to play the mixed Nash equilibrium solution it computes by calling Gambit in a subprocess in game states where most information has been revealed. While this experiment into nondeterminism might help with exploitability, ultimately the Nash equilibrium bot proves much weaker overall than the vanilla approach. An alternative experiment attempts to enhance the default agent with dynamic search depths, possibly doubling search depth when run via PyPy in scenarios where the number of joint actions is small.

For each decision, many different determinizations for unknowns of the opponent’s active Pokémon are searched. Usage statistics are parsed ahead of time are used to produce possible combinations of spreads, items, abilities, and moves. Cutoffs and bucketing in addition to heuristics are used to pare down the number of possible states to only those with are logical and most likely. During the expectiminimax search the usage statistics also factor in as the “effectiveness” of each matchup as computed from the Checks and Counters statistics are included in the scoring function. The scoring does not place value on preserving hidden information or scouting, though in the future hopes to explicitly consider the notion of win conditions and give bonuses in attempts to better improve chances of particular Pokémon sweeping. Tweaks to evaluation may also be necessary to address the agents’ exhibited issues around closing out games.

The agents can be configured to play with fixed teams in non-random formats, but also include special support for Random Battle formats, tracking random battle sets and statistics in addition to slightly tweaking the scoring of healthy Pokémon during evaluation in random formats. In non-random formats, an augmented version of the safe expectiminimax agent that is configured with a team scouting dataset gleaned from scraping Smogon University’s forums can utilize team prediction to considerably improve playing performance.

Future Sight. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque ultrices, tortor sed iaculismollis, odio ex porta ante, ac malesuada est elit eu elit. Etiam dapibus quam id lorem condimentummollis. Donec non dolor ipsum. Praesent vel ultrices ex. Donec nibh dui, hendrerit a erat et,tincidunt semper orci. Cras in lectus sapien. Proin vehicula eros varius, eleifend neque laoreet,iaculis leo. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse in quamcondimentum, eleifend felis vitae, congue lectus. Maecenas arcu urna, dictum et auctor quis,vestibulum sit amet odio. Aliquam felis neque, fringilla et porttitor a, posuere at magna. Morbisodales maximus sapien commodo venenatis.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque ultrices, tortor sed iaculismollis, odio ex porta ante, ac malesuada est elit eu elit. Etiam dapibus quam id lorem condimentummollis. Donec non dolor ipsum. Praesent vel ultrices ex. Donec nibh dui, hendrerit a erat et,tincidunt semper orci. Cras in lectus sapien. Proin vehicula eros varius, eleifend neque laoreet,iaculis leo. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse in quamcondimentum, eleifend felis vitae, congue lectus. Maecenas arcu urna, dictum et auctor quis,vestibulum sit amet odio. Aliquam felis neque, fringilla et porttitor a, posuere at magna. Morbisodales maximus sapien commodo venenatis.In nunc felis, porta non diam in, faucibus condimentum nulla.

Technical Machine. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque ultrices, tortor sed iaculismollis, odio ex porta ante, ac malesuada est elit eu elit. Etiam dapibus quam id lorem condimentummollis. Donec non dolor ipsum. Praesent vel ultrices ex. Donec nibh dui, hendrerit a erat et,tincidunt semper orci. Cras in lectus sapien. Proin vehicula eros varius, eleifend neque laoreet,iaculis leo. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse in quamcondimentum, eleifend felis vitae, congue lectus. Maecenas arcu urna, dictum et auctor quis,vestibulum sit amet odio. Aliquam felis neque, fringilla et porttitor a, posuere at magna. Morbisodales maximus sapien commodo venenatis.In nunc felis, porta non diam in, faucibus condimentum nulla. Quisque eu vulputate odio. Sedpharetra massa et venenatis dignissim. Vivamus aliquet consequat nibh, non tristique liberoullamcorper eu. Fusce eget suscipit metus.

Tse built a Generation VIII VGC agent that leveraged both supervised learning and deep Q-learning with poke-env to achieve success against $RandomPlayer$ and $MaxDamagePlayer$ baselines.

Tse used a dataset of 700,000 Pokémon Showdown server battle logs gathered from the top 20% of ranked VGC players on the site to train a 6-layer neural network with supervised learning. The network’s output layer consisted of two softmax units producing action and value estimates, with a move selection and move target being considered separately. Despite using categorical encodings for the input embedding (as opposed to a one-hot key embedding) the initial input size of 4186 features resulted in overfitting. As a result, Tse used the supervised network to learn a smaller embedding of 254 features based on the output of the first layer which was designed to not be fully connected so as to be used for this purpose.

The learned embedding was used as input to a second RL network which used deep Q-learning and an ε-greedy policy to learn action selection. The reward for this network was computed based on a linear combination of the battle outcome, the current percentage HP for both sides’ active Pokémon, and the value output from the supervised learning network. The training of the RL network was performed against the $RandomPlayer$ and $MaxDamagePlayer$ and this led to a learned policy that proved to be fairly exploitable by less contrived opponents.

Battle simulation performance was cited as the main bottleneck, discouraging the addition of Monte Carlo tree search and serving as a barrier to increased training time.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque ultrices, tortor sed iaculismollis, odio ex porta ante, ac malesuada est elit eu elit. Etiam dapibus quam id lorem condimentummollis. Donec non dolor ipsum. Praesent vel ultrices ex. Donec nibh dui, hendrerit a erat et,tincidunt semper orci. Cras in lectus sapien. Proin vehicula eros varius, eleifend neque laoreet,iaculis leo. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse in quamcondimentum, eleifend felis vitae, congue lectus. Maecenas arcu urna, dictum et auctor quis,vestibulum sit amet odio. Aliquam felis neque, fringilla et porttitor a, posuere at magna. Morbisodales maximus sapien commodo venenatis.In nunc felis, porta non diam in, faucibus condimentum nulla. Quisque eu vulputate odio. Sedpharetra massa et venenatis dignissim. Vivamus aliquet consequat nibh, non tristique liberoullamcorper eu.

reuniclusVGC extends poke-env’s double battles support to target Generation VIII VGC formats. An experimental agent was built using TensorFlow and Keras and featured a small DQN model that was trained to play a fixed team against baseline agents. While this agent eventually learned to roughly approximate $MaxDamagePlayer$ (though with the ability to switch when in an obviously disadvantageous position and to Dynamax immediately), the agent did not generalize well to new teams or opponents. This initial foray into the field helped motivate a new approach based on a “model-based search algorithm with depth-limited and heavily pruned search, not assuming unobservable information, and a Nash Equilibrium solver” that will serve as the direction for future work. Furthermore, the initial reuniclusVGC training revealed limitations in the standard singles baseline agents leading to the versions of $RandomPlayer$ and $MaxDamagePlayer$ which were augmented with heuristics to allow for more realistic training in the doubles formats being targeted.

The first competitive Pokémon AI agent to be described in a paper, Percymon targeted the Generation VI Random Battle format. The bot relied on pure depth 2 pessimistic sequentialized minimax with alpha-beta pruning and move ordering (e.g., preferring to explore super effective or status moves before those which are not very effective) with a hand-crafted evaluation function based on an early version of Technical Machine’s features and weights. Percymon also included several hardcoded special cases for certain moves which have intricate game mechanics that were hard for the agent to properly evaluate.

Due to Percymon being built around an unmodified copy of Pokémon Showdown it was unable to easily make use of the expectiminimax algorithm. Furthermore, while the agent was mostly deterministic, the damage and turn simulations it ran via Pokémon Showdown did not fix a random seed and thus the bot’s policies could exhibit a small amount of non-determinism. The Pokémon Showdown engine was found to be a major performance bottleneck, with the cloning of the engine’s state between simulations cited as the major limiting factor on depth as even with pruning turns could frequently take in excess of 40 seconds to evaluate.

Percymon chose to simplify the action space by always mega evolving when available as a form of forward pruning. It also chose to simplify the question of imperfect information by assuming that an opponent’s unrevealed Pokémon do not exist and that an opponent has all possible moves from their random move pool until all of their moves have been revealed (as opposed to leveraging usage stats). In order to work around the increased move acion space resulting from this approach, Percymon only looked at the top 10 opponent choices (i.e, forward pruning via beam search).

Percymon planned to use TD-learning for the evaluation function weights and considered supporting non-random formats by adding in a usage statistics-backed Bayesian network for team prediction coupled with a CSP solver to build robust teams.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque ultrices, tortor sed iaculismollis, odio ex porta ante, ac malesuada est elit eu elit. Etiam dapibus quam id lorem condimentummollis. Donec non dolor ipsum. Praesent vel ultrices ex. Donec nibh dui, hendrerit a erat et,tincidunt semper orci. Cras in lectus sapien. Proin vehicula eros varius, eleifend neque laoreet,iaculis leo. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse in quamcondimentum, eleifend felis vitae, congue lectus. Maecenas arcu urna, dictum et auctor quis,vestibulum sit amet odio. Aliquam felis neque, fringilla et porttitor a, posuere at magna. Morbisodales maximus sapien commodo venenatis.In nunc felis, porta non diam in, faucibus condimentum nulla. Quisque eu vulputate odio.

Inspired by Synedh/showdown-battle-bot, the hsahovic/reinforcement-learning-pokemon-bot is the precursor to poke-env. Perhaps most notable as the origin of various patches to Pokémon Showdown that were eventually upstreamed Pokémon Showdown to make it more suitable for reinforcement learning training, the project uses Keras and Tensorflow to experiment with reinforcement learning approaches. The project features a client representation that is used to produce a description of the state to be fed into models and contains support for self-play and concurrent training. The project contains two models built with a fully-connected multi-layer perceptron architecture - an incomplete larger model that utilizes the full feature set and a functional model model that uses the Actor-Critic method with a pruned number of features. The latter agent’s reward depends on the difference in respective total party hit points.

alphaPoke. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque ultrices, tortor sed iaculismollis, odio ex porta ante, ac malesuada est elit eu elit. Etiam dapibus quam id lorem condimentummollis. Donec non dolor ipsum. Praesent vel ultrices ex. Donec nibh dui, hendrerit a erat et,tincidunt semper orci. Cras in lectus sapien. Proin vehicula eros varius, eleifend neque laoreet,iaculis leo. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse in quamcondimentum, eleifend felis vitae, congue lectus. Maecenas arcu urna, dictum et auctor quis,vestibulum sit amet odio. Aliquam felis neque, fringilla et porttitor a, posuere at magna. Morbisodales maximus sapien commodo venenatis.

The Showdown AI Competition pitted a wide variety of agents against each other within a custom Generation VII Random Battle format with Zoroark, Ubers, and Pokémon below NU banned, 20 seconds allotted time per decision and a hard 500-turn limit. The competition involved multiple rounds of two best-of-3 matches with team swap to reduce variance. The following approaches were compared:

• non-adversarial breadth-first search that greedily assumes an opponent does nothing and a variant that prunes moves which are resisted / switches into poor type matchups and assumes the opponent is a one-turn look-ahead agent instead.
• pure pessimistic minimax with an evaluation function that favors dealing damage while rewarding survival and a depth penalty to promote exploration and limit deep searches.
• both single-layer and multi-layer Q-learning networks.
• one-turn look-ahead that computes estimated damage and uses the max-damaging move or switches to the Pokémon that has the max-damaging move and a variant that includes a killer move heuristic as well as additional rules and heuristics around type matchups and switching.

Every approach was an improvement over $RandomPlayer(\widetilde{100\%})$, but the basic one-turn look-ahead agent ended up with the best results followed minimax and pruned breadth-first-search which both also significantly outperformed alternatives. High simulation cost was deemed a major limiting factor to agent performance.

Chen et al. trained an agent using proximal policy optimization (PPO), anticipating an approach using Q-learning would struggle to learn a value function for a game as complicated as Pokémon. While their bot targeted the Generation I Random Battle format the agent was generalized and could be trained on arbitrary formats.

The agent utilized Tensorfloww, the PPO implementation from OpenAI and a custom OpenAI Gym environment that works with a modified Pokémon Showdown server. The server was relaxed to provide additional information about features and to determine valid actions ahead of time which are used to mask off the output from the RL network to force the logits of invalid actions to negative infinity before applying the softmax.

While Chen et al. discussed using the node2vec algorithm to create embeddings for various species, it seems as though a basic one-hot encoding for each Pokémon’s attributes was ultimately used. The model’s network architecture featured 3 fully-connected layers each with 512 units and a ReLU activation with only terminal states being rewarded. The agent trained against $RandomPlayer(\widetilde{100\%})$, $RandomPlayer(0\%)$, and a depth 1 minimax agent with alpha-beta pruning with an evaluation function that compared the difference in the sum of HP percentages for the full team of each side.

The agent eventually learned a policy that preferentially chose the move in the 4th moveslot, calling into question the results. Chen et al. reveal they were largely bottlenecked by computational resources, in particular repeatedly cloning the simulator state, and their ideas for future work included self-play and improved visualization to enhance tuning and debugging.

alexzhang13/reward-shaping-rl demonstrated the use of large language models (LLMs) for reward shaping, offering three different methods for providing feedback: sequential, tree-based and moving target. These methods were compared while training agents via proximal policy optimization (PPO1 from stable_baselines) and deep Q-learning with experience replay (a handwritten sequential 512x512x256 network with ReLU activations between layers) and appeared to boost sample efficiency. The model was trained on a single Generation VIII Ubers team, allowing it to ignore abilities and items and instead treat them as properties of the specific Pokémon. The input embedding was limited due to resource constraints but included the active moves and (unboosted) stats, HP, and status for both sides. Results indicated the final models were superior to $RandomPlayer(\widetilde{100\%})$, roughly on par with $RandomPlayer(0\%)$, and significantly worse than a base-power variant of $MaxDamagePlayer$.

Having adapted the toy engine from DanielAlcocerSoto/Pokemon-Python to singles, blue-sky-sea/Pokemon-MCTS-AI-Master implemented an agent based on UCT that reported positive results versus $RandomPlayer$ and $MaxDamagePlayer$.

Kalose et al. implemented a custom deterministic simulator for the Generation I Challenge Cup format and evaluated the efficacy of an agent trained on self-play utilizing Q-learning. The agent’s feature vector was limited to just the active Pokémon from either side’s types and corresponding “HP bucket” (an abstraction of a Pokémon’s remaining HP percentage into 10 evenly distributed bins), deliberately avoiding any information about either side’s inactive party due to performance concerns. In addition to the large terminal reward based on the battle’s final result, smaller intermediate rewards were provided to the agent based on the remaining HP for both sides. A softmax exploration strategy was seen to outperform an ε-greedy one versus $RandomPlayer$, though overall playing strength was poor compared to a benchmark minimax solution. Suggestions for future work included introducing eligibility traces and expanding the size of the feature vector.

One of the earliest known examples of a competitive Pokémon AI, Shanai was most notable for the creation of the “Shanai Cup”, a simplified variant of Pokémon that hoped to serve as the testbed for AI development. Shanai only supported this restricted Generation V variant and its core algorithm can be recognized as an extension of the rudimentary $MaxDamagePlayer$ benchmark. The project also contains the beginnings of a reverse damage calculator, though ultimately the project was abandoned before this feature was incorporated.

An early agent targeting the Generation VI OU format that leveraged depth 2 determinized and sequentialized pure minimax search with alpha-beta pruning and a hand-crafted evaluation function to play with predefined teams. While eschewing expectiminimax or a simultaneous-move minimax, the AI experimented between both optimistic and pessimistic sequentializations, seemingly preferring the latter.

The AI took an unorthodox approach towards integrating with the Pokémon Showdown platform, relying on a scriptable headless browser instead of the more obvious WebSocket API, complicating connection and parsing, possibly due to misunderstandings or platform limitations that are no longer relevant.

The agent relied on a custom engine for simulating actions, replete with simplified move handler and damage calculator implementations. The engine removed all stochastic elements (e.g., by ignoring accuracy checks or the contribution of damage rolls) and as such the AI’s game state was patched between actions to better keep it synchronized with the true server state.

The AI featured relatively sophisticated team prediction capabilities: movesets were scraped from Smogon University, with the agent falling back on data from the prior generation where necessary. These served as the basis for an opponent’s predicted set, though with the set’s moves instead replaced with the moves that occur most frequently as computed from a replay database consisting of logs from top ladder players. This database was also used to produce $P(Move \mid Move)$ and $P(Move \mid Species \land Move)$ correlations that allowed the agent to use Bayesian inference as further moves get revealed to better predict an opponent’s moveset.

The first framework to be published for competitive Pokémon artificial intelligence, Leftovers Again aimed to lower the barrier of entry for beginners. Supporting Generation VII (both random and team-based formats), Leftovers Again provided a client and damage calculator along with instructions for getting started and infrastructure for running and debugging agents. Leftovers Again featured support for round-robin tournaments against the various (mostly rules-based) sample bots it bundled, including $RandomPlayer$ (“randumb”) and $MaxDamagePlayer$ (“stabby”) implementations. Leftovers Again also included replay scraping and processing functionality, the latter of which could be used to produce $P(Move \mid Species \land Move)$ bigrams.

Bill’s PC was a promising but ultimately unrealized and overlooked attempt at implementing a competitive Pokémon AI based on more contemporary game theory techniques. Bill’s PC was written in Python and targeted the Generation VI Random Battle format, though hoped to eventually support Generation VII and recognized a rewrite in C++ would likely be necessary to achieve the performance required for a high level of play.

Bill’s PC featured several novel ideas — it was the first known example of producing usage statistics for the Random Battle formats by leveraging Pokémon Showdown’s team generation logic, a technique independently rediscovered and popularized by pkmn/randbats half a decade later. However, Bill’s PC’s processed statistics were significantly richer than those offered by pkmn/randbats, approaching the depth of those offered by Smogon University for non-random formats.

Additionally, Bill’s PC demonstrated the ability to reuse its own high-quality custom engine implementation for damage calculation, overriding certain functions to force specific damage rolls or critical hit outcomes in order to compute damage ranges — a primitive but powerful initial version of the functionality that later could be found in libpkmn.

Bill’s PC’s client included a distinct opponent representation tracking known and possible moves and attempted to deduce an opponent’s Hidden Power type, deterministically filled in unknown attributes with their most likely value based on the usage statistics, and also tried to leverage typing information to determine which unknown species of Pokémon might “balance” an opponent’s team composition.

Bill’s PC’s roadmap showed plans to implement “SM-MCTS for stacked matrix games (Tak 2014) and/or LP solutions for Nash equilibria with learned valuation functions”, but despite the solid foundational work on the client and engine, the project appears to have been abandoned shortly after the initial skeletons of an agent were implemented.

A competitive Pokémon battling AI that existed to serve as the in-game computer opponent for a ROM-hack when run in a custom emulator, Deep Red is a typical rules-based agent with a hand-crafted evaluation function. Due to the unique platform it was built for it could leverage perfect information to simplify its implementation, though targeting an in-game battle system also meant it had to support the additional action of using items in battle, a game element not present in the multiplayer link battle system online play is based around. Deep Red’s algorithm was mostly deterministic, primarily leveraging a damage calculator that computed expected damage, though it also featured a notion of exploring “combos” computed via limited look-ahead and contained basic implementations of a killer move heuristic and its inverse, changing behavior of the AI when its own HP was low.

An early rule-based agent that variably focused on the then-current generation of random random formats (Generation VI–VIII). Synedh/showdown-battle-bot featured limited state tracking, augmenting Pokémon Showdown’s request JSON with simple battle log parsing to form a more complete client representation. The agent’s evaluation scored actions in terms of speed and expected damage computed by a basic damage calculator, with special handling for boosting moves and moves which inflict status. The AI preferred attacking but would switch if it lacked sufficiently good options for moves.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque ultrices, tortor sed iaculismollis, odio ex porta ante, ac malesuada est elit eu elit. Etiam dapibus quam id lorem condimentummollis. Donec non dolor ipsum. Praesent vel ultrices ex. Donec nibh dui, hendrerit a erat et,tincidunt semper orci. Cras in lectus sapien. Proin vehicula eros varius, eleifend neque laoreet,iaculis leo. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse in quamcondimentum, eleifend felis vitae, congue lectus. Maecenas arcu urna, dictum et auctor quis,vestibulum sit amet odio. Aliquam felis neque, fringilla et porttitor a, posuere at magna.

CynthiAI targeted the Generation VII Random Battle format using a mixture of heuristics and depth 1 pure optimistic minimax search with a hand-crafted evaluation function. CynthiAI deterministically used the highest accuracy move that would knock out its opponent (killer move heuristic) otherwise selected the most damaging move (ignoring accuracy), finally falling back on the result of its minimax search. Like Percymon, to better account for game mechanics and prevent suboptimal decisions, CynthiAI applied further hardcoded modifiers to the minimax result. In lieu of team prediction via usage statistics, CynthiAI worked off of an opponent’s known moves if enough were known otherwise defaulted to random move pool for the species.

SutadasutoIA was a knowledge-based expert system designed to play a variant of Generation VI BSS without status moves, EVs, or items. The agent was intended for repeated battles against the same opponent akin to Battle Maison and used a hill-climbing algorithm to optimize team selection from a pool of Pokémon between battles. Like vasumv/pokemon_ai, SutadasutoIA used a webdriver (Selenium) for play on Pokémon Showdown as opposed to communicating directly via the WebSocket API. SutadasutoIA’s hand-crafted evaluation function used a combination of rules and a scoring system to determine actions, focusing on classifying Pokémon by their role (“Special Sweeper”) and choosing suitable actions based on this designation.

Rill-García followed up their work on SutadasutoIA with an agent developed via reinforcement learning. Designed for the same constrained Generation VI BSS format and repeated battle challenge as SutadasutoIA, the RL agent used data from 500 high-ranked human replays gathered from Pokémon Showdown which were then also used to bootstrap the agent’s learned evaluation function. The bulk of training was conducted via a Q-learning approach with self-play and a threshold-greedy policy. Actions were rewarded based on whether they resulted in an OHKO, KO, the percentage damage they caused their opponent or whether they prevented the opponent from acting. During evaluation matches with SutadasutoIA where agents were allowed to learn and adapt to their opponent’s team between battles, the RL agent did well when using a single team but did poorly when forced to play with several teams.

DanielAlcocerSoto/Pokemon-Python experimented with a reinforcement learning model written with Keras and Tensorflow as the opponent trainer featured in a custom engine and accompanying pygame-based UI supporting Generation IV Double Battles. While this domain is quite ambitious, the engine was extremely simple; lacking support for switching, effects, or move accuracy. A Q-learning approach was used to train the agent’s small neural network on a fairly minimal embedding against a random agent following ε-greedy policy using mini-batch gradient descent and a reward function that favored maximum damage. The network’s architecture was a straightforward multi-layer perceptron with layers of 24, 24, and 8 units with ReLU activation between layers and a linear output.

The PokeML project contained a collection of several Generation VII Pokémon battle agents that simulated battles via Pokémon Showdown and whose client representations tracked state using an extraction from its client. PokeML’s agents included:

• A basic $RandomPlayer$ (RandomAgent) and a $RandomPlayer(switch=\widetilde{0\%}, mega=100\%)$ implementation (SemiRandomAgent).
• The classic $MaxDamagePlayer$ (MaxDamageAgent) and a more simplified version that only considered a move’s base power (MaxBasePowerAgent).
• SimAgent, an agent that performed a one-turn look-ahead by simulating each possible joint action, evaluating the new state based purely on the sum of HP percentages of Pokémon on each side, using lrsnash to solve the computed payoff matrix and making a decision based on a weighted random sample from this solution.
• ChecksSwitchAgent which chose actions based on hard-coded checks and counters table and the typechart, switching to whichever Pokémon on the team has the best matchup versus the current opponent Pokémon (or attacking randomly if already out).
• TestAgent, a work-in-progress agent utilizing machine learning which targeted a reduced conmeta to make encoding the game states into fixed-size numeric vectors more tractable.

Ihara et al. compared information set MCTS (ISMCTS) to determinized conventional MCTS in the Generation VI Pokémon BSS format. Using a custom simulator and a fixed pool of six Pokémon combined with a client representation that leveraged usage statistics from Pokémon Global Link and heuristics fill in unknown information for an opponent’s team, Ihara et al. found that ISMCTS was superior to determinized MCTS when the number of iterations was fixed. Both approaches were significantly weaker than agents leveraging MCTS with perfect information which were used as a control, though the gap decreased in further experiments where the battle rules were simplified.

Supporting perfect information variants of assorted Generation VII singles and doubles formats, samhippie/shallow-red experimented with Decoupled UCT (DUCT), Monte Carlo tree search with Exp3, Monte Carlo Regret Matching, Online Outcome Sampling (OOS), and MC-CFR before settling on an approach using Deep MC-CFR. Built with PyTorch, the agent contained no client respresentation, working instead off of the tokenized protocol augmented with information extracted from Pokémon Showdown’s data files and leveraging Pokémon Showdown’s CLI for simultations. The model featured a LSTM network, used a hash embedding, and could be configured to enable convolutions and attention.

Showdown AI Bot targeted random battle formats from Generation I-VII and featured policies for varying difficulty levels. Backed by a custom engine with a basic damage calculator, the engine’s scraped data was inserted into a MySQL database that was then queried at runtime and included information about random movepools. The bot’s client representation featured request parsing and differentiated between an opponent’s possible and actual moves. The bot was mostly rules-based but featured different heuristics for each difficulty:

• Easy: a $MaxDamagePlayer$ that will only switch when forced, but when switching considers the species type matchup of prospective switches.
• Normal: still focused on maximizing damage, but move decisions also relied on hardcoded logic for status and more nuanced game mechanics as well as limited look-ahead to avoid being knocked out by an opponent, possibly switching to preserve a Pokémon. Switching decisions considered both species and move type matchups.
• Hard: depended on the same logic as “Normal” for switching but move decisions were based on a max depth 2 iterative deepening pure pessimistic minimax search with a complicated hand-crafted evaluation function.

0 ERROR exists as the theoretical motivating use case behind the pkmn project. While the actual agent is currently vaporware, pkmn has been building foundational Pokémon infrastructure that has been used within multiple other Pokémon AI projects:

• pkmn/ps: Pokémon Showdown’s simulator and client broken up into composable libraries.
• data.pkmn.cc: An aggregated source exposing all of Smogon University’s data in machine-readable form.
• pkmn/stats: A framework for efficient Pokémon Showdown battle log processing and usage stats generation.
• pkmn/engine: A minimal, complete, battle simulation engine optimized for performance.
• EPOKé: An enhanced client library which tracks not only the observed state of the battle, but also uses the mechanics of the game, reverse damage calculation, and usage statistics to infer as much as possible about the state of play.

Norström compared the $RandomPlayer$, learned linear combinations of features, and Monte Carlo tree search across various scenarios in Generation I including a limited version of the OU format implemented with a custom game engine. While MCTS proved to be the strongest in relative terms across the scenarios tested, strength in absolute terms was unimpressive. Combined approaches and increased engine and training performance were cited as the most promising avenues for improvement for future work.

The premier Python reinforcement learning framework for competitive Pokémon AIs, poke-env supports both singles & doubles battles and random & non-random Generation IV-IX formats. Documentation and examples help make it easy for beginners to get started, though poke-env’s fully-featured client representation makes it valuable to any Python project.

poke-env doesn’t embed Pokémon Showdown so can’t easily perform simulations (or leverage the simulator for damage calculations), but handles connecting to a local Pokémon Showdown instance and includes data extracted from the simulator as well as team parsing and packing logic to provide the foundation most agents need.

Beyond this, poke-env also offers support for training agents concurrently and utilities for evaluating agent performance (e.g., cross evaluation). While self-play support is still experimental, poke-env provides a MaxBasePowerPlayer and SimpleHeuristicsPlayer (both simplified versions of an actual $MaxDamagePlayer$) for training. Finally, poke-env contains an OpenAI Gym environment implementation and helpers including one for rewards.

True to its name, the Simplified Pokemon Environment proposed an incredibly restricted toy OpenAI Gym environment that reduces competitive Pokémon to simply a question of type matchups and damaging moves as a baseline for evaluating Pokémon agents. In addition to the environment, the project evaluates the performance of two deep learning approaches both which use ε-greedy exploration policies - Generalized Infinitesimal Gradient Ascent using the Win or Lose Fast principle (GIGA-WoLF) adapted to deep learning as GIGAθ and a asynchonous deep version of a Weighted Policy Learner (WPLθ). Both approaches converged to successful strategies, though in the limited environment GIGAθ converges faster and ended up outperforming WPLθ.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque ultrices, tortor sed iaculismollis, odio ex porta ante, ac malesuada est elit eu elit. Etiam dapibus quam id lorem condimentummollis. Donec non dolor ipsum. Praesent vel ultrices ex. Donec nibh dui, hendrerit a erat et,tincidunt semper orci. Cras in lectus sapien. Proin vehicula eros varius, eleifend neque laoreet,iaculis leo.

Pokemon Battle Predictor was a browser extension that surfaced predictions at the start of every turn about the current battle state to Pokémon Showdown players inferred from a collection of models built with supervised learning techniques and battle logs data. These models (which would eventually be used to power initial versions of the Future Sight AI) included a value network which would predict winning percentage as well as a decomposed policy network made up of three components: a network that predicted the chance of a player choosing to switch or move and then separate networks that would determine which move the player might choose or Pokémon the player might switch to if they chose to do so.

Initially trained on approximately 10,000 Generation VIII OU Pokémon Showdown replay logs scraped from players with an Elo rating of at least 1250, the models’ accuracies were greatly improved after Pokémon Showdown staff provided the project with a corpus of around 2 million server logs. This additional data also allowed for the creation of models for the majority of non-random Smogon University singles formats. The project originally performed model inference client-side with TensorFlow.js, though later on anonymized data from extension users was sent to a server and inference was moved server-side in order to support larger models. These server-side models could then be tuned with regularly scraped replay data to respond to shifts in the various supported metagames.

The original neural network model implementation involved an input vector of 6815 features extracted from the battle state, with seemingly important features like abilities, items, or types being absent, justified by the observation that these could instead be learned as innate characteristics summarized purely by a Pokémon’s species. A unique feature included was a “Switch Coefficient” that aimed to capture how often one Pokémon species switches out when faced off against an opposing Pokémon. These inputs were used to train the value network and high-level policy network while the finer-grained networks relied on a slightly different approach. The switch choice model involved a layer to detect which Pokémon would switch in as well as a layer that uses that output to learn which Pokémon were commonly brought in under similar situations. The move choice model was trained to output a probability distribution across all moves which then got masked off by observed probabilities of each Pokémon having said move derived from moveset usage statistics. Updated versions of the model were also trained to predict based on a player’s Elo rating, allowing it to account for players of different skill levels exhibiting different behaviors.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque ultrices, tortor sed iaculismollis, odio ex porta ante, ac malesuada est elit eu elit. Etiam dapibus quam id lorem condimentummollis. Donec non dolor ipsum. Praesent vel ultrices ex. Donec nibh dui, hendrerit a erat et,tincidunt semper orci. Cras in lectus sapien. Proin vehicula eros varius, eleifend neque laoreet,iaculis leo. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse in quamcondimentum, eleifend felis vitae, congue lectus. Maecenas arcu urna, dictum et auctor quis,vestibulum sit amet odio. Aliquam felis neque, fringilla et porttitor a, posuere at magna. Morbisodales maximus sapien commodo venenatis.In nunc felis, porta non diam in, faucibus condimentum nulla. Quisque eu vulputate odio. Sedpharetra massa et venenatis dignissim. Vivamus aliquet consequat nibh, non tristique liberoullamcorper eu. Fusce eget suscipit metus. Quisque volutpat ultricies erat sit amet dignissim.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque ultrices, tortor sed iaculismollis, odio ex porta ante, ac malesuada est elit eu elit. Etiam dapibus quam id lorem condimentummollis. Donec non dolor ipsum. Praesent vel ultrices ex. Donec nibh dui, hendrerit a erat et,tincidunt semper orci. Cras in lectus sapien. Proin vehicula eros varius, eleifend neque laoreet,iaculis leo. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse in quamcondimentum, eleifend felis vitae, congue lectus.

Youngster Joey targeted a perfect information variant of Generation I with a greedy policy that determines its action by maximizing a set of hand-crafted heuristics. A slightly more sophisticated $MaxDamagePlayer$, Youngster Joey deliberately eschewed minimax and expectiminimax due to concerns over the game tree size even after pruning and the algorithms not being able to play optimally against opponents who were also expected to play sub-optimally. The addition of some form of look-ahead was considered for future work.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque ultrices, tortor sed iaculismollis, odio ex porta ante, ac malesuada est elit eu elit. Etiam dapibus quam id lorem condimentummollis. Donec non dolor ipsum. Praesent vel ultrices ex. Donec nibh dui, hendrerit a erat et,tincidunt semper orci. Cras in lectus sapien. Proin vehicula eros varius, eleifend neque laoreet,iaculis leo. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse in quamcondimentum, eleifend felis vitae, congue lectus. Maecenas arcu urna, dictum et auctor quis,vestibulum sit amet odio. Aliquam felis neque, fringilla et porttitor a, posuere at magna. Morbisodales maximus sapien commodo venenatis.In nunc felis, porta non diam in, faucibus condimentum nulla. Quisque eu vulputate odio. Sedpharetra massa et venenatis dignissim. Vivamus aliquet consequat nibh, non tristique liberoullamcorper eu. Fusce eget suscipit metus. Quisque volutpat ultricies erat sit amet dignissim.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque ultrices, tortor sed iaculismollis, odio ex porta ante, ac malesuada est elit eu elit. Etiam dapibus quam id lorem condimentummollis. Donec non dolor ipsum. Praesent vel ultrices ex. Donec nibh dui, hendrerit a erat et,tincidunt semper orci. Cras in lectus sapien. Proin vehicula eros varius, eleifend neque laoreet,iaculis leo. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Pokemon Trainer AI used a bespoke Generation I engine to evaluate several different agents in a level 50 3v3 format:

• RBES, a rules-based expert system that decided on actions based on its hand-crafted evaluation function that primarily focused on maximizing damage.
• DT, a decision tree based approach produced for a specific team configuration.
• RL, a state-based Q-learning model also trained for a specific team configuration.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque ultrices, tortor sed iaculismollis, odio ex porta ante, ac malesuada est elit eu elit. Etiam dapibus quam id lorem condimentummollis. Donec non dolor ipsum. Praesent vel ultrices ex. Donec nibh dui, hendrerit a erat et,tincidunt semper orci. Cras in lectus sapien. Proin vehicula eros varius, eleifend neque laoreet,iaculis leo. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse in quamcondimentum, eleifend felis vitae, congue lectus. Maecenas arcu urna, dictum et auctor quis,vestibulum sit amet odio. Aliquam felis neque, fringilla et porttitor a, posuere at magna. Morbisodales maximus sapien commodo venenatis.

The Pokémon Simulator Environment is an environment for running a large number of Pokémon Showdown battle simulations. While not intended as a serious attempt at a competitive Pokémon battling agent, the project features an elaborate rules-based heuristic agent that can serve as a training baseline for other agents in singles formats. Supporting all generations, the HeuristicPlayerAI’s extends past the $MaxDamagePlayer$, considering elements beyond just the potential damage its moves may do. The agent evaluates matchups based on type advantage, speed, and remaining HP percentage, considers switching out, and contains handwritten cases to handle specific game mechanics or to be able to use moves which indirectly benefit the player.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque ultrices, tortor sed iaculismollis, odio ex porta ante, ac malesuada est elit eu elit. Etiam dapibus quam id lorem condimentummollis. Donec non dolor ipsum. Praesent vel ultrices ex. Donec nibh dui, hendrerit a erat et,tincidunt semper orci. Cras in lectus sapien. Proin vehicula eros varius, eleifend neque laoreet,iaculis leo. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse in quamcondimentum, eleifend felis vitae, congue lectus. Maecenas arcu urna, dictum et auctor quis,vestibulum sit amet odio. Aliquam felis neque, fringilla et porttitor a, posuere at magna. Morbisodales maximus sapien commodo venenatis.In nunc felis, porta non diam in, faucibus condimentum nulla. Quisque eu vulputate odio. Sedpharetra massa et venenatis dignissim. Vivamus aliquet consequat nibh, non tristique liberoullamcorper eu.

PokéLLMon explores using large language models (LLMs) to make decisions in the Generation VIII Random Battle format. The agent translates its interpretation of the battle state from poke-env into a text description that is then provided as a prompt to an LLM to generate the next action. A small amount of heuristics are involved: the decision to Dynamax depends on hardcoded rules and if the LLM is unable to produce an action the agent falls back on the policy of choosing the move with the highest base power. PokéLLMon leverages In-Context Reinforcement Learning (ICRL) by including information about historical turns and feedback on changes in HP, effectiveness of attacking moves, priority of move execution, and the effects of executed moves to allow the LLM learn from its past actions. The expected type effectiveness of moves and possible effects of its moves are also provided to the LLM for the current turn as a form of Knowledge-Augmented Generation (KAG). One-shot, Self-consistency (SC), Chain-of-thought (COT), Tree-of-thought (TOT) prompting approaches are all evaluated, with self-consistency proving to be the most effective. Overall the agent struggles with long-term planning, acting mostly greedily and performing poorly against stall strategies.