A first preliminary version of my paper can be found here. You might notice that the most important part seems missing, namely the content about act-r and soar. This is because I am working on those right now and I am in the process of reworking them completely. Regular updates will follow in the coming days. The intent of this document is to give people an impression of the content present in the paper.

Parameters and methodology for evaluating the performance of a Soar-powered computer controlled character.

Using soar and Quake II

As I have stated before, there have been studies in which SOAR is used to model behavior of computer controlled characters (NPC's). This was done using the game Quake II. In the multiplayer component of this game, players compete against each other in a so called deathmatch. The goal is simple: players have to kill opponents from the opposing team. This is done on a map consisting of different rooms, hallways, doors, ... On different locations power-ups can be found such as new weapons, health, shields and the like. The game allows for players to play agains bots. These have the same function as human players, but are controlled by the computer. Ideally these behave as human-like as possible, but are adaptable in regard to difficulty so that more experienced players get a decent challenge as well.

It has been previously shown that SOAR can be used to model the behavior of such a bot. A single entity is controlled by rules defined in SOAR. This bot has then been used to evaluate different parameters concerning it's behavior. One such parameter is playability. Is this game possible to win for a human? Is it not too hard or too easy? Is it rewarding? Does it pose a challenge? In the case of the Quake 2 bot it is important that it behaves in a human like way, that it does not act stupid (walking into walls for example), but that it does not have obvious super-human abilities (shots always hit). The good thing is that it is possible to parameterize this, so it is possible to evaluate what parameters have an impact on this.

Different evaluation criteria:

The goal was to evaluate the humanness of the bot using some basic parameters:

1. Decision time: how fast can a bot make decisions?
2. Aggressiveness: how likely is it that the bot will directly attack a player?
3. Complexity of tactics: different tactics, corresponding to different skill levels
4. Level of expertise: basically, how good can the bot shoot?

Methodology: Judging the bot

Judging the bots was done by taping the matches between humans and the bot and then showing those tapes to a jury that had different levels of experience with quake. They had to answer a simple question: is this character human or not? Different combinations of parameters were used. In the end it was shown that not all of the evaluation criteria had an effect on the perception of the bot.

Concerning my own research

Once I have a system up and running for creating AI's using rule-based models, I think it is a good idea to test at the very least the parameters described here. They are logical parameters, correspond to the most basic abilities a decent AI should have and are easy to test. On top of that the testing methodology is clearly described and is not hard to reproduce.
Source:Creating Human-like Synthetic Characters with Multiple Skill Levels : A Case Study using the Soar Quakebot

I discovered that both Act-R and SOAR can be used from python. With Act-R this is easy, there is a python implementation (more information can be found in the Links and Literature section) available. SOAR is a different story. It is a bit harder to use since there is no python implementation available. There is a java version available though. This implementation allows a user to use different kinds of languages from inside the Java code. Another possibility is using wrappers to wrap the c-headers and use those.