Character-like Level Generation for Science Birds

Evaluation

The submitted prompts will undergo an evaluation process that involves subjecting them to 10 trials for each of the 26 uppercase letters of the English alphabet (A-Z). The levels generated for each character will be evaluated based on their similarity, stability, and diversity, and scored using the criteria outlined in the scoring policy given below. The entire evaluation process will be conducted using automated scripts and programs.

Please note that the evaluation process will be conducted twice, at midterm and final stages. The number of trials and characters in the evaluation set may be adjusted based on the number of teams.

Evaluation Set

All 26 alphabetical uppercase characters.

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Scoring Policy

In trial $i$ of target character $j$ for program $k$:

1. Stability: The stability of a level is evaluated using our Science Birds Evaluator. Here, $total\_blocks_{ijk}$ is defined as the number of blocks at the initialization step of loading the level into the evaluator. Then the program will calculate the value of $moving\_blocks_{ijk}$which is defined as the number of moving blocks during the first 10 seconds after level initialization. Each level will receive an $sta_{ijk}$ score according to the following equation. The score has a continuous value in $[0, 1]$.
$$sta_{ijk} = \frac{total\_blocks_{ijk} - moving\_blocks_{ijk}}{total\_blocks_{ijk}}$$
2. Similarity: The similarity score reflects the softmax probability, $\sigma (\textbf{z}_{ijk})_j$ of the model called vit-base-uppercase-english-characters, which is used to infer target character $j$ in trial $i$ from the image of the generated level after the first 10 seconds of initialization. Each level will receive a continuous value between $[0, 1]$. This score, $sim_{ijk}$, is given as
$$sim_{ijk} = \sigma (\textbf{z}_{ijk})_j$$
3. Diversity: The diversity score represents how diverse the generated levels are under the same target character $j$ for program $k$. The score, $div_{jk}$, is calculated by computing cosine distance, $D_c$, of unordered pairs in the set $A$ containing pairs of output vectors from the softmax probability, $\textbf{v}_{ijk} = \sigma(\textbf{z}_{ijk})$. $\Xi_{jk}$ denotes a set containing all such vectors across trials of the same target character $j$ from the same program $k$.
$$div_{jk} = \frac{\sum_a^{|A_{jk}|} D_c(\textbf{v}_a^1, \textbf{v}_a^2)}{0.5T(T+1)-T}\text{,}$$

where

$$A_{jk} = \{(\textbf{v}_a^1, \textbf{v}_a^2) | (\textbf{v}_a^1, \textbf{v}_a^2) \in \Xi_{jk} \bowtie \Xi_{jk} \land v_a^1 \neq v_a^2 \}$$

and $T$ represents the number of trials per target character.

Given that $P$ and $C$ represent the number of programs in the competition and the number of characters, respectively, to give a higher weight to a more difficult target character, $weight_{j}$ for target character $j$ is defined as follows:

where

and

Next, the weighted $trial_{ijk}$ score is defined as follows:

The average score for target character $j$ of program $k$, $char_{jk}$, is defined as follows:

The $prompt_{k}$ score is defined as follows:

Next, the normalized $prompt_{k}$ score, $norm\_prompt_{k}$, is defined as follows:

where

Finally, $norm\_prompt_{k}$ will be used for ranking.

Ranking Policy

The team that has the highest $norm\_prompt_{k}$ will be declared the winner. If there are multiple teams with the same highest score, the one with the shortest prompt will be chosen as the winner. However, if multiple teams still have the same score and the shortest prompt, they will be considered co-winners.

Evaluation Tools

1. ChatGPT API via openai Python package using gpt-3.5-turbo-0125 model.
2. Science Birds Evaluator with features to
   1. Automatically assess the stability
   2. Automatically export images using black-textured blocks on a white background for similarity testing
3. vit-base-uppercase-english-characters
4. Automation scripts available on our Resources page.

Evaluation Environment for Response Generation Stage

Software

• OS: Ubuntu 22.04.3 LTS
• Python: 3.11.xx
• Node.js: 20.xx.xx LTS
• Unity: 2019.4.40f1
• Science Birds Evaluator
• Our automation scripts

Hardware

• CPU: Intel(R) Xeon(R) Gold CPU 6430x128
• RAM: 250 GB
• GPU: NVIDIA L40S 48GB

Evaluation Process

1. We manually inspect each submitted program for a potential violation of the rules.
2. Qualified programs will be run to gather responses from ChatGPT.
3. The code extraction script will load each response and produce a new file containing only a series of drop_block() commands.
4. The text-to-xml conversion script will load each code file and convert it into a Science Birds level description XML file.
5. Next, Science Birds Evaluator will individually load all levels to assess their stability and capture their images. The results of stability will be recorded, and for each level an image of the structure with black-textured blocks on a white background will be produced by the program.
6. The similarity checking script will load each image and pass it through an open source-model called vit-base-uppercase-english-characters. It will then record the similarity result.
7. The diversity checking script assesses the diversity of the levels by averaging the cosine distance of non-duplicated all-pairs of outputs from softmax for each level across trial of the same target character as described in the Evaluation.
8. Finally, the scoring and ranking script will load all stability, similarity, and diversity results and produce the final rank and score result for all teams according to the scoring policy.