UnrealAction Dataset

UnrealAction is a dataset that contains the videos generated from embodied agents. Since the generated videos looks largely different from the realworld ones, we term this dataset as "UnrealAction". As shown in the above figure, UnrealAction has 14 actions classes, and each action class has 100 videos from the virtual domainand 10 videos from the realworld domain.

1. Virtual: actions of the agent captured by virtual cameras in Unreal Engine 4.
2. Realworld: videos from Youtube and published datasets: UCF101, and Kinetics.

Action Classes: jazz dancing, samba dancing, break dancing, house dancing, ymca dancing, capoeira, chicken dance, bow, clapping, holding a baby, saluting, show bicep, throwing, waving hand.

waving hand	throwing

saluting	break dancing

>> How we create this dataset?

We take advantage of Unreal Engine 4, a popular game engine, to build our simulator. Unreal Engine 4 provides Blueprint, a visual script, to control the virtual world. As shown in the figure, our simulator is mainly composed of several agents and virtual environments. Generally, we first let the agentes to perform actions in the virtual enviroments, and then record the actions of the agent by virtual cameras. We use BluePrint to define the actions of the agent, the motion of the virtual camera, and the reaction of the environment. Specifically,

• Enviroments: We take several game maps as our environment. The game maps include indoor scenes, urban scenes, and natural scenes.
• Agents: We collect tens of character models as alternative appearances of our agent. The agents are in different clothes, hairstyles, genders, and races. They also have skeletons, which enable the characters to perform skeletal animations.
• Actions: We collect 14 action classes of animations, all of the actions are atom i.e, perform one well-defined action. We make these animations compatible with all of our agent skeletons. Therefore, all the characters can perform all the actions.

>> Dataset Folders

Once you download our dataset from Google Drive or 百度云盘, you will see these sub folders:

• DatasetInfo: It contains the "ClassInd.txt", "RealVideoList.text", and "VirtualVideoList.txt".
• RealVideos: It contains 14 sub folders, each is named by the action class with 10 videos.
• VirtualVideos: It contains 14 sub folders, each is named by the action class with 100 videos.

The UnrealAction dataset is characterized by its virtual action videos generated by embodied agents, as well as the unique design of atomic actions within the videos. Thus, we suppose it could be useful as a testbed for:

• cross-domain related video tasks;
• fine-grained ralted video tasks;
• ...