In addition to performing detailed analysis with test frames taken from the same video as the training data, we also sought to assess performance across 10 more challenging “situational” test cases. A brief explanation of each test case is given below. Note: all of the cases with the exception of the domain shift were captured in the ACL Motion Capture Studio.

• • Drone Decoy: Our dataset only includes a puck class and, furthermore, only imagery of pucks close up without any other puck-sized objects in the immediate vicinity representing true-negative cases. To determine how resilient our models were at rejecting false-positives, we collected imagery when a drone was also in frame and near the pucks. This is an especially tricky case for the network, as both the puck and drone share some of the same circuit boards, so there are some visually-similar features between the drone and the puck.
  • Crashed Puck: All of the pucks imaged in our dataset were fully assembled, however we may want to identify pucks which are broken and may not visually resemble the original puck. To this end, we used a puck which was broken into several pieces to see how our models performed when the puck was in a broken state rather than fully assembled.
  • Tumbled Pucks: All of the pucks imaged in our dataset were upright in their operating state, and so we wanted to determine how our models performed when the pucks had fallen over. Similar to the above test, we collected imagery of our pucks in a tumbled state. 
  • Overhead: Since our dataset was collected from an egocentric viewpoint (i.e. from the ground), we sought to find how robust our models are to a change in viewpoint, so we collected imagery from an overhead viewpoint.
  • Puck Swarm: We only included, at a max, four pucks in one image simultaneously in our dataset. To this end, we wanted to determine if the model performance was affected when significantly (>30) more pucks were in frame. 
  • Solid Occlusion: In our dataset, there were only cases of occlusion when one puck occluded another or one puck was partially out of the frame. We wanted to see how our model performance was affected when solid obstacles were introduced into the environment to provide occluded imagery.
  • Diffuse Occlusion: We also wanted to see how the model performance was affected when the pucks were occluded by a net to represent a diffuse occlusion setting. A practical setting where a case like this might be encountered in looking through vegetation or branch-like objects at another puck.
  • Low light: Tested model performance when lighting within the studio was partially-diminished. 
  • Very low light: Tested model performance when lighting within the studio was fully-diminished (except for the blue light emitted by the motion capture cameras). 
  • Domain Shift: Since our dataset was collected entirely in the ACL Motion Capture Studio, we wanted to determine how the performance of the three models would be affected by a domain shift. To this end, we collected imagery from a puck on grass outdoors.

Using aforementioned post-processing tools, all 10 test-cases are demoed below.