For example, to distinguish between opening and closing a box, reversing the order will give opposite results, so temporal modeling is critical.

One key difference between video recognition and image recognition is the need for temporal modeling. Hours of videos uploaded to YouTube every day to be processed for recommendation and ads ranking tera-bytes of sensitive videos in hospitals need to be processed locally on edge devices to protect privacy.Īll these industry applications require both accurate and efficient video understanding.ĭeep learning has become the standard for video understanding over the years. Hardware-efficient video understanding is an important step towards real-world deployment, both on the cloud and on the edge. Spatial-temporal action detector emerges during the training of classification Lastly, TSM enables action concepts learning, whichĢD networks cannot model we visualize the category attention map and find that Scalability compared to 3D networks, enabling large-scale Kinetics training onġ,536 GPUs in 15 minutes. Recognition on Jetson Nano and Galaxy Note8. Secondly, TSM has highĮfficiency it achieves a high frame rate of 74fps and 29fps for online video Something-Something leaderboard upon submission. Firstly, TSM has high performance it ranks the first on the Temporal modeling at zero computation and zero parameters. It can be inserted into 2D CNNs to achieve The channels along the temporal dimension, thus facilitate informationĮxchanged among neighboring frames. Propose a generic and effective Temporal Shift Module (TSM) that enjoys both Conventional 2D CNNs are computationallyĬheap but cannot capture temporal relationships 3D CNN-based methods canĪchieve good performance but are computationally intensive. The explosive growth in video streaming requires video understanding at highĪccuracy and low computation cost.

Speed up tsm 4