Human-object interaction prediction in videos through gaze following

Understanding the human–object interactions (HOIs) from a video is essential to fully comprehend a visual scene. The proposed approach leverages human gaze information, scene contexts, and visual appearances of human-object pairs through a spatio-temporal transformer to detect and anticipate HOIs in videos. The model outperforms state-of-the-art in HOI detection in the largest video HOI dataset, VidHOI, by large margins.

Ni, Zhifan., Esteve Valls Mascaró., Hyemin Ahn, & Dongheui Lee.  Human-object interaction prediction in videos through gaze following, Computer Vision and Image Understanding,  2023. Webpage, opens an external URL in a new window

HOI prediction Paper

© Esteve Valls Mascaro

HOI prediction Paper

HOI prediction Paper

Long-Horizon Planning and Execution with Functional Object-Oriented Networks

Following work on joint object-action representations, functional object-oriented networks, opens an external URL in a new window (FOON) were introduced as a knowledge graph representation for robots. A FOON contains symbolic concepts useful to a robot's understanding of tasks and its environment for object-level planning, opens an external URL in a new window. We thereby introduce the idea of exploiting object-level knowledge as a FOON for task planning and execution.

David Paulius, Alejandro Agostini, Dongheui Lee, Long-Horizon Planning and Execution with Functional Object-Oriented Networks, IEEE Robotics and Automation Letters (RA-L), 2023. DOI: 10.1109/LRA.2023.3285510 . Webpage, opens an external URL in a new window

[Translate to English:] Long-Horizon Planning and Execution with Functional Object-Oriented Networks

© Dongheui Lee

[Translate to English:] PDDL

[Translate to English:] Long-Horizon Planning and Execution with Functional Object-Oriented Networks

Vision-based Approximate Estimation of Muscle Activation Patterns for Tele-Impedance

Our goal is to transfer the human ability of end-point stiffness adaptation to the robot solely based on RGB images. For that, we introduce a model named muscle activation pattern estimator (MAPE). MAPE can infer:

  1. Whether the muscle is activated or inactivated (static states).
  2. Whether the degree of muscle activation is increasing, stable, or decreasing (temporal states).

Hyemin Ahn, Youssef Michel, Thomas Eiband and Dongheui Lee, Vision-based Approximate Estimation of Muscle Activation Patterns for Tele-Impedance, IEEE Robotics and Automation Letters (RA-L), pp. - , 2023

 

Vision-based Approximate Estimation of Muscle Activation Patterns for Tele-Impedance

© Hyemin Ahn

Vision-based Approximate Estimation of Muscle Activation Patterns for Tele-Impedance

Vision-based Approximate Estimation of Muscle Activation Patterns for Tele-Impedance

Coordination tending towards an anti-phase relationship determines greater sway reduction during entrainment with a simulated partner

To develop effective balance support robotic devices for older adults at risk of falls, understanding the simultaneous occurrence of entrainment and sway reduction in human-human interaction is crucial. This study investigates the effects of different simulated sway-responsive interaction partners on sway entrainment, sway reduction, relative interpersonal coordination, and individual body schema accuracy. Our findings reveal that certain interaction modes, such as the Repulsor, lead to reduced body sway and improved interpersonal coordination, emphasizing the significance of accurate body schema and anti-phase relationships in facilitating sway reduction.

Youssef Michel, Katrin Schulleri, Leif Johannsen, and Dongheui Lee, Coordination tending towards an anti-phase relationship determines greater sway reduction during entrainment with a simulated partner, Human Movement Science, vol. 89, 103090, 2023. doi: 10.1016/j.humov.2023.103090

 

Coordination tending towards an anti-phase relationship determines greater sway reduction during entrainment with a simulated partner

© Youssef Michel, Katrin Schulleri

Coordination tending towards an anti-phase relationship determines greater sway reduction during entrainment with a simulated partner

Coordination tending towards an anti-phase relationship determines greater sway reduction during entrainment with a simulated partner

Estimation of 6D pose of objects based on a variant adversarial autoencoder

This paper proposes a method for estimating the 6D pose of objects using a texture-less dataset, where the pose is obtained by rendering the 3D model and the orientation is represented by a latent space from RGB images, utilizing a variant Adversarial Autoencoder network. Experimental results demonstrate that the proposed method achieves better accuracy in object pose estimation compared to a state-of-the-art method.

Huang Dan, Hyemin Ahn, Shile Li, Yueming Hu, and Dongheui Lee, Estimation of 6D pose of objects based on a variant adversarial autoencoder, Neural Processing Letters, 2023. DOI: 10.1007/s11063-023-11215-2

 

[Translate to English:] Vision-based Approximate Estimation of Muscle Activation Patterns for Tele-Impedance

© Dongheui Lee

[Translate to English:] Vision-based Approximate Estimation of Muscle Activation Patterns for Tele-Impedance