Our method was alike, when I draw an person his/her actions comes along. This will affect my response to the real action I want to combine. For example my 45 21 Einstein is drawing a big 
on the blackboard.
And I had overcame your problem by granting characteristics to the person, it means despite they are doing the same action, but they have their styles.
Please take a look into Mnemonic Architecture - #7 by Antelex
deep learning in PA0-24