There is some debate regarding what is the function of human gesticulation. Some believe it’s a tool we use to communicate with others [1], and some that it’s part of the language [2]. Regardless of the function behind gesturing, I decided to take on a study about gesticulation to explore and delimit what a gesture is, possible categories and to find some gesticulation patterns. Therefore, I conducted a small experiment that allowed me to delve into this exploration of human gesticulation.

Experiment Design

I devised an experiment in which three participants watched a short animation episode and, afterwards, they were interviewed with questions about said episode while being recorded. The questions were divided into summary, descriptive, detail and movement questions. In addition, the recorded interview was processed by a video analysis program that also provided some movement data. This, together with the comparison of the interview answers provided enough material for an in-depth gesticulation analysis.

Types of questions and gestures

I categorized the interview questions as well as the gestures to facilitate the discussion of the results later.

Interview questions

  • Summary question: asks the participants to narrate something in general terms, hoping for a summary of events. There was one summary question in the interview that asked participants to narrate everything they remembered from what they had watched.
  • Descriptive question: asks for a scenery description. There was one descriptive question included asking to describe the inside of the pyramid that appears in the episode.
  • Detail questions: ask for a specific detail, usually a visual element. Eight of them were included.
  • Movement questions: ask about a scene that contains movement (objects/characters moving, other actions that involve movement). Six of them were included. Detail and movement questions were included in the interview together as a set of short questions.

Gestures

I devised the categories for the gestures according to the motion included in the content of the participants’ speech.

  • Detail gestures: those used to depict details or still visual elements as can be shapes, sizes, locations etc.
  • Movement gestures: those used to depict moving bodies or actions that involve movement.

Motion Capture Tools

For this experiment, I made a recording of each participant’s interview and processed the videos through a program that analyzes the movement frame per frame called Video Analysis. From this program, csv files are exported containing data on timestamps, centroid of motion, quantity of motion and other motion-related data.

Quantity of Motion

The most relevant motion data feature for this experiment exported from the Video Analysis program is the Quantity of Motion (QoM). The term refers to a calculation of “the overall movement of the motion” [4]. There are different approaches to achieving this calculation depending on the tools used to capture the motion. I used QoM to compare the total amount of movement in each participant’s interview to be able to draw informed conclusions from that.

Experiment Results

Observations from the videos and the QoM.

Participant 1

  • Participant 1 was sitting in a somewhat stiff position and did not gesture much throughout the interview.
  • Most of the detail gestures depicted objects rather than characters, however, most of the movement gestures were mostly about actions/movement carried out by characters.
  • There were about the same amount of detail and movement gestures.
  • Sometimes, P1 would use detail gestures to describe movements and vice versa.
  • There were some head movements whenever P1 was trying to recall something, since they either moved their pupils or their entire head left or right.
  • There was plenty of fidgeting movements with hands, legs and feet which are justified since P1 has been diagnosed with ADHD.

Participant 2

  • Participant 2 is the participant that gestured the most and also whose interview is longer. When asked the summary question, P2 went into a lot of detail.
  • Their position was more relaxed and comfortable.
  • The participant would sometimes use movement gestures to depict details and vice versa.
  • P2’s gestures were more complex and specific, specially the detail gestures.
  • They performed about the same amount of detail and movement gestures
  • There was hand fidgeting whenever the participant was not gesturing.

Participant 3

  • Participant 3 was also sitting in a straight, somewhat stiff position and did not gesticulate in abundance.
  • P3’s interview was the shortest out of the three, lasting only around half of the length of the other two.
  • Some of P3’s gestures were performed with the head instead of the hands and arms but were still clearly recognizable as a gesture of the aforementioned categories.
  • Most of the gestures were movement gestures and most were pretty discreet and small.

 

After going through the results of the experiment various times, that’s when I really felt the need to delimit what a gesture is. There were so many types of movements recorded throughout the interview that I couldn’t help but wonder which would make the cut and be categorized as such.

What are gestures?

The Merriam-Webster dictionary defines gestures as “a movement usually of the body or limbs that expresses or emphasizes an idea, sentiment, attitude” [4], which I believe is an accurate enough definition. However, and with this in mind, when we think of a gesture we generally think of hand, finger or arm movements. If I present a case in which a participant, as it was the case with P3, gestures the action of “returning” with a semi-circular motion of their head, a fair amount of people would also agree that head movements can be used to gesticulate. But what about facial expressions? When narrating the summary, P1 was describing that a character was shocked at the same time as P1’s eyes widened and their eyebrows raised. Same happened with P2. This specific use of a facial expression to accompany the content of the narration, specifically depicting the character’s emotion, fits with the definition of what a gesture is. Additionally, when indicating an object’s location, P3’s limbs and head did not move, but their pupils did. Would this mean that fidgeting, posture checks or clothes re-arrangement [5] count as gesturing too? These other movements can have a meaning behind them sometimes, e.g.: fidgeting out of nervousness, changes of posture due to insecurity or discomfort, or fixing one’s clothing because of self-awareness. However, they do not always have a specific meaning and whatever they may communicate is not an intentional communication. Therefore, and after a careful consideration of everything mentioned above, I conclude that a gesture can be considered any movement that expresses a message or that fulfills a goal intentionally.

Types of gestures (reprise)

I have already delimited and defined two very clear types of gestures at the beginning of this post: detail and movement gestures, both related to motion within the speech content. In their paper, Alibali et al. [1] use two different categories instead, one of which I found especially interesting: representational gestures which are “gestures that represent some aspect of the content of the speech” [1]. Within that classification, both of my gesture categories, detail and movement, belong to the representational gestures. Nonetheless, after observing more carefully the participants’ gestures, I noticed that there was a third type of gesture I had overlooked and that needed its own category:

  • Additional gestures: those that communicate something to the listener in addition to or regardless of what the speaker is saying. An example of this type would be when P3 was summarizing the episode: They did so within a sentence, then they ended the summary with an “In short” accompanied by a gesture with both open hands moving outwards towards the interviewer. By saying “in short” they were specifying that they were doing a short summary, but their hands were not gesturing something small or short. Instead, their hands were metaphorically handing to the interviewer their response, saying something like “that’s it” or “there you have it”. The meaning behind the words and the meaning behind the gesture were different. This last category is the only one with no relation to motion compared to the other two.

With this, I can conclude my gesture classification with three distinct categories: detail, movement and additional gestures.

Gesticulation patterns

Commonalities

Getting back to the analysis results, I think some interesting patterns came to light in the common factors that the participants shared throughout their interviews:

  1. When narrating in general terms, participants didn’t feel the need to gesture. P1 and P2 gestured during the summarizing question only because they decided to narrate in detail certain parts, taking their time to recall scenes and images etched in their heads. However, whenever they didn’t detail the events, the gesticulation stopped. Moreover, P3, who did the most general summary, didn’t gesticulate at all during that section.
  2. If the participants were unsure about an answer, no detail nor movement gesticulation was produced. They did raise their eyebrows or squint their eyes in what we could maybe consider an additional gesture, denoting that they were doubtful.
  3. No gesticulation (including facial expression) was made by the participants when talking about emotions or facial expressions, with one exception from P1 and P2 when describing “shock”. No gesticulation was performed either when talking about sound-related actions like “screaming” or “crying”. But it’s true that I failed to ask specific questions about sounds.
  4. No gesticulation was made when describing colors, most likely because colors are an abstract concept hard to describe with movement or shapes.
  5. Whenever the participants were trying to recall something, they would look away from the interviewer either with their pupils or with their entire head. This might be due to a need to feel ‘safer’ or less pressured while thinking, away from the interviewer’s stare.
  6. All participants (ADHD or not) fidgeted to some degree when they weren’t gesticulating. Nevertheless, it’s visible in the general overview that P1’s fidgeting was more intense.
  7. All participants used detail gestures to depict movement and movement gestures to depict details. This, I believe, would be the equivalent of using nouns as verbs and verbs as nouns when one cannot find a fitting word within the language’s vocabulary.

Differences

There were many difference as well, but the most relevant ones were these:

  1. While P1 and P3 were in a stiff, straight posture and gesticulated little, P2 was in a comfortable position and gesticulated more. This is not consistent evidence that the posture is related to the amount of gesticulation but there might be a correlation. If the participant felt more comfortable or less aware of the interview (and therefore was sitting more comfortably), they might gesticulate more and vice versa.
  2. While P1 and P2 used at some point a facial expression as a representational gesture, P3 didn’t at any point. This might be related to each individual’s personality, perhaps.
  3. While P1 and P3 performed an additional gesture when told the interview was over, P2 didn’t and just sat up from their relaxed position.
  4. While P1 and P2 used a balanced amount of movement and detail gestures, P3 used a majority of movement gestures. This as well might be related to the participants having different types of personalities or mental processes.

Final Conclusions

After conducting this study, I established that a gesture carries an intentional meaning or use and I found three types of gestures: detail, movement and additional. Moreover, I traced some patterns between the participants’ gesticulation, such as gesturing while detailing, no gesticulation for colors or when unsure of an answer, general fidgeting and movement (detail) gestures being used to depict detail (movement) elements. I also found some differences such as posture and gesticulation, facial expressions as gestures and amount of detail and movement gestures which might be related to the participants’ different personalities or mental processes.

I hope this blogpost has been an interesting read and thank you for reading!  

 

 

 

 

References

[1] M.W. Alibali, D.C. Heath, and H.J. Myers, “Effects of Visibility between Speaker and Listener on Gesture Production: Some Gestures Are Meant to Be Seen” in Journal of Memory and Language, no.44, pp.169-188 2001. doi:10.1006/jmla.2000.2752, [Online]. Available: http://www.academicpress.com

[2] L. Mol, “Language in the Hands”, PhD thesis, Tilburg University, Tilburg, 2010. [Online]. Available: https://research.tilburguniversity.edu/en/publications/language-in-the-hands

[3] Oggy, (NEW SEASON 5) Oggy and the Cockroaches The Great Pyramid Mystery (S05-E01) Full Episode in HD. (Sep. 05, 2017), Accessed: May 12, 2022. [Online Video]. Available: https://youtu.be/XJYxOumSiIY

[4] “Gesture.” Merriam-Webster.com Dictionary, Merriam-Webster. Accessed May. 10, 2022. [Online Dictionary]. Available: https://www.merriam-webster.com/dictionary/gesture.

[5] I.H. Smith and K. Larry, “Gesticulation and Effective Communication” in International Journal of Linguistics and Communication, vol. 3, no. 1, June 2015, doi: 10.15640/ijlc.v3n1a1. [Online]. Available: http://dx.doi.org/10.15640/ijlc.v3n1a1