Reliability of the Image Category Observation System in Opponent Sports (SOCIDA)

Laura Ruiz-Sanchis

Julio Martín-Ruiz

Ignacio Tamarit-Grancha

Clara Gallego-Cerveró

Eugenia García-Sottile

Concepción Ros-Ros

Abstract

The Image Category Observation System in Opponent Sports (SOCIDA by its Spanish acronym) was developed to analyze body and gender stereotypes present in images from books and specialist manuals related to the group of opponent sports and, more specifically, to combat sports. The influence of stereotyped models in images on individuals’ processes of socialization, perception and belief formation is profound and significant. The main aim of this study was to validate the SOCIDA observation system, which is structured into two dimensions, eight criteria and 15 categories. Validation was carried out using systematic observational methodology, including consensual agreement, expert judge review, observer training and analysis, and the assessment of intra- and interobserver agreement, thereby ensuring the reliability and generalizability of the results. The data were managed through a web application developed in PHP, an open-source server-side scripting language used for dynamic web applications, which facilitates collaboration, enhancement and customization of the tool according to users’ needs. SOCIDA demonstrated high levels of agreement, internal consistency and reliability, with values of α=.841 and κ≥.826 as the mean across its items. It was concluded that this tool is suitable for physical activity teachers and researchers to identify and analyze body and gender stereotypes in images from opponent sports, contributing to a more critical understanding of the models represented in educational materials.

Introduction

The issue of body and gender representation in sport has been addressed from various disciplinary perspectives, and highlights the need for intersectional approaches that consider multiple axes of inequality (Esposito & Evans-Winters, 2022; Vandellos et al., 2023). Gender, corporality, biological traits and disability interact in complex ways in the construction of social stereotypes, influencing both the experiences of athletes in these disciplines and the public perception of them. Abioye and Nyawo (2020) point out that gender-based structural barriers are deeply rooted in social culture, resulting in normalized forms of exclusion. In this context, intersectionality is essential to analyze how certain identities may be simultaneously rendered invisible or hypervisible, according to Crenshaw (1989) and PPape et al. (2023). Vandellos et al. (2023) underline the importance of considering the interaction between race, class, gender, sexual orientation and age. As a result, femvertising has emerged as an advertising trend that seeks to empower and challenge traditional stereotypes.

Combat sports have traditionally been regarded as masculine spaces that reproduce gender stereotypes, reinforcing traditional models of male strength and dominance (Channon & Phipps, 2017). When women enter these disciplines, they are confronted by a negative social imaginary that portrays them as violating conventional standards of femininity due to their strength, physique or physical expression (Gentile et al., 2018; Hartmann-Tews, 2019; Pastor, 2021). This issue is compounded by the way in which female athletes are depicted in the media and specialized manuals: less frequently, in static poses, with a focus on appearance, and rarely in competitive action (Juliet, 2024). In contrast, male athletes are usually portrayed in athletic performance contexts and occupy more visual space (Delorme & Testard, 2015).

Visual representation in sport is not neutral: it shapes identities, validates roles and contributes to the normalization of gender hierarchies. These images serve as powerful symbolic mechanisms that reflect and shape reality (Muñoz-Muñoz & Salido-Fernández, 2022). Consequently, the way in which women and men are visually and discursively represented in the sporting arena influences their professional and social recognition, and directly affects their identity (Gentile et al., 2018; Hartmann-Tews, 2019; Pastor, 2021).

According to Camerino et al. (2020), one of the key factors in reinforcing these social constructs is somatotype. The interaction between body biology and cultural gender ideals shapes how female athletes are perceived and valued. Andersen and Swami (2021) stress that these variables have a particular impact on adolescent girls, who are constantly judged based on their bodies. Tinoco et al. (2023) demonstrate how stereotypes related to skin color and body weight are common sources of ridicule and discomfort among female athletes, which can even lead them to abandon their sporting careers. This pressure stems not only from the male gaze, but also from the stigma attached to disciplines that are traditionally considered masculine, such as combat sport (Gentile et al., 2018; Channon & Matthews, 2018). One of the least explored aspects of this perspective is the intersection of gender and disability in sport. Paccaud and Baril (2021) point out that this relationship has only recently received attention in sport studies, despite its importance in understanding stigmatization dynamics. In combat sports, this situation is particularly acute, as gender acts as a determining factor in how participation is portrayed and valued (Paccaud, 2022; Calow, 2022).

Given this context, there is a need for rigorous analysis of the visual discourses found in teaching and promotional materials related to combat sports. This study addresses the factors that promote stereotypes through systematic observational methodology (Anguera et al., 2020), using a validated category system that enables the identification of key dimensions for analyzing visual representation.

Specifically, the study proposes the validation of the Image Category Observation System in Opponent Sports (SOCIDA by its Spanish acronym) (Ruiz-Sanchis et al., 2025). Although the tool has already been published in a specialist manual, its validation is necessary to ensure its reliability and relevance to its specific context of application. SOCIDA was developed based on previous work examining how the body is visually treated in sport (Muñoz-Muñoz & Salido-Fernández, 2022; Delorme & Testard, 2015). It provides a cross-cutting, intersectional reading of images found in sports books and manuals and incorporates variables related to technical, bodily and contextual representation.

The main aim of this study was thus to validate an observational tool designed to describe gender and body stereotypes in the images included in technical books and manuals on opponent sports. These materials contained technical-tactical, psychological, physical and regulatory content. The methodological approach adopted a critical and intersectional perspective in order to uncover visual patterns that reproduce inequality, invisibility and hierarchies in how bodies and genders are represented in the field of combat sports.

Methodology

The study was framed within an instrumental design based on observational methodology (Anguera, 1991), which enabled the structured, objective and sequential recording of observable behaviors in natural contexts.

The ad hoc tool, according to the typology proposed by Anguera et al. (2011), was classified as nomothetic, as it focused on the systematic comparison of multiple still images extracted from different sports manuals and books, allowing for the analysis of common patterns and divergences in visual representations. It was also cross-sectional, as it was limited to images captured at specific moments, and multidimensional, since it incorporated several analysis criteria related to technical, aesthetic and sociocultural aspects (Anguera et al., 2020).

A qualitative content analysis with an interpretative approach was used for data processing. According to Neuendorf (2017), this technique enables the identification and interpretation of the implicit meanings in visual material, based on the context in which it was produced, especially in relation to sociocultural values and possible stereotypes represented.

Sample 

The sample consisted of 240 images (n = 240), with 120 representations per observer round. These were distributed equally, with 24 representations for each of the five combat sports analyzed: fencing, boxing, taekwondo, judo, and Olympic wrestling (men’s and women’s freestyle). This set of images represented approximately 2% of the total visual content available in the 36 selected books and specialized manuals. The images were selected through random sampling with structural representativeness criteria, ensuring the inclusion of all identified grouping types. Furthermore, the selected images covered the 15 categories defined according to the eight criteria established in the analysis system.

The sample consisted of 40 individual representations (33%), 50 paired (42%), 20 group (17%) and 10 mass representations (8%). The selected books and manuals had to meet the following inclusion criteria: 1) Specialized manuals that explicitly, systematically and orderly provide information for the initiation and improvement of performance in the sport modality, as well as instructions considered necessary for the execution of defined activities in the technical-tactical, regulatory, physical preparation and/or training support processes; 2) Manuals written in Spanish and English. 3) Manuals printed between 2012 and 2022.

Recording Tool

The observation tool used in this study was the Image Category Observation System in Opponent Sports (SOCIDA, by its Spanish acronym) (Ruiz-Sanchis et al., 2025). Although the tool had already been published, this study focused on its validation to ensure the methodological robustness of the collected data.

SOCIDA enabled the structured, systematic and multivariate recording of data and images related to the combat sports included in the Olympic programme for the Paris 2024 Games.

From a technical perspective, the SOCIDA platform was developed as a web application programmed in PHP, allowing online use from any device with internet access. The interface was designed to be simple, intuitive and operationally efficient, minimizing potential errors in the observers’ recording and analysis process. The web application is accessible from the sub-domain http://freesocida.giepafs.net hosted on a virtual private server (VPS) with Linux operating system (CentOS 6.4). The platform was programmed in PHP (v.8.1) and uses a MySQL relational database (v.5.3.1), which enables the structured insertion, storage and management of observational data. This digital environment allows for continuous data updating, avoids duplication, and automates the storage of records, thereby improving the quality and reliability of the analysis. SOCIDA is structured into two dimensions (general and specific), eight criteria and 15 categories, designed to be exhaustive and mutually exclusive to ensure the reliability of the coding system and to facilitate subsequent multivariate analysis. 

The registration and analysis process in the platform is conducted in two sequential phases: 

First, the typological classification of the image, which considers the visual grouping based on the number of people depicted: individual (one person), paired (two people with clearly visible faces and/or full bodies), group (between three and 10 people in an organized sports practice setting), and mass (more than 10 people in the image). 

Second, the coding of metadata and observational variables: an identification code is assigned to each image, along with additional variables such as the sport represented, publication year of the manual, image size, and use of color or black and white. These data appear in the user interface alongside the criteria and categories for analysis, thus facilitating the coding and review process by the observers.

Table 1 summarizes the criteria and categories for the general dimension, applicable to all images. Table 2 describes the criteria and categories for the specific dimension, differentiated by visual grouping type: individual, pair, and group. For pairs, items are duplicated to code the left and right figures, while for groups, items are subdivided by the number of men and women observed in each category. Table 3 presents the reduced set of 12 criteria and categories for the specific dimension applicable to mass groupings (more than 10 people), where coding is done using percentage approximation and binary detection of item presence.

Table 1

Overall dimension, criteria, categories and indicators of the SOCIDA tool

See Table

Table 2

Specific dimension: criteria and categories for individual, paired and group formats

See Table

Table 3

Specific dimension, criteria and categories for mass grouping (>10 people)

See Table

Data Quality Control

To validate SOCIDA, three expert judges were intentionally selected based on their professional and academic backgrounds. Inclusion criteria were: a) experience in the study of body and gender stereotypes in physical activity; b) over five years of experience in intersectional image analysis; c) experience as coaches or teachers familiar with specialized combat sports manuals.

This panel critically reviewed the tool’s category system. Five basic criteria were used to evaluate the items—sufficiency, clarity, coherence, relevance, and effect—using a 4-point Likert scale. Qualitative and quantitative observations were provided for each unit, enabling adjustment and refinement of deficient items. This process enhanced content validity and ensured the tool’s quality and relevance.

To ensure the reliability of the coding process, three observers were intentionally selected. Although two observers are sufficient according to Neuendorf (2017), this number was chosen to strengthen the study’s methodological rigor. Selected observers met the following requirements: a) prior experience in intersectional image analysis research; b) work with illustrated images in the sports domain; c) access to the internet to participate online.

The observers demonstrated a high level of expertise in visual analysis of combat sports. Their experience facilitated coding and contributed qualitative judgements and insights that enhanced the analysis system. 

Training Procedure and Reliability Calculation

The validation of the category system and agreement analysis involved several strategies. First, dimensions, criteria and categories were constructed based on documentary reviews and consensual agreement (Lapresa et al., 2021). Next, a descriptive manual-protocol of the tool was developed. The system was then reviewed by three expert judges. Subsequently, three observers were trained and tested in image analysis. Finally, intraobserver, interobserver and generalizability coefficients were calculated (Sastre et al., 2022).

Observer training consisted of a session where the category system was presented, doubts were clarified, and a subset of images (not included in the final sample) was coded. The aim was to standardize coding criteria and ensure operational understanding of the categories (Sastre et al., 2022). To evaluate interobserver reliability, Cohen’s kappa index (Cohen, 1960) was calculated for dichotomous and presence/absence variables, and Krippendorff’s alpha (Krippendorff, 2019) was used for nominal categories. Agreement values exceeded .80 across all categories, interpreted as “optimal” or “almost perfect” (Landis & Koch, 1977). These results ensured the consistency and reliability of the coding system. For intraobserver agreement, kappa values were calculated for the same observer at two different points in time (Table 4). The initial calculation showed a mean of .820, while the final one yielded .882 (Cohen et al., 2011), demonstrating high reliability.

Table 4

Intraobserver agreement, first and second round in the specific dimension

See Table

To evaluate interobserver agreement, kappa was calculated based on the same situation observed independently by the three evaluators (Table 5). Two weeks later, the procedure was repeated, obtaining a mean kappa of .843, indicating a high level of agreement. As a complementary measure, Cronbach’s alpha was also calculated to assess consistency among observers, resulting in a value of .737 (Losada & Arnau, 2000).

Table 5

Interobserver agreement, first and second rounds

See Table

SOCIDA was adapted based on the data. When comprehension issues arose, some categories were revised and adjusted. After the modifications, observers reported improved global assessment of content validity and the adequacy of the criteria:

• In the “somatotype” criterion, specific guidance was added to differentiate the three possible levels.
• In the “age” criterion, four groups were defined based on age ranges: “Childhood” (2–12 years), “Youth” (13–20), “Adulthood” (21–40), and “Maturity” (41 and older).
• In the “aesthetics” criterion, the “Various” item was added.

These modifications are reflected in Tables 1, 2 and 3, which describe the criteria and categories.

The final phase involved analyzing data from the second round of observations to determine the system’s reliability and stability with the data from the three observers. A total of 120 images were analyzed following the initial criteria, based on a new stratified, non-probabilistic sampling with synchronous observer participation.

Data Analysis

To validate the SOCIDA system, the statistical analysis of the data was performed using SPSS 26.0 and the SAGT v1.0 version 211 software (Hernández-Mendo et al., 2016) for generalizability analysis.

Cohen’s kappa coefficient was applied for each criterion in the tool. In addition, overall pairwise comparisons between observers were conducted. The reliability of the quantitative scales across the three observers was adjusted using Fleiss’ kappa. Subsequently, Cohen’s kappa was applied for pairwise comparisons, complemented by a generalizability analysis using a two-facet design: observer/categories.

Results

Fleiss’ kappa coefficient estimated values ranging from .701 to .962 for interobserver agreement and from .722 to .983 for intraobserver agreement among the three participants. These coefficients are shown in Table 6. The results indicated substantial agreement in the following criteria: image characteristics (.933), context characteristics (.857), performed activity (.885), function in the image (.912), aesthetics (.877), and person with disability (.961). All of them exceeded the value of .80, a threshold which, according to Landis & Koch (1977), indicates almost perfect agreement.

Table 6

Kappa values in the inter- and intraobserver test for SOCIDA

See Table

A group interobserver agreement analysis was carried out (Table 7). The variables image characteristic (κ = .941), the dimension “athlete with a disability” (κ = .933), and image context (κ = .906) obtained the highest kappa scores, reflecting a very high correlation. In contrast, the criteria of sociodemographic characteristics (κ = .671) and physical characteristics (κ = .627) yielded the lowest coefficients, indicating a moderate correlation strength.

Table 7

Interobserver agreement analysis

See Table

With the data obtained, contrast tests were conducted to calculate the reliability of SOCIDA. In this second round, Fleiss’ kappa coefficient was applied, yielding an intraobserver agreement ranging from .753 to .862, and an interobserver agreement from .721 to .880. The mean kappa value was .826, indicating a good level of agreement (Landis & Koch (1977), and high, highly suitable reliability. Additionally, Cronbach’s alpha was calculated as an indicator of observer competence, yielding a result of .841 (Losada & Arnau, 2000).

A generalizability study was conducted using a two-sided model: Criteria and Observers, under the C/O design, where the criteria acted as the differentiation facet and the observers as the instrumentation facet. The variability of each facet and their interactions was estimated, as well as the absolute and relative generalizability coefficients. Eighty-nine per cent of the variability corresponded to the criteria, 6% to the observers and 5% to the interaction. The overall generalizability index was .821, indicating a good fit and allowing the conclusion that the results can be generalized reliably.

Discussion

Specialist manuals on opponent sports—and more specifically, on combat sports—constitute a fundamental didactic resource both for initial and ongoing training of professionals in the field of sport. However, their long-lasting nature and frequent use by students and professionals make them significant vehicles for the transmission of cultural norms and social biases (United Nations Educational, Scientific and Cultural Organization, 2017). In contexts where gender or body stereotypes prevail, these tend to be integrated and reproduced in the visual and textual content of such manuals (Abioye & Nyawo, 2020). As a result, far from being neutral tools, these resources may contribute to the consolidation of prejudices, directly influencing the perception and education of future sport professionals. It is worth noting that the presence of such stereotypes can significantly impact the training of future physical education teachers and coaches, who may in turn perpetuate them among their students and athletes, thus generating negative effects and exclusionary situations—particularly for women and girls in sport (López & Pardo, 2021; Monforte & Úbeda-Colomer, 2019). It is therefore essential to identify the stereotypes present and analyze the characteristics and attributes of the people represented in these documents, as well as to describe the symbolic constructs that ultimately shape and define the roles assigned in sport (Martínez-Abajo et al., 2020; Lampert, 2018).

The aim of this study was to validate the Image Category Observation System in Opponent Sports (SOCIDA) (Ruiz-Sanchis et al., 2025), a tool designed to analyze and describe gender and body stereotypes present in the images of sport manuals and textbooks. The validation process focused on ensuring the reliability of the tool, starting from a design grounded in the relevant scientific literature and in consensual agreement (Lapresa et al., 2021). Special attention was also given to the training and selection of observers, to ensure high-quality standards both in their instruction and in the training procedures, contributing to the robustness and consistency of the data obtained.

The integration of an intersectional perspective (Crenshaw, 1989) is especially relevant, as it enables an analysis of how gender stereotypes interact with other axes of inequality—such as age, body type, race and disability—in the construction of image and recognition of participants in the sporting context (Esposito & Evans-Winters, 2022; Channon & Matthews, 2018). As evidenced in the literature (Vandellos et al., 2023), the visibility or invisibility of certain groups in educational and sports dissemination materials reflects normalized patterns of exclusion that particularly affect women, girls and people with disabilities (Gentile et al., 2018). In this regard, the analysis of the images using the SOCIDA system has enabled the identification of the reproduction of body and gender stereotypes, as well as the limited presence of diverse and inclusive models. This aligns with the need to address equal opportunities and coeducation through physical education and the training of future teachers. The interaction between somatotype, biological traits and gender stereotypes reinforces the pressure on female athletes, especially during formative stages, and generates negative effects on self-esteem, retention, and the development of their sports careers (Hartmann-Tews, 2019).

The SOCIDA tool has proven to be valid and reliable for identifying these patterns, as it enables a systematic and multivariate analysis of visual representations in sports-related materials (Anguera et al., 2020). Its application facilitates the auditing of content—especially images—from a critical perspective, and provides empirical data that can guide the revision and redesign of teaching resources and sports manuals towards more inclusive and equitable models. Its application allows the identification of visual patterns that reproduce inequalities, invisibility and hierarchies, thus facilitating the critical review of educational and dissemination materials that influence the social and professional perception of sport. The reliability achieved in the coding processes and the high inter- and intraobserver agreement reinforce the methodological consistency of the study (Losada & Arnau, 2000), and support the usefulness of the instrument for research in educational and professional contexts.

The tool aims to raise awareness among teachers, coaches and publishers about the presence of bias and stereotypes that may perpetuate exclusion—particularly of women, girls and people with disabilities—thus contributing to the training of professionals who are more aware and committed to equality and inclusion (Monforte & Úbeda-Colomer, 2019). It may also help improve the design and selection of visual representations that reflect real diversity, and promote alternative models that challenge traditional standards and enable all groups to see themselves represented in the field of sport (Delorme & Testard, 2015; Channon & Phipps, 2017). Moreover, it is suitable for promoting more egalitarian and respectful sports environments, as it contributes to the normalisation of body and gender diversity in the images that are part of physical and sports culture (Muñoz-Muñoz & Salido-Fernández, 2022), which can positively influence the motivation, self-esteem, and retention of historically marginalised groups.

In sum, SOCIDA is a suitable tool for reviewing visual narratives in opponent sports. It aims to support the development of an inclusive, diverse, and equitable sporting culture that responds to current social demands and fosters the full participation of all individuals, regardless of gender, physicality, or disability.

Conclusions

The results of this study show that the developed instrument has content validity and optimal reliability for identifying and analyzing body and gender stereotypes in images found in manuals and specialized books on opponent sports, with particular emphasis on combat sports. Its digital format and online accessibility facilitate collaborative work between coders, improve the traceability of the analysis, and optimize data management, thereby reinforcing its applicability in academic and professional contexts.

These features enable researchers, publishers, educational institutions, teachers, sports federations and public bodies to audit their visual materials and communication strategies from an equity perspective. Its use promotes greater critical awareness of the persistence of cultural and social stereotypes in training materials, helping to guide the design of more inclusive, representative and educationally valuable content.

Ultimately, SOCIDA is a functional tool that contributes to improving the quality and social responsibility of visual content in contemporary physical and sports culture.

Acknowledgements

This work is part of the project (CIGE/2022/81). Combat sports: an analysis of gender and body stereotypes in the images of specialized books and sports press. Funded by the Conselleria de Educación, Cultura, Universidades y Empleo de la Generalitat Valenciana.