Skin-cancer AI can outperform dermatologists in carefully controlled experiments—but what happens when it sees the whole patient, unfamiliar lesions, and real clinical context? This protocol builds the data pipeline to find out. Artificial Intelligence has proven effective in classifying skin cancers using dermoscopy images, and in experimental settings algorithms have outperformed expert dermatologists in classifying melanoma and keratinocyte cancers.

However, clinical application is limited when algorithms encounter untrained or out-of-distribution lesion categories, sometimes misclassifying benign lesions as malignant or malignant lesions as benign. Another limitation is the lack of clinical context, such as medical history, as input for the AI decision process.

Total Body Photography creates an opportunity for holistic analysis of the whole patient rather than a single lesion, but standards for annotating TBP images and preserving patient privacy are lacking. Convolutional neural networks recognize patterns in medical images, and their accuracy has surpassed that of clinical specialists in experimental settings across several medical fields.

Algorithms have been applied to image analysis in radiology, cardiology, ophthalmology, and dermatology. Translation into clinical practice remains in its early stages because researchers must address informed consent, privacy, representative training datasets, trust, explainable AI, and clinical workflow.

Only a limited number of AI models in dermatology have been applied to Total Body Photography or wide-field photography. TBP objectively captures the entire skin surface and allows sequential images to be compared between appointments. TBP platforms can also combine whole-patient image input with common patient-level metadata recorded in their user interfaces.

The protocol describes the Clinical Data Acquisition Study, including the collection, annotation, and use of clinical data to develop the iToBoS Cognitive Assistant. The study aims to present solutions, set standards for the international AI research community, and release an anonymized dataset for an international image analysis challenge.

The first objective is to build a database of skin images enriched with annotations for skin lesion diagnostic categories, skin phenotype information, and patient-level metadata. The dataset is intended to support algorithms for both lesion-specific and patient-level risk assessment for melanoma.

The master dataset will be partitioned into separate training, test, and validation datasets. The training dataset will be used for the design and development of deep machine learning models. The protocol therefore treats the training dataset as the basis for developing the planned machine learning objectives.

The methods cover data acquisition, data anonymization, image annotation, and algorithm development. A flow chart of the steps involved is provided in Figure 1. Figure one maps the authors’ end-to-end workflow for machine learning with multimodal skin-cancer data.

Patient metadata, including medical records and genetic information, and images from total-body photography and dermoscopy are anonymized, stored, and annotated by HITers and dermatologists, including lesion categories and skin phenotype. These inputs form an annotated dataset for machine-learning development, producing both individual risk stratification and risk scores for individual lesions.

The clinical data acquisition study is carried out at the Barcelona Hospital Clinic in Spain and the University of Queensland in Australia, with prospective local Human Research Ethics Committee approval at each site. Both studies require written, informed consent and collect standardized data.

The two sites follow distinct protocols for recruitment and data acquisition. The study sets a target sample size of 500 patients because optimal sample sizes for AI-based image classification tasks are difficult to estimate.

The expected risk distribution is 15 percent low or average risk, 50 percent moderate or high risk, and 35 percent very high risk for melanoma. Most patients will have up to six sequential TBP captures at intervals of every six to twelve months.

Each TBP capture is expected to produce an average of 500 image tiles for annotation and development of the photodamage convolutional neural network algorithm. All participants underwent three-dimensional total body photography using the VECTRA WB360 whole body three-dimensional imaging system.

The VECTRA WB360 system uses 92 cameras, which simultaneously capture images while the participant holds a specific anatomical pose. Software constructs a three-dimensional avatar from those images, while an attached dermoscopy camera records and maps high-resolution images of single lesions to the avatar.

The system documents skin lesions, excluding the scalp, soles of the feet, and skin covered by underwear, and facilitates monitoring changes over time. For anonymization, the original two-dimensional macro images are divided into tiles of approximately eight by ten centimeters, with a five percent overlap in each direction to ensure coverage of all lesions.

Each tile receives masks for its best-view regions, its area excluding padding, and the initial AI-based lesion detection produced by the VECTRA WB360. Average resolution and primary anatomic location are recorded for each tile, with anatomic location specified using ten categories.

Figure two shows the V7 software’s work view for reviewing a three-dimensional total-body photograph. The left toolbar provides annotation and editing tools, while lesion outlines, a ruler, and the red “area not valid” mask appear on the image and are listed in the annotations panel.

The interface also supports lesion labels and one sun-damage tag per image, making the visual important for producing standardized masks and classifications for later analysis. HITers, meaning Human Intelligent Tasks workers, are responsible for annotating skin lesions and sun-damage levels on each image tile.

The workforce is composed of women from Africa, Southeast Asia, and South America, who receive extensive training and support in image annotation and digital skills. A second annotation round is completed by expert dermatologists or dermatologically trained clinicians for the diagnostic categories displayed in Table 1.

This process is carried out by the recruitment sites in Barcelona and Brisbane, as well as the University of Trieste in Italy. Table one defines the diagnostic categories used for lesion annotation and pairs each with a shorthand label, from benign melanocytic lesion, or BM, through melanoma, or MM, to non-melanocytic diagnoses such as basal cell carcinoma, or BCC, and squamous cell carcinoma, or SCC.

The authors use these categories in a second annotation round conducted across the participating clinical sites. This shared vocabulary supports stratified division into training, validation, and test datasets, helping ensure that all diagnostic classes are represented during model development and evaluation.

The algorithms are intended to apply automatic annotations to previously unseen VECTRA WB360 images and identify evolving changes in a patient’s skin images over time. Task-specific algorithms include lesion detection and segmentation, inter-exploration lesion matching, ugly duckling detection, lesion classification, intra-exploration lesion matching, and lesion change detection.

Imaging phenotypes extracted from these algorithm results are used to compute lesion risk assessment. The data are divided by stratified randomization, with 65 percent allocated to training, 20 percent to validation, and 15 percent to testing.

Stratification ensures that all diagnostic classes are sufficiently represented in each dataset. The purpose is to prevent class imbalances that could adversely affect model performance and generalization. Figure three illustrates the authors’ three-step pipeline.

Panel A applies an HSV-based skin mask to remove background, panel B uses a YOLO detector, initialized from COCO pretraining, to place labeled bounding boxes around suspected lesions, and panel C passes those boxes to SAM as prompts for more precise lesion segmentation. This matters because it combines background filtering, detection, and pixel-level delineation before later analysis such as lesion cropping and super-resolution.

The lesion detection and segmentation task uses a three-step approach because image tiles may include background other than skin. A skin mask is obtained by thresholding within the HSV color space, followed by morphological operations to enhance and clean the segmentation result.

The resulting skin mask is applied to the final lesion segmentation to eliminate inaccurate detections outside the skin region. Figure four illustrates the full workflow for comparing the same lesion across sequential skin examinations. Panels A and B show the first and second image explorations; panel C overlays matched features, and panel D transforms the first image to align with the second.

The authors then compare segmentation masks in panels E and F, producing the detected difference map in panel G to identify changes in lesion shape and size. Figure five shows two original skin-lesion images in panels A and B, followed by pixel-based color detections in panels C and D.

The detected regions are grouped into black, blue-grey, dark brown, light brown, red, white, and skin-color categories. This matters because the authors use these color maps to compare sequential examinations of the same lesion and identify changes in color, alongside changes in shape and size.

Blood or saliva samples are used to extract DNA for genetic risk assessment for melanoma. Genome Wide Association Studies have identified single nucleotide polymorphisms associated with melanoma risk, including variants in genes associated with skin pigmentation, nevus lifecycle, DNA repair, and telomere length.

Each individual single nucleotide polymorphism produces only a small incremental change, but their cumulative effect, calculated as a polygenic risk score, is estimated to contribute as high as 30 percent of melanoma susceptibility. The applicability of AI-based methods for genetic risk assessment will also be explored, with preliminary results showing effectiveness in discovering potential complex interaction patterns among many single nucleotide polymorphisms.

The paper’s central contribution is a detailed, privacy-aware protocol for turning total body images, lesion annotations, clinical metadata, and genetic risk factors into datasets for patient-level and lesion-level melanoma risk assessment.