Reproducibility of the clinical criteria (ABCDE rule) and dermatoscopic features (7FFM) for the diagnosis of malignant melanoma

ARTICLE

The diagnosis of melanoma is a challenge that dermatologists face every day. The criteria for the clinical diagnosis of melanoma are mainly founded on the geometric characteristics of the lesion under examination: the shape and the symmetry of the lesion, the features of its margin, the characteristics of its colors, its dimensions. The ABCD rule, developed by Rigel et al. [1], a simple checklist of the signs of early melanoma, is based on four macroscopical criteria: A, asymmetry in shape; B, border irregularity; C, variation in color; D, diameter > 6 mm.

Afterwards, a new criterion, E, standing for evolution, was added to improve sensitivity in the diagnosis: evolution is an anamnestic criterion based on the patient's description, a term that includes elevation, enlargement or change in the color of the lesion [2].

Melanoma can be cured definitely with surgery if discovered at an early stage, i.e. when maximum thickness is less than 0.75 mm. According to a number of statistics, the clinical diagnosis of melanoma has a mean sensitivity of 67% (range 48 to 81% according both to the physician's ability, dermatologists having better results than general practitioners, and to the type of melanoma, the thinner the melanoma the more difficult the diagnosis) [3, 4].

Among the new techniques developed to ameliorate the clinical diagnosis dermatoscopy, dermoscopy or epiluminescence microscopy is one of the more promising. Epiluminescence microscopy can be performed either with a simple or a compound microscope. The simple microscope yields an image at a fixed 10X magnification, it is small, handy and suitable to be used in outpatient practice for the screening of pigmented skin lesions. Already in 1994 W. Stolz proposed a diagnostic method for the dermatoscopic diagnosis of pigmented skin lesions using a simple microscope that he called the ABCD rule of dermatoscopy [5-7]. In 1996 we developed a simplified diagnostic dermatoscopic method which evaluates the presence or the absence of only seven dermatoscopic features: we called this method Seven features for melanoma (7FFM) [8-11].

The clinical and the dermatoscopic diagnoses of pigmented skin lesions are based on the analysis of a visual image and are subjective diagnoses. As known, the reliability of a diagnostic test or method mainly depends on the reproducibility of the interpretation of the results on a given specimen. Therefore, the usefulness of a diagnostic dermatoscopic method for the diagnosis of pigmented skin lesions is mainly dependent on the rate of concordance, among different observers, in the interpretation of the dermatoscopic features present in the lesion under examination. These data should be compared to the rate of concordance, among the same observers, in the interpretation of the clinical criteria. As the clinical diagnosis with ABCDE criteria is the only way to diagnose melanoma by naked eye examination alone, data about concordance with the ABCDE rule should be considered as the "gold standard". In order to propose the use of dermatoscopy for the diagnosis of pigmented skin tumors, the reproducibility of the dermatoscopic features should not be inferior to the reproducibility of the clinical criteria.

To inform dermatologists about our method and to evaluate its reproducibility we held three Dermatoscopy Courses, from April 1999 to May 1999, which were attended by 73 dermatologists.

At the beginning of the course the participants had a self-examination test where they evaluated 50 clinical slides of pigmented skin tumors with the ABCDE rule. At the end of the course the participants evaluated the dermatoscopic slides of the same skin tumors with our method 7FFM.

This article presents an analysis of their evaluations.

Materials

From April 16, 1999 to May 29, 1999 we held three courses in dermatoscopy to inform dermatologists about our dermatoscopic diagnostic method. A total of 73 dermatologists attended our courses. Each course lasted about six hours, from 9.30 a.m. to 17.30 p.m. In the morning session the participants were asked to execute a pre-test evaluating 50 clinical slides of pigmented skin tumors with the clinical ABCDE rule. Afterwards, we presented the principles of dermatoscopy, the correlation between dermatoscopic features and histopathology, the dermatoscopic features of non melanocytic and melanocytic pigmented skin tumors.

In the afternoon session, the various diagnostic dermatoscopic methods were shown, including the one developed by us. Finally, the participants were asked to execute a post-test evaluating the dermatoscopic slides of the 50 pigmented skin tumors, previously evaluated clinically, using our diagnostic method.

The pre-test and post-test were formed of the clinical and dermatoscopic slides of 50 pigmented skin tumors. None of the lesions was located on the head, on the palms or on the soles.

Two of us (C. B. and E. R.) selected the fifty pigmented skin tumors. Histologically the lesions proved to be: seborrhoeic keratoses n. 3, pigmented basal cell carcinoma n. 2, blue nevus n. 1, angiokeratoma n. 2, Spitz nevus n. 5, junctional nevi n. 5, compound nevi n. 9, nevi undergoing regression n. 10, melanomas n. 12.

The maximum diameter of the lesions evaluated was ¾ 10 mm; 36 lesions (9 melanomas) presented a maximum diameter ¾ 6 mm. The maximum thickness of melanomas was 1 mm and the minimum thickness was 0.10 mm; two melanomas were in situ.

During the tests each slide was projected onto a screen of 2 x 2 m with the observers at a distance of 3-6 m.

Thirty seconds were given to the observers to evaluate the clinical slides with the ABCDE rule, while 45 seconds were allowed to evaluate the dermatoscopic slides with our 7FFM method.

The courses were held in the following Italian cities, on the dates indicated with the respective number of dermatologists:

Course held in Padova, 04/16/99 attended by 37 dermatologists.

Course held in Milan, 05/07/99 attended by 17 dermatologists.

Course held in Milan, 05/28/99 attended by 19 dermatologists.

Diagnostic methods

The clinical evaluation has two steps: the first, a decision based on one's own experience whether the lesion under examination is melanocytic or non melanocytic. Subsequently, those identified as melanocytic are clinically evaluated on the five ABCDE criteria. Criterion A is the geometrical asymmetry on the two axes of the tumor. Criterion B is an irregular, ragged or indented border. Criterion C is the presence of at least two different colors within the lesion, except the usual symmetrical darkening of the lesion in its center, typical of juctional nevi. Criterion D is a maximum diameter > 6 mm. Criterion E, as evolution, is an anamnestic criterion based on patient's description, including in this term elevation, enlargement or change in the color of the lesion [2]. Criteria D and E were assessed on the basis of the case records and, as they are necessary for a correct diagnosis, their presence in the lesion under examination was reported on the test sheet. A present clinical criterion is rated as 1 and an absent criterion as 0. The score of each lesion is the sum of the clinical criteria present in the lesion: therefore the clinical score may range from 0 to 5. In relation to the number of criteria, i.e. the score, whose presence is considered necessary to classify a lesion as malignant, various diagnostic methods can be obtained. According to data reported in literature the most used scores in the diagnosis of melanoma are score 2 and score 3 (i.e. the presence of 2 or 3 clinical criteria is considered necessary to diagnose melanoma) [12].

Dermatoscopically, the lesions were evaluated with the method we have developed (7FFM).

Our method has two steps: in the first step one decides if the lesion under examination is melanocytic in nature, following the dermatoscopic algorithm used to distinguish melanocytic from non-melanocytic lesions [5-10].

The algorithm is as follows: the lesions showing network or globules are regarded as melanocytic. The presence of horny pseudocysts and comedo-like openings, without a network or globules, suggests seborrheic keratosis. Maple leaf-like areas at the periphery suggest basal cell carcinoma, a homogeneous blue coloring points to blue nevus, while red-blue areas are typical of angioma and angiokeratoma. If none of these features is present the lesion is regarded and evaluated as melanocytic.

The second step, used to evaluate only the lesions considered as melanocytic, is based on seven dermatoscopic features shown to be significant for malignancy by a statistical analysis on a training set of 218 cutaneous pigmented lesions. The features were divided into major and minor according to statistical significance, sensitivity and specificity. The major features were pseudopods, radial streaming, regression-erythema and gray-blue veil; the minor features were unhomogeneity, irregular pigment network and sharp margin.

These features are regarded as present or absent in the lesion under examination. The regular network has thin lines, a close mesh net and is uniform throughout the lesion [13]. The irregular network is thick, has a wide mesh net and shows varying features in the same lesion.

We added two new dermatoscopic features we detected during our experience to the classical ones, namely regression-erythema and unhomogeneity, The term regression-erythema defines the disappearance of dermatoscopic features in a given area of the lesion, while diffuse erythema, possibly with a few angiectases, is observed.

Unhomogeneity is an asymmetrical or irregular distribution in the lesion of at least two dermatoscopic features not necessarily predictive of malignancy.

Other authors in previous studies have proposed dermatoscopic features similar to unhomogeneity.

Nilles et al. [14] considered an asymmetrical pigment distribution (no relation with dermatoscopic features) with four different grade of severity. Kenet et al. [15] described a multicompetent pattern that consists of three or more discrete regions with different ELM appearances, including a darkly pigmented region with broadened network.

The sharp margin is regarded as such when an area of diffuse pigmentation with abrupt ending is present on at least one fourth of the margin of the lesion.

The pigment network and other dermatoscopic features of this method are not taken into consideration in determining the sharp margin.

Pseudopods are considered predictive of malignancy when they display an irregular distribution: in fact, epithelioid and/or spindle cells nevi usually present regularly distributed pseudopods [16].

Following such selection we attributed a score of 2 to the major features and a score of 1 to the minor features. The lesions where the sum of the features gave a score of >= 2 were diagnosed as being malignant, therefore to make a diagnosis of melanoma, the presence of one major feature or the concurrent presence of two minor features was regarded as sufficient.

Statistical analysis

Development of tests

Each author alone took the pre-test and the post test and evaluated the clinical criteria and the dermatoscopic features as present or absent in the 50 pigmented skin tumors selected.

The agreement % and the concordance of the three authors on the clinical criteria and the dermatoscopic features was calculated.

To evaluate the concordance among observers, i.e. inter rater reliability, a kappa statistic was used.

Kappa statistic is the statistic most often used to measure agreement between two observers on a binary variable. Kappa is defined as the agreement beyond chance (i.e. observed agreement minus chance agreement), divided by the amount of agreement possible beyond chance (i.e. 100% agreement minus chance agreement).

Kappa statistic is calculated as follows.

The answers given from a judge 1 and the answers given from a judge 2 were displayed on a 2 x 2 table.

The observed agreement is O = (a + d)/n.

The chance (expected) agreement is C = [(a + c)(a + b) + (b + d)(c + d)]/n².

Kappa is k = (O - C)/(1 - C).

Kappa values may range between - 1 and 1: negative results are obtained when the agreement occurs less often than expected by chance, K values 0-0.40 indicate poor agreement, 0.41-0.75 from fair to good agreement, > 0.75 excellent agreement [17-18].

Each author was considered alternatively as judge 1 or judge 2, and the mean kappa values obtained among the three authors, for each clinical criterion and for each dermatoscopic feature, were considered as the kappa among the authors.

On the basis of the authors' analysis the clinical criteria and the dermatoscopic features were considered present or absent when at least two of the three authors agreed about their presence or absence.

One of us (V.D. P.) did not participate in the selection of the tumors and she was not aware of the histopathologic diagnosis of the 50 pigmented skin tumors when she took the tests. The values of the clinical and dermatoscopic sensitivity and specificity that she obtained were considered representative of the Authors' diagnosis. The sensitivity of the diagnostic method is the proportion of all cases of histologically proved melanomas that have been diagnosed as melanomas. The specificity of the diagnostic method is the proportion of all cases histologically proved not to be melanomas, that were diagnosed as not melanomas. The formulas are:

Sensitivity = (diagnosed melanomas)/(diagnosed melanomas + not diagnosed melanomas),

Specificity = (diagnosed non melanomas)/(diagnosed non melanomas + not diagnosed non melanomas).

Evaluation of tests

An answer was considered as correct if the diagnosis obtained with the clinical and dermatoscopic evaluation was concordant with the one proposed by the three authors. The clinical and the dermatoscopic agreement % were calculated. A kappa statistic was calculated: the correct answer was considered as judge 1, while each dermatologist was considered as judge 2. Therefore, a K value was calculated for each dermatologist. The mean and the 95% confidence intervals of values obtained from participants were reported.

The clinical (score 2 and score 3) and the dermatoscopic (7FFM) sensitivity and the specificity obtained by each participant were calculated. The mean and the 95% confidence intervals of values obtained were reported.

The mean and the 95% confidence intervals of agreement % and of kappa obtained from the participants for each clinical criterion and for each dermatoscopic feature were also calculated.

The calculations were performed with Microsoft Excel 4.0.

Results

Development of tests

The three authors presented the following agreement % and kappa for the clinical criteria: the agreement for the clinical first step, discriminating between melanocytic and non melanocytic pigmented skin tumors, was 100%, K = 1. The three authors were concordant in considering as melanocytic false positive, in the clinical evaluation, two seborrhoeic keratoses.

The agreement % and kappa for the clinical criteria were: for criterion A, agreement = 79%, K = 0.55; for criterion B, agreement = 76%, K = 0.51; for criterion C, agreement = 75%, K = 0.50.

The agreement among the three authors about the dermatoscopic first step discriminating between melanocytic and non melanocytic pigmented skin tumors was = 100%, K = 1.

The dermatoscopic features of our method 7FFM presented the following values of agreement % and kappa: unhomogeneity, agreement = 83%, K = 0.64. Irregular pigment network, agreement = 85%, K = 0.44. Sharp margin, agreement = 92%, K = 0.26. Irregularly distributed pseudopods, agreement = 92%, K = 0.59. Radial streaming, agreement = 96%, K = 0.57; Regression erythema, agreement = 99%, K = 0.77. Gray-blue veil, agreement = 91%, K = 0.59.

The sensitivity and the specificity obtained by one of us (VDP), unaware of the diagnosis, were: for clinical score 3, sensitivity 33%, specificity 71%; for clinical score 2, sensitivity 58%, specificity 53%; for the dermatoscopic diagnosis, sensitivity 91%, specificity 92%.

The three authors were concordant in considering one melanoma as a dermatoscopic false negative and two nevi with regression and inflammatory infiltrate as dermatoscopic false positive.

The clinical criteria were considered as present on the basis of the authors' evaluation and of the case records. Criterion A was considered present in 22 lesions, criterion B in 17 lesions, criterion C in 23 lesions, criterion D in 16 lesions and criterion E in 20 lesions. The dermatoscopic features discriminating between melanocytic and non melanocytic pigmented skin tumors were present in the following number of lesions: horny pseudocysts and comedo-like openings in 3 lesions, maple leaf-like areas and telangectasies in 2 lesions, red blue areas in 2 lesions, homogeneous blue coloring in 1 lesion. The dermatoscopic features of our method 7FFM were represented in the following quantity: unhomogeneity in 20 lesions, irregular pigment network in 6 lesions, sharp margin in 3 lesions, irregularly distributed pseudopods in 6 lesions, radial streaming in 3 lesions, regression erythema in 1 lesion, gray-blue veil in 6 lesions.

Evaluation of tests

A total of 65 tests out of 73 were valuable for statistical analysis. Clinically and dermatoscopically an answer was considered correct when the score obtained gave a diagnosis concordant with the one proposed by the authors. Therefore, for the clinical score 2 the lesions with a score < 2 were considered non melanomas while the lesions with a score >= 2 were considered melanomas; the clinical score 3 was evaluated in the same way. Dermatoscopically the lesions with a score < 2 were considered non melanomas while the lesions with a score >= 2 were considered melanomas. The test included two dermatoscopic false positives and one dermatoscopic false negative.

The mean and the 95% confidence intervals of agreement % for the clinical and dermatoscopic 1st steps are: clinical mean 69% (95%CI 66%-71%), dermatoscopic mean 88% (95%CI 87%-90%).

The mean and the 95% confidence intervals of kappa for the clinical and dermatoscopic 1st steps are: clinical kappa 0.2 (95%CI 0.16-0.23), dermatoscopic kappa 0.54 (95%CI 0.5-0.58).

The mean and the 95% confidence intervals of agreement % and kappa for the clinical score 2 and 3 and for the dermatoscopic method 7FFM are reported in Table II.

The mean and the 95% confidence intervals of the sensitivity and the specificity for the diagnosis of melanoma of clinical score 2 and 3 and of our method 7FFM are reported in Table III.

The mean and the 95% confidence intervals of agreement % and kappa for the clinical criteria A, B and C and for the dermatoscopic features of our method 7FFM are reported respectively in Tables IV and V.

Discussion

Five diagnostic methods for the simple microscope or dermatoscope have been reported in literature. These methods are: the ABCD rule of dermatoscopy [5-7], the methods developed by Nilles et al. [14] and by Menzies et al. [19-21], the seven features for melanoma (7FFM) developed by us [8-11], and the more recent 7 Point Checklist based on pattern analysis [22]. These diagnostic dermatoscopic methods, utilized by their own authors, have both values of sensitivity (97.9%, 90%, 92%, 94.6% and 95% respectively) and specificity (90.3%, 85%, 70%, 85.5% and 75% respectively), which are not significantly different from one another.

In spite of these good values of sensitivity and specificity dermatoscopy is not extensively used by dermatologists and is considered a technique for experts.

In fact, as shown by Binder et al. [23], ELM criteria are often confusing for non-experienced observers. These authors showed that ELM pattern analysis has a low kappa value for ELM non-experts, median K = 0.36, while ELM experts presented a higher value median K = 0.56.

These authors underlined the fact that almost all studies about the value of ELM are based on data derived from ELM experts and concluded that a formal broad-based training in ELM should be offered to the dermatologic community.

In a subsequent paper [24] they evaluated the effect of short formal ELM training on the diagnostic performance of 11 previously untrained dermatologists obtaining a significant improvement of their diagnostic performance.

A few studies have been published about the reproducibility of dermatoscopic features among different observers. In a study between two observers, Stanganelli et al. [25] concluded that the best K value (median K = 0.77) was obtained in the evaluation of the presence or the absence of ELM features. Lower values were obtained for distribution, for width, thickness and size, and for pigmentation, of ELM features (median K 0.47, 0.39 and 0.21 respectively). In a study involving 12 dermatologists about reproducibility on ELM features and ELM diagnosis Stanganelli et al. [26] found better values for ELM diagnosis (mean K 0.65) than for ELM features (mean K range 0.14 - 0.49). Carli et al. [27] found good K values for ELM criteria among 9 ELM experts (K range 0.607-0.827 except pigment network K = 0.009).

All the studies reported so far evaluated the reproducibility of the dermatoscopic features without evaluating the reproducibility of their obvious terms of comparison, i.e. the clinical criteria. As the clinical diagnosis is the only way widely accepted to diagnose melanoma, a new technique should have an inter-observer variability at least equal to the one obtained with the clinical ABCDE rule. Besides, no study about the reproducibility of a dermatoscopic diagnostic method has been reported in literature.

The mean kappa values obtained show a good reproducibility of our dermatoscopic method, step 1 K = 0.54, step 2 K = 0.64 compared to the clinical method, step 1 K = 0.2, score 2 K = 0.44, score 3 K = 0.61.

The seven dermatoscopic features of step 2 of our method presented a good reproducibility compared with the three clinical criteria. In fact the range of the mean kappa values obtained for the dermatoscopic features is not lower than the values obtained for the clinical criteria: 0.62-0.25 versus 0.35-0.25. If we had not compared the reproducibility of our dermatoscopic features to the reproducibility of the clinical criteria we could have concluded that the reproducibility of some of our features was poor. These low values may be due to the small diameter of the lesions, 36 lesions presented a maximum diameter ¾ 6 mm, the smaller the lesion the hazier the criterion or the feature. In fact lesions with a diameter from 6 to 10 mm were evaluated in a previous study and we obtained higher kappa values for our dermatoscopic features (K range 0.636-0.34) [28]. Besides, the short time allowed for the evaluation, 30 seconds per slide in the clinical test and 45 seconds per slide in the dermatoscopic test, may explain these low values.

We chose small, clinically equivocal lesions for our tests and allowed a short time of examination in the attempt to reproduce a real clinical situation.

The number of cases tested was a compromise between the requirements of the study and compliance of the participants to the test.

Our data confirm that the clinical and the dermatoscopic diagnoses of pigmented skin lesions present a better reproducibility than that of the single clinical criteria and dermatoscopic features, which was also confirmed by Stanganelli et al. [26].

Unhomogeneity and irregular pigment network are frequently confused and this explains the lower mean percentages of agreement (70% and 75%) among the features of our method. These two features may seem over-represented in our test but this is due to the fact that all melanomas of our test were thin melanomas (maximum thickness 1 mm.) and, as shown by Argenziano et al. [29], unhomogeneity and irregular pigment network are often present in thin melanomas.

Pseudopods and radial streaming are other two features that are frequently mistaken for each other.

The high value of agreement and the low value of kappa for regression- erythema are due to the fact that only one lesion of our test presented this feature. For this reason, the chance of not detecting it when present, was higher than that of detecting it when absent, and this explains the high value of agreement. Kappa statistic gives equal importance to the one lesion with the feature and the 49 lesions without the feature and, as many participants did not detect regression-erythema, the kappa value is low.

The results of the self-examination tests point to a good reproducibility of our method among dermatologists of our community.

We were interested in a global evaluation of the reproducibility of our method from dermatologists of our community and for this reason we did not evaluate their level of knowledge in dermatoscopy.

As the lesions were chosen to show the dermatoscopic features, any inference from these data about an improvement given from our method compared to clinical diagnosis would be biased.

In fact these lesions are not representative of daily routine.

But the improvement given from our method compared to clinical diagnosis has already been evaluated and published [11].

The mean values of sensitivity and specificity obtained from participants to our courses have been reported to show that they are similar to those obtained by the authors.

CONCLUSION

In conclusion the results obtained in our study indicate that our method 7FFM is easy to learn after a short training program, that it presents a reproducibility similar to clinical diagnosis, and that the clinical criteria and the dermatoscopic features show a similar reproducibility.

These data suggest that dermatoscopy can be extensively used for the diagnosis of melanoma.

Acknowledgements

The Dermoscopic courses were held with grants from:
Course held in Padova, 04/16/99, Restiva Pharmaceutical Industry,
Course held in Milan, 05/07/99, Mipharm Pharmaceutical Industry,
Course held in Milan, 05/28/99, Mipharm Pharmaceutical Industry.

Article accepted on 6/3/01

REFERENCES

1. Rigel DS, Friedman RJ, Kopf AW. Early detection of malignant melanoma: the role of physician examination and self-examination of the skin. Cancer 1985; 35: 130-51.

2. Carli P, Borgognoni L, Reali UM, Giannotti B. Clinico-pathological features of small diameter malignant melanoma. Eur J Dermatol 1994; 4: 440-2.

3. Piepkorn M, Odland PB. Quality of Care in the Diagnosis of Melanoma and related Melanocytic Lesions. Arch Dermatol 1997; 133: 1393-6.

4. Miller M, Ackerman AB. How accurate are dermatologists in the diagnosis of melanoma? Degree of accuracy and implications. Arch Dermatol 1997; 133: 1393-6.

5. Stolz W, Braun-Falco O, Bilek P, Landthaler M, Cognetta AB. Color Atlas of Dermatoscopy. Blackwell Science Ltd 1994.

6. Stolz W, Riemann A, Cognetta A B, Pillet l, Abmayr W, Holzel D, Bilek P, Nachbar F, Landthaler M, Braun-Falco O. ABCD rule of dermatoscopy: a new practical method for early recognition of malignant Melanoma. Eur J Dermatol 1994; 4: 521-7.

7. Nachbar F, Stolz W, Merkle T. The ABCD rule of dermatoscopy: high prospective value in the diagnosis of doubtful melanocytic skin lesions. J Am Acad Dermatol 1994; 30: 551-9.

8. Dal Pozzo V, Benelli C. Atlante di Dermoscopia. Milan: Edra Publishing House, 1996.

9. Dal Pozzo V, Benelli C. Atlas of Dermoscopy. Milan: Edra Publishing House, 1997.

10. Dal Pozzo V, Roscetti E, Benelli C. The seven features for melanoma (7FFM) a new dermoscopic algorithm for the diagnosis of melanoma. Eur J Dermatol 1999; 9: 303-8.

11. Benelli C, Roscetti E, Dal Pozzo V, Gasparini G, Cavicchini S. The dermoscopic (7FFM) versus the clinical (ABCDE) diagnosis of melanoma. Eur J Dermatol 1999; 9: 470-6.

12. Thomas L, Tranchand P, Berard F, Secchi T, Colin C, Moulin G. Semiological value of ABCDE criteria in the diagnosis of cutaneous pigmented tumors. Dermatology 1998; 197: 11-7.

13. Bahmer FA, Fritsch P, Kreusch J, Pehamberger H, Rohrer C, Schindera I. Terminology in surface microscopy. J Am Acad Dermatol 1990; 23: 1159-62.

14. Nilles M, Boedeker RH, Schill WB. Surface microscopy of naevi and Melanomas-clues to Melanoma. Br J Dermatol 1994; 130: 349-55.

15. Kenet RO, Kang S, Kenet BJ, Fitzpatrick TB, Sober AJ, Barnhill RL. Clinical diagnosis of pigmented lesions using digital epiluminescence microscopy. Arch Dermatol 1993; 129: 157-74.

16. Steiner A, Pahemberger H, Binder M, Wolff K. Pigmented Spitz Nevi: Improvement of the diagnostic accuracy by epiluminescence microscopy. J Am Acad Dermatol 1992; 27: 697-701.

17. Armitage P, Berry G. Statistical methods in Medical Research. Oxford: Blackwell Scientific Publication Limited, 1994.

18. Dawson-Saunders B, Trapp RG. Basic & Clinical Biostatistics. Appleton & Lange, 1994.

19. Menzies SW, Crotty KA, Ingvar C, et al. An atlas of surface microscopy of pigmented skin lesions. Sidney, Australia: McGraw-Hill, 1996.

20. Menzies SW, Ingvar C, McCarty WH. A sensitivity and specificity analysis of the surface microscopy features of invasive Melanoma. Melanoma Res 1996; 6: 55-62.

21. Menzies SW, Ingvar C, Crotty KA, et al. Frequency and morphologic characteristics of invasive Melanomas lacking specific surface microscopic features. Arch Dermatol 1996; 132: 1178-82.

22. Argenziano G, Fabbroncini G, Carli P, De Giorgi V, Sammarco E, Delfino M. Epiluminescence microscopy for the diagnosis of doubtful melanocytic skin lesions. Arch Dermatol 1998; 134: 1563-70.

23. Binder M, Schwarz M, Winkler A, Steiner A, Kaider A, Wolff K, Pehamberger H. Epiluminescence Microscopy. Arch Dermatol 1995; 131: 286-91.

24. Binder M, Puespoeck-Schwarz M, Steiner A, Kittlewr H, Muellner M, Wolff K, Pehamberger H. Epiluminescence microscopy of small pigmented skin lesions: short-term formal training improves the diagnostic performance of dermatologists. J Am Acad Dermatol 1997; 36: 197-202.

25. Stanganelli I, Burroni M, Rafanelli S, Bucchi L. Intraobserver agreement in interpretation of digital epiluminescence microscopy. J Am Acad Dermatol 1995; 33: 584-9.

26. Stanganelli I, Bucchi L. Epiluminescence microscopy versus clinical evaluation of pigmented skin lesions: effects of operator's training on reproducibility and accuracy. Dermatology 1998; 196: 199-203.

27. Carli P, De Giorgi V, Naldi L, Dosi G on behalf of the Dermatoscopy Panel. Reliability and inter-observer agreement of dermatoscopic diagnosis of melanoma and melanocytic naevi. Eur J Cancer Prev 1998; 7: 397-402.

28. Benelli C, Roscetti E, Dal Pozzo V. Reproducibility of a dermoscopic method (7FFM) for the diagnosis of malignant melanoma. Eur J Dermatol 2000; 10: 110-4.

29. Argenziano G, Fabbroncini G, Carli P, De Giorgi V, Delfino M. Epiluminescence microscopy: criteria of cutaneous melanoma progression. J Am Acad Dermatol 1997; 37: 68-74.