Recently, as available computational power has increased dramatically, patch-/pixel-based machine learning (PML) [114] emerged in medical image processing/analysis which uses values in image patches (i.e., pixel values and/or features calculated from the image patches), instead of features calculated from segmented regions, as input information; thus, segmentation is not required. PML has been used in the classification of the detected lesion candidates in CADe schemes. Recently in the computer-vision field, deep learning and deep neural networks [11, 54] have been attracting researchers’ attentions as a breakthrough technology in computer vision. Deep learning and deep neural networks use PML architecture.

PMLs were first developed for tasks in medical image processing/analysis and computer vision. There are three classes of PMLs: (1) neural filters [126, 129] including neural edge enhancers [128, 130], (2) convolution neural networks (NNs) [62, 68, 69, 71, 73, 88, 100] including shift-invariant NNs [153, 171, 172], and (3) massive-training artificial neural networks (MTANNs) [89, 111, 120, 121, 140] including multiple MTANNs [3, 121, 126, 129, 131, 134], a mixture of expert MTANNs [132, 139], a multiresolution MTANN [120], a Laplacian eigenfunction MTANN (LAP-MTANN) [141], and a massive-training support vector regression (MTSVR) [159]. The class of neural filters was used for image-processing tasks such as edge-preserving noise reduction in fluoroscopy, radiographs and other digital pictures [126, 129], edge enhancement from noisy images [128], and enhancement of subjective edges traced by a physician in cardiac images [130]. The class of convolution NNs was applied to classification tasks such as false-positive (FP) reduction in CAD schemes for the detection of lung nodules in chest radiographs (CXRs) [68, 69, 73], FP reduction in CAD schemes for the detection of microcalcifications [71] and masses [100] in mammography, face recognition [62], and character recognition [88]. The class of MTANNs was used for classification, such as FP reduction in CAD schemes for the detection of lung nodules in CXR [134] and thoracic CT [3, 65, 121], distinction between benign and malignant lung nodules in CT [131], and FP reduction in a CAD scheme for polyp detection in CT colonography [132, 139–141, 159]. The MTANNs were also applied to pattern enhancement and suppression such as separation of bones from soft tissue in CXR [19, 89, 120], and enhancement of lung nodules in CT [111]. There are other PML approaches in the literature. An iterative, pixel-based, supervised, statistical classification method called iterated contextual pixel classification has been proposed for segmenting posterior ribs in CXR [74]. A pixel-based, supervised regression filtering technique called filter learning has been proposed for separation ribs from soft tissue in CXR [75].

An MTANN was developed by extension of neural filters to accommodate various pattern-recognition tasks [121]. A two-dimensional (2D) MTANN was first developed for distinguishing a specific opacity from other opacities in 2D images [121]. The 2D MTANN was applied to reduction of FPs in computerized detection of lung nodules on 2D CT images in a slice-by-slice way [3, 65, 121] and in CXR [134], the separation of ribs from soft tissue in CXR [89, 119, 120], and the distinction between benign and malignant lung nodules on 2D CT slices [131]. For processing of three-dimensional (3D) volume data, a 3D MTANN was developed by extending the structure of the 2D MTANN, and it was applied to 3D CT colonography data [132, 139–141, 159] in CADe of polyps.

The generalized architecture of an MTANN is shown in Fig. 7.3. An MTANN consists of an ML model (typically a regression model) such as a linear-output ANN regression model [128] and a support vector regression model [159], which is capable of operating on pixel/voxel data directly [128]. The linear-output ANN regression model uses a linear function instead of a sigmoid function as the activation function of the output-layer unit because the characteristics of an ANN were improved significantly with a linear function when applied to the continuous mapping of values in image processing [128]. Note that the activation functions of the hidden layer units are a sigmoid function for nonlinear processing, and those of the input layer units an identity function, as usual. The pixel/voxel values of the input images/volumes may be normalized from 0 to 1. The input to the MTANN consists of pixel/voxel values in a subregion/subvolume (image patch or local window), R, extracted from an input image/volume. The output of the MTANN is a continuous scalar value, which is generally associated with the center voxel in the subregion (image patch), and is represented by

where x, y, and z or t are the coordinate indices, ML(·) is the output of the ML model, and I(x, y, z or t) is a pixel/voxel value of the input image/volume. A three-layer structure may be selected as the structure of the ANN, because it has been proved that any continuous mapping can be approximated by a three-layer ANN [10, 55]. More layers can be used for efficient solving of a complicated problem. The structure of input units and the number of hidden units in the ANN may be designed by the use of sensitivity-based unit-pruning methods [110, 125]. Other ML models such as support vector regression [145, 146] can be used as a core part of the MTANN. ML regression models rather than ML classification models would be suited for the MTANN framework, because the output of the MTANN are continuous scalar values (as opposed to nominal categories or classes, e.g., 0 or 1). The entire output image/volume is obtained by scanning with the input subvolume (local window) of the MTANN on the entire input image/volume. The input subregion/subvolume and the scanning with the MTANN can be analogous to the kernel of a convolution filter and the convolutional operation of the filter, respectively.

The MTANN is trained with input images/volumes and the corresponding “teaching” (designed) images/volumes for enhancement of a specific pattern and suppression of other patterns in images/volumes. The “teaching” images/volumes are ideal or desired images for the corresponding input images/volumes. For enhancement of lesions and suppression of nonlesions, the teaching volume contains a map for the “likelihood of being lesions,” represented by
To enrich the training samples, a training region, R _T , extracted from the input images is divided pixel by pixel into a large number of overlapping subregions. Single pixels are extracted from the corresponding teaching images as teaching values. The MTANN is massively trained by the use of each of a large number of input subregions (image patches) together with each of the corresponding teaching single pixels, hence the term “massive-training ANN.” The error to be minimized by training of the MTANN is represented by
where c is a training case number, O _c is the output of the MTANN for the cth case, T _c is the teaching value for the MTANN for the cth case, and P is the number of total training voxels in the training region for the MTANN, R _T . The expert 3D MTANN is trained by a linear-output back-propagation (BP) algorithm [128] which was derived for the linear-output ANN model by the use of the generalized delta rule [98]. After training, the MTANN is expected to output the highest value when a lesion is located at the center of the subregion of the MTANN, a lower value as the distance from the subregion center increases, and zero when the input subregion contains a nonlesion.

In 1994, Giger et al. [38] developed a CADe scheme for the detection of lung nodules in CT based on comparison of geometric features. They applied their CADe scheme to a database of thick-slice diagnostic CT scans. In 1999, Armato et al. [5, 6] extended the method to include 3D feature analysis, a rule-based scheme, and LDA for classification. They tested their CADe scheme with a database of thick-slice (10 mm) diagnostic CT scans. They achieved a sensitivity of 70 % with 42.2 FPs per case in a leave-one-out cross-validation test. Gurcan et al. [45] employed a similar approach, i.e., a rule-based scheme based on 2D and 3D features, followed by LDA for classification. They achieved a sensitivity of 84 % with 74.4 FPs per case for a database of thick-slice (2.5–5 mm, mostly 5 mm) diagnostic CT scans in a leave-one-out test. Lee et al. [63] employed a simpler approach which is a rule-based scheme based on 13 features for classification. They achieved a sensitivity of 72 % with 30.6 FPs per case for a database of thick-slice (10 mm) diagnostic CT scans.

Suzuki et al. [121] developed a PML technique called an MTANN for reduction of a single source of FPs and a multiple MTANN scheme for reduction of multiple sources of FPs that had not been removed by LDA. They achieved a sensitivity of 80.3 % with 4.8 FPs per case for a database of thick-slice (10 mm) screening LDCT scans of 63 patients with 71 nodules with solid, part-solid, and nonsolid patterns, including 66 cancers in a validation test. This MTANN approach did not require a large number of training cases: the MTANN was able to be trained with ten positive and ten negative cases [17, 99, 123], whereas feature-based classifiers generally require 400–800 training cases [17, 99, 123]. Arimura et al. [3] employed a rule-based scheme followed by LDA or by the MTANN [121] for classification. They tested their scheme with a database of 106 thick-slice (10 mm) screening LDCT scans of 73 patients with 109 cancers, and they achieved a sensitivity of 83 % with 5.8 FPs per case in a validation test (or a leave-one-patient-out test for LDA). Farag et al. [32] developed a template-modeling approach that uses level sets for classification. They achieved a sensitivity of 93.3 % with an FP rate of 3.4 % for a database of thin-slice screening LDCT scans of 16 patients with 119 nodules and 34 normal patients. Ge et al. [36] incorporated 3D-gradient field descriptors and ellipsoid features in LDA for classification. They employed Wilks’ lambda stepwise feature selection for selecting features before the LDA classification. They achieved a sensitivity of 80 % with 14.7 FPs per case for a database of 82 thin-slice CT scans of 56 patients with 116 solid nodules in a leave-one-patient-out test. Matsumoto et al. [79] employed LDA with eight features for classification. They achieved a sensitivity of 90 % with 64.1 FPs per case for a database of thick-slice diagnostic CT scans of five patients with 50 nodules in a leave-one-out test.

Yuan et al. [170] tested a commercially available CADe system (ImageChecker CT, LN-1000, by R2 Technology, Sunnyvale, CA; Hologic now). They achieved a sensitivity of 73 % with 3.2 FPs per case for a database of thin-slice (1.25 mm) CT scans of 150 patients with 628 nodules in an independent test. Pu et al. [93] developed a scoring method based on the similarity distance of medial axis-like shapes for classification. They achieved a sensitivity of 81.5 % with 6.5 FPs per case for a database of thin-slice screening CT scans of 52 patients with 184 nodules, including 16 nonsolid nodules. Retico et al. [94] used a voxel-based neural approach (i.e., a class of the MTANN approach) with pixel values in a subvolume as input for classification. They obtained sensitivities of 80–85 % with 10–13 FPs per case for a database of thin-slice screening CT scans of 39 patients with 102 nodules. Ye et al. [163] used a rule-based scheme followed by a weighted SVM for classification. They achieved a sensitivity of 90.2 % with 8.2 FPs per case for a database of thin-slice screening CT scans of 54 patients with 118 nodules including 17 nonsolid nodules in an independent test. Golosio et al. [44] used a fixed-topology ANN for classification, and they evaluated their CADe scheme with a publicly available database from the Lung Image Database Consortium (LIDC) [8]. They achieved a sensitivity of 79 % with four FPs per case for a database of thin-slice CT scans of 83 patients with 148 nodules that one radiologist detected from an LIDC database in an independent test.

Murphy et al. [86] used a k-nearest-neighbor classifier with features selected from 135 features for classification. They achieved a sensitivity of 80 with 4.2 FPs per case for a large database of thin-slice screening CT scans of 813 patients with 1,525 nodules in an independent test. Tan et al. [143] developed a feature-selective classifier based on a genetic algorithm and ANNs for classification. They achieved a sensitivity of 87.5 % with four FPs per case for a database of thin-slice CT scans of 125 patients with 80 nodules that four radiologists agreed from the LIDC database in an independent test. Messay et al. [80] developed a sequential forward selection process for selecting the optimum features for LDA and quadratic discriminant analysis (QDA). They obtained a sensitivity of 83 % with three FPs per case for a database of thin-slice CT scans of 84 patients with 143 nodules from the LIDC database in a sevenfold cross-validation test. Riccardi et al. [95] used a heuristic approach based on geometric features, followed by an SVM for classification. They achieved a sensitivity of 71 % with 6.5 FPs per case for a database of thin-slice CT scans of 154 patients with 117 nodules that four radiologists agreed on from the LIDC database in a twofold cross-validation test.

Thus, various approaches have been proposed for CADe schemes for lung nodules in CT. Sensitivities for the detection of lung nodules in CT range from 70 to 95 %, with from a few to 70 FPs per case. Major sources of FPs are various-sized lung vessels. Major sources of false negatives are ground-glass nodules, nodules attached to vessels, and nodules attached to the lung wall (i.e., juxtapleural nodules). Ground-glass nodules are difficult to detect, because they are subtle, are of low contrast, and have ill-defined boundaries. The MTANN approach was able to enhance and thus detect ground-glass nodules [121]. The cause of false negatives due to vessel-attached nodules and juxtapleural nodules is mis-segmentation and thus inaccurate feature calculation. Because the MTANN approach does not require segmentation or feature calculation, it was able to detect such nodules [121].

Figure 7.4a illustrates an axial slice of a CT scan of the lungs with a lung cancer. The lung cancer on the CT image is the target that we want to detect with a CADe scheme. As shown in the flowchart in Fig. 7.1, the first step in a CADe scheme is segmentation of the organ of interest, in this case, the lungs. For a high-contrast image with a stable gray scale over different patients like the lung CT image, thresholding often works. To avoid missing nodules attached to the lung walls, mathematical morphology operations are often performed. Figure 7.4b illustrates lung segmentation by simple thresholding followed by mathematical morphology filtering.

To improve the performance of CADe schemes, an optional step of enhancement of lesions is sometimes employed. Suzuki [111] developed a supervised “lesion enhancement” filter based on an MTANN for enhancing lesions and suppressing nonlesions in medical images. Figure 7.5b illustrates the enhancement of a lung nodule in a CT image by means of a trained MTANN lesion-enhancement filter for the original axial CT slice shown in Fig. 7.5a. In the output image, the lung nodule in the original CT image is enhanced, while normal structures such as lung vessels are suppressed substantially. Figure 7.5c shows the detection and segmentation result for the lung nodule by using simple thresholding followed by removal of small regions. After thresholding, connected-component labeling [49, 50, 127] was performed to calculate the area of each isolated region (i.e., connected component). By removing small regions, the lung nodule was detected correctly with no FP detection. By the use of the MTANN lesion-enhancement filter, the performance of the initial nodule candidate detection step was substantially improved from a 96 % sensitivity with 19.3 FPs per section to a 97 % sensitivity with 6.7 FPs per section.

Morphologic and gray-level-based features such as contract, area, and circularity were calculated from the segmented nodule candidates. The extracted features were then inputted to a classifier (feature-based ML) to classify the candidates into nodules or non-nodules. At this stage, there were a lot of FPs (non-nodules) that the classifier had not been able to distinguish from nodules.

To reduce remaining FPs, Suzuki et al. developed an FP reduction technique based on MTANNs [121]. The architecture of the MTANN for FP reduction is shown in Fig. 7.6. For enhancement of nodules (i.e., true positives) and suppression of non-nodules (i.e., FPs) on CT images, the teaching image contains a distribution of values that represent the “likelihood of being a nodule.” For example, the teaching volume contains a 3D Gaussian distribution with standard deviation σ _T for a lesion and zero (i.e., completely dark) for nonlesions, as illustrated in Fig. 7.6. This distribution represents the “likelihood of being a lesion”:

A scoring method is used for combining of output voxels from the trained MTANNs, as illustrated in Fig. 7.7. A score for a given region-of-interest (ROI) from the MTANN is defined as

where
is a 3D Gaussian weighting function with standard deviation σ, and with its center corresponding to the center of the volume for evaluation, R _E , and O is the output image of the trained MTANN, where its center corresponds to the center of R _E . The use of the 3D Gaussian weighting function allows us to combine the responses (outputs) of a trained MTANN as a 3D distribution. A 3D Gaussian function is used for scoring, because the output of a trained MTANN is expected to be similar to the 3D Gaussian distribution used in the teaching images. This score represents the weighted sum of the estimates for the likelihood that the ROI (lesion candidate) contains a lesion near the center, i.e., a higher score would indicate a lesion, and a lower score would indicate a nonlesion. Thresholding is then performed on the scores for distinction between lesions and nonlesions.

The MTANNs were trained to enhance lung nodules and suppress various types of FPs (i.e., non-nodules) such as lung vessels. Figure 7.8 shows the results of the enhancement of various lung nodules such as nonsolid (ground-glass), part-solid (mixed-ground-glass), and solid nodules (a) and those of the suppression of various-sized lung vessels (b). Figure 7.9 shows a free-response receiver operating characteristic (FROC) curve [13] indicating the performance of the trained MTANNs in the CADe scheme. With the MTANNs, the specificity of the CADe scheme was improved from 0.98 to 0.18 FPs per case without sacrificing the original sensitivity of 80.3 %.

Figure 7.10 shows an example of CADe outputs on a CT image of the lungs. A CADe scheme detected a lung nodule correctly with one FP which was a branch of the lung vessels.