### Abstract

For statistical design of an optimal filter, it is probabilistically advantageous to employ a large number of observation random variables; however, estimation error increases with the number of variables, so that variables not contributing to the determination of the target variable can have a detrimental effect. In linear filtering, determination involves the correlation coefficients among the input and target variables. This paper discusses use of the more general coefficient of determination in nonlinear filtering. The determination coefficient is defined in accordance with the degree to which a filter estimates a target variable beyond the degree to which the target variable is estimated by its mean. Filter constraint decreases the coefficient, but it also decreases estimation error in filter design. Because situations in which the sample is relatively small in comparison with the number of observation variables are of salient interest, estimation of the determination coefficient is considered in detail. One may be unable to obtain a good estimate of an optimal filter, but can nonetheless use rough estimates of the coefficient to find useful sets of observation variables. Since minimal-error estimation underlies determination, this material is at the interface of signal processing, computational learning, and pattern recognition. Several signal-processing factors impact application: the signal model, morphological operator representation, and desirable operator properties. In particular, the paper addresses the VC dimension of increasing operators in terms of their morphological kernel/basis representations. Two applications are considered: window size for restoring degraded binary images; finding sets of genes that have significant predictive capability relative to target genes in genomic regulation.

Original language | English (US) |
---|---|

Pages (from-to) | 2219-2235 |

Number of pages | 17 |

Journal | Signal Processing |

Volume | 80 |

Issue number | 10 |

DOIs | |

State | Published - Oct 2000 |

Externally published | Yes |

### Fingerprint

### ASJC Scopus subject areas

- Signal Processing
- Electrical and Electronic Engineering

### Cite this

*Signal Processing*,

*80*(10), 2219-2235. https://doi.org/10.1016/S0165-1684(00)00079-7

**Coefficient of determination in nonlinear signal processing.** / Dougherty, Edward R.; Kim, Seungchan; Chen, Yidong.

Research output: Contribution to journal › Article

*Signal Processing*, vol. 80, no. 10, pp. 2219-2235. https://doi.org/10.1016/S0165-1684(00)00079-7

}

TY - JOUR

T1 - Coefficient of determination in nonlinear signal processing

AU - Dougherty, Edward R.

AU - Kim, Seungchan

AU - Chen, Yidong

PY - 2000/10

Y1 - 2000/10

N2 - For statistical design of an optimal filter, it is probabilistically advantageous to employ a large number of observation random variables; however, estimation error increases with the number of variables, so that variables not contributing to the determination of the target variable can have a detrimental effect. In linear filtering, determination involves the correlation coefficients among the input and target variables. This paper discusses use of the more general coefficient of determination in nonlinear filtering. The determination coefficient is defined in accordance with the degree to which a filter estimates a target variable beyond the degree to which the target variable is estimated by its mean. Filter constraint decreases the coefficient, but it also decreases estimation error in filter design. Because situations in which the sample is relatively small in comparison with the number of observation variables are of salient interest, estimation of the determination coefficient is considered in detail. One may be unable to obtain a good estimate of an optimal filter, but can nonetheless use rough estimates of the coefficient to find useful sets of observation variables. Since minimal-error estimation underlies determination, this material is at the interface of signal processing, computational learning, and pattern recognition. Several signal-processing factors impact application: the signal model, morphological operator representation, and desirable operator properties. In particular, the paper addresses the VC dimension of increasing operators in terms of their morphological kernel/basis representations. Two applications are considered: window size for restoring degraded binary images; finding sets of genes that have significant predictive capability relative to target genes in genomic regulation.

AB - For statistical design of an optimal filter, it is probabilistically advantageous to employ a large number of observation random variables; however, estimation error increases with the number of variables, so that variables not contributing to the determination of the target variable can have a detrimental effect. In linear filtering, determination involves the correlation coefficients among the input and target variables. This paper discusses use of the more general coefficient of determination in nonlinear filtering. The determination coefficient is defined in accordance with the degree to which a filter estimates a target variable beyond the degree to which the target variable is estimated by its mean. Filter constraint decreases the coefficient, but it also decreases estimation error in filter design. Because situations in which the sample is relatively small in comparison with the number of observation variables are of salient interest, estimation of the determination coefficient is considered in detail. One may be unable to obtain a good estimate of an optimal filter, but can nonetheless use rough estimates of the coefficient to find useful sets of observation variables. Since minimal-error estimation underlies determination, this material is at the interface of signal processing, computational learning, and pattern recognition. Several signal-processing factors impact application: the signal model, morphological operator representation, and desirable operator properties. In particular, the paper addresses the VC dimension of increasing operators in terms of their morphological kernel/basis representations. Two applications are considered: window size for restoring degraded binary images; finding sets of genes that have significant predictive capability relative to target genes in genomic regulation.

UR - http://www.scopus.com/inward/record.url?scp=0034302007&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0034302007&partnerID=8YFLogxK

U2 - 10.1016/S0165-1684(00)00079-7

DO - 10.1016/S0165-1684(00)00079-7

M3 - Article

AN - SCOPUS:0034302007

VL - 80

SP - 2219

EP - 2235

JO - Signal Processing

JF - Signal Processing

SN - 0165-1684

IS - 10

ER -