Coefficient of determination in nonlinear signal processing

Edward R. Dougherty, Seungchan Kim, Yidong Chen

Research output: Contribution to journalArticlepeer-review

169 Scopus citations


For statistical design of an optimal filter, it is probabilistically advantageous to employ a large number of observation random variables; however, estimation error increases with the number of variables, so that variables not contributing to the determination of the target variable can have a detrimental effect. In linear filtering, determination involves the correlation coefficients among the input and target variables. This paper discusses use of the more general coefficient of determination in nonlinear filtering. The determination coefficient is defined in accordance with the degree to which a filter estimates a target variable beyond the degree to which the target variable is estimated by its mean. Filter constraint decreases the coefficient, but it also decreases estimation error in filter design. Because situations in which the sample is relatively small in comparison with the number of observation variables are of salient interest, estimation of the determination coefficient is considered in detail. One may be unable to obtain a good estimate of an optimal filter, but can nonetheless use rough estimates of the coefficient to find useful sets of observation variables. Since minimal-error estimation underlies determination, this material is at the interface of signal processing, computational learning, and pattern recognition. Several signal-processing factors impact application: the signal model, morphological operator representation, and desirable operator properties. In particular, the paper addresses the VC dimension of increasing operators in terms of their morphological kernel/basis representations. Two applications are considered: window size for restoring degraded binary images; finding sets of genes that have significant predictive capability relative to target genes in genomic regulation.

Original languageEnglish (US)
Pages (from-to)2219-2235
Number of pages17
JournalSignal Processing
Issue number10
StatePublished - Oct 2000
Externally publishedYes

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Software
  • Signal Processing
  • Computer Vision and Pattern Recognition
  • Electrical and Electronic Engineering


Dive into the research topics of 'Coefficient of determination in nonlinear signal processing'. Together they form a unique fingerprint.

Cite this