2007 Fiscal Year Final Research Report Summary
Aresearch of statistical properties of singular models such as a multi-layer perceptron
| Project/Area Number |
18500171
|
|---|
| Research Category |
Grant-in-Aid for Scientific Research (C)
|
| Allocation Type | Single-year Grants |
|---|
| Section | 一般 |
|---|
| Research Field |
Sensitivity informatics/Soft computing
|
|---|
| Research Institution | Mie University |
|---|
Principal Investigator |
HAGIWARA Katsuyuki Mie University, Faculty of Education, Associate Prof. (60273348)
|
|---|
| Project Period (FY) |
2006 – 2007
|
| Keywords | singular model / multi-laver perceptron / model selection / training error / generalization error / shrinkage estimator / over-fitting / extreme value theory |
|---|
| Research Abstract |
The multi-layer perceptron is known to be a singular model in which the Fisher information matrix can be singular in some cases. The singularity comes from parametric basis functions by which the shapes of basis functions vary. To reveal statistical properties of a singular model, in this research, we focus on the variable basis functions. We derived a upper bound of the degree of over-fitting under a restriction on basis function outputs. For example, the restriction can be achieved by restricting an input weight space in a multi-layer perceptron. By applying this result, we showed that, for a regression using one Gaussian unit, a very small value is frequently obtained for a width parameter in training. On the other hand, we derived the expectations of the training error and generalization error of learning machine in which basis functions which are chosen from a finite set of orthogonal functions. This clarifies that AIC type model selection criteria for machines with variable basis functions need an information of a target function. We solve this problem by applying a shrinkage method. From a viewpoint of variable basis functions, furthermore, we proposed a shrinkage method for a nonparametric regression. The method produces a machine with low generalization error in less computational time. The results obtained in this research helps for clarifying statistical properties of a machine with variable basis functions, thus, a singular model such as multi-layer perceptrons.
|
