4.6 Nongradient-Based Training: Simplex Method

The simplex method [43] is a representative method of the nongradient-based training techniques, which only require function evaluations. It uses theerror function E _{T r}( w), without the derivative information . We define a simplex formed by N _w + 1 points in w-space, where N _w is the total number of neural network weights. The N _w + 1 points, w ⁽ⁱ⁾, i = 1, 2, , N _w + 1, are the vertices of the simplex.

The training error E _{T r}( w ⁽ⁱ⁾) is computed at all vertices of the simplex. Comparing these values, the worst point ( w ^(h)) (i.e., E _{T r} ( w ^(h)) > E _{T r}( w (i)) for all i, i ? h), can be selected. The centroid of all the points excluding w ^(h) (see Figure 4.26) can be computed as

(4.58)

Figure 4.26: A simplex with three vertices in a two-dimensional w-space. Corresponding centroid and reflection points are also shown.

In the simplex method, optimization means searching for new points where the training error is minimal. There are three operations used to create a new point namely the reflection, the expansion, and the contraction. The reflection operation is given by

(4.59)

If w ^(r) turns out to be the best point of the present simplex, then use the expansion operation

(4.60)

Otherwise,...