On stability analysis in shape optimisation : critical shapes for Neumann problem

The stability issue of critical shapes for shape optimization problems with the state function given by a solution to the Neumann problem for the Laplace equation is considered. To this end, the properties of the shape Hessian evaluated at critical shapes are analysed. First, it is proved that the stability cannot be expected for the model problem. Then, the new estimates for the shape Hessian are derived in order to overcome the classical two norms-discrepancy well know in control problems, Malanowski (2001). In the context of shape optimization, the situation is similar compared to control problems, actually, the shape Hessian can be coercive only in the norm strictly weaker with respect to the norm of the second order differentiability of the shape functional. In addition, it is shown that an appropriate regularization makes possible the stability of critical shapes.