If you have followed the special steps in our documentation for configuring memory debugging on AIX 5.3 and you are still having problems you may be missing some required patches. On AIX 5.3 the memory debugger requires that the fixes for APARs IY66938 and IY78436 are installed on the machine. The minimum version of fileset bos.rte.libc is 5.3.0.41, the instfix command should tell you that these fixes are installed on the system:

% instfix -k IY66938 -i -v
IY66938 Abstract: user-defined malloc does not work in AIX 53

Fileset bos.adt.prof:5.3.0.30 is applied on the system.
Fileset bos.rte.libc:5.3.0.30 is applied on the system.

% instfix -k IY78436 -i -v
IY78436 Abstract: User Defined MALLOC doesnt work with _malloc_user_defined_

Fileset bos.adt.prof:5.3.0.41 is applied on the system.
Fileset bos.rte.libc:5.3.0.41 is applied on the system.