This error message generally means that you need to resize your MTA character set internal tables and then rebuild the compiled character set tables with the following commands:
imsimta chbuild -noimage -maximum -option imsimta chbuild |
Verify that nothing else needs to be recompiled or restarted before making this change. Refer to imsimta chbuild in Sun Java System Messaging Server 6.3 Administration Reference for more information on imsimta chbuild.