You may wish to try XLARGE instead, as the memory allocated to static arrays for XXLARGE may be interfering with your need for dynamic memory for the calculation.
Another option is to count the number of VEHEAP> statements in the output, and see if you can reduce them via NBSCAL and IMSCAL (see nbonds.doc). Another way to limit VEHEAP> statements is to pre-allocate a large memory block via GETHEAP (test.doc). (Each VEHEAP> statement represents a memory allocation; the problem is that unused memory blocks from earlier allocations count against the program total.)
You could also try reducing CUTNB (keep CTOFNB and CTONNB the same), and increasing INBFRQ (if not -1; likewise for CUTIM/IMGFRQ).