Mystery solved:

CHARMM formula for the volume of the trunc.cuboctahedron:
V = 4*sqrt(3)/9 * a**3
'a' is the distance between two hexagonal faces and not as I initially thought the edge length.

'b' is the length between two quadratic faces (truncated cube length).
b = a*sqrt(4/3.)
'a' can also be seen as
a = b*sqrt(3) / 2
which is half of the space diagonal.

Unfortunately, I haven't found a proof for the CHARMM formula but numerical results are equal to results of V=8*sqrt(2)*a**3