Should I set IsReferenceMode to True when I use custom sessionSecurityTokenCache?

If I cached the Security Token in a custom sessionSecurityTokenCache, should I use IsReferenceMode?

Because without setting IsReferenceMode to true, the FedAuth cookie is still large although the cache service is being called!