A transactional cache being full means that some code / logic has been loading a high number of elements, which were previously not cached, from the DB within a single transaction. Transactional cache sizes are limited for a reason (to avoid runaway resource consumption and overly expensive cache update processing at commit time), and once exceeded will cause the global shared cache to be cleared instead of updated, to ensure there are no stale data entries.
It is generally NOT a good idea to just increase the maxItems configuration without really understanding the cause, i.e. which piece of code / logic is causing an excessive load of data in a single transaction. Ideally, such cases should be identified and either be rectified via configuration / code change. One example would be a background job processing data which should be configured to use smaller batches, e.g. synchronising users from LDAP/AD or activity feed processing.
In your example, you aren't actually changing the size of the transactional cache, because you are using the wrong properties. The correct properties have a slightly different name pattern, ending in *.tx.maxItems
