Static records limit in TaskEntityManager

[savanladani]

Hi,

I don't understand why there is 20000 max records limit in TaskEntityManager while fetching task list. why we can't pass dynamic pagination value to query?

// paging doesn't work for combining task instances and variables due to an outer join, so doing it in-memory
    if (taskQuery.getFirstResult() < 0 || taskQuery.getMaxResults() <= 0) {
      return Collections.EMPTY_LIST;
    }
    
    int firstResult = taskQuery.getFirstResult();
    int maxResults = taskQuery.getMaxResults();
    
    // setting max results, limit to 20000 results for performance reasons
    taskQuery.setMaxResults(20000);
    taskQuery.setFirstResult(0);
    
    List<Task> instanceList = getDbSqlSession().selectListWithRawParameterWithoutFilter(query, taskQuery, taskQuery.getFirstResult(), taskQuery.getMaxResults());
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[jbarrez]

This is for fetching variables only when doing task queries. 20K should be enough (you'll have performance issues otherwise).

But you are right, it should be configurable.

[arandall]

Hey there,

We hit this issue with historic and active tasks in an environment were we had created a lot of tasks with variables. The highest index you can get back decreases the start index you can pull back and creates unexpected REST responses.

Max results returned = 20,000 / (average task variables + average process variables)

A static/configurable option here wouldn't help if you want to go past the 20,000 limit it will only return the count.
eg. <blockcode>
{"taskVariables" : [], "start" : 20001, "size" : 10, "includeTaskLocalVariables" : true}
</blockcode>

I have a fix for this which uses the same behaviour as includeTaskLocalVariables":false and then makes calls out to get the Task and Process variables for each task returned.
It is available on https://github.com/dgit-hub/Activiti/commit/7452c4555c35a3cf7bbdcda9b0aaad4df1760f11 and should merge cleanly to any of the 5.x releases.

When querying with "start" > 10,000 the added db hit of the extra queries is still significantly faster than the memory filtering overhead of processing 10,000 results.

After applying the fix it looks like Executions and Process Instances will suffer from the same issue. It wouldn't take much to extend the fix to include them too. <blockcode>
engine/impl/persistence/entity/ExecutionEntityManager.java:    executionQuery.setMaxResults(20000);
engine/impl/persistence/entity/HistoricProcessInstanceEntityManager.java:      historicProcessInstanceQuery.setMaxResults(20000);
</blockcode>

[jbarrez]

would this solve your problem (just been merged to the engine: https://github.com/Activiti/Activiti/pull/891)?