v23.1 - Notifications don't seem to work if they use properties where a value is not present

[MVannLeicester]

Problem

If a formatted notification uses a property for any part of it - recipient, subject, body, etc. - they don't get sent by the system and just error.  The file remains in the cache directory. 

Here's an example error that gets logged to Event Viewer when this happens:

[02/01/2024 15:23:27 -06:00 Error] An error occurred while loading the request with ID '60259'. Message: 'Unexpected character encountered while parsing value: Ā. Path '', line 0, position 0.'.

However, if I manually run the Unity Scheduler service using an EP3 application server, it sends.

I've eliminated everything except the one thing in common which is that the affected notifications depend on properties that should be getting set in Workflow.  In a couple of situations the value doesn't get set, and that's being addressed upstream.

Regular text notifications appear to be unaffected. It's only formatted ones.

EDIT: More context.  For some reason the system seems to think a recipient isn't there when it is.  Here's the Diagnostics:

Workflow Exception Report: Inner Exceptions 'True' Stack Trace 'True'Exception Level 0: Type 'Hyland.Workflow.TaskExecution.TaskExecutionException' Message 'Task list execution failed: Exception Type Hyland.Core.Distribution.NoRecipientsException: Cannot enqueue the email since the recipient list is empty.Action: [2100] Send notification: document received (Send Notification)'Exception Level 0: Stack Trace   at Hyland.Workflow.Core.TaskExecution.TaskExecutor.HandleExecuteOrResume(SuspendedTask resumeTask)   at Hyland.Workflow.Core.TaskExecution.TaskExecutor.Execute()   at Hyland.Workflow.Core.Scheduler.SchedulerTimer.ProcessItem(ISession session, IQueue queue, IWorkItem workitem, TaskExecutorSettings executorSettings, Int64 itemsprocessed, Boolean isfirst, Boolean islast)Exception Level 1: Type 'Hyland.Workflow.Tasks.TaskException' Message 'Exception Type Hyland.Core.Distribution.NoRecipientsException: Cannot enqueue the email since the recipient list is empty.Action: [2100] Send notification: document received (Send Notification)'Exception Level 1: Stack Trace   at Hyland.Workflow.Core.ActionAndRuleBase`1.HandleTaskExecutionException(IWorkflowTrace trace, Exception e)   at Hyland.Workflow.Core.ActionAndRuleBase`1.Execute(ISession session, IWorkItem item, ITaskExecutionArgs executionArgs)   at Hyland.Workflow.Core.TaskExecution.ActionExecutor.ExecuteOrResume(IAction action, ActionState actionState, SuspendedTask resumeTask, String resumeTaskState, ITaskExecutionArgs executionArgs)   at Hyland.Workflow.Core.TaskExecution.ActionExecutor.Execute(IAction action, TaskExecutorState executorState, ITaskExecutionArgs executionArgs)   at Hyland.Workflow.Core.TaskExecution.TaskListExecutor.ExecuteOrResumeImpl(ITaskList taskList, TaskListState taskListState, SuspendedTask resumeTask, ITaskExecutionArgs executionArgs)   at Hyland.Workflow.Core.TaskExecution.TaskListExecutor.InternalExecute(ITaskList taskList, TaskExecutorState executorState, ITaskExecutionArgs executionArgs)   at Hyland.Workflo…

However, in distributionqueue there are clearly recipients, and if I manually release the messages using EP3, they send fine (prefixes edited out for privacy):

The logic that eventually sends this notification is inefficient and will be rebuilt at some point here soon because it's doing a lot of unnecessary work.  But I'm still not sure why the EP3 application server can send it fine and the 23.1 application server cannot.

Question

Is there any known issue regarding Notifications requiring all properties to exist/have a value before sending?  I don't know if this was an issue in EP4, 5 or v22.

This is specifically 23.1.3.  I haven't yet tried 23.1.2.

[Edward_Hudson · Accepted solution]

Thanks for the reply, I have a feeling this might be related as well:

https://community.hyland.com/forum/threads/82474-parallel-upgrade-distribution-services-not-sending-...

[MVannLeicester]

@Edward Hudson I believe I've figured this out (theory).  The link is definitely related.  Really depends on how you have Unity Scheduler configured - I am currently doing some tweaks to see if it resolves the issue.

There are three connections to consider: the user's connection, the one that the Workflow Timer (Unity Scheduler) is using and the one that Distribution (Unity Scheduler) is using.

Some customers run everything through one Unity Scheduler (the default) - which limits their ability to do IPUP. 

I've always separated the services out so that I could control up and downgrade as well as being able to intervene for "runaway" executions and do monitoring for things like memory leaks for specific jobs.  So in my case I have at least 5-6 separate Unity Scheduler instances on the server.

I did find that one of the instances that runs that specific set of timers that create the distribution request, was not fully upgraded; some of the files still show as EP3.  So I'm going through to swap that service over to a fresh service folder to see if that resolves it for the most affected notifications.  If so, that's the key in my case, and I'll provide the findings here as an update.

But it also means that for anyone running a single Unity Scheduler instance for all tasks, they really can't do IPUP, and the link is on the right track with what broke.

Why this only affects Formatted notifications is still a mystery - because everything else has never had an issue.  It's only Formatted notifications.

[MVannLeicester · Accepted solution]

Confirmed:

For whatever reason, the Formatted (only) notifications are processed differently in the newer version.

If a notification was requested (Workflow Timer) by a pre 23.1 Unity Scheduler but then processed (Distribution) by a newer Unity Scheduler, the send fails.  Only for Formatted notifications and seemingly, when there's a property that isn't there.

This makes me think that the newer version is composing the outbound message completely differently than the older version.  But it's not clear why that only applies to Formatted notifications and not all of them.

The application server seems to have nothing to do with this specifically, but I did notice different behavior when changing the entry in Config.

So my thoughts:

If you're running everything through one Unity Scheduler, whatever version that's installed appears to need to be the same as the entry in Config.

If you're running multiple Unity Schedulers, any that are requesting (via Workflow Timer) Formatted notifications needs to be the same as the entry in Config.

[Robert_King5]

Today our Distribution Service was locked up.  This sql is usually executed to verify how many notifications are queued up before restarting.  After upgrading to Foundation 22, this SQL is not showing any results.  Has the table name changed?

select * from hsi.distributionqueue

[MVannLeicester]

Same table.  But be aware that this table only shows what the service didn't process.  

In EP3, if a notification was missing a recipient, it would still send it to basically nobody (so basically it "processed").

In 23.1, it errors and doesn't actually send.  Gets left in distributionqueue.

If you see recent results in distributionhist, they got processed, thus distributionqueue being empty.

[Can_Ergin]

Hi everyone,

I have replied here regarding this issue.