you can not have a rule (folder rule) that triggers a dialog to the user. You need to implement a custom uploader that has to react to the duplicate file name error and trigger the dialog.
Thank you Afaust! You mean web script can't use popup method like "alert("xxxx is error!")" And I need to custom uploader, but I have no technical knowledge about it..