You can avoid this problem by naming one or more people as your “alternate” Agent. The alternate can act if your Agent is unable to assume the role or chooses not to. If you have not named an alternate you should consider making a new Personal Directive.
If all of the possible Agents named in your Personal Directive are unable or unwilling to act, your Personal Directive will cease to have effect.