Prince Harry felt a sense of 'wariness' as he met with crowds after royal snub