Hello

Just seems to happen when using inPrivate browsing yet all other websites I go to show their favicon just fine on all pages using inPrivate browsing.

Ok.

I did not change the theme, it says in the header element on affected pages

<link href="http://mydomain.com/themes/default/images/favicon.ico" rel="shortcut icon">

Ok, so you use the original theme. It's the original favicon.ico too or did you changed that file?

This is the original one (in Firefox):

Tschö, Auge