When does a smooth $\mathrm{SO}(n)$-action extend to an $\mathrm{O}(n)$-action?

$\def\SU{\text{SU}}\def\SO{\text{SO}}\def\Id{\text{Id}}\def\ZZ{\mathbb{Z}}\def\Spin{\text{Spin}}\def\Pin{\text{Pin}}$I have a counter-example for $n \equiv 0 \bmod 4$, and a strategy for general even $n$. In particular, I have also found an example when $n=10$ using this strategy, so it is not just about the value of $n \bmod 4$.

Suppose that we can find a Lie subgroup (not necessarily connected) $H$ of $\SO(n)$ such the normalizer of $H$ in $O(n)$ is contained in $\SO(n)$. Then I claim that the action of $\SO(n)$ on $\SO(n)/H$ doesn't extend to $O(n)$.

Indeed, since the action of $\SO(n)$ on $\SO(n)/H$ is transtive, the extended action would also have to be transitive. Let $H'$ be the $O(n)$ stabilizer of the coset $eH$. Then we would have to have $[H':H] = 2$, so $H$ would have to be normal in $H'$. In particular, $H'$ would have to be contained in $N(H)$. But, by hypothesis, $N(H) \subseteq \SO(n)$.

Now, suppose that $n = 2m \equiv 0 \bmod 4$. Let $J$ be the block diagonal matrix, made up of $m$ blocks of the form $\begin{bmatrix} 0&-1 \\ 1&0 \end{bmatrix}$. I claim that $H = \langle J \rangle$ has the desired property.

The centarlizer of $J$ in $\text{GL}_n(\mathbb{R})$ is $\text{GL}_m(\mathbb{C})$ and, if $g \in \text{GL}_m(\mathbb{C})$ has determinant $z$ as a complex matrix, then it has determinant $|z|^2 > 0$ as a real matrix. So the centralizer of $J$ within $O(n)$ lies in $\SO(n)$.

We want the normalizer, not the centralizer, so we must also consider matrices which conjugate $J$ to $J^{-1}$. It is enough to check that one such matrix has determinant $1$. Indeed, $J^{-1} = B J B^{-1}$ where $B$ is the block diagonal matrix, made up of $m$ blocks of the form $\begin{bmatrix} 0&1 \\ 1&0 \end{bmatrix}$. Using that $m \equiv 0 \bmod 2$, we have $\det B = 1$.

Here is the $n=10$ example. Let $W$ be the $10$-dimensional representation of $S_6$ labeled as "second exterior power of standard" in this character table. Like all representations of the symmetric group, this is defined over $\mathbb{R}$, so we get an injection $S_6 \to O(10)$.

We claim that $S_6$ lands in $\SO(10)$. We know that $A_6$ lands in $\SO(10)$, so it is enough to compute the determinant of a transposition. According to this character table, the transposition has trace $2$, so it must have eigenvlues $1$ and $-1$ with multiplicities $6$ and $4$ respectively, so the determinant is $(-1)^4=1$, and $S_6$ lands in $\SO(10)$. We also note for future reference that $W$ is not isomorphic to its image under the outer automorphism of $S_6$, since this automorphism switches the conjugacy classes $(12)$ and $(12)(34)(56)$, and those entries in the character table are not equal in row $W$.

Now, suppose that $g \in O(10)$ normalizes $S_6$. Then $g$ must act on $S_6$ by an automorphism. This automorphism is not outer, or else $W$ would be isomorphic (by $g$) to its image under the outer automorphism. So $g$ acts on $S_6$ by an inner automorphism, say conjugation by $h$ and, replacing $g$ by $gh^{-1}$, we can assume that $g$ centralizes $S_6$.

But $W$ is an irreducible real representation so, by Schur's lemma, the centralizer of $S_6$ in $\text{GL}_{10}(\mathbb{R})$ is $\mathbb{R}^{\times} \Id$. Since $10$ is even, this has positive determinant.

I suspect that, for $n = 10+4k$, it works to take $H = S_6 \times \mathbb{Z}/4 \mathbb{Z}$, with the $S_6$ acting on the first $10$ coordinates and $\mathbb{Z}/4 \mathbb{Z}$ acting on the other $4k$ coordinates. But $\text{Aut}(S_6 \times \mathbb{Z}/4 \mathbb{Z})$ is a bit larger than I want to work out.

This strategy can't work for $n=2$, though. Every subgroup of $\SO(2)$ is normal in the whole of $O(2)$. I think this case requires a very different strategy.

I can give an example for $SO(2)$ in dimension $5$.

Let $N$ be a fake projective plane: This is a quotient of the unit ball in $\mathbb C^2$ by a fixed-point-free group of automorphisms such that the quotient has the rational cohomology of a projective plane.

Let $M$ be the associated circle bundle of a complex line bundle on $N$ of nontrivial first Chern class. This has a natural action of $U(1)=SO(2)$. I claim this does not extend to $O(2)$.

If the action extended to $O(2)$, a nontrivial element of $O(2)$ would send $SO(2)$ orbits to $SO(2)$ orbits and thus act on $N$. This would give a diffeomorphism from $N$ to itself which sends the first Chern class of this complex line bundle to its negation. This would give an isomorphism from $\pi_1(N)$ to itself that sends the first Chern class of this complex line bundle, in group cohomology, to its negation. By Mostow rigidity, any isomorphism from $\pi_1(N)$ to itself arises from a isometry of the complex hyperbolic manifold $N$. Kharlamov and Kulikov proved that there is no isometry from $N$ to itself that reverses the complex structure, so it must preserve the complex structure.

Since the isometry preserves the complex structure, it preserves the first Chern class of the complex tangent bundle, which generates the second cohomology, so it preserves the second group cohomology, and thus does not negate any nontrivial cohomology class.

I want to record an idea about how we might be able to build a counterexample for $SO(2)$. Completing this approach will require someone with more differential geometry knowledge than I have.

Let $M$ be a $(4k+2)$-fold with no orientation reversing diffeomorphisms. For example, there are exotic $10$-spheres with this property. See this answer of Jason DeVito, and the table of exotic sphere groups at Wikipedia.

I now want to write down a circle action on $M$ which is generic in a certain sense. If we have an $SO(2)$ action on $M$, and $x$ is an isolated fixed point, then $SO(2)$ acts on $T_x M$. There is a sequence of positive integers $a_1 \leq a_2 \leq \cdots a_{2k+1}$ such that this action on $T_x M$ decomposes as a direct sum of $2$-dimensional subspaces, where the action on the $j$-th subspace is by $\begin{bmatrix} \cos (a_j \theta) & - \sin (a_j \theta) \\ \sin (a_j \theta) & \cos (a_j \theta) \end{bmatrix}$. I'll define $(a_1, a_2, \ldots, a_{2k+1})$ to be the "trace" of $x$. (I don't know if this is a standard term.)

The condition that I want is that our action has finitely many fixed points, but at least one, and that the various fixed points are all isolated with distinct traces. Such actions are easy to write down on various common manifolds that I understand, but I have no idea how to concretely write down an exotic $10$-sphere.

In this case, I claim that there is not a diffeomorphism $\sigma : M \to M$, conjugating the circle action to its inverse. If there were, it would have to take fixed points to fixed points and, by the assumption on traces, it would have to take each fixed point to itself. At a fixed point $x$, we would have an invertible linear map $D \sigma: T_x M \to T_x M$ which conjugated the action by $\theta$ to the action by $-\theta$. But all such maps have determinant $-1$ (this uses that $2k+1$ is odd), so $\sigma$ must be orientation reversing, contradicting the choice of $M$.