Here is a ZFC counterexample.
Let $\mathbb{P}$ be the forcing to add a Cohen real $c$, and let $\omega_1=\bigsqcup_n S_n$ be a partition of $\omega_1$ into disjoint stationary sets $S_n$. Let $\dot{\mathbb{Q}}$ be the forcing to kill the stationarity of all $S_n$, for $n\in c$, the generic Cohen real added by the $\mathbb{P}$ forcing. It is not difficult to see that $\dot{\mathbb{Q}}$ is forced to be $(\omega,\infty)$-distributive, since we are really just shooting a club through the other $S_m$'s.
The term forcing for $\dot{\mathbb{Q}}$ over $\mathbb{P}$ will have to add a generic for $\mathbb{Q}$ over any Cohen real in the extension, and so it will have to kill the stationarity of all the $S_n$'s. Thus, it will have to collapse $\omega_1$. So it cannot be $\omega$-distributive.
A modification to the argument, using higher cardinals instead of $\omega_1$, will have the property that $\mathbb{P}$ is small relative to the distributivity, as you asked for in the comments. For example, let $\kappa$ be any uncountable regular cardinal and consider an $\omega$-partition of the cofinality $\omega$ ordinals up to $\kappa^+$. The $\mathbb{Q}$ forcing should kill parts of the partition, depending on the digits of $c$, and this will be fine, but the term forcing will have to kill them all, since it must add generics over any Cohen real in the extension. So $|\mathbb{P}|<\kappa$ and $\dot{\mathbb{Q}}$ is $(\kappa,\infty)$-distributive, but the term forcing collapses $\kappa^+$ and hence is not $\kappa$-distributive.
Here is a more extreme example, which makes the same point perhaps more clearly. Let $S\subset\omega_1$ be a stationary co-stationary set. Let $\mathbb{P}$ be the lottery sum $\{\text{yes}\}\oplus\{\text{no}\}$ of two trivial forcing notions, so that the generic filter selects either the point yes or the point no, and both of these are generic. Let $\dot{\mathbb{Q}}$ be the forcing that kills the stationarity of $S$, if the generic filter for $\mathbb{P}$ selected yes, and otherwise kills the stationarity of the complement of $S$. This forcing is $\omega$-distributive. But the term forcing has to add a generic for both the generics for $\mathbb{P}$, and so it will have to kill the stationarity both of $S$ and its complement. So it will collapse $\omega_1$.