We have $$\alpha = \left\{\begin{array}{ll} \sqrt{a^2 + b^2} & \mathrm{if} ~ (a^2 + b^2)R^2 \ge a^2 + cb^2, \\[6pt] R\sqrt{a^2 + b^2/c}& \mathrm{if} ~ a^2c^2 + b^2c \ge (a^2c^2 + b^2)R^2, \\[6pt] |a|\sqrt{\frac{c-R^2}{c-1}} + |b|\sqrt{\frac{R^2 - 1}{c-1}} & \mathrm{otherwise}. \end{array} \right.$$
Proof:
If $a^2 + b^2 = 0$, it is easy. In the following, assume that $a^2 + b^2 > 0$.
We split into three cases:
Case 1: $(a^2 + b^2)R^2 \ge a^2 + cb^2$
By Cauchy-Bunyakovsky-Schwarz inequality, we have $(ax + by)^2 \le (a^2 + b^2)(x^2 + y^2) \le a^2 + b^2$.
On the other hand, letting $x = \frac{a}{\sqrt{a^2+b^2}}$ and $y = \frac{b}{\sqrt{a^2+b^2}}$, we have $x^2 + y^2 = 1$ and $x^2 + cy^2 \le R^2$ and $ax + by = \sqrt{a^2 + b^2}$. The desired result follows.
Case 2: $a^2c^2 + b^2c \ge (a^2c^2 + b^2)R^2$,
By Cauchy-Bunyakovsky-Schwarz inequality, we have $(ax + by)^2 \le (a^2 + b^2/c)(x^2 + cy^2) \le R^2(a^2 + b^2/c)$.
On the other hand, letting $x = \frac{acR}{\sqrt{a^2c^2 + b^2c}}$ and $y = \frac{bR}{\sqrt{a^2c^2 + b^2c}}$, we have $x^2 + y^2 \le 1$ and $x^2 + cy^2 = R^2$ and $ax + by = R\sqrt{a^2 + b^2/c}$. The desired result follows.
Case 3: $(a^2 + b^2)R^2 < a^2 + cb^2$ and $a^2c^2 + b^2c < (a^2c^2 + b^2)R^2$
Clearly, $c \ne 1$. It is easy to prove that $$\alpha = \sup_{0 \le y \le \min(1, R/\sqrt c)} |a|\min(\sqrt{1-y^2}, ~ \sqrt{R^2 - cy^2}) + |b| y.$$
(i) If $c > 1$, from $(a^2 + b^2)R^2 < a^2 + cb^2$ and $a^2c^2 + b^2c < (a^2c^2 + b^2)R^2$, we have $R > 1$ and $c > R^2$ and $$\frac{|b|R}{\sqrt{a^2c^2 + b^2c}} < \sqrt{\frac{R^2-1}{c-1}} < \frac{|b|}{\sqrt{a^2+b^2}} . \tag{1}$$
We have $$\alpha = \sup_{0 \le y \le R/\sqrt c} |a|\min(\sqrt{1-y^2}, ~ \sqrt{R^2 - cy^2}) + |b| y = \max(\alpha_1, ~ \alpha_2)$$ where $$\alpha_1 = \sup_{0 \le y \le \sqrt{(R^2-1)/(c-1)}} |a|\sqrt{1-y^2} + |b| y$$ and $$\alpha_2 = \sup_{ \sqrt{(R^2-1)/(c-1)} \le y \le R/\sqrt c} |a|\sqrt{R^2 - cy^2} + |b| y.$$
Using (1), it is not difficult to prove that $$\alpha_1 = \alpha_2 = |a|\sqrt{\frac{c-R^2}{c-1}} + |b|\sqrt{\frac{R^2 - 1}{c-1}}.$$ (Note: For example, consider $\alpha_1$. Let $f(y) = |a|\sqrt{1-y^2} + |b| y$. Using $\frac{|b|}{\sqrt{a^2 + b^2}} > \sqrt{(R^2-1)/(c-1)}$, we have $f'(y) = - \frac{|a| y}{\sqrt{1 - y^2}} + |b| > 0$ on $[0, \sqrt{(R^2-1)/(c-1)}]$. Thus, $\alpha_1 = f(\sqrt{(R^2-1)/(c-1)}) = |a|\sqrt{\frac{c-R^2}{c-1}} + |b|\sqrt{\frac{R^2 - 1}{c-1}}$.)
The desired result follows.
(ii) If $0 < c < 1$, from $(a^2 + b^2)R^2 < a^2 + cb^2$ and $a^2c^2 + b^2c < (a^2c^2 + b^2)R^2$, we have $R < 1$ and $R^2 > c$ and $$\frac{|b|R}{\sqrt{a^2c^2 + b^2c}} > \sqrt{\frac{R^2-1}{c-1}} > \frac{|b|}{\sqrt{a^2+b^2}} . \tag{2}$$
We have $$\alpha = \sup_{0 \le y \le 1} |a|\min(\sqrt{1-y^2}, ~ \sqrt{R^2 - cy^2}) + |b| y = \max(\alpha_3, ~ \alpha_4)$$ where $$\alpha_3 = \sup_{0 \le y \le \sqrt{(1-R^2)/(1-c)}} |a|\sqrt{R^2 - cy^2} + |b| y,$$ and $$\alpha_4 = \sup_{\sqrt{(1-R^2)/(1-c)} \le y \le 1} |a|\sqrt{1-y^2} + |b| y$$
Using (2), it is not difficult to prove that $$\alpha_3 = \alpha_4 = |a|\sqrt{\frac{c-R^2}{c-1}} + |b|\sqrt{\frac{R^2 - 1}{c-1}}.$$
The desired result follows.
$\phantom{2}$
We are done.
