The relationship between the two pseudometrics is the following: $\rho(x,y)=2\cdot d(x,y)$ for all $x$ and $y$.
We compare the definitions of the pseudometrics in both books. Both start with a sequence $\langle U_n:n\in\omega\rangle$ of sets that contain the diagonal. The demands on the sequence are: $U_0=X\times X$, and $U_{n+1}\circ U_{n+1}\circ U_{n+1}\subseteq U_n$ for all $n$.
Engelking specifies that the $U_n$ are members of a uniformity $\mathcal{U}$ and thus, according to his definition, the $U_n$ are, in addition, symmetric. Kelley imposes no further restrictions.
Kelley defines an auxiliary function $f$ on $X\times X$ by $$ f(x,y)=\begin{cases} 2^{-n} & (x,y)\in U_{n-1}\setminus U_n\\ 0 & (x,y)\in \bigcap_n U_n \end{cases} $$ and defines $d(x,y)$ to be the infimum of the set of all sums $$ \sum_{j=0}^n f(x_j,x_{j+1}) $$ over all finite sequences $\langle x_i:i\le n+1\rangle$ with $x_0=x$ and $x_{n+1}=y$.
Engelking's definition of $\rho(x,y)$ is a bit different: for every finite sequence $\langle x_i:i\le n+1\rangle$ of points in $X$ with $x_0=x$ and $x_{n+1}=y$ he takes all possible sums $$ \sum_{j=1}^{n+1} 2^{-i_j} $$ subject to $(x_{j-1},x_j)\in U_{i_j}$. One can rewrite these sums as
$$ \sum_{j=0}^{n} 2^{-i_j} $$ subject to $(x_j,x_{j+1})\in U_{i_j}$, this merely involves shifting the sequence $\langle i_j:1\le j\le n+1\rangle$ down by one.
Now observe that, in general, $(x,y)\in U_i$ iff $f(x,y)\le 2^{-(i+1)}$ Indeed: if $(x,y)\in U_i$ then $(x,y)\in U_{n-1}\setminus U_n$ for some $n$ with $n-1\ge i$, that is for some $n>i$. And that implies that $f(x,y)=2^{-n}\le2^{-(i+1)}$. Conversely, if $f(x,y)\le2^{-(i+1)}$ then $(x,y)\in U_{n-1}\setminus U_n$ for some $n\ge i+1$, that is, for some $n>i$.
This means that for every sequence $\langle x_i:i\le n+1\rangle$ and every associated sum $\sum_{j=0}^{n} 2^{-i_j}$ we have $$ 2\cdot \sum_{j=0}^n f(x_j,x_{j+1}) \le 2\cdot\sum_{j=0}^{n} 2^{-(i_j+1)} =\sum_{j=0}^{n} 2^{-i_j} $$ it also implies that $2\sum_{j=0}^n f(x_j,x_{j+1})$ is one of the associated sums, namely the one where every $i_j$ is such that $(x_j,x_{j+1})\in U_{i_j}\setminus U_{i_j+1}$.
This implies that $\rho(x,y)=2\cdot d(x,y)$ for all $x$ and $y$.
