You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
"""This function deploys python ml in PySpark using the `predict` method of `model.
16
16
17
17
Args:
18
18
model: python ml model with sklearn API
19
-
*cols (list-like): Features used for predictions, required to be present as columns in the spark DataFrame used to make predictions.
19
+
cols (list-like): Features used for predictions, required to be present as columns in the spark DataFrame used to make predictions.
20
20
"""
21
21
@sf.pandas_udf(returnType=DoubleType())
22
22
def predict_pandas_udf(*cols):
23
23
# cols will be a tuple of pandas.Series here.
24
-
X = pd.concat(cols, axis=1)
25
-
return pd.Series(model.predict(X))
24
+
x = pd.concat(cols, axis=1)
25
+
return pd.Series(model.predict(x))
26
26
27
27
return predict_pandas_udf(*cols)
28
28
@@ -40,4 +40,4 @@ The code used in the [deploying-python-ml-in-pyspark](deploying-python-ml-in-pys
40
40
41
41
# Further Reading
42
42
- The code used in is based on the excellent excellent blog post ["Prediction at Scale with scikit-learn and PySpark Pandas UDFs"](https://medium.com/civis-analytics/prediction-at-scale-with-scikit-learn-and-pyspark-pandas-udfs-51d5ebfb2cd8) written by **Michael Heilman**.
43
-
-[sklearn](https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html) has more information on column transformers with mixed types.
43
+
-[sklearn](https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html) has more information on column transformers with mixed types.
0 commit comments