Skip to content

Scala Wart: Convoluted de-sugaring of for-comprehensions #2573

@lihaoyi

Description

@lihaoyi

Opening this issue, as suggested by Martin, to provide a place to discuss the individual warts brought up in the blog post Warts of the Scala Programming Language and the possibility of mitigating/fixing them in Dotty (and perhaps later in Scala 2.x). These are based on Scala 2.x behavior, which I understand Dotty follows closely, apologies in advance if it has already been fixed


Scala lets you write for-comprehensions, which are converted into a chain
of flatMaps an maps as shown below:

@ val (x, y, z) = (Some(1), Some(2), Some(3)) x: Some[Int] = Some(1) y: Some[Int] = Some(2) z: Some[Int] = Some(3) @ for{ i <- x j <- y k <- z } yield i + j + k res40: Option[Int] = Some(6) @ desugar{ for{ i <- x j <- y k <- z } yield i + j + k } res41: Desugared = x.flatMap{ i => y.flatMap{ j => z.map{ k => i + j + k } } }

I have nicely formatted the desugared code for you, but you can try this
yourself in the Ammonite Scala REPL to
verify that this is what the for-comprehension gets transformed into.

This is a convenient way of implementing nested loops over lists, and happily
works with things that aren't lists: Options (as shown above), Futures,
and many other things.

You can also assign local values within the for-comprehension, e.g.

@ for{ i <- x j <- y foo = 5 k <- z } yield i + j + k + foo res42: Option[Int] = Some(11)

The syntax is a bit wonky (you don't need a val, you can't define defs or
classes or run imperative commands without _ = println("debug")) but for
simple local assignments it works. You may expect the above code to be
transformed into something like this

res43: Desugared = x.flatMap{ i => y.flatMap{ j => val foo = 5 z.map{ k => i + j + k } } }

But it turns out it instead gets converted into something like this:

@ desugar{ for{ i <- x j <- y foo = 5 k <- z } yield i + j + k + foo } res43: Desugared = x.flatMap(i => y.map{ j => val foo = 5 scala.Tuple2(j, foo) }.flatMap((x$1: (Int, Int)) => (x$1: @scala.unchecked) match { case Tuple2(j, foo) => z.map(k => i + j + k + foo) } ) )

Although it is roughly equivalent, and ends up with the same result in most
cases, this output format is tremendously convoluted and wastefully inefficient
(e.g. creating and taking-apart unnecessary Tuple2s). As far as I can tell,
there is no reason at all not to generated the simpler version of the code
shown above.

PostScript:

Notably, these two desugarings do not always produce the same results! The current desugaring behaves weirdly in certain cases; here is one that just bit me an hour ago:

Welcome to Scala 2.11.11 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_112). Type in expressions for evaluation. Or try :help. scala> for{ | x <- Right(1).right | y = 2 | z <- Right(3).right | } yield x + y + z <console>:13: error: value flatMap is not a member of Product with Serializable with scala.util.Either[Nothing,(Int, Int)] x <- Right(1).right ^ <console>:16: error: type mismatch; found : Any required: String } yield x + y + z ^

This specific problem has gone away in 2.12 because Either doesn't need .right anymore, but the language-level problem is still there: y = 2 ends up causing strange, difficult-to-debug errors due to the weirdness of the desugaring. This would not be an issue at all given the desugaring I proposed.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions