Pre-SIP: Reference-able Package Objects

Scala Improvement Process
1

Reference-able Package Objects

One limitation with package objects is that we cannot currently assign them to values: a.b fails to compile when b is a package object, even though it succeeds when b is a normal object. The workaround is to call a.b.package, which is ugly and non-obvious, or to use a normal object, which is not always possible. This proposal is to allow a.b to automatically expand into a.b.package when b is a package object. Such usage will simplify the language, simplify IDE support for the language, and generally make things more uniform and regular.

Why?

package objects are the natural “entry point” of a package. While top-level declarations reduce their need somewhat, they do not replace it: package objects are still necessary for adding package-level documentation or having the package-level API inherit from traits or classes. Other languages have equivalent constructs (module-info.java or __init__.py) that fulfil the same need, so it’s not just a quirk of the Scala language.

Notably, normal objects are not a replacement for package objects, because only package objects allow the package contents to be defined in other files. Normal objects would require that the package contents be all defined in a single file in the object body, or scattered into other files as traits in different packages and mixed into the object, both of which are messy and sub-optimal.

It’s possible to have a convention “the object named foo is always going to be the primary entrypoint for a package”, but that is just a poor-man’s package object with worse syntax and less standardization.

Many libraries use package objects to expose the “facade” of the package hierarchy:

  • Mill uses package objects to expose the build definitions within each package, and each one is an instance of mill.Module

  • Requests-Scala uses a package object to represent the default requests.BaseSession instance with the default configuration for people to use

  • PPrint uses a package object to expose the pprint.log and other APIs for people to use directly, as a default instance of PPrinter

  • OS-Lib uses a package object to expose the primary API of the os.* operations

None of these use cases can be satisfied by normal objects or by top-level declarations, due to the necessity of documentation and inheritance. They need to be package objects.

However, the fact that you cannot easily pass around these default instances as values e.g. val x: PPrinter = pprint without calling pprint.package is a source of friction.

This source of friction is not just for humans, but for tools as well. For example, IntelliJ needs a special case and special handling in the Scala plugin specifically to support this irregularity:

What

This proposal is meant to allow the following:

package a package object b val z = a.b // Currently fails with "package is not a value"

Currently the workaround is to use a .package suffix:

val z = a.b.`package`

This proposal is to make it such that given a.b, if b is a package containing a package object, expands to a.b.package automatically

Limitations

  • a.b only expands to a.b.package when used “standalone”, i.e. not when part of a larger select chain a.b.c or equivalent postfix expression a.b c, prefix expression !a.b, or infix expression a.b c d.

  • a.b expands to a.b.package of the type a.b.package.type, and only contains the contents of the package object. It does not contain other things in the package a.b that are outside of the package object

Both these requirements are necessary for backwards compatibility, and anyway do not impact the main goal of removing the irregularity between package objects and normal objects.

Alternatives

The two main alternatives now are to use .package suffixes, e.g. in Mill writing:

def moduleDeps = Seq(foo.`package`, bar.`package`, qux.baz.`package`)

Or to use normal objects. Notably, normal objects do not allow packages of the same name, which leads to contortions. e.g. Rather than:

package object foo extends _root_.foo.bar.Qux{ val bar = 1 } 
package foo.bar class Qux

We need to move the package foo contents into package foo2 to avoid conflicts with object foo, and then we need to add back aliases to all the declarations in foo2 to make them available in foo:

object foo extends foo2.bar.Qux{ val bar = 1 object bar{ type Qux = foo2.bar.Qux } } 
package foo2.bar class Qux

Both of these workarounds are awkward and non-idiomatic, but are necessary due to current limitations in referencing package objects directly

Implementations

Mill since version 0.12.0 already emulates this proposed behavior in Scala 2 using source-code mangling hacks, with custom support in IntelliJ. It works great and does what it was intended to do (allow passing around package objects as values without having to call .package every time)

We have a prototype Scala3 implementation here Expand value references to packages to their underlying package objects by odersky · Pull Request #22011 · scala/scala3 · GitHub

3 Likes
2

This is what I read in Scala 3 docs:

https://docs.scala-lang.org/scala3/reference/dropped-features/package-objects.html

will be dropped. They are still available, but will be deprecated and removed at some point in the future

While I like package objects in theory, I remember when using them with Scala 2 I have encountered so many implementation glitches, esp. in the compiler, but in other tooling as well, than I have learned to avoid them as a plague.

I understand you like to keep them (e.g. Plan for Package Objects in Scala 3). If it possible to implement them in a way which is glitchless, I am all for it, but considering backwards compatibility burden, I really doubt it it possible (and I have not seen much interest from other relevant people involved).

1 Like
3

Scala is the language of second chances for any feature who is down and out.

It would be nice if they just got rid of the backticks in the selection, since we’re so soft on soft keywords, package is a natural, see how naturally I put it in backticks for this post.

And it would be nice if they sorted out “my package object has a class C and so does my package!” as we hear on daytime talk shows where people confess their strangely intertwined relations.

Far from going extinct, package objects named after files have overrun the ecosystem, so what’s one more package object named after the package?

Loving inheritance is a bit surprising, because a few years ago, that was the most pernicious of behaviors, but hey, love the one you’re with.

1 Like
4

The Scala 3 compiler treats package objects and toplevel definitions mostly in the same way. So the added overhead of package objects is very low, in both spec and implementation. I am not sure what problems were encountered in Scala 2, but it would be good to check each of them whether they still persist in Scala 3.

About dropping package objects: Yes, that was the original intention. But since we still don’t have good alternative solutions to the points mentioned in Plan for Package Objects in Scala 3 it seems the best way forward is to keep them.

For instance, the concern how to document a package requires a solution where there is one specific artefact standing for a package. We have package objects, and moving to something else would be painful.

1 Like
5

For me that observation convinced me that the change is right.

a.b.`package`

is an encoding of a package object reference. We should not be forced to drop down to the level of encodings to express that; there should be a way to do it using higher-level concepts.

2 Likes
6

More recently zinc issues with them:

As I said, theoretically I like them, but my practical experience with them is not encouraging, too many issues which make work harder. If they are fixable, great - however they need to be reliable, and work well even for complex real-life code, not only for simple examples.

7

Without a self-contained minimization it’s impossible to see whether they would be issues in Scala 3. I have not seen anything out of the ordinary in the example code you gave. The companion symbol logic which failed in the first issue has been made much more robust in Scala 3. So I don’t forsee a problem from that side.

8

Opened a SIP SIP-XX: Reference-able Package Objects by lihaoyi · Pull Request #100 · scala/improvement-proposals · GitHub

5 Likes
9

About the motivation that package objects are necessary to use inheritance and top-level fields are not enough.

Can’t the inheritance problem in the top-level fields be solved with exports?

package a.b trait Foo { def foo = "Foo" var bar = 1 val zoo = "zoo" } private object StuffToExport extends Foo export StuffToExport.*

Should be equivalent to:

package a.b trait Foo { def foo = "Foo" var bar = 1 val zoo = "zoo" }

and

package a package object b extends Foo

If it’s equivalent, I think it should be removed from the motivations since it can’t be a reason to stop deprecating package objects

Also exports have the advantage that can be used multiple times to export fields from multiple objects, while you can have only one package object, which is not consistent with the fact that you can have top-level definitions from multiple files.

2 Likes