esm: add initialize hook, integrate with register

Follows @giltayar's proposed API: > `register` can pass any data it wants to the loader, which will be passed to the exported `initialize` function of the loader. Additionally, if the user of `register` wants to communicate with the loader, it can just create a `MessageChannel` and pass the port to the loader as data. The `register` API is now: ```ts interface Options { parentUrl?: string; data?: any; transferList?: any[]; } function register(loader: string, parentUrl?: string): any; function register(loader: string, options?: Options): any; ``` This API is backwards compatible with the old one (new arguments are optional and at the end) and allows for passing data into the new `initialize` hook. If this hook returns data it is passed back to `register`: ```ts function initialize(data: any): Promise<any>; ``` **NOTE**: Currently there is no mechanism for a loader to exchange ownership of something back to the caller. Refs: nodejs/loaders#147 PR-URL: #48842 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>

Author: izaakschroeder

doc/api/esm.md

@@ -685,6 +685,9 @@ of Node.js applications.
 <!-- YAML
 added: v8.8.0
 changes:
+ - version: REPLACEME
+ pr-url: https://github.com/nodejs/node/pull/48842
+ description: Added `initialize` hook to replace `globalPreload`.
  - version:
  - v18.6.0
  - v16.17.0
@@ -739,6 +742,69 @@ different [realm](https://tc39.es/ecma262/#realm). The hooks thread may be
 terminated by the main thread at any time, so do not depend on asynchronous
 operations (like `console.log`) to complete.
 
+#### `initialize()`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+> The loaders API is being redesigned. This hook may disappear or its
+> signature may change. Do not rely on the API described below.
+
+* `data` {any} The data from `register(loader, import.meta.url, { data })`.
+* Returns: {any} The data to be returned to the caller of `register`.
+
+The `initialize` hook provides a way to define a custom function that runs
+in the loader's thread when the loader is initialized. Initialization happens
+when the loader is registered via [`register`][] or registered via the
+`--loader` command line option.
+
+This hook can send and receive data from a [`register`][] invocation, including
+ports and other transferrable objects. The return value of `initialize` must be
+either:
+
+* `undefined`,
+* something that can be posted as a message between threads (e.g. the input to
+ [`port.postMessage`][]),
+* a `Promise` resolving to one of the aforementioned values.
+
+Loader code:
+
+```js
+// In the below example this file is referenced as
+// '/path-to-my-loader.js'
+
+export async function initialize({ number, port }) {
+ port.postMessage(`increment: ${number + 1}`);
+ return 'ok';
+}
+```
+
+Caller code:
+
+```js
+import assert from 'node:assert';
+import { register } from 'node:module';
+import { MessageChannel } from 'node:worker_threads';
+
+// This example showcases how a message channel can be used to
+// communicate between the main (application) thread and the loader
+// running on the loaders thread, by sending `port2` to the loader.
+const { port1, port2 } = new MessageChannel();
+
+port1.on('message', (msg) => {
+ assert.strictEqual(msg, 'increment: 2');
+});
+
+const result = register('/path-to-my-loader.js', {
+ parentURL: import.meta.url,
+ data: { number: 1, port: port2 },
+ transferList: [port2],
+});
+
+assert.strictEqual(result, 'ok');
+```
+
 #### `resolve(specifier, context, nextResolve)`
 
 <!-- YAML
@@ -949,8 +1015,8 @@ changes:
  description: Add support for chaining globalPreload hooks.
 -->
 
-> The loaders API is being redesigned. This hook may disappear or its
-> signature may change. Do not rely on the API described below.
+> This hook will be removed in a future version. Use [`initialize`][] instead.
+> When a loader has an `initialize` export, `globalPreload` will be ignored.
 
 > In a previous version of this API, this hook was named
 > `getGlobalPreloadCode`.
@@ -1609,13 +1675,16 @@ for ESM specifiers is [commonjs-extension-resolution-loader][].
 [`import.meta.resolve`]: #importmetaresolvespecifier-parent
 [`import.meta.url`]: #importmetaurl
 [`import`]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import
+[`initialize`]: #initialize
 [`module.createRequire()`]: module.md#modulecreaterequirefilename
 [`module.register()`]: module.md#moduleregister
 [`module.syncBuiltinESMExports()`]: module.md#modulesyncbuiltinesmexports
 [`package.json`]: packages.md#nodejs-packagejson-field-definitions
+[`port.postMessage`]: worker_threads.md#portpostmessagevalue-transferlist
 [`port.ref()`]: https://nodejs.org/dist/latest-v17.x/docs/api/worker_threads.html#portref
 [`port.unref()`]: https://nodejs.org/dist/latest-v17.x/docs/api/worker_threads.html#portunref
 [`process.dlopen`]: process.md#processdlopenmodule-filename-flags
+[`register`]: module.md#moduleregister
 [`string`]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String
 [`util.TextDecoder`]: util.md#class-utiltextdecoder
 [cjs-module-lexer]: https://github.com/nodejs/cjs-module-lexer/tree/1.2.2

doc/api/module.md

@@ -175,6 +175,28 @@ globalPreload: http-to-https
 globalPreload: unpkg
 ```
 
+This function can also be used to pass data to the loader's [`initialize`][]
+hook; the data passed to the hook may include transferrable objects like ports.
+
+```mjs
+import { register } from 'node:module';
+import { MessageChannel } from 'node:worker_threads';
+
+// This example showcases how a message channel can be used to
+// communicate to the loader, by sending `port2` to the loader.
+const { port1, port2 } = new MessageChannel();
+
+port1.on('message', (msg) => {
+ console.log(msg);
+});
+
+register('./my-programmatic-loader.mjs', {
+ parentURL: import.meta.url,
+ data: { number: 1, port: port2 },
+ transferList: [port2],
+});
+```
+
 ### `module.syncBuiltinESMExports()`
 
 <!-- YAML
@@ -364,6 +386,7 @@ returned object contains the following keys:
 [`--enable-source-maps`]: cli.md#--enable-source-maps
 [`NODE_V8_COVERAGE=dir`]: cli.md#node_v8_coveragedir
 [`SourceMap`]: #class-modulesourcemap
+[`initialize`]: esm.md#initialize
 [`module`]: modules.md#the-module-object
 [module wrapper]: modules.md#the-module-wrapper
 [source map include directives]: https://sourcemaps.info/spec.html#h.lmz475t4mvbx

lib/internal/modules/esm/hooks.js

@@ -2,6 +2,7 @@
 
 const {
  ArrayPrototypePush,
+ ArrayPrototypePushApply,
  FunctionPrototypeCall,
  Int32Array,
  ObjectAssign,
@@ -47,8 +48,10 @@ const {
  validateObject,
  validateString,
 } = require('internal/validators');
-
-const { kEmptyObject } = require('internal/util');
+const {
+ emitExperimentalWarning,
+ kEmptyObject,
+} = require('internal/util');
 
 const {
  defaultResolve,
@@ -83,6 +86,7 @@ let importMetaInitializer;
 
 // [2] `validate...()`s throw the wrong error
 
+let globalPreloadWarned = false;
 class Hooks {
  #chains = {
  /**
@@ -127,31 +131,43 @@ class Hooks {
  * Import and register custom/user-defined module loader hook(s).
  * @param {string} urlOrSpecifier
  * @param {string} parentURL
+ * @param {any} [data] Arbitrary data to be passed from the custom
+ * loader (user-land) to the worker.
  */
- async register(urlOrSpecifier, parentURL) {
+ async register(urlOrSpecifier, parentURL, data) {
  const moduleLoader = require('internal/process/esm_loader').esmLoader;
  const keyedExports = await moduleLoader.import(
  urlOrSpecifier,
  parentURL,
  kEmptyObject,
  );
- this.addCustomLoader(urlOrSpecifier, keyedExports);
+ return this.addCustomLoader(urlOrSpecifier, keyedExports, data);
  }
 
  /**
  * Collect custom/user-defined module loader hook(s).
  * After all hooks have been collected, the global preload hook(s) must be initialized.
  * @param {string} url Custom loader specifier
  * @param {Record<string, unknown>} exports
+ * @param {any} [data] Arbitrary data to be passed from the custom loader (user-land)
+ * to the worker.
+ * @returns {any} The result of the loader's `initialize` hook, if provided.
  */
- addCustomLoader(url, exports) {
+ addCustomLoader(url, exports, data) {
  const {
  globalPreload,
+ initialize,
  resolve,
  load,
  } = pluckHooks(exports);
 
- if (globalPreload) {
+ if (globalPreload && !initialize) {
+ if (globalPreloadWarned === false) {
+ globalPreloadWarned = true;
+ emitExperimentalWarning(
+ '`globalPreload` will be removed in a future version. Please use `initialize` instead.',
+ );
+ }
  ArrayPrototypePush(this.#chains.globalPreload, { __proto__: null, fn: globalPreload, url });
  }
  if (resolve) {
@@ -162,6 +178,7 @@ class Hooks {
  const next = this.#chains.load[this.#chains.load.length - 1];
  ArrayPrototypePush(this.#chains.load, { __proto__: null, fn: load, url, next });
  }
+ return initialize?.(data);
  }
 
  /**
@@ -553,15 +570,30 @@ class HooksProxy {
  }
  }
 
- async makeAsyncRequest(method, ...args) {
+ /**
+ * Invoke a remote method asynchronously.
+ * @param {string} method Method to invoke
+ * @param {any[]} [transferList] Objects in `args` to be transferred
+ * @param {any[]} args Arguments to pass to `method`
+ * @returns {Promise<any>}
+ */
+ async makeAsyncRequest(method, transferList, ...args) {
  this.waitForWorker();
 
  MessageChannel ??= require('internal/worker/io').MessageChannel;
  const asyncCommChannel = new MessageChannel();
 
  // Pass work to the worker.
- debug('post async message to worker', { method, args });
- this.#worker.postMessage({ method, args, port: asyncCommChannel.port2 }, [asyncCommChannel.port2]);
+ debug('post async message to worker', { method, args, transferList });
+ const finalTransferList = [asyncCommChannel.port2];
+ if (transferList) {
+ ArrayPrototypePushApply(finalTransferList, transferList);
+ }
+ this.#worker.postMessage({
+ __proto__: null,
+ method, args,
+ port: asyncCommChannel.port2,
+ }, finalTransferList);
 
  if (this.#numberOfPendingAsyncResponses++ === 0) {
  // On the next lines, the main thread will await a response from the worker thread that might
@@ -593,12 +625,19 @@ class HooksProxy {
  return body;
  }
 
- makeSyncRequest(method, ...args) {
+ /**
+ * Invoke a remote method synchronously.
+ * @param {string} method Method to invoke
+ * @param {any[]} [transferList] Objects in `args` to be transferred
+ * @param {any[]} args Arguments to pass to `method`
+ * @returns {any}
+ */
+ makeSyncRequest(method, transferList, ...args) {
  this.waitForWorker();
 
  // Pass work to the worker.
- debug('post sync message to worker', { method, args });
- this.#worker.postMessage({ method, args });
+ debug('post sync message to worker', { method, args, transferList });
+ this.#worker.postMessage({ __proto__: null, method, args }, transferList);
 
  let response;
  do {
@@ -708,6 +747,7 @@ ObjectSetPrototypeOf(HooksProxy.prototype, null);
  */
 function pluckHooks({
  globalPreload,
+ initialize,
  resolve,
  load,
 }) {
@@ -723,6 +763,10 @@ function pluckHooks({
  acceptedHooks.load = load;
  }
 
+ if (initialize) {
+ acceptedHooks.initialize = initialize;
+ }
+
  return acceptedHooks;
 }