feat: Use nixpkgs to resolve with liveness

[booxter]

Since 7683220 we always mark nested with expressions alive. This is
technically sound because, indeed, there's no general way to determine
whether a particular attribute comes from one of the nested withs.

Sadly, this change made it impossible to detect common problematic
patterns in nixpkgs tree like:

meta = with lib; { maintainers = with lib.maintainers; [ ... ]; }; 

This would be very useful in our ongoing attempt to automate cleanup of
these patterns from the whole nixpkgs tree, see:

technowledgy/refactor-tractor#14

This change attempts to address this problem. It is observed that nixd
completion controllers already use nixpkgs attrset and heuristics to
look for known attributes in pkgs and lib, assuming these belong to
nixpkgs. We can do the same when analysing with scopes.

1. This patch reuses the existing nixpkgs client infrastructure for
   Sema/VariableLookup.cpp. Since variable analysis is part of
   libnixf while nixpkgs client and LSP are part of nixd, we have to
   relocate a number of modules plus the nixd-attrset-eval tool
   outside the nixd directory under common.

2. The nixd-attrset-eval tool was truncating the number of returned
   values by 30, which is useful for interactive use in completion
   helper context, but is not enough for comprehensive variable scope
   analysis. For this reason, the json protocol for the tool was
   extended to support MaxItems attribute that can be set to 0 to
   indicate that no truncation should occur.

3. On performance optimizations: The new nixpkgs client is initialized
   once per call to VLA. It will load and cache discovered attributes
   as needed for the with resolution procedure. (In the future, it may
   be used for other needs as needed.)

   Note: It could be beneficial to allow reuse of the same nixpkgs
   client with its cached results across multiple calls to VLA. This
   is left for a future exercise, if and when consuming tools like
   nixd-tidy gain support for multi-file operation.

4. On build system: Because the project supports "global" meson build
   from the root directory as well as component specific builds for
   libnixf and nixd, we have to setup scaffolding to reuse the
   common library in both. This is achieved with subprojects and a
   bunch of symlinks.

5. Because libnixf / nixd-tiny can be used interactively, and each
   call to nixd-attrset-eval is logged in extreme details (including
   complete json protocol comms between the tool and its driver), we
   have to suppress this output not to pollute the caller's stdout with
   these debug messages. Some adjustments were done to allow to disable
   these log messages when the tool is called from VLA nixpkgs client.

6. VLA test timeout is bumped from 30s to 120s because it now has to run
   nixd-attrset-eval. The Sema test now takes time comparable to nixd
   regression suite for this reason. 120s is the timeout taken from nixd
   regression suite.

[booxter]

@wolfgangwalther FYI. I tried to capture the design of the feature in the commit message in detail.

I still need to fix some compilation errors that are, apparently, now caught because I moved some files from nixd to libnixf that - I assume - has slightly different compilation regime (?)

Full disclosure: I used gpt5 codex LLM to bridge the gap between me and my lack of Meson (and to a lesser extent - C++) knowledge. I am not sure if the approach taken here in regards to cross-linking multiple "subprojects" to a "common" subproject is solid. If someone can point me to a better way of achieving the same, please let me know.

[inclyc]

Overall, I believe this PR fully addresses the issues encountered in the refactoring scenario. And I agree that we should do this. However, this patch is relatively large in scale and involves significant changes to the overall structure. Below are some of my thoughts:

1. This patch reuses the existing nixpkgs client infrastructure for Sema/VariableLookup.cpp. Since variable analysis is part of libnixf while nixpkgs client and LSP are part of nixd, we have to relocate a number of modules plus the nixd-attrset-eval tool outside the nixd directory under common.

When I initially designed libnixf, its "product positioning" was to be a "foundational library with minimal dependencies" (only relying on header-only libraries) and with runtime dependencies limited to the standard library. I made this decision because I hoped this library would ultimately be maximally reusable, for example:

1. Writing an independent linter based on this library
2. Replacing the official NixOS/nix parser with libnixf
3. Enabling the development of an LSP based on this library
4. Providing language bindings like Python

However, the current design introduces full LSP-required dependencies into libnixf, which might be inappropriate.

4. On build system: Because the project supports "global" meson build from the root directory as well as component specific builds for libnixf and nixd, we have to setup scaffolding to reuse the common library in both. This is achieved with subprojects and a bunch of symlinks.

I'm not very fond of meson's "subprojects" feature, as it forces all projects to have a hard-coded "subprojects" directory. Emotionally, I don't want this project to become like that.

I haven't carefully reviewed the specific code implementation yet, but perhaps we can first discuss how to technically solve the "with" liveness issue while keeping libnixf from introducing excessive dependencies?

[booxter]

@inclyc Thanks for the comment — I’ve seen it and appreciate it. I just need a bit more time to think it through. I share your uneasiness about the Meson subproject situation.

I’ll follow up in a week or two with more concrete details once I’ve had time to organize my thoughts. As a prelude, I’m thinking that proper decoupling would require introducing modes for the VLA code — one that runs with the nix-attrset-eval helper, and one that runs without it. VLA would still need to implement the JSON nix-attrset-eval protocol, but beyond that, there wouldn’t be a hard dependency.

nixf-tidy might be the right place to implement this mode selection. The workflow could look like: nixf-tidy --with-nixpkgs path/to/nixpkgs. The tool would then run nix-attrset-eval and prepare a NixpkgsClient object for VLA to use during variable analysis. If --with-nixpkgs isn’t passed, both the tool and VLA would behave as usual.

One open question: since nixf-tidy currently lives under libnixf, it doesn’t have in-PATH access to nix-attrset-eval. Is its placement under libnixf a strict requirement, or could it be moved closer to nix-attrset-eval — for example, under nixd directory?

[inclyc]

Thanks for your detailed sharing earlier! Regarding the discussion about decoupling the VLA code and tool dependencies you mentioned, I've sorted out a specific plan to sync with you:

Here's my idea: we could create a new directory that depends on both the attrset-eval tool and libnixf (this way, libnixf itself won't need any modifications, maintaining its existing stability). Then, within this new directory, we'll specifically implement refactoring functionality based on nixpkgs. To be specific, we'll develop an independent tool (named nixpkgs-refactor, for example) that depends on both libnixf and attrset-eval, with the core function of supporting systematic refactoring of nixpkgs.

However, I have a concern to discuss with you: is there sufficient demand to support this work? Could it be that these needs can be addressed without developing a dedicated tool, perhaps just by submitting a PR to nixpkgs, with no need for ongoing maintenance or checks afterward? Additionally, I noticed that you're preparing a refactoring project. I wonder if there's a significant demand for such systematic refactoring within that project? If the demand is indeed clear and frequent, the value of creating a new tool might be more prominent.

[booxter]

I believe @wolfgangwalther was thinking of making the refactoring suite integrated in nixpkgs. Perhaps we should sync in a group to see how to slice the problem better. There are a number of tools involved right now (nixd, nixf-tidy, nixf-diagnose, refactor-tractor, nixpkgs' fmt target...) and they may start looking like a hodgepodge. :)

[wolfgangwalther]

However, I have a concern to discuss with you: is there sufficient demand to support this work? Could it be that these needs can be addressed without developing a dedicated tool, perhaps just by submitting a PR to nixpkgs, with no need for ongoing maintenance or checks afterward? Additionally, I noticed that you're preparing a refactoring project. I wonder if there's a significant demand for such systematic refactoring within that project? If the demand is indeed clear and frequent, the value of creating a new tool might be more prominent.

Once we are through the whole migration, we can just check for meta = with lib; and forbid this. This PR is only required for the migration and it's quite specific. Personally, I don't think it's required to merge this into nixd at all - I don't see much use beyond this migration effort.

It would be great however, if you, @inclyc, could check the code here as the expert on it. I wouldn't care much about the structure for meson or whatever - but it would be great if you could check whether there is anything wrong with the actual "with liveness check" implemented here.

[booxter]

FWIW I am fine with keeping it in an unmerged branch and pinning to it in refactor-tractor repository while we need the rule (and dropping it afterwards). If we find more use cases for nixpkgs aware VLA, we could then revive the PR.

[booxter]

@inclyc please let us know if you'll be able to take an expert eye look at VLA changes.

[inclyc]

I just took a brief look at this implementation, and I think its overall correctness should be fine.

[wolfgangwalther]

Which checkout of Nixpkgs does this use? The one that it is running in or one that it was built with?

[booxter]

I would have to double check but I think it will use <nixpkgs>. I don't think nixd encloses nixpkgs lib on build. @wolfgangwalther

I may give an answer in a day or two if you don't find out yourself before then.

[wolfgangwalther]

I would have to double check but I think it will use <nixpkgs>.

Ah, that makes sense. Annoying, but likely. Will update my system to make sure I get all the latest data.

[booxter]

@wolfgangwalther it's actually an interesting concern. Wonder if we should pin nixpkgs for the refactor tractor using NIX_PATH so that it's pure in this way.

[wolfgangwalther]

Wonder if we should pin nixpkgs for the refactor tractor using NIX_PATH so that it's pure in this way.

This works in principle, but one needs to be careful - it needs a second checkout with the same base commit, because (I assume) it ends in some cycle of reloading a new changes nixpkgs after every modification the script makes, otherwise...