IPIP-499: UnixFS CID Profiles

[mishmosh]

Currently, CIDs can be generated with a variety of settings and optimizations for chunking, DAG width, and more. This means the same file can yield multiple, different CIDs depending on which tools and settings are used, and it is not possible to reliably reproduce or verify the CID.

This proposal introduces profiles for IPFS CIDs. Profiles explicitly define CID version, hash algorithm, chunk size, DAG width, layout, and other parameters. They can be used to verify data across implementations, provide recommended settings depending on retrieval performance goals, and more.

[lidel]

Thank you for kicking this off, and filling initial state.

I've incorporated specific "dag width" settings for File, Directory and HAMTDirectory nodes,
and updated the table to reflect state from ipfs/kubo#10774
and profiles that exist in Kubo master branch: legacy-cid-v0, test-cid-v1 and test-cid-v1-wide:

• https://github.com/ipfs/kubo/blob/master/config/profile.go#L268-L307

Next:

• agree what "cid-2025" profile should look like
  • this will be new default in "Kubo v1.0"
  • we have test-cid-v1 and test-cid-v1-wide in Kubo as potential candidates
• switch to PR from local branch (so we have build preview)
• figure out how to render the information (currently the table is not supported by https://github.com/ipfs/spec-generator)

[darobin]

Suggested change

	1. `HAMTDirectory` threshold (max `Directory` size before switching to `HAMTDirectory`): based on an estimate of the block size by counting the size of PNNode.Links
	1. `HAMTDirectory` threshold (max `Directory` size before switching to `HAMTDirectory`): based on an estimate of the block size by counting the size of PNNode.Links. We do not include details about the estimation algorithm as we do not encourage implementations to support it.

[lidel]

Bit odd to discourage, when both most popular implementations in GO and JS use size-based heurstic - #499 (comment)

Unsure how to handle this. Perhaps clarify the heuristic is implementation-specific, and when deterministic behavior is expected, a specific heuristic should be used?

[achingbrain]

I don't think we should be estimating the block size as it's trivial to calculate it exactly. Can we not just define this (and punt to the spec for the details) to make it less hand-wavey?

Suggested change

	1. `HAMTDirectory` threshold (max `Directory` size before switching to `HAMTDirectory`): based on an estimate of the block size by counting the size of PNNode.Links
	1. `HAMTDirectory` threshold (max `Directory` size before switching to `HAMTDirectory`): based on the final size of the serialized form of the [PBNode protobuf message](https://specs.ipfs.tech/unixfs/#dag-pb-node) that represents the directory.

[rvagg]

Hey, I'd love to be able to reference this, even if it's in "draft" form, could we just merge it and continue to iterate on top of it to get it right?

[github-actions]

🚀 Build Preview on IPFS ready

• 🔎 Commit: 123be3d
• 🔏 CID bafybeig3s4k2ppn6olfk56r3yrcyfuxlygevn2bra4vq77hazd5wcw6vmu
• 📦 Preview:
  • dweb.link
  • w3s.link
  • inbrowser.link

[mishmosh]

I made a few changes/fixes, aiming to land this early next week.

• Added links to UnixFS spec (now that it exists)
• Specified calendar versioning for profile names (line 64), per @b5 suggestion
  • @lidel I gave the 3 kubo profiles names that matched the naming scheme. This would mean minor updates to kubo, but is probably better for future-proofing. Acceptable? Also happy to discuss live.
• Changed the "current defaults" section into a series of legacy profile names, that implementations MAY support. This allows those profile sets to be referenced/used across implementations.
• We were using fanout and bitwidth interchangeably. I changed them all to fanout, in keeping with the UnixFS terminology. If we prefer bitwidth, I can PR that to UnixFS spec and then also here.
• Streamlined lots of duplicate language from Summary and Motivation sections

Open questions:

• How to handle Test fixtures section (line 120)? (Not-blocking, IMO)
• Thread on empty directory filtering (blocking)
• Thread on threshold size (blocking)

[lidel]

Just noting this is (imo) a blocker.

We did not merge UnixFS spec until we had sensible set of fixtures that people could use as reference.

The spec may be incomplete, but a fixture will let people reverse-engineer any details, and then PR improvement to spec.

Without fixtures for each UnixFS node type, we risk unknown unknown silently impacting final CID (e.g. because we did not know that someone may decide to place leaves one level sooner as "optimization" and someone else always at bottom, as "formal consistency")

[mishmosh]

Tracking this in ipfs/kubo#11071

[lidel]

Thanks!

• I will implement kubo-* profiles as part of 0.40 and test fixtures will be part of that work.
• Then we will be able to link to them form spec, like we did in https://specs.ipfs.tech/unixfs/#appendix-test-vectors

[mishmosh]

Just synced with @lidel. He wants to ship this with test fixtures in place, (tracked in kubo/issues/11071). In the meantime, we don't anticipate changes to the profiles themselves so you can can reference this PR.

[icidasset]

Great work, glad to see this!

Couple notes/questions:

• The profiles (legacy + new) don't say if the chunks are of a fixed size, or which algorithm they use.
• Small typo under "Compatibility": "support the the set of" (double the)
• Would it also be interesting to note if an implementation respects symlinks and if so, how the different kinds of symlinks are translated?

[lidel]

Quick update: I've pushed several commits addressing feedback and gaps in the document:

Resolved / Research done

• Documented Syhmlink behavior as suggested by @icidasset
  • Only Kubo 0.39 preserves symlinks, everything else dereferences on the fly by default, turning symlinks into real files and directories a symlink pointed at
• Added Chunking algorithm row to both profile tables for completeness
• Fixed kubo-legacy-2025 profile: corrected Leaves from raw to dag-pb (verified against kubo v0.39 legacy-cid-v0 profile where UnixFSRawLeaves=false)
• Documented filtering behavior with clear terminology:
  • included: always in DAG (no option to exclude)
  • excluded: always excluded (no option to include)
  • opt-in: excluded by default, flag to include (e.g., --hidden)
  • opt-out: included by default, flag to exclude
• Added Based on row with implementation versions and kubo profile names (legacy-cid-v0, test-cid-v1, test-cid-v1-wide)
• Clarified HAMTDirectory threshold estimation methods in the parameters section: link count (naive), PBNode.Links size (name + CID), or full dag-pb block size (most accurate)
• Noted that legacy table includes non-UnixFS implementations (DASL) in Summary section
• Added estimation method suffix (est:links[name+cid]) to kubo profiles in legacy table

Remaining TODOs in unixfs-2025

Parameter	Status
HAMTDirectory threshold	TODO - fix kubo: likely based on full block size estimation (Helia approach)
Empty directories	TODO - use kubo? needs opt-out flag + Import.*
Hidden entities	TODO - use kubo? needs opt-in flag + Import.*
Symlinks	TODO - use kubo? needs flag + Import.* for controlling if all symlinks in imported directory tree are preserved or dereferenced)
Test fixtures	TODO - reuse kubo: will reuse once kubo has them for *-2025 profiles

Other:

• Look into data-preservation-programs/singularity#525

Implementation Plan (Kubo 0.40, ETA 2026 Q1)

To finalize this IPIP, Kubo needs to support additional Import.* configuration flags for:

1. Empty directories: opt-out flag to exclude them from DAG
2. Hidden files: already has --hidden, just need to wire it up from config
3. HAMTDirectory threshold: configurable to support both legacy estimation (name + CID size) and Helia-style full block size calculation

Test fixtures will likely be included in the same Kubo PR that adds these missing features.

I also think we may replace two kubo-2025 and kubo-2025-wide profiles with a single one, that makes decision on what remains narrow and what is wide, but will update once Kubo changes land. (now that we have convention of doing IPIPs with profiles, we can always course-correct in `-202