Release 8

Author: HenrikPoulsen

.gitignore

@@ -23,12 +23,16 @@
 build/*
 TwoStickShooter/Pure/Library/AnnotationManager
 *.rsp
+*.pyc
 
 #generated by performance framework
 */PerformanceTestRunInfo.json
+Performance/Assets/StreamingAssets.meta
+Performance/Assets/StreamingAssets
 
-# Files to unignore that would otherwise have been ignored
-!PackageRepos/ECSJobDemos/UnstablePrototypes/Asteriods/Assets/Packages/Unity.Multiplayer/Plugins/gamesocket.native.pdb
-
-*.pyc
 StreamingAssets.meta
+PerformanceTestRunInfo.json
+PerformanceTestRunInfo.json.meta
+
+**/InitTestScene*
+Samples/Logs/*

Documentation/content/chunk_iteration.md

@@ -0,0 +1,336 @@
+# Chunk Iteration
+
+## Motivation
+
+If, for example, there are three components Position, Rotation, and Scale and the output of any combination of these three components should write to a LocalToWorld component. The approach using component group injection might look something like:
+
+```
+struct PositionToLocalToWorld
+{
+ ComponentDataArray<Position> Position;
+ SubtractiveComponent<Rotation> Rotation;
+ SubtractiveComponent<Scale> Rotation;
+ ComponentDataArray<LocalToWorld> LocalToWorld;
+}
+[Inject] PositionToLocalToWorld positionToLocalToWorld;
+
+struct PositionRotationToLocalToWorld
+{
+ ComponentDataArray<Position> Position;
+ ComponentDataArray<Rotation> Rotation;
+ SubtractiveComponent<Scale> Rotation;
+ ComponentDataArray<LocalToWorld> LocalToWorld;
+}
+[Inject] PositionRotationToLocalToWorld positionRotationToLocalToWorld;
+
+struct PositionRotationScaleToLocalToWorld
+{
+ ComponentDataArray<Position> Position;
+ ComponentDataArray<Rotation> Rotation;
+ ComponentDataArray<Scale> Rotation;
+ ComponentDataArray<LocalToWorld> LocalToWorld;
+}
+[Inject] PositionRotationScaleToLocalToWorld positionRotationScaleToLocalToWorld;
+
+struct PositionScaleToLocalToWorld
+{
+ ComponentDataArray<Position> Position;
+ SubtractiveComponent<Rotation> Rotation;
+ ComponentDataArray<Scale> Rotation;
+ ComponentDataArray<LocalToWorld> LocalToWorld;
+}
+[Inject] PositionScaleToLocalToWorld positionScaleToLocalToWorld;
+
+struct RotationToLocalToWorld
+{
+ SubtractiveComponent<Position> Position;
+ ComponentDataArray<Rotation> Rotation;
+ SubtractiveComponent<Scale> Rotation;
+ ComponentDataArray<LocalToWorld> LocalToWorld;
+}
+[Inject] RotationToLocalToWorld rotationToLocalToWorld;
+
+struct RotationScaleToLocalToWorld
+{
+ SubtractiveComponent<Position> Position;
+ ComponentDataArray<Rotation> Rotation;
+ ComponentDataArray<Scale> Rotation;
+ ComponentDataArray<LocalToWorld> LocalToWorld;
+}
+[Inject] RotationScaleToLocalToWorld rotationScaleToLocalToWorld;
+
+struct ScaleToLocalToWorld
+{
+ SubtractiveComponent<Position> Position;
+ SubtractiveComponent<Rotation> Rotation;
+ ComponentDataArray<Scale> Rotation;
+ ComponentDataArray<LocalToWorld> LocalToWorld;
+}
+[Inject] ScaleToLocalToWorld scaleToLocalToWorld;
+```
+
+ComponentGroup is a utility which simplifies iteration over same component type values independent of the archetypes those components belong to. ComponentGroup accomplishes this by contraining the types of archetypes that are queried: Either components which must exist in all matching archetypes or components which exist in none of the matching archetypes (subtractive). By contrast, chunks can be iterated in a way that matches how the data is layed out in memory without those same contraints. (At the cost of foregoing ComponentGroup, automatic injection and and other associated utilities.)
+
+Direct chunk iteration allows for "optional" components or managing component combinations more directly.
+
+Another alternative might be to use ComponentDataFromEntity and check for the existence of the components on a per-entity basis, as in:
+```
+[Inject] [ReadOnly] ComponentDataFromEntity<Position> positions;
+[Inject] [ReadOnly] ComponentDataFromEntity<Rotation> rotations;
+[Inject] [ReadOnly] ComponentDataFromEntity<Scale> scales;
+
+struct LocalToWorldGroup
+{
+ EntityArray Entities;
+ ComponentDataArray<LocalToWorld> LocalToWorld;
+}
+[Inject] LocalToWorldGroup localToWorldGroup; 
+```
+
+An advantage of direct chunk iteration is that any branching that needs to be done based on the existance of a particular component type can be done on a per-chunk basis rather than a per-entity basis.
+
+## Querying matching archetypes
+
+Each Chunk belongs to a specific Archetype. In order to iterate Chunks, a set of archetypes must be selected. This is an `EntityArchetypeQuery`.
+
+```
+public class EntityArchetypeQuery
+{
+ public ComponentType[] Any;
+ public ComponentType[] None;
+ public ComponentType[] All;
+}
+```
+
+An example might look like:
+```
+var RootLocalToWorldQuery = new EntityArchetypeQuery
+{
+ Any = new ComponentType[] {typeof(Rotation), typeof(Position), typeof(Scale)}, 
+ None = new ComponentType[] {typeof(Frozen), typeof(Parent)},
+ All = new ComponentType[] {typeof(LocalToWorld)},
+};
+```
+
+Which means RootLocalToWorldQuery will request all archetypes which meet the conditions:
+1. Archetype has at least one of Rotation, Position, or Scale component tyoe.
+2. Archetype does not have Frozen or Parent component types.
+3. Archetype must have LocalToWorld component tyoe.
+
+The query can be resolved by a call to `EntityManager.AddMatchingArchetypes(EntityArchetypeQuery query, NativeList<EntityArchetype> foundArchetypes)`
+
+Additional calls to `AddMatchingArchetypes` passing in the results of previous calls to foundArchetypes will append additional results to the NativeList. (i.e. The logical-or of multiple queries.)
+
+Note that EntityArchetypeQuery uses managed arrays so they should not be created per frame. (OnCreate in a ComponentSystem or JobComponentSystem is more appropriate.)
+
+## Getting array of Chunks
+
+From a `NativeList<EntityArchetype>` the list of Chunks in those archetypes can be retrieved.
+
+This is done by a call to `EntityManager.CreateArchetypeChunkArray(NativeList<EntityArchetype> archetypes, Allocator allocator)` which will return a `NativeArray<ArchetypeChunk>`.
+
+There is also a utility function `EntityManager.CreateArchetypeChunkArray(EntityArchetypeQuery query, Allocator allocator)` which takes a single EntityArchetypeQuery directly and will return a NativeArray<ArchetypeChunk>. (Simplifying the case where no logical-or between multiple EntityArchetypeQuery is needed.)
+
+The caller is responsible for calling Dispose() on the `NativeArray<ArchetypeChunk>`.
+
+An ArchetypeChunk type and by extension a `NativeArray<ArchetypeChunk>` is always read-only. (And should be marked [ReadOnly] when used in jobs.)
+
+However the arrays of component data within those chunks can be retrieved as either read-write or read-only, as needed.
+
+## Accessing component data in chunks
+
+To access data within a chunk, a `ChunkComponentType` is required which represents the specific component type and read-only attribute requested. From within a ComponentSystem or JobComponentSystem, this is retrieved by a call to `GetArchetypeChunkComponentType<T>(bool isReadOnly = false)` which returns a `ArchetypeChunkComponentType<T>`.
+
+For instance, in order to gain read-only access to the Position component data in the Chunks matching the archetypes above: 
+```
+var RotationTypeRO = GetArchetypeChunkComponentType<Rotation>(true);
+```
+Or read-write access to the LocalToWorld component data:
+```
+var LocalToWorldTypeRW = GetArchetypeChunkComponentType<LocalToWorld>(false);
+```
+
+When used in a Job, the \[ReadOnly\] attribute must match the type. e.g.
+```
+ [ReadOnly] public ArchetypeChunkComponentType<Rotation> rotationType;
+ public ArchetypeChunkComponentType<LocalToWorld> localToWorldType;
+```
+
+To retrieve the actual component data for reading or editing, `ArchetypeChunk.GetNativeSlice<T>(ArchetypeChunkComponentType<T> chunkComponentType)` is used, which returns `NativeSlice<T>`
+
+e.g. For a given ArchetypeChunk (chunk), the Position and LocalToWorld data can be retrieved as:
+```
+var chunkPositions = chunk.GetNativeSlice(positionType);
+var chunkLocalToWorlds = chunk.GetNativeSlice(localToWorldType);
+```
+
+Implicit to an `EntityArchetypeQuery` is that every chunk may not have the same components available. In this case, for instance, chunks coming from different archetypes may or may not have Position components. In that case, the length of the returned array will be zero. e.g. Existance of Position component data in a chunk can be confirmed by:
+```
+var chunkPositionsExist = chunkPositions.Length > 0;
+```
+
+For iteration, the number of instances in a chunk can be retrieved with `ArchetypeChunk.Count`.
+
+After confirming existance of the component data for the chunk, the data can be read/written as expected. e.g.
+```
+for (int i = 0; i < chunk.Count; i++)
+{
+ chunkLocalToWorlds[i] = new LocalToWorld
+ {
+ Value = float4x4.translate(chunkPositions[i].Value)
+ };
+}
+```
+
+## Accessing entity data in chunks
+
+Iterating Entity values in chunks is very similar to Components.
+
+The Entity type is requested from within a ComponentSystem or JobComponentSystem by `GetArchetypeChunkEntityType()` with returns a `ArchetypeChunkEntityType`. Entity type is always read-only.
+
+e.g.
+```
+var EntityTypeRO = GetArchetypeChunkEntityType();
+```
+
+Similarly ArchetypeChunkEntityType should always include the \[ReadOnly\] attribute when used in a Job. e.g.
+```
+[ReadOnly] public ArchetypeChunkEntityType entityType;
+```
+
+To retrieve the Entity values given a specific ArchetypeChunk, GetNativeSlice is used as with component data: 
+```
+var chunkEntities = chunk.GetNativeSlice(entityType);
+```
+
+## Accessing SharedComponent (index) data in chunks
+
+SharedComponent data cannot be directly accessed in a chunk. SharedComponent data is not stored in the chunks. However, each archetype contains the index of the specific value of the SharedComponent which is part of its definition, and indexes into the global list of SharedComponent values.
+
+Retrieving the shared component index works very much like retrieving component data. An `ArchetypeChunkSharedComponentType<T>` is returned by a call to `GetArchetypeChunkSharedComponentType<T>()` within a ComponentSystem or JobComponentSystem. 
+
+Like `ArchetypeChunkEntityType`, `ArchetypeChunkSharedComponentType` is always read-only.
+
+The index of the shared component is returned by `ArchetypeChunk.GetSharedComponentIndex<T>(ArchetypeChunkSharedComponentType<T> chunkSharedComponentData)`
+
+e.g.
+```
+var chunkDepthSharedIndex = chunk.GetSharedComponentIndex(depthType);
+```
+
+Where depthType is:
+```
+[ReadOnly] public ArchetypeChunkSharedComponentType<Depth> depthType;
+```
+
+The total number of shared component instances the archetype the chunk belongs to includes can also be retrieved by: `ArchetypeChunk.NumSharedComponents()`
+
+## Accessing SharedComponent data from ArchetypeChunk.GetSharedComponentIndex
+
+Shared component values can be retrieved from EntityManager via `EntityManager.GetAllUniqueSharedComponentData<T>(List<T> sharedComponentValues)`. However, this list is per SharedComponent type. The indices returned by `ArchetypeChunk.GetSharedComponentIndex` refer to the global shared component list and are not per-type.
+
+In order to resolve these indices, a mapping from global to per-type index is needed.
+
+Both the shared component values and the remapping can be retrieved from EntityManager via `EntityManager.GetAllUniqueSharedComponentData<T>(List<T> sharedComponentValues, List<int> sharedComponentIndices)` 
+
+For each `sharedComponentValue[i]`, the `sharedComponentIndices[i]` stores the global index of the shared component value.
+
+e.g.
+```
+var sharedDepths = new List<Depth>();
+var sharedDepthIndices = new List<int>();
+EntityManager.GetAllUniqueSharedComponentData(sharedDepths, sharedDepthIndices);
+
+...
+
+var chunkDepthSharedIndex = chunk.GetSharedComponentIndex(depthType);
+var chunkDepthIndex = sharedDepthIndices.IndexOf(chunkDepthSharedIndex);
+var chunkDepth = sharedDepths[chunkDepthIndex];
+```
+
+Additionally, the total count of all shared components in the global list can be retrieved with `EntityManager.GetSharedComponentCount()`
+
+## Common use in Jobs
+
+It's expected to iterate over `NativeArray<ArchetypeChunk>` in an IJobParallelFor. Then within the Job Execute, for each ArchetypeChunk, based on what Components exist, the code would loop over all the appropriate components. i.e. It's a batch operation.
+
+Additionally:
+
+- `ArchetypeChunkArray.CalculateEntityCount(NativeArray<ArchetypeChunk> chunks)` is a utility which returns the complete entity count for all chunks in the array.
+- Each ArchetypeChunk contains a StartIndex value which is the entity count offset within the NativeArray<ArchetypeChunk>.
+
+e.g.
+```
+struct CollectValues : IJobParallelFor
+{
+ [ReadOnly] public NativeArray<ArchetypeChunk> chunks;
+ [ReadOnly] public ArchetypeChunkComponentType<EcsTestData> ecsTestData;
+
+ [NativeDisableParallelForRestriction] public NativeArray<int> values;
+
+ public void Execute(int chunkIndex)
+ {
+ var chunk = chunks[chunkIndex];
+ var chunkStartIndex = chunk.StartIndex;
+ var chunkCount = chunk.Count;
+ var chunkEcsTestData = chunk.GetNativeSlice(ecsTestData);
+
+ for (int i = 0; i < chunkCount; i++)
+ {
+ values[chunkStartIndex + i] = chunkEcsTestData[i].value;
+ }
+ }
+}
+
+public void TestCollectValues()
+{
+ var query = new EntityArchetypeQuery
+ {
+ Any = Array.Empty<ComponentType>(),
+ None = Array.Empty<ComponentType>(),
+ All = new ComponentType[] {typeof(EcsTestData)}
+ };
+ var chunks = m_Manager.CreateArchetypeChunkArray(query, Allocator.Temp);
+ var ecsTestData = m_Manager.GetArchetypeChunkComponentType<EcsTestData>(true);
+ var entityCount = ArchetypeChunkArray.CalculateEntityCount(chunks);
+ var values = new NativeArray<int>(entityCount, Allocator.TempJob);
+ var collectValuesJob = new CollectValues
+ {
+ chunks = chunks,
+ ecsTestData = ecsTestData,
+ values = values
+ };
+ var collectValuesJobHandle = collectValuesJob.Schedule(chunks.Length, 64);
+ collectValuesJobHandle.Complete();
+ chunks.Dispose();
+
+ // Use values here...
+
+ values.Dispose();
+}
+```
+
+## Change Versions
+
+When a type is changed within a chunk, the version for that type within the chunk is assigned to the EntityManager.GlobalSystemVersion. By comparing the version number of a type within a chunk to the current GlobalSystemVersion in a ComponentSystem or JobComponentSystem, it can be inferred whether or not the type values have changed.
+
+Special value: If the version number is zero, the chunk is new.
+
+The version number is returned by `ArchetypeChunk.GetComponentVersion<T>(ArchetypeChunkComponentType<T> chunkComponentType)` e.g. `chunk.GetComponentVersion(positionType)`
+
+Utilities are provided to compare version numbers and determine change:
+
+`ChangeVersionUtility.DidChange(uint changeVersion, uint requiredVersion)` given the chunk type version number and the expected system version, respectively, will return whether or not the specified type in the chunk has been changed. New chunks will return false.
+
+`ChangeVersionUtility.DidAddOrChange(uint changeVersion, uint requiredVersion)` given the chunk type version number and the expected system version, respectively, will return whether or not the specified type in the chunk has been changed or is new.
+
+By way of example, in this case a chunk is skipped if no change was made to the specified types since the last iteration:
+```
+var chunkRotationsChanged = ChangeVersionUtility.DidAddOrChange(chunk.GetComponentVersion(rotationType), lastSystemVersion);
+var chunkPositionsChanged = ChangeVersionUtility.DidAddOrChange(chunk.GetComponentVersion(positionType), lastSystemVersion);
+var chunkScalesChanged = ChangeVersionUtility.DidAddOrChange(chunk.GetComponentVersion(scaleType), lastSystemVersion);
+var chunkAnyChanged = chunkRotationsChanged || chunkPositionsChanged || chunkScalesChanged;
+
+if (!chunkAnyChanged)
+ return;
+```