Ingestion & Semantic Reconciliation Protocol

Applies to the 11 official shallow ZIP inputs. This protocol is deterministic-first and loss-averse.

0. Input authority

The outer transport container is not an input package. Exactly 11 shallow inner ZIPs are the official current input set.

No final artifact may be generated until all 11 have a physical identity and every recursively discovered manifestation has a terminal disposition.

1. Archive safety before extraction

For outer and every nested archive:

  1. compute SHA-256 of archive bytes;
  2. verify archive structure/CRC where format supports it;
  3. inventory entries without extracting;
  4. reject/flag absolute paths, .., drive paths, ADS/colon hazards, symlink/special-device entries, Windows reserved names, trailing dot/space, duplicate paths, Unicode normalization collision and case-fold collision;
  5. enforce entry count, total uncompressed size, per-entry size and suspicious compression-ratio controls;
  6. use clean staging directory; never extract over an existing authoritative tree;
  7. hash archive before and after validation/extraction to detect archive TOCTOU.

Archive failure becomes evidence and disposition; it is not silent omission.

2. Manifestation-level ledger

Every physical file occurrence receives its own manifestation_id even when its bytes duplicate another file.

Required fields: source_zip_id, nested_archive_chain, original_path, original_filename, sha256, bytes, mime_guess, extension, detected_format, encoding_state, parse_state, read_state, semantic_family, topics, lineage, exact_duplicate_group, semantic_cluster, conflict_group, canonical_object_id, destination_package, destination_path, transformation, applied_state, test_state, manual_destination, provenance, disposition.

Invariant: CANONICAL_OBJECT_COUNT != SOURCE_COVERAGE.

Deduplication may reduce canonical content objects but never source manifestations or provenance paths.

3. Actual-format detection

Do not trust filename extension alone.

Use, in order when practical:

  1. magic/header bytes;
  2. container internals;
  3. structural parser probe;
  4. extension;
  5. MIME guess.

Record EXTENSION_FORMAT_MISMATCH instead of silently renaming originals.

Original bytes and source name stay preserved. Normalized derivatives get separate IDs and provenance.

4. Encoding protocol

UTF8_DECODE_FAILURE != FILE_CORRUPTION.

For textual candidates:

  1. BOM-aware UTF-8/UTF-16/UTF-32 probe;
  2. strict UTF-8;
  3. format parser on candidate decode;
  4. controlled CP1252/Latin-1 compatible fallback when evidence supports it;
  5. record detected/ambiguous encoding and confidence;
  6. binary/unreadable remains byte-read with explicit parse blocker.

Do not lossy-replace undecodable bytes to manufacture a successful parse.

5. Recursive nested-archive exhaustion

For each relevant archive manifestation: archive -> inventory -> nested archive discovery -> safe recurse -> member ledger.

No artificial depth cutoff. Resource controls may pause/checkpoint a branch, but do not redefine deep entries as absent.

Detect repeated-content archive recursion by archive hash lineage to avoid cycles/redundant work.

6. Deterministic analysis before LLM semantic reading

PHYSICAL INVENTORY -> HASH/NORMALIZED HASH -> STRUCTURAL PARSE -> EXACT DUP GROUPS -> STRUCTURAL FINGERPRINTS -> VERSION/LINEAGE SIGNALS -> WRITER/IMPORT/REFERENCE GRAPH -> STATIC DEFECT SCAN -> PRIORITY QUEUE -> SEMANTIC REASONING.

Never use an LLM for hashing, counting, joins or deterministic parsing when code can do it reliably.

7. Read-state machine

States remain independent: FOUND -> CENSUSED -> BYTE_READ -> PARSED -> SEMANTICALLY_READ -> CLASSIFIED -> RECONCILED -> APPLIED -> TESTED -> CURRENT.

A file may be BLOCKED_WITH_REASON at any stage, but never silently disappear.

FOUND != READ != UNDERSTOOD != CLASSIFIED != APPLIED != TESTED.

8. Duplicate model

EXACT_DUPLICATE

Same SHA-256. Share canonical content if useful; retain every manifestation/path.

SEMANTIC_DUPLICATE

Different bytes but materially same responsibility. Merge only after comparing purpose, scope, triggers, preconditions, authority, workflow, outputs, verification, failures, stop conditions, examples, platform semantics and version lineage.

Overlap exists but responsibilities differ. Preserve separate identities with typed relation.

VERSIONED_SUPERSESSION

Newer/current object supersedes older authority while older evidence remains available in provenance/history.

CONFLICT

Material claims/semantics disagree. Preserve both, identify authority/currentness, and resolve explicitly or keep unresolved state.

Never merge by filename, topic label or embedding proximity alone.

9. Canonical semantic object

Minimum fields: canonical_id, canonical_name_ptbr, english_term, object_class, definition, scope, what_it_is, what_it_is_not, aliases, positive_examples, counterexamples, parents, children, related_objects, source_manifestations, platform_variants, lineage, conflicts, authority_state, currentness_state, operational_destinations.

10. Semantic role precedes file format

Determine responsibility before projecting a platform-specific file.

Examples of distinct roles: persistent instruction/context, Skill, Rule/Policy, Agent/Subagent, Hook, Workflow/Command, MCP config, Memory, Retriever index, Validator, Test, Template, Generator, Ledger, Receipt, Report, Status Doc.

PLATFORM_FILE_FORMAT != SEMANTIC_ROLE.

Adapters are generated from canonical roles; canonical knowledge is not mutated to mimic one vendor filesystem.

11. Source-to-operation closure

For every material reusable concept: SOURCE -> CONCEPT -> WHAT IT CHANGES -> COMPONENT CLASS -> LIVE COMPONENT -> CONSUMER -> ROUTER/TRIGGER -> ENFORCEMENT -> DETECTOR -> TEST -> MUTATION -> RUNTIME EVIDENCE -> STATUS.

Reverse audit: ACTIVE COMPONENT -> WHY IT EXISTS -> SOURCE KNOWLEDGE -> CONTRACT -> CONSUMER -> TRIGGER -> OUTPUT -> ENFORCEMENT -> DETECTOR -> EVIDENCE.

Orphans remain REVIEW/UNKNOWN and block unrestricted completeness claims.

12. Non-regression

Compare current input/final packages against historical version-scoped baselines.

Cardinality is a signal, not a completion oracle. A smaller canonical representation may pass only when:

  • every source manifestation is dispositioned;
  • provenance remains complete;
  • material deep knowledge is local/referenced/routable;
  • routing/retrieval behavior is proven;
  • removed/superseded identities have explicit crosswalk;
  • no operational capability disappears without justified disposition.

13. Currentness

Historical provider/platform syntax never proves current adapter behavior.

Separate: DOC_CURRENT -> BINARY_INSTALLED -> CONNECTED -> AUTHENTICATED -> AUTHORIZED -> ENTITLED -> QUOTA_AVAILABLE -> RUNTIME_PROBED -> VERIFIED_FOR_TASK.

Adapter generation uses current official docs; runtime claims require actual probes when the runtime is available.

14. Incremental checkpoints

Persist after each atomic phase: INPUT_CENSUS, PARSE, SEMANTIC_READ, RECONCILIATION, ARCHITECTURE, IMPLEMENTATION, TEST, PREFREEZE, FREEZE, CLEANROOM, FINAL_RECONCILIATION.

A timeout/stream interruption after a persisted phase is not task failure and does not justify repeating completed work.

15. Terminal condition

Zero silent omissions means:

  • every official ZIP identified;
  • every recursive manifestation terminally dispositioned;
  • all current-input bytes covered by source ledger;
  • all canonical objects source-bound;
  • all material conflicts resolved or explicitly blocking;
  • operational destinations/test evidence closed where applicable;
  • final packages and manual reconcile back to source manifestations.