DataGator

What Makes DataGator Unique?

• 🧠 Forensic Precision — Cryptographically fingerprints every file, stream, and path segment using MD5 + SHA256
• 🧬 Unicode-Aware Indexing — Handles normalization, control chars, raw vs canonical paths, and metadata-safe comparisons
• 🔁 Deduplication Engine — Detects and deduplicates identical content across nested archives, volumes, and snapshots
• 📁 Composite Archive Scanning — Recursively explores .zip, .iso, .dmg, .vhd, MDOS/QIC, even inside disk images
• 🧾 Time-Stamped, Immutable Snapshots — For audit logs, forensic trails, backup validation, and SBOM generation
• 📊 Human-Friendly Dashboards — Local HTML summaries of indexed content, snapshots, diffs, and drift
• 🔐 ACL-Aware Permissions — Captures and compares ACLs across NTFS, APFS, ext4, and NFSv4
• 🧪 Test-Driven Shell & AWK Codebase — Every core module ships with unit tests for sanity and longevity

🐊 DataGator Core vs Pro

DataGator Core is built for power users, researchers, and forensic tinkerers who want full control using CLI tools and containers.
Pro adds real-time indexing, dashboards, syncing, and team workflows — built for compliance, operations, and distributed environments.

Use Cases Across Domains

• 🕵️ Digital forensics and chain-of-custody proof
• 💾 Smart backup with dedup + drift detection + rehydration
• 👩‍⚖️ Legal compliance (21 CFR Part 11, GDPR, audit diffs)
• 📦 Container and firmware reverse engineering
• 🧠 Academic and AI dataset lineage attestation
• 📚 Long-term archival + digital preservation (NARA-grade)
• 🔍 Reproducible build tracing (distro package walker)

Comparison: DataGator vs the World

• ✅ Versioned Index — Unlike rsync or basic backup tools, DG tracks exact changes and content hashes
• ✅ Cross-Platform — POSIX-native, works in containers, bind-mounts, and loopback devices
• ✅ Filesystem-Agnostic — NTFS, ext4, APFS, HFS+, FAT, ZFS, raw .img/.dsk/.iso — all ingestible
• ✅ Audit-Ready — All snapshots include cryptographic identity, timestamp, and provenance chains
• ❌ No Vendor Lock-In — 100% local, no cloud required, open output formats

Get Started in 3 Steps

All you need is a bind-mount and Docker or Podman.

docker run --rm \
  -v /your/data:/mnt:ro \
  -v ~/.DataGator.id:/etc/DataGator.id:ro \
  datagator scan /mnt

Want to detect changes over time?

./drift_detector.sh

Need a web report?

open __postgresql/dashboard/index.html

How DataGator Sees Your Filesystem

DataGator doesn’t just index file paths — it models your storage with cryptographic and structural fidelity.

• 📁 Canonical Paths: Paths are stored as UTF-8 encoded, percent-escaped, and normalized segments
• 🔡 PUA-stripping and normalization: Ensures consistency across case/encoding quirks
• 🔍 Hashing Policy: Each file, segment, stream, and directory gets an MD5 + SHA256 identity
• 🧬 Raw vs Canonical: Stores both for forensic transparency

Test-Driven from Day One

Every core shell or AWK module in DataGator is paired with a unit test. No test, no merge. Run everything at once:

find scripts -name '*_test.sh' -exec bash {} \;

This ensures stability across updates, formats, and edge cases.

Our Philosophy

“We do not store files — we store truth.
Names are not paths — they are sequences of intention.”
— DataGator Design Memo #001

Edge Cases and Real-World Gotchas

• 🔣 Filenames with control characters are safely encoded, raw form preserved
• 📂 File deletions are never removed — they're tracked as disappeared
• 📑 NTFS alternate data streams are treated as child files
• 🧲 Snapshots and USB mounts are treated as distinct indexable volumes
• 🔀 Sync logic maps chunk hashes and path segments, not whole files

Who Is DataGator For?

• 🛠 Home Lab Architects — Multi-device ZFS setups, Pi clusters, NAS sprawl
• 🧪 Researchers & Developers — Indexing legacy data and academic corpora
• 🔍 Digital Forensics Pros — Path+hash snapshots for provable evidence trails
• 💾 Backup & Archival Tinkerers — Smart deduping, drift detection, and restoration
• 🧑‍⚖️ Compliance Officers — 21 CFR Part 11, GDPR, and audit traceability

What DataGator Replaces

• 🛠 Dev Environments — replaces Dockerfiles and Nix with rehydrated snapshots
• 📥 Package Managers — segment-based deduped installs instead of dependency hell
• 🧾 SBOM Tools — cryptographic manifests at source, not post-facto guesswork
• 🔐 Vaults/KMS — signed secrets with ACL lineage and timestamped provenance
• 🧠 ML Registries — hash-chain verified corpora and models for reproducibility
• 📦 Backup/Sync Tools — smarter than rsync, more forensic than NAS snapshots

CLI Command Overview

Built for Compliance and Courtroom

• 📅 Immutable, time-sealed manifests for 21 CFR Part 11, GDPR
• 🔏 Signed content blobs for tamper-evident trails
• 📜 Full ACL lineage and permission diffs for NTFS/APFS/ext4
• 🧾 Court-admissible snapshots with hash+path chains
• 📈 Machine-verifiable diff reports over time

Beyond Indexing: What Sets DataGator Apart

• 🧬 Recursive archive extraction inside disk images and formats like QIC/MDOS
• 🧊 Sparse file detection using native system APIs
• 🔀 Overlay fingerprinting for containers, VMs, and firmware
• 🔡 Metadata normalization for broken encodings and legacy filesystems
• 🌡️ Entropy heatmaps to visualize compressed or encrypted regions

AI Dataset Integrity & Model Provenance

• ✅ Legal authorship tracking with input → model → export hashes
• ✅ GDPR/EU AI Act compliance via full source traceability
• ✅ Poisoned dataset detection with diffable manifest snapshots
• ✅ Reproducibility for academic or benchmark use

Extend DataGator with Custom Parsers

Want to analyze a weird format or ancient archive? You can build your own extractor.

• 📦 Add support with infogator add-parser my_format.py
• 📜 Use declarative format schemas for reproducible exports
• 🔌 Join a growing community of open-source plugin developers

Future & Expansion Plans

• 🌐 Web UI Explorer with overlay tree viewer
• 🧠 AI summarization of suspicious archives or entropy zones
• 📊 PostgreSQL/SQLite backend for diffable index snapshots
• 🧬 FUSE-based virtual mount of indexed archives (planned)
• 🔥 Entropy heatmap per file and segment (with SVG export)