feat(bitnet/model): implement model weights and tokenizer integration

[thejhh]

Changes

• Added tokenizer support with BPE (Byte Pair Encoding) implementation
• Updated model loader to handle GGUF format
• Added comprehensive tests and benchmarks for both tokenizer and model loader
• Created download script for model weights and tokenizer
• Fixed path handling in model loader and tokenizer

Test Coverage

• Current coverage: 77.9% for model package
• Coverage changes: → 77.9%
• Untested areas:
  • Internal config package (0% coverage)
  • Math operations package (0% coverage)

Performance Metrics

Memory Usage

• Allocations per operation:
  • Tokenizer operations: 0-2 allocs/op
  • Model loading: 1-3 allocs/op
  • Chunk loading: 1 alloc/op

CPU Performance

• Operation timing:
  • Tokenization: ~100ns/op
  • Model loading: ~1ms/op
  • Chunk loading: ~10μs/op

Areas for Improvement

High Priority

• Add tests for internal packages
• Optimize tokenizer memory usage
• Implement memory pooling for large tensors

Medium Priority

• Improve error handling in model operations
• Add more comprehensive benchmarks
• Enhance documentation

Low Priority

• Consider SIMD optimizations
• Add more tensor operations
• Improve test organization

[thejhh]

Fix review comments

[thejhh]

We need to make changes I've described in the full review comments.

[thejhh]

Error handling requires some work still. See full review comments.

[thejhh]

We need to test all code:

?   	github.com/hyperifyio/gnd/pkg/bitnet/internal/config	[no test files]
?   	github.com/hyperifyio/gnd/pkg/bitnet/internal/math	[no test files]

[thejhh]

Some tests are also failing:

=== RUN   TestNewTokenizerErrors/file_not_found
--- FAIL: TestNewTokenizerErrors (0.00s)
    --- PASS: TestNewTokenizerErrors/nil_filesystem (0.00s)
    --- FAIL: TestNewTokenizerErrors/empty_model_path (0.00s)
    --- PASS: TestNewTokenizerErrors/file_not_found (0.00s)

[thejhh]

This is failing too:

=== RUN   TestNewModelLoaderErrors/file_not_found
--- FAIL: TestNewModelLoaderErrors (0.00s)
    --- PASS: TestNewModelLoaderErrors/nil_filesystem (0.00s)
    --- FAIL: TestNewModelLoaderErrors/empty_model_path (0.00s)
    --- PASS: TestNewModelLoaderErrors/file_not_found (0.00s)

[thejhh]

Still some review comments unresolved.

Also fix missing tests:

?   	github.com/hyperifyio/gnd/pkg/bitnet/internal/config	[no test files]
?   	github.com/hyperifyio/gnd/pkg/bitnet/internal/math	[no test files]

[Copilot]

Pull Request Overview

This PR restructures and enhances the BitNet project’s .cursor/rules documentation by removing outdated guidelines, adding new standardized procedures, and expanding workflow and benchmark guidance.

• Removed deprecated rule files.
• Introduced new PR update, review, and creation templates.
• Updated project overview, environment setup, branching strategy, and benchmarking guidelines.

Reviewed Changes

Copilot reviewed 38 out of 38 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
.cursor/rules/bitnet-pr-update-procedures.mdc	New standardized PR update procedures
.cursor/rules/bitnet-pr-review-workflow.mdc	Added PR review workflow guidelines
.cursor/rules/bitnet-pr-creation-description.mdc	Added PR creation & description format template
.cursor/rules/bitnet-overview.mdc	Updated high-level project overview and goals
.cursor/rules/bitnet-environment.mdc	Updated development environment setup instructions
.cursor/rules/bitnet-branching-strategy.mdc	Defined branch creation and naming conventions
.cursor/rules/bitnet-benchmarks.mdc	Defined benchmark file placement and naming rules
.cursor/rules/bitnet-benchmark-invocation.mdc	Added benchmark invocation and profiling commands
.cursor/rules/bitnet-benchmark-categories.mdc	Introduced benchmark semantic categories
.cursor/rules/bitnet-benchmark-analysis.mdc	Added guidance for interpreting and tracking results

Comments suppressed due to low confidence (8)

[thejhh]

We still have broken tests:

--- FAIL: TestTokenize (0.00s)
    --- PASS: TestTokenize/known_words (0.00s)
    --- FAIL: TestTokenize/unknown_word (0.00s)
    --- PASS: TestTokenize/empty_text (0.00s)

We also are missing tests for pkg/bitnet/internal/math/ops.go and pkg/bitnet/internal/config/config.go.

[thejhh]

Ok , finally we can move on :)