176 set model constants architecture hyperparameters

[thejhh]

Changes

• Updated model architecture constants in pkg/bitnet/internal/config/config.go to match BitNet b1.58-2B specification:
  • Set HiddenSize to 2560
  • Set IntermediateSize to 6912
  • Set NumHiddenLayers to 30
  • Set NumAttentionHeads to 20
  • Set NumKeyValueHeads to 5
  • Set MaxPositionEmbeddings to 4096
  • Added HiddenAct as "relu2" for squared ReLU activation
  • Added NormType as "rms" for RMS normalization
  • Added RMSNormEps as 1e-6 for RMS normalization epsilon

Test Coverage

• Current coverage: 83.0%
• Coverage changes: 83.0% → 83.0%

Performance Metrics

Memory Usage

Tensor Operations

• Allocations per operation:
  • New tensor creation: 120 allocs/op
  • Get/Set operations: 0 allocs/op
  • Parallel operations: 160749 allocs/op

BitNet Model Operations

• Allocations per operation:
  • Model weights loading: N/A allocs/op (TODO Implement BitLinear Layer #178)
  • Model inference: N/A allocs/op (TODO Token Decoding (Inference Loop) #190)
  • Ternary weights reading: N/A allocs/op (TODO Implement BitLinear Layer #178)

CPU Performance

Tensor Operations

• Operation timing:
  • Basic operations: 11.87 ns/op
  • Parallel operations: 97795 ns/op
  • Large tensor operations: 1068 ns/op

BitNet Model Operations

• Operation timing:
  • Model weights loading: N/A ns/op (TODO Implement BitLinear Layer #178)
  • Model inference: N/A ns/op (TODO Token Decoding (Inference Loop) #190)
  • Ternary weights reading: N/A ns/op (TODO Implement BitLinear Layer #178)

Areas for Improvement

High Priority

• Optimize memory allocations in model operations (TODO Parallelize with Goroutines #191)
• Implement proper self-attention (TODO Integrate Attention Sublayer (Pre-Norm & Residual) #186)

Medium Priority

• Improve error handling in model operations (TODO Testing & Performance Tuning #192)
• Add more comprehensive benchmarks (TODO Testing & Performance Tuning #192)
• Enhance documentation
• Implement proper feed-forward network (TODO Integrate Feed-Forward Sublayer (Pre-Norm & Residual) #187)

Low Priority

• Consider SIMD optimizations (TODO Parallelize with Goroutines #191)
• Add more model operations (TODO Token Decoding (Inference Loop) #190)
• Improve test organization (TODO Testing & Performance Tuning #192)
• Implement proper output generation (TODO Final Output Layer (LM Head) #189)

Closes #176

[thejhh]

Looks fine