Skip to content

188 stack transformer blocks - #216

Merged
thejhh merged 44 commits into
bitnetfrom
188-stack-transformer-blocks
May 23, 2025
Merged

188 stack transformer blocks#216
thejhh merged 44 commits into
bitnetfrom
188-stack-transformer-blocks

Conversation

@thejhh

@thejhh thejhh commented May 22, 2025

Copy link
Copy Markdown
Contributor

Changes

  • Implemented transformer block stacking functionality in the BitNet model
  • Added final normalization layer with proper weight handling
  • Enhanced tensor operations with improved thread safety and memory management
  • Added comprehensive documentation for tensor package
  • Improved error handling and validation in tensor operations
  • Added new tensor operations: Transpose, Repeat, and Add
  • Optimized memory allocations and parallel processing in BitLinear operation
  • Added debug logging for better observability

Test Coverage

  • Current coverage: 88.9%
  • Coverage changes: 88.6% → 88.9%

Areas for Improvement

High Priority

Medium Priority

Low Priority

Implementation Details

  • Added final normalization layer with proper weight conversion from int8 to float32
  • Enhanced tensor operations with improved thread safety using mutex locks
  • Implemented efficient parallel processing in BitLinear operation
  • Added comprehensive documentation for tensor package and its operations
  • Improved error handling with proper validation and panic messages
  • Added new tensor operations for better flexibility in model operations

Closes #188

@thejhh thejhh linked an issue May 22, 2025 that may be closed by this pull request
@thejhh
thejhh changed the base branch from main to bitnet May 22, 2025 20:57
@thejhh thejhh added the bitnet BitNet implementation label May 22, 2025
@thejhh thejhh self-assigned this May 22, 2025
Jaakko Heusala added 17 commits May 23, 2025 00:02
- Fix tensor reshaping logic in QKV projection\n- Maintain proper int8 type consistency throughout\n- Improve tensor data copying efficiency\n- Fix invalid shape dimension error
- Update attention and FFN layer implementations\n- Add test data for model validation\n- Modify tensor operations and model tests\n- Update BitNet-related scripts
Replace direct logging with structured logging using loggers package for better debugging and consistency
- Replace direct fmt/os usage with loggers package\n- Remove unused ErrInvalidHeadDimension error
- Add specific error types for attention and tensor operations\n- Replace generic error wrapping with typed errors\n- Add debug logging before error returns\n- Remove unused fmt import
@thejhh
thejhh marked this pull request as ready for review May 23, 2025 12:09
Jaakko Heusala added 19 commits May 23, 2025 19:47
- Improve attention output computation efficiency\n- Enhance FFN sublayer performance\n- Update model tests for optimized operations\n- Refine sublayer normalization implementation
@thejhh
thejhh merged commit 9ba4b49 into bitnet May 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bitnet BitNet implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stack Transformer Blocks

1 participant