Skip to content
SimpleLLaMA Documentation
Notes
Initializing search
SimpleLLaMA Documentation
Home
Pretraining
Pretraining
Overview
Dataset Preparation
Tokenization
Tokenization
Overview
Algorithms
Examples
Project Usage
Model Architecture
Model Architecture
Overview
Embeddings
Attention
Normalization
FeedForward
Layer Block
End To End
Training Process (Beginner)
Training Process (Beginner)
Introduction
Model Configurations
Dataset and Batching
Scheduler
Training Loop
Training Process (Advanced)
Training Process (Advanced)
Basics Recap
Optimizer Details
Cross Entropy Loss
Gradient Accumulation
Distributed Data Parallel
Throughput Optimizations
Checkpointing and Evaluations
Final Walkthrough
Supervised Fine-Tuning
Supervised Fine-Tuning
Overview
Dataset
Prompt Formatting
Utilities
Finetuning Process
RLHF
RLHF
Overview
Preference Optimization
Performance & Applications
Performance & Applications
Benchmarking
Inference
Notes
Custom Training
Custom Training
Pretraining
Supervised Fine Tuning
RLHF
Additional Notes
NA