Understanding the Geneformer Algorithm

• 1. Motivation
• 2. Background
  • 2.1 Single-Cell Transcriptomics
  • 2.2 Transformer Models
  • 2.3 Why Foundation Models for Biology
• 3. Core Idea
• 4. Input Representation
  • 4.1 Raw Expression Data
  • 4.2 Tokenization Strategy
  • 4.3 Sequence Construction
• 5. Model Architecture
  • 5.1 Embedding Layer
  • 5.2 Transformer Encoder
  • 5.3 Output Representations
• 6. Training Objective
  • 6.1 Pretraining Task
  • 6.2 Fine-Tuning Tasks
• 7. Inference Workflow
• 8. In Silico Perturbation
• 9. Strengths
• 10. Limitations
• 11. Comparison With Related Methods
• 12. Practical Notes
  • 12.1 Data Requirements
  • 12.2 Compute Requirements
  • 12.3 Reproducibility Checklist
• 13. Summary
• References

1. Motivation

2. Background

2.1 Single-Cell Transcriptomics

2.2 Transformer Models

2.3 Why Foundation Models for Biology

3. Core Idea

4. Input Representation

4.1 Raw Expression Data

4.2 Tokenization Strategy

4.3 Sequence Construction

5. Model Architecture

5.1 Embedding Layer

5.2 Transformer Encoder

5.3 Output Representations

6. Training Objective

6.1 Pretraining Task

6.2 Fine-Tuning Tasks

7. Inference Workflow

8. In Silico Perturbation

9. Strengths

10. Limitations

11. Comparison With Related Methods

12. Practical Notes

12.1 Data Requirements

12.2 Compute Requirements

12.3 Reproducibility Checklist

13. Summary

References