heterogeneous-distributed-training-framework/docs/source/api-guide/models.gpt.rst

models.gpt package

This is the implementation of the popular GPT model. It supports several features like model parallelization (Tensor Parallel, Pipeline Parallel, Data Parallel) , mixture of experts, FP8 , Distributed optimizer etc. We are constantly adding new features. So be on the lookout or raise an issue if you want to have something added.

Submodules

models.gpt.gpt_model module

core.models.gpt.gpt_model

Module contents

core.models.gpt