48 lines
1.7 KiB
ReStructuredText
48 lines
1.7 KiB
ReStructuredText
pipeline\_parallel package
|
|
==========================
|
|
|
|
This package contains implementations for two different pipeline parallelism
|
|
schedules (one without interleaving and one with interleaving, see `Efficient
|
|
Large-Scale Language Model Training on GPU Clusters Using Megatron-LM <https://arxiv.org/abs/2104.04473>`_
|
|
for details), and a default no-pipelining schedule. It also contains methods
|
|
for the point-to-point communication that is needed between pipeline stages.
|
|
|
|
Submodules
|
|
----------
|
|
|
|
pipeline\_parallel.p2p\_communication module
|
|
--------------------------------------------
|
|
|
|
Contains implementations for the various point-to-point communication needed
|
|
(e.g., `recv_forward` and `recv_backward`) in the different pipeline parallelism
|
|
schedules.
|
|
|
|
.. automodule:: core.pipeline_parallel.p2p_communication
|
|
:members:
|
|
:undoc-members:
|
|
:show-inheritance:
|
|
|
|
pipeline\_parallel.schedules module
|
|
-----------------------------------
|
|
|
|
Contains implementations for two pipeline parallelism schedules
|
|
(`forward_backward_pipelining_with_interleaving`for pipeline parallelism with
|
|
interleaving, `forward_backward_pipelining_without_interleaving` for pipeline
|
|
parallelism without interleaving) and a default no-pipelining schedule
|
|
(`forward_backward_no_pipelining`). `get_forward_backward_func` returns the right
|
|
scheduling function to use based on the configuration being trained
|
|
(e.g., if pipeline-parallel size is 1, use `forward_backward_no_pipelining`).
|
|
|
|
.. automodule:: core.pipeline_parallel.schedules
|
|
:members:
|
|
:undoc-members:
|
|
:show-inheritance:
|
|
|
|
Module contents
|
|
---------------
|
|
|
|
.. automodule:: core.pipeline_parallel
|
|
:members:
|
|
:undoc-members:
|
|
:show-inheritance:
|