Class AggregateBatchingPool

Class Documentation

class AggregateBatchingPool

Aggregates request queueing and generation of batches from multiple TranslationModels (BatchingPools within, specifically), thereby acting as an intermediary to enable multiple translation model capability in BlockingService and AsyncService.

A simple queue containing shared owning references to TranslationModels are held here from which batches are generated on demand. Since a queue is involved, the ordering is first-come first serve on requests except there are leaks effectively doing priority inversion if an earlier request with the same TranslationModel is pending to be consumed for translation. Actual storage for the request and batch generation are within the respective TranslationModels, which owns its own BatchingPool.

Matches API provided by BatchingPool except arguments additionally parameterized by TranslationModel.

Note: This class is not thread-safe. You may use this class wrapped with ThreadsafeBatchingPool for a thread-safe equivalent of this class, if needed.

Public Functions

AggregateBatchingPool()

Create an AggregateBatchingPool with (tentatively) global (across all BatchingPools) limits imposed here.

size_t enqueueRequest(Ptr<TranslationModel> model, Ptr<Request> request)

Enqueue an existing request onto model, also keep account of that this model and request are now pending.

Return

number of sentences added for translation.

Parameters

  • [in] model: Model to use in translation. A shared ownership to this model is accepted by this object to keep the model alive until translation is complete.

  • [in] request: A request to be enqueued to model.

size_t generateBatch(Ptr<TranslationModel> &model, Batch &batch)

Generate a batch from pending requests, obtained from available TranslationModels.

Return

Number of sentences in the generated batch.

Parameters

  • [out] model: TranslationModel

  • [out] batch: Batch to write onto, which is consumed at translation elsewhere.

void clear()

Clear the aggregate queue.

Does not clear the underlying model/request pairs but the next call to generateBatch() will return 0. (Unless enqueueRequest() was called in the mean time.)