C++ Example

The following steps can be followed to create a minimal C++ end to end setup using realtime Whisper transcribed text and performing inference on the text with Flora Engine models.

1. Create a C++ Actor from the content browser in Unreal Engine
2. In the .Build.cs of the C++ project add "FloraEngine" and "WhisperASR" to PublicDependencyModuleNames
3. In your Actor's header file add #include "FloraEngineSubsystem" and #inclue "WhisperSubsystem" in the .cpp file
4. Add a UNLM* variable and UFloraEngineSubsystem* to hold references to the loaded model and subsystem for ease of use
5. Create the function void OnWhisperProcessed(const TArray<FString>& TranscribedText) and add the macro UFUNCTION() above it
6. In the BeginPlay() in the Actor's .cpp file get a reference to the FloraEngineSubsystem and initialize it as well as the NLM you would like to perform inference on
7. Get a reference to the WhisperSubsystem and initialize it with WhisperInit()
8. Then bind OnWhisperProcessed to OnRealtimeWhisperProcessed in the subsystem and call RealtimeCaptureStart(true, false, "")
9. Create your OnWhisperProcessed function body
10. Append the transcribed text together into one string
11. Create a FModelAsyncPrompt* with WorldContextObject set to your actor, Model set to your NLM*, Prompt set to the transcribed string and InstructionLine set to the instruction in the .json template file that you wish to use (for .csv models, "StateMachineInstruction" must be used)
12. Call AsyncInfer on the FloraEngineSubsystem using the FModelAsyncPrompt. A function delegate must be provided that will run on the game thread after inference
13. After inference is complete, call GetOutput with the created model to retrieve the output and reaction

void AMyActor::BeginPlay()
{
    Super::BeginPlay();

    FloraSubsystem = GetGameInstance()->GetSubsystem<UFloraEngineSubsystem>();
    FloraSubsystem->Init();
    Model = FloraSubsystem->InitNLM(this, "ModelName", TArray<FString>());

    UWhisperSubsystem* WhisperSubsystem = GetGameInstance()->GetSubsystem<UWhisperSubsystem>();
    WhisperSubsystem->WhisperInit();
    WhisperSubsystem->OnRealtimeWhisperProcessed.AddDynamic(this, &AMyActor::OnWhisperProcessed);
    WhisperSubsystem->RealtimeCaptureStart(true, false, "");
}

void AMyActor::OnWhisperProcessed(const TArray<FString>& TranscribedText)
{
    FString FullText;
    for (const FString& Text : TranscribedText)
    {
        FullText.Append(Text);
    }

    FModelAsyncPrompt* Prompt = new FModelAsyncPrompt{
				.WorldContextObject = this,
				.Model = Model,
				.Prompt = FullText,
				.InstructionLine = "StateMachineInstruction"
};

    FloraSubsystem->AsyncInfer(Prompt, [this, Prompt]() {
        uint8 InstructionIndex, ReactionIndex;
        FString Output, Reaction;
        float TokenSpeed;
        FloraSubsystem->GetOutput(this, Prompt->Model, Output, InstructionIndex, ReactionIndex, Reaction, TokenSpeed);

        UE_LOG(LogTemp, Warning, TEXT("Output: %s"), Output);
        UE_LOG(LogTemp, Warning, TEXT("Reaction: %s"), Reaction);
        });
}