Inference Information Reference

This chapter is for understanding the inference SDK more in depth.

Section contents are usually structured this way:

1. Giving a brief summary and remarks about the topic if the topic is universal and not specific to the inference SDK.

2. Giving a detailed explanation about the topic if the topic is specifically related to the inference SDK.

3. Showing how the inference SDK is related to the topic, useful objects and their locations and brief examples of how to use it.

4. Short SDK usage summary.

5. Header file synopsis.

Sections

• SDK Structure
  • File Naming and Purpose
  • Dependencies
  • Finding the SDK
  • Inference Part
    • Naming
  • GGUF Configuration Part
  • Common Part
• Parallel State Machine
  • About
  • Identifying Expensive Operations
  • SDK Location
    • Processor Signal Object
    • Logical Processor Object
  • Flow of Operations
  • Applied Example
    • Timing Diagram
• Inference Workflow in General
• Obtaining Hardware Information
  • Device Description Object
  • Strange Device Types
  • Internal Device Handle
  • SDK Usage
  • Header Synopsis
• About GGUF Files
  • Quantization Support
  • Important Metadatas
    • General Metadata
    • Architectural Metadata
  • Converting to GGUF
    • Acquiring the Model
    • Using the Conversion Script
    • Remarks on GGUF Naming
  • SDK Usage
    • Reading Key/Value
      • Displaying General Metadata
      • Displaying Architectural Metadata
      • Displaying All Strings
    • Writing Key/Value
  • Header Synopsis
• Message Preparation
  • About Chat Templates
    • Sophisticated Templates
  • Tokenization and Input Preperation
    • Context Lines
  • SDK Usage
  • Header Synopsis
    • Context line
    • Hard-coded Templates
• Model Object in Detail
  • Naming
  • Identifying the Expensive Operations
  • Essential Callbacks
  • Essential Signals
  • General Operation Workflow
    • Initialization Example
    • Processor Registration Example
  • Synopsis
    • Base Model Object
      • begin()
      • begin()
      • end()
      • begin()
      • end()
      • cbegin()
      • cend()
      • rbegin()
      • rend()
      • crbegin()
      • crend()
      • get_registered_processors()
      • get_model_category()
      • is_initialize_failed()
      • is_initialized()
      • signal_state_initializing()
      • signal_state_destroying()
      • signal_initializing()
      • signal_destroying()
      • reset_base_signals()
    • TextToText Model Object
      • signal_lora_operation()
      • signal_state_lora_operation()
      • is_available()
      • is_embedding_model()
      • has_lora_adapter()
      • get_raw_model()
      • get_special_tokens()
      • get_special_tokens_string()
      • get_model_name()
      • get_architecture()
      • get_sys_start()
      • get_assistant_start()
      • get_usr_start()
      • get_sys_end()
      • get_assistant_end()
      • get_usr_end()
      • get_eot_token()
      • get_lf_token()
      • get_vocab_count()
      • get_size()
      • get_embedding_length()
      • get_head_count()
      • get_layer_count()
      • get_max_embedding_context()
      • is_token_eof_generation()
      • is_token_special()
      • is_token_control()
      • get_quantization_string()
      • get_total_context_size()
      • get_occupied_context_size()
      • initialize_model_ex()
      • initialize_model()
      • initialize_model_ex_sync()
      • initialize_model_sync()
      • destroy()
      • destroy_sync()
      • register_context_process()
      • register_context_process()
      • declare_lora_remove()
      • declare_lora_adapter()
      • start_lora_operation()
      • tokenize_input()
      • on_lora_operate()
      • update()
      • update_t()
• Processor Object in Detail
  • Naming
  • Identifying the Expensive Operations
  • Essential Callbacks
  • Essential Signals
  • TextToText Execution Flow
    • Client Registration Example
    • Input Execution and Token Generation
  • Advanced
    • Decode Behavior Description
    • Manual Caching
    • Context Shifting
• Client Object in Detail
• About Sampling