Edge Rewrite
pun Source code View original
Jump to content

Draft:TensorSharp

From Wikipedia, the free encyclopedia

TensorSharp is an open-source inference engine for running large language models and other generative artificial intelligence models in the GGUF format. Written primarily in C# for the .NET platform, it is intended for local inference on personal computers, workstations, and servers.[1]

The software includes a command-line application, a browser-based chat server, a programming interface for integration into .NET applications, and HTTP endpoints implementing subsets of the OpenAI and Ollama application programming interfaces.[2][3] TensorSharp supports execution on Windows, macOS, and Linux, with processor backends for CPUs, CUDA, Metal, Vulkan, and Apple's MLX framework.[2]

History

[edit]

TensorSharp was created by software developer Zhongkai Fu and is developed publicly in a GitHub repository.[4] Its public development history documents the addition of multiple execution backends, model architectures, multimodal input, continuous batching, speculative decoding, image generation and editing, and server interfaces.[5]

Source-code releases are distributed through GitHub.[6] The project is licensed under the three-clause BSD license.[7]

Architecture

[edit]

TensorSharp is organized as a set of separate .NET projects representing different parts of the inference system. The project's documentation describes the principal components as follows:[2]

  • TensorSharp.Core defines the tensor, storage, device, and operation abstractions and includes CPU implementations of tensor operations.
  • TensorSharp.Runtime provides GGUF parsing, tokenization, prompt-template processing, sampling, output parsing, key–value caching, and request scheduling.
  • TensorSharp.Models contains implementations of supported neural-network architectures and multimodal encoders.
  • TensorSharp.Backends.GGML connects the .NET runtime to native operations based on the ggml tensor library.
  • TensorSharp.Backends.Cuda provides a direct CUDA implementation using the CUDA Driver API, cuBLAS, and custom kernels.
  • TensorSharp.Backends.MLX provides an execution path for Apple silicon using MLX.
  • TensorSharp.Server is an ASP.NET Core application that hosts the web interface and HTTP APIs.
  • TensorSharp.Cli provides command-line inference, batch processing, interactive sessions, and benchmarking functions.

When a model is opened, the runtime reads its GGUF metadata, identifies the model architecture, loads its quantized weights into the selected backend, applies the model's tokenizer and chat template, and performs prompt processing and token generation.[2]

Execution backends

[edit]

The project documents several execution backends:[2][8]

  • a CPU backend implemented in C#;
  • a direct CUDA backend for supported Nvidia graphics processors;
  • an MLX backend for Apple silicon;
  • a ggml CPU backend;
  • a ggml Metal backend for macOS;
  • a ggml CUDA backend; and
  • a ggml Vulkan backend for supported graphics processors on Windows and Linux.

According to the project's architecture documentation, operations not implemented by a selected accelerator backend may fall back to the CPU implementation.[2]

Request scheduling and memory management

[edit]

TensorSharp implements continuous batching, in which requests from multiple users are interleaved while a model remains loaded. It also implements a paged key–value cache and prefix sharing intended to reuse cached prompt data between requests.[9]

The project describes this design as being influenced by the paged-memory approach used by vLLM. PagedAttention, introduced as part of vLLM, divides the attention key–value cache into blocks that can be allocated without requiring a single contiguous region of memory.[10]

TensorSharp also includes optional speculative-decoding implementations for models that provide multi-token prediction or additional draft-model heads. In speculative decoding, a draft component proposes one or more tokens that are subsequently checked by the primary model.[9]

Interfaces

[edit]

Command-line interface

[edit]

The command-line application can perform one-shot text generation, interactive conversations, multimodal inference, batch processing from JSON Lines files, prompt inspection, and benchmark execution.[11]

Models, execution backends, sampling parameters, context length, output length, multimodal projectors, and related settings can be selected through command-line arguments.[11]

Server and web interface

[edit]

TensorSharp.Server is an ASP.NET Core application that loads a model and exposes a browser-based chat interface. The same server provides streaming and non-streaming HTTP endpoints.[12]

The OpenAI-compatible portion of the API includes Chat Completions and model-listing endpoints, including /v1/chat/completions and /v1/models. The Ollama-compatible portion includes generation, chat, model-listing, and model-information endpoints.[3]

The interfaces support features including streaming generation, image input for compatible multimodal models, function or tool calling, and structured output validated against a JSON Schema.[3]

The server documentation states that the application does not provide built-in API-key authentication or TLS and listens on all network interfaces by default. The documentation recommends restricting it to a trusted network or deploying it behind an authenticated TLS reverse proxy.[3]

.NET integration

[edit]

TensorSharp can be incorporated into a C# application by referencing its source projects and calling the model-loading and inference APIs directly.[13] This permits applications to embed inference without communicating with a separate Python process or external model server.

The project's documentation stated in July 2026 that several runtime, model, backend, command-line, and server packages had not yet been published separately through NuGet. Source-based project references were therefore the documented integration method.[2][13]

Model and modality support

[edit]

The set of supported architectures changes as the project is developed. Project documentation has listed implementations for model families including Gemma, Qwen, GPT-OSS, Nemotron-H, Mistral, DiffusionGemma, and Qwen-Image.[14]

Depending on the model architecture and the availability of a compatible multimodal projection file, TensorSharp can process text, images, video, and audio. Its server and command-line interfaces can also process PDF documents by extracting text or, for scanned pages, rendering pages for processing by a vision-capable model.[2][14]

The project supports quantized model weights stored in GGUF files. Supported quantization formats and accelerated operations vary by model architecture and execution backend.[14]

TensorSharp also includes support for image generation and image editing through architectures documented by the project, including Qwen-Image-Edit.[14]

Testing and performance evaluation

[edit]

The repository includes unit tests, integration tests, and a model-and-backend test matrix used to run combinations of models, execution backends, and inference features.[4][15]

The project publishes benchmarks comparing TensorSharp with inference engines including llama.cpp and stable-diffusion.cpp. The tests cover measurements such as prompt-processing throughput, time to first token, token-generation throughput, structured-output generation, tool calling, and image-editing latency.[15]

The published benchmark figures are produced and reported by the TensorSharp project rather than by an independent benchmarking organization. The project provides test scripts and describes the models, hardware, quantization formats, prompts, and execution settings used in its comparisons.[15]

Distribution and licensing

[edit]

TensorSharp is distributed as source code and requires the .NET software development kit. Some native backends additionally require platform-specific development tools, such as the CUDA Toolkit, Vulkan development files, CMake, or Apple's command-line development tools.[8]

The software is released under the permissive three-clause BSD license.[7]

See also

[edit]

References

[edit]
  1. ^ "TensorSharp". TensorSharp. Retrieved July 10, 2026.
  2. ^ a b c d e f g h "Overview and Architecture". TensorSharp. Retrieved July 10, 2026.
  3. ^ a b c d "HTTP API (Ollama / OpenAI)". TensorSharp. Retrieved July 10, 2026.
  4. ^ a b Fu, Zhongkai. "TensorSharp". GitHub. Retrieved July 10, 2026.
  5. ^ "Commit history for TensorSharp". GitHub. Retrieved July 10, 2026.
  6. ^ "TensorSharp releases". GitHub. Retrieved July 10, 2026.
  7. ^ a b "TensorSharp license". GitHub. Retrieved July 10, 2026.
  8. ^ a b "Getting Started". TensorSharp. Retrieved July 10, 2026.
  9. ^ a b "Advanced Features". TensorSharp. Retrieved July 10, 2026.
  10. ^ Kwon, Woosuk; Li, Zhuohan; Zhuang, Siyuan; Sheng, Ying; Zheng, Lianmin; Yu, Cody Hao; Gonzalez, Joseph E.; Zhang, Hao; Stoica, Ion (2023). "Efficient Memory Management for Large Language Model Serving with PagedAttention". Proceedings of the 29th Symposium on Operating Systems Principles. pp. 611–626. doi:10.1145/3600006.3613165.
  11. ^ a b "Command Line Interface". TensorSharp. Retrieved July 10, 2026.
  12. ^ "Server and Web UI". TensorSharp. Retrieved July 10, 2026.
  13. ^ a b "Using TensorSharp from C#". TensorSharp. Retrieved July 10, 2026.
  14. ^ a b c d "Features". TensorSharp. Retrieved July 10, 2026.
  15. ^ a b c "Benchmarks and Testing". TensorSharp. Retrieved July 10, 2026.
[edit]

Category:2026 software Category:Free software programmed in C Sharp Category:Free and open-source software Category:Software using the BSD license Category:Cross-platform free software

Retrieved from "https://en.wikipedia.org/w/index.php?title=Draft:TensorSharp&oldid=1363605771"