LLM-d infrastructure using Kubernetes

 Here is one of the best video tutorials for this

https://www.youtube.com/watch?v=hBzUokVYQkI

What is Perplexity Lily inference engine?

 

Perplexity Open Sources Lily: A Rust + Metal Inference Engine for Qwen3.6-35B-A3B on Apple Silicon

By Asif Razzaq -September 2, 2026

Perplexity has open sourced Lily, the local inference engine behind Hybrid Compute in Perplexity Computer. It is a single-process runtime: a Rust layer loads the checkpoint and drives the generation loop, an OpenAI-compatible chat-completions API streams tokens, and hand-written Metal kernels execute the model. Neither PyTorch nor MLX sits in the execution path. Lily is deliberately narrow with one model, Qwen3.6-35B-A3B, on one hardware family and that narrowness is the performance argument.

Is it deployable? Yes. A standalone demo is public in the pplx-garden repository. A Rust and Metal inference server offering greedy text generation through a minimal OpenAI-compatible HTTP API. The 4-bit checkpoint is 19.4 GB, so an Apple silicon Mac with 32 GB or more of unified memory is the realistic floor; Perplexity’s shipping Hybrid Compute product lists macOS 15+, 24 GB minimum and 32 GB for best results.

Why specialize at all?

The default Mac stack is MLX plus MLX-LM, which already ships a Qwen implementation with grouped expert work, a fused recurrent Metal kernel, and GQA-aware attention. But its operations must stay reusable across architectures. Lily gives that up and puts model structure, execution plans, and kernel selection in one runtime.

Three workload shapes

Qwen3.6-35B-A3B stores 35B parameters and activates roughly 3B per token. A router scores 256 experts and picks eight, alongside one shared expert that sees every token. It also mixes 10 full-attention layers using grouped-query attention (16 query heads, two KV heads) with 30 Gated DeltaNet layers. That yields three patterns: uneven expert groups, attention over a growing KV cache, and a fixed-size recurrence.

Prefill: keep weights packed, keep routing on the GPU

The checkpoint uses groupwise affine 4-bit quantization, every group of 64 weights sharing a bfloat16 scale and bias, about 70 GB of bfloat16 weights compressed to 19.4 GB. Metal 4 tensor operations consume bfloat16, so weights must be reconstructed first. Lily does that one tile at a time inside the grouped GEMM, holding results in threadgroup memory and accumulating in FP32, so the expanded array never reaches unified memory. In Perplexity’s ablation that fusion raised end-to-end prefill 77.4% at a 512-token prompt.

Keeping the routing histogram, prefix scan, scatter and block map inside a single GPU command buffer added 89% at 512 tokens by removing CPU synchronization inside each MoE layer. Moving from 16-row to 32-row tiles with four simdgroups added 13.2% at 2K; a register-resident Gated DeltaNet scan added 5.6%. Expert GEMMs are roughly 90% of prefill time. Long prompts run in bounded chunks so temporary activations do not compete with weights and cache for memory.

Decode: minimize bytes moved per token

Batch-1 decode has almost no weight reuse, so bandwidth sets the ceiling. One recorded step launched 795 kernels forming 555 sequential stages; Lily records real dependencies in a concurrent Metal pass so independent kernels overlap. The selected token is written straight into the next step’s GPU-resident input slot, removing a per-token CPU round trip, and four kernel chains are fused to keep intermediates in registers.

Coalesced cache reads lifted key bandwidth from 33.8 to 47.9 GB/s and value bandwidth from 42.0 to 61.8 GB/s. GQA packing, four query heads sharing one threadgroup so each KV row loads once, improved decode 23.8% at 32K. A fixed-block attention layout at 32K and above improved decode 7.7% at 32K, 27.4% at 64K, and 40.2% at 128K.

Results

On one 40-core, 128 GB M5 Max at batch 1, loading identical 4-bit checkpoint bytes against MLX-LM’s fastest direct-generation path across ten lengths from 256 to 128K tokens, Lily averaged 4,156 prefill tokens/s versus 3,388 (1.23x) and 170.0 decode tokens/s versus 126.4 (1.35x). At a 4K prompt and 4K context it reached 5,749.9 and 186.6 tokens/s against 4,737.5 and 140.9, and was faster at every recorded point: 1.12–1.42x prefill, 1.31–1.37x decode. A teacher-forced check across 192 positions put Lily’s perplexity 0.04% higher, with the same top-ranked token 96.35% of the time.

Google Proto sample file with Message and Service.

syntax = "proto3";

package library.v1;

import "google/protobuf/timestamp.proto";

import "google/protobuf/empty.proto";

option go_package = "github.com/example/library/v1;libraryv1";

// Service definition housing multiple RPC endpoints

service LibraryService {

  rpc GetBook (GetBookRequest) returns (Book);

  rpc ListBooks (google.protobuf.Empty) returns (ListBooksResponse);

  rpc CreateBook (CreateBookRequest) returns (Book);

}

// Data message definition for input requests

message GetBookRequest {

  string isbn = 1;

}

// Data message definition for book resources

message Book {

  string id = 1;

  string title = 2;

  string author = 3;

  int32 publication_year = 4;

  google.protobuf.Timestamp added_at = 5;

}

// Data message definition for creation requests

message CreateBookRequest {

  string title = 1;

  string author = 2;

  int32 publication_year = 3;

}

// Data message definition for collection responses

message ListBooksResponse {

  repeated Book books = 1;

}

What is Google Protobuf definition file?

 

A `.proto` file serves as the contract and Interface Definition Language (IDL) for gRPC, defining data structures (**messages**) and remote service endpoints (**services**) in a language-agnostic format.

**Why Proto Files Are Significant in gRPC**

Protocol buffers are the operational backbone of gRPC. Instead of relying on dynamic JSON payloads or ad-hoc REST endpoints, gRPC uses the `.proto` file to compile strongly-typed client stubs and server boilerplate code for dozens of programming languages, including Go, Java, Python, and C++. This ensures strict adherence to the API contract at compile time rather than runtime.

**Key Advantages of Using Proto Files**

* **High Performance:** Data is serialized into a compact binary format, resulting in smaller network payloads and significantly faster serialization and parsing speeds compared to text-based formats like JSON.

* **Language Agnosticism:** A service defined in a single `.proto` file can seamlessly connect a Python microservice to a Go backend or a C++ client without custom translation layers.

* **Backward and Forward Compatibility:** Because fields are identified by unique numeric tags rather than string names, you can safely add or deprecate fields without breaking legacy clients.

* **Automatic Code Generation:** Compiling tools like `protoc` eliminate manual network boilerplate writing, drastically reducing human error.

**Best Practices for Defining Proto Files**

* **Always use unique, immutable field numbers:** The numbers assigned to fields (e.g., `id = 1`) are permanent binary tags. Never change a field's number once deployed, and avoid reusing numbers of deleted fields; use the `reserved` keyword instead.

* **Optimize tag numbers 1 through 15:** Field numbers 1 through 15 take only one byte to encode. Assign your most frequently transmitted fields to these lower numbers to minimize wire size.

* **Adopt versioned package names:** Use structured naming conventions (e.g., `package demo.v1;`) to prevent naming collisions and manage breaking API changes cleanly over time.

* **Keep messages cohesive:** Avoid creating monolithic "god messages." Design granular, focused messages tailored to specific service actions to maintain clarity and reusability.

* **Leverage well-known types:** Utilize standard Google proto types (like `google.protobuf.Timestamp` or `google.protobuf.Empty`) instead of reinventing common data structures.

What is ConvertX

ConvertX is a free, open-source, self-hosted online file converter that you can deploy as a single Docker container. Developed by user C4illin on GitHub, it is designed to replace sketchy third-party file conversion websites, giving you 100% data privacy by processing all your files locally on your own server or computer. [1] (https://www.youtube.com/watch?v=3xQvxa7WGFc&t=54), [2] (https://daily.dev/posts/this-docker-container-converts-any-file-to-any-format-so-i-stopped-trusting-sketchy-websites-with-m-9ykzj06el), [3] (https://www.makeuseof.com/ditched-online-file-converters-for-a-docker-container-with-1000-formats/), [4] (https://www.youtube.com/watch?v=aqRr0cijPv0)The technical stack is built on TypeScript, Bun, and Elysia, ensuring the application is incredibly fast and lightweight. [1] (https://www.youtube.com/watch?v=0INt3gFFoEY), [2] (https://www.youtube.com/watch?v=3xQvxa7WGFc&t=54)

How It Works: "20 Engines in a Trenchcoat"Rather than writing conversion tools from scratch, ConvertX wraps over 20 well-known open-source backend conversion engines into a single unified graphical user interface (GUI). When you upload a file, ConvertX automatically passes it to the correct specialized tool: [1] (https://www.xda-developers.com/this-one-docker-container-converts-any-file-to-any-format/), [2] (https://www.makeuseof.com/ditched-online-file-converters-for-a-docker-container-with-1000-formats/), [3] (https://www.youtube.com/watch?v=3xQvxa7WGFc&t=54)🎬 FFmpeg: For high-speed audio and video conversions (e.g., MKV to MP4).🖼️ ImageMagick & GraphicsMagick: For handling complex images and vector assets.📄 LibreOffice & Pandoc: For text documents, spreadsheets, and LaTeX files.📚 Calibre: For e-book conversions.📐 Assimp: For 3D assets and modeling files.Other engines include Inkscape, libheif, Vips, XeLaTeX, Potrace, and Markitdown. [1] (https://www.makeuseof.com/ditched-online-file-converters-for-a-docker-container-with-1000-formats/), [2] (https://www.youtube.com/watch?v=3xQvxa7WGFc&t=54), [3] (https://www.youtube.com/watch?v=0INt3gFFoEY), [4] (https://daily.dev/posts/this-docker-container-converts-any-file-to-any-format-so-i-stopped-trusting-sketchy-websites-with-m-9ykzj06el), [5] (https://www.xda-developers.com/this-one-docker-container-converts-any-file-to-any-format/)Because it links all these engines together, it supports over 1,000 different format combinations

What is NextCloud Self hosting

 Nextcloud is a free, open-source Nextcloud Platform that lets you host your own personal cloud storage, file sync, and collaboration services on your own hardware or server

What is Nextcloud?Nextcloud functions similarly to commercial services like Google Drive or Dropbox, but it gives you complete control and privacy over your data. [1] (https://www.youtube.com/watch?v=jT7CUK5UNrw), [2] (https://www.digitalocean.com/community/tutorial-collections/how-to-install-and-configure-nextcloud), [3] (https://medium.com/@ayanpande/how-i-set-up-my-own-personal-cloud-with-nextcloud-ddb56bd00963)Core Features: File storage, automatic device syncing, calendar, contacts, and notes.Nextcloud Hub: Combines file sharing with online office editing, text chat, and video conferencing (Talk).Self-Hosted: Runs on a local Linux server, a Network Attached Storage (NAS) device, or a rented cloud Virtual Private Server (VPS)

How to Set It Up (Using Docker All-in-One)The easiest way to set up Nextcloud without dealing with complex web server or PHP configurations is using Docker and the official All-in-One (AIO) image

Step 1: Install DockerInstall Docker and Docker Compose on your host system (such as an Ubuntu/Debian Linux server or a compatible NAS). [1] (https://www.youtube.com/watch?v=cSpTo8b7RLs&t=17), [2] (https://nextcloud.com/home-users/)

Step 2: Run the Master ContainerStart the Nextcloud AIO master container using your terminal with a command exposing ports 8080 and 8443:

docker run -d \

  --init \

  --sig-proxy=false \

  --name nextcloud-aio-mastercontainer \

  --restart always \

  -p 8080:8080 \

  -v nextcloud_aio_mastercontainer:/mnt/docker-aio-config \

  -v /var/run/docker.sock:/var/run/docker.sock:ro \

  nextcloud/all-in-one:latest

Step 3: Access the AIO InterfaceOpen your web browser and navigate to https://your-server-ip:8080 or https://localhost:8080.Follow the on-screen setup interface to configure your domain or local address, and let it automatically pull and start the dependent containers (Apache proxy, Nextcloud app server, PostgreSQL database, and Redis cache)

tep 4: Complete the InstallationOnce the containers are running, open the Nextcloud interface on the configured port/domain.Create your admin username and secure password.Choose whether to install recommended productivity apps, then finish and access your new cloud dashboard. [1] (https://www.youtube.com/watch?v=cSpTo8b7RLs&t=17), [2] (https://github.com/nextcloud/all-in-one)

What are Causal AI Models

 Causal AI models are artificial intelligence systems designed to understand and model cause-and-effect relationships rather than just statistical correlations. [1] (https://datapoem.ai/resources/article/what-is-causal-ai)Correlation vs. CausationTraditional AI: Finds patterns and correlations (variables that move together). For example, it might notice that high ice cream sales correlate with high shark attacks. [1] (https://datapoem.ai/resources/article/what-is-causal-ai)Causal AI: Understands the underlying mechanism and determines if one variable actually causes a change in another (the heat drives ice cream and swimming, but ice cream doesn't cause shark attacks). [1] (https://datapoem.ai/resources/article/what-is-causal-ai)How Causal AI WorksCausal Discovery: Algorithms look at data patterns to find potential cause-and-effect links.Structural Causal Models (SCMs): Uses tools like Directed Acyclic Graphs (DAGs) to map dependencies.Counterfactual Reasoning: Asks "what if" scenarios to simulate hypothetical interventions and predict outcomes before they happen in the real world. [1] (https://medium.com/@alexglee/causal-ai-current-state-of-the-art-future-directions-c17ad57ff879), [2] (https://www.infobip.com/glossary/causal-ai)Key BenefitsExplainability: Clearly explains why a decision or prediction was made.Reduced Bias: Explicitly models sensitive variables to detect and mitigate hidden biases.Better Decision-Making: Helps organizations test interventions safely in simulations before real-world deployment. [1] (https://kanerika.com/blogs/causal-ai/), [2] (https://www.techtarget.com/whatis/video/An-explanation-of-causal-AI)Common ApplicationsHealthcare: Evaluating how specific treatments directly impact patient recovery.Sales & Marketing: Figuring out if an ad campaign actually drove a purchase or if the buyer would have bought the product anyway.Supply Chain: Identifying the exact root cause of a production line breakdown. [1] (https://www.techtarget.com/whatis/video/An-explanation-of-causal-AI), [2] (https://kanerika.com/blogs/causal-ai/), [3] (https://datapoem.ai/resources/article/what-is-causal-ai)If you'd like, I can share:Specific Python libraries used to build causal models (like DoWhy or Pyro)How it differs from Generative AI