Cache-to-Cache: Direct Semantic Communication Between LLMs (2025)

📝 Discussion Summary (Click to expand)

Prevalent themes in the discussion

Neuralese threatens model interpretability
cubefox: “It's not looking good for monitorability.”
sparky_twofort: “Is Neuralese in no way decodable into a human‑interpretable system? Genuine question …”
Y_Y: “Definitely decodable, that's what's being done now.”
cubefox: “It's not.”
Whether semantic/embedding representations should replace natural language for model communication
gavinray: “I asked why semantic representation rather than text wasn't used, since natural language seems quite a lossy representation for semantic concepts.”
foota: “Why don't multimodal models learn to interpret images themselves without an embedding? … I think this is similar to how Gemma 4 12B is implemented …”
thfuran: “What does interpreting images mean in practice if you exclude the possibility of feature extraction or any other sort of implicit embedding?”
KV‑cache alignment as a way to share knowledge and delegate subgoals between models
ACCount39: “If multiple models can use cache representations … producing a model family where all models are 'KV aligned', and each model can utilize the KV cache of other models directly …”
refibrillator: points to the Universal Weight Subspace Hypothesis as related work on weight‑space sharing.

🚀 Project Ideas

Provides interactive visualization and translation of internal hidden states (Neuralese) from LLMs into natural language explanations, addressing interpretability concerns.
Enables developers to audit chain-of-thought reasoning, debug model behavior, and build trust in AI systems.

Key	Value
Target Audience	AI researchers, ML engineers, AI safety practitioners
Core Feature	Real-time probing of model activations, mapping to human-readable concepts via probing classifiers and attention visualization
Tech Stack	Python, PyTorch/HuggingFace, React, D3.js, ONNX for model export
Difficulty	Medium
Monetization	Revenue-ready: SaaS subscription tiered by model size/API calls

HN user sparky_twofort asked: “Is Neuralese in no way decodable into a human‑interpretable system? Genuine question.” This tool directly answers that query.
Offers a concrete way to discuss and improve model transparency, potentially becoming a standard auditing utility.

Library enabling seamless sharing of key‑value caches between disparate LLM sizes, allowing cheap models to delegate subgoals to expensive ones without recompute.
Reduces latency and compute cost for hierarchical reasoning pipelines, addressing the need for KV‑aligned model families.

Key	Value
Target Audience	Researchers building model ensembles, developers of agent frameworks
Core Feature	Cache alignment adapter that projects KV cache from one model to another's space using learned linear transforms
Tech Stack	Python, Triton for CUDA kernels, HuggingFace Transformers, FAISS for similarity
Difficulty	High
Monetization	Hobby (open‑source) with optional enterprise support license

ACCount39 mused: “If multiple models can use cache representations… What stops us then from going a step further…?” This project provides the missing step.
Enables new agent architectures with minimal handover latency, sparking discussion on efficient model chaining.

Interactive tool that lets users inject textual prompts into the image encoder of multimodal models to steer embedding generation, testing the hypothesis of guided embeddings.
Helps practitioners understand and improve multimodal model interpretability, addressing confusion about embeddings being unaware of context.

Key	Value
Target Audience	Multimodal ML researchers, product teams building vision‑language apps
Core Feature	UI to adjust prompt text, visualize resulting image embeddings, and retrieve nearest‑neighbor captions or generated text
Tech Stack	Streamlit or Gradio, PyTorch, CLIP or similar model, FAISS for search, React for UI
Difficulty	Medium
Monetization	Hobby (open‑source) or Revenue‑ready: paid plugin for cloud notebooks

Foota questioned: “Why don't multimodal models learn to interpret images themselves without an embedding?” This explorer lets users test prompt‑guided embeddings as an alternative.
Provides a playground for discussing new training objectives and improving multimodal model transparency.