IdeaWell

Project ideas from Hacker News discussions.

Why building a Rust LSP is hard

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

Theme 1 – Criticism of LSP’s overhead and preference for direct, in‑process calls
Many commenters argue that shuttling JSON over a TCP socket (or even stdin/stdout) is needlessly heavy compared to the native plugin models that existed before LSP.

  • “Using a JSON TCP connection for what on Windows would be direct function calls (COM) or in Eclipse would be direct function calls between Java modules always felt a bit gross.” — mitxela
  • “I agree, the only reason LSP exists as it does is for a world running on electron based applications. Plug‑ins and application extensions are not new technology and they are nearly universally more efficient in the forms designed and used prior to 2005‑ish.” — throwaway17_17
  • “IPC is the natural solution to that … TCP is not most optimal, but it is ubiquitous. Same thing with JSON.” — bheadmaster
  • “Moreover, it's not just stupid, it also does not actually solve anything. A synchronous function can also have obsolete indexing information if it races with the code updates.” — cyberax

Theme 2 – Defense of LSP: decoupling, stability, and cross‑editor language support
Supporters highlight that LSP’s client/server split prevents editor crashes, lets any editor reuse the same language server, and provides a lingua‑franca for tooling.

  • “LSP was (to my knowledge) the first language/platform/usecase agnostic protocol intended for use in code editors.” — throwaway17_17 (acknowledging the upside)
  • “In LSP, most of the processing happens in the LSP server anyway – communication overhead between client and server is negligible in comparison to it.” — bheadmaster
  • “It only makes sense in the context of host security and stability, as proven by plugin issues in those IDEs.” — pjmlp
  • “I’m glad that the code editors out there didn’t wait for your theoretical better designed protocol and decided to adopt LSP. Otherwise we’d still have editor that only support one language properly, and the rest is treated like text.” — tredre3
  • “LSP is not that bad. They tried to decouple IDE features from a language features.” — ivanjermakov

Theme 3 – Discussion of possible improvements / alternative designs
Several participants suggest tweaks or wholly different approaches (pipes, AST‑based protocols, better synchronization) that could keep LSP’s benefits while reducing its drawbacks.

  • “TCP isn’t really necessary. Usually a language server just uses stdin/stdout, which are just pipes.” — Panzerschrek
  • “Wouldn’t a normalized protocol based on AST vocabulary and tree operations make more sense?” — lenkite
  • “One of the largest issues of LSP is that each language implementation needs to do everything separately… What you’d actually want to do is parse each language into your AST, … and have all the ‘go to definition’, … done generically by the IDE on top.” — kuschku
  • “Well for one I wouldn’t design it so both the LSP and the editor need a synchronized view of the underlying file.” — packetlost
  • “The hardest thing was to find a way allowing providing useful autocompletion for a document in edited state, when it’s not syntactically‑correct.” — Panzerschrek (pointing to a concrete pain‑point that any redesign must address)

These three threads—criticism of inefficiency, praise for decoupling and safety, and ideas for refinement—dominate the conversation.

🚀 Project Ideas

BinaryLSP: High‑Performance Language Server Protocol

Summary

  • Replace JSON‑over‑TCP with a binary RPC (Protobuf/Cap'n Proto) over Unix domain sockets or shared memory to cut serialization overhead.
  • Keeps the LSP decoupling benefits while delivering near‑native function‑call speed.
  • Core value: lower latency and CPU usage for language features, especially in large projects or on resource‑constrained machines.

Details

Key Value
Target Audience IDE/editor builders, language tooling authors, developers frustrated by LSP latency
Core Feature Binary‑encoded LSP messages (requests, responses, notifications) using Protobuf; optional shared‑memory transport for zero‑copy
Tech Stack Protobuf (or Cap'n Proto), gRPC‑like async framework, language‑agnostic server stubs, C++/Rust/Runtime for client shim
Difficulty Medium
Monetization Revenue‑ready: SaaS offering of optimized LSP binaries + support, or dual‑license (open core)

Notes

  • HN commenters lamented JSON overhead and wished for synchronous‑like performance (mitxela, bheadmaster). BinaryLSP directly answers “I wouldn't design it so both the LSP and the editor need a synchronized view…” by keeping the decoupled model but removing the costly text parsing step.
  • Enables cross‑platform use (Windows named pipes, Linux sockets) while preserving the existing LSP spec, so adoption is incremental.
  • Potential for discussion: benchmarking against vanilla LSP, exploring security sandboxing for shared‑memory transport.

Polyglot AST Toolkit: Language‑Agnostic Refactoring Engine

Summary

  • Build a reusable library that operates on a concrete syntax tree (e.g., Tree‑sitter) to provide generic “go‑to‑definition”, rename, find‑usages, and refactoring actions across any language with a grammar.
  • Eliminates duplicated effort where each language server re‑implements the same analyses.
  • Core value: one implementation, many languages; faster tooling development and consistent behavior.

Details

Key Value
Target Audience Language tooling authors, editor plugin developers, companies maintaining multiple language servers
Core Feature Generic AST‑based operations (symbol resolution, cross‑language reference tracking) pluggable via language‑specific parsers
Tech Stack Tree‑sitter grammars, Rust (or Go) core library, WASM bindings for editor integration, JSON config for language‑specific rules
Difficulty Medium
Monetization Hobby (open‑source) with optional paid premium grammars or enterprise support tiers

Notes

  • Commenters like kuschku wished for “parse each language into your AST… have all the ‘go to definition’, UI rendering, highlighting, refactoring, etc all done generically”. This kit realizes that vision.
  • Enables editors to share symbol indexes between Rust, HTML, JS, CSS, etc., addressing the pain of separate tooling per language.
  • Practical utility: reduces maintenance burden, encourages community‑driven grammar contributions, and can be demoed in a Hacker News “Show HN” post.

In‑Process Language Server Shim (WASM‑Sandboxed)

Summary

  • Run language servers inside the editor process as WebAssembly modules, communicating via fast synchronous function calls instead of JSON‑over‑TCP, while retaining sandboxing for safety.
  • Provides the simplicity of direct function calls (like COM/OLE) without sacrificing isolation or multi‑language support.
  • Core value: near‑zero IPC latency, minimal CPU overhead, and easy debugging.

Details

Key Value
Target Audience Editor developers (VS Code, NeoVim, Emacs), language tooling creators seeking lower latency
Core Feature WASM‑hosted language server exposing a simple synchronous API (e.g., defn_at(position), references(symbol)) with memory‑safe sandbox
Tech Stack WebAssembly runtime (Wasmtime or Wasmer), ABI generation from existing LSP servers via thin wrapper, Rust/C++ to WASM compilation
Difficulty High
Monetization Revenue‑ready: Licensing the shim as a proprietary performance add‑on for commercial editors, or offering a hosted WASM‑LSP service

Notes

  • Direct function calls were praised by mitxela and criticized by others for safety; this approach gives the performance benefit while isolating the server in a WASM sandbox, addressing bheadmaster’s concern about malicious plugins.
  • Allows using the same language server from multiple editor instances (solving cyberax’s multi‑client issue) because each instance loads its own WASM copy.
  • Would likely spark lively HN discussion about trade‑offs of WASM vs native plugins and about extending the LSP spec with a synchronous mode.

Read Later