Is this because of a max latency setting, or the internal prompt, or am I doing something wrong? Or is it only really make to try to autocomplete lines and not blocks like Copilot will.

Thanks :)

Just curious: how much of your code nowadays completed by LLM?

If so, what's ollama missing?

Agreed but he's an international treasure (his Github profile states Bulgaria).

Currently, there isn't a user-friendly way to disable the stats from showing apart from modifying the "'show_info': 0" value directly in the plugin implementation. These things will be improved with time and will become more user-friendly.

A few extra optimizations will soon land which will further improve the experience:

- Speculative FIM

- Multiple suggestions

I think this will improve significantly over time when when hardware becomes cheaper. As long as newer languages can map to older languages (syntax/function wise), we should be able generate enough synthetic data to make working with less known languages easier.

> During LLM inference, this context is utilized for code completion

Hmmm... I wonder what's better. As I'm coding I jump and search to relevant parts of the codebase to build up my own context for solving the problem, and I expect that's likely better than RAG. Llama.vim seems to follow this model, while tabby could theoretically get at things I'm not looking at/haven't looked at recently...

It seems very logical to me that there'd be orders of magnitude more training data for some domains than others, and that existing models' skill is not evenly distributed cross-domain.

You're not alone :-) I asked a very similar question about a month ago: https://news.ycombinator.com/item?id=42552653 and have continued researching since.

My takeaway was that autocomplete, boiler plate, and one-off scripts are the main use cases. To use an analogy, I think the code assistants are more like an upgrade from handsaw to power tools and less like hiring a carpenter. (Which is not what the hype engine will claim).

For me, only the one-off script (write-only code) use-case is useful. I've had the best results on this with Claude.

Emacs abbrevs/snippets (+ choice of language) virtually eliminate the boiler plate problem, so I don't have a use for assistants there.

For autocomplete, I find that LSP completion engines provide 95% of the value for 1% of the latency. Physically typing the code is a small % of my time/energy, so the value is more about getting the right names, argument order, and other fiddly details I may not remember exactly. But I find, that LSP-powered autocomplete and tooltips largely solve those challenges.

I've found incredible value in having LLMs help me write unit tests. The quality of the test code is far from perfect, but AI tooling - Claude Sonnet specifically - is good at coming up with reasonable unit test cases after I've written the code under test (sue me, TDD zealots). I probably have to fix 30% of the tests and expand the test cases, but I'd say it cuts the number if test code lines I author by more than 80%. This has decreased the friction so much, I've added Continuous Integration to small, years-old personal projects that had no tests before.

I've found lesser value with refactoring and adding code docs, but that's more of autocomplete++ using natural language rather than AST-derived code.

“One”? Wired up how? There is a huge difference between the best and worst. They aren’t fungible. Which one? How long ago? Did it even support FIM (fill in middle), or was it blindly guessing from the left side? Did the plugin even gather appropriate context from related files, or was it only looking at the current file?

If you try Copilot or Cursor today, you can experience what “the best” looks like, which gives you a benchmark to measure smaller, dumber models and plugins against. No, Copilot and Cursor are not available for emacs, as far as I know… but if you want to understand if a technology is useful, you don’t start with the worst version and judge from that. (Not saying emacs itself is the worst… just that without more context, my assumption is that whatever plugin you probably encountered was probably using a bottom tier model, and I doubt the plugin itself was helping that model do its best.)

There are some local code completion models that I think are perfectly fine, but I don’t know where you will draw the line on how good is good enough. If you can prove to yourself that the best models are good enough, then you can try out different local models and see if one of those works for you.

Copilot, Ollama, and the others have all been strictly inferior to rust-analyzer. The suggested code is often just straight up invalid and takes just long enough to be annoying. Compare that to just typing '.'/'::' + a few characters to fuzzy-select what I'm looking for + enter.

ETA: Both did save me a few seconds here and there when obvious "repetition with a few tweaks each line" was involved, but to me that is not worth a monthly subscription or however much wall power my GPU was consuming.

So maybe it is just a difference in perspective? Even incorrect code and bad ideas can still be helpful. It is only useless if you expect them to hand you working code.

I can run 2b-14b models just fine on the CPU on my laptop (framework 13 with 32gb ram). They aren't super fast, and the 14b models have limited context length unless I run a quantized version, but they run.

If you just want generation and it doesn't need to be fast... drop the $200 for 128gb of system ram, and you can run the vast majority of the available models (up to ~70b quantized). Note - it won't be quick (expect 1-2 tokens/second, sometimes less).

If you want something faster in the "low end" range still - look at picking up a pair of Nvidia p40s (~$400) which will give you 16gb of ram and be faster for 2b to 7b models.

If you want to hit my level for "moderate", I use 2x3090 (I bought refurbed for ~$1600 a couple years ago) and they do quite a bit of work. Ex - I get ~15t/s generation for 70b 4 quant models, and 50-100t/s for 7b models. That's plenty usable for basically everything I want to run at home. They're faster than the m2 pro I was issued for work, and a good chunk cheaper (the m2 was in the 3k range).

That said - the m1/m2 macs are generally pretty zippy here, I was quite surprised at how well they perform.

Some folks claim to have success with the k80s, but I haven't tried and while 24g vram for under $100 seems nice (even if it's slow), the linux compatibility issues make me inclined to just go for the p40s right now.

I run some tasks on much older hardware (ex - willow inference runs on an old 4gb gtx 970 just fine)

So again - I'm not really sure we'd agree on moderate (I generally spend ~$1000 every 4-6 years to build a machine to play games, and the machine you're describing would match the specs for a machine I would have built 12+ years ago)

But you just need literal memory. bumping to 32gb of system ram would unlock a lot of stuff for you (at low speeds) and costs $50. Bumping to 124gb only costs a couple hundred, and lets you run basically all of them (again - slowly).

If "moderate hardware" is your average office PC then it's unlikely to be very usable. Anyone with a gaming GPU from the last several years should be workable though.

[1] https://news.ycombinator.com/item?id=42069453

Larger models work but slow down. I do have 64GB of RAM but I think 32 could work. 16GB is pushing is, but should be possible if you don't have anything else open.

Memory requirements depend on numerous factors. 2GB VRAM is not enough for most GenAI stuff today.

This way I can choose to just ignore it for the most part, but if I see something interesting, I can refine it, prompt, cherry pick, etc. I really don't want autocomplete for anything.

This will help for offline support (on planes and such).

Obviously, newer GPUs will run faster than older GPUs, but you need more VRAM to be able to run larger models. A small LLM that fits into an RTX 4060's 8GB of VRAM will run faster there than it would on an older RTX 3090. But the 3090 has 24GB of VRAM, so it can run larger LLMs that the 4060 simply can't handle.

Llama.cpp can split your LLM onto multiple GPUs, or split part of it onto the CPU using system RAM, though that last option is much much slower. The more of the model you can fit into VRAM, the better.

The Apple M-series macbooks have unified memory, so the GPU has higher bandwidth access to system RAM than would be available over a PCIe card. They're not as powerful as Nvidia GPUs, but they're a reasonable option for running larger LLMs. It's also worth considering AMD and Intel GPUs, but most of the development in the ML space is happening on Nvidia's CUDA architecture, so bleeding edge stuff tends to be Nvidia first and other architectures later, if at all.

    VRAM (GB) = 1.2 * number of parameters (in billions) * bits per parameter / 8

Because quantization is often nearly free in terms of output quality, you should usually look for quantized versions. For example, Llama 3.2 uses 16-bit parameters but has a 4-bit quantized version, and looking at the formula above you can see that will allow you to run a 4x larger model.

Having enough VRAM will allow you to run a model, but performance is dependent on a lot of other factors. For a much deeper dive into how all of this works along with price/dollar recommendations (though from last year!), Tim Dettmers wrote this excellent article: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...

Worth mentioning for the benefit of those who don't want to buy a GPU: there are also models which have been converted to run on CPU.

[0] https://blog.runpod.io/understanding-vram-and-how-much-your-...

Not so great. Zero legal status. No blessing by Apple. Can be taken down whenever Apple execs decide to.

So all depends on how it fits in context and it will for the foreseeable time be like it as training is absolutely expensive compared to RAG.