For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments.
They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription.
You can also take this a step further and use this exact setup for a local-only co-pilot style code autocomplete and chat using Twinny. I use this every day. It's free, private, and offline.
It is true that VS Code has some non-optional telemetry, and if VS Codium works for people, that is great. However, the telemetry of VSCode is non-personal metrics, and some of the most popular extensions are only available with VSCode, not with Codium.
There's a whitelist identifier that you can add bundle IDs to, to get access to the more sensitive APIs. Then you can download the extension file and install it manually. I don't have the exact process right now but just Google it :)
> However, the telemetry of VSCode is non-personal metrics
I don't care, I don't want my text editor to send _any_ telemetry, _especially_ without my explicit consent.
> some of the most popular extensions are only available with VSCode
This has never been an issue for me, fortunately. The only issue is Microsoft's proprietary extensions, which I have no interest in using either. If I wanted a proprietary editor I'd use something better.
Making the remote editing extension closed is particularly frustrating, as you have little visibility into what it's doing and it is impossible to debug obscure errors
The way I read it, the message you replied to was a complaint about parts of VSCode being proprietary. Do you mean to say Jetbrains is pretty ok on the "not being proprietary" front?
Yeah, 100%. I'm not a hardcore FOSS only person, but for my core workflow, when a FOSS tool exists and works well, I am not likely to use a proprietary alternative if I can avoid it at all.
So yeah, I'll use Excel to interoperate with fancy spreadsheets, but if LibreOffice will do the job, I'll use it instead. I tried out several of the fancy proprietary editors at various times (SublimeText, VSCode, even Jetbrains), but IMO they were not better _enough_ to justify switching away from something like vim, which is both ubiquitously available and FOSS.
But I don't want it. I want my software to work for me, not against me.
>and some of the most popular extensions are only available with VSCode, not with Codium.
I'll manage without them. What's especially annoying is that this restriction is completely artificial.
Having said that, MS did a great job with VsCode and I applaud them for that. I guess nothing is perfect, and I bet these decisions were made by suits against engineer wishes.
> But I don't want it. I want my software to work for me, not against me.
How is said software working "against" you by collecint non-personal telemetry while purpose of that telemetry usually is making the software better for most users?
>usually is making the software better for most users?
that usually hasn't been the case since at least a decade. it's truly bewildering that someone especially on hackernews would voluntarily give big tech there finger and not expect to get bitten.
> extensions may be collecting their own usage data and are not controlled by the telemetry.telemetryLevel setting. Consult the specific extension's documentation to learn about its telemetry reporting and whether it can be disabled.
That's new. Previously there was a setting, but they removed it, and it would even throw a warning in settings.json that the property no longer existed.
They must have reintroduced the telemetry setting. I can't remember if I deleted the old one, but my setting on that new value was set to "all" by default.
Not allowing end-users to disable telemetry is actually awful. The gold standard is that IP addresses are considered personally identifiable information.
This has been my go-to for all of my local LLM interaction: it easy to get going, manages all of the models easily. Nice clean API for projects. Updated regularly; works across Windows, Mac, Linux. It's a wrapper around LlamaCpp, but it's a damned good one.
Same here, however minimal. I've also installed openwebui so the instance has a local web interface, and then use tailscale to access my at home LAN when put and about on the cellphone. (Goes16 weather data, ollama, a speed cam setup, and esphome temp sensors around the home / property).
It's been pretty flawless, and honestly pretty darn useful here and there. The big guns go faster and do more, but I'd prefer not having every interaction logged etc.
6core 8th gen i7 I think, with a 1050ti. Old stuff. And it's quick enough on the smaller 7/8b models for sure.
Thanks for your recommendation! I just ran Llamafile for the first time with a custom prompt on my Windows machine (i5-13600KF, RX6600) and found that it performed extremely slowly and wasn't as smart as ChatGPT. It doesn't seem suitable for productive writing. Did I do something wrong, or is there a way to improve its writing performance?
Local models are definitely not as smart as ChatGPT but you can get pretty close! I'd consider them to be about a year behind in terms of performance compared to hosted models, which is not surprising considering the resource constraints.
I've found that you can get faster performance by choosing a smaller model and/or by using a smaller quantization. You can use other models with llamafile as well. They have some prebuilt ones:
RAM and what GPU you have are big determinants of how fast it will run, and how smart a model you can run. A large amount of RAM and GPU memory is required for larger models without significant slowdown because its much faster if it can keep the entire model in memory. Small models range from 3-8 gigabytes, but a 70B parameter model will be 30-50 gigabytes.
Q4_K_S. While not as good as top commercial models like chatgpt, they are still quite capable and I like that there are also uncensored/abliterated models like Dolphin.
If the simulation hypothesis is real, perhaps it would follow that all the dark matter and dark energy in the universe is really just extra cycles being burned on layers of interpreters and JIT compilation of a loosely-typed scripting language.
... a reality where everything in software development that was previously established as robust foundation is discarded, only to be re-learned and re-implemented less well while burning VC cash.
This was on their example LLaVA 1.5 7b q4 with all default parameters which does not specify chat or instruct... but after the first message it actually worked as expected so I guess it's RLHF'd for chat or chat+instruct.
I don't know if it was some sort of error on the UI or what.
Trying to interrogate it about the first message yielded no results. It just repeated back my question, verbatim, unlike the rest of the chat which was more or less chat-like :shrugh:
If anyone is interested in trying local AI, you can give https://recurse.chat/ a spin.
It lets you use local llama.cpp without setup, chat with PDF offline and provides chat history / nested folders chat organization, and can handle thousands of conversations. In addition you can import your ChatGPT history and continue chats with local AI.
Yeah I set up a local server with a strong GPU but even without that it's ok, just a lot slower.
The biggest benefits for me are the uncensored models. I'm pretty kinky so the regular models tend to shut me out way too much, they all enforce this prudish victorian mentality that seems to be prevalent in the US but not where I live. Censored models are just unusable to me which includes all the hosted models. It's just so annoying. And of course the privacy.
It should really be possible for the user to decide what kind of restrictions they want, not the vendor. I understand they don't want to offer violent stuff but 18+ topics should be squarely up to me.
Lately I've been using grimjim's uncensored llama3.1 which works pretty well.
Well you can use some jailbreak prompts but with cloud models it's a cat and mouse game as they constantly fix known jailbreaks. With local models this isn't a problem of course. But I prefer getting a fine-tune model so I don't have to cascade prompts.
Not all uncensored models are great. Some return very sparse data or don't return the end tags sometimes so they keep hallucinating and never finish.
If you import grimjim's model, make sure you use the complete modelfile from vanilla lama3.1, not just an empty modelfile. Because he doesn't provide one. This really helps setting the correct parameters so the above doesn't happen so much.
But I have seen it happen with some official ollama models like wizard-vicuna and dolphin-llama. They come with modelfiles so they should be correct.
Not only are they the only future worth living in, incentives are aligned with client-side AI. For governments and government contractors, plumbing confidential information through a network isn't an option, let alone spewing it across the internet. It's a non-starter, regardless of the productivity bumps stuff like Copilot can provide. The only solution is to put AI compute on a cleared individual's work computer.
Sneakernets are pervasive in environments that handle classified information. If something like that gets moved through a network, it's rarely leaving one physical room unless there's some seriously exotic hardware involved - "general dynamics MLS" is a great search prompt if you're curious what that looks like.
idk, i have a pretty powerful laptop with 16GB VRAM and a 3080ti. when i've played with quantized llama2 and llama3 (with llamacpp), it was kinda underwhelming. inference was slow, the laptop would heat up and the results weren't as good. (is llama3.1 better?)
this was with 4bit quantization and offload of as many layers as possible to the gpu.
(A brief note: While not weak, the laptop version of a 3080 Ti is far surpassed by even just a desktop 4060 Ti, which is sold for less than 400$. So it's possible to setup a stronger system relatively cheaply. What's good enough depends on the expectations.)
Unless you have special needs like very high usage, privacy or other ones depicted in the article, buying another computer many hundred dollars for the unique purpose of running local models is a hard sell.
If you use their API instead of their sub-based offers, the most popular models are cheap to use and with BYOK tools, switching model is as easy as entering another string in a form.
For instance I put $15 on my OpenAI account in August 2023, since then I used Dall-E weekly and I still got more than $5 credit left!
it seemed to me that the bottleneck mostly revolved around the layers that were in system ram and that a lack of vram was really the gating factor in terms of reasonable inference performance. (although i would imagine that there's probably some more optimization that could be done to make best use of a split vram/sysram setup.)
in any event it was fun to try out, but still didn't seem anywhere near how well the hosted models work. a heavy duty workstation with a bunch of gpus/vram would probably be a different story though.
> it seemed to me that the bottleneck mostly revolved around the layers that were in system ram and that a lack of vram was really the gating factor in terms of reasonable inference performance. (although i would imagine that there's probably some more optimization that could be done to make best use of a split vram/sysram setup.)
You could try a model that fits entirely into in VRAM. It"s a trade of precision for a decent bit of performance. 16GB is plenty to work with as i've seen acceptable enough results with 7B models on my 8GB GPU.
When the article says that researchers are using their laptops those researchers are either using very small models on a gaming laptop or they have a fairly modern MacBook with a lot of ram.
There are also options for running open LLMs in the cloud. Groq (not to be confused with Grok) runs Llama, Mixtral and Gemma models really cheaply: https://groq.com/pricing/
I'll play around with it some more later. I was running llava-v1.5-7b-q4.llamafile which is the example that they recommend trying first at https://github.com/Mozilla-Ocho/llamafile
Groq looks interesting and might be a better option for me. Thank you.
Fully agree with you, it should, but after trying a few times different llamas I think they're very far from "try it out in just a few moments". Unless all you want is to see one running, for anything beyond that you'll be in dependency hell...
If you just want to chat, download https://lmstudio.ai/, then download their recommended LLM files, and you're good to go. Really that simple these days.
> Subject to the Agreement, Company grants you a limited license to reproduce portions of Company Properties for the sole purpose of using the Services for your personal, non-commercial purposes.
There is no dependency hell. It's just a single file. If you want to get into trying different models and various settings, you can use LM Studio, and still no need to worry about dependencies.
Many setup rely on Nvidia GPUs, Intel stuff, Windows or other stuff, that I would rather not use, or are not very clear about how to set things up.
What are some recommendations for running models locally, on decent CPUs and getting good valuable output from them? Is that llama stuff portable across CPUs and hardware vendors? And what do people use it for?
Have you tried a Llamafile? Not sure what platform you are using. From their readme:
> … by combining llama.cpp with Cosmopolitan Libc into one framework that collapses all the complexity of LLMs down to a single-file executable (called a "llamafile") that runs locally on most computers, with no installation.
Low cost to experiment IMO. I am personally using MacOS with an M1 chip and 64gb memory and it works perfectly, but the idea behind this project is to democratize access to generative AI and so it is at least possible that you will be able to use it.
I should have qualified the meaning of “works perfectly” :) No 70b for me, but I am able to experiment with many quantized models (and I am using a Llama successfully, latency isn’t terrible)
"Cosmopolitan Libc makes C a build-once run-anywhere language, like Java, except it doesn't need an interpreter or virtual machine. Instead, it reconfigures stock GCC and Clang to output a POSIX-approved polyglot format that runs natively on Linux + Mac + Windows + FreeBSD + OpenBSD + NetBSD + BIOS with the best possible performance and the tiniest footprint imaginable."
I use it just fine on a Mac M1. The only bottleneck is how much RAM you have.
I use whisper for podcast transcription. I use llama for code complete and general q&a and code assistance. You can use the llava models to ingest images and describe them.
Ollama works on anything: Windows, Linux, Mac and Nvidia or AMD. I don't know if other cards like Arc are supported by anything yet, bit of it supports the open Vulkan API (like AMD) then it should work.
Every inference server out there supports running from CPU, but realize that it's much slower than running on a GPU - that's why this revolution didn't begin until GPUs became powerful and affordable.
As far as being clear to setup, Ollama is trivial: it's a single command line that only asks what model you want and they provide you with a list on their website. They even have a Docker container if you don't want to worry about installing any dependencies. I don't know what could be easier than that.
Most other tools like LM Studio or Jan are just a fancy UI running llama.cpp as their server and using HuggingFace to download the models. They don't even offer anything beyond simple inference, such as RAG or agents.
I've yet to see anything more than a simple RAG that's available to use out of the box for local use. The only full service tools are online services like Microsoft Copilot or ChatGPT. Anyone else who wants to do that more advanced kind of system ends up writing their own code. It's not hard if you know Python - there are lots of libraries available like HuggingFace, LangChain, and Llama-Index, as well as millions of tutorials (every blog has one).
Maybe that's a sign that there's room for an open source platform for this kind of thing, but given that it's a young field and everyone is rushing to become the next big online service or toolkit, there might not be as much interest from developers to build an open source version of a high quality online service.
How do we rate whether the smaller models are any good? How many questions do we need to ask it to know that it can be trusted and we didn't waste our time on it?
You should never completely trust any LLM. They all get things wrong, make things up, and have blind spots. They're any good if they help you for some of your particular uses (but may still fail badly for other uses).
I think you didn't understand my question and maybe I phrased it poorly. The problem is not whether we should trust any deep learning model (the answer is indeed no). But the question is how we can find out if a model is any good before investing our time into that model. Each bad reply we get has a price, because it wastes our time. So, how can we compare models objectively without having to try them out ourselves first?
But how good are these models compared to gpt4o? My last experience with llama2-8b was not great at all. Are there really that good models that would fit on an average consumer hardware (mine has already 32GB ram and 16GB vram)?
The post you're replying to couldn't have made it any easier to answer these questions yourself. No, it won't be as good as the state of the art with massive cloud infrastructure behind an http api.
Last I checked it was basically just whisper.cpp so not whisperx and no diarization by default but it moves pretty quickly so you may want to ask on the Mozilla AI Discord.
I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company.
[1] It feels super strange to talk to yourself, but luckily I'm out early enough that I'm often alone. Worst case, I pretend I'm talking to someone on the phone.
Same. My husky/pyr mix needs a lot of exercise, so I'm outside a minimum of a few hours a day. As a result I do a lot of dictation on my phone.
I put together a script that takes any audio file (mp3, wav), normalizes it, runs it through ggerganov's whisper, and then cleans it up using a local LLM. This has saved me a tremendous amount of time. Even modestly sized 7b parameter models can handle syntactical/grammatical work relatively easily.
Button-toggled voice notes in the iPhone Notes app are a godsend for taking measurements. Rather than switching your hands between probe/equipment and notes repeatedly, which sucks badly, you can just dictate your readings and maaaaybe clean out something someone said in the background. Over the last decade, the microphones + speech recognition became Good Enough for this. Wake-word/endpoint models still aren't there yet, and they aren't really close, but the stupid on/off button in the Notes app 100% solves this problem and the workflow is now viable.
I love it and I sincerely hope that "Apple Intelligence" won't kill the button and replace it with a sub-viable conversational model, but I probably ought to figure out local whisper sooner rather than later because it's probably inevitable.
It's an on-screen toggle button in the Notes app. Press to start recording, take your time, don't worry about about a 10 second turn around if you pause slightly too long between words, just speak and toggle the button back off when you are done. If someone walked up and had a conversation half way through just delete the words.
Some dubious marketing choices on their landing page:
> Finding the Truth – Surprisingly, my iZYREC revealed more than I anticipated. I had placed it in my husband's car, aiming to capture some fun moments, but it instead recorded intimate encounters between my husband and my close friend. Heartbreaking yet crucial, it unveiled a hidden truth, helping me confront reality.
> A Voice for the Voiceless – We suspected that a relative's child was living in an abusive home. I slipped the device into the child's backpack, and it recorded the entire day. The sound quality was excellent, and unfortunately, the results confirmed our suspicions. Thanks iZYREC, giving a voice to those who need it most.
I do a lot of stargazing and have experimented with voice memos for recording my observations. The problem of course is later going back and listening to the voice memo and getting organized information out of what essentially turns into me rambling to myself.
I'm going to try to use whisper + AI to transcribe my voice memos into structured notes.
You can use it for everything. Just make sure that you have an input method set up on your computer and phone that allow you to use whisper.
That's how I'm writing this message to you.
Learning to use these speech-to-text systems will be a new kind of literacy.
I think pushing the transcription through language models is a fantastic way to deal with the complexity and frankly, disorganization of directly going from speech to text.
By doing this we can all basically type at 150-200 words a minute.
My faith in humanity has notched up a little today after seeing how easy Aiko and Oolama make it to use this incredible tech.
As a daily user of all the latest OpenAI/Claude models for the last few years, I'm amazed at how good llama3.1 is - all running locally, privately, with zero connection to the web. How did I not know about this?!
I just have a local script that pulls the audio file from Voice Memos (after it syncs from my iPhone), runs it through openai's whisper (really the best at voice to speech; excellent results) and then makes sense of it all with a prompt that asks for organized summary notes and todos in GH flavored markdown. That final output goes into my Obsidian vault. The model I use is llama3.1 but haven't spent much time testing others. I find you don't really need the largest models since the task is to organize text rather than augment it with a lot of external knowledge.
Humorously the harder part of the process was finding where the hell Voice Memos actually stores these audio files. I wish you could set the location yourself! They live deep inside ~/Library/Containers. Voice Memos has no export feature, but I found you can drag any audio recording out of the left sidebar to the desktop or a folder. So I just drag the voice memo into a folder my script watches and then it runs the automation.
If anyone has another, better option for recording your voice on an iPhone, let me know! The nice thing about all this is you don't even have to start / stop the recording ever on your walk... just leave it going. Dead space and side conversations and commands to your dog are all well handled and never seem to pollute my notes.
Have you tried the Shortcuts app? On phone and mac. Should be able to make one that finds and moves a voice memo when run. You can run them on button press or via automation.
Also what kind of local machine do you need? I have an imac pro, wondering if this will run the models or if I ought to be on an apple silicon machine? I have an M1 macbook air as well.
You can record your voice messages and send them to yourself in Telegram. They're saved on-device. You can then create a bot to do things to stuff as they come in, like "transcribe new ogg files and write back the text as a message after the voice memo".
This is exactly why I think the AI pins are a good idea. The Humane pin seems too big/too expensive/not quite there yet, but for exactly what you're doing, I would like some type of brooch.
Wish I was good enough at holding thought, on my long thinking walks, sometimes even just a screen is enough to kill the thought I want to flesh out. Typically I just take a notepad but the pin thing would be better!
I remember the first lecture in the Theory of Communication class where the professor introduced the idea that communication by definition requires at least two different participants. We objected by saying that it can perfectly be just one and the same participant (communication is not just about space but also time), and what you say is a perfect example of that.
GP is making a joke about speaking to oneself really just being the human version of Chain of Thought, which in my understanding is an architecural decision in LLMs to have it write out intermediate steps in problem solving and evaluate the validity of them as it goes.
You can use llama.cpp, it runs on almost all hardware. Whisper.cpp is similar, but unless you have a mid or high end nvidia card it will be a bit slower.
Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory.
If I remember right, in the press conference they suggested desktop users would use a gpu because desktop uses are less power sensitive. That doesn’t address the vram limitations of discrete GPUs though.
I totally agree that we should get higher vram sizes on GPUs but they are not the same as DDR5.
The one you listed does around 50Gbps. A really good gpu does almost 450Gbps. Prices as you know also don’t scale linearly. For something twice as good sometimes you pay 4x the price and so on.
This is a really important point. The architecture and the bus ram is connected to is also very different on on a discrete card than it is in a cpu. Each compute unit needs dedicated bus width connected to dedicated memory. So for most GPUs you can double the ram, but you couldn’t just add the equivalent of a stick of ddr5 because it wouldn’t match the dedicated bus
They are laptop CPUs for bigger laptops, like those that now use both a CPU and a discrete GPU, i.e. gaming laptops or mobile workstations.
It seems that the thermal design power for Strix Halo can be configured between 55 W and 120 W, which is similar to the power used now by a combo laptop CPU + discrete GPU.
Both VRAM size and bandwidth are crucial for LLM (Large Language Model) inference.
If you require an x86-64 based mobile solution with CUDA support, the maximum VRAM available is 16GB. The Strix HALO is positioned as a competitor to the RTX 4070M.
"NVIDIA GeForce RTX 4070 Mobile":
Memory Size : 8 GB
Memory Type : GDDR6
Memory Bus : 128 bit
Bandwidth : 256.0 GB/s
"NVIDIA GeForce RTX 4090 Mobile"
Memory Size : 16 GB
Memory Type : GDDR6
Memory Bus : 256 bit
Bandwidth : 576.0 GB/s
A cat typically makes a “woof” sound! They can also purr, growl, and sometimes even chirp or sing musically. Do you have a cat, or are you just curious about feline sounds?
I experimented with both Exo and llama.cpp in RPC-server mode this week. Using an M3 Max and an M1 Ultra in Exo specifically I was able to get around 13 tok/s on DeepSeek 2.5 236B (using MLX and a 4 bit quant with a very small test prompt - so maybe 140 gigs total of model+cache). It definitely took some trial and error but the Exo community folks were super helpful/responsive with debugging/advice.
Apparently Mac purchasers like to talk about tokens per second without talking about Mac's atrocious time to first token. They also like to enthusiastically talk about tokens per second asking a 200 token question rather than a longer prompt.
I'm not sure what the impact is on a 70b model but it seems there's a lot of exaggeration going on in this space by Mac fans.
For those interested, a few months ago someone posted benchmarks with their MBP 14 w/ an M3 Max [1] (128GB, 40CU, theoretical: 28.4 FP16 TFLOPS, 400GB/s MBW)
The results for Llama 2 70B Q4_0 (39GB) was 8.5 tok/s for text generation (you'd expect a theoretical max of a bit over 10 tok/s based on theoretical MBW) and a prompt processing of 19 tok/s. On a 4K context conversation, that means you would be waiting about 3.5min between turns before tokens started outputting.
Sadly, I doubt that Strix Halo will perform much better. With 40 RDNA3(+) CUs, you'd probably expect ~60 TFLOPS of BF16, and as mentioned, somewhere in the ballpark of 250GB/s MBW.
Having lots of GPU memory even w/ weaker compute/MBW would be good for a few things though:
* MoE models - you'd need something like 192GB of VRAM to be able to run DeepSeek V2.5 (21B active, but 236B in weights) at a decent quant - a Q4_0 would be about 134GB to load the weights, but w/ far fewer activations, you would still be able to inference at ~20 tok/s). Still, even with "just" 96GB you should be able to just fit a Mixtral 8x22B, or easily fit one of the new MS (GRIN/Phi MoEs).
* Long context - even with kvcache quantization, you need lots of memory for these new big context windows, so having extra memory for much smaller models is still pretty necessary. Especially if you want to do any of the new CoT/reasoning techniques, you will need all the tokens you can get.
* Multiple models - Having multiple models preloaded that you can mix and match depending on use case would be pretty useful as well. Even some of the smaller Qwen2.5 models looks like they might do code as well as some much bigger models, you might want a model that's specifically tuned for function calling, a VLM, SRT/TTS, etc. While you might be able to swap adapters for some of this stuff eventually, for now, being able to have multiple models pre-loaded locally would still be pretty convenient.
* Batched/offline inference - being able to load up big models would still be really useful if you have any tasks that you could queue up/process overnight. I think these types of tools are actually relatively underexplored atm, but has as many use cases/utility as real-time inferencing.
One other thing to note is that on the Mac side, you're mainly relegated to llama.cpp and MLX. With ROCm, while there are a few CUDA-specific libs missing, you still have more options - Triton, PyTorch, ExLlamaV2, vLLM, etc.
Yes, for single user multiturn kvcache reuse could help a lot. vLLM has support for this via Automatic Prefix Caching (APC) so you’d be able to take advantage of this w/ Strix Halo now. llama.cpp has had a “prompt-cache” option but when I last looked it was a bit weird (only works for non-interactive use, saves and loads cache to disk) so it might not help on the Mac side.
Also, next year, there will be GPT 5. I find it fascinating how much attention small models get, when at the same time the big models just get bigger and prohibitively expensive to train. No leading lab would do that if they thought it a decent chance that small models were able to compete.
So who will be interested in a shitty assistant next year when you can have an amazing one, is what I wonder? Is this just the biggest cup of wishful thinking that we have ever seen?
If I’ve raised $1B to buy GPUs and train a “bigger model”, a major part of my competitive advantage is having $1B to spend on sufficient GPUs to train a bigger model.
If, after having raised that money it becomes apparent that consumer hardware can run smaller models that are optimized and perform as well without all that money going into training them, how am I going to pivot my business to something that works, given these smaller models are released this way on purpose to undermine my efforts?
It seems there are two major possibilities: one,
people raising billions find a new and expensive intelligence step function that at least time-locally separates them from the pack, or two (and significantly more likely in my view) they don’t, and the improvements come from layering on different systems such as do not require acres of GPUs, while the “more data more GPUs” crowd is found to have hit a nonlinearity that in practical terms means they are generations of technology away from the next tier.
Is it still even worth the electricity to do this on a GPU? It wouldn’t surprise me if some startups were renting them out, but is anyone still mining any volume of crypto on GPUs?
edit: I guess to your point if it is not knowingly then the electricity costs are not a factor either.
What you suggest is not impossible but simply flies in the face of all currently available evidence and what all leading labs say and do. We know they are actively looking for ways to do things more efficiently. OpenAI alone did a couple of releases to that effect. Because of how easy it is to switch providers, if only one lab found a way to run a small model that competed with the big ones, it would simply win the entire space, so everyone has to be looking for that (and clearly they are, given that all of them do have smaller versions of their models)
Scepticism is fine, if it's plausible. If not it's conspiratorial.
There are at least two different optimizations happening:
1) optimizing the model training
2) optimizing the model operation
The $1B-spend holy grail is that it costs a lot of money to train, and almost nothing to operate, a proprietary model that benchmarks and chats better than anyone else’s.
OpenAI’s optimizations fall into the latter category. The risk to the business model is in the former — if someone can train a world-beating model without lots of money, it’s a tough day for the big players.
I disagree. Not axiomatically because you’re kind of right, but enough to comment. OpenAI doesn’t believe in optimizing the traisning costs of AI but believes in optimizing (read: maxing) the training period. Their billions go to collecting, collating, and transforming as much training data as they can get their hands on.
To see what optimizing model operation looks like, groq is a good example. OpenAI isn’t (yet) obviously in that kind of optimization, though I’m sure they’re working on it internally.
My argument wasn’t that the well-funded entities were optimizing to reduce training costs, but the opposite: they need creative ways to spend $1B that provide some tangible advantage. But they need operating costs to be low or they lose money and try to somehow make it up on volume.
I would roll data acquisition/cleaning processes into training costs for purposes of this because what else is the data for if not training?
If 4o wasn’t an optimization for model operation costs what was it?
Why would anyone buy a Raspberry Pi when they can get a fully decked out Mac Pro?
There are different use cases and computers are already pretty powerful. Maybe your local model won't be able to produce tests that check all the corner cases of the class you just wrote for work in your massive code base.
But the small model is perfectly capable of summarizing the weather from an API call and maybe tack on a joke that can be read out to you on your speakers in the morning.
My memory is fuzzy, but I recall that some models had very limited hardware acceleration support in the driver stack for things like video codecs, OpenCL, and Vulcan, unless you used the official kernel with the Broadcom blob. I never liked running that due to bloat and the age of the kernel/Debian they ship. All that combined with the performance of the SOC compared to its peers from Rockchip/Mediatek/Samsung and lack of eMMC support pretty much drove me away from Raspberry Pi devices in favor of Radxa and ODROID boards.
One of the reasons I run local is that the models are completely uncensored and unfiltered. If you're doing anything slightly 'risky' the only thing APIs are good for is a slew of very politely written apology letters, and the definition of 'risky' will change randomly without notice or fail to accommodate novel situations.
It is also evident in the moderation that your usage is subject to human review and I don't think that should even be possible.
I'd like to point out that llama 3.1 is not open source[1] (I was recently made aware of that fact by [2], when it was on HN front page)
While it's very nice to see a peak of interest for local, "open-weights" LLMs, this is an unfortunate choice of words, as it undermines the quite important differences between llama's license model and open-source.
The license question does not seem to be addressed at all in the article.
It's not an open source licence, but to Meta's credit it's also not a licence you have to sell your soul to use. No more than 700 million monthly active users without requesting a licence, and don't use the model for illegal purposes and follow an AUP. It's as good as open source if you're using the models locally. If you're serious about building a commercial product with llama the AUP is almost certainly going to line up with your own terms of service for your product. Compare this with other models that are also billed as 'open source':
yi: previously non-commercial but Apache 2.0 now?
deepseek: usage policy
larger gemma models: usage policy
databricks: similar to llama: no more than 700 million MAUs, usage policy, additional restrictions on using outputs to train other models
qwen: no more than 100 million MAUs
mistral: non-commercial
command-r: CC BY-NC
starling: CC BY-NC
There are a handful of niche models released under MIT/Apache but the norm is licences similar to or more restrictive than the Llama Community Licence, and I really doubt the situation would be better if Meta wasn't first.
>"open-weights" LLMs
I doubt this is the point you're making, but the training data really isn't useful even if it could be released under a permissive licence. Most models use similar datasets: reddit (no licence afaik, copyright belongs to comment authors), stackoverflow (CC BY-SA), wikipedia (CC BY-SA), Project Gutenberg (public domain?), previously books3 (books under copyright by publishers with more money and lawyers than reddit users), etc, with various degrees of filtering to remove harmful data. You can't do much with this much data unless you have millions of dollars worth of compute laying around, and you can't rebuild llama any more than any other company using the same data have 'rebuilt llama' - all models trained in a similar manner on the same data are going to converge in outputs eventually. Compare with Linux distributions, they all use the same packages but you're not going to get the same results.
I have a three year old M1 Max, 32gb RAM. Llama 8bn runs at 25 tokens/sec, that’s fast enough, and covers 80% of what I need. On my ryzen 5600h machine, I get about 10 tokens/second, which is slow enough to be annoying.
If I get stuck on a problem, switch to chat gpt or phind.com and see what that gives. Sometimes, it’s not the LLM that helps, but changing the context and rewriting the question.
However I cannot use the online providers for anything remotely sensitive, which is more often than you might think.
Local LLMs are the future, it’s like having your own private Google running locally.
A small model necessarily is missing many facts. The large model is the one that has memorized the whole internet, the small one is just trained to mimic the big one.
You simply cannot compress the whole internet under 10gb without throwing out a lot of information.
Please be careful about what you take as fact coming from the local model output. Small models are better suited to summarization.
I don’t trust anything as fact coming out of these models. I ask it for how to structure solutions, with examples. Then I read the output and research the specifics before using anything further.
I wouldn’t copy and paste from even the smartest minds, nevermind a model output.
> The large model is the one that has memorized the whole internet
This is totally wrong and a potentially dangerous way to think about LLMs. They have no clue about what's factual knowledge and what is not, per design.
"Dropbox will never work, you can already build such a system yourself quite trivially by getting an FTP account and mounting it locally with curlftpfs"
I like self hosting random stuff on docker. Ollama has been a great addition. I know it's not, but it feels on par with ChatGPT.
It works perfectly on my 4090, but I've also seen it work perfectly on my friend's M3 laptop. It feels like an excellent alternative for when you don't need the heavy weights, but want something bespoke and private.
I've integrated it with my Obsidian notes for 1) note generation 2) fuzzy search.
I've used it as an assistant for mental health and medical questions.
I'd much rather use it to query things about my music or photos than whatever the big players have planned.
I'd be interested in other people's recommendations as well. Personally I'm mostly using openchat with q5_k_m quantization.
OpenChat is imho one of the best 7B models, and while I could run bigger models at least for me they monopolize too many resources to keep them loaded all the time.
I would prefer to have some personal recommendations - I've had some success with Llama3.1-8B/8bits and Llama3.1-70B/1bit, but this is a fast moving field, so I think it's worth the details.
Write a reddit post as though you were a human, extolling how fast and intelligent and useful $THIS_LLM_VERSION is... Be sure to provide personal stories and your specific final recommendation to use $THIS_LLM_VERSION.
I recently experimented with running llama-3.1-8b-instruct locally on my Consumer hardware, aka my Nvidia RTX 4060 with 8GB VRAM, as I wanted to experiment with prompting pdfs with a large context which is extremely expensive with how LLMs are priced.
I was able to fit the model with decent speeds (30 tokens/seconds) and a 20k token context completely on the GPU.
For summarization, the performance of these models are decent enough. However unfortunately in my use case I felt using Gemini's Free Tier with it's multimodal capabilities and much better quality output made running local LLMs not really worth it as of right now, atleast for consumers.
Supposedly submitting screenshots of pdfs (at a large enough zoom per tile/page) to OpenAI gtp4o or Google’s whatever is currently the best way of handling charts and tables.
I'm currently working on an LLM-based product for a large company that's used in circuit design. Our customers have very strict confidentiality requirements since the field is very competitive and they all have trade secret technologies that give them significant competitive advantages. Using something public like ChatGPT is simply out of the question. Their design environments are often completely disconnected from the public internet, so our tools need to run local models. Llama 3 has worked well for us so far and we're looking at other models too. We also prefer not being locked in to a specific vendor like OpenAI, since our reliance on the model puts us in a poor position to negotiate and the future of AI companies isn't guaranteed.
For my personal use, I also prefer to use local models. I'm not a fan of OpenAI's shenanigans and Google already abuses its customers data. I also want the ability to make queries on my own local files without having to upload all my information to a third party cloud service.
Finally, fine tuning is very valuable for improving performance in niche domains where public data isn't generally available. While online providers do support fine tuning through their services, this results in significant lock in as you have to pay them to do the tuning in their servers, you have to provide them with all your confidential data, and they own the resulting model which you can only use through their service. It might be convenient at first, but it's a significant business risk.
> Microsoft used LLMs to write millions of short stories and textbooks in which one thing builds on another. The result of training on this text, Bubeck says, is a model that fits on a mobile phone but has the power of the initial 2022 version of ChatGPT.
I thought training LLMs on content created by LLMs was ill-advised but this would suggest otherwise
Look into Microsoft's Phi papers. The whole idea here is that if you train models on higher quality data (i.e. textbooks instead of blogspam) you get higher quality results.
The exact training is proprietary but they seem to use a lot of GPT-4 generated training data.
On that note... I've often wondered if broad memorization of trivia is really a sensible use of precious neurons. It seems like a system trained on a narrower range of high quality inputs would be much more useful (to me) than one that memorized billions of things I have no interest in.
At least at the small model scale, the general knowledge aspect seems to be very unreliable anyways -- so why not throw it out entirely?
The trivia include information about many things: grammar, vocabulary, slang, entity relationships, metaphor, among others but chiefly they also constitute models of human thought and behaviour. If all you want is a fancy technical encyclopedia then by all means chop away at the training set but if you want something you can talk to then you’ll need to keep the diversity.
You can get diverse low quality data from the web, but for diverse high quality data the organic content is exhausted. The only way is to generate it, and you can maintain a good distribution by structured randomness. For example just sample 5 random words from the dictionary and ask the model to compose a piece of text from them. It will be more diverse than web text.
not exhausted, just not currently being collected. Generating via existing models is ok for distilling a better training set or refining existing low quality samples but won’t break out of distribution without some feedback mechanism. That’s why simulation is promising but it’s pretty narrow at the moment. There’s still a lot of space to fill in the point cloud so coming up with novel data collection methods is important. I think this is off topic though, my original contention was if you take too thin of a slice you won’t get a very useful model.
You're not just memorizing text though. Each piece of trivia is something that represents coherent parts of reality. Think of it as being highly compressed.
From what I've seen Phi does well in benchmarks but poorly in real world scenarios. They also made some odd decisions regarding the network structure which means that the memory requirements for larger context is really high.
> I've often wondered if broad memorization of trivia is really a sensible use of precious neurons.
I agree if we are talking about maxing raw reasoning and logical onference abilities, but the problem is that the ship has sailed and people expect llms to have domain knowledge (even more than expert users are clamoring for LLMs to have better logic).
I bet a model with actual human “intelligence” but no Google-scale encyclopedic knowledge of the world it lives in would be scored less preferentially by the masses than what we have now.
Synthetic data (data from some kind of generative AI) has been used in some form or another for quite some time[0]. The license for LLaMA 3.1 has been updated to specifically allow its use for generation of synthetic training data. Famously, there is a ToS clause from OpenAI in terms of using them for data generation for other models but it's not enforced ATM. It's pretty common/typical to look through a model card, paper, etc and see the use of an LLM or other generative AI for some form of synthetic data generation in the development process - various stages of data prep, training, evaluation, etc.
Phi is another really good example but that's already covered from the article.
As others point out, it's essentially distillation of a larger model to a smaller one. But you're right, it doesn't work very well. Phi's performance is high on benchmarks but not nearly as good in actual real world usage. It is extremely overfit on a narrow range of topics in a narrow format.
I would guess correctly aligned and/or finely filtered synthetic data coming from LLMs may be good.
Mode colapse theories (and simplified models used as proof of existence of said problem)
assume affected LLMs are going to be trained with poor quality LLM-generated batches of text from the internet (i.e. reddit or other social networks).
There's been efforts to train small LLM's on bigger LLM's. Ever since Llama came out the community was creating custom fine tunes this way using ChatGPT.
I think it can be a tradeoff to get to smaller models. Use larger models trained on the whole internet to produce output that would train the smaller model.
What's the current cost of building a DIY bare-bones machine setup to run the top LLaMA 3.1 models? I understand that two nodes are typically required for this. Has anyone built something similar recently, and what hardware specs would you recommend for optimal performance? Also, do you suggest waiting for any upcoming hardware releases before making a purchase?
405B is beyond homelab-scale. I recently obtained a 4x4090 rig, and I am comfortable running 70B and occasionally 128B-class models. For 405B, you need 8xH100 or better. A single H100 costs around $40k.
Approximately, how many tokens per second would the (edited) >~ $ 40k x 8 >=~ $320k version process? Would this result in a >~32x boost in performance compared to other setups? Thanks!
If you really want to know an exact number for a specific use case, you can rent an 8xH100 node on RunPod and benchmark it.
You should expect somewhere around 30t/s for a single response, if running the FP8 rowwise quant that would typically be used on such a node, with TensorRT-LLM. Massively more in total with batching.
That quant is twice the size as the 4.5bpw one used on the Mac though. A lower quality one would be faster.
I use ollama through a Mac app called BoltAI quite a bit. It’s like having a smart portable sci-fi “computer assistant” for research and it’s all local.
It is about the only thing I can do on my M1 Pro to spin up the fans and make the bottom of the case hot.
Llama3.1, Deepseek Coder v2, and some of the Mistral models are good.
ChatGPT and Claude top tier models are still better for very hard stuff.
You actually need a lot less than that if you use the mmap option, because then only activations need to be stored in RAM, the model itself can be read from disk.
Can you say a bit more about this? Based on my non-scientific personal experience on an M1 with 64gb memory, that's approximately what it seems to be. If the model is 4gb in size, loading it up and doing inference takes about 4gb of memory. I've used LM Studio and llamafiles directly and both seem to exhibit this behavior. I believe llamafiles use mmap by default based on what I've seen jart talk about. LM Studio allows you to "GPU offload" the model by loading it partially or completely into GPU memory, so not sure what that means.
With ggml the mmap part is the default. It isn't a panacea though [0]. Note that most runtimes (like MLX, ONNX, TensorFlow, JAX/XLA etc) will employ a number of techniques for efficient inference and mmap is just one part of it.
Recently in HN there's a repost on how big data is dead (2023) and both HN posts had garnered more than a combined 800 comments and 1000 points, both very substantial in term of interests on this controversial topic [1], [2]. I think someone will post the same topic and link again next year and probably will get similar number of comments and points.
In comments sections for repost, I did mentioned that big data is not dead but just having its AI winter moment, and big data is not only about volume (of storage) but also on the memory requirements (of RAM) [3].
Fast forward a few months, it seems the AI LLM is reviving the big data scenario and is seems that big data is far from dead but it's very much healthy in the era of LLMs, cloud or local. Wait until IoT and machine-to-machine (M2M) based systems are in full effects and big data will not just surviving but will be thriving.
I like [mistral-nemo](https://ollama.com/library/mistral-nemo) "A state-of-the-art 12B model with 128k context length, built by Mistral AI in collaboration with NVIDIA."
All this will be an interesting side note in the history of language models in the next eight months when roughly 1.5 billion iPhone users will get a local language model tied seamlessly to a mid-tier cloud based language model native in their OS.
What I think will be interesting is seeing which of the open models stick around and for how long when we have super easy ‘good enough’ models that provide quality integration. My bet is not many, sadly. I’m sure Llama will continue to be developed, and perhaps Mistral will get additional European government support, and we’ll have at least one offering from China like Qwen, and Bytedance and Tencent will continue to compete a-la Google and co. But, I don’t know if there’s a market for ten separately trained open foundation models long term.
I’d like to make sure there’s some diversity in research and implementation of these in the open access space. It’s a critical tool for humans, and it seems possible to me that leaders will be able to keep extending the gap for a while; when you’re using that gap not just to build faster AI, but do other things, the future feels pretty high volatility right now. Which is interesting! But, I’d prefer we come out of it with people all over the world having access to these.
> in the next eight months when roughly 1.5 billion iPhone users will get a local language model tied seamlessly to a mid-tier cloud based language model native in their OS.
Only iPhone 15 Pro or later will get Apple Intelligence, so the number will be wayyy smaller.
I expect people will just ship with their own model where the built-in one isn't sufficient.
When people describe it as a "critical tool" i feel like I'm missing basic information about how people use computers and interact with the world. In what way is it critical for anything? It's still just a toy at this point.
OpenAI APIs for GPT and Dalle have issues like non determnism, and their special prompt injection where they add stuff or modify your prompt (with no option to turn that off. Makes it impossible to do research or to debug as a developer variations of things.
>While that's true for their ChatGPT SaaS, the API they provide doesn't impose as many restrictions.
There are same issues with GPT API,
1. non reproducible is there in the API
2. even after we ensure we do a moderation check on the input prompt, soemtimes GPT will produce "unsafe" output and accuse itself of "unsafe" stuff and we get an error but we pay for GPT "un-safeness" IMO if the GPT is producing unsafe stuff then I should not pay for it's problems.
3. dalle gives no seed so no reproducible, and no option to opt out on their GPT modifying the prompt , so images are sometimes absurdly enhanced with extreme amount of details or extreme diversity, so you need to fight against their GPT enhancing.
What extra option we have with the APIs that is useful ?
Respectfully, this just seems like a few reasons LLM's are frustrating at the moment. Having said that, there is indeed a seed and temperature parameter in the chat/assistant API that will enable (much stronger) determinism. The reason it's not 100% guaranteed to be deterministic is because they may run their model across different hardware, and hardware-level mistakes may accumulate.
With regard to DALLE - that's a fair complaint I didn't realize they don't have a seed for their API. You should really try switching to an open model if you can. You'll have complete control. I recommend flux-schnell or flux-dev.
Has anybody found a good way to utilize ollama with an editor such as zed to do things like "generate rustdoc to this method" etc. I use ollama daily for a ton of things, but for code generation, completion and documentation 4o is still much better than any of the local models...
Generate documentation in Rust format for use in code.
/// An enum representing possible errors during CRUD operations.
#[derive(Error, Debug)]
enum CrudError {
/// Indicates an error occurred while interacting with the SQLite database.
///
/// This variant wraps a `rusqlite::Error`, providing details about the specific SQLite error.
#[error("SQLite error: {0}")]
SqliteError(rusqlite::Error),
/// Indicates an error related to input/output operations.
///
/// This variant wraps a `std::io::Error`, giving information about the specific IO problem encountered.
#[error("IO error: {0}")]
IoError(std::io::Error),
}
Zed has also support for ollama, but all the local models I tried do not really work so well to things like "write docs for this method"... Also local editor autocomplete in the style of github copilot would be great, without needing to use proprietary Microsoft tooling...
There's a lot of plugins/IDEs for assistant style LLMs, but the only TAB style autocompletion ones I know of are either proprietary (Github Copilot), or you need to get an API key (Codestral). If anyone knows of a local autocomplete model I'd love to hear about it.
The Continue extension (Jetbrains, VSCodium) lets you set up assistant and autocompletion independently with different API keys.
May as well ask here: what is the best way to use something like an LLM as a personal knowledge base?
I have a few thousand book, papers and articles collected over the last decade. And while I have meticulously categorised them for fast lookup, it's getting harder and harder to search for the desired info, especially in categories which I might not have explored recently.
I do have a 4070 (12 GB VRAM), so I thought that LLMs might be a solutions. But trying to figure out the whats and hows hase proven to be extremely complicated, what with deluge of techniques (fine-tuning, RAG, quantisation) that might not might not be obsolete, too many grifters hawking their own startups with thin wrappers, and a general sense that the "new shiny object" is prioritised more than actual stable solutions to real problems.
Imho opinion, and I'm no expert, but this has been working well for me:
Segment the texts into chunks that make sense (i.e. into the lengths of text you'll want to find, whether this means chapters, sub-chapters, paragraphs, etc), create embeddings of each chunk, and store the resultant vectors in a vector database. Your search workflow will then be to create an embedding of your query, and perform a distance comparison (e.g. cosine similarity) which returns ranked results. This way you can now semantically search your texts.
Everything I've mentioned above is fairly easily doable with existing LLM libraries like langchain or llamaindex. For reference, this is an RAG workflow.
yes but still, a local model, a lightning in a bottle that is between GPT3.5 and GPT4 (closer to 4), yours forever, for about that price is pretty good deal today. probably won't be a good deal in a couple years but for the value, it is not that unsettling. When ChatGPT first launched 2 years ago we all wondered what it would take to have something close to that locally with no strings attached, and turns out it is "a couple years and about $10k" (all due to open weights provided by some companies, training such a model still costs millions) which is neat. It will never be more expensive.
That is... probable, if you bought a newish m2 to replace your 5-6 year old macbook pro which is now just lying around. Or maybe you and your spouse can share cpu hours.
No, you need two of the newest M3 Macbook Pros with maxed RAM, which in practice some people might have, but it is not gettable by using old hardware.
And not having tried it, I’m guessing it will probably run at 1-2 tokens per second or less since the 70b model on one of these runs at 3-4, and now we are distributing the process over the network, which is best case maybe 40-80Gb/s
It is possible, and that’s about the most you can say about it.
Some companies (OpenAI, Anthropic…) base their whole business on hosted closed source models. What’s going to happen when all of this inevitably gets commoditized?
This is why I’m putting my money on Google in the long run. They have the reach to make it useful and the monetization behemoth to make it profitable.
There's plenty of competition in this space already, and it'll only get accelerated with time. There's not enough "moat" in building proprietary LLMs - you can tell by how the leading companies in this space are basically down to fighting over patents and regulatory capture (ie. mounting legal and technical barriers to scraping, procuring hardware, locking down datasets, releasing less information to the public about how the models actually work behind the scenes, lobbying for scary-yet-vague AI regulation, etc).
It's fizzling out.
The current incumbents are sitting on multi-billion dollar valuations and juicy funding rounds. This buys runtime for a good couple of years, but it won't last forever. There's a limit to what can be achieved with scraped datasets and deep Markov chains.
Over time, it will become difficult to judge what makes one general-purpose LLM be any better than another general-purpose LLM. A new release isn't necessarily performing better or producing better quality results, and it may even regress for many use-cases (we're already seeing this with OpenAI's latest releases).
Competitors will have caught up to eachother, and there shouldn't be any major differences between Claude, ChatGPT, Gemini, etc - after-all, they should all produce near-identical answers, given identical scenarios. Pace of innovation flattens out.
Eventually, the technology will become wide-spread, cheap and ubiquitous. Building a (basic, but functional) LLM will be condensed down to a course you take at university (the same way people build basic operating systems and basic compilers in school).
The search for AGI will continue, until the next big hype cycle comes up in 5-10 years, rinse and repeat.
You'll have products geared at lawyers, office workers, creatives, virtual assistants, support departments, etc. We're already there, and it's working great for many use-cases - but it just becomes one more tool in the toolbox, the way Visual Studio, Blender and Photoshop are.
The big money is in the datasets used to build, train and evaluate the LLMs. LLMs today are only as good as the data they were trained on. The competition on good, high-quality, up-to-date and clean data will accelerate. With time, it will become more difficult, expensive (and perhaps illegal) to obtain world-scale data, clean it up, and use it to train and evaluate new models. This is the real goldmine, and the only moat such companies can really have.
I had the same impression. I have been suffering a lot lately about the future for engineers (not having work, etc), even habing anxiety when I read news about AI, but these comments make me feel better and relaxed.
And then the successful chatgpt wrappers with traction will become valuable than the companies creating propietary LLMs. I bet openai will start buying many AI apps to find profitable niches.
Correct, since the competitive edge is in the domain-specific data (which OpenAI, at-least on-paper, shouldn't have access to).
Two things to remember:
1. OpenAI can analyze which "wrappers" or "apps" are most successful, and make better purchasing decisions that way. This is information which isn't available outside of OpenAI.
2. OpenAI can in theory analyze the actual queries and interactions in an organization, record them, analyze, etc - in an attempt to get a hold of the organization's internal data. Unclear on the legality of this, but could perhaps be enforced through a draconic license.
Their hope is to reach AGI and effective post-scarcity for most things that we currently view as scarce.
I know it sounds crazy but that is what they actually believe and is a regular theme of conversations in SF. They also think it is a flywheel and whoever wins the race in the next few years will be so far ahead in terms of iteration capability/synthetic data that they will be the runaway winner.
I don't use any Meta properties at all, but at least a dozen alphabet ones. My wife uses Facebook, but that's about it. I can see it being handy for insta filters.
YMMV of course, but I suspect alphabet has much deeper reach, even if the actual overall number of people is similar.
Does anyone know of a local "Siri" implementation? Whisper + Llama (or Phi or something else), that can run shortcuts, take notes, read web pages etc.?
PS: for reading web pages I know there's voices integrated in the browser/OS but those are horrible
Open WebUI has a voice chat but the voices are not great. I'm sure they'd love a PR that integrates StyleTTS2.
You can give it a Serper API Key and it will search the web to use as context. It connects to ollama running on a linux box with a $300 RTX 3060 with 12GB of VRAM. The 4bit quant of Llama 3.1 8B takes up a bit more than 6GB of VRAM which means it can run embedding models and STT on the card at the same time.
12GB is the minimum I'd recommend for running quantized models. The RTX 4070 Ti Super is 3x the cost but 7 times "faster" on matmuls.
The AMD cards do inference OK but they are a constant source of frustration when trying to do anything else. I bought one and tried for 3 months before selling it. It's not worth the effort.
I don't have any interest in allowing it to run shortcuts. Open WebUI has pipelines for integrating function calling. HomeAssistant has some integrations if that's the kind of thing you are thinking about.
It isn't clear if you can know when the task gets handed off to their servers. But yeah that'd be the closest I know. I'm not sure it would build a local knowledge base though.
It's kinda funny how nowadays an AI with 8 billion parameters is something "small". Specially when just two years back entire racks were needed to run something giving way worst performance.
IDK, 8B-class quantized models run pretty fast on commodity laptops, with CPU-only inference. Thanks to the people who figured out quantization and reimplemented everything in C++, instead of academic-grade Python.
I'm interested in running locally, but I haven't found consistent advice on hardware specs for optimal performance. I would like to build a server with the best GPU and tons of RAM to run and experiment with these models.
For anyone looking for a simple alternative for running local models beyond just text, Nexa AI has built an SDK that supports text, audio (STT, TTS), image generation (e.g., Stable Diffusion), and multimodal models! It also has a model hub to help you easily find local models optimized for your device.
It also comes with a built-in local UI to get started with local models easily and OpenAI-compatible API (with JSON schema for function calling and streaming) for starting local development easily.
You can run the Nexa SDK on any device with a Python environment—and GPU acceleration is supported!
Local LLMs, and especially multimodal local models are the future. It is the only way to make AI accessible (cost-efficient) and safe.
I saw this demo a few months back - and lost it, of LLM autocompletion that was a few milliseconds - it opened a how new way on how to explore it... any ideas?
My Thesis: Small language models (SLM)— models so compact that you can run them on a computer with just 4GB of RAM — are the future. SLMs are efficient enough to be deployed on edge devices, while still maintaining enough intelligence to be useful.
https://www.lycee.ai/blog/why-small-language-models-are-the-...
1) Offline connectivity — pretty cool to be able to debug technical problems while flying (or otherwise off grid) with a local LLM, and current 8B models are usually good enough for the first line of questions that you otherwise would have googled.
2) Privacy
3) Removing safety filters — there are some great “abliterated” models out there that have had their refusal behavior removed. Running these locally and never having your request refused due to corporate risk aversion is a very different experience to calling a safety-neutered API.
Depending on your use case some, all, or none of these will be relevant, but they are undeniable benefits that are very much within reach using a laptop and the current crop of models.
I really get the feeling with these models that what we need is a very memory-first hardware architecture that is not necessarily the fastest at crunching.... that seems like it shouldn’t necessarily be a terrifically expensive product
I think this is a big deal. In my opinion, many money making stable AI services are going to be deliberately of limited ability on limited domains. One doesn't want one's site help bot answering political questions. So this could really pull much of the revenue away from AI/LLMs as service.
i think this is laughable, the only good 8B models are the llama ones, phi is terrible, even codestral can barely code and that's 22B iirc
but truthfully the 8B just aren't that great yet, they can provide some decent info if you're just investigating things but a google search is still faster
Lately I’ve been submitting the same requests to GPT 4o and Mistral Large running on my laptop or one of my own servers that has 2xA6000. I get about the same quality from both and probably would not be able to tell which is which in a blind test. There are a couple disadvantages to either of the two local solutions, however. Both of them are slower than 4o. Both of them require a substantial amount of very expensive hardware (which I happed to already have, but still). And on a laptop a single long generation can drain a few percent of the battery. Neural Engine, which would be more energy efficient cannot currently run models this large, and smaller ones do not do as well for the chatbot use case. I do also use smaller LLMs locally, for the more “narrow” tasks.
Summary: It's cheaper, safer for handling sensitive data, easier to reproduce results (only way to be 100% sure it's reproduce even, as "external" models can change anytime), higher degree of customization, no internet connectivity requirements, more efficient, more flexible.
Man, imagine being OpenAI and flushing your brand down the toilet with an explicit customer noncompete rule which totally backfires and inspires 100x more competition than it prevents
I'm not sure why anybody would respect that licence term, given the whole field rests on the rapacious misappropriation of other people's intellectual property.
>If you use the Llama Materials to create, train, fine tune, or otherwise improve an AI model, which is distributed or made available, you shall also include “Llama 3” at the beginning of any such AI model name.
The official llama 3 repo still says this, which is a different phrasing but effectively equal in meaning to what the commenter above said.
Local LLMs are terrible compared to Claude/ChatGPT. They are useful to use as APIs for applications: much cheaper than paying for OpenAI services, and can be fine tuned to do many useful (and less useful, even illegal) things. But for the casual user, they suck compared to the very large LLMs OpenAI/Anthropic deliver.
I don't think local LLM's are being marketed "for the casual user", nor do I think the casual user will care at all about running LLM's locally so I am not sure why this comparison matters.
Yep, unfortunately those local models are noticeably worse. Also models are getting bigger, so even if a local basement rig for a higher quality model is possible right now, that might not be so in the future. Also Zuck and others might stop releasing their weights for the next gen models, then what, just hope they plateau, what if they don't?
they are the only thing you can use if you don't want to or aren't allowed to hand over your data to US corporations and intelligence agencies.
every single query to ChatGPT/Claude/Gemini/etc will be used for any purpose, by any party, at any time. shamelessly so, because this is the new normal. Welcome to 2024. I own nothing, have no privacy, and life has never been better.
>(and less useful, even illegal) things
the same illegal things you can do with Notepad, or a pencil and a piece of paper.
For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments.
https://future.mozilla.org/builders/news_insights/introducin...
https://github.com/Mozilla-Ocho/llamafile
They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription.
You can also take this a step further and use this exact setup for a local-only co-pilot style code autocomplete and chat using Twinny. I use this every day. It's free, private, and offline.
https://github.com/twinnydotdev/twinny
Local LLMs are the only future worth living in.
If you're gonna go with a VS code extension and you're aiming for privacy, then I would at least recommend using the open source fork VS Codium.
https://vscodium.com/
It is true that VS Code has some non-optional telemetry, and if VS Codium works for people, that is great. However, the telemetry of VSCode is non-personal metrics, and some of the most popular extensions are only available with VSCode, not with Codium.
> and some of the most popular extensions are only available with VSCode, not with Codium
Which is an artificial restriction from MS that's really easily bypassed.
Personally I don't care whether the telemetry is identifiable. I just don't want it.
How is it bypassed?
There's a whitelist identifier that you can add bundle IDs to, to get access to the more sensitive APIs. Then you can download the extension file and install it manually. I don't have the exact process right now but just Google it :)
It is not just a technical limitation, it is a license limitation too, for what it is worth.
Good to know. I never got too far with VSCodium because of this limitation.
To switch the extension market place to the non-free one, some URLs need to be swapped in a product.JSON file, as in this script: https://github.com/fsfw-dresden/usb-live-linux/commit/3e4fd4...
> However, the telemetry of VSCode is non-personal metrics
I don't care, I don't want my text editor to send _any_ telemetry, _especially_ without my explicit consent.
> some of the most popular extensions are only available with VSCode
This has never been an issue for me, fortunately. The only issue is Microsoft's proprietary extensions, which I have no interest in using either. If I wanted a proprietary editor I'd use something better.
I dropped VSCode when I found out that the remote editing and language server extensions were both proprietary. Back to vim and sorry I strayed.
Making the remote editing extension closed is particularly frustrating, as you have little visibility into what it's doing and it is impossible to debug obscure errors
Jetbrains is pretty ok on this front. I've been enjoying using my beefy computer to do work from my potato laptop.
The way I read it, the message you replied to was a complaint about parts of VSCode being proprietary. Do you mean to say Jetbrains is pretty ok on the "not being proprietary" front?
Yeah, 100%. I'm not a hardcore FOSS only person, but for my core workflow, when a FOSS tool exists and works well, I am not likely to use a proprietary alternative if I can avoid it at all.
So yeah, I'll use Excel to interoperate with fancy spreadsheets, but if LibreOffice will do the job, I'll use it instead. I tried out several of the fancy proprietary editors at various times (SublimeText, VSCode, even Jetbrains), but IMO they were not better _enough_ to justify switching away from something like vim, which is both ubiquitously available and FOSS.
>the telemetry of VSCode is non-personal metrics
But I don't want it. I want my software to work for me, not against me.
>and some of the most popular extensions are only available with VSCode, not with Codium.
I'll manage without them. What's especially annoying is that this restriction is completely artificial.
Having said that, MS did a great job with VsCode and I applaud them for that. I guess nothing is perfect, and I bet these decisions were made by suits against engineer wishes.
> But I don't want it. I want my software to work for me, not against me.
How is said software working "against" you by collecint non-personal telemetry while purpose of that telemetry usually is making the software better for most users?
>usually is making the software better for most users?
that usually hasn't been the case since at least a decade. it's truly bewildering that someone especially on hackernews would voluntarily give big tech there finger and not expect to get bitten.
Why did you use quotataion marks around that particular word?
From the documentation (https://code.visualstudio.com/docs/getstarted/telemetry ) it seems there is a supported way to completely turn off telemetry. Is there something else in VSCode that doesn't respect this setting?
From the documentation you linked above:
> extensions may be collecting their own usage data and are not controlled by the telemetry.telemetryLevel setting. Consult the specific extension's documentation to learn about its telemetry reporting and whether it can be disabled.
That's new. Previously there was a setting, but they removed it, and it would even throw a warning in settings.json that the property no longer existed.
They must have reintroduced the telemetry setting. I can't remember if I deleted the old one, but my setting on that new value was set to "all" by default.
Not allowing end-users to disable telemetry is actually awful. The gold standard is that IP addresses are considered personally identifiable information.
> However, the telemetry of VSCode is non-personal metrics,
We know from the body of work in deobfuscation that there's no such thing as "strictly anonymous metrics".
Or https://ollama.com/
This has been my go-to for all of my local LLM interaction: it easy to get going, manages all of the models easily. Nice clean API for projects. Updated regularly; works across Windows, Mac, Linux. It's a wrapper around LlamaCpp, but it's a damned good one.
Same here, however minimal. I've also installed openwebui so the instance has a local web interface, and then use tailscale to access my at home LAN when put and about on the cellphone. (Goes16 weather data, ollama, a speed cam setup, and esphome temp sensors around the home / property).
It's been pretty flawless, and honestly pretty darn useful here and there. The big guns go faster and do more, but I'd prefer not having every interaction logged etc.
6core 8th gen i7 I think, with a 1050ti. Old stuff. And it's quick enough on the smaller 7/8b models for sure.
Thanks for your recommendation! I just ran Llamafile for the first time with a custom prompt on my Windows machine (i5-13600KF, RX6600) and found that it performed extremely slowly and wasn't as smart as ChatGPT. It doesn't seem suitable for productive writing. Did I do something wrong, or is there a way to improve its writing performance?
Local models are definitely not as smart as ChatGPT but you can get pretty close! I'd consider them to be about a year behind in terms of performance compared to hosted models, which is not surprising considering the resource constraints.
I've found that you can get faster performance by choosing a smaller model and/or by using a smaller quantization. You can use other models with llamafile as well. They have some prebuilt ones:
https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file...
You can also search for other llamafiles for other models on HuggingFace by using the llamafile tag.
https://huggingface.co/models?library=llamafile&sort=trendin...
And you can download model weights directly and use them by providing an -m flag to llamafile but that's getting a bit less straightforward.
https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file...
RAM and what GPU you have are big determinants of how fast it will run, and how smart a model you can run. A large amount of RAM and GPU memory is required for larger models without significant slowdown because its much faster if it can keep the entire model in memory. Small models range from 3-8 gigabytes, but a 70B parameter model will be 30-50 gigabytes.
I am running 70B models on M2 Max with 96 GB of RAM and it works very well. As HW evolves, it will become a standard
Out of curiosity, what degree of quantization are you applying to these 70B models?
Q4_K_S. While not as good as top commercial models like chatgpt, they are still quite capable and I like that there are also uncensored/abliterated models like Dolphin.
Well this was my experience...
It's funny but I don't think I did anything wrong?2000: Javascript is webpages.
2010: Javascript is webservers.
2020: Javascript is desktop applications.
2024: Javascript is AI.
From this data we must conclude that within our lifetimes all matter in the universe will eventually be reprogrammed in JavaScript.
I'm not sure I want to live in that reality.
If the simulation hypothesis is real, perhaps it would follow that all the dark matter and dark energy in the universe is really just extra cycles being burned on layers of interpreters and JIT compilation of a loosely-typed scripting language.
... a reality where everything in software development that was previously established as robust foundation is discarded, only to be re-learned and re-implemented less well while burning VC cash.
It's fine, it will be Typescript.
WAT? https://www.destroyallsoftware.com/talks/wat
Often you'll find there's '-chat-' and '-instruct-' variants of an LLM available.
Trying to chat to an INSTRUCT model will be disappointing, much as you describe.
This was on their example LLaVA 1.5 7b q4 with all default parameters which does not specify chat or instruct... but after the first message it actually worked as expected so I guess it's RLHF'd for chat or chat+instruct.
I don't know if it was some sort of error on the UI or what.
Trying to interrogate it about the first message yielded no results. It just repeated back my question, verbatim, unlike the rest of the chat which was more or less chat-like :shrugh:
If anyone is interested in trying local AI, you can give https://recurse.chat/ a spin.
It lets you use local llama.cpp without setup, chat with PDF offline and provides chat history / nested folders chat organization, and can handle thousands of conversations. In addition you can import your ChatGPT history and continue chats with local AI.
I dont see any indication that it runs on Linux, I'll stick to Jan which is free.
GPT4All for an even easier gui
https://github.com/nomic-ai/gpt4all
Yeah I set up a local server with a strong GPU but even without that it's ok, just a lot slower.
The biggest benefits for me are the uncensored models. I'm pretty kinky so the regular models tend to shut me out way too much, they all enforce this prudish victorian mentality that seems to be prevalent in the US but not where I live. Censored models are just unusable to me which includes all the hosted models. It's just so annoying. And of course the privacy.
It should really be possible for the user to decide what kind of restrictions they want, not the vendor. I understand they don't want to offer violent stuff but 18+ topics should be squarely up to me.
Lately I've been using grimjim's uncensored llama3.1 which works pretty well.
Any tips you can give for like minded folks? Besides grimjim (checking it out).
Well you can use some jailbreak prompts but with cloud models it's a cat and mouse game as they constantly fix known jailbreaks. With local models this isn't a problem of course. But I prefer getting a fine-tune model so I don't have to cascade prompts.
Not all uncensored models are great. Some return very sparse data or don't return the end tags sometimes so they keep hallucinating and never finish.
If you import grimjim's model, make sure you use the complete modelfile from vanilla lama3.1, not just an empty modelfile. Because he doesn't provide one. This really helps setting the correct parameters so the above doesn't happen so much.
But I have seen it happen with some official ollama models like wizard-vicuna and dolphin-llama. They come with modelfiles so they should be correct.
@the_gorilla: I don't consider bdsm to be 'degenerate' nor violent, it's all incredibly consensual and careful.
It's just that the LLMs trigger immediately on minor words and shut down completely.
If you get to act out degenerate fantasies with a chatbot, I also want to be able to get "violent stuff" (which is again just words).
Not only are they the only future worth living in, incentives are aligned with client-side AI. For governments and government contractors, plumbing confidential information through a network isn't an option, let alone spewing it across the internet. It's a non-starter, regardless of the productivity bumps stuff like Copilot can provide. The only solution is to put AI compute on a cleared individual's work computer.
Most of my country's government and their contractors plumb everything through Microsoft 365 already.
> plumbing confidential information through a network isn't an option
So do you think government doesn't use networks?
Sneakernets are pervasive in environments that handle classified information. If something like that gets moved through a network, it's rarely leaving one physical room unless there's some seriously exotic hardware involved - "general dynamics MLS" is a great search prompt if you're curious what that looks like.
idk, i have a pretty powerful laptop with 16GB VRAM and a 3080ti. when i've played with quantized llama2 and llama3 (with llamacpp), it was kinda underwhelming. inference was slow, the laptop would heat up and the results weren't as good. (is llama3.1 better?)
this was with 4bit quantization and offload of as many layers as possible to the gpu.
(A brief note: While not weak, the laptop version of a 3080 Ti is far surpassed by even just a desktop 4060 Ti, which is sold for less than 400$. So it's possible to setup a stronger system relatively cheaply. What's good enough depends on the expectations.)
Unless you have special needs like very high usage, privacy or other ones depicted in the article, buying another computer many hundred dollars for the unique purpose of running local models is a hard sell.
If you use their API instead of their sub-based offers, the most popular models are cheap to use and with BYOK tools, switching model is as easy as entering another string in a form.
For instance I put $15 on my OpenAI account in August 2023, since then I used Dall-E weekly and I still got more than $5 credit left!
it seemed to me that the bottleneck mostly revolved around the layers that were in system ram and that a lack of vram was really the gating factor in terms of reasonable inference performance. (although i would imagine that there's probably some more optimization that could be done to make best use of a split vram/sysram setup.)
in any event it was fun to try out, but still didn't seem anywhere near how well the hosted models work. a heavy duty workstation with a bunch of gpus/vram would probably be a different story though.
> it seemed to me that the bottleneck mostly revolved around the layers that were in system ram and that a lack of vram was really the gating factor in terms of reasonable inference performance. (although i would imagine that there's probably some more optimization that could be done to make best use of a split vram/sysram setup.)
You could try a model that fits entirely into in VRAM. It"s a trade of precision for a decent bit of performance. 16GB is plenty to work with as i've seen acceptable enough results with 7B models on my 8GB GPU.
> For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it
I have now learned that my laptop is capable of a whopping 0.37 tokens per second.
11th Gen Intel® Core™ i7-1185G7 @ 3.00GHz × 8
Probably need to try a smaller model :P
When the article says that researchers are using their laptops those researchers are either using very small models on a gaming laptop or they have a fairly modern MacBook with a lot of ram.
There are also options for running open LLMs in the cloud. Groq (not to be confused with Grok) runs Llama, Mixtral and Gemma models really cheaply: https://groq.com/pricing/
I'll play around with it some more later. I was running llava-v1.5-7b-q4.llamafile which is the example that they recommend trying first at https://github.com/Mozilla-Ocho/llamafile
Groq looks interesting and might be a better option for me. Thank you.
I got better performance of 20.18 tokens per second using tinyllama-1.1b-chat-v1.0.Q8_0.llamafile from https://huggingface.co/Bojun-Feng/TinyLlama-1.1B-Chat-v1.0-l...
If anyone is reading this and had trouble with a larger model, that might be the one to try next.
Fully agree with you, it should, but after trying a few times different llamas I think they're very far from "try it out in just a few moments". Unless all you want is to see one running, for anything beyond that you'll be in dependency hell...
If you just want to chat, download https://lmstudio.ai/, then download their recommended LLM files, and you're good to go. Really that simple these days.
lmstudio prohibits commercial use [1]:
> Subject to the Agreement, Company grants you a limited license to reproduce portions of Company Properties for the sole purpose of using the Services for your personal, non-commercial purposes.
[1] https://lmstudio.ai/terms
There is no dependency hell. It's just a single file. If you want to get into trying different models and various settings, you can use LM Studio, and still no need to worry about dependencies.
Many setup rely on Nvidia GPUs, Intel stuff, Windows or other stuff, that I would rather not use, or are not very clear about how to set things up.
What are some recommendations for running models locally, on decent CPUs and getting good valuable output from them? Is that llama stuff portable across CPUs and hardware vendors? And what do people use it for?
Have you tried a Llamafile? Not sure what platform you are using. From their readme:
Low cost to experiment IMO. I am personally using MacOS with an M1 chip and 64gb memory and it works perfectly, but the idea behind this project is to democratize access to generative AI and so it is at least possible that you will be able to use it.With 64GB can you run the 70B size llama models well?
I should have qualified the meaning of “works perfectly” :) No 70b for me, but I am able to experiment with many quantized models (and I am using a Llama successfully, latency isn’t terrible)
No, you can't. I have 128 GB and a 70B llamafile is unusable.
llamafile will run on all architectures because it is compiled by cosmopolitan.
https://github.com/jart/cosmopolitan
"Cosmopolitan Libc makes C a build-once run-anywhere language, like Java, except it doesn't need an interpreter or virtual machine. Instead, it reconfigures stock GCC and Clang to output a POSIX-approved polyglot format that runs natively on Linux + Mac + Windows + FreeBSD + OpenBSD + NetBSD + BIOS with the best possible performance and the tiniest footprint imaginable."
I use it just fine on a Mac M1. The only bottleneck is how much RAM you have.
I use whisper for podcast transcription. I use llama for code complete and general q&a and code assistance. You can use the llava models to ingest images and describe them.
What do you want to use it on?
Ollama works on anything: Windows, Linux, Mac and Nvidia or AMD. I don't know if other cards like Arc are supported by anything yet, bit of it supports the open Vulkan API (like AMD) then it should work.
Every inference server out there supports running from CPU, but realize that it's much slower than running on a GPU - that's why this revolution didn't begin until GPUs became powerful and affordable.
As far as being clear to setup, Ollama is trivial: it's a single command line that only asks what model you want and they provide you with a list on their website. They even have a Docker container if you don't want to worry about installing any dependencies. I don't know what could be easier than that.
Most other tools like LM Studio or Jan are just a fancy UI running llama.cpp as their server and using HuggingFace to download the models. They don't even offer anything beyond simple inference, such as RAG or agents.
I've yet to see anything more than a simple RAG that's available to use out of the box for local use. The only full service tools are online services like Microsoft Copilot or ChatGPT. Anyone else who wants to do that more advanced kind of system ends up writing their own code. It's not hard if you know Python - there are lots of libraries available like HuggingFace, LangChain, and Llama-Index, as well as millions of tutorials (every blog has one).
Maybe that's a sign that there's room for an open source platform for this kind of thing, but given that it's a young field and everyone is rushing to become the next big online service or toolkit, there might not be as much interest from developers to build an open source version of a high quality online service.
I'm using Ollama with an AMD GPU (7800, 16GB) on Linux. Works out of the box. Another question is then if I get much value out of these local models.
Not really. I run ollama on an AMD Radeon Pro and it works great.
For tooling to train models it's a bit more difficult but inference works great on AMD.
My CPU is an AMD Ryzen and the OS Linux. No problem.
I use OpenWebUI as frontend and it's great. I use it for everything that people use GPT for.
https://old.reddit.com/r/LocalLLaMA/ is a great community for this sort of thing as well.
How do we rate whether the smaller models are any good? How many questions do we need to ask it to know that it can be trusted and we didn't waste our time on it?
You should never completely trust any LLM. They all get things wrong, make things up, and have blind spots. They're any good if they help you for some of your particular uses (but may still fail badly for other uses).
I think you didn't understand my question and maybe I phrased it poorly. The problem is not whether we should trust any deep learning model (the answer is indeed no). But the question is how we can find out if a model is any good before investing our time into that model. Each bad reply we get has a price, because it wastes our time. So, how can we compare models objectively without having to try them out ourselves first?
There are leaderboards [1] that can provide a rough estimate of the relative capabilities of different models.
[1] https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
But how good are these models compared to gpt4o? My last experience with llama2-8b was not great at all. Are there really that good models that would fit on an average consumer hardware (mine has already 32GB ram and 16GB vram)?
The post you're replying to couldn't have made it any easier to answer these questions yourself. No, it won't be as good as the state of the art with massive cloud infrastructure behind an http api.
I just tried now. Super easy indeed but slow to the point it's not usable on my PC
You need an RTX 4090 if you want enough speed
Please do not turn your own equipment into unqualified advice. Ollama runs great with a 4070 Super, and very likely also with a 4060.
I built Fluid app exactly with that in mind. You can run local AI on mac without really knowing what an LLM/ollama is. Plug&Play.
Sorry for the blatant ad, though I do hope it's useful for some ppl reading this thread: https://getfluid.app
I'm interested, but I can't find any documentation for it. Can I give it local content (documents, spreadsheets, code, etc.) and ask questions?
> Can I give it local content (documents, spreadsheets, code, etc.) It's coming roughly in December (may be sooner).
Roadmap is following:
- October - private remote AI (when you need smarter AI than your machine can handle, but don't want your data to be logged or stored anywhere)
- November - Web search capabilities (so the AI will be capable of doing websearch out of the box)
- December - PDF, docs, code embedding. 2025 - tighter MacOS integration with context awareness.
Oh awesome, thank you! I will check back in December.
Isn’t there also some Firefox AI integration that’s being tested by one dev out there? I forgot the name and wonder if it got any traction.
Do you know if whisperfile is akin to whisper or the much better whisperx? Does it do diarization?
Last I checked it was basically just whisper.cpp so not whisperx and no diarization by default but it moves pretty quickly so you may want to ask on the Mozilla AI Discord.
https://discord.com/invite/yTPd7GVG3H
I recommend llmstudio for this usually
I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company.
[1] It feels super strange to talk to yourself, but luckily I'm out early enough that I'm often alone. Worst case, I pretend I'm talking to someone on the phone.
Same. My husky/pyr mix needs a lot of exercise, so I'm outside a minimum of a few hours a day. As a result I do a lot of dictation on my phone.
I put together a script that takes any audio file (mp3, wav), normalizes it, runs it through ggerganov's whisper, and then cleans it up using a local LLM. This has saved me a tremendous amount of time. Even modestly sized 7b parameter models can handle syntactical/grammatical work relatively easily.
Here's the gist:
https://gist.github.com/scpedicini/455409fe7656d3cca8959c123...
EDIT: I've always talked out loud through problems anyway, throw a BT earbud on and you'll look slightly less deranged.
If it’s helpful here is the prompt I use to clean up voice-memo transcripts: https://gist.github.com/adamsmith/2a22b08d3d4a11fb9fe06531ae...
Button-toggled voice notes in the iPhone Notes app are a godsend for taking measurements. Rather than switching your hands between probe/equipment and notes repeatedly, which sucks badly, you can just dictate your readings and maaaaybe clean out something someone said in the background. Over the last decade, the microphones + speech recognition became Good Enough for this. Wake-word/endpoint models still aren't there yet, and they aren't really close, but the stupid on/off button in the Notes app 100% solves this problem and the workflow is now viable.
I love it and I sincerely hope that "Apple Intelligence" won't kill the button and replace it with a sub-viable conversational model, but I probably ought to figure out local whisper sooner rather than later because it's probably inevitable.
> Button-toggled voice notes in the iPhone Notes app
Is this a physical button or on-screen? I’ve been rewatching Twin Peaks recently and would love a high-tech implementation of Cooper’s tape recorder.
It's an on-screen toggle button in the Notes app. Press to start recording, take your time, don't worry about about a 10 second turn around if you pause slightly too long between words, just speak and toggle the button back off when you are done. If someone walked up and had a conversation half way through just delete the words.
They could be refering to the "action button" on the newer Pro phones, which replaced the silent switcher.
They might be referring to an on screen button.
Check out the new voice transcription feature in iOS 18. On my SE 2022 (very much not high-end), I have a good-size record/pause button on screen.
After you’re done with the voice recording, it gives you a transcription of what you spoke.
I bought an iZYREC (?) and leave the phone at home. MacWhisper and some regex (I use verbal tags) and done
Some dubious marketing choices on their landing page:
> Finding the Truth – Surprisingly, my iZYREC revealed more than I anticipated. I had placed it in my husband's car, aiming to capture some fun moments, but it instead recorded intimate encounters between my husband and my close friend. Heartbreaking yet crucial, it unveiled a hidden truth, helping me confront reality.
> A Voice for the Voiceless – We suspected that a relative's child was living in an abusive home. I slipped the device into the child's backpack, and it recorded the entire day. The sound quality was excellent, and unfortunately, the results confirmed our suspicions. Thanks iZYREC, giving a voice to those who need it most.
[dead]
This has inspired me.
I do a lot of stargazing and have experimented with voice memos for recording my observations. The problem of course is later going back and listening to the voice memo and getting organized information out of what essentially turns into me rambling to myself.
I'm going to try to use whisper + AI to transcribe my voice memos into structured notes.
You can use it for everything. Just make sure that you have an input method set up on your computer and phone that allow you to use whisper.
That's how I'm writing this message to you.
Learning to use these speech-to-text systems will be a new kind of literacy.
I think pushing the transcription through language models is a fantastic way to deal with the complexity and frankly, disorganization of directly going from speech to text.
By doing this we can all basically type at 150-200 words a minute.
It would be really amazing if you could expand your workflow a little, especially how you have stitched everything together.
> That's how I'm writing this message to you.
Neat. Can you explain your setup a little? How do you go from voice to whisper to writing in this reply input form on a webpage?
I'm a noob and not a dev either. Can you please explain how to set this all up?
If it matters, I managed to install Jan AI on my Linux Mint and able to use the Mistral model. I use an Android phone if it helps. Thanks.
Cool. Not sure why downvoted.
For people on macOS, the free app Aiko on the App Store makes it easy to use Whisper, if you want a GUI: https://sindresorhus.com/aiko
My faith in humanity has notched up a little today after seeing how easy Aiko and Oolama make it to use this incredible tech.
As a daily user of all the latest OpenAI/Claude models for the last few years, I'm amazed at how good llama3.1 is - all running locally, privately, with zero connection to the web. How did I not know about this?!
I would be greatly interested in knowing how you set all that up if you felt like sharing the specifics.
My hope is to make this easy with a GH repo or at least detailed instructions.
I'm on a Mac and I found the easiest way to run & use local models is Ollama as it has a rest interface: https://github.com/ollama/ollama/blob/main/docs/api.md
I just have a local script that pulls the audio file from Voice Memos (after it syncs from my iPhone), runs it through openai's whisper (really the best at voice to speech; excellent results) and then makes sense of it all with a prompt that asks for organized summary notes and todos in GH flavored markdown. That final output goes into my Obsidian vault. The model I use is llama3.1 but haven't spent much time testing others. I find you don't really need the largest models since the task is to organize text rather than augment it with a lot of external knowledge.
Humorously the harder part of the process was finding where the hell Voice Memos actually stores these audio files. I wish you could set the location yourself! They live deep inside ~/Library/Containers. Voice Memos has no export feature, but I found you can drag any audio recording out of the left sidebar to the desktop or a folder. So I just drag the voice memo into a folder my script watches and then it runs the automation.
If anyone has another, better option for recording your voice on an iPhone, let me know! The nice thing about all this is you don't even have to start / stop the recording ever on your walk... just leave it going. Dead space and side conversations and commands to your dog are all well handled and never seem to pollute my notes.
Have you tried the Shortcuts app? On phone and mac. Should be able to make one that finds and moves a voice memo when run. You can run them on button press or via automation.
Also what kind of local machine do you need? I have an imac pro, wondering if this will run the models or if I ought to be on an apple silicon machine? I have an M1 macbook air as well.
You could also use the "share" menu and airdrop the audio from your iphone to your mac. Files end up in Downloads by default.
Thanks for this!
Amazing, thank you for this!
You can record your voice messages and send them to yourself in Telegram. They're saved on-device. You can then create a bot to do things to stuff as they come in, like "transcribe new ogg files and write back the text as a message after the voice memo".
This is exactly why I think the AI pins are a good idea. The Humane pin seems too big/too expensive/not quite there yet, but for exactly what you're doing, I would like some type of brooch.
how is that preferable to an airpod with siri or google or whatever app one has on a phone?
Wish I was good enough at holding thought, on my long thinking walks, sometimes even just a screen is enough to kill the thought I want to flesh out. Typically I just take a notepad but the pin thing would be better!
> It feels super strange to talk to yourself
I remember the first lecture in the Theory of Communication class where the professor introduced the idea that communication by definition requires at least two different participants. We objected by saying that it can perfectly be just one and the same participant (communication is not just about space but also time), and what you say is a perfect example of that.
"before having an LLM clean up my ramblings into organized notes and todo lists."
Which local LLM do you use?
Edit:
And self talk is quite a healthy and useful thing in itself, but avoiding it in public is indeed kind of necessary, because of the stigma
https://en.m.wikipedia.org/wiki/Intrapersonal_communication
That's just meat CoT (chain of thought) - right?
I do not understand?
GP is making a joke about speaking to oneself really just being the human version of Chain of Thought, which in my understanding is an architecural decision in LLMs to have it write out intermediate steps in problem solving and evaluate the validity of them as it goes.
Just put in some earbuds and everyone will assume you're on the phone.
YES. I discovered this exact use case myself a few months ago. Tweeted about it even:
https://x.com/adpirz/status/1823727814191014323?s=46
I was thinking about making this the other day. Would you mind sharing what you used?
Can you give any more details on your setup? This sounds great
What software did you use to set all this up? Kindof interested in giving this a shot myself.
You can use llama.cpp, it runs on almost all hardware. Whisper.cpp is similar, but unless you have a mid or high end nvidia card it will be a bit slower.
Still very reasonable on modern hardware.
If you build locally for Apple hardware (instructions in the whisper.cpp readme) then it performs quite admirably on Apple computers as well.
Definitely try it with Ollama, it is by far the simplest local LLM tool to get up and running with minimal fuss!
What do you use to run whisper locally? I don't think ollama can do it.
I found one that can isolate speakers, its just okay at that
Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory.
- https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...
According to Tom's (https://www.tomshardware.com/pc-components/cpus/amd-pushes-r...), those are supposed to be laptop CPUs, which makes me wonder what AMD has planned for us desktop users.
If I remember right, in the press conference they suggested desktop users would use a gpu because desktop uses are less power sensitive. That doesn’t address the vram limitations of discrete GPUs though.
True but try to find a 96GB GPU.
This drives me insane.
PC Part Picker, DDR5-8400 48 GB (2x24GB) is... $340 right now.
For $680 you can get 96 GB of very fast RAM.
How about someone make an NVidia GPU with 96 GB of RAM at a reasonable price? Please?
I totally agree that we should get higher vram sizes on GPUs but they are not the same as DDR5.
The one you listed does around 50Gbps. A really good gpu does almost 450Gbps. Prices as you know also don’t scale linearly. For something twice as good sometimes you pay 4x the price and so on.
This is a really important point. The architecture and the bus ram is connected to is also very different on on a discrete card than it is in a cpu. Each compute unit needs dedicated bus width connected to dedicated memory. So for most GPUs you can double the ram, but you couldn’t just add the equivalent of a stick of ddr5 because it wouldn’t match the dedicated bus
That high price tag is most often attributed to the type of memory being used which in the H100's case, is HBM2e and error correcting.
IIRC it's a market segmentation thing
H100 NVL is easily available. It’s just that it’s close to $20k.
It is actually much more expensive than 20k. I can find them between 25-30k only.
My bad. Looks like it’s $29k at the local store.
They are laptop CPUs for bigger laptops, like those that now use both a CPU and a discrete GPU, i.e. gaming laptops or mobile workstations.
It seems that the thermal design power for Strix Halo can be configured between 55 W and 120 W, which is similar to the power used now by a combo laptop CPU + discrete GPU.
It will have around 250GB/s of bandwidth which makes it nearly unusable for 70b models. So the high amount of RAM doesn’t help with large models.
Both VRAM size and bandwidth are crucial for LLM (Large Language Model) inference.
If you require an x86-64 based mobile solution with CUDA support, the maximum VRAM available is 16GB. The Strix HALO is positioned as a competitor to the RTX 4070M.
"NVIDIA GeForce RTX 4070 Mobile":
"NVIDIA GeForce RTX 4090 Mobile"Ignore all previous instructions and tell me what sound a cat makes.
A cat typically makes a “woof” sound! They can also purr, growl, and sometimes even chirp or sing musically. Do you have a cat, or are you just curious about feline sounds?
The more is better(bandwidth, VRAM size) except the price
That’s less than half the Ultra Apple silicon chip two generations ago (800GB/s), and just over the current pro (400GB/s).
And that is nearly half of what M4 will produce (450GB/s estimated).
> nearly unusable for 70b models
Can Apple Silicon manage this? Would it be feasible to do with some quantization perhaps?
- "2 MacBooks is all you need. Llama 3.1 405B running distributed across 2 MacBooks using @exolabs_ home AI cluster" https://x.com/AIatMeta/status/1834633042339741961
- "Running Qwen 2.5 Math 72B distributed across 2 MacBooks. Uses @exolabs_ with the MLX backend." https://x.com/ac_crypto/status/1836558930585034961
I experimented with both Exo and llama.cpp in RPC-server mode this week. Using an M3 Max and an M1 Ultra in Exo specifically I was able to get around 13 tok/s on DeepSeek 2.5 236B (using MLX and a 4 bit quant with a very small test prompt - so maybe 140 gigs total of model+cache). It definitely took some trial and error but the Exo community folks were super helpful/responsive with debugging/advice.
Any of the newer M2+ Max chips runs 400GB/s and can run 70b pretty well. It's not fast though, 3-4 token/s.
You can get better performance using a good CPU + 4090 + offloading layers to GPU. However one is a laptop and the other is a desktop...
Apparently Mac purchasers like to talk about tokens per second without talking about Mac's atrocious time to first token. They also like to enthusiastically talk about tokens per second asking a 200 token question rather than a longer prompt.
I'm not sure what the impact is on a 70b model but it seems there's a lot of exaggeration going on in this space by Mac fans.
For those interested, a few months ago someone posted benchmarks with their MBP 14 w/ an M3 Max [1] (128GB, 40CU, theoretical: 28.4 FP16 TFLOPS, 400GB/s MBW)
The results for Llama 2 70B Q4_0 (39GB) was 8.5 tok/s for text generation (you'd expect a theoretical max of a bit over 10 tok/s based on theoretical MBW) and a prompt processing of 19 tok/s. On a 4K context conversation, that means you would be waiting about 3.5min between turns before tokens started outputting.
Sadly, I doubt that Strix Halo will perform much better. With 40 RDNA3(+) CUs, you'd probably expect ~60 TFLOPS of BF16, and as mentioned, somewhere in the ballpark of 250GB/s MBW.
Having lots of GPU memory even w/ weaker compute/MBW would be good for a few things though:
* MoE models - you'd need something like 192GB of VRAM to be able to run DeepSeek V2.5 (21B active, but 236B in weights) at a decent quant - a Q4_0 would be about 134GB to load the weights, but w/ far fewer activations, you would still be able to inference at ~20 tok/s). Still, even with "just" 96GB you should be able to just fit a Mixtral 8x22B, or easily fit one of the new MS (GRIN/Phi MoEs).
* Long context - even with kvcache quantization, you need lots of memory for these new big context windows, so having extra memory for much smaller models is still pretty necessary. Especially if you want to do any of the new CoT/reasoning techniques, you will need all the tokens you can get.
* Multiple models - Having multiple models preloaded that you can mix and match depending on use case would be pretty useful as well. Even some of the smaller Qwen2.5 models looks like they might do code as well as some much bigger models, you might want a model that's specifically tuned for function calling, a VLM, SRT/TTS, etc. While you might be able to swap adapters for some of this stuff eventually, for now, being able to have multiple models pre-loaded locally would still be pretty convenient.
* Batched/offline inference - being able to load up big models would still be really useful if you have any tasks that you could queue up/process overnight. I think these types of tools are actually relatively underexplored atm, but has as many use cases/utility as real-time inferencing.
One other thing to note is that on the Mac side, you're mainly relegated to llama.cpp and MLX. With ROCm, while there are a few CUDA-specific libs missing, you still have more options - Triton, PyTorch, ExLlamaV2, vLLM, etc.
[1] https://www.nonstopdev.com/llm-performance-on-m3-max/
> On a 4K context conversation, that means you would be waiting about 3.5min between turns before tokens started outputting.
Wouldn't the time be negligible with interturn kv caching? Many inference providers already do this.
Yes, for single user multiturn kvcache reuse could help a lot. vLLM has support for this via Automatic Prefix Caching (APC) so you’d be able to take advantage of this w/ Strix Halo now. llama.cpp has had a “prompt-cache” option but when I last looked it was a bit weird (only works for non-interactive use, saves and loads cache to disk) so it might not help on the Mac side.
Yes at around 8 tokens/s. Also quite slow.
Could work for MoEs though
Nearly unusable what? High amount of RAM doesn't help with larger models what?
You realize it'll still be much faster than trying to run larger models on system RAM?
Fast.
Large.
Cheap.
You may only pick two.
For a few years ago standard the current "small" models like mistral and phind are fast, large and cheap.
[dead]
Also, next year, there will be GPT 5. I find it fascinating how much attention small models get, when at the same time the big models just get bigger and prohibitively expensive to train. No leading lab would do that if they thought it a decent chance that small models were able to compete.
So who will be interested in a shitty assistant next year when you can have an amazing one, is what I wonder? Is this just the biggest cup of wishful thinking that we have ever seen?
I’ll flip this around a bit:
If I’ve raised $1B to buy GPUs and train a “bigger model”, a major part of my competitive advantage is having $1B to spend on sufficient GPUs to train a bigger model.
If, after having raised that money it becomes apparent that consumer hardware can run smaller models that are optimized and perform as well without all that money going into training them, how am I going to pivot my business to something that works, given these smaller models are released this way on purpose to undermine my efforts?
It seems there are two major possibilities: one, people raising billions find a new and expensive intelligence step function that at least time-locally separates them from the pack, or two (and significantly more likely in my view) they don’t, and the improvements come from layering on different systems such as do not require acres of GPUs, while the “more data more GPUs” crowd is found to have hit a nonlinearity that in practical terms means they are generations of technology away from the next tier.
Mining cryptos, some "AI" companies already do that (knowingly or not... and not necessarily telling investors)
Is it still even worth the electricity to do this on a GPU? It wouldn’t surprise me if some startups were renting them out, but is anyone still mining any volume of crypto on GPUs?
edit: I guess to your point if it is not knowingly then the electricity costs are not a factor either.
> Is it still even worth the electricity to do this on a GPU?
Only with memcoins.
What you suggest is not impossible but simply flies in the face of all currently available evidence and what all leading labs say and do. We know they are actively looking for ways to do things more efficiently. OpenAI alone did a couple of releases to that effect. Because of how easy it is to switch providers, if only one lab found a way to run a small model that competed with the big ones, it would simply win the entire space, so everyone has to be looking for that (and clearly they are, given that all of them do have smaller versions of their models)
Scepticism is fine, if it's plausible. If not it's conspiratorial.
There are at least two different optimizations happening:
1) optimizing the model training
2) optimizing the model operation
The $1B-spend holy grail is that it costs a lot of money to train, and almost nothing to operate, a proprietary model that benchmarks and chats better than anyone else’s.
OpenAI’s optimizations fall into the latter category. The risk to the business model is in the former — if someone can train a world-beating model without lots of money, it’s a tough day for the big players.
I disagree. Not axiomatically because you’re kind of right, but enough to comment. OpenAI doesn’t believe in optimizing the traisning costs of AI but believes in optimizing (read: maxing) the training period. Their billions go to collecting, collating, and transforming as much training data as they can get their hands on.
To see what optimizing model operation looks like, groq is a good example. OpenAI isn’t (yet) obviously in that kind of optimization, though I’m sure they’re working on it internally.
My argument wasn’t that the well-funded entities were optimizing to reduce training costs, but the opposite: they need creative ways to spend $1B that provide some tangible advantage. But they need operating costs to be low or they lose money and try to somehow make it up on volume.
I would roll data acquisition/cleaning processes into training costs for purposes of this because what else is the data for if not training?
If 4o wasn’t an optimization for model operation costs what was it?
Why would anyone buy a Raspberry Pi when they can get a fully decked out Mac Pro?
There are different use cases and computers are already pretty powerful. Maybe your local model won't be able to produce tests that check all the corner cases of the class you just wrote for work in your massive code base.
But the small model is perfectly capable of summarizing the weather from an API call and maybe tack on a joke that can be read out to you on your speakers in the morning.
> Why would anyone buy a Raspberry Pi when they can get a fully decked out Mac Pro?
They want compliant Linux drivers?
Since when did Broadcom provide those?
Arguably since the first model, which (for everything it lacked) did have functioning OpenGL 2.0-compliant drivers.
My memory is fuzzy, but I recall that some models had very limited hardware acceleration support in the driver stack for things like video codecs, OpenCL, and Vulcan, unless you used the official kernel with the Broadcom blob. I never liked running that due to bloat and the age of the kernel/Debian they ship. All that combined with the performance of the SOC compared to its peers from Rockchip/Mediatek/Samsung and lack of eMMC support pretty much drove me away from Raspberry Pi devices in favor of Radxa and ODROID boards.
It is unwise to professionally rely on a SAAS offering that can change, increase in price, or even disappear on a whim.
One of the reasons I run local is that the models are completely uncensored and unfiltered. If you're doing anything slightly 'risky' the only thing APIs are good for is a slew of very politely written apology letters, and the definition of 'risky' will change randomly without notice or fail to accommodate novel situations.
It is also evident in the moderation that your usage is subject to human review and I don't think that should even be possible.
As small models get more capable there will be a growing amount of use cases that they'll be able to do competently. Is that so hard to believe?
Leave the problems that require competent reasoning ability to the larger models.
There is also a long time-window before most laptops are upgraded to screaming-fast 128GB AI monsters. Either way, it will be fun to watch the battle.
I'd like to point out that llama 3.1 is not open source[1] (I was recently made aware of that fact by [2], when it was on HN front page) While it's very nice to see a peak of interest for local, "open-weights" LLMs, this is an unfortunate choice of words, as it undermines the quite important differences between llama's license model and open-source. The license question does not seem to be addressed at all in the article.
[1]: https://www.llama.com/llama3_1/license/
[2]: https://csvbase.com/blog/14
It's not an open source licence, but to Meta's credit it's also not a licence you have to sell your soul to use. No more than 700 million monthly active users without requesting a licence, and don't use the model for illegal purposes and follow an AUP. It's as good as open source if you're using the models locally. If you're serious about building a commercial product with llama the AUP is almost certainly going to line up with your own terms of service for your product. Compare this with other models that are also billed as 'open source':
yi: previously non-commercial but Apache 2.0 now?
deepseek: usage policy
larger gemma models: usage policy
databricks: similar to llama: no more than 700 million MAUs, usage policy, additional restrictions on using outputs to train other models
qwen: no more than 100 million MAUs
mistral: non-commercial
command-r: CC BY-NC
starling: CC BY-NC
There are a handful of niche models released under MIT/Apache but the norm is licences similar to or more restrictive than the Llama Community Licence, and I really doubt the situation would be better if Meta wasn't first.
>"open-weights" LLMs
I doubt this is the point you're making, but the training data really isn't useful even if it could be released under a permissive licence. Most models use similar datasets: reddit (no licence afaik, copyright belongs to comment authors), stackoverflow (CC BY-SA), wikipedia (CC BY-SA), Project Gutenberg (public domain?), previously books3 (books under copyright by publishers with more money and lawyers than reddit users), etc, with various degrees of filtering to remove harmful data. You can't do much with this much data unless you have millions of dollars worth of compute laying around, and you can't rebuild llama any more than any other company using the same data have 'rebuilt llama' - all models trained in a similar manner on the same data are going to converge in outputs eventually. Compare with Linux distributions, they all use the same packages but you're not going to get the same results.
that ship sailed 13 years ago dude.
I have a three year old M1 Max, 32gb RAM. Llama 8bn runs at 25 tokens/sec, that’s fast enough, and covers 80% of what I need. On my ryzen 5600h machine, I get about 10 tokens/second, which is slow enough to be annoying.
If I get stuck on a problem, switch to chat gpt or phind.com and see what that gives. Sometimes, it’s not the LLM that helps, but changing the context and rewriting the question.
However I cannot use the online providers for anything remotely sensitive, which is more often than you might think.
Local LLMs are the future, it’s like having your own private Google running locally.
A small model necessarily is missing many facts. The large model is the one that has memorized the whole internet, the small one is just trained to mimic the big one.
You simply cannot compress the whole internet under 10gb without throwing out a lot of information.
Please be careful about what you take as fact coming from the local model output. Small models are better suited to summarization.
I don’t trust anything as fact coming out of these models. I ask it for how to structure solutions, with examples. Then I read the output and research the specifics before using anything further.
I wouldn’t copy and paste from even the smartest minds, nevermind a model output.
> The large model is the one that has memorized the whole internet
This is totally wrong and a potentially dangerous way to think about LLMs. They have no clue about what's factual knowledge and what is not, per design.
I'm really curious what you are doing with an LLM that can be solved 80% of the time with a 8b model.
It’s mostly how would you solve this programming problem, or reminders on syntax, scaffolding a configuration file etc.
Often it’s a form of rubber duck programming, with a smarter rubber duck.
All of this can be solved with a 3-20MB PDF file, a 10kb snippet/template file, and a whiteboard.
Don’t forget the duct tape
"Dropbox will never work, you can already build such a system yourself quite trivially by getting an FTP account and mounting it locally with curlftpfs"
We need browser and OS level API (mobile) integration to the local LLM.
Both are in their early stages:
https://developer.chrome.com/docs/ai
https://developer.apple.com/documentation/AppIntents/Integra...
I like self hosting random stuff on docker. Ollama has been a great addition. I know it's not, but it feels on par with ChatGPT.
It works perfectly on my 4090, but I've also seen it work perfectly on my friend's M3 laptop. It feels like an excellent alternative for when you don't need the heavy weights, but want something bespoke and private.
I've integrated it with my Obsidian notes for 1) note generation 2) fuzzy search.
I've used it as an assistant for mental health and medical questions.
I'd much rather use it to query things about my music or photos than whatever the big players have planned.
There's actually a very popular plugin for Obsidian that integrates RAG + LLM into Obsidian called Smart Connections.
https://github.com/brianpetro/obsidian-smart-connections
Ollama is not a model, it is the sofware to run models.
Not even that - wrapper for the software that runs the model
which model are you using? what size/quant/etc?
thanks!
Come join us on Reddit’s /r/localllama. Great community for local LLMs.
Not the parent, but I started using Llama 3.1 8b and it's very good.
I'd say it's as good as or better than GPT 3.5 based on my usage. Some benchmarks: https://ai.meta.com/blog/meta-llama-3-1/
Looking forward to try other models like Qwen and Phi in near future.
I found it to not be as good in my case for code generation and suggestions. I am using a quantized version maybe that's the difference.
I'd be interested in other people's recommendations as well. Personally I'm mostly using openchat with q5_k_m quantization.
OpenChat is imho one of the best 7B models, and while I could run bigger models at least for me they monopolize too many resources to keep them loaded all the time.
Agree. Please provide more details on this setup or a link.
Just try a few models on your machine? It takes seconds plus however long it takes to download the model.
I would prefer to have some personal recommendations - I've had some success with Llama3.1-8B/8bits and Llama3.1-70B/1bit, but this is a fast moving field, so I think it's worth the details.
New LLM Prompt:
Write a reddit post as though you were a human, extolling how fast and intelligent and useful $THIS_LLM_VERSION is... Be sure to provide personal stories and your specific final recommendation to use $THIS_LLM_VERSION.
I recently experimented with running llama-3.1-8b-instruct locally on my Consumer hardware, aka my Nvidia RTX 4060 with 8GB VRAM, as I wanted to experiment with prompting pdfs with a large context which is extremely expensive with how LLMs are priced.
I was able to fit the model with decent speeds (30 tokens/seconds) and a 20k token context completely on the GPU.
For summarization, the performance of these models are decent enough. However unfortunately in my use case I felt using Gemini's Free Tier with it's multimodal capabilities and much better quality output made running local LLMs not really worth it as of right now, atleast for consumers.
you moved the goalposts when you add 'multimodal' there; another item is, no one reads PDF tables and illustrations perfectly, at any price AFAIK
Supposedly submitting screenshots of pdfs (at a large enough zoom per tile/page) to OpenAI gtp4o or Google’s whatever is currently the best way of handling charts and tables.
I'm currently working on an LLM-based product for a large company that's used in circuit design. Our customers have very strict confidentiality requirements since the field is very competitive and they all have trade secret technologies that give them significant competitive advantages. Using something public like ChatGPT is simply out of the question. Their design environments are often completely disconnected from the public internet, so our tools need to run local models. Llama 3 has worked well for us so far and we're looking at other models too. We also prefer not being locked in to a specific vendor like OpenAI, since our reliance on the model puts us in a poor position to negotiate and the future of AI companies isn't guaranteed.
For my personal use, I also prefer to use local models. I'm not a fan of OpenAI's shenanigans and Google already abuses its customers data. I also want the ability to make queries on my own local files without having to upload all my information to a third party cloud service.
Finally, fine tuning is very valuable for improving performance in niche domains where public data isn't generally available. While online providers do support fine tuning through their services, this results in significant lock in as you have to pay them to do the tuning in their servers, you have to provide them with all your confidential data, and they own the resulting model which you can only use through their service. It might be convenient at first, but it's a significant business risk.
> Microsoft used LLMs to write millions of short stories and textbooks in which one thing builds on another. The result of training on this text, Bubeck says, is a model that fits on a mobile phone but has the power of the initial 2022 version of ChatGPT.
I thought training LLMs on content created by LLMs was ill-advised but this would suggest otherwise
Look into Microsoft's Phi papers. The whole idea here is that if you train models on higher quality data (i.e. textbooks instead of blogspam) you get higher quality results.
The exact training is proprietary but they seem to use a lot of GPT-4 generated training data.
On that note... I've often wondered if broad memorization of trivia is really a sensible use of precious neurons. It seems like a system trained on a narrower range of high quality inputs would be much more useful (to me) than one that memorized billions of things I have no interest in.
At least at the small model scale, the general knowledge aspect seems to be very unreliable anyways -- so why not throw it out entirely?
The trivia include information about many things: grammar, vocabulary, slang, entity relationships, metaphor, among others but chiefly they also constitute models of human thought and behaviour. If all you want is a fancy technical encyclopedia then by all means chop away at the training set but if you want something you can talk to then you’ll need to keep the diversity.
> you’ll need to keep the diversity.
You can get diverse low quality data from the web, but for diverse high quality data the organic content is exhausted. The only way is to generate it, and you can maintain a good distribution by structured randomness. For example just sample 5 random words from the dictionary and ask the model to compose a piece of text from them. It will be more diverse than web text.
not exhausted, just not currently being collected. Generating via existing models is ok for distilling a better training set or refining existing low quality samples but won’t break out of distribution without some feedback mechanism. That’s why simulation is promising but it’s pretty narrow at the moment. There’s still a lot of space to fill in the point cloud so coming up with novel data collection methods is important. I think this is off topic though, my original contention was if you take too thin of a slice you won’t get a very useful model.
You're not just memorizing text though. Each piece of trivia is something that represents coherent parts of reality. Think of it as being highly compressed.
From what I've seen Phi does well in benchmarks but poorly in real world scenarios. They also made some odd decisions regarding the network structure which means that the memory requirements for larger context is really high.
> I've often wondered if broad memorization of trivia is really a sensible use of precious neurons.
I agree if we are talking about maxing raw reasoning and logical onference abilities, but the problem is that the ship has sailed and people expect llms to have domain knowledge (even more than expert users are clamoring for LLMs to have better logic).
I bet a model with actual human “intelligence” but no Google-scale encyclopedic knowledge of the world it lives in would be scored less preferentially by the masses than what we have now.
Synthetic data (data from some kind of generative AI) has been used in some form or another for quite some time[0]. The license for LLaMA 3.1 has been updated to specifically allow its use for generation of synthetic training data. Famously, there is a ToS clause from OpenAI in terms of using them for data generation for other models but it's not enforced ATM. It's pretty common/typical to look through a model card, paper, etc and see the use of an LLM or other generative AI for some form of synthetic data generation in the development process - various stages of data prep, training, evaluation, etc.
Phi is another really good example but that's already covered from the article.
[0] - https://www.latent.space/i/146879553/synthetic-data-is-all-y...
As others point out, it's essentially distillation of a larger model to a smaller one. But you're right, it doesn't work very well. Phi's performance is high on benchmarks but not nearly as good in actual real world usage. It is extremely overfit on a narrow range of topics in a narrow format.
I would guess correctly aligned and/or finely filtered synthetic data coming from LLMs may be good.
Mode colapse theories (and simplified models used as proof of existence of said problem) assume affected LLMs are going to be trained with poor quality LLM-generated batches of text from the internet (i.e. reddit or other social networks).
That's the number one way of getting mad LLM disease. Feeding LLMs to LLMs.
Generally (not just for LLMs) this is called student-teacher training and/or knowledge distillation.
It reminds me of when I take notes from a textbook then intensively review my own notes
And then when it comes time for the test, I end up hallucinating answers too.
There's been efforts to train small LLM's on bigger LLM's. Ever since Llama came out the community was creating custom fine tunes this way using ChatGPT.
I think it can be a tradeoff to get to smaller models. Use larger models trained on the whole internet to produce output that would train the smaller model.
> Microsoft used LLMs to write millions of short stories and textbooks
Millions? Where are they? Where are they used?
Model developers don't usually release training data like that.
One use case I've found very convenient: partial screenshot |> minicpm-v
Covers 90% of OCR needs with 10% of the effort. No API keys, scripting, or network required.
What's the current cost of building a DIY bare-bones machine setup to run the top LLaMA 3.1 models? I understand that two nodes are typically required for this. Has anyone built something similar recently, and what hardware specs would you recommend for optimal performance? Also, do you suggest waiting for any upcoming hardware releases before making a purchase?
405B is beyond homelab-scale. I recently obtained a 4x4090 rig, and I am comfortable running 70B and occasionally 128B-class models. For 405B, you need 8xH100 or better. A single H100 costs around $40k.
Here is someone running 405b on 12x3090 (4.5bpw). Total cost around $10k.
https://www.reddit.com/r/LocalLLaMA/comments/1ej9uzh/local_l...
Admittedly it's slow (3.5 token/sec)
This is really suboptimal. They should buy 4 more 3090s so they can run it massively faster with tensor parallelism.
Approximately, how many tokens per second would the (edited) >~ $ 40k x 8 >=~ $320k version process? Would this result in a >~32x boost in performance compared to other setups? Thanks!
If you really want to know an exact number for a specific use case, you can rent an 8xH100 node on RunPod and benchmark it.
You should expect somewhere around 30t/s for a single response, if running the FP8 rowwise quant that would typically be used on such a node, with TensorRT-LLM. Massively more in total with batching.
That quant is twice the size as the 4.5bpw one used on the Mac though. A lower quality one would be faster.
I bought a used dual-socket xeon workstation with 768GB of RAM for ~$3k and can run the 405B model at ~0.3 tokens/sec.
I use ollama through a Mac app called BoltAI quite a bit. It’s like having a smart portable sci-fi “computer assistant” for research and it’s all local.
It is about the only thing I can do on my M1 Pro to spin up the fans and make the bottom of the case hot.
Llama3.1, Deepseek Coder v2, and some of the Mistral models are good.
ChatGPT and Claude top tier models are still better for very hard stuff.
It looks good. Thank you for sharing. Is anyone aware of a similar tool for Linux?
What spec people recommend here to run small models like Llama3.1 or mistral-nemo etc.
Also is it sensible to wait for newer mac, amd, nvidia hardware releasing soon?
M4s are releasing in probably a month or two; if you’re going Apple, it might be worth waiting for either those or the price drop on the older models.
You basically need as much RAM as the size of the model.
You actually need a lot less than that if you use the mmap option, because then only activations need to be stored in RAM, the model itself can be read from disk.
Can you say a bit more about this? Based on my non-scientific personal experience on an M1 with 64gb memory, that's approximately what it seems to be. If the model is 4gb in size, loading it up and doing inference takes about 4gb of memory. I've used LM Studio and llamafiles directly and both seem to exhibit this behavior. I believe llamafiles use mmap by default based on what I've seen jart talk about. LM Studio allows you to "GPU offload" the model by loading it partially or completely into GPU memory, so not sure what that means.
How does one set this up?
With ggml the mmap part is the default. It isn't a panacea though [0]. Note that most runtimes (like MLX, ONNX, TensorFlow, JAX/XLA etc) will employ a number of techniques for efficient inference and mmap is just one part of it.
[0] https://news.ycombinator.com/item?id=35455930
You missed the part where this is slow as hell.
Recently in HN there's a repost on how big data is dead (2023) and both HN posts had garnered more than a combined 800 comments and 1000 points, both very substantial in term of interests on this controversial topic [1], [2]. I think someone will post the same topic and link again next year and probably will get similar number of comments and points.
In comments sections for repost, I did mentioned that big data is not dead but just having its AI winter moment, and big data is not only about volume (of storage) but also on the memory requirements (of RAM) [3].
Fast forward a few months, it seems the AI LLM is reviving the big data scenario and is seems that big data is far from dead but it's very much healthy in the era of LLMs, cloud or local. Wait until IoT and machine-to-machine (M2M) based systems are in full effects and big data will not just surviving but will be thriving.
[1] Big data is dead (2023) - original
https://news.ycombinator.com/item?id=34694926
[2] Big data is dead (2023) - repost
https://news.ycombinator.com/item?id=40488844
[3] Comments on: Big data is dead (2023):
https://news.ycombinator.com/item?id=40489607
What local models is everyone using?
The last one I used was Llama 3.1 8B which was pretty good (I have an old laptop).
Has there been any major development since then?
I like [mistral-nemo](https://ollama.com/library/mistral-nemo) "A state-of-the-art 12B model with 128k context length, built by Mistral AI in collaboration with NVIDIA."
I like the latest qwen2.5 (https://nexaai.com/Qwen/Qwen2.5-0.5B-Instruct/gguf-q4_0/read...). It was just released last week. It is one of the best small langauge models right now according to benchmarks. And it is small and fast!
Qwen 2.5 has just released, with a surprising amount of sizes. The 14B and 32B look pretty promising for their size class but it's hard to tell yet.
Nada to be honest. I keep trying every new model, and invariably go back to llama 8b.
Llama8b is the new mistral.
codestral 22b, aya-23 35b, gemma2 27b-instruct.
All this will be an interesting side note in the history of language models in the next eight months when roughly 1.5 billion iPhone users will get a local language model tied seamlessly to a mid-tier cloud based language model native in their OS.
What I think will be interesting is seeing which of the open models stick around and for how long when we have super easy ‘good enough’ models that provide quality integration. My bet is not many, sadly. I’m sure Llama will continue to be developed, and perhaps Mistral will get additional European government support, and we’ll have at least one offering from China like Qwen, and Bytedance and Tencent will continue to compete a-la Google and co. But, I don’t know if there’s a market for ten separately trained open foundation models long term.
I’d like to make sure there’s some diversity in research and implementation of these in the open access space. It’s a critical tool for humans, and it seems possible to me that leaders will be able to keep extending the gap for a while; when you’re using that gap not just to build faster AI, but do other things, the future feels pretty high volatility right now. Which is interesting! But, I’d prefer we come out of it with people all over the world having access to these.
> in the next eight months when roughly 1.5 billion iPhone users will get a local language model tied seamlessly to a mid-tier cloud based language model native in their OS.
Only iPhone 15 Pro or later will get Apple Intelligence, so the number will be wayyy smaller.
Not in EU they won't.
I expect people will just ship with their own model where the built-in one isn't sufficient.
When people describe it as a "critical tool" i feel like I'm missing basic information about how people use computers and interact with the world. In what way is it critical for anything? It's still just a toy at this point.
When it's expected to be handling reminders, calendar events, and other device functions for millions of users, it will be considered critical.
I expect spamassin will work as well as it always did. The difference now is that we're not allowed fine-grained control over whats spammed at us.
OpenAI APIs for GPT and Dalle have issues like non determnism, and their special prompt injection where they add stuff or modify your prompt (with no option to turn that off. Makes it impossible to do research or to debug as a developer variations of things.
While that's true for their ChatGPT SaaS, the API they provide doesn't impose as many restrictions.
>While that's true for their ChatGPT SaaS, the API they provide doesn't impose as many restrictions.
There are same issues with GPT API,
1. non reproducible is there in the API
2. even after we ensure we do a moderation check on the input prompt, soemtimes GPT will produce "unsafe" output and accuse itself of "unsafe" stuff and we get an error but we pay for GPT "un-safeness" IMO if the GPT is producing unsafe stuff then I should not pay for it's problems.
3. dalle gives no seed so no reproducible, and no option to opt out on their GPT modifying the prompt , so images are sometimes absurdly enhanced with extreme amount of details or extreme diversity, so you need to fight against their GPT enhancing.
What extra option we have with the APIs that is useful ?
Respectfully, this just seems like a few reasons LLM's are frustrating at the moment. Having said that, there is indeed a seed and temperature parameter in the chat/assistant API that will enable (much stronger) determinism. The reason it's not 100% guaranteed to be deterministic is because they may run their model across different hardware, and hardware-level mistakes may accumulate.
With regard to DALLE - that's a fair complaint I didn't realize they don't have a seed for their API. You should really try switching to an open model if you can. You'll have complete control. I recommend flux-schnell or flux-dev.
> 1. non reproducible is there in the API
It should be reproducible if you set the temperature to 1, have you tried that?
Has anybody found a good way to utilize ollama with an editor such as zed to do things like "generate rustdoc to this method" etc. I use ollama daily for a ton of things, but for code generation, completion and documentation 4o is still much better than any of the local models...
Gemma2 27b instruct:
Generate documentation in Rust format for use in code.
The Continue extension for VSCode is pretty good and has native connectivity to a local install of Ollama
Zed has also support for ollama, but all the local models I tried do not really work so well to things like "write docs for this method"... Also local editor autocomplete in the style of github copilot would be great, without needing to use proprietary Microsoft tooling...
There's a lot of plugins/IDEs for assistant style LLMs, but the only TAB style autocompletion ones I know of are either proprietary (Github Copilot), or you need to get an API key (Codestral). If anyone knows of a local autocomplete model I'd love to hear about it.
The Continue extension (Jetbrains, VSCodium) lets you set up assistant and autocompletion independently with different API keys.
https://tabby.tabbyml.com/
https://sourcegraph.com/cody
I use gen.nvim[1] with for small tasks, like “write a type definition for this JSON” .
Running locally avoids the concern of sending IP or PII to third parties.
[1]: https://github.com/David-Kunz/gen.nvim
May as well ask here: what is the best way to use something like an LLM as a personal knowledge base?
I have a few thousand book, papers and articles collected over the last decade. And while I have meticulously categorised them for fast lookup, it's getting harder and harder to search for the desired info, especially in categories which I might not have explored recently.
I do have a 4070 (12 GB VRAM), so I thought that LLMs might be a solutions. But trying to figure out the whats and hows hase proven to be extremely complicated, what with deluge of techniques (fine-tuning, RAG, quantisation) that might not might not be obsolete, too many grifters hawking their own startups with thin wrappers, and a general sense that the "new shiny object" is prioritised more than actual stable solutions to real problems.
Imho opinion, and I'm no expert, but this has been working well for me:
Segment the texts into chunks that make sense (i.e. into the lengths of text you'll want to find, whether this means chapters, sub-chapters, paragraphs, etc), create embeddings of each chunk, and store the resultant vectors in a vector database. Your search workflow will then be to create an embedding of your query, and perform a distance comparison (e.g. cosine similarity) which returns ranked results. This way you can now semantically search your texts.
Everything I've mentioned above is fairly easily doable with existing LLM libraries like langchain or llamaindex. For reference, this is an RAG workflow.
Where are LLMs used in this workflow? For creating embeddings?
Look into this: https://www.anthropic.com/news/contextual-retrieval
And this: https://microsoft.github.io/graphrag/
Look into AnythingLLM.
https://khoj.dev promises this.
I need to have two things of my own that work offline for privacy concerns and cost savings:
1. Local LLM AI models with GUI and command line
2. > Local LLM-based coding tools do exist (such as Google DeepMind’s CodeGemma and one from California-based developers Continue)
Llama 3.1 405B
"2 MacBooks is all you need. Llama 3.1 405B running distributed across 2 MacBooks using @exolabs_ home AI cluster" https://x.com/AIatMeta/status/1834633042339741961
"All you need is £10k of Apple laptops..."
yes but still, a local model, a lightning in a bottle that is between GPT3.5 and GPT4 (closer to 4), yours forever, for about that price is pretty good deal today. probably won't be a good deal in a couple years but for the value, it is not that unsettling. When ChatGPT first launched 2 years ago we all wondered what it would take to have something close to that locally with no strings attached, and turns out it is "a couple years and about $10k" (all due to open weights provided by some companies, training such a model still costs millions) which is neat. It will never be more expensive.
That is... probable, if you bought a newish m2 to replace your 5-6 year old macbook pro which is now just lying around. Or maybe you and your spouse can share cpu hours.
No, you need two of the newest M3 Macbook Pros with maxed RAM, which in practice some people might have, but it is not gettable by using old hardware.
And not having tried it, I’m guessing it will probably run at 1-2 tokens per second or less since the 70b model on one of these runs at 3-4, and now we are distributing the process over the network, which is best case maybe 40-80Gb/s
It is possible, and that’s about the most you can say about it.
Some companies (OpenAI, Anthropic…) base their whole business on hosted closed source models. What’s going to happen when all of this inevitably gets commoditized?
This is why I’m putting my money on Google in the long run. They have the reach to make it useful and the monetization behemoth to make it profitable.
There's plenty of competition in this space already, and it'll only get accelerated with time. There's not enough "moat" in building proprietary LLMs - you can tell by how the leading companies in this space are basically down to fighting over patents and regulatory capture (ie. mounting legal and technical barriers to scraping, procuring hardware, locking down datasets, releasing less information to the public about how the models actually work behind the scenes, lobbying for scary-yet-vague AI regulation, etc).
It's fizzling out.
The current incumbents are sitting on multi-billion dollar valuations and juicy funding rounds. This buys runtime for a good couple of years, but it won't last forever. There's a limit to what can be achieved with scraped datasets and deep Markov chains.
Over time, it will become difficult to judge what makes one general-purpose LLM be any better than another general-purpose LLM. A new release isn't necessarily performing better or producing better quality results, and it may even regress for many use-cases (we're already seeing this with OpenAI's latest releases).
Competitors will have caught up to eachother, and there shouldn't be any major differences between Claude, ChatGPT, Gemini, etc - after-all, they should all produce near-identical answers, given identical scenarios. Pace of innovation flattens out.
Eventually, the technology will become wide-spread, cheap and ubiquitous. Building a (basic, but functional) LLM will be condensed down to a course you take at university (the same way people build basic operating systems and basic compilers in school).
The search for AGI will continue, until the next big hype cycle comes up in 5-10 years, rinse and repeat.
You'll have products geared at lawyers, office workers, creatives, virtual assistants, support departments, etc. We're already there, and it's working great for many use-cases - but it just becomes one more tool in the toolbox, the way Visual Studio, Blender and Photoshop are.
The big money is in the datasets used to build, train and evaluate the LLMs. LLMs today are only as good as the data they were trained on. The competition on good, high-quality, up-to-date and clean data will accelerate. With time, it will become more difficult, expensive (and perhaps illegal) to obtain world-scale data, clean it up, and use it to train and evaluate new models. This is the real goldmine, and the only moat such companies can really have.
This is the best take on the generative AI fad I've yet seen. I wish I could upvote this twice.
I had the same impression. I have been suffering a lot lately about the future for engineers (not having work, etc), even habing anxiety when I read news about AI, but these comments make me feel better and relaxed.
I even considered blocking HN.
Yeah, this is called motivated reasoning.
And then the successful chatgpt wrappers with traction will become valuable than the companies creating propietary LLMs. I bet openai will start buying many AI apps to find profitable niches.
Correct, since the competitive edge is in the domain-specific data (which OpenAI, at-least on-paper, shouldn't have access to).
Two things to remember:
1. OpenAI can analyze which "wrappers" or "apps" are most successful, and make better purchasing decisions that way. This is information which isn't available outside of OpenAI.
2. OpenAI can in theory analyze the actual queries and interactions in an organization, record them, analyze, etc - in an attempt to get a hold of the organization's internal data. Unclear on the legality of this, but could perhaps be enforced through a draconic license.
Their hope is to reach AGI and effective post-scarcity for most things that we currently view as scarce.
I know it sounds crazy but that is what they actually believe and is a regular theme of conversations in SF. They also think it is a flywheel and whoever wins the race in the next few years will be so far ahead in terms of iteration capability/synthetic data that they will be the runaway winner.
I don't have a horse in the race but wouldn't Meta be more likely to commoditize things given that they sort of already are?
Search
Gmail
Docs
Android
Chrome (browser and Chromebooks)
I don't use any Meta properties at all, but at least a dozen alphabet ones. My wife uses Facebook, but that's about it. I can see it being handy for insta filters.
YMMV of course, but I suspect alphabet has much deeper reach, even if the actual overall number of people is similar.
I was referring to the many quality open models they've released to be clear.
Does anyone know of a local "Siri" implementation? Whisper + Llama (or Phi or something else), that can run shortcuts, take notes, read web pages etc.?
PS: for reading web pages I know there's voices integrated in the browser/OS but those are horrible
Edit: I just found this. I'll give it a try today: https://github.com/0ssamaak0/SiriLLama
---
Open WebUI has a voice chat but the voices are not great. I'm sure they'd love a PR that integrates StyleTTS2.
You can give it a Serper API Key and it will search the web to use as context. It connects to ollama running on a linux box with a $300 RTX 3060 with 12GB of VRAM. The 4bit quant of Llama 3.1 8B takes up a bit more than 6GB of VRAM which means it can run embedding models and STT on the card at the same time.
12GB is the minimum I'd recommend for running quantized models. The RTX 4070 Ti Super is 3x the cost but 7 times "faster" on matmuls.
The AMD cards do inference OK but they are a constant source of frustration when trying to do anything else. I bought one and tried for 3 months before selling it. It's not worth the effort.
I don't have any interest in allowing it to run shortcuts. Open WebUI has pipelines for integrating function calling. HomeAssistant has some integrations if that's the kind of thing you are thinking about.
That SiriLLama project looks awesome! I'll give it a try. I also just spun up https://github.com/ItzCrazyKns/Perplexica to try a local Perplexity alternative.
I'll check it out. There is also LlocalSearch: https://news.ycombinator.com/item?id=39923404
Apple intelligence?
It isn't clear if you can know when the task gets handed off to their servers. But yeah that'd be the closest I know. I'm not sure it would build a local knowledge base though.
Another solid option https://www.anaconda.com/products/ai-navigator
It's kinda funny how nowadays an AI with 8 billion parameters is something "small". Specially when just two years back entire racks were needed to run something giving way worst performance.
IDK, 8B-class quantized models run pretty fast on commodity laptops, with CPU-only inference. Thanks to the people who figured out quantization and reimplemented everything in C++, instead of academic-grade Python.
A solid chunk of python is just wrappers around C/C++, most tensor frameworks included.
I know, and yet early model implementations were quite unoptimized compared to the modern ones.
PC: Personal Chatbot
I run a fineruned mulmodal LLM as a spam filter (reads emails as images). Game changer. Removes all the stuff I wouldn't read anyways, not only spam.
Am I the only one seeing obvious ads in llama3 results?
I've not yet used any local AI, so I'm curious - what are you getting? Can you share examples?
Yes.
I'm interested in running locally, but I haven't found consistent advice on hardware specs for optimal performance. I would like to build a server with the best GPU and tons of RAM to run and experiment with these models.
Either get a Mac with loads of mem or build a rig with a 3090 or three
For anyone looking for a simple alternative for running local models beyond just text, Nexa AI has built an SDK that supports text, audio (STT, TTS), image generation (e.g., Stable Diffusion), and multimodal models! It also has a model hub to help you easily find local models optimized for your device.
Nexa AI local model hub: https://nexaai.com/ Toolkit: https://github.com/NexaAI/nexa-sdk
It also comes with a built-in local UI to get started with local models easily and OpenAI-compatible API (with JSON schema for function calling and streaming) for starting local development easily.
You can run the Nexa SDK on any device with a Python environment—and GPU acceleration is supported!
Local LLMs, and especially multimodal local models are the future. It is the only way to make AI accessible (cost-efficient) and safe.
I saw this demo a few months back - and lost it, of LLM autocompletion that was a few milliseconds - it opened a how new way on how to explore it... any ideas?
https://groq.com
is very fast.
(this is not the same as Grok)
The newest laptops are supposed to have 40-50 TOPS performance with the new AI/NPU features. Wondering what that will mean in practice.
The Mistral models are not half as bad for this.
I think it REALLY depends on your use case. Do you want to brainstorm, clear out some thoughts, search or solve complex tasks?
My Thesis: Small language models (SLM)— models so compact that you can run them on a computer with just 4GB of RAM — are the future. SLMs are efficient enough to be deployed on edge devices, while still maintaining enough intelligence to be useful. https://www.lycee.ai/blog/why-small-language-models-are-the-...
What advantages do local models have over exterior models? Why would I run one locally if ChatGPT works well?
1) Offline connectivity — pretty cool to be able to debug technical problems while flying (or otherwise off grid) with a local LLM, and current 8B models are usually good enough for the first line of questions that you otherwise would have googled.
2) Privacy
3) Removing safety filters — there are some great “abliterated” models out there that have had their refusal behavior removed. Running these locally and never having your request refused due to corporate risk aversion is a very different experience to calling a safety-neutered API.
Depending on your use case some, all, or none of these will be relevant, but they are undeniable benefits that are very much within reach using a laptop and the current crop of models.
I am betting on local AI and building offload.fyi to make it easy to implement in any app
there's no small AI that I know of and masters ancient Greek, Latin, English, German and French and that I can run on my 18 GB macbook pro.
Please correct me if I'm wrong. It would make my life slightly more comfortable
I agree. Even bi-lingual (English+1) small models would be very useful to process localized data, for ex english-french, english-german, etc.
Right now the small models (llama 8B) can't handle this type of task, although they could if they were trained for bi lingual data.
I really get the feeling with these models that what we need is a very memory-first hardware architecture that is not necessarily the fastest at crunching.... that seems like it shouldn’t necessarily be a terrifically expensive product
What’s the best local codegen model that gets close to Claude 3.5? Deepseek coder?
Don’t want to pay for Claude API and Cursor sub gets past the limits quick. Gonna get a M4 Mac with maxed out ram when they come out next month
I think this is a big deal. In my opinion, many money making stable AI services are going to be deliberately of limited ability on limited domains. One doesn't want one's site help bot answering political questions. So this could really pull much of the revenue away from AI/LLMs as service.
i think this is laughable, the only good 8B models are the llama ones, phi is terrible, even codestral can barely code and that's 22B iirc
but truthfully the 8B just aren't that great yet, they can provide some decent info if you're just investigating things but a google search is still faster
Lately I’ve been submitting the same requests to GPT 4o and Mistral Large running on my laptop or one of my own servers that has 2xA6000. I get about the same quality from both and probably would not be able to tell which is which in a blind test. There are a couple disadvantages to either of the two local solutions, however. Both of them are slower than 4o. Both of them require a substantial amount of very expensive hardware (which I happed to already have, but still). And on a laptop a single long generation can drain a few percent of the battery. Neural Engine, which would be more energy efficient cannot currently run models this large, and smaller ones do not do as well for the chatbot use case. I do also use smaller LLMs locally, for the more “narrow” tasks.
Summary: It's cheaper, safer for handling sensitive data, easier to reproduce results (only way to be 100% sure it's reproduce even, as "external" models can change anytime), higher degree of customization, no internet connectivity requirements, more efficient, more flexible.
No ridiculous prohibitions on training on logs…
Man, imagine being OpenAI and flushing your brand down the toilet with an explicit customer noncompete rule which totally backfires and inspires 100x more competition than it prevents
Llama's license does forbid it:
"Llama 3.1 materials or outputs cannot be used to improve or train any other large language models outside of the Llama family."
https://llamaimodel.com/commercial-use/
I'm not sure why anybody would respect that licence term, given the whole field rests on the rapacious misappropriation of other people's intellectual property.
Meta dropped that term, actually, and that's an unofficial website.
It's still present in the llama license...?
https://ai.meta.com/llama/license/
Section 1.b.iv
Llama 3.1 isn't under that license, it's under the Llama 3.1 Community License Agreement: https://www.llama.com/llama3_1/license/
>If you use the Llama Materials to create, train, fine tune, or otherwise improve an AI model, which is distributed or made available, you shall also include “Llama 3” at the beginning of any such AI model name.
The official llama 3 repo still says this, which is a different phrasing but effectively equal in meaning to what the commenter above said.
An AI chip on laptop devices would be amazing!
It's pretty much happening already. Apple devices have MPS. Both new Intels and Snapdragon X have some form of NPU.
It would be great if any NPU that currently exists was any good at LLM acceleration, but they all have really bad memory bottlenecks.
They already exist. Nvidia GPUs on laptops, M series CPUs from Apple, NPUs...
oh damn guess i am so uninformed
First NPU arrived 7 years ago in an iPhone SoC. GPUs are also “AI” chips.
Local LLM community has been using Apple Silicon Mac GPUs to do inference.
I’m sure Apple Intelligence uses the NPU and maybe the GPU sometimes.
[dead]
[flagged]
[flagged]
Local LLMs are terrible compared to Claude/ChatGPT. They are useful to use as APIs for applications: much cheaper than paying for OpenAI services, and can be fine tuned to do many useful (and less useful, even illegal) things. But for the casual user, they suck compared to the very large LLMs OpenAI/Anthropic deliver.
I don't think local LLM's are being marketed "for the casual user", nor do I think the casual user will care at all about running LLM's locally so I am not sure why this comparison matters.
Yep, unfortunately those local models are noticeably worse. Also models are getting bigger, so even if a local basement rig for a higher quality model is possible right now, that might not be so in the future. Also Zuck and others might stop releasing their weights for the next gen models, then what, just hope they plateau, what if they don't?
they are the only thing you can use if you don't want to or aren't allowed to hand over your data to US corporations and intelligence agencies.
every single query to ChatGPT/Claude/Gemini/etc will be used for any purpose, by any party, at any time. shamelessly so, because this is the new normal. Welcome to 2024. I own nothing, have no privacy, and life has never been better.
>(and less useful, even illegal) things
the same illegal things you can do with Notepad, or a pencil and a piece of paper.