Ilya Ganelin

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

16

2 things: - become FAANG engineer - make 3,251xxx+ per year Both goals are worth admiration but not tied to each other. Your income not always tied to your qualifications and you may be prone to survivor bias. We may never know the stories of those other amazing engineers with same traits that did not get there.

14

1 Comment

Hallucinations in LLMs are inevitable, no seriously! Josh Fritzsche and I have worked hard at Voze to find a way to tune, prompt and even plead 🙏 with LLMs to prevent them from the occasional 🍄 hallucination. It seemed like the better we have done the harder the edge cases have been to solve and eventually we realized that they tend to frequently happen when LLMs predict or make anticipations on "nothing". I remember at one point nearly going insane with frustration because when LLMs had a hallucination, they were EPIC! 😱 Then we pushed through it and benefited for the effort! How did we solve this common problem? We didn't (Not completely), we got better at identifying the issues as they happened. Once we identified in the data we were able to start looking for patterns. The same catalysts that cause it to happen once if left unchanged caused it over and over again. Our solution is astoundingly simple. Once you know that it's likely to happen..... let it! Accept it will 🍄, but also be preemptive in not focusing on it and use alternative data sources, generators or providers. Change the conditions that cause it but continue to monitor it as it happens. As Dan Caffee is fond of saying to me on a regular bases. "Dusty we need the system to be adaptive and provide mechanisms to learn from it's past." Well he is right! All we needed was data and the people that contribute to it's quality. Huge props 👏 to strong support from Kathy and Janelles audit teams. We were armed with a knowledge and a desire to learn from past mistakes. Armed with confidence that we could correct these problems we could focus on the data to prevent their impact going forward. When doing millions of LLM (Generative AI) calls regularly Josh and I have learned that hallucinations will happen. We also know exactly how we will handle and prevent it going forward! Failure is not doing anything and accepting defeat.

4

2 Comments

A couple of recent college graduates from India against AI finetuning giants like Anyscale and Run:ai? I met Arko C at a NASSCOM event last October. He was building Xylem AI (YC S24) , a platform that allows you to train and deploy LLMs in production. So naturally most Indian VCs rejected him. Because how could a team of three recent college grads compete with the likes SV startups like Anyscale that have raised hundreds of millions of dollars, have teams of experienced AI researchers and a stash of H100 GPUs? Arko hustled and closed two Fortune 500 companies for paid contracts. I personally know unicorns that haven't managed to do this. He then got into YC in the last round in May. There's a lot to still be proved out, but Arko exemplifies how valuable grit is in building startups. I’m doing a webinar with him tomorrow on challenges of using LLMs in production. Link to register in the comments. Disclaimer: I'm a small investor in Xylem.

153

5 Comments

This is a great first step in understanding how scaling dynamics extend beyond language to other types of sequential data. Time is a universal axis, understanding how models learn optimally over data collected over time is a crucial step in the broader AI research agenda.

26

LLM powered AI agents have been a hot topic this year. Inspired by John Yang's SWE-agent paper which framed agents as their own class of end users and showed a path to improving agent performance through agent-computer interfaces (ACI), I wanted to explore the parallels between AI agents and consumers. In my latest article, I lay out my vision of Web 4.0--a web characterized by AI operability and agentic markets, and how ACIs will enable these. Through examining some recent LLM research papers, I outline: - Why some AI agents are consumers - How AI agents being consumers is currently a problem - Why allowing AI agents to spend will lead to new economies and define Web 4.0 - Why agent-computer interfaces (ACI) is needed for Web 4.0 to come to life Check out the piece below and let me know your thoughts! Huge thanks to my friends Kevin Zhang, Hanyi . , Robin Huang, 🐙 Marc Mengler, Sheng Ho, Andrew Tan, Phillip An , Nadja Carina Reischel, and Luc de Leyritz for reviewing this piece and valuable suggestions! #AI #Agents #LLM #ACI #Web4

41

2 Comments

Tech doesn’t rest! With multiple cloud vendors came the concept of “supercloud” which is just a fancy term for a conceptual architecture that provides a unified and flexible abstraction layer on top of the various cloud environments to enable companies leverage the best from each of the many cloud vendors. With LLMs poping up everyday; there is an equally strong imperative to use the best of each LLMs, and so enters the concept of “Mixture -of -Agents", (MoA) as an architecture My take is that tech doesn’t really like monopoly; tech likes options and engineers, through open source projects will ensure no one gets locked into a single AI ecosystem - we want the the option of picking from a vendor only that which the vendor is great at. #SuperCloud #MixtureOfAgents #CloudComputing #AI #LLM

12

2 Comments

🚀 Good folks at NVIDIA just dropped: "ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities" 🧠💡 In the past few months, the open LLM community has made significant progress in releasing open models (Llama-3-70B-Instruct (Meta -AI) 🦙, QWen2-72BInstruct (Alibaba Group ) 🌐, Nemotron-4-340B-Instruct (NVIDIA ) ⚙️, and Mixtral-8x22BInstruct-v0.1 (Mistral AI ) 🌪️) that are at par with proprietary models! 📈 But top models like GPT-4 are still outperforming them in certain domains! 🔝💪 This led us to having domain-focused open-LLMs (DeepSeek-Coder-V2 for coding and math 👨‍💻➕, ChatQA 1.5 for conversational QA and retrieval-augmented generation (RAG) 💬🔍, and InternVL 1.5 for vision-language tasks 🖼️🗣️) The challenge that ChatQA 2 focuses on is of context length and RAG! 📏🔗 These are the two capabilities essential for LLMs to process large volumes of information that cannot fit into a single prompt and are complementary to each other, depending on the downstream tasks and computational budgets. 🧩📊 The solution is a detailed continued training recipe to extend the context window of Llama3-70B-base from 8K to 128K tokens, along with a three-stage instruction tuning process to enhance the model's instruction-following, RAG performance, and long-context understanding capabilities. 🔄🔧 📄 Paper: https://lnkd.in/gnUqMyXE The interesting thing to notice from benchmarks was, how good QWen 2 is out of the box! 👏✨

15

Just read Farquhar et al's Nature paper on detecting hallucinations in large language models using semantic entropy. Interesting result on quantifying uncertainty in LLMs. When you ask an LLM a question, its answers may look correct, but be made up - a confabulation. For example, the LLM may answer incorrectly that the STARD10 protein is a "negative regulator of the mTOR pathway" instead of its true role as a lipid transfer protein. When an answer is confabulated, multiple subsequent generations will have different meanings. For example, a second answer of the LLM may incorrectly say that STARD10 is a "negative regulator of the meiotic recombination process". The variability in meaning can be quantified using semantic entropy, which measures variability over multiple generated answers. Higher entropy suggests greater variability and potential confabulation. The LLM can then refuse questions with high semantic entropy. To calculate semantic entropy, you generate 5-10 answers. You then cluster the answers based on semantic meaning using bi-directional entailment. Two answers are part of the same cluster if they entail - or imply - each other (e.g. "Joe buys a red apple" implies "A red apple was bought by Joe" and vice-versa). The entailment is calculated using another, possibly less powerful LLM, so it's an interesting applications of using language models to correct other language models. After clustering, the probabilities of each cluster are calculated from the output probabilities of the LLM, and then the resulting entropy. Thresholding the entropy can classify a question as likely to produce confabulations with an AUROC of around 0.75. Link to paper below.

63

2 Comments

Over the past period I had an immense privilege of working with Andrej Karpathy (ex director of AI at Tesla, founding member of OpenAI, and our AI sensei :)), and a few cracked GPU engineers (Arun Demeure ex GPU Platform Architect at Apple, ngc92 et al) on the llm.c project. And I must admit I was pleasantly surprised yesterday getting a shout-out from Andrej and seeing my Twitter blow up with notifications 😅 (see the attached screen shot) As someone I've been looking up to since i started my ML journey up to a point where i'm casually chatting with him on a daily basis it all feels very unreal. Arun & ngc92 are extremely knowledgeable, and i learned a ton from them building llm.c. They're our CUDA whisperers. :) But, before continuing, what is llm.c? from Andrej: "...Because llm.c is a direct implementation of GPT training in C/CUDA, the requirements are minimal - there is no need for conda environments, Python interpreters, pip installs, etc. You spin up a cloud GPU node (e.g. on Lambda), optionally install NVIDIA cuDNN, NCCL/MPI, download the .bin data shards, compile and run, and you're stepping in minutes. You then wait 24 hours and enjoy samples about English-speaking Unicorns in the Andes." original tweet here: https://lnkd.in/dU93zyZU I've quickly contributed 113 commits (+ 40 pending once i merge my muP PR), among those: * I added multi-node support with NCCL initialization going via MPI/TCP/FS so that we can support both slurm and non-slurm scheduling envs * Helped make our code bitwise deterministic - so that after the training crashes we have guarantees that starting from the last checkpoing will give **precisely** the same loss & gradient numbers! * Implemented our random shuffling logic in the data loader, added various lr schedulers, and finally caught many bugs (e.g. periodic loss spikes, distributed norm computation bug, memory leaks), etc. I have learned a ton about CUDA, contributed to our layernorm kernel, and finally I now feel comfortable with basically every single part of LLM training since llm.c is such a "bare metal" project. I really feel i can build anything at this point. But I also learned you can't do it without a cracked team. :)) And, of course, it's more fun, and you make friends along the way. Over the past 2 weeks I mostly worked on adding maximum update parametrization support to llm.c: https://lnkd.in/dB2MuCH3 ran a ton of experiments & ablations and we're getting close to merging it in. The issue is that mup's benefits will become more apparent at bigger scales, while currently we're mostly in the ~1B regime. Today we celebrate llm.c milestone: 5 years after OpenAI announced GPT-2, with all the surrounding safety talks and staged release -> today you can now train your own GPT-2 in ~650$ on a single 8xH100 nodes in 24 hours. See this post for more info: https://lnkd.in/dD6Du4se Extremely excited for the future!

922

35 Comments

Many researchers are building agents to improve the capabilities of LLMs at long-range tasks, like writing a pull request for an issue in a code repository. Typical agents use the LLM in a loop, possibly with human feedback, where previous answers are an input to the LLM and are used to inform next actions. Other promising agents use tree searches, where multiple trains of thought are sampled in parallel. MOLLEO is a recent algorithm that caught my eye, as the authors wrapped an LLM in an evolutionary algorithm to generate SMILES strings. An evolutionary algorithm goes like this. Start with a group of candidates and calculate their fitness with an oracle. Then, sample two parents proportional to the fitness. Combine the two parents with a Crossover operator to create an offspring and then edit the offspring with a Mutation operator. Remove the lowest fitness members of the population and repeat the process. MOLLEO uses an LLM to do the Crossover and Mutation. Interestingly, evolutionary algorithms with LLMs benchmark better than without the LLM, and better than alternatives such as reinforcement learning. Paper in the comments 👇

15

3 Comments

Unit Tests Part 2: Cartesian Parameterization! TIL about multiplying parameterizations in pytest. Now, we all know and love pytests parameterization, where you can pass in parameters to a test and thereby re-use the same test code for multiple situations. For example you can pass in these parameters: @pytest.mark.parametrize("input, output", [('hello', 'world'), ('goodbye', 'mom')]) Then in your pytest you will get these two parameter scenarios to use as variables and execute your tests against. - input: hello, output: world - input: goodbye, output: mom It's so sweet I can taste it just writing it! Now - I had no idea but I found out (thank you ChatGPT) today that you can multiple those parameters to get a cartesian product of the sets which will give you every possible permutations of your test parameters. So if you do this: @pytest.mark.parametrize("fruit", ['apple', 'orange']) @pytest.mark.parametrize("ripeness", ['underripe', 'ripe']) You will get these test cases: - fruit: apple, ripeness: underripe - fruit: apple, ripeness: ripe - fruit: orange, ripeness: underripe - fruit: orange, ripeness: ripe This might seem trivial at first - but this is so powerful for test code re-use and flexibility. Below you can see how in the actual test code I am using for one of my projects: At first we had one file to render with 4 different parameter sets. Easy! Then we added another file but we want to test the same 4 parameters against each file rendering. Super easy with pytest, check it out and I hope you try it yourself. That's all for now, go forth and code with delight!

1

There is so much research going into bringing the power of LLMs into the Enterprise context. The main techniques are k-shot learning or prompt engineering, RAG (Retrieval Augmented Generation) & Model Fine tuning (not to mention many variations of these e.g. Agentic RAG, GraphRAG etc.) However, I came across an interesting take on this from IBM and they are calling it "InstructLab" - where the claim is that they are able to teach the LLM "new skills" grounded in an Enterprise context. On the surface it looks like a k-shot learning + fine turning on steroids. Approach seems to be to utilize a small "skills training dataset" in order to generate a large synthetic dataset (perhaps utilizing a specialized LLM) that is then incorporated into the LLM weights natively. This could be a very interesting concept as long as the cost of adding new skills is not very high (when compared to fine tuning). Exciting times! #genAI #enterpriseAI

40

2 Comments

Without some method of language integration, scaling self-driving is more difficult and requires larger amounts of driving data. In contrast, a human who has learned driving in San Francisco doesn’t really need new driving experience in New York City, just to learn the local rule that you shouldn’t make a right turn at red lights there. LLMs appear to be excellent means for this and they will have a significant role in the future of robotics and self-driving. We are sharing a technical blog post (https://lnkd.in/enfjXkVy) on LAMBDA, our Multimodal LLM integrated with our autonomy stack, that runs on our in-vehicle hardware platform. We haven’t given direct control of the car to LAMBDA yet, but it already demonstrates advanced understanding and reasoning in various road situations.

31

In an innovative exploration at Google Research, Alexey Kurakin and Natalia Ponomareva delve into the creation and utility of differentially private synthetic data. This innovative approach leverages differentially private stochastic gradient descent (DP-SGD) to fine-tune large language models (LLMs), ensuring robust privacy protections. The synthetic data generated maintains the key characteristics of original datasets without compromising user privacy, proving invaluable for training predictive models, feature engineering, and model evaluation. Key Insights - Differential Privacy: A mathematical framework ensuring user data privacy during model training. - Synthetic Data Generation: Using DP-SGD and LLMs to create high-quality, privacy-preserving synthetic datasets. - Parameter-Efficient Fine-Tuning: Techniques like LoRa and prompt fine-tuning optimize the balance between model quality and privacy. - Empirical Results: Synthetic data often enhances model performance compared to directly using sensitive data. This approach is particularly beneficial for applications like on-device safety classification, where privacy and performance are paramount. For more details, read the full blog here. Read the full write-up for a detailed exploration: https://lnkd.in/gjBjh35m Paper is accessible at: https://lnkd.in/gKyggk4d P.S. The illustration attached has been obtained from Google Research's Blog. #artificialintelligence #MachineLearning #DifferentialPrivacy #SyntheticData #GoogleResearch #DataPrivacy

10

Excited to announce another major upgrade to WhyHow.AI’s Knowledge Graph SDK - tying vector chunks to graph nodes automatically, for a more deterministic and richer context window. Check out how we do it, why we did it, and an example benchmark of the increased completeness of the answer. Tired of just returning single-word triples from your knowledge graph? WhyHow.AI’s latest upgrade with vector chunk linking now lets you use a graph structure to determine which raw vector chunks to return to the context window, combining the best of knowledge graphs and vector search. “While the triples in a Knowledge Graph are useful in providing specific information that semantic similarity was unable to retrieve, we wanted to also allow leeway in the information represented and retrieved from the graph, to include the surrounding words and retrieving the relevant raw vector chunk tied to that graph node as well. By tying vector chunks to a knowledge graph, we get the advantages that lie in both vector and graph search.” - WhyHow.AI Design Partner WhyHow.AI builds workflow tools for data orchestration, and graph creation, and we work on top of any data extraction model you want to bring. In this case, we work on top of OpenAI, Neo4j, and Pinecone, and will be supporting the most popular data extraction models, LLMs, graph and vector databases. https://lnkd.in/eEJdUNPi

68

3 Comments

Linear attention is an hot research topic with important industrial applications. The problem is inherently quadratic, but there are certain linear solutions under given hyphotesis. This is a a good repo for SOTA, including evaluation benchmarks. If you are a Researcher don't miss the opportunity.

21

In-Context Learning at Extreme Scales: My Thoughts As large language models (LLMs) continue to expand their context capacities, the domain of in-context learning (ICL) at extreme scales has piqued my interest. A recent paper by researchers at Carnegie Mellon University and Tel Aviv University provides a detailed exploration into this area, unveiling both impressive capabilities and notable challenges. 🔍 Key Insights: Performance Improvement: The study demonstrates that ICL can improve significantly by integrating hundreds to thousands of examples into the context window—far beyond traditional few-shot settings. This is particularly impactful in complex tasks like 77-way banking intent classification, where long-context ICL matches or surpasses traditional finetuning. Input Sensitivity: Long-context ICL shows reduced sensitivity to input shuffling compared to shorter contexts. However, organizing examples by label dramatically reduces performance, emphasizing the need for diverse context configurations. Source of Gains: Performance gains primarily arise from accessing more relevant examples, rather than aggregating knowledge across many examples during encoding. 📉 Challenges & Drawbacks: Sensitivity to Example Order: Despite reduced sensitivity, the performance still varies with input order, indicating potential instability in practical scenarios. Performance Saturation: The models often do not fully leverage their extended context capacities, showing performance saturation before reaching the maximum context length. Computational Cost: The increased computational cost associated with using large context windows could limit practical deployment, particularly in resource-constrained environments. Efficiency Issues: There is a noted inefficiency in how models utilize the encoded information within large datasets, suggesting that current architectures may need refinement to better use available data. 🚀 Future Directions: The paper suggests several exciting avenues for future research, including optimizing the trade-offs between computational costs and performance, exploring model adaptation strategies, and refining ICL approaches for specific applications or datasets. As we push the boundaries of what LLMs can achieve, effectively leveraging their full context could unlock new capabilities. This area is ripe for further exploration and could lead to major advancements in how we deploy AI. For those at the cutting edge of language model development, I highly recommend this paper for its thorough analysis and forward-looking perspective. Check out the full work at https://lnkd.in/d7AmGh4y. What are your thoughts on the future of in-context learning at such scales and the challenges discussed? #AI #MachineLearning #DeepLearning #LanguageModels #InContextLearning

17

Excited to be back in the Bay Area this week to represent Caper and Instacart at the Embedded Vision Summit! Ever curious how to deploy and optimize AI models running on edge devices? I’ll break down some tips and techniques in my talk, "Real-Time Retail Product Classification on Android Devices inside the Caper AI Cart” If you're into AI, machine learning, or just want to learn about the Caper Cart or how we're improving our user's experience by leveraging AI, please feel free to come say hi!

61

2 Comments