Anyscale

💥 Another feature update💥 Excited to announce Anyscale Job Queues Job Queues deliver improved utilization and simplified cluster management by running multiple concurrent jobs on a single cluster. • Users submit jobs to a specified queue, Anyscale automatically prioritizes and schedules them. • Jobs are executed based on their queue position, with limits on concurrent jobs per cluster. • Jobs run to completion, including retries, repeating until all jobs in the queue are completed or errored. Read full details here: https://lnkd.in/g3VfuApt And watch how it works below:

🚀 Exciting news! Anyscale and MongoDB are empowering enterprises to build, deploy, and scale AI-enriched applications with the launch of the MongoDB AI Applications Program (MAAP.) We’re thrilled to be part of this ecosystem and to help enterprises take advantage of AI. hashtag #MAAPLaunch

📣 Meta-Llama-3.1-405B is now available on Anyscale! 📣 Get started and deploy yourself here: https://lnkd.in/gfywap9J

🚀 Excited to announce Elastic Distributed Training is now on Anyscale! 🔍 With Elastic Training, you’ll see up to 60% lower cloud costs using spot instances and faster training with uninterrupted progress, even as computational resources come and go during training. Elastic training adjusts to dynamic computational resources during the training process. Training can recover from spot instances preemptions and hardware failures. Instead of waiting potentially hours for a fixed number of GPUs to be available, training can continue on the resources that are already available. ⚒ You can try this out on Anyscale with minimal code changes. This includes a simple one line code change to specify (min_workers, max_workers) as a tuple instead of a fixed worker group size and adding checkpointing. Read more in our announcement: https://lnkd.in/gK64MEiS

Introducing the Anyscale LLM Router template. This notebook helps you build your own LLM Router to optimize LLM applications and save up to 2X on LLM applications. The LLM router dynamically sends queries to the most cost-efficient LLM while maintaining quality and latency. In Gen AI and LLMs, model size, quality, inference speed, and cost vary widely. Some models can be 200X more expensive than others. Using a single model can lead to expensive queries or low-quality responses. LLM Routing ensures you get the best balance of cost and performance. With the template, you can easily start training your own LLM Router and optimize your LLM Applications to reduce costs. Click to learn more: https://lnkd.in/gZK73Z8s Let us know your results 👇

We’ve recently introduced the Anyscale Unified Log Viewer, which simplifies and streamlines the debugging and optimization of distributed Ray applications! Key features include: • Structured logging with actor, instance, and request attributes for effective searching across the cluster • UI and log persistence • Comprehensive search and filtering capabilities • Scalable to 100s of GB of logs and thousands of nodes Check out the video: https://lnkd.in/gwr5hsTf  🛠️ To get started, log in to Anyscale, run your Ray application, and navigate to the new log viewing section in your dashboard. Start debugging and optimizing with ease! 🌟 Read more in our announcement: https://lnkd.in/gpbEvr2h

Python's default logging library has a lot of advantages, but logs are often very unstructured, which adds complexity for downstream parsing, querying, managing, and exporting. We're excited to announce Structured Logging is now available on Anyscale. Logs are now in machine-readable JSON format, simplifying filtering, searching, and analyzing. No more messy text logs and writing regex to debug. Read more in our announcement: https://lnkd.in/gUt9e7sV Try on Anyscale: https://bit.ly/3Y2BnHJ

Excited to partner with PingCAP on the #TiDB Future App Hackathon!

🚀 Follow along this week as we highlight some of the features we shipped over the past quarter! Up first: Anyscale Replica Compaction. Anyscale’s Replica Compaction optimizes resource scheduling in Ray Serve, solving resource fragmentation and enhancing efficiency. Why it matters:  Resource fragmentation happens when scaling leads to uneven utilization across nodes. As traffic decreases, previously needed nodes become underutilized, leading to increased cost and poor efficiency. Ray Serve scales each model independently, leading to this issue. Solution: Replica Compaction automatically migrates replicas into fewer nodes, optimizing resource use and reducing costs with zero downtime. Results: On an Anyscale Service with multiple LLMs (Mistral, Mixtral, Llama3), efficiency increased by 10.7% (measured by Tokens / GPU Second).  We found instance seconds declined by 3.7% despite an 11.2% traffic increase. Learn more in our blog: https://lnkd.in/e5WjVSFH Get started today: https://bit.ly/3Y6qHrx

It was great to see everyone at the Ray in Financial Services Meetup in New York 🏙 A big thank you to all the community members who made it possible. 👏 We discussed Ray and it's use cases, got a hands on look at running parallel experiments with cpus and gpus, and shared more on the upcoming roadmap. Looking forward to our next meetup - training day in New York this July 31st! Register here:https://lnkd.in/ePbX2KYs