AI Evaluation Tools Archives

21Oct

Google on Google AI

As a leading cloud provider, Google Cloud is also a major player in the generative AI market. Google on Google AI provides insights into this new tool. In the past two years, Google has been in a competitive battle with AWS, Microsoft, and OpenAI to gain dominance in the generative AI space. Recently, Google introduced several generative Artificial Intelligence products, including its flagship large language model, Gemini, and the Vertex AI Model Garden. Last week, it also unveiled Audio Overview, a tool that transforms documents into audio discussions. Despite these advancements, Google has faced criticism for lagging in some areas, such as issues with its initial image generation tool, like X’s Grok. However, the company remains committed to driving progress in generative AI. Google’s strategy focuses not only on delivering its proprietary models but also offering a broad selection of third-party models through its Model Garden. Google’s Thoughts on Google AI Warren Barkley, head of product for Google Cloud’s Vertex AI, GenAI, and machine learning, emphasized this approach in a recent episode of the Targeting AI podcast. He noted that a key part of Google’s ongoing effort is ensuring users can easily transition to more advanced models. “A lot of what we did in the early days, and we continue to do now, is make it easy for people to move to the next generation,” Barkley said. “The models we built 18 months ago are a shadow of what we have today. So, providing pathways for people to upgrade and stay on the cutting edge is critical.” Google is also focused on helping users select the right AI models for specific applications. With over 100 closed and open models available in the Model Garden, evaluating them can be challenging for customers. To address this, Google introduced evaluation tools that allow users to test prompts and compare model responses. In addition, Google is exploring advancements in Artificial Intelligence reasoning, which it views as crucial to driving the future of generative AI. Like Related Posts Salesforce OEM AppExchange Expanding its reach beyond CRM, Salesforce.com has launched a new service called AppExchange OEM Edition, aimed at non-CRM service providers. Read more The Salesforce Story In Marc Benioff’s own words How did salesforce.com grow from a start up in a rented apartment into the world’s Read more Salesforce Jigsaw Salesforce.com, a prominent figure in cloud computing, has finalized a deal to acquire Jigsaw, a wiki-style business contact database, for Read more Health Cloud Brings Healthcare Transformation Following swiftly after last week’s successful launch of Financial Services Cloud, Salesforce has announced the second installment in its series Read more

October 21, 2024in Generative AI, Google

19Oct

LLMs and AI

Large Language Models (LLMs): Revolutionizing AI and Custom Solutions Large Language Models (LLMs) are transforming artificial intelligence by enabling machines to generate and comprehend human-like text, making them indispensable across numerous industries. The global LLM market is experiencing explosive growth, projected to rise from $1.59 billion in 2023 to $259.8 billion by 2030. This surge is driven by the increasing demand for automated content creation, advances in AI technology, and the need for improved human-machine communication. Several factors are propelling this growth, including advancements in AI and Natural Language Processing (NLP), large datasets, and the rising importance of seamless human-machine interaction. Additionally, private LLMs are gaining traction as businesses seek more control over their data and customization. These private models provide tailored solutions, reduce dependency on third-party providers, and enhance data privacy. This guide will walk you through building your own private LLM, offering valuable insights for both newcomers and seasoned professionals. What are Large Language Models? Large Language Models (LLMs) are advanced AI systems that generate human-like text by processing vast amounts of data using sophisticated neural networks, such as transformers. These models excel in tasks such as content creation, language translation, question answering, and conversation, making them valuable across industries, from customer service to data analysis. LLMs are generally classified into three types: LLMs learn language rules by analyzing vast text datasets, similar to how reading numerous books helps someone understand a language. Once trained, these models can generate content, answer questions, and engage in meaningful conversations. For example, an LLM can write a story about a space mission based on knowledge gained from reading space adventure stories, or it can explain photosynthesis using information drawn from biology texts. Building a Private LLM Data Curation for LLMs Recent LLMs, such as Llama 3 and GPT-4, are trained on massive datasets—Llama 3 on 15 trillion tokens and GPT-4 on 6.5 trillion tokens. These datasets are drawn from diverse sources, including social media (140 trillion tokens), academic texts, and private data, with sizes ranging from hundreds of terabytes to multiple petabytes. This breadth of training enables LLMs to develop a deep understanding of language, covering diverse patterns, vocabularies, and contexts. Common data sources for LLMs include: Data Preprocessing After data collection, the data must be cleaned and structured. Key steps include: LLM Training Loop Key training stages include: Evaluating Your LLM After training, it is crucial to assess the LLM’s performance using industry-standard benchmarks: When fine-tuning LLMs for specific applications, tailor your evaluation metrics to the task. For instance, in healthcare, matching disease descriptions with appropriate codes may be a top priority. Conclusion Building a private LLM provides unmatched customization, enhanced data privacy, and optimized performance. From data curation to model evaluation, this guide has outlined the essential steps to create an LLM tailored to your specific needs. Whether you’re just starting or seeking to refine your skills, building a private LLM can empower your organization with state-of-the-art AI capabilities. For expert guidance or to kickstart your LLM journey, feel free to contact us for a free consultation. Like Related Posts Salesforce OEM AppExchange Expanding its reach beyond CRM, Salesforce.com has launched a new service called AppExchange OEM Edition, aimed at non-CRM service providers. Read more The Salesforce Story In Marc Benioff’s own words How did salesforce.com grow from a start up in a rented apartment into the world’s Read more Salesforce Jigsaw Salesforce.com, a prominent figure in cloud computing, has finalized a deal to acquire Jigsaw, a wiki-style business contact database, for Read more Health Cloud Brings Healthcare Transformation Following swiftly after last week’s successful launch of Financial Services Cloud, Salesforce has announced the second installment in its series Read more

October 19, 2024in Artificial Intelligence, Data, Google, Salesforce, Technology

12Oct

Salesforce AI Introduces SFR-Judge

Salesforce AI Introduces SFR-Judge: A Family of Three Evaluation Models with 8B, 12B, and 70B Parameters, Powered by Meta Llama 3 and Mistral NeMO The rapid development of large language models (LLMs) has transformed natural language processing, making the need for accurate evaluation of these models more critical than ever. Traditional human evaluations, while effective, are time-consuming and impractical for the fast-paced evolution of AI models. Salesforce AI Introduces SFR-Judge. To address this, Salesforce AI Research has introduced SFR-Judge, a family of LLM-based judge models designed to revolutionize how AI outputs are evaluated. Built using Meta Llama 3 and Mistral NeMO, the SFR-Judge family includes models with 8 billion (8B), 12 billion (12B), and 70 billion (70B) parameters. These models are designed to handle evaluation tasks such as pairwise comparisons, single ratings, and binary classifications, streamlining the evaluation process for AI researchers. Overcoming Limitations in Traditional Judge Models Traditional LLMs used for evaluation often suffer from biases such as position bias (favoring responses based on their order) and length bias (preferring longer responses regardless of their accuracy). SFR-Judge addresses these issues by leveraging Direct Preference Optimization (DPO), a training method that enables the model to learn from both positive and negative examples, reducing bias and ensuring more consistent and accurate evaluations. Performance and Benchmarking SFR-Judge has been rigorously tested across 13 benchmarks covering three key evaluation tasks. It outperformed existing judge models, including proprietary models like GPT-4o, achieving top performance on 10 of the 13 benchmarks. Notably, on the RewardBench leaderboard, SFR-Judge achieved a 92.7% accuracy, marking a new high in LLM-based evaluation and demonstrating its potential not only as an evaluation tool but also as a reward model for reinforcement learning from human feedback (RLHF) scenarios. Innovative Training Approach The SFR-Judge models were trained using three distinct data formats: These diverse data formats allow SFR-Judge to generate well-rounded, accurate evaluations, making it a more reliable and robust tool for model assessment. Bias Mitigation and Robustness SFR-Judge was tested on EvalBiasBench, a benchmark designed to measure six types of bias. The results demonstrated significantly lower bias levels compared to competing models, along with high consistency in pairwise order comparisons. This robustness ensures that SFR-Judge’s evaluations remain stable, even when the order of responses is altered, making it a scalable and reliable alternative to human annotation. Key Takeaways: Conclusion Salesforce AI Research’s introduction of SFR-Judge represents a breakthrough in the automated evaluation of large language models. By incorporating Direct Preference Optimization and a diverse training approach, SFR-Judge sets a new standard for accuracy, bias reduction, and consistency. Its ability to provide detailed feedback and adapt to various evaluation tasks makes it a powerful tool for the AI community, streamlining the process of LLM assessment and setting the stage for future advancements in AI evaluation. Like Related Posts Salesforce OEM AppExchange Expanding its reach beyond CRM, Salesforce.com has launched a new service called AppExchange OEM Edition, aimed at non-CRM service providers. Read more The Salesforce Story In Marc Benioff’s own words How did salesforce.com grow from a start up in a rented apartment into the world’s Read more Salesforce Jigsaw Salesforce.com, a prominent figure in cloud computing, has finalized a deal to acquire Jigsaw, a wiki-style business contact database, for Read more Health Cloud Brings Healthcare Transformation Following swiftly after last week’s successful launch of Financial Services Cloud, Salesforce has announced the second installment in its series Read more

October 12, 2024in Data, Salesforce

AI evolves with tools like Agentforce and Atlas

02Oct

AI Evolves With Agentforce and Atlas

Not long ago, financial services companies were still struggling with the challenge of customer data trapped in silos. Though it feels like a distant issue, this problem remains for many large organizations unable to integrate different divisions that deal separately with the same customers. Salesforce AI evolves with tools like Agentforce and Atlas. The solution is a concept known as a “single source of truth.” This theme took center stage at Dreamforce 2024 in San Francisco, hosted by Salesforce (NYSE). The event showcased Salesforce’s latest AI innovations, including Agentforce, which is set to revolutionize customer engagement through its advanced AI capabilities. Agentforce, which becomes generally available on October 25, enables businesses to deploy autonomous AI agents to manage a wide variety of tasks. These agents differ from earlier Salesforce-based AI tools by leveraging Atlas, a cutting-edge reasoning engine that allows the bots to think like human beings. Unlike generative AI models, which might write an email based on prompts, Agentforce’s AI agents can answer complex, high-order questions such as, “What should I do with all my customers?” The agents break down these queries into actionable steps—whether that’s sending emails, making phone calls, or texting customers—thanks to the deep capabilities of Atlas. Atlas is at the heart of what makes these AI agents so powerful. It combines multiple large language models (LLMs), large action models (LAMs), and retrieval-augmented generation (RAG) modules, along with REST APIs and connectors to various datasets. This robust system processes user queries through multiple layers, checking for validity and then expanding the query into manageable chunks for processing. Once a query passes through the chit-chat detector—which filters out non-relevant inputs—it enters the evaluation phase, where the AI determines if it has enough data to provide a meaningful answer. If not, the system loops back to the user for more information in a process Salesforce calls the agentic loop. The fewer loops required, the more efficient the AI becomes, making the experience seamless for users. Phil Mui, Senior Vice President of Salesforce AI Research, explained that the AI agents created via Agentforce are powered by the Atlas reasoning engine, which makes use of several key tools like a re-ranker, a refiner, and a response synthesizer. These tools ensure that the AI retrieves, ranks, and synthesizes relevant information to generate high-quality, natural language responses for the user. But Salesforce’s AI agents don’t stop at automation—they also emphasize trust. Before responses reach users, they go through additional checks for toxicity detection, bias prevention, and personally identifiable information (PII) masking. This ensures that the output is both accurate and safe. The potential of Agentforce is massive. According to Wedbush, Salesforce’s AI strategy could generate over $4 billion annually by 2025. Wedbush analysts recently increased their price target for Salesforce stock to $325, reflecting the strong customer reception of Agentforce’s AI ecosystem. While some analysts, such as Yiannis Zourmpanos from Seeking Alpha, have expressed caution due to Salesforce’s high valuation and slower revenue growth, the company’s continued focus on AI and multi-cloud solutions places it in a strong position for the future. Robin Fisher, Salesforce’s head of growth markets for Europe, the Middle East, and Africa, highlighted two major takeaways from Dreamforce for African businesses: the Data Cloud and AI. Data Cloud provides a 360-degree view of the customer, consolidating data into a single source of truth without requiring full data migration. Meanwhile, Agentforce’s autonomous AI agents will drive operational efficiency across industries, especially in markets like Africa. Zuko Mdwaba, Salesforce’s managing director for South Africa, added that the company’s decade-long AI journey is culminating in its most advanced AI offerings yet. This new wave of AI, he said, is transforming not just customer engagement but also internal operations, empowering employees to focus on more strategic tasks while AI handles repetitive ones. The future is clear: as AI evolves with tools like Agentforce and Atlas, businesses across sectors, from banking to retail, are poised to harness the transformative power of autonomous technology and data-driven insights, finally breaking free from the silos of the past. Like1 Related Posts Salesforce OEM AppExchange Expanding its reach beyond CRM, Salesforce.com has launched a new service called AppExchange OEM Edition, aimed at non-CRM service providers. Read more The Salesforce Story In Marc Benioff’s own words How did salesforce.com grow from a start up in a rented apartment into the world’s Read more Salesforce Jigsaw Salesforce.com, a prominent figure in cloud computing, has finalized a deal to acquire Jigsaw, a wiki-style business contact database, for Read more Health Cloud Brings Healthcare Transformation Following swiftly after last week’s successful launch of Financial Services Cloud, Salesforce has announced the second installment in its series Read more

October 2, 2024in Agentforce Platform, AI Tools, Data, Generative AI, Salesforce, Salesforce Implementation Services, Technology

AI Evaluation Tools

Salesforce AI Introduces SFR-Judge

Recent Posts

Native Salesforce App to Simplify Event Management

Generative AI in CX

From Generative AI to Agentic AI

Python-Based Reasoning

Salesforce Foundations

Contact Us

Be in touch today — and start your business on a path to success.

Category

Archives