Context Window Archives - gettectonic.com
26Dec

Why Build a General-Purpose Agent?

A general-purpose LLM agent serves as an excellent starting point for prototyping use cases and establishing the foundation for a custom agentic architecture tailored to your needs. What is an LLM Agent? An LLM (Large Language Model) agent is a program where execution logic is governed by the underlying model. Unlike approaches such as few-shot prompting or fixed workflows, LLM agents adapt dynamically. They can determine which tools to use (e.g., web search or code execution), how to use them, and iterate based on results. This adaptability enables handling diverse tasks with minimal configuration. Agentic Architectures Explained:Agentic systems range from the reliability of fixed workflows to the flexibility of autonomous agents. For instance: Your architecture choice will depend on the desired balance between reliability and flexibility for your use case. Building a General-Purpose LLM Agent Step 1: Select the Right LLM Choosing the right model is critical for performance. Evaluate based on: Model Recommendations (as of now): For simpler use cases, smaller models running locally can also be effective, but with limited functionality. Step 2: Define the Agent’s Control Logic The system prompt differentiates an LLM agent from a standalone model. This prompt contains rules, instructions, and structures that guide the agent’s behavior. Common Agentic Patterns: Starting with ReAct or Plan-then-Execute patterns is recommended for general-purpose agents. Step 3: Define the Agent’s Core Instructions To optimize the agent’s behavior, clearly define its features and constraints in the system prompt: Example Instructions: Step 4: Define and Optimize Core Tools Tools expand an agent’s capabilities. Common tools include: For each tool, define: Example: Implementing an Arxiv API tool for scientific queries. Step 5: Memory Handling Strategy Since LLMs have limited memory (context window), a strategy is necessary to manage past interactions. Common approaches include: For personalization, long-term memory can store user preferences or critical information. Step 6: Parse the Agent’s Output To make raw LLM outputs actionable, implement a parser to convert outputs into a structured format like JSON. Structured outputs simplify execution and ensure consistency. Step 7: Orchestrate the Agent’s Workflow Define orchestration logic to handle the agent’s next steps after receiving an output: Example Orchestration Code: pythonCopy codedef orchestrator(llm_agent, llm_output, tools, user_query): while True: action = llm_output.get(“action”) if action == “tool_call”: tool_name = llm_output.get(“tool_name”) tool_params = llm_output.get(“tool_params”, {}) if tool_name in tools: try: tool_result = tools[tool_name](**tool_params) llm_output = llm_agent({“tool_output”: tool_result}) except Exception as e: return f”Error executing tool ‘{tool_name}’: {str(e)}” else: return f”Error: Tool ‘{tool_name}’ not found.” elif action == “return_answer”: return llm_output.get(“answer”, “No answer provided.”) else: return “Error: Unrecognized action type from LLM output.” This orchestration ensures seamless interaction between tools, memory, and user queries. When to Consider Multi-Agent Systems A single-agent setup works well for prototyping but may hit limits with complex workflows or extensive toolsets. Multi-agent architectures can: Starting with a single agent helps refine workflows, identify bottlenecks, and scale effectively. By following these steps, you’ll have a versatile system capable of handling diverse use cases, from competitive analysis to automating workflows. Like Related Posts Salesforce OEM AppExchange Expanding its reach beyond CRM, Salesforce.com has launched a new service called AppExchange OEM Edition, aimed at non-CRM service providers. Read more Salesforce Jigsaw Salesforce.com, a prominent figure in cloud computing, has finalized a deal to acquire Jigsaw, a wiki-style business contact database, for Read more Health Cloud Brings Healthcare Transformation Following swiftly after last week’s successful launch of Financial Services Cloud, Salesforce has announced the second installment in its series Read more Top Ten Reasons Why Tectonic Loves the Cloud The Cloud is Good for Everyone – Why Tectonic loves the cloud You don’t need to worry about tracking licenses. Read more

Read More
December 26, 2024in Data0 Comments
Where LLMs Fall Short
15Nov

LLM Economies

Throughout history, disruptive technologies have been the catalyst for major social and economic revolutions. The invention of the plow and irrigation systems 12,000 years ago sparked the Agricultural Revolution, while Johannes Gutenberg’s 15th-century printing press fueled the Protestant Reformation and helped propel Europe out of the Middle Ages into the Renaissance. In the 18th century, James Watt’s steam engine ushered in the Industrial Revolution. More recently, the internet has revolutionized communication, commerce, and information access, shrinking the world into a global village. Similarly, smartphones have transformed how people interact with their surroundings. Now, we stand at the dawn of the AI revolution. Large Language Models (LLMs) represent a monumental leap forward, with significant economic implications at both macro and micro levels. These models are reshaping global markets, driving new forms of currency, and creating a novel economic landscape. The reason LLMs are transforming industries and redefining economies is simple: they automate both routine and complex tasks that traditionally require human intelligence. They enhance decision-making processes, boost productivity, and facilitate cost reductions across various sectors. This enables organizations to allocate human resources toward more creative and strategic endeavors, resulting in the development of new products and services. From healthcare to finance to customer service, LLMs are creating new markets and driving AI-driven services like content generation and conversational assistants into the mainstream. To truly grasp the engine driving this new global economy, it’s essential to understand the inner workings of this disruptive technology. These posts will provide both a macro-level overview of the economic forces at play and a deep dive into the technical mechanics of LLMs, equipping you with a comprehensive understanding of the revolution happening now. Why Now? The Connection Between Language and Human Intelligence AI did not begin with ChatGPT’s arrival in November 2022. Many people were developing machine learning classification models in 1999, and the roots of AI go back even further. Artificial Intelligence was formally born in 1950, when Alan Turing—considered the father of theoretical computer science and famed for cracking the Nazi Enigma code during World War II—created the first formal definition of intelligence. This definition, known as the Turing Test, demonstrated the potential for machines to exhibit human-like intelligence through natural language conversations. The test involves a human evaluator who engages in conversations with both a human and a machine. If the evaluator cannot reliably distinguish between the two, the machine is considered to have passed the test. Remarkably, after 72 years of gradual AI development, ChatGPT simulated this very interaction, passing the Turing Test and igniting the current AI explosion. But why is language so closely tied to human intelligence, rather than, for example, vision? While 70% of our brain’s neurons are devoted to vision, OpenAI’s pioneering image generation model, DALL-E, did not trigger the same level of excitement as ChatGPT. The answer lies in the profound role language has played in human evolution. The Evolution of Language The development of language was the turning point in humanity’s rise to dominance on Earth. As Yuval Noah Harari points out in his book Sapiens: A Brief History of Humankind, it was the ability to gossip and discuss abstract concepts that set humans apart from other species. Complex communication, such as gossip, requires a shared, sophisticated language. Human language evolved from primitive cave signs to structured alphabets, which, along with grammar rules, created languages capable of expressing thousands of words. In today’s digital age, language has further evolved with the inclusion of emojis, and now with the advent of GenAI, tokens have become the latest cornerstone in this progression. These shifts highlight the extraordinary journey of human language, from simple symbols to intricate digital representations. In the next post, we will explore the intricacies of LLMs, focusing specifically on tokens. But before that, let’s delve into the economic forces shaping the LLM-driven world. The Forces Shaping the LLM Economy AI Giants in Competition Karl Marx and Friedrich Engels argued that those who control the means of production hold power. The tech giants of today understand that AI is the future means of production, and the race to dominate the LLM market is well underway. This competition is fierce, with industry leaders like OpenAI, Google, Microsoft, and Facebook battling for supremacy. New challengers such as Mistral (France), AI21 (Israel), and Elon Musk’s xAI and Anthropic are also entering the fray. The LLM industry is expanding exponentially, with billions of dollars of investment pouring in. For example, Anthropic has raised $4.5 billion from 43 investors, including major players like Amazon, Google, and Microsoft. The Scarcity of GPUs Just as Bitcoin mining requires vast computational resources, training LLMs demands immense computing power, driving a search for new energy sources. Microsoft’s recent investment in nuclear energy underscores this urgency. At the heart of LLM technology are Graphics Processing Units (GPUs), essential for powering deep neural networks. These GPUs have become scarce and expensive, adding to the competitive tension. Tokens: The New Currency of the LLM Economy Tokens are the currency driving the emerging AI economy. Just as money facilitates transactions in traditional markets, tokens are the foundation of LLM economics. But what exactly are tokens? Tokens are the basic units of text that LLMs process. They can be single characters, parts of words, or entire words. For example, the word “Oscar” might be split into two tokens, “os” and “car.” The performance of LLMs—quality, speed, and cost—hinges on how efficiently they generate these tokens. LLM providers price their services based on token usage, with different rates for input (prompt) and output (completion) tokens. As companies rely more on LLMs, especially for complex tasks like agentic applications, token usage will significantly impact operational costs. With fierce competition and the rise of open-source models like Llama-3.1, the cost of tokens is rapidly decreasing. For instance, OpenAI reduced its GPT-4 pricing by about 80% over the past year and a half. This trend enables companies to expand their portfolio of AI-powered products, further fueling the LLM economy. Context Windows: Expanding Capabilities

Read More
November 15, 2024in Artificial Intelligence, Google, Technology
guide to RAG
30Aug

Tectonic Guide to Rag Part 2

The first notable change in the field of language models is the significant expansion of context window sizes and a reduction in token costs. For instance, Anthropic’s largest model, Claude, has a context window exceeding 200,000 tokens, while recent reports indicate that Gemini’s context window can reach up to 10 million tokens. Under such circumstances, Retrieval-Augmented Generation (RAG) may no longer be necessary for many tasks, as all required data can be accommodated within the expanded context window. Several financial and analytical projects have already demonstrated that tasks can be solved without needing a vector database as intermediate storage. This trend of reducing token costs and increasing context window sizes is likely to continue, potentially decreasing the need for external mechanisms in LLMs, although they are still relevant for the time being. If the context window remains insufficient, methods for summarization and context compression have been introduced. LangChain, for example, offers a class called ConversationSummaryMemory to address this challenge. pythonCopy codellm = OpenAI(temperature=0) conversation_with_summary = ConversationChain( llm=llm, memory=ConversationSummaryMemory(llm=OpenAI()), verbose=True ) conversation_with_summary.predict(input=”Hi, what’s up?”) Knowledge Graphs As the volume of data continues to grow, navigating through it efficiently becomes increasingly critical. In certain cases, understanding the structure and attributes of data is essential for effective use. For example, if the data source is a company’s wiki, an LLM might not recognize a phone number unless the structure or metadata indicates that it’s the company’s contact information. Humans can infer meaning from conventions, such as the subdirectory “Company Information,” but standard RAG may miss such connections. This challenge can be addressed by Knowledge Graphs, also known as Knowledge Maps, which provide both raw data and metadata that illustrates how different entities are interconnected. This method is referred to as Graph Retrieval-Augmented Generation (GraphRAG). Graphs are excellent for representing and managing structured, interconnected information. Unlike vector databases, they excel at capturing complex relationships and attributes among diverse data types. Creating a Knowledge Graph The process of creating a knowledge graph typically involves collecting and structuring data, which requires expertise in both the subject matter and graph modeling. However, LLMs can automate a significant portion of this process by analyzing textual data, identifying entities, and recognizing their relationships, which can then be represented in a graph structure. In many cases, an ensemble of vector databases and knowledge graphs can improve accuracy, as discussed previously. For example, search functionality might combine keyword search through a regular database (e.g., Elasticsearch) and graph-based queries. LangChain can also assist in extracting structured data from entities, as demonstrated in this code example: pythonCopy codedocuments = parse_and_load_data_from_wiki_including_metadata() graph_store = NebulaGraphStore( space_name=”Company Wiki”, tags=[“entity”] ) storage_context = StorageContext.from_defaults(graph_store=graph_store) index = KnowledgeGraphIndex.from_documents( documents, max_triplets_per_chunk=2, space_name=space_name, tags=[“entity”] ) query_engine = index.as_query_engine() response = query_engine.query(“Tell me more about our Company”) Here, searching is conducted based on attributes and related entities, instead of similar vectors. If set up correctly, metadata from the company’s wiki, such as its phone number, would be accessible through the graph. Access Control One challenge with this system is that data access may not be uniform. For instance, in a wiki, access could depend on roles and permissions. Similar issues exist in vector databases, leading to the need for access management mechanisms such as Role-Based Access Control (RBAC), Attribute-Based Access Control (ABAC), and Relationship-Based Access Control (ReBAC). These access control methods function by evaluating paths between users and resources within graphs, such as in systems like Active Directory. To ensure the integrity of data during the ingestion phase, metadata related to permissions must be preserved in both the knowledge graph and vector database. Some commercial vector databases already have this functionality built in. Ingestion and Parsing Data needs to be ingested into both graphs and vector databases, but for graphs, formatting is especially critical since it reflects the data’s structure and serves as metadata. One particular challenge is handling complex formats like PDFs, which can contain diverse elements like tables, images, and text. Extracting structured data from such formats can be difficult, and while frameworks like LLama Parse exist, they are not always foolproof. In some cases, Optical Character Recognition (OCR) may be more effective than parsing. Enhancing Answer Quality Several new approaches are emerging to improve the quality of LLM-generated answers: While these advancements in knowledge graphs, access control, and retrieval mechanisms are promising, challenges remain, particularly around data formatting and parsing. However, these methods continue to evolve, enhancing LLM capabilities and efficiency. Like Related Posts Salesforce OEM AppExchange Expanding its reach beyond CRM, Salesforce.com has launched a new service called AppExchange OEM Edition, aimed at non-CRM service providers. Read more Salesforce Jigsaw Salesforce.com, a prominent figure in cloud computing, has finalized a deal to acquire Jigsaw, a wiki-style business contact database, for Read more Health Cloud Brings Healthcare Transformation Following swiftly after last week’s successful launch of Financial Services Cloud, Salesforce has announced the second installment in its series Read more Alphabet Soup of Cloud Terminology As with any technology, the cloud brings its own alphabet soup of terms. This insight will hopefully help you navigate Read more

Read More
August 30, 2024in Data
AI Then and Now
30Mar

AI Then and Now

AI: Transforming User Interactions and Experiences Have you ever been greeted by a waitress who already knows your breakfast order? It’s a relief not to detail every aspect — temperature, how do you want your eggs, what kind of juice, bacon or sausage, etc. This example encapsulates the journey we’re navigating with AI today. AI Then and Now. This article isn’t about ordering breakfast; it’s about the evolution of user interactions, particularly how generative AI might evolve based on past trends in graphical user interfaces (GUIs) and emerging trends in AI interactions. We’ll explore the significance of context bundling, user curation, trust, and ecosystems as key trends in AI user experience in this Tectonic insight. From Commands to Conversations Let’s rewind to the early days of computing when users had to type precise commands in a Command-Line Interface (CLI). Imagine the challenge of remembering the exact command to open a file or copy data. This complexity meant that only a few people could use computers effectively. To reach a broader audience, a shift was necessary. You might think Apple’s creation of the mouse and drop down menues was the pinnacle of success, but truly the evolution predates Apple. Enter ELIZA in 1964, an early natural language processing program that engaged users in basic conversations through keyword recognition and scripted responses. Although groundbreaking, ELIZA’s interactions were far from flexible or scalable. Around the same time, Xerox PARC was developing the Graphical User Interface (GUI), later popularized by Apple in 1984 and Microsoft shortly thereafter. GUIs transformed computing by replacing complex commands with icons, menus, and windows navigable by a mouse. This innovation made computers accessible and intuitive for everyday tasks, laying the groundwork for technology’s universal role in our lives. Not only did it make computing accessible to the masses but it layed the foundation upon which every household would soon have one or more computers! The Evolution of AI Interfaces Just as early computing transitioned from the complexity of CLI to the simplicity of GUIs, we’re witnessing a parallel evolution in generative AI. User prompts are essentially mini-programs crafted in natural language, with the quality of outcomes depending on our prompt engineering skills. We are moving towards bundling complex inputs into simpler, more user-friendly interfaces with the complexity hidden in the background. Context Bundling Context bundling simplifies interactions by combining related information into a single command. This addresses the challenge of conveying complex instructions to achieve desired outcomes, enhancing efficiency and output quality by aligning user intent and machine understanding in one go. We’ve seen context bundling emerge across generative AI tools. For instance, sample prompts in Edge, Google Chrome’s tab manager, and trigger-words in Stable Diffusion fine-tune AI outputs. Context bundling isn’t always about conversation; it’s about achieving user goals efficiently without lengthy interactions. Context bundling is the difference in ordering the eggs versus telling the cook how to crack and prepare it. User Curation Despite advancements, there remains a spectrum of needs where users must refine outputs to achieve specific goals. This is especially true for tasks like researching, brainstorming, creating content, refining images, or editing. As context windows and multi-modal capabilities expand, guiding users through complexity becomes even more crucial. Humans constantly curate their experiences, whether by highlighting text in a book or picking out keywords in a conversation. Similarly, users interacting with ChatGPT often highlight relevant information to guide their next steps. By making it easier for users to curate and refine their outputs, AI tools can offer higher-quality results and enrich user experiences. User creation takes ordering breakfast from a manual conversational process to the click of a button on a vending-like system. Designing for Trust Trust is a significant barrier to the widespread adoption of generative AI. To build trust, we need to consider factors such as previous experiences, risk tolerance, interaction consistency, and social context. Without trust, in AI or your breakfast order, it becomes easier just to do it yourself. Trust is broken if the waitress brings you the wrong items, or if the artificial intelligence fails to meet your reasonable expectations. Context Ecosystems Generative AI has revolutionized productivity by lowering the barrier for users to start tasks, mirroring the benefits and journey of the GUI. However, modern UX has evolved beyond simple interfaces. The future of generative AI lies in creating ecosystems where AI tools collaborate with users in a seamless workflow. We see emergent examples like Edge, Chrome, and Pixel Assistant integrating AI functionality into their software. This integration goes beyond conversational windows, making AI aware of the software context and enhancing productivity. The Future of AI Interaction Generative AI will likely evolve to become a collaborator in our daily tasks. Tools like Grammarly and Github Copilot already show how AI can assist users in creating and refining content. As our comfort with AI grows, we may see generative AI managing both digital and physical aspects of our lives, augmenting reality and redefining productivity. The evolution of generative AI interactions is repeating the history of human-computer interaction. By creating better experiences that bundle context into simpler interactions, empower user curation, and augment known ecosystems, we can make generative AI more trustworthy, accessible, usable, and beneficial for everyone. Like Related Posts Salesforce OEM AppExchange Expanding its reach beyond CRM, Salesforce.com has launched a new service called AppExchange OEM Edition, aimed at non-CRM service providers. Read more The Salesforce Story In Marc Benioff’s own words How did salesforce.com grow from a start up in a rented apartment into the world’s Read more Salesforce Jigsaw Salesforce.com, a prominent figure in cloud computing, has finalized a deal to acquire Jigsaw, a wiki-style business contact database, for Read more Health Cloud Brings Healthcare Transformation Following swiftly after last week’s successful launch of Financial Services Cloud, Salesforce has announced the second installment in its series Read more

Read More
March 30, 2024in AI Tools, Artificial Intelligence, Data, Generative AI, Google, Salesforce, Salesforce Einstein, Technology
Communicating With Machines
20Sep

Communicating With Machines

For as long as machines have existed, humans have struggled to communicate effectively with them. The rise of large language models (LLMs) has transformed this dynamic, making “prompting” the bridge between our intentions and AI’s actions. By providing pre-trained models with clear instructions and context, we can ensure they understand and respond correctly. As UX practitioners, we now play a key role in facilitating this interaction, helping humans and machines truly connect. The UX discipline was born alongside graphical user interfaces (GUIs), offering a way for the average person to interact with computers without needing to write code. We introduced familiar concepts like desktops, trash cans, and save icons to align with users’ mental models, while complex code ran behind the scenes. Now, with the power of AI and the transformer architecture, a new form of interaction has emerged—natural language communication. This shift has changed the design landscape, moving us from pure graphical interfaces to an era where text-based interactions dominate. As designers, we must reconsider where our focus should lie in this evolving environment. A Mental Shift In the era of command-based design, we focused on breaking down complex user problems, mapping out customer journeys, and creating deterministic flows. Now, with AI at the forefront, our challenge is to provide models with the right context for optimal output and refine the responses through iteration. Shifting Complexity to the Edges Successful communication, whether with a person or a machine, hinges on context. Just as you would clearly explain your needs to a salesperson to get the right product, AI models also need clear instructions. Expecting users to input all the necessary information in their prompts won’t lead to widespread adoption of these models. Here, UX practitioners play a critical role. We can design user experiences that integrate context—some visible to users, others hidden—shaping how AI interacts with them. This ensures that users can seamlessly communicate with machines without the burden of detailed, manual prompts. The Craft of Prompting As designers, our role in crafting prompts falls into three main areas: Even if your team isn’t building custom models, there’s still plenty of work to be done. You can help select pre-trained models that align with user goals and design a seamless experience around them. Understanding the Context Window A key concept for UX designers to understand is the “context window“—the information a model can process to generate an output. Think of it as the amount of memory the model retains during a conversation. Companies can use this to include hidden prompts, helping guide AI responses to align with brand values and user intent. Context windows are measured in tokens, not time, so even if you return to a conversation weeks later, the model remembers previous interactions, provided they fit within the token limit. With innovations like Gemini’s 2-million-token context window, AI models are moving toward infinite memory, which will bring new design challenges for UX practitioners. How to Approach Prompting Prompting is an iterative process where you craft an instruction, test it with the model, and refine it based on the results. Some effective techniques include: Depending on the scenario, you’ll either use direct, simple prompts (for user-facing interactions) or broader, more structured system prompts (for behind-the-scenes guidance). Get Organized As prompting becomes more common, teams need a unified approach to avoid conflicting instructions. Proper documentation on system prompting is crucial, especially in larger teams. This helps prevent errors and hallucinations in model responses. Prompt experimentation may reveal limitations in AI models, and there are several ways to address these: Looking Ahead The UX landscape is evolving rapidly. Many organizations, particularly smaller ones, have yet to realize the importance of UX in AI prompting. Others may not allocate enough resources, underestimating the complexity and importance of UX in shaping AI interactions. As John Culkin said, “We shape our tools, and thereafter, our tools shape us.” The responsibility of integrating UX into AI development goes beyond just individual organizations—it’s shaping the future of human-computer interaction. This is a pivotal moment for UX, and how we adapt will define the next generation of design. Content updated October 2024. Like Related Posts Salesforce OEM AppExchange Expanding its reach beyond CRM, Salesforce.com has launched a new service called AppExchange OEM Edition, aimed at non-CRM service providers. Read more Salesforce Jigsaw Salesforce.com, a prominent figure in cloud computing, has finalized a deal to acquire Jigsaw, a wiki-style business contact database, for Read more Health Cloud Brings Healthcare Transformation Following swiftly after last week’s successful launch of Financial Services Cloud, Salesforce has announced the second installment in its series Read more Top Ten Reasons Why Tectonic Loves the Cloud The Cloud is Good for Everyone – Why Tectonic loves the cloud You don’t need to worry about tracking licenses. Read more

Read More
September 20, 2023in Data

Subscribe to our mailing list. Join our mail list to receive our newsletter

We'll keep you up to date with our latest news, insights, free resources, and much more.
Don't worry, we won't spam you.
gettectonic.com