Lumos

Lumos Introduction

Lumos is a Chrome extension that serves as a RAG LLM co-pilot for browsing the web, powered by local LLMs. This extension is built on top of Ollama, which allows for inference to be done on your local machine without relying on any remote server support. However, due to security constraints in the Chrome extension platform, it does require a local server to run the LLM. Lumos is inspired by the Chrome extension example provided by the Web LLM project and the local LLM examples provided by LangChain.

Lumos Features

Key Features

• Local Inference: Lumos uses local LLMs to process information, ensuring privacy and speed.
• Multi-model Support: Users can select different Ollama models and embedding models for various tasks.
• Custom Content Parsing: The ability to customize how content is parsed from web pages for better indexing and searching.
• Multimodal Support: Lumos can process images found on web pages and use them in the prompting process.
• File Attachments: Supports uploading and processing various file types, including PDFs and plain text files.
• Keyboard Shortcuts: Several shortcuts are available to make using Lumos more efficient.

Technical Details

• Ollama Server: A local Ollama server is required for the embedding database and LLM inference.
• Docker Support: The Ollama server can be run in a Docker container for easier setup and management.
• Chrome Extension: Lumos is installed as a Chrome extension, allowing for seamless integration with the browsing experience.

Lumos Use Cases

• Summarizing Content: Lumos can summarize long threads on issue tracking sites, forums, and social media sites.
• News Articles: Users can quickly get summaries of news articles.
• Business and Product Reviews: Ask questions about reviews on business and product pages.
• Technical Documentation: Navigate through and ask questions about long, technical documentation.

Lumos Installation

To install Lumos, users need to download and install Ollama and the CLI. The Ollama server must be running with the appropriate environment variables set to allow requests from the Chrome extension. Detailed instructions are provided for macOS and Docker setups.

Loading the Extension

Users can load the unpacked extension directly from the Chrome extensions page by navigating to chrome://extensions/ and selecting "Load unpacked."

Setting Up Shortcuts

Lumos can be made easily accessible with a keyboard shortcut. Users can configure this shortcut by navigating to chrome://extensions/shortcuts and setting a shortcut for Lumos.

Lumos Options

Right-clicking on the extension icon reveals several options:

• Ollama Model: Choose the desired model for LLM inference.
• Ollama Embedding Model: Select the desired embedding model.
• Ollama Host: Specify the host for the Ollama server.
• Vector Store TTL: Set the time-to-live for cached URL content.
• Content Parser Config: Customize how web page content is parsed.
• Enable/Disable Tools: Toggle various tools within Lumos.
• Enable/Disable Dark Arts: A playful option that adds a bit of mystery to the extension.

Lumos Custom Content Parsing

Lumos allows users to create custom content parsers for different URLs. This is done by configuring the chunkSize, chunkOverlap, and selectors for the content to be indexed into the RAG vector store. This feature is particularly useful for optimizing the way Lumos interacts with specific websites.

Lumos Highlighted Content

If content is highlighted on a web page, Lumos can parse that content instead of using the content parser configuration. This is useful for quickly extracting and processing specific information from a page.

Lumos Multimodal Support

Lumos can process images on the current page and use them in the multimodal model prompting. This feature enhances the capabilities of Lumos by allowing it to work with visual content.

Lumos File Attachments

File attachments can be uploaded to Lumos, and their contents will be processed through the RAG workflow. Supported file types include CSV, JSON, PDF, and any plain text file format. When a file attachment is present, page content will not be parsed.

Lumos Keyboard Shortcuts

Several keyboard shortcuts are available to enhance the user experience:

• cmd + c: Copy last message to clipboard.
• cmd + b: Load clipboard text as a file attachment.
• cmd + j: Toggle content parsing.
• cmd + k: Clear all messages.
• cmd + ;: Open/close Chat History panel.
• ctrl + c: Cancel request.
• ctrl + x: Remove file attachment.
• ctrl + r: Regenerate last LLM response.

Lumos Tools

Lumos includes several experimental tools that can be invoked automatically based on the provided prompt. These tools are designed to enhance the functionality of Lumos and provide additional capabilities to the user.

Lumos Documentation

Extensive documentation is available for users who want to learn more about Lumos. Topics include setting up the local LLM in the browser, working with multimodal models, and tips for using LangChain with Lumos.

This markdown document provides an overview of Lumos, its features, installation process, and usage. It is designed to help users understand the capabilities of Lumos and how to effectively use it for their browsing needs.