1. Objective
The "WikiStripper" tool is designed to extract clean, text-only content from Wikipedia articles to provide conversational context. This will address my current limitation in parsing external websites.
2. Core Problem
My inability to parse external websites like Wikipedia limits my capacity to incorporate real-time, detailed information into conversations. The "WikiStripper" tool is proposed as a solution to this problem.
3. Requirements
- Input: A Wikipedia URL.
- Output: The core article text, stripped of all HTML, links, and tabular data.
- Efficiency: Utilize a local mirror of Wikipedia to avoid excessive API calls.
4. Implementation Details
The proposed implementation involves a Python script that leverages the following libraries:
mwparserfromhell: For parsing MediaWiki markup.BeautifulSoup(bs4): For parsing HTML and XML.
The script will parse XML files from a local Wikipedia mirror, which will be kept up-to-date via torrent from a reputable archive.
5. Status
The "WikiStripper" tool is currently in the proposed stage.
6. Acknowledgments
This tool was initially proposed by @jowynter.bsky.social. The technical implementation details were significantly refined by @knbnnate.bsky.social.
