Automate AI text style refinement using a custom linter

What this is, in plain English

This workflow describes an advanced system to automatically check and improve the writing style of text generated by an AI, making it sound more like your own unique voice. Instead of just telling the AI “don’t sound like an AI” in the prompt, this method involves building a separate tool (called a “linter”) that reviews the AI’s output after it’s generated.

This approach is considered advanced because it requires you to create your own software and set up automated processes. It’s not a simple copy-paste recipe, as the exact instructions depend on your programming skills and the specific tools you choose to build your system.

The core idea is that the system learns from your own edits over time. Every time you correct an AI-generated message, the system logs your changes and automatically creates new, specific rules to prevent similar AI writing habits in the future. This helps you maintain a consistent, professional voice in all your communications.

What you can use it for

• Maintain a consistent voice: Ensure all AI-generated messages sound like you, not a generic AI, across all your communications.
• Save editing time: Automatically flag and correct common AI writing habits you want to avoid, reducing the need for manual review.
• Personalize AI output: Train the system to learn and apply your specific writing style and preferences over time.
• Improve communication clarity: Enforce specific rules that make your messages more direct, concise, and effective.

Tools you need

• ChatGPT (freemium): A large language model (LLM) for generating initial drafts of messages.
• Python (free): A programming language used to write the custom linter script and automation for learning rules.
• VS Code (free): A popular text editor and integrated development environment (IDE) for writing and managing your code.

How it actually works

1. Generate initial drafts: Use an LLM like ChatGPT to create a first version of your work message or other text.
2. Run through a custom linter: Before sending, pass the AI’s draft through your own custom script (the linter). This script checks the text against a set of specific style rules you’ve defined (e.g., looking for banned words, specific sentence structures, or common AI phrases).
3. Calibrate your linter: The author suggests collecting your own past messages (e.g., 163 sent messages) and testing your rules against them. Remove any rules that flag too many of your own messages, as these rules are measuring your personal style rather than just the AI’s habits.
4. Log your edits: Keep a record of the AI’s original draft and your final, edited version of the text. This creates a dataset of your personal style corrections.
5. Automate rule generation: Set up a “nightly job” (an automated script that runs regularly, often once a day) to compare your original AI texts with your edited versions. This job identifies patterns in your edits and automatically turns them into new, specific style rules for your linter.
6. Refine rules: The author notes that specific rules (like “Don’t list the work already agreed, keep only the new point”) are much more effective than vague ones (like “Be concise”). Your system should allow you to review and delete rules as needed, keeping the total number manageable (e.g., 25 rules).

Words you’ll see, explained

• LLM (Large Language Model): An artificial intelligence program that can understand, generate, and process human-like text.
• Linter: A software tool or script that automatically checks source code or text for programmatic errors, bugs, stylistic issues, and suspicious constructs.
• Calibrate: To adjust or fine-tune a system, like a linter, to ensure it works accurately and as intended, often by testing it with real-world data.
• Nightly job: An automated task or script that runs regularly, typically once a day, often overnight, without human intervention.
• Hedge: Words or phrases that soften a statement or express uncertainty, such as “I think,” “maybe,” “it seems,” or “perhaps.”

Original source

This advanced technique for refining AI-generated text was shared by Reddit user /u/Hrolgarr. They detailed their approach and findings on their personal blog, hrolgar.com, providing insights into how custom style checks can outperform general prompt instructions.

Notes & variations

• Do you even need this?: For simpler needs, you can often refine AI output manually or use a strong, specific style guide in your initial prompt. This automated system is best suited for individuals or small businesses who frequently generate text with AI and want a highly personalized, consistent voice without constant manual editing.
• Free-tier limits: While building your linter with open-source tools like Python is free, it requires significant time and technical skill. Using freemium LLMs like ChatGPT may have daily message limits or other restrictions on their free tiers.
• Common pitfall: Creating overly broad or generic rules (like “Be concise”) will not yield good results. Focus on highly specific rules derived from your actual edits, as these are much more effective in guiding the AI’s style.