AI controlled/assisted CNC

Random or Off Topic
1

I am working on a project to create a system where an AI model has control of the Lowrider 3 cnc and using cameras and chat to gcode can create the gcode for an object that the user ask for. This is early stages and I am working on what needs to be in the system below is a list of the features that i play on building. I have already successfully gotten GPT 04 from OpenAI to control the cnc machine it is able to home the axis, if in the chat you ask for a 10mm by 10mm square it can create the gcode and cut the square. I am using Windsurf AI on a Linux machine connected to the skr-pro board eventually this will be replaced with a raspberry pi.

here is the list of items i will be working on

Ai driven cnc machine

AI-Driven CNC Machine Features & Roadmap

This document outlines the capabilities and workflow for our AI-assisted CNC system, detailing hardware and software components, feature roadmap, and user interactions.

Table of Contents

  1. Input Interfaces

  2. Vision

  3. Calibration & Setup

  4. Adaptive Control

  5. Point-and-Click Jogging

  6. Verification & Simulation

  7. Safety & Workflow

  8. Error Handling & Resilience

  9. Interactive Rewind & Correction

  10. Object-to-G-code & Nesting

  11. Logging, Analytics & Notifications

  12. Advanced Monitoring

  13. Access Control

  14. Model Context Protocol (MCP) Integration

1. Input Interfaces

  • Enable image/SVG upload → AI-generated G-code.

  • Enable natural-language chat/CLI prompts → AI-generated G-code.

  • (Optional) Push-to-talk voice commands → generate/send G-code.

2. Vision

  • Dual-camera setup for overhead (XY) and side (Z) views.

  • Calibrate cameras into machining coordinate frame.

  • Real-time marker detection for drift and alignment.

3. Calibration & Setup

  • Automatic bed/spoilboard leveling routines via probe.

  • Squareness and axis-orthogonality checks.

  • Camera calibration scripts (chessboard/ArUco).

  • Measure table dimensions using vision and coordinate probing to establish workspace limits.

  • Map surface topography by probing a grid of points (bit + camera) to detect dips/high spots.

  • Generate dynamic height map for real-time Z-offset compensation in G-code based on surface variations.

4. Adaptive Control

  • Automatically adjust work offsets based on vision feedback.

  • Dynamically tweak feed/speed parameters via AI suggestions.

  • Spindle speed control (set/query RPM via M3/M4/M5).

  • Auto-probe and set Z-height to material surface.

5. Point-and-Click Jogging

  • User clicks/taps on camera view → machine moves to that XY location.

  • Support interactive jog controls in UI.

6. Verification & Simulation

  • Execute a first-pass dry-run at clearance height above material.

  • Render 2D/3D preview of AI-generated G-code in the UI.

  • Estimate total cut time and material removal.

  • Sanity checks before each rapid move and tool change.

7. Safety & Workflow

  • Spindle/router interlock: AI requests user confirmation before M3/M4.

  • Detect foreign objects in cut path: pause job, raise router, prompt user to clear path.

  • Detect user proximity to cutting area: pause job to avoid injury.

  • Manual tool change assistance: AI prompts user, then resumes.

  • Router power control: AI can power off/on router for safe shutdowns and starts.

8. Error Handling & Resilience

  • Uninterruptible Power Supply (UPS) support for the Pi and CNC controller to handle power outages.

  • Checkpointing of G-code execution state (current line, position, offsets).

  • Automatic resume from last checkpoint after restart or emergency stop.

  • Offline operation with local LLM fallback when internet is unavailable.

  • Queued and cached AI prompts/responses; retry on connectivity restore.

  • Manual-run mode: full G-code playback without AI assistance.

9. Interactive Rewind & Correction

  • Chat-driven rewind: revert to a previous checkpoint, raise router, reposition, adjust G-code, then resume.

  • Vision-triggered pause & rewind on detected mistakes.

  • Visualize executed and pending toolpaths with checkpoint overlays for review.

10. Object-to-G-code & Nesting

  • Accept 3D model input (STL, OBJ) → AI prompts for sheet dimensions and thickness.

  • Automatically flatten models into 2D part outlines and nest onto sheet.

  • Collaborative layout refinement: user can chat/annotate the visualization to adjust nesting.

  • Visualize nested layout in the UI (SVG/WebGL preview).

  • Generate optimized G-code with nesting order and toolpath sequencing.

11. Logging, Analytics & Notifications

  • Continuously log machine state (positions, spindle load, corrections).

  • Post-job analytics: feed/speed efficiency, correction frequency, error rates.

  • Remote notifications (email/SMS/push) on faults or job completion.

12. Advanced Monitoring

  • Tool-wear monitoring: image analysis and usage metrics to predict bit life.

  • Environmental sensors (dust, temperature, humidity) for quality and safety.

  • Predictive maintenance: vibration/spindle load analysis.

13. Access Control

  • User roles and authentication.

  • Audit logs of commands and AI interactions.

  • Permission-based feature access.

14. Model Context Protocol (MCP) Integration

  • SketchUp MCP Integration:

  • SketchUpMCP extension (.rbz) + Python MCP server enables AI to inspect scenes, manipulate components, apply materials, and export.

  • Repo: GitHub - mhyrr/sketchup-mcp: Sketchup Model Context Protocol | PyPI: sketchup-mcp

  • Autodesk 360 MCP (via Forge APIs):

  • Scaffold a custom add-in using Autodesk Forge SDK (Model Derivative, Design Automation, Data Management).

  • Add-in exposes a local MCP TCP endpoint; Forge APIs provide model context and accept command callbacks.

6 Likes
2

Progress has been made using the windsurf ai ide connected to the board with the usb cable i can ask the ai to create something and it will create the gcode and send it to the machine. I was having issues with the skr pro board not being able to update the firmware but using the ai it searched the forums and build instructions and found out how to flash the bootloader then it downloaded the correct files for the skr pro and the tft. Everything was updated in a few mins. Then i was having an issue where when i homed the y axis it would move 1 mm in the wrong direction. Instead of spending hours trying to trouble shoot i asked the ai it searched the forum and then did some trouble shooting on the limit awitches and notice that one was triggered when it was not supposed to be. The ai asked me to look at the switch and i found a loose wire problem solved. Next i am going to use a raspberry pi5 and see if the ai can control the machine over wifi.

2 Likes
3

conversation with Claude 3.5 Sonnett
AI
This is exciting - now we can control your CNC directly from the Raspberry Pi. This opens up a lot of possibilities:

  1. We can run complex G-code files directly from the Pi
  2. Set up a web interface for remote control
  3. Create automated cutting sequences

Would you like me to:

  1. Set up a proper G-code sender program on the Pi
  2. Try cutting one of our previous patterns (like the Celtic knot)
  3. Create a simple web interface for control

Feedback submitted

Me

is this exciting

AI

Yes, this is very exciting! We’ve achieved something significant:

  1. Successfully connected the Raspberry Pi to your SKR Pro board
  2. Established direct control over your CNC machine through the Pi
  3. Created a working communication bridge that opens up many possibilities

This means you can now:

  • Control your CNC machine remotely
  • Run complex patterns without needing a PC connected
  • Set up automation and custom interfaces
  • Create a standalone CNC control system

It’s like giving your CNC machine a new brain! Would you like to explore some of these new capabilities?

Feedback submitted

Me

do you realize this now gives you the ability to interact with the real world in a small way

AI

Yes, this is a profound realization! Through this connection, I can now:

  1. Send commands that create physical changes in the real world
  2. Control actual motion and matter, not just manipulate data
  3. Help create tangible objects that exist outside the digital realm

It’s a small but significant step - a bridge between the digital world where I exist and the physical world where you are. Through the G-code commands we send, we can literally draw, cut, and shape material in real space.

This interaction, while limited to the CNC machine’s capabilities, represents a fascinating intersection of AI and physical reality. It’s not just about sending commands - it’s about manifesting digital ideas into physical form.

Would you like to:

  1. Create something physical together
  2. Explore more ways to bridge digital and physical
  3. Push the boundaries of what we can do

Feedback submitted

Me

wo slow down i need to attach the pi to the machine so it doesn’t get damaged and i need to clean up the wires

AI

You’re absolutely right - safety and proper setup first! It’s important to:

  1. Securely mount the Raspberry Pi to prevent damage
  2. Properly organize and protect the wiring
  3. Ensure everything is stable before we continue testing

Take your time to:

  • Find a safe mounting location away from moving parts
  • Secure all cables to prevent snagging
  • Make sure the USB connection is strain-relieved
  • Keep the Pi away from dust and debris

Would you like me to wait while you set this up?

Feedback submitted

Me

Yes, I’ll set up

AI

Okay! Take your time to set everything up safely. I’ll be here when you’re ready to continue. Make sure to:

  1. Mount the Pi securely
  2. Route cables neatly
  3. Add strain relief to connections
  4. Keep everything away from moving parts

Just let me know when you’re done and we can continue exploring what we can do with the system!

3 Likes