3. Object Recognition

3.1 Description

Object detection based on YOLO Fastest and NanoDet. Supports V-Training.

3.2 Web Operations

1. Click Add to upload a model file in .tar format. For training custom models, please see the tutorial UnitV2 V-Training
2. After selecting a model, click Run to execute the specified model. (Built-in models: nanodet_80class, yolo_20class can be used directly)

3.3 Serial Operations

// Select the parameter "yolo_20class" to switch to this function
{
    "function": "Object Recognition",
    "args": ["yolo_20class"]
}
// Select the parameter "nanodet_80class" to switch to this function
{
    "function": "Object Recognition",
    "args": ["nanodet_80class"]
}

yolo_20class: [
    "aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow", "diningtable", "dog", 
    "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor"
]

nanodet_80class: [
        "person","bicycle","car","motorbike","aeroplane","bus","train","truck","boat","traffic light",
        "fire hydrant","stop sign","parking meter","bench","bird","cat","dog","horse","sheep","cow",
        "elephant","bear","zebra","giraffe","backpack","umbrella","handbag","tie","suitcase","frisbee",
        "skis","snowboard","sports ball","kite","baseball bat","baseball glove","skateboard","surfboard",
        "tennis racket","bottle","wine glass","cup","fork","knife","spoon","bowl","banana","apple",
        "sandwich","orange","broccoli","carrot","hot dog","pizza","donut","cake","chair","sofa","pottedplant",
        "bed","diningtable","toilet","tvmonitor","laptop","mouse","remote","keyboard","cell phone","microwave",
        "oven","toaster","sink","refrigerator","book","clock","vase","scissors","teddy bear","hair drier","toothbrush"
]

3.4 Sample Output

{
    "num": 1,
    "obj": [
        {
            "prob": 0.938137174,
            "x": 179,
            "y": 186,
            "w": 330,
            "h": 273,
            "type": "person"
        }
    ],
    "running": "Object Recognition"
}

4. Color Tracker

4.1 Description

Detects specified color regions and returns the coordinates of the color area.

4.2 Web Operations

You can directly adjust the LAB threshold slider to filter out the background and get the interested color area. You can also directly frame the interested color area in the picture, the system will automatically calculate the color occupying the most proportion of the target area and filter out the background. You can further adjust the slider based on the calculation to get better filtering effect. Clicking the "To Mask Mode" button will switch to Mask mode, where you can directly see the filtering effect. Clicking the "To RGB Mode" button will switch back to RGB mode.

• About CIELAB color space
• LAB threshold is mapped to 0~255.
• L in LAB represents brightness, usually this threshold is not set (0~255), indicating that the impact of brightness is ignored. However, this will cause the algorithm to be unable to distinguish between black and white, please be aware.
• The algorithm will only return the largest target

4.3 Serial Operations

the following operations, please switch the function to Color Tracker

{
    "function": "Color Tracker",
    "args": ""
}

4.3.1 Specify LAB Threshold

• Send

  // * LAB threshold mapped to 0~255
  {
      "config":"Color Tracker",
      "l_min":0, // 0 ~ 255
      "l_max":0, // 0 ~ 255
      "a_min":0, // 0 ~ 255
      "a_max":0, // 0 ~ 255
      "b_min":0, // 0 ~ 255
      "b_max":0 // 0 ~ 255
  } 
  

• Receive

  {
      "running":"Color Tracker",
      "msg":"Data updated."
  }
  

4.3.2 Specify ROI (Automatic Threshold Calculation)

• Send

  {
  {
      "config":"Color Tracker",
      "x":0, // 0 ~ 640
      "y":0, // 0 ~ 480
      "w":30,
      "h":30,
  }
  

• Receive

  // * va and vb refer to the color dispersion degree within the ROI. If the dispersion is high, the tracking effect will be poor.
  {
      "running":"Color Tracker",
      "a_cal":0.0,
      "b_cal":0.0, // calculated threshold
      "va":0.0,
      "vb":0.0, // color dispersion rate
      "l_min":0, // fixed value 0
      "l_max":255, // fixed value 255
      "a_min":0, // a_cal - (10 + (int)(va / 2.0f))
      "a_max":0, // a_cal + (10 + (int)(va / 2.0f))
      "b_min":0, // b_cal - (10 + (int)(vb / 2.0f))
      "b_max":0  // b_cal + (10 + (int)(vb / 2.0f))
  } 
  

4.4 Sample Output

{
    "running":"Color Tracker",
    "cx": 0, // center X coordinate
    "cy": 0, // center Y coordinate
    "r": 0, // radius
    "mx": 0, // moment x position
    "my": 0 // moment y position
}

5. Lane Line Tracker

5.1 Description

Detects road lines in the frame, fits them into straight lines, and returns the line angle and coordinates.

5.2 Web Operations

You can directly adjust the LAB threshold slider to filter out the background and get the interested color area. You can also directly frame the interested color area in the picture, the system will automatically calculate the color occupying the most proportion of the target area and filter out the background. You can further adjust the slider based on the calculation to get better filtering effect. Clicking the "To Mask Mode" button will switch to Mask mode, where you can directly see the filtering effect. Clicking the "To RGB Mode" button will switch back to RGB mode.

• About CIELAB color space
• LAB threshold is mapped to 0~255.
• L in LAB represents brightness, usually this threshold is not set (0~255), indicating that the impact of brightness is ignored. However, this will cause the algorithm to be unable to distinguish between black and white, please be aware.

5.3 Serial Operations

{
    "function": "Lane Line Tracker",
    "args": ""
}

5.3.1 Specify LAB Threshold

• Send ```cpp  // * LAB threshold mapped to 0~255 {     "config":"Lane Line Tracker",     "l_min":0, // 0

5.3.1 Specify LAB Threshold

• Send

   // * Map LAB thresholds to 0~255
  {
      "config": "Lane Line Tracker",
      "l_min": 0, // 0 ~ 255
      "l_max": 0, // 0 ~ 255
      "a_min": 0, // 0 ~ 255
      "a_max": 0, // 0 ~ 255
      "b_min": 0, // 0 ~ 255
      "b_max": 0, // 0 ~ 255
  }
  

• Receive

  {
      "running": "Lane Line Tracker",
      "msg": "Data updated."
  }
  

5.3.2 Specify ROI (Automatically Calculate Thresholds)

• Send

  {
      "config": "Lane Line Tracker",
      "x": 0, // 0 ~ 640
      "y": 0, // 0 ~ 480
      "w": 30,
      "h": 30,
  }
  

• Receive

  //* va and vb refer to the color dispersion within the ROI. A higher dispersion indicates poorer segmentation results.
  {
      "running": "Lane Line Tracker",
      "a_cal": 0.0,
      "b_cal": 0.0, // Calculated thresholds
      "va": 0.0,
      "vb": 0.0, // Color dispersion rates
      "l_min": 0, // Fixed value 0
      "l_max": 255, // Fixed value 255
      "a_min": 0, // a_cal - (10 + (int)(va / 2.0f))
      "a_max": 0, // a_cal + (10 + (int)(va / 2.0f))
      "b_min": 0, // b_cal - (10 + (int)(vb / 2.0f))
      "b_max": 0  // b_cal + (10 + (int)(vb / 2.0f))
  } 
  

5.4 Example Output

{
    "running": "Lane Line Tracker",
    "x": 0,
    "y": 0, // Base point of the fitted line
    "k": 0  // Slope of the fitted line
} 

6. Target Tracker

6.1 Description

Track a selected target in the frame, using the MOSSE algorithm.

6.2 Web Operation

Simply box out the target of interest on the screen.

6.3 Serial Operation

{
    "function": "Target Tracker",
    "args": ""
}

6.4 Example Output

{
    "running": "Target Tracker",
    "x": 0,//0~640
    "y": 0,//0~480
    "w": 0,
    "h": 0
} 

7. Motion Tracker

7.1 Description

Detect and track moving targets, returning the coordinates and angle of the target.

7.2 Web Operation

Click the 'Set as background' button to determine the background. The algorithm can adapt to slowly changing backgrounds.

7.3 Serial Operation

{
    "function": "Motion Tracker",
    "args": ""
}

7.3.1 Determine Background

• Send

  // Sending this command will determine the background
  {
      "config": "Motion Tracker",
      "operation": "update"
  } 
  

• Receive

  {
      "running": "Motion Tracker",
      "msg": "Background updated."
  }
  

7.4 Example Output

{
    "running": "Motion Tracker",
    "num": 2,
    "roi": [
        {
            "x": 0,
            "y": 0,
        

    "w": 0,
            "h": 0,
            "angle": 0.0,
            "area": 0
        },
        {
            "x": 0,
            "y": 0,
            "w": 0,
            "h": 0,
            "angle": 0.0,
            "area": 0
        }
    ]
} 

8. Online Classifier

8.1 Description

This function can real-time train and classify objects within the green target box. The trained feature values can be saved on the device for future use.

8.2 Web Operation

1. Click the 'Reset' button to clear existing categories and enter training mode.
2. Click the '+' button to create a new category.
3. Select the category you want to train.
4. Place the object to be trained inside the green target box.
5. Click the 'Train' button to complete one training.
6. Change the angle of the object and repeat the training until you achieve the expected effect.
7. Click the 'save&run' button to save the training results on the device and exit training mode to start object classification.

8.3 Serial Operation

{
    "function": "Online Classifier",
    "args": ""
}

8.3.1 Train

• Send

  //This command will put the device into training mode and extract features once to be stored under the specified class. If the class_id does not exist, it will create this class.
  {
      "config": "Online Classifier",
      "operation": "train",
      "class_id": 1, // Integer (0 ~ N), ID of the class
      "class": "class_1" // String, name of the class
  }
  

• Receive

  {
      "running": "Online Classifier",
      "msg": "Training [class name] [num of training] times"
  }
  

8.3.2 Save&Run

• Send

  {
      "config": "Online Classifier",
      "operation": "saverun",
  }
  

• Receive

  {
      "running": "Online Classifier",
      "msg": "Save and run."
  }
  

8.3.3 Reset

• Send

  //This command will put the device into training mode and clear all categories.
  {
      "config": "Online Classifier",
      "operation": "reset",
  } 
  

• Receive

  {
      "running": "Online Classifier",
      "msg": " Please take a picture."
  }
  

8.4 Example Output

{
    "running": "Online Classifier",
    "class_num": 2, // Number of recognized classes
    "best_match": "class_1", // Best matching class
    "best_score": 0.83838, // Best matching score
    "class": [ // Score of each class
        {
            "name": "class_1",
            "score": 0.83838
        },
        {
            "name": "class_2",
            "score": 0.66244
        }
    ]
}

9. Face Recognition

9.1 Description

Detect and recognize faces.

9.2 Web Operations

1. Clicking the Reset button clears all existing faces.
2. Click the + button to create a new face.
3. Select the face you want to train.
4. Look at the camera, ensuring the face to train is in the correct position.
5. Click the Train button.
6. During training, a yellow boundary box means it is training. You can slowly turn your head to sample different angles for better recognition.
7. If the boundary box turns red, it means the target is lost, usually because the face has changed too much. Adjust the face position until it is found again.
8. Click Stop when you think the desired effect has been achieved. Now, the device can recognize this face.
9. Click the Save button to save the feature data on the device for future use.

9.3 Serial Port Operations

{
    "function": "Face Recognition",
    "args": ""
}

9.3.1 Train

• Send

  // To create a new face, please provide the face_id in sequence (0 ~ N).
  {
      "config":"Face Recognition",
      "operation":"train",
      "face_id":1, // Integer (0 ~ N), the ID of the face
      "name":"tom" // String, the name of the face
  }
  // For example, if there are already 3 faces (0~2), to create a new face, the id should be specified as 3.
  

• Receive (Success)

  {
      "running":" Face Recognition ",
      "msg":"Training tom" // Training face name
  }
  

• Receive (Error)

  {
      "running":"Face Recognition",
      "msg":"Invalid face id"
  }
  

9.3.2 Stop Train

• Send

  {
      "config":" Face Recognition ",
      "operation":" stoptrain",
  }
  

• Receive

  {
      "running":"Face Recognition",
      "msg":"Exit training mode."
  }
  

9.3.3 Save&Run

• Send

  {
      "config":" Face Recognition ",
      "operation":"saverun",
  }
  

• Receive

  {
      "running":"Face Recognition",
      "msg":"Faces saved."
  }
  

9.3.4 Reset

• Send

  // This command will delete all faces.
  {
      "config":"Face Recognition",
      "operation":"reset",
  }
  

• Receive

  {
      "running":"Face Recognition",
      "msg":"Reset success"
  }
  

9.4 Sample Output

9.4.1 Training Mode

{
    "running":"Face Recognition",
    "status":"training", // training or missing
    "x":0,
    "y":0,
    "w":0,
    "h":0, //  face detection bounding box
    "prob":0, // detection confidence
    "name":0,
}

9.4.2 Normal Mode (Matching Score > 0.5)

{
    "running":"Face Recognition",
    "num":1, // number of faces recognized
    "face":[
        {
            "x":0, // 0 ~ 320
            "y":0, // 0 ~ 240
            "w":30,
            "h":30, // face detection bounding box
            "prob":0, // detection confidence
            "match_prob":0.8, // matching confidence
            "name": "tom",
            "mark":[ // landmarks
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {


                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
            ]
        },
    ]
} 

9.4.3 Normal Mode (Matching Score <= 0.5)

{
    "running":"Face Recognition",
    "num":1, //  number of faces recognized
    "face":[
        {
            "x":0, // 0 ~ 320
            "y":0, // 0 ~ 240
            "w":30,
            "h":30, // face detection bounding box
            "prob":0, // confidence
            "name": "unidentified",
            "mark":[ // landmarks
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
            ]
        },
    ]
} 

10. Face Detector

10.1 Description

Detect faces in the frame and provide 5-point landmarks.

10.2 Web Operations

10.3 Serial Port Operations

{
    "function": "Face Detector",
    "args": ""
}

10.4 Sample Output

{
    "running":"Face Detector",
    "num":1, //  number of faces recognized
    "face":[
        {
            "x":0,
            "y":0,
            "w":30,
            "h":30, // face detection bounding box
            "prob":0, // confidence
            "mark":[ // landmarks
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                }
            ]
        }
    ]
}

11. Shape Detector

11.1 Description

Detect shapes in the frame and calculate their area. It can recognize squares, rectangles, triangles, pentagons, and circles.

11.2 Web Operations

Click the 'Set as background' button to set the background. This algorithm can adapt to slowly changing backgrounds.

11.3 Serial Port Operations

{
    "function": "Shape Detector",
    "args": ""
}

• Send

  // Sending this command will set the background
  {
      "config":"Shape Detector",
      "operation":"update"
  } 
  

• Receive

  {
      "running":"Shape Detector",
      "msg":"Background updated."
  }
  

11.4 Sample Output

{
    "running":"Shape Detector",
    "num":2,
    "shape":[
        {
            "name":"Rectangle", // "unidentified", "triangle", "square", "rectangle", "pentagon", "circle"
            "x":0,
            "y":0,
            "w":0,
            "h":0,
            "angle":0.0, // can be used when the shape is a square or rectangle
            "area":0
        },
        {
            "name":"Rectangle", // "unidentified", "triangle", "square", "rectangle", "pentagon", "circle"
            "x":0,
            "y":0,
            "w":0,
            "h":0,
            "angle":0.0, // can be used when the shape is a square or rectangle
            "area":0
        }
    ]
} 

12. Shape Matching

12.1 Description

Match any given shape (although it is better if the shape does not contain curves), the uploaded shape will be converted into feature data and saved on the device for future use.

12.2 Web Operation

1. Click the add button to add a shape, you need to upload an image of the shape template as shown below (PNG format, shape in black and background in white), the file name will be the shape's name.
   
2. Click the reset button to clear all uploaded shapes.
3. Click the 'Set as background' button to set the background. The algorithm can adapt to slowly changing backgrounds.

12.3 Serial Operation

To be developed, not supported yet.

12.4 Sample Output

// The shape returned here is the filename of the uploaded template, please note if the confidence rate is below 30% it will be labeled as unidentified.
{
    "running":"Shape Matching",
    "num":2,
    "shape":[
        {
            "name":"arrow", // Your custom shape name, when confidence rate is less than 30 it cannot be identified
            "max_score":83, // Confidence rating score, if the shape is unknown, it is not present
            "x":0,
            "y":0,
            "w":0,
            "h":0,
            "area":0
        },
        {
            "name":"unidentified", // Your custom shape name, when the confidence score is less than 30 it cannot be identified
            "x":0,
            "y":0,
            "w":0,
            "h":0,
            "area":0
        },
    ]
} 

13. Audio FFT

13.1 Description

Capture audio through the microphone on the device, perform real-time FFT (Fast Fourier Transform), and plot a time-frequency graph. The green chart below is the audio's RMS, indicating the current loudness level.

• The microphone's response cut-off frequency is about 10KHz.

13.2 Web Operation

13.3 Serial Operation

None

13.4 Sample Output

None

Serial Reading

Arduino

void setup() {

  Serial.begin(115200);
  Serial2.begin(115200, SERIAL_8N1, 16, 17);

}

void loop() {

 if(Serial2.available()) {
   String recvStr = Serial2.readStringUntil('/n');
   if(recvStr[0] == '{'){
     Serial.print(recvStr);
   }
 }
  
}

Micropython

import machine
import json

uart1 = machine.UART(1, tx=16, rx=17)
uart1.init(115200, bits=8, parity=None, stop=1)

PROTOCOL_START = b'{'[0]

while True:
 

 if uart1.any():
    data = uart1.readline()
    if data[0] == PROTOCOL_START:
        json_data = json.loads(data)

Using Python to Call Model Files

from json.decoder import JSONDecodeError
import subprocess
import json
import base64
import serial
import time
from datetime import datetime
from PIL import Image
import os
import io

uart_grove = serial.Serial('/dev/ttyS0', 115200, timeout=0.1)
reconizer = subprocess.Popen(['/home/m5stack/payload/bin/object_recognition', '/home/m5stack/payload/uploads/models/nanodet_80class'],
                             stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

reconizer.stdin.write("_{\"stream\":1}\r\n".encode('utf-8'))
reconizer.stdin.flush()

img = b''

while True:
    today = datetime.now()
    path = str(today.strftime("%Y_%m_%d") + "/")
    newpath = "/media/sdcard/" + path

    line = reconizer.stdout.readline().decode('utf-8').strip()
    if not line:
        break  # Process finished or empty line

    try:
        doc = json.loads(line)
        if 'img' in doc:
            byte_data = base64.b64decode(doc["img"])
            img = bytes(byte_data)
        elif 'num' in doc:
            for obj in doc['obj']:
                uart_grove.write(str(obj['type'] + '\n').encode('utf-8'))
                if obj['type'] == "aeroplane":
                    print('aeroplane ' + today.strftime("%Y_%m_%d_%H_%M_%S"))
                    if os.path.exists(newpath):
                        image_path = newpath + today.strftime("%Y_%m_%d_%H_%M_%S") + ".jpg"
                        img = Image.open(io.BytesIO(byte_data))
                        img.save(image_path, 'jpeg')
                    else:
                        os.mkdir(newpath)
                        image_path = newpath + today.strftime("%Y_%m_%d_%H_%M_%S") + ".jpg"
                        img = Image.open(io.BytesIO(byte_data))
                        img.save(image_path, 'jpeg')
                    time.sleep(1)
                else:
                    print('Not detect '+ today.strftime("%Y_%m_%d_%H_%M_%S"))
    except JSONDecodeError as e:
        print("Error: Invalid JSON string")
        print("JSONDecodeError:", str(e))