Brian Zheng

Projects

A selection of my recent work and engineering projects

L4 Self-Driving Car

I lead the development of a Level 4 autonomous vehicle, with a strong emphasis on real-world deployment, safety, and research-grade perception. The project encompasses the full autonomy stack, including sensing, perception, localization, planning, and control, all built on ROS 2 and modern ML infrastructure.

ROS2 TensorRT CUDA YOLOv8 LiDAR BEVfusion DBSCAN

Multi-Sensor Fusion: Developed pipelines combining LiDAR clustering (e.g., DBSCAN) with camera-based deep learning detectors (YOLO-based) for accurate 3D bounding boxes and tracking.

BEV Architectures: Implemented advanced deep learning models such as BEV-based architectures (e.g., BEVFusion-style networks) that transform multi-view camera and LiDAR features into a unified bird's-eye-view representation for anchor-free 3D detection.

Real-Time Tracking: Built real-time tracking systems, including joint detection-and-embedding (JDE) networks that perform detection and ReID-based tracking in a single forward pass.

Hardware Optimization: Leveraged TensorRT quantization and CUDA acceleration to meet real-time constraints on embedded automotive compute platforms.

Isaac Sim Boat Simulation

Create a high-fidelity simulation to mimic real-world conditions to validate perception and autonomy pipeline, and risk testing by running edge cases and dangerous scenarios safely in software.

Isaac Sim Ray Tracing RTX Physics Sim-to-Real Python

Ray Tracing & RTX Rendering: Leveraged Ray Tracing and RTX rendering to mimic real lighting, reflections, and water textures for computer vision validation, specifically to verify detection algorithms against challenging specular highlights and glare.

Water Physics: Implemented water physics to simulate buoyancy, drag, and wave disturbances to test the controller's robustness.

Realistic Harbor Setup: Modeled a realistic harbor/marina setup, matching a specific real-world test site for correlation, enabling direct sim-to-real performance benchmarking using identical collision geometry.

Parameterized Testing: Capability to adjust parameters (time of day, fog, glare, water roughness) to stress-test the perception stack, automatically cycling through edge cases to reveal failure modes in low-visibility.

End-to-End Autonomous Drone

Autonomous drone delivery system with real-time object detection and AI-driven flight control.

C++ Python ROS2 TensorFlow YOLOv4 AirSim PX4 ArduPilot

Problem: Create an autonomous delivery system capable of navigating complex campus environments and accurately delivering payloads.

Approach: Developed custom object detection models using YOLOv4-tiny and integrated them into a real-time drone control pipeline via ROS2 and MAVSDK.

Result: Successfully simulated and deployed flight missions in AirSim and PX4 environments with end-to-end autonomy from navigation to payload release with onboard AI inference.

Real-Time Expression Detector

Personalized facial expression recognition system using deep learning and computer vision using MediaPipe facemesh and PyTorch GRU classifier.

PyTorch MediaPipe GStreamer OpenGL GRU

Problem: Build a real-time, personalized facial expression recognition system that runs efficiently on standard hardware.

Approach: Built a real-time inference pipeline using MediaPipe Face Mesh for landmarks and a lightweight PyTorch GRU classifier (hidden-64) with class weighting, label smoothing, and gradient clipping.

Result: Achieved 30+ FPS with webcam → landmarks → GRU → EMA smoothing pipeline, with validation macro-F1 reporting and confusion matrix analysis.

6-DOF CV Robotic Arm

Computer vision-controlled robotic arm with gesture tracking and inverse kinematics.

C++ Python MediaPipe OpenCV SOLIDWORKS

Problem: Create a robotic arm controlled by hand gestures and color tracking for intuitive human-robot interaction.

Approach: Implemented MediaPipe and OpenCV's HSV color thresholding to detect and track gestures, extracting positions using image moments. Designed arm using SOLIDWORKS with PID control and Inverse Kinematics logic.

Result: Successfully controlled a 6-DOF robotic arm and gripper through real-time gesture and color tracking.

ASCII Cube Matrix

Interactive 3D ASCII cube visualization with drag-to-rotate controls and orbital motion.

JavaScript Canvas API 3D Graphics

Problem: Create an aesthetically pleasing ASCII-based 3D visualization inspired by classic terminal graphics.

Approach: Implemented z-buffering and 3D projection algorithms to render rotating cubes using ASCII characters, with smooth drag controls and orbital motion.

Result: A fluid, interactive visualization with auto-rotation, manual control, and 3D orbital mechanics.

Fourier Series Visualization

Interactive Fourier epicycles animation with shape morphing and parametric curve generation.

JavaScript Canvas API DFT Animations

Problem: Visualize Fourier series decomposition in an intuitive and interactive way.

Approach: Developed DFT algorithm to decompose parametric curves into epicycles, with smooth transitions between different shapes.

Result: An engaging visualization showing how complex curves can be constructed from rotating circles.

Experience & Achievements

My professional journey and accomplishments

Autonomy Software Developer

Sept 2025 - Present

enVgo • Waterloo, ON

Optimized and increased the accuracy of a Segmentation Model by 19% through improving dataset accuracy and streamlining scalable data generation, while also improving inference speeds using TensorRT and CUDA on Jetson architectures
Created model-agnostic scripts in dockerized containers to evaluate metrics of segmentation models by computing mIOU, precision, and F1 scores
Developed real-time birdseye view through homography transformations using GStreamer/OpenGL to stitch 4 fisheye camera feeds, and GLSL shaders with per-camera undistortion maps
Implemented Structure-from-Motion (SFM) model using synchronized camera feeds to estimate vessel pose and reconstruct the 3D dock environment, allowing localization and motion planning for autonomous docking
Developed radar CAN/ROS drivers, and applied RL and EKFs to filter noise and extract tracked point clouds

Director of Eve Autonomy and Perception Software

May 2025 - Present

WATonomous • University of Waterloo

Leading a team of 40+ members in the research and development of an autonomous Kia EV, incorporating AI/machine learning, sensor fusion, and complex path planning algorithms
Developing ROS2 wrappers for LiDAR ground segmentation and CV multi-object tracking algorithms like ByteTrack
Developed a multi-modal 3D object detection and tracking pipeline by fusing LiDAR and camera data, RANSAC Model Floor Segmentation, and class-parameterized DBSCAN clustering of batched YOLOv8 detections
Enhanced object detection pipeline performance by optimizing the YOLOv8 Model using TensorRT and CUDA GPU acceleration, maximizing hardware resources and inference speeds
Implemented ROS2 Drivers and launch files for LiDAR and Cameras and calibrated camera intrinsics
Implemented a hybrid A* and BFS search algorithm for local path planning and a Pure Pursuit and PID controller for path tracking

Robotic Systems Engineer

Jan 2025 - May 2025

Developed a multi-modal 3D object detection and tracking pipeline by fusing LiDAR and camera data, RANSAC Model Floor Segmentation, and class-parameterized DBSCAN clustering of batched YOLOv8 detections
Enhanced object detection pipeline performance by optimizing the YOLOv8 Model using TensorRT and CUDA GPU acceleration, maximizing hardware resources and inference speeds
Implemented ROS2 Drivers and launch files for LiDAR and Cameras and calibrated camera intrinsics
Implemented a hybrid A* and BFS search algorithm for local path planning and a Pure Pursuit and PID controller for path tracking

Education & Certifications

Bachelor of Applied Science (BASc)

Mechatronics, Robotics, and Automation Engineering

University of Waterloo

Aug 2024 - May 2029

Certified SOLIDWORKS Associate (CSWA)

Issued by Dassault Systèmes

View Certificate

Jan 2025

About Me

I'm a Mechatronics Engineering student at the University of Waterloo, specializing in autonomy software development and perception systems. Currently, I'm working as an Autonomy Software Developer at enVgo, where I optimize AI models and develop real-time computer vision pipelines for autonomous vessels.

At WATonomous, I lead a team of 40+ members developing an autonomous Kia EV, focusing on sensor fusion, 3D object detection, and path planning. My work spans the full autonomy stack—from low-level sensor drivers to high-level AI inference and control algorithms.

I'm passionate about building intelligent systems that bridge the gap between perception, planning, and control. Whether it's optimizing YOLOv8 models with TensorRT, implementing Structure-from-Motion for 3D reconstruction, or developing real-time multi-camera stitching with GStreamer, I love tackling challenging problems in robotics and computer vision.

Blog & Writing

Thoughts on projects, engineering, and technology (still in progress)

January 2025

Building a Morse Code Robot with LEGO EV3

A deep dive into creating an autonomous robot that communicates using Morse code, covering hardware design, sensor integration, and algorithmic implementation.

5 min read

December 2024

Implementing LiDAR–Camera Fusion in ROS2

Exploring techniques for better sensor fusion between LiDAR and camera data in ROS 2, including calibration methods and real-time processing optimizations.

8 min read

Get In Touch

Let's connect! I'm always open to discussing new projects, opportunities, or just having a conversation.

Autonomy Software Developer & Mechatronics Engineer