Baidu robotics project: construction, mining, and industrial machinery autonomy.
Robotics Application
Autonomous Machinery System
Applying robotics, AI, and edge computing to make traditional heavy machinery more intelligent, safer, and more productive.
RoboForce Engineering Interview discussion
Jason Ye, Project Core Contributor, previous work at Baidu USA, 2020-2024
1 / 5
Project Context
Background & role
From lab research to real industrial autonomy deployment.
About me
- Zhixian Ye, or Jason
- UCSD ECE robotics track
- Baidu Research RAL, Sensing World AI/product, GM perception
Why it mattered
- Labor shortage in harsh industrial sites
- Improve safety without replacing people
- Jiangsu waste-factory excavator ran 24/7
Engineering goal
- Parse high-level customer tasks into common autonomy commands
- Share perception, planning, control, and safety modules
- Reuse simulation and replay across machine types
The goal was not to replace people, but to help smaller teams operate heavy machinery more safely, consistently, and profitably.
2 / 5
System Architecture
Robot stack
A hierarchical stack turned operator intent into coordinated base and arm execution.
Human-Robot Interface
Operator dispatches high-level work: move, dig, trench, supervise.
Task Resolver
Parses commands into segments, work cycles, and executable robot behaviors.
Base Loop
- Encoders, cameras, LiDAR, RTK localization
- Global route, local mapping, path following
- Obstacle avoidance while repositioning the machine
Arm & Digging Loop
- Inclination sensors and hydraulic valve readings
- Camera/LiDAR material-height and pile-shape sensing
- Dig, move, dig cycle with calibrated MPC/PID-style controls
3 / 5
Generalization
Reusable platform
The platform generalized from one machine to multiple industrial robots.
1
Machine boundary
- Reused excavator perception on wheel loader and crane
- Normalized sensors, image sizes, mapping, and localization
2
Autonomy stack
- Shared perception and edge-computing pipeline
- Common ROS-style middleware and A-to-B planning
3
Operations & safety
- Shared fleet dispatcher UI and business metrics
- Safety zones, stop-sign detection, state machines, fallback
4
Sim/replay loop
- Vendor simulator for base motion, arm motion, and materials
- Unit, replay, and CI tests before field deployment
4 / 5
Project Impact
Research to product
The project created research visibility, IP, commercial traction, and a reusable autonomy platform.
40+ papers
Research visibility
Top robotics and automation venues across control, planning, perception, and autonomy.
Patents
Defensible IP
Motion execution, machine interfaces, perception deployment, and safety workflows.
Deals
China + overseas
Local China deployments plus overseas commercial opportunities, including Brazil.
20+ machines
Platform scale
Reusable autonomy foundation scaled across excavator, wheel loader, crane, and others.
My contribution
- Core contributor: motion planning, execution cycles, HRI/frontend, perception streaming, frame buffers, AI deployment
- Supervised a research intern and led a 5-person SmartCrane project team
- Balanced customer requirements, ops deployment/testing, and resource-limited delivery
Personally authored 5+ papers; 3 patents filed, 2 granted.
5 / 5
Part 2
Technical Deep Dive
Imitation Learning and Model Integrated Excavator Trajectory Planning
IROS 2022, IEEE/RSJ International Conference on Intelligent Robots and Systems
Qiangqiang Guo, Zhixian Ye, Liyang Wang, Liangjun Zhang · RAL, Baidu Research
6 / 18
Problem Motivation
Trajectory planning
Excavator autonomy depends on trajectories that are productive, safe, and repeatable under changing terrain.
Operational pressure
- Increasing labor cost and skilled-operator shortage
- Hazardous environments and repetitive cycles
Technology timing
- Better sensors, compute, and AI made closed-loop autonomy practical
- The planner sits between perception and hydraulic execution
Planning challenge
- Pure rules are stable but brittle
- Pure learning adapts but needs feasibility guarantees
The core technical question: can we combine learned operator priors with a model-based optimizer, instead of choosing only one side?
7 / 18
Method Landscape
Why hybridize
Each planning family wins on a different axis, which made a hybrid method attractive.
| Criterion | Rule | Optimization | Imitation learning | Reinforcement learning |
|---|---|---|---|---|
| Optimality | * | **** | *** | ***** |
| Generalization | * | **** | *** | ***** |
| Stability | ***** | **** | *** | **** |
| Training / data efficiency | **** | ** | **** | ** |
| Application efficiency | ***** | ** | **** | ** |
8 / 18
9 / 20
10 / 20
11 / 20
12 / 20
13 / 20
STOMP Optimizer
Gradient-free refinement
STOMP iteratively samples trajectory perturbations and moves toward lower-cost feasible motion.
Why STOMP
- Gradient-free method
- Flexible with mixed objectives
- Can refine non-smooth trajectory costs
Main risk
- Sensitive to initial trajectory
- Requires parameter tuning
- Pure linear initialization can be weak
Hybrid fix
- Use the IL actor trajectory as the initial reference
- Let optimization handle constraints, volume, and smoothness
Guidance from the IL actor
Resample the trajectory generated by the actor and use it as the optimizer's reference.
IL prior supplies initial shape; STOMP optimizes feasible execution.
12 / 18
Cost Design
Objectives & constraints
The optimizer balances feasibility, human-like guidance, smoothness, and excavation volume.
Point + trajectory cost
Each sampled trajectory is scored point-by-point, then aggregated across the whole bucket path.
C = hard constraints + guidance + smoothness + volume
Hard constraints
Keep trajectory points inside the predefined 3D excavation area and away from invalid states.
Penalty rises with distance outside the allowed region.
Human-like guidance
Use the IL actor trajectory as a reference so optimization starts from an operator-like motion.
Deviation from IL reference becomes part of the cost.
Smoothness + volume
Prefer smooth bucket motion while targeting the desired amount of material per excavation.
Stable motion plus productive digging, not one at the expense of the other.
13 / 18
Experiment Settings
Ablation setup
The evaluation compared the hybrid method against pure learning and pure optimization baselines.
Tested algorithms
- IL+STOMP: proposed hybrid algorithm
- Pure-IL: actor trajectory revised for kinematic feasibility
- Pure-STOMP: linearly initialized optimizer without guidance cost
Protocol
- 3 episodes for each method
- 15 excavations per episode
- Same initial terrain state per episode; POA selected by rules
Physical setup
Depth digital twin
14 / 18
Experiment Results
Method comparison
IL+STOMP improved excavation weight while retaining high success and lower variance.
+24.77%
Avg. weight vs pure-STOMP
+18.42%
Success rate vs pure-STOMP
-35.53%
Deviation vs pure-IL
| Epi | Index | pure-STOMP | pure-IL | IL + STOMP | ||
|---|---|---|---|---|---|---|
| value | vs STOMP | vs IL | ||||
| 1 | Avg-W | 0.763 | 0.905 | 0.969 | 26.96% | 7.15% |
| Stdev | 0.176 | 0.193 | 0.148 | -15.59% | -23.09% | |
| Suc-R | 0.867 | 1.000 | 1.000 | 15.38% | 0.00% | |
| 2 | Avg-W | 0.742 | 0.978 | 0.964 | 29.94% | -1.35% |
| Stdev | 0.146 | 0.185 | 0.143 | -1.75% | -22.62% | |
| Suc-R | 0.867 | 1.000 | 1.000 | 15.38% | 0.00% | |
| 3 | Avg-W | 0.813 | 0.920 | 0.960 | 18.00% | 4.30% |
| Stdev | 0.101 | 0.236 | 0.108 | 6.95% | -54.30% | |
| Suc-R | 0.800 | 1.000 | 1.000 | 25.00% | 0.00% | |
| Total | Avg-W | 0.773 | 0.932 | 0.965 | 24.77% | 3.25% |
| Stdev | 0.144 | 0.204 | 0.131 | -8.78% | -35.52% | |
| Suc-R | 0.844 | 1.000 | 1.000 | 18.42% | 0.00% | |
15 / 18
Trajectory Comparison
Same terrain state
Trajectory shapes under the same terrain state.
16 / 18
Material Generalization
Generic vs specialized
Generic and specialized IL models reduced weight variance, while rocky terrain required more conservative behavior.
| Model | Baseline | Wooden block | Generic | ||
|---|---|---|---|---|---|
| value | improvement | value | improvement | ||
| Avg-W | 0.21248 | 0.22882 | 7.69% | 0.22754 | 7.09% |
| Min-W | 0.02530 | 0.12740 | 403.56% | 0.08190 | 223.72% |
| Max-W | 0.39370 | 0.32060 | -18.57% | 0.32160 | -18.31% |
| Stdev | 0.08332 | 0.05127 | -38.47% | 0.05391 | -35.30% |
| Suc-R | 0.5556 | 0.8444 | 52.00% | 0.7442 | 33.95% |
Interpretation
- Material type changes the relationship between trajectory shape and moved volume
- Specialized learning can reduce variance when material behavior is consistent
- Rocky terrain such as wooden block pushes the planner toward conservative choices
For field deployment, material awareness matters as much as geometry: the same bucket path can behave differently in sand, soil, or rock-like debris.
17 / 18
Conclusion & Discussion
Technical takeaways
The method improved excavation weight up to 24.77% while keeping low variance and high success rate.
What worked
Learning supplied a strong initial trajectory prior; optimization made it constraint-aware and physically safer.
What remains hard
A larger and more diverse dataset is needed for broader terrain, material, and machine variation.
Where it goes next
More advanced learning methods and real excavator experiments close the loop from benchtop to field deployment.
Discussion anchor: how I think about hybrid autonomy systems where learned priors accelerate planning, but model-based structure keeps the robot accountable.
18 / 18