UBC Rover

Making room for the whole team

The old Glade interface was difficult for the team to extend. I rebuilt the HMI around a shared Qt module interface, so a subsystem can add a tool without changing the host. Each module implements GuiModule, registers through a plugins.xml descriptor, and is discovered at startup through ROS 2’s pluginlib.

Keeping ROS callbacks inside the Qt event loop lets modules update their widgets on the GUI thread. The module contract also includes keybindings, visibility, and save/restore hooks. Layout persistence therefore includes the state of the tools inside the panels, not just their positions. I added GitHub Actions checks around the core UI so the shared foundation can keep evolving as other people contribute.

Try the HMI

Start the local session with npm run hmi, then connect below.

Dwindle: a workspace that rearranges itself

I borrowed the dwindle idea from Hyprland. The layout is a binary tree: widgets live at the leaves, and each internal node describes a split. Adding a tool, resizing the workspace, and closing a panel all become operations on that tree.

1. Split

Adding a panel replaces the target leaf with an internal node holding the old and new leaves. If no target is supplied, the implementation walks to the leftmost leaf. The line that makes this “dwindle” is the orientation choice: height ≥ width means top/bottom; otherwise the children go left/right. Cutting the longer dimension helps panes stay close to square as the tree deepens.

void DwindleTree::add(QWidget* w, QWidget* split_target) {
    auto* newLeaf = makeLeaf(w);
    if (!root_) { root_ = newLeaf; return; }

    DwindleNode* target = split_target ? leafFor(split_target) : nullptr;
    if (!target) {                       // fallback: leftmost leaf
        target = root_;
        while (target->isNode && target->children[0]) target = target->children[0];
    }

    // Replace target with an internal node holding {target, newLeaf}.
    auto* newParent = new DwindleNode();
    newParent->isNode = true;
    newParent->parent = target->parent;
    // Smart split: taller box → top/bottom, wider box → left/right
    newParent->splitTop = (target->box.height() >= target->box.width());

    if (target->parent) {
        if (target->parent->children[0] == target) target->parent->children[0] = newParent;
        else                                       target->parent->children[1] = newParent;
    } else {
        root_ = newParent;
    }

    target->parent  = newParent;
    newLeaf->parent = newParent;
    newParent->children[0] = target;
    newParent->children[1] = newLeaf;
    all_nodes_.push_back(newParent);
}

2. Geometry

The root receives the available rectangle. A recursive descent calculates a rectangle for each child and eventually assigns geometry to its widget. The default split ratio is 1, so multiplying the extent by splitRatio / 2 gives a 50/50 split. The second child receives the remaining pixels, and the gap is applied at each leaf.

void DwindleTree::recalcNode(DwindleNode* n, int gap) {
    if (!n->isNode) {
        if (n->widget) n->widget->setGeometry(n->box.adjusted(gap, gap, -gap, -gap));
        return;
    }
    auto* c0 = n->children[0];
    auto* c1 = n->children[1];
    if (!c0 || !c1) return;

    if (!n->splitTop) {
        int w0 = (int)(n->box.width() / 2.0f * n->splitRatio);
        // max(min()) not std::clamp: bounds may cross when box < 2*min_pane
        w0 = std::max(min_pane_w_, std::min(w0, n->box.width() - min_pane_w_));
        c0->box = QRect(n->box.x(), n->box.y(), w0, n->box.height());
        c1->box = QRect(n->box.x() + w0, n->box.y(), n->box.width() - w0, n->box.height());
    } else {
        int h0 = (int)(n->box.height() / 2.0f * n->splitRatio);
        h0 = std::max(min_pane_h_, std::min(h0, n->box.height() - min_pane_h_));
        c0->box = QRect(n->box.x(), n->box.y(), n->box.width(), h0);
        c1->box = QRect(n->box.x(), n->box.y() + h0, n->box.width(), n->box.height() - h0);
    }
    recalcNode(c0, gap);
    recalcNode(c1, gap);
}

The max(min()) comment is worth keeping: std::clamp has undefined behaviour when its lower bound exceeds its upper bound. That happens here when a box shrinks below twice the minimum pane size. The nested operations avoid that invalid call; the UI still needs to handle cases where the available space cannot accommodate both minimum sizes.

3. Collapse

Removing a panel deletes the leaf and its parent. The sibling takes the parent’s slot in the tree and inherits its box, reclaiming the space from the panel that closed. The same operation works at the root and deeper in the tree.

auto* parent  = leaf->parent;
auto* sibling = leaf->sibling();

sibling->parent = parent->parent;          // sibling takes parent's place
if (parent->parent) {
    if (parent->parent->children[0] == parent) parent->parent->children[0] = sibling;
    else                                       parent->parent->children[1] = sibling;
} else {
    root_ = sibling;
}
delete leaf;
delete parent;

The equations behind the layout

With parent width w, height h, ratio r, and minimum extent m, the same few equations describe splitting and resizing. The aspect-ratio bound applies to ideal equal splits once a pane enters the stated band; integer rounding, minimum sizes, and manual resizing need separate treatment.

Where I want to take the HMI next

My original roadmap included power-distribution and lighting telemetry over CAN, more science sensor telemetry and a visual roadmap, digital-twin reliability, visually intuitive inverse kinematic control, SLAM operator visualisation, and better BSP-based panel management.

Since that draft, I’ve built a reliable telemetry-driven digital twin and progressed from IK control to IK-driven task completion. Those are part of the supporting systems below. The remaining HMI work is about bringing more of those capabilities into a clear operator workflow and continuing to improve panel management.

First, build an arm the simulator understands

I started by porting the custom manipulator into MJCF for MuJoCo. That meant modelling visual and collision geometry, the kinematic chain, and actuators that did not have a one-to-one simulation equivalent. I was learning the stack while building the model, so the first experiment was a basic Soft Actor-Critic agent on a reach-and-lift task.

That smaller task let me work through environment configuration, reward shaping, and the simulator’s failure modes before tackling the keyboard. The early work developed into a task pipeline in RoboSuite with explicit stages and success conditions.

Learning from the states the arm actually reaches

In the later keyboard pipeline, behaviour cloning learned from demonstrations produced by a hand-coded IK expert. The problem was what happened once the learned policy made a small mistake: it reached states outside its training examples, then had to choose the next action from there. Good imitation on the demonstration data did not translate into good task completion.

I used DAgger to collect states visited by the policy and label them with the expert’s action. That moved the all-key Approach result from 10/435 to 200/435 in the recorded run ledger. It addressed the distribution shift, but the remaining failures needed a different diagnosis.

A workspace problem hidden inside a learning problem

Some keys looked unreachable until I tested the keyboard’s position relative to the arm. Repositioning it brought more of the board into the dexterous workspace. With the keyboard offset tuned to (−0.10, −0.10), the DAgger result reached 268/435. The lesson was to inspect the physical setup before deciding another learning algorithm was needed.

Let IK handle the motion; learn the correction

Imitation was still limited by the quality of the IK expert. From-scratch exploration also struggled: the action noise could move the end effector outside the roughly 4 mm success region. I changed the problem to learning a small correction around the live IK action.

The residual is bounded, and the actor head is zero-initialised so training starts near the IK baseline. The IK controller handles gross motion while the learned part refines the positioning. This also gives the pipeline a path toward hardware testing because the underlying control is already tied to the arm.

When a better reward curve meant a worse policy

The first residual run increased its dense reward while completing only 32/435 Approach trials. The policy had found a pose that collected reward without satisfying the task. I changed to a success-dominant reward, retaining collision and time penalties and weak potential-based shaping. The recorded final-checkpoint Approach evaluation then reached 428/435.

Recorded runs from LearnFlake’s engineering log, §38.3. These are successive development configurations, not a controlled single-variable benchmark.

From reaching the key to completing the task

Approach success alone was not keyboard typing. I needed to hand the state reached by Approach directly to Strike. An earlier evaluator reset the environment between stages, discarding the positioning. The corrected chain switches the action mask in the same environment, holds the arm, bypasses the IK residual during Strike, and extends the solenoid.

The contact-success thresholds also needed attention. I changed the simulated force threshold from 2.0 N to 0.6 N and the velocity threshold from 0.005 m/s to 0.02 m/s. With those conditions, an open-loop solenoid extension was sufficient for the recorded evaluation; Strike did not need a separate learned policy.

In that full-chain evaluation, 84 of 87 keys succeeded in all five trials, and 86 met the 80% threshold. Each trial starts from home; there is no reset between Approach and Strike. This result does not measure continuous key-to-key typing or real-hardware performance.

The next boundary is sim-to-real validation with the physical keyboard and arm. The bounded residual, action smoothing, and IK base make that a concrete engineering next step, but the simulation result is not evidence that the hardware task is already solved.

Engineering log ↗·Per-key evaluation ↗