CUHK-X Competition | A small model for Multimodal Action Recognition

Picture a flat where an elderly person lives alone. A system that notices a fall, or a slow change in how someone washes their face or gets out of a chair, would be genuinely useful. Nobody wants an RGB camera in that flat, though. The CUHK-X challenge starts from this tension: recognize what a person is doing from depth, thermal, infrared, inertial, radar and skeleton data, with no RGB at all [1].

This post walks through my entry to the Small Model Track of the challenge, organized at UbiComp / ISWC 2026. The full code is on GitHub [4]. The idea is easy to state: train one small network per sensor, turn each network's output into a handful of tokens, and let a tiny transformer decide how much to trust each sensor for every clip. The details are where it gets interesting, because the test set has missing sensors, the evaluation is cross-subject, and everything has to fit in 100 MB.

The competition in brief

The challenge has two parallel Kaggle tracks. The Large Model Track asks for visual question answering with large multimodal models. The Small Model Track, which is the one discussed here, is a plain 40-class classification problem: given a multimodal clip, predict which of 40 daily activities is happening [1]. It targets edge deployment in smart-home and healthcare scenarios, where the model must run on low-power devices with limited memory and compute.

The rules are what make the track interesting. Models must be conventional architectures (CNN, RNN or Transformer), their size must not exceed 100 MB, and large pretrained backbones are forbidden. Closed-source APIs and LLMs are not allowed either, even for labeling training data [1]. Everything has to be trained from scratch on the competition data.

The evaluation is strictly cross-subject. Training uses users 1 to 9 and 16 to 24, while testing uses users 10, 11, 25 and 26 [1]. A model that memorizes the gait or the body proportions of a participant will look great on a random split and fail on the leaderboard. For finalists, the Kaggle score is only 20% of the final grade. Organizer-run private evaluation weighs 30%, and model efficiency has its own 10% share [1], so staying small is a goal and not only a constraint.

The data

CUHK-X contains 64,267 synchronized samples across seven modalities, collected from 30 participants performing 40 activities in two indoor environments [1]. The dataset paper also defines action understanding and reasoning tasks on top of plain recognition [2], but the small track only uses the classification part. RGB is removed. The Hugging Face release lists six modalities [3]:

The training labels live in the folder structure, HAR/data/<modality>/<action>/<user>/<trial>, and the test set has 405 anonymized clips [3]. Two details of the dataset card shape the whole design. Sampling rates differ across modalities, and not every clip has every modality [3]. A clip can be a few hundred radar frames, sixteen useful depth frames and no thermal data at all, and the model has to cope with all of these cases.

Three constraints, one architecture

Most of my design choices trace back to three problems:

Size. The budget is 100 MB. The repository counts size as fp32 parameters times four bytes, so the ceiling is about 25 million parameters across all branches, the fusion stage and every auxiliary head.

Cross-subject generalization. Every preprocessing step asks the same question: is this signal about the action, or about the person? Body size, facing direction, sensor mounting and absolute position are all cues that identify a subject, and none of them helps on unseen users.

Missing modalities. The fusion model cannot assume that all six inputs exist. It must work for a clip with six sensors and for a clip with two.

The answer is a late-fusion design. Each modality gets its own backbone, trained on its own, which outputs a small set of tokens of shape [B, K, D]. A fusion transformer then reads the tokens of all present modalities, plus a learned classification token, and predicts the action. The figure below shows the whole pipeline.

Turning folders into samples

The first job is to reassemble clips. The dataset stores each modality in its own folder tree, so one physical recording is spread over up to six directories. The DataIndex class walks every modality tree and merges folders that share the same (action, user, trial) key into a single record, which keeps a dictionary from modality name to folder path:

for modality in tqdm.tqdm(MODALITIES):
    data_root = os.path.join(self.data_root, modality)
    # ... walk action / user / trial directories ...
    action_id = int(action.split("_")[0])
    # ...
    key = (action_id, user, trial)
    entry = self._samples.setdefault(
        key, {"label": action_id, "user": user, "paths": {}})
    entry["paths"][modality] = tdir

The label comes from the numeric prefix of the action folder, such as 0_Wash_face. A missing modality simply means a missing key in paths. The class also has a statistics() method that prints how many samples have all six modalities and which combinations of missing ones occur, which is the first thing worth looking at before trusting any fusion result.

A validation split that behaves like the test set

Since the leaderboard is cross-subject, the local validation must be too. A random split of clips would put the same person in both training and validation, and the accuracy would be inflated by subject identity. The split is done on users instead:

def split_by_user(samples, val_fraction=0.2, seed=0):
    users = sorted({s["user"] for s in samples})
    rng = random.Random(seed)
    rng.shuffle(users)
    n_val = max(1, int(round(len(users) * val_fraction)))
    val_users = set(users[:n_val])
    train = [s for s in samples if s["user"] not in val_users]
    val = [s for s in samples if s["user"] in val_users]
    return train, val, sorted(val_users)

With 18 training users and a fraction of 0.2, this holds out four whole users. The validation set is small and noisy, but it is honest, and every number shown by the training scripts comes from it.

Fixed shapes and a presence mask

Batching clips of different modalities needs fixed tensor shapes. The loaders map every clip to a constant shape: 16 frames of 112 by 112 pixels for the three image modalities, 64 frames of 17 joints for the skeleton, a 30 by 400 grid for the IMU and a 32 by 8 point grid per radar bin. The MultiModalityDataset then fills absent modalities with zeros and returns a binary mask next to the data, so that the default collate function works and the model still knows what is real:

def __getitem__(self, i):
    s = self.items[i]
    out, mask = {}, torch.zeros(len(self.modalities))
    for j, m in enumerate(self.modalities):
        if m in s["paths"]:
            out[m] = Loaders.get(m)(s["paths"][m], self.train)
            mask[j] = 1.0
        else:
            out[m] = torch.zeros(SHAPES[m])
    return out, mask, s["label"]

One loader per modality

Each modality has its own loader, with its own normalization and its own augmentation. This is only possible because the backbones are trained independently first. As a comment in the config file puts it, there is no need to keep augmented data consistent across channels, so a skeleton can be rotated freely without caring what the depth camera would have seen.

Depth, infrared and thermal frames

The three image modalities share one loader. Clips are short and neighboring frames are nearly identical, so the loader samples 16 frames uniformly in relative clip time and adds a small random jitter during training as temporal augmentation:

def sample_indices(n, k, jitter):
    idx = np.linspace(0, n - 1, k)
    if jitter and n > k:
        idx = idx + np.random.uniform(-0.5, 0.5, size=k) * (n / k)
    return np.clip(np.round(idx), 0, n - 1).astype(int)

Spatial augmentation parameters are drawn once per clip, not once per frame. A random crop, a flip and a brightness gain are applied identically to all 16 frames, because otherwise the augmentation would invent motion that does not exist. At validation time the crop is centered and nothing else changes:

load_size = int(VISION_SIZE * AUG_CROP_FACTOR)
max_off = load_size - VISION_SIZE
if train:
    ox = np.random.randint(0, max_off + 1)
    oy = np.random.randint(0, max_off + 1)
    flip = np.random.rand() < 0.5
    gain = np.random.uniform(*AUG_GAIN_RANGE)
else:
    ox = oy = max_off // 2
    flip = False
    gain = 1.0

Frames are loaded at 1.15 times the target size and then cropped, and pixel values stay in [0, 1]. There is no mean and standard deviation normalization here on purpose. The networks are trained from scratch and start with a BatchNorm layer, which absorbs any fixed affine change of the input anyway.

Skeleton

The skeleton loader reads one JSON file per frame and takes the first detected person. Pose estimators fail from time to time, so a corrupt or non-finite frame repeats the last valid pose instead of injecting garbage:

for i in idx:
    try:
        with open(files[i]) as f:
            data = json.load(f)
        kp = np.asarray(data[0]["keypoints"], dtype=np.float32)
        if kp.shape == (NUM_JOINTS, 3) and np.isfinite(kp).all():
            last = kp
    except Exception:
        pass
    seq.append(last.copy())

Then come the two normalizations that matter most for cross-subject transfer. Subtracting the root joint removes where the person stands, and dividing by the mean joint distance removes how large the person is. During training, a random rotation around the vertical axis (up to 30 degrees) and a little Gaussian noise are added on top:

x = np.stack(seq)
x = x - x[:, :1, :]
scale = np.linalg.norm(x.reshape(-1, 3), axis=1).mean()
if scale > 1e-4:
    x = x / scale
if train:
    R = rot_about_axis(np.random.uniform(-AUG_SKEL_ROT_RAD, AUG_SKEL_ROT_RAD), SKELETON_UP_AXIS)
    x = x @ R.T
    x = x + np.random.normal(0, AUG_SKEL_NOISE, x.shape).astype(np.float32)

The file also contains a variant, load_skeleton_2, that rotates each clip so that the average hip-to-hip vector points in a fixed direction. It would remove the facing direction deterministically instead of relying on augmentation, but in the current code that step is commented out, because the hip vector is itself noisy.

Radar

Radar is the strangest modality. Each raw frame is a small point cloud, typically a handful of points with position, velocity, signal-to-noise ratio and a noise level. An earlier version of my loader collapsed every frame to a mean, a standard deviation and a maximum. With so few points per frame, standard deviation and maximum are mostly noise, and the collapse happened before any learnable layer could tell signal from noise. The current loader keeps the raw points instead. It splits the sequence of raw frames into 32 contiguous temporal bins, concatenates all points that fall in one bin, and keeps at most 8 of them, preferring the strongest signal-to-noise ratio:

n = len(frames)
edges = np.linspace(0, n, RADAR_LEN + 1).astype(int)
pts = np.zeros((RADAR_LEN, RADAR_MAX_PTS, RADAR_POINT_FEATS), dtype=np.float32)
mask = np.zeros((RADAR_LEN, RADAR_MAX_PTS), dtype=bool)

for i in range(RADAR_LEN):
    lo, hi = edges[i], max(edges[i + 1], edges[i] + 1)
    merged = np.concatenate(frames[lo:hi], axis=0)
    # ... point dropout (train only) ...
    if len(merged) > RADAR_MAX_PTS:
        order = np.argsort(-merged[:, 4])   # snr_norm column
        merged = merged[order[:RADAR_MAX_PTS]]
    # ... xyz jitter (train only) ...
    k = len(merged)
    pts[i, :k] = merged
    mask[i, :k] = True

Binning has a clear advantage over the naive alternative. The raw recording has hundreds of frames and we want 32 steps. Nearest-frame sampling would throw most of the points away, while merging keeps all of them until the strongest ones are selected. The output has shape [32, 8, 7]: six point features plus a validity flag for the padded slots.

IMU

The IMU data comes from five wearable units, each providing a three-axis accelerometer and a three-axis gyroscope, which gives 30 channels. The loader is designed to keep only those two sensor types. Absolute angles, quaternions and magnetometer readings encode how a unit was mounted and in which session it was recorded, so they leak subject identity under a cross-subject split. Each device stream is resampled onto a grid of 400 steps spanning the clip, which normalizes every clip to the same length in relative time, and a device that is absent stays at zero. Channel-wise mean and standard deviation, fitted on the training split only, are applied afterwards by a small Standardizer class, since accelerometer and gyroscope channels use very different units.

Backbones: one small network per sensor

All backbones follow one contract. They take the tensor of their modality and return [B, K, D], which is K tokens of width D, and they expose embed_dim and num_tokens as attributes. A dictionary maps modality names to builder functions, so the joint model can construct any subset of branches:

BUILDERS = {
    "Depth_Color": depth_channel_backbone.build,
    "IR": ir_channel_backbone.build,
    "Thermal": thermal_channel_backbone.build,
    "IMU": imu_channel_backbone.build,
    "Radar": radar_channel_backbone.build,
    "Skeleton": skeleton_transformer_backbone.build,
}

The budget is not split evenly. Depth gets the widest network, thermal frames are described in the code as low information and often missing, so that branch is narrower, and radar is called the weakest signal and gets the smallest budget.

Frames: a 2D CNN followed by temporal convolutions

For image sequences I avoided 3D convolutions, which are expensive, and split the problem in two. A shared 2D CNN encodes every frame independently, then a short stack of dilated 1D convolutions mixes the per-frame features along time. An adaptive pooling layer finally squeezes the time axis to K tokens:

class DepthColorBackbone(nn.Module):
    def __init__(self, in_channels=3, widths=(32, 64, 128, 256),
                 embed_dim=256, num_tokens=4):
        super().__init__()
        blocks, c = [], in_channels
        for w in widths:
            blocks += [ConvBlock2d(c, w, stride=2), ConvBlock2d(w, w)]
            c = w
        self.frame_net = nn.Sequential(*blocks)
        self.temporal = nn.Sequential(
            ConvBlock1d(c, embed_dim, k=3, dilation=1),
            ConvBlock1d(embed_dim, embed_dim, k=3, dilation=2))
        self.pool = nn.AdaptiveAvgPool1d(num_tokens)
        self.embed_dim = embed_dim
        self.num_tokens = num_tokens

    def forward(self, x):
        b, t, c, h, w = x.shape
        f = self.frame_net(x.reshape(b * t, c, h, w))
        f = f.mean(dim=(2, 3)).reshape(b, t, -1).transpose(1, 2)
        f = self.temporal(f)
        return self.pool(f).transpose(1, 2)

Folding time into the batch dimension lets the whole clip go through the CNN in one call. Four stride-2 stages bring a 112 by 112 frame down to 7 by 7, and a spatial mean turns it into one vector per frame. The infrared and thermal backbones are subclasses that only change the input channels and the widths, which keeps them independently tunable without touching the depth branch.

IMU: dilated convolutions

For one-dimensional signals, dilated convolutions are a cheap way to see a long window [5]. The IMU backbone stacks four blocks with kernel size 5 and dilations 1, 2, 4 and 8, the first two with stride 2:

class IMUBackbone(nn.Module):
    def __init__(self, in_channels=30, widths=(64, 128, 128, 256),
                 dilations=(1, 2, 4, 8), strides=(2, 2, 1, 1),
                 embed_dim=256, num_tokens=3):
        super().__init__()
        layers, c = [], in_channels
        for w, d, s in zip(widths, dilations, strides):
            layers.append(ConvBlock1d(c, w, k=5, stride=s, dilation=d))
            c = w
        layers.append(ConvBlock1d(c, embed_dim, k=3))
        self.net = nn.Sequential(*layers)
        self.pool = nn.AdaptiveAvgPool1d(num_tokens)

By my count, the receptive field of this stack is 221 steps of the 400-step grid, a bit more than half of the clip, which is enough to cover a whole short motion such as a sit-to-stand transition. Each block also contains dropout and a leaky ReLU, which is useful regularization for wearable data from a small number of subjects.

Radar: pool the points, then attend over time

The radar branch has two stages. The first one summarizes the points of each temporal bin using a small 1D convolution over the point axis, followed by masked average and max pooling. The mask matters, because most bins have padded slots that must not take part in the max:

class CoarsePointPool(nn.Module):
    def __init__(self, in_feats, hidden=32, out_dim=32):
        super().__init__()
        self.conv1 = ConvBlock1d(in_feats, hidden, k=3, stride=1, dilation=1)
        self.conv2 = ConvBlock1d(hidden, hidden, k=3, stride=1, dilation=1)
        self.proj = nn.Linear(hidden * 2, out_dim)  # avg + max concat

    def forward(self, pts, mask):
        B, T, N, F = pts.shape
        x = (pts * mask.unsqueeze(-1)).view(B * T, N, F).transpose(1, 2)
        h = self.conv1(x)
        h = self.conv2(h) + h                       # skip connection
        h = h.transpose(1, 2).view(B, T, N, -1)

        m = mask.unsqueeze(-1)
        count = m.sum(2).clamp(min=1)
        avg = (h * m).sum(2) / count
        mx = h.masked_fill(~m, float("-inf")).max(2).values
        mx = torch.where(torch.isinf(mx), torch.zeros_like(mx), mx)

        pooled = self.proj(torch.cat([avg, mx], dim=-1))
        return torch.cat([pooled, count.log1p()], dim=-1)

Note the last line. The number of points in a bin is itself informative, since a person who moves a lot reflects more radar points than a person sitting still, so its logarithm is appended as an extra feature. The second stage is a small transformer [7] over the 32 per-bin embeddings, with learned positional embeddings and a skip connection around the whole stack, followed by pooling to tokens:

def forward(self, x):
    pts = x[..., :-1]                       # [B, T, N, F]
    mask = x[..., -1] > 0.5                 # [B, T, N] bool
    frame_emb = self.point_pool(pts, mask)  # [B, T, point_out+1]
    h = self.in_proj(frame_emb)             # [B, T, embed_dim]
    h = self.transformer(h)                 # [B, T, embed_dim]
    f = h.transpose(1, 2)
    return self.pool(f).transpose(1, 2)     # [B, num_tokens, embed_dim]

Skeleton: a transformer with a learned resampler

The skeleton is the cleanest modality and gets the deepest network. Each frame is flattened to 51 numbers (17 joints times three coordinates) and linearly embedded, then a pre-norm transformer encoder [7][8] mixes frames across time. The default depth is 13 layers of width 128, which is cheap, because the sequence is only 64 frames long and the width is small.

The interesting part is how the variable-length sequence becomes a fixed number of tokens. Instead of average pooling, a small set of learned query vectors cross-attends into the encoded sequence, in the style of the Perceiver [9]. The backbone also accepts a padding mask, so clips of different lengths could be padded instead of resampled, and two skip connections keep the deep stack easy to optimize:

def forward(self, x, mask=None):
    b, t, j, c = x.shape
    feats = x.reshape(b, t, j * c)
    h0 = self.input_proj(feats)
    h0 = h0 + sinusoidal_position_encoding(t, self.d_model, x.device, h0.dtype)

    key_padding_mask = None if mask is None else ~mask.bool()
    enc = self.encoder(h0, src_key_padding_mask=key_padding_mask)
    enc = self.encoder_norm(enc + h0)        # stack-level residual

    q = self.query.unsqueeze(0).expand(b, -1, -1)
    pooled, _ = self.resampler(q, enc, enc, key_padding_mask=key_padding_mask)
    pooled = self.resampler_norm(pooled + q) # resampler residual
    return self.out_proj(pooled)

The sinusoidal position encoding is computed for whatever length arrives, so there is no learned table with a hard maximum. The idea is that fixed-size requirements are pushed to the very last step and handled by attention, instead of by frame-index interpolation upstream.

Using the skeleton graph

A flat vector of 51 numbers ignores the fact that joints are connected. The repository therefore contains a second skeleton backbone, spatial_skeleton.py, that embeds each joint separately and applies one or two graph convolution hops over the bone graph, per frame, before the temporal transformer. This is the standard graph convolution of Kipf and Welling [10], used on skeletons in the spirit of ST-GCN [11]. A residual connection keeps the per-joint identity alive:

class SpatialGCNBlock(nn.Module):
    def __init__(self, channels, num_joints, edges, dropout=0.1):
        super().__init__()
        self.register_buffer("A", build_adjacency(num_joints, edges))
        self.lin = nn.Linear(channels, channels)
        self.norm = nn.LayerNorm(channels)
        self.drop = nn.Dropout(dropout)

    def forward(self, x):  # x: [B, T, J, C]
        agg = torch.einsum("ij,btjc->btic", self.A.to(x.dtype), x)
        out = self.drop(torch.relu(self.lin(agg)))
        return self.norm(x + out)  # residual

A companion file, spatialized_skeleton_transformer.py, goes one step further. It uses the symmetric normalization of the adjacency matrix and adds a small learnable edge gate on top of the anatomical graph. The gate is passed through a tanh and scaled by 0.1, so the network can add weak long-range links, for example between the two hands, but it cannot override the skeleton structure.

The graph variant also has a pretraining path, inspired by masked autoencoders [12]. Random (frame, joint) entries are replaced by a learnable mask token, and a throwaway decoder reconstructs the hidden coordinates from the temporal encoder output. Because the decoder is discarded before fine-tuning, it does not count against the size budget:

def _embed_joints(self, x, joint_frame_mask=None):
    h0 = self.joint_embed(x)  # [B, T, J, joint_ch]
    if joint_frame_mask is not None:
        h0 = torch.where(joint_frame_mask.unsqueeze(-1), self.mask_token, h0)
    h = h0
    for block in self.spatial_blocks:
        h = block(h)
    return h + h0

def random_joint_frame_mask(b, t, j, ratio, device):
    return torch.rand(b, t, j, device=device) < ratio

This is an experimental branch of the project, and the joint model currently builds the plain transformer skeleton backbone shown before.

Fusion: tokens in, one decision out

With six trained backbones, the JointModel class has two jobs: put the tokens in a common space and decide how to combine them. Each modality's tokens are projected to a shared width of 256 and tagged with two learned embeddings. One identifies the modality, the other the position of the token inside its modality. A learned CLS token is prepended, so the sequence has 1 + 6 x 4 = 25 entries, and a two-layer transformer encoder reads them:

self.branches = nn.ModuleDict({
    m: BUILDERS[m](num_tokens=num_tokens, **bk.get(m, {}))
    for m in self.modalities})
self.projs = nn.ModuleDict({
    m: nn.Linear(self.branches[m].embed_dim, fusion_dim)
    for m in self.modalities})
self.aux_heads = nn.ModuleDict({
    m: nn.Linear(self.branches[m].embed_dim, n_classes)
    for m in self.modalities})
self.modality_emb = nn.Parameter(torch.zeros(len(self.modalities), fusion_dim))
self.segment_emb = nn.Parameter(torch.zeros(num_tokens, fusion_dim))
self.cls = nn.Parameter(torch.zeros(1, 1, fusion_dim))
layer = nn.TransformerEncoderLayer(
    fusion_dim, fusion_heads, fusion_dim * 2, dropout=fusion_dropout,
    batch_first=True, norm_first=True)
self.fusion = nn.TransformerEncoder(layer, fusion_layers, enable_nested_tensor=False)
self.fusion_head = nn.Linear(fusion_dim, n_classes)

The forward pass shows how missing data is handled. Each branch only runs on the rows of the batch where its modality exists, so zero-filled placeholders never touch BatchNorm statistics. Absent modalities are then excluded from attention through the key padding mask:

def forward(self, inputs, mask):
    b = mask.size(0)
    fusion_mask = self._dropped_mask(mask)
    tokens, aux = [], {}
    for j, m in enumerate(self.modalities):
        br = self.branches[m]
        t = inputs[m].new_zeros(b, br.num_tokens, br.embed_dim)
        present = mask[:, j] > 0
        if present.any():
            t[present] = br(inputs[m][present])
        aux[m] = self.aux_heads[m](t.mean(dim=1))
        tokens.append(self.projs[m](t)
                      + self.modality_emb[j]
                      + self.segment_emb)
    x = torch.cat(tokens, dim=1)
    pad = fusion_mask.repeat_interleave(self.num_tokens, dim=1) == 0
    x = torch.cat([self.cls.expand(b, -1, -1), x], dim=1)
    pad = torch.cat(
        [torch.zeros(b, 1, dtype=torch.bool, device=x.device), pad], dim=1)
    x = self.fusion(x, src_key_padding_mask=pad)
    return self.fusion_head(x[:, 0]), aux

Modality dropout

There is a catch. In the training set most clips have most modalities, but the test set has organically missing ones. A fusion layer trained only on complete inputs would learn to rely on its strongest sensor and collapse when it disappears. The remedy is modality dropout [13]: during training, each present modality is hidden from the fusion stage with probability 0.3, while at least one is always kept:

def _dropped_mask(self, mask):
    if not self.training or self.modality_dropout <= 0:
        return mask
    drop = (torch.rand_like(mask) < self.modality_dropout).float()
    kept = mask * (1.0 - drop)
    dead = kept.sum(dim=1) == 0
    kept[dead] = mask[dead]
    return kept

Only the fusion mask is changed. The branches still run on every present modality, so the auxiliary losses below keep receiving a signal from all of them.

Auxiliary heads

Every branch has its own linear classifier on its mean-pooled tokens. Their cross-entropy is added to the main loss with weight 0.3, restricted to the samples where the modality really exists. This keeps each branch individually discriminative, so the fusion transformer cannot lazily ignore a weak modality and let its backbone drift to useless features:

def aux_loss(aux, mask, y, loss_fn):
    total, count = 0.0, 0
    for j, m in enumerate(MODALITIES):
        pm = mask[:, j] > 0
        if pm.any():
            total = total + loss_fn(aux[m][pm], y[pm])
            count += 1
    return total / max(count, 1)

Training in three steps

The model is not trained end to end from random weights. Training happens in three stages, each with a script of its own.

Stage 1: every backbone alone. The script train_backbone_individual.py trains one modality at a time with a linear head on the mean of its tokens. It uses AdamW [14] with weight decay applied only to matrices, so biases and normalization parameters are not decayed, label smoothing of 0.1 [15], and a loss weighted by inverse class frequency:

counts = Counter(entry['label'] for entry in train_s)
weights = torch.tensor([1.0 / counts[c] for c in range(NUM_CLASSES)])
weights = weights / weights.sum() * NUM_CLASSES
loss_fn = nn.CrossEntropyLoss(weight=weights.to(DEVICE), label_smoothing=LABEL_SMOOTHING)

def build_param_groups(model, weight_decay):
    decay, no_decay = [], []
    for name, p in model.named_parameters():
        if not p.requires_grad:
            continue
        (no_decay if p.ndim < 2 else decay).append(p)
    return [
        {"params": decay, "weight_decay": weight_decay},
        {"params": no_decay, "weight_decay": 0.0},
    ]

The learning rate follows a linear warmup over the first 5% of steps and then a cosine decay to 1% of its peak, stepped per batch. The best checkpoint by validation accuracy is saved, with both the backbone and its head, so that the head can later serve as the auxiliary classifier.

After this stage, a small script reads all the saved histories and prints a table of size and accuracy per modality. It also flags runs that do not learn, defined as a best accuracy below twice the chance level of 1/40, and runs that overfit, defined as a final validation loss more than 1.5 above the training loss:

def status(h):
    best = h["best_val_acc"]
    if best < FAIL_FACTOR * CHANCE:
        return "NOT LEARNING"
    if h["val_loss"] and h["train_loss"]:
        gap = h["val_loss"][-1] - h["train_loss"][-1]
        if gap > 1.5:
            return "OVERFITTING"
    return "OK"

Stage 2: fusion with frozen backbones. The joint model loads every best.pt, freezes all parameters whose name starts with branches., and trains only the projections, the embeddings, the fusion transformer and the heads. Modality dropout is active here, which is the stage where robustness to missing inputs is learned.

Stage 3: joint fine-tuning. Finally everything is trained together, with the backbones at a learning rate thirty times smaller than the fusion layers:

backbone_params, fusion_params = [], []
for name, p in model.named_parameters():
    (backbone_params if name.startswith("branches.")
     else fusion_params).append(p)
if STAGE == "fusion":
    for p in backbone_params:
        p.requires_grad = False
    groups = [{"params": fusion_params, "lr": LR_FUSION}]
else:
    groups = [{"params": fusion_params, "lr": LR_FUSION},
              {"params": backbone_params, "lr": LR_BACKBONE}]
opt = torch.optim.AdamW(groups, weight_decay=WEIGHT_DECAY)

The learning rates are 3e-4 for the fusion layers and 1e-5 for the backbones. Total loss in both joint stages is the main cross-entropy plus 0.3 times the auxiliary term, with gradient clipping at 1.0. The reason for the staged recipe is practical. Backbones are easy to debug alone, a dead branch shows up immediately in the comparison table, and the fusion stage starts from features that are already discriminative instead of from noise.

Keeping an eye on the budget

Every script reports size using the same helper, which counts parameters in fp32:

def param_size_mb(module):
    return sum(p.numel() for p in module.parameters()) * 4 / 1e6

The joint model's size_report() prints the cost of every branch, including its projection and auxiliary head, and then the fusion stage and the total. This is the number to compare against the 100 MB limit. Since the auxiliary heads are only useful for training, they could be dropped from the submitted weights, which gives some extra room. Running the report is also the quickest way to see which modality is expensive for what it brings.

Limitations and what comes next

This post describes the architecture and not its leaderboard result. Validation accuracy comes from a user-held-out split of the training users, which is small, so differences of a point or two between variants should not be trusted. Some ideas also remain unexplored in the final pipeline. The graph-based skeleton backbone and its masked pretraining are not part of the joint model yet, the radar and IMU branches could use stronger augmentation, and the late-fusion design only combines modalities at the token level, so it cannot learn fine-grained cross-modal timing.

Conclusion

The core of this entry is a division of labor. Loaders remove subject-specific cues one modality at a time, small backbones turn each signal into a few tokens, and a tiny transformer learns how to combine whatever is available. The presence mask, the modality dropout and the auxiliary heads are what make the system tolerant to missing sensors, which is the main practical difference between this dataset and a clean benchmark.

The broader lesson is the same one that makes older detectors such as Viola-Jones fun to study: with a hard compute budget, the structure of the model matters as much as its size. Here the structure is a set of cheap, specialized encoders and a small, flexible mixer, and it fits the problem well.