Learning Structural Regularities over Mobile UI Flows from Interaction Traces

Abstract: Mobile applications exhibit rich user interface (UI) flows composed of sequences of screens connected through user interactions. While prior work has made significant progress in understanding individual UI screens, reusable flow-level regularities across applications remain underexplored. In this study, we learn a role-based probabilistic prior over mobile UI flows from large-scale interaction traces by mapping each screen to an abstract screen role via unsupervised clustering of multimodal screen embeddings derived from screenshots and view-hierarchy text. Interaction traces are then converted into sequences of screen roles, enabling probabilistic modeling of flow structure beyond app-specific identifiers and layouts. We study two complementary tasks. First, for next-step prediction on unseen applications and held-out categories, simple n-gram baselines already capture meaningful cross-app regularities, and a causal Transformer further improves performance, achieving R@1 = 0.2598 and R@10 = 0.7268 on unseen test applications at K = 40 while outper-forming the trigram baseline across all reported cutoffs. Second, we study atypical-transition scoring under a fixed sampled-candidate protocol, where the same learned flow prior is used to rank observed transitions against sampled alternatives. Under this controlled setting, the Transformer achieves the strongest ranking performance among the compared models. These results indicate that role-based abstractions provide a promising basis for modeling UI flow regularities across applications and for ranking-oriented sequence prediction under cross-application generalization. The anomaly results are encouraging under the sampled evaluation protocol, but broader validation will require more realistic evaluation settings and structured human assessment.

Iskandar Salama and Masayasu Atsumi. “Learning Structural Regularities over Mobile UI Flows from Interaction Traces”. International Journal of Advanced Computer Science and Applications (IJACSA) 17.4 (2026). http://dx.doi.org/10.14569/IJACSA.2026.0170483

@article{Salama2026,
title = {Learning Structural Regularities over Mobile UI Flows from Interaction Traces},
journal = {International Journal of Advanced Computer Science and Applications},
doi = {10.14569/IJACSA.2026.0170483},
url = {http://dx.doi.org/10.14569/IJACSA.2026.0170483},
year = {2026},
publisher = {The Science and Information Organization},
volume = {17},
number = {4},
author = {Iskandar Salama and Masayasu Atsumi}
}