In this paper, we note that despite being a multimodal phenomenon, turn-taking has still been investigated mostly as being unimodal. Based on theoretical positions emphasizing that communication is organized jointly by interaction partners, we identify the challenge of assessing human sequential behavior: 1) spread across different modalities and 2) co-constructed with a partner. By analyzing a corpus of mother-child dyads with cross recurrence quantification analysis and frequent pattern mining, we offer novel steps toward understanding multimodal turn-taking.