I prefer a non linear game, a place where my decision at least seem to matter, where I can roam freely no matter how "urgent" the calls of the main quest line seem to be (you soon enough get it, that this "urgency" is only there for dramatic reasons).

Think that also is exactly what the devs had in mind: handing out a game, almost open world, where the player can play every way she/he wants.

Funny enough, this includes even you (side quests must be a nightmare for you) :)!

I am playing from an end game save so I've no idea if/what the explanation given when you start it without the choices carrying over, but I'd say it is detached enough just take it as a side story separate from Wild Hunt.

But In an attempt to answer your question, depending on where you are you can justify it as Geralt has hit a dead end in his search for Ciri Vellen. He hasn't money (a common problem for Geralt in universe, not necessarily in game) for the only boat to Skellige, so he takes the only job available, this leads him to Toussaint. From our knowledge of how it all plays out after that we (the Player) know that Ciri certainly isn't going anywhere or in much danger.

This is of course not a great explanation (The war affecting travel), but it is the sort of what I kept falling back on a lot when doing some of the long side quest in the main game.