Line a large rimmed baking sheet with parchment paper.
Roll out puff pastry, one block at a time, on a lightly floured surface to make a 12 x 10 inch rectangle. Repeat with the second block. Cut each piece into eight 5×3 inch rectangles and refrigerate for 15 minutes.
Arrange 8 of the pieces onto the prepared baking sheet. Divide the jam into the center of each pastry. Top each with the remaining pastry pieces and seal the edges with the tines of a fork.
Transfer the tray to the freezer and freeze for 10 minutes.
In small bowl, whisk egg yolk with milk. Remove the pastries from the freezer and brush egg wash over each pastry and sprinkle with sugar.
Bake in the center of a 375F (190C) oven until pastries are puffed and golden, 20 to 25 minutes. Allow to cool on the baking sheet for 10 minutes and transfer to a wire rack.
Sift sugar into a medium bowl. Gradually whick in milk and vanilla to make a smooth icing. Drizzle over cooled pop tarts.