It does matter, but not as much as people often suggest. Even if the Suffering Servant is Israel or a remnant within the nation, there would be overwhelming odds against Jesus' life aligning as closely with the passage as it does. Jesus' fulfillment could be typological, yet still be highly evidential.
I don't think the passage is referring to Israel, a remnant, or any other group. It's referring to an individual. We know that Isaiah was expecting an individual Davidic Messiah, as other ancient Jews were (9:6-7). Prior to the Suffering Servant prophecy, there's reference to a Servant who's distinct from Israel and distinct from a Jewish remnant (49:5-6). The Suffering Servant passage uses individual language ("he", etc.), so taking it as referring to an individual is the most natural way of reading it. The reference to the Servant's being "high and lifted up and greatly exalted" (52:13) is language used to describe God elsewhere in Isaiah (6:1, 33:10, 57:15), so 52:13 probably identifies the Servant as God incarnate, much like 9:6-7. God is an individual being, not a group, so the opening verse of the prophecy (52:13) supports an individual interpretation. The prophecy goes on to distinguish the Servant from Israel (53:8). And the individual view was widespread in ancient Israel, including among non-Christian Jews. You can read a more extensive argument for the individual reading and Jesus' fulfillment of it in Darrell Bock and Mitch Glaser, edd., The Gospel According To Isaiah 53 (Grand Rapids, Michigan: Kregel, 2012).
But even if the passage weren't referring to an individual, Jesus' fulfillment of it would carry a lot of evidential weight. No other individual's life has even come close to aligning with the passage, whether typologically or non-typologically. It's not as though it would be easy to make yourself be widely perceived as God incarnate (52:13), sinless (53:9), making atonement for the sins of others (53:6), and risen from the dead (53:10), for example. The idea that Jesus was delusional or dishonest enough to try to fulfill the prophecy by naturalistic means is unlikely. But even if he had attempted it, there would have been factors he wouldn't have had much control over. The transformation of the Gentile world, including Gentile rulers, described in 52:13-5 is something that's occurred since Jesus' death. And some of the people who reported seeing him risen from the dead (53:10) were enemies (James, Saul of Tarsus) whose experiences occurred after Jesus' death, of course. Jesus didn't have much of an ability to orchestrate such factors in order to make himself seem to have fulfilled Isaiah's prophecy. If Jesus had attempted such a naturalistic fulfillment, how would Isaiah have known that there would be such an individual, who would not only be willing to try to fulfill such an unusual and difficult prophecy, but would even be so successful in doing it? How many people other than Jesus have even attempted a fulfillment? The lack of any other individual who even comes close to fulfilling the passage is a reflection of the unlikelihood of a fulfillment by naturalistic means.
When people argue that the Suffering Servant is some entity other than Jesus, our first response shouldn't be to argue for an individual Messianic interpretation of the passage. Rather, our first response should be to ask how they explain Jesus' remarkable typological fulfillment of the passage under such a scenario. Then, after explaining that Jesus' fulfillment of the passage is highly evidential either way, we can go on to argue for an individual interpretation.