Levi/Matthew the public tax collector was one of the original twelve. What's interesting is among the twelve he was likely the least liked. I wonder what went through the others' minds when Jesus said to Matthew: "Follow me!" The Word of God (The Bible) is silent about it, perhaps it's because God wants us to focus what was said and recorded.
Tax collectors in those times were hated and probably the only thing worse than that was a Samaritan (a half-breed). I wonder how the other disciple felt initially about the inclusion of Matthew into their midst. I figure that they were plenty uncomfortable about it. On the top of having a publican added to them now they had to go to his place for dinner. They likely very uncomfortable at the dinner... cause there was all this low life sinners around them. I'm guessing that when Jesus responded to the Pharisees, He was not just addressing them but also the disciples who were silent.
It is also interesting to note that, in both Matthew and Luke's accounts, this event takes place BEFORE Jesus selected the twelve. This would mean then that the disciples referred to in Luke 5 and Matthew 9 were not just the twelve but a whole bunch of people. Perhaps the incident at Matthew's banquet had something to do with how and who Jesus eventually choose as the twelve.
Was Jesus also watching how the disciples treated Matthew and the others? Did the other disciples shun him or avoid talking to him. How did they respond to the other "rift-raft" of society at that banquet?
Whatever the case, Matthew a flop and outcast of society, was called and chosen by Jesus to be one of the twelve. Matthew who was likely shun by his family and peers was asked to be part of what would be the world-changing movement.