Scholars evaluated hadiths by asking two major questions. One question is about the chain of narrators and the other question is about the text that is being related.
Analysis of chain of narrators:
A chain of narrators looks like this: Person A says that he heard from person B, who heard from person C, who heard from person D, that the Prophet (peace be upon him) said XYZ. To evaluate the chain, scholars would ask about persons A, B, C and D.
First, is the person known to be trustworthy and morally upright?
There were what is referred to in the Quran as munafiqun, hypocrites, who lived in the community but were not good Muslims. Technically, they weren’t even Muslim, because though they professed the faith, inwardly they were not really Muslims. That's not for us to judge, but that's the Quranic judgment on them. The hadith scholars had to take the necessary steps to evaluate them. They felt uneasy about this at first, because it would mean that they were trying to judge individuals, and how can one know what's in their heart? But for the sake of evaluating the hadith, they chose to take from those whose piety and devotion to the faith were well-established. They left off those they were unsure about. They wrote biographical dictionaries with lists of hadith narrators. They asked, what do other evaluators say about this person? Somebody may say that this person is honest, another may say he is trustworthy, and yet another may say that he's a liar, or maybe worse than a liar, he's Dajjal, which means something like the anti-Christ. So looking at these evaluative snippets that have been given by other experts in the field, a person like Al-Imam Bukhari would then be confident in including some narrators and excluding some others. That's about the honesty of the unreliability of the narrator as a person himself, given his trustworthiness as an individual.
Second, what about his trustworthiness in terms of relating hadiths? Because he could be a good individual, righteous, pious and honest, but maybe he doesn't have a good memory. And this too was noted by the scholars. They looked at a teacher having, let's say, six students. If five of them narrate a certain saying one way, and the sixth one narrates it differently, they would know he was the odd man out. He perhaps misremembered and maybe his memory was faulty. He was considered a less reliable narrator, not because of anything about his person, but because of his human failing.
Third, what about the chain of narrators itself? Even if each individual by himself is a good and upright person with good memory, the scholars would ask whether they actually met each other? One person may say I got it from that person and person A says I got it from B, but maybe between A and B there is another person who was not mentioned. Maybe A didn't really get it from B, maybe A got it from A prime, who got it from B. This would be noted as a lacuna in that chain of narrators, as scholars wouldn’t be able to tell who that person was and would not be able to evaluate the chain. So they’d leave the hadith aside.
Evaluating the chain of narrators was a very thorough and painstaking process. A lot of work has gone into that, so we must admire the scholars who did this great work.
Hadith scholars also looked at the text of the narrative itself. They asked, does this seem congruent with what the Prophet (peace be upon him) might've said? Or could it be that one of the enemies of Islam invented the saying for their own purposes and foisted it on unsuspecting Muslims? Was the saying an anomaly, which they refer to as a shaadh, or is there some islaah (cause or a reason) for rejecting it? There could be something in the narrative itself which conveyed to the hadith collector that it was not a genuine narrative from the Prophet, (peace be upon him).
In theory, the textual analysis should have been rigorous. Aisha, the mother of the believers, would sometimes hear the text of the narrative and indicate that it didn’t sound right. Sometimes she would compare it with the Quran and she would say the Quran says one thing and people are narrating almost the opposite of what the Quran says. And she would say they have misremembered, because the Quran says this and it couldn't be that the Prophet said the other thing. So this practice of evaluating the text was there from the inception.
But eventually what happened in Islamic history is that people started to move away from using reason towards a reliance on narrative. If two people were having a discussion about what to do, and one quoted from the Quran, that would resolve the discussion immediately. Because as Muslims, we are required to submit to what the Quran says. A similar thing happened with the introduction of hadith. As soon as someone said, you know, the Prophet (peace be upon him), said this is what we should do, that settled the matter. The more hadiths came to be introduced into public discussions, the less Muslims had scope to use reason to think about how to do things. Hadiths tended to take over. If these were all genuine narratives from the Prophet (peace be upon him), we would have felt comforted by the fact that we are doing the right thing by following the hadith. But a lot has been forged, as we already mentioned, and when hadith scholars were doing this evaluation, they were already in the mood to just simply take the hadith without scrutinizing the texts so much, especially as many generations had passed and there were a lot of hadiths to deal with.
In addition, there was also the sense that they wanted hadiths to speak about the wide scope of Muslim activity in all areas. A desire arose among Muslims in this period; there were differences of opinions, and to settle the issue, it seemed best to find hadiths on the subject. If no one come forward with the hadith, they would go out searching for the hadith. And in the midst of that query, people might have been in the mood to come forward to settle the issue with a hadith of their own invention, not necessarily with nefarious purposes, but because they wanted to be of service.
There are six books of hadiths which are generally referred to as the sihah sitta. Among them is the collection of Al-Imam Ibn Majah. Ibn Majah's collection has been criticized by many scholars. Dr. Israr Ahmad Khan, in his recent book, "Authentication of Hadith: Revaluating the Criteria," noted that Ibn Majah's collection contains many fabricated hadiths. Among the others you might find hadiths which are on a spectrum from being authentic on the one hand to being weak on the other hand. Hardly does anyone say that there were fabrications in the others, partly because we respect the books. But careful scholars may weed out one or two because the chain of narrators is faulty. Yet there remains a lot of hesitation in using reason to evaluate the text of the narratives, and because reason has not been used as much as we would like, even though in principle it was always acknowledged as a valid method of evaluating the hadith, we have hadiths in the major collections which have given cause for pause to some of our greatest scholars. We spoke about the different factions inventing things. We have a hadith which speaks about twelve imams, and that's in the Sunni collections!