Ultimately the strength of a material comes down to the strength with which two neighbouring atoms are stuck together. The electrical attraction between carbon atoms is very strong and that is the source of strength for both graphene and diamond.
However, most materials are hundreds of times weaker than the bonds between atoms would lead people to expect.
The explanation of this has to do with ‘defects’ in the materials where the perfect atomic arrangements are disrupted. Defects can dramatically weaken a material. When stressed, the material breaks at the defect – colloquially a chain breaks at its weakest link.
In a material this initial break enlarges the defect. The stress is then passed on to neighbouring atoms which break too because of the enlarged defect. Think about how tough the cellophane covering is on a CD – but how once a defect is introduced – it rips easily.
However, the strength of a single atom-thick layer of graphene isn’t available to build big things – because when you put many sheets of graphene together you get graphite, which we all know is ‘crumbly’ – it’s what pencil lead is made from.