As a non colour scientist I’d say “What colour’s that black cat there?” to which someone would answer “Black”. So black’s a label as valid as any other colour label.
Scientifically, ‘colourfulness’ scales how colourful something is. Zero colourfulness things include white, grey and black. Though the ‘neutrality’ of whites and greys is rather tricky – often the most neutral thing visible wil be judged as being neutral, so it’s a self calibrating relative white point thing rather than an absolute.
In as much as there are many ‘acceptable’ whites, there’s a volume (was colour is a 3D thing) of colours one could describe as black, with the blackest black being the one which emits or reflects no light at all, and a purist might say that’s the only black the original question was all about. Then we’re at the hub of this perceptual and philosophical territory.
Looking at analogues with other senses and logic, is total silence a sound? Is zero a number? Your answer lies somewhere between these two.