Gnomon:  The short answer is that in a figure such as the one below, the green area ADCFGE is a gnomon.  The green "L" is a gnomon.  But there is much more to know about this word that has puzzled many readers. 

The thirteen Euclid's Elements, perhaps the best selling mathematics book in the world, defines gnomon at the beginning of Book II.  (The translation below, by Sir Thomas L. Heath, had not been published when Joyce, or the boy in the story, was in school.  Heath's 1908 translation supplanted one made by Simson, which Joyce surely did see, for it was widely used in schools.  The concepts in both books are identical; I've used the more modern translation because it's easier for us to follow.)

Definition 1.  Any rectangular parallelogram is said to be contained by the two straight lines containing the right angle.

Definition 2.  And in any parallelogrammic area let any one whatever of the parallelograms about its diameter with the two complements be called a gnomon.

Exactly what this means is not obvious to the non-mathmatician and has confused literary scholars. The gnomon, Heath explains, is the figure which remains of a square when a smaller square is cut out of one corner.

In the diagram below, the gnomon, by this classic definition, is the green L-shape: AEGFCD.

• Another way to describe it is to say, as Aristotle does, a gnomon is the figure which when added to a square increases its size but does not alter its form.
• An example: to the yellow square below, the green gnomon (L shaped piece) can be added to make another similar square. 

There is more to say, however. about the gnomon.