If your potatoes stay small, there can be several explanations for the plant's beautiful vegetation. Assuming that you have carried out the cultivation correctly and that you have harvested at the right time, I make two hypotheses that can explain the poor size of the tubers.
Small potatoes for nutrients
You guessed the first explanation yourself: it lies in the nutrients available to the plant. The main elements that allow the life and development of crops are three: nitrogen, phosphorus and potassium. Among these, nitrogen is a great stimulator of leaf growth, while potassium is useful for the formation of the tuber. Given your results I would say that it could be an imbalance in the presence of the elements. There may be an excess of nitrogen that has "diverted" the resources to the aerial part of the plant, to the detriment of the potatoes. There may be a shortage of potassium which has made it lack the necessary material to swell the tubers. Third possibility you could have an alkaline soil (pH too high), which leads to difficulties in the absorption of some substances by the plant, in this case the potassium is therefore there but your potatoes are unable to take it.
In this case I recommend that you first measure the pH of the soil, secondly you will have to be careful when fertilizing (avoiding excess manure or liquid fertilizers for example). If the pH is low, you can add some ash which brings potassium but not nitrogen.
Small potatoes because of the soil
Second explanation for the failure of your vegetables to grow is the soil. A clayey and compact soil hinders the development of the tubers, which remain small and sometimes even deformed. The solution in this case is to dig more, incorporate organic substance into the soil and possibly also river sand, hoe it often, cultivate in raised stalks.