The Scaling Lesson. What are we missing?

Consider the difference between GPT 1 and GPT 6. They are both decoder-only language models, yet there's so much difference between them. Data scale, data quality, sft, rl, etc. But the important lesson, I think, is that the preliminary capabilities of a specific manifestation of a machine learning architecture should not be viewed as the absolute limit of that architecture's potential, seeing as we went from 'autocomplete on steroids...