Truthfully, we have no idea if Retro has been working on a project and it has been scrapped to make way for Prime 4. For all we know, Retro is finished or is in the stages of finishing a project that Nintendo still might be ready to show as a surprise in a Direct real soon. Because of that, they may be in the best position to take on and start work on Prime 4 now because of that. Or maybe it is similar to when Nintendo bought Retro and then had them scrap all their other projects to just focus on Prime 1.
I think with all the speculation people had at the start of the year that there would be a new Direct in January and hopes of news on Prime 4, NoA probably conveyed to NoJ that they should get ahead of this delay and let people know now instead of more future freakouts at lack of news from fan's expectations and hopes. And while there will probably be some snarky takes about how Other M was released if Nintendo has such high standards for the franchise, I do take that as a good thing in that Nintendo is being a bit proactive with the Metroid community by expressing that they really do care about the franchise even if some of the worst elements of the fanbase have been overboard in their criticisms of other efforts like Other M and Federation Force to try new things in it. Nintendo is saying they recognize what fan expectations are and want a product to meet that. This could be the sort of Twilight Princess moment for the Metroid series where Nintendo went back to the Ocarina formula and realistic Link art style after the first responses to Wind Waker.
With Ian's point (and this is something that has been discussed over a decade ago on these forums) while Metroid gets a lot of critical praise, for some reason, the games just don't sell as well as other Nintendo franchises. Therefore, as a business, it makes sense to put more resources and releases into the series that will sell a higher volume before getting around to some of the other franchises that sell modestly. Sure, it would be great if Nintendo of Japan took on another Metroid game but if that delays Mario, Zelda, Animal Crossing, Splatoon now and heck, even Kirby then it is not a sound business decision. With the port of DKC: TF to the Switch then I believe the combined sales of it on Switch and Wii U has meant both of Retro's DKC releases have sold better than any of their Metroid Prime releases even if the MP games are probably the stronger games overall. It was you that talked before about voicing your opinion through your wallet and what you purchase. Based on sales, the people have spoken and Nintendo is listening to that opinion by focusing a bit less on that franchise. That's why the idea of voting with one's wallet has always been flawed.