⚠️ Partially solved, the search index isn't as large. Writeup documented in #fedwiki on element.io. Conversation 
I've decided I'm not the right person to solve this. I spent quite a lot of time trying to figure out the pearl and ruby scraper scripts but the problem was fixed while I was researching.
I'd like to contribute, but now isn't the right time.
There's some kind of photo gallery that Ward put together that he was trying to use as a sample data set for scaling search but I find it hard to follow.
I raised this under Projects after a suggestion by Ward on the back of Open Search for Fed Wiki.
There's a full search running on search.fed.wiki.org. Search
Ward suggests his whole federation search could be made faster and more friendly many ways. Issue#283
Ward has made incremental improvements to this project privately. Pulls