Forum:Template Configuration Changes Coming

Hello,

In order to improve site performance and guarantee site health, Wikia has decided to lower the maximum number of templates that can be transcluded into a page. Even if the templates are fairly straight forward, pulling them into a page can still be a technically costly process when the number of transcluded templates reaches the hundreds of thousands. Wikia is thus setting the limit at 300,000 template transclusions, a number that 99.99% of pages on our network fall below.

That said, some of the pages that are above 300,000 transclusions exist on this particular wiki. Once this limit is changed, pages which are "above" the limit will display an error on the page where the 300,001st template call is made.

General suggestions for effectively curtailing the number of templates on these pages is to divide the page into appropriate subpages or possibly to combine a number of templates together instead of reusing the same template over and over. Another common issue we see is “nested” templates (templates within templates within templates). Merging some of the code into a single, “flat” template will dramatically decrease the number of transclusions.

If you would like specific advice regarding any of these pages, feel free to leave a reply here and we will look into it further.

Here are the pages on your wiki that have template transclusions over the forthcoming limit:


 * http://onepiece.wikia.com/?curid=1572
 * http://onepiece.wikia.com/?curid=3067
 * http://onepiece.wikia.com/?curid=3210
 * http://onepiece.wikia.com/?curid=3212
 * http://onepiece.wikia.com/?curid=4004
 * http://onepiece.wikia.com/?curid=4468
 * http://onepiece.wikia.com/?curid=4611
 * http://onepiece.wikia.com/?curid=4612
 * http://onepiece.wikia.com/?curid=4613
 * http://onepiece.wikia.com/?curid=4862
 * http://onepiece.wikia.com/?curid=5014
 * http://onepiece.wikia.com/?curid=5085
 * http://onepiece.wikia.com/?curid=5087
 * http://onepiece.wikia.com/?curid=5089
 * http://onepiece.wikia.com/?curid=5092
 * http://onepiece.wikia.com/?curid=5133
 * http://onepiece.wikia.com/?curid=5185
 * http://onepiece.wikia.com/?curid=5186
 * http://onepiece.wikia.com/?curid=6277
 * http://onepiece.wikia.com/?curid=6639
 * http://onepiece.wikia.com/?curid=7045
 * http://onepiece.wikia.com/?curid=7117
 * http://onepiece.wikia.com/?curid=7516
 * http://onepiece.wikia.com/?curid=7517
 * http://onepiece.wikia.com/?curid=7710
 * http://onepiece.wikia.com/?curid=7813
 * http://onepiece.wikia.com/?curid=8122
 * http://onepiece.wikia.com/?curid=8272
 * http://onepiece.wikia.com/?curid=8625
 * http://onepiece.wikia.com/?curid=8626
 * http://onepiece.wikia.com/?curid=8627
 * http://onepiece.wikia.com/?curid=8628
 * http://onepiece.wikia.com/?curid=8629
 * http://onepiece.wikia.com/?curid=8631
 * http://onepiece.wikia.com/?curid=8870
 * http://onepiece.wikia.com/?curid=11212
 * http://onepiece.wikia.com/?curid=11213
 * http://onepiece.wikia.com/?curid=11232
 * http://onepiece.wikia.com/?curid=11294
 * http://onepiece.wikia.com/?curid=11325
 * http://onepiece.wikia.com/?curid=11327
 * http://onepiece.wikia.com/?curid=11328
 * http://onepiece.wikia.com/?curid=11329
 * http://onepiece.wikia.com/?curid=11331
 * http://onepiece.wikia.com/?curid=11332
 * http://onepiece.wikia.com/?curid=11333
 * http://onepiece.wikia.com/?curid=11334
 * http://onepiece.wikia.com/?curid=11335
 * http://onepiece.wikia.com/?curid=11337
 * http://onepiece.wikia.com/?curid=11342
 * http://onepiece.wikia.com/?curid=11391
 * http://onepiece.wikia.com/?curid=11392
 * http://onepiece.wikia.com/?curid=11393
 * http://onepiece.wikia.com/?curid=11394
 * http://onepiece.wikia.com/?curid=11395
 * http://onepiece.wikia.com/?curid=11396
 * http://onepiece.wikia.com/?curid=11397
 * http://onepiece.wikia.com/?curid=11399
 * http://onepiece.wikia.com/?curid=11400
 * http://onepiece.wikia.com/?curid=11401
 * http://onepiece.wikia.com/?curid=11431
 * http://onepiece.wikia.com/?curid=11459
 * http://onepiece.wikia.com/?curid=11460
 * http://onepiece.wikia.com/?curid=11469
 * http://onepiece.wikia.com/?curid=11470
 * http://onepiece.wikia.com/?curid=11471
 * http://onepiece.wikia.com/?curid=11527
 * http://onepiece.wikia.com/?curid=11528
 * http://onepiece.wikia.com/?curid=11529
 * http://onepiece.wikia.com/?curid=11530
 * http://onepiece.wikia.com/?curid=11535
 * http://onepiece.wikia.com/?curid=11536
 * http://onepiece.wikia.com/?curid=11546
 * http://onepiece.wikia.com/?curid=11558
 * http://onepiece.wikia.com/?curid=11559
 * http://onepiece.wikia.com/?curid=11567
 * http://onepiece.wikia.com/?curid=11568
 * http://onepiece.wikia.com/?curid=11569
 * http://onepiece.wikia.com/?curid=11570
 * http://onepiece.wikia.com/?curid=11571
 * http://onepiece.wikia.com/?curid=11572
 * http://onepiece.wikia.com/?curid=11573
 * http://onepiece.wikia.com/?curid=11575
 * http://onepiece.wikia.com/?curid=11576
 * http://onepiece.wikia.com/?curid=11595
 * http://onepiece.wikia.com/?curid=11597
 * http://onepiece.wikia.com/?curid=11602
 * http://onepiece.wikia.com/?curid=11619
 * http://onepiece.wikia.com/?curid=11620
 * http://onepiece.wikia.com/?curid=11621
 * http://onepiece.wikia.com/?curid=14802
 * http://onepiece.wikia.com/?curid=16514
 * http://onepiece.wikia.com/?curid=17166
 * http://onepiece.wikia.com/?curid=17419
 * http://onepiece.wikia.com/?curid=17461
 * http://onepiece.wikia.com/?curid=17462
 * http://onepiece.wikia.com/?curid=17463
 * http://onepiece.wikia.com/?curid=17464
 * http://onepiece.wikia.com/?curid=17465
 * http://onepiece.wikia.com/?curid=17466
 * http://onepiece.wikia.com/?curid=17540
 * http://onepiece.wikia.com/?curid=17543
 * http://onepiece.wikia.com/?curid=17544
 * http://onepiece.wikia.com/?curid=17545
 * http://onepiece.wikia.com/?curid=17546
 * http://onepiece.wikia.com/?curid=17547
 * http://onepiece.wikia.com/?curid=17589
 * http://onepiece.wikia.com/?curid=17593
 * http://onepiece.wikia.com/?curid=17594
 * http://onepiece.wikia.com/?curid=17595
 * http://onepiece.wikia.com/?curid=17596
 * http://onepiece.wikia.com/?curid=17597
 * http://onepiece.wikia.com/?curid=17598
 * http://onepiece.wikia.com/?curid=17599
 * http://onepiece.wikia.com/?curid=17600
 * http://onepiece.wikia.com/?curid=17601
 * http://onepiece.wikia.com/?curid=17602
 * http://onepiece.wikia.com/?curid=17603
 * http://onepiece.wikia.com/?curid=17604
 * http://onepiece.wikia.com/?curid=17606
 * http://onepiece.wikia.com/?curid=17608
 * http://onepiece.wikia.com/?curid=17615
 * http://onepiece.wikia.com/?curid=17616
 * http://onepiece.wikia.com/?curid=17617
 * http://onepiece.wikia.com/?curid=17618
 * http://onepiece.wikia.com/?curid=17619
 * http://onepiece.wikia.com/?curid=17642
 * http://onepiece.wikia.com/?curid=17643
 * http://onepiece.wikia.com/?curid=17645
 * http://onepiece.wikia.com/?curid=17646
 * http://onepiece.wikia.com/?curid=17647
 * http://onepiece.wikia.com/?curid=17648
 * http://onepiece.wikia.com/?curid=17649
 * http://onepiece.wikia.com/?curid=17650
 * http://onepiece.wikia.com/?curid=21316
 * http://onepiece.wikia.com/?curid=23091
 * http://onepiece.wikia.com/?curid=23640
 * http://onepiece.wikia.com/?curid=23652
 * http://onepiece.wikia.com/?curid=23653
 * http://onepiece.wikia.com/?curid=36648
 * http://onepiece.wikia.com/?curid=37644
 * http://onepiece.wikia.com/?curid=71386
 * http://onepiece.wikia.com/?curid=79796
 * http://onepiece.wikia.com/?curid=86381
 * http://onepiece.wikia.com/?curid=90691
 * http://onepiece.wikia.com/?curid=113926
 * http://onepiece.wikia.com/?curid=142705
 * http://onepiece.wikia.com/?curid=163426
 * http://onepiece.wikia.com/?curid=170422
 * http://onepiece.wikia.com/?curid=174262
 * http://onepiece.wikia.com/?curid=174263
 * http://onepiece.wikia.com/?curid=174526
 * http://onepiece.wikia.com/?curid=176693
 * http://onepiece.wikia.com/?curid=177008
 * http://onepiece.wikia.com/?curid=177816
 * http://onepiece.wikia.com/?curid=177817
 * http://onepiece.wikia.com/?curid=180765
 * http://onepiece.wikia.com/?curid=180928
 * http://onepiece.wikia.com/?curid=180929
 * http://onepiece.wikia.com/?curid=184434
 * http://onepiece.wikia.com/?curid=185065
 * http://onepiece.wikia.com/?curid=189218
 * http://onepiece.wikia.com/?curid=189900
 * http://onepiece.wikia.com/?curid=190748
 * http://onepiece.wikia.com/?curid=194324
 * http://onepiece.wikia.com/?curid=196338
 * http://onepiece.wikia.com/?curid=200571
 * http://onepiece.wikia.com/?curid=201094
 * http://onepiece.wikia.com/?curid=201095
 * http://onepiece.wikia.com/?curid=205677
 * http://onepiece.wikia.com/?curid=212110
 * http://onepiece.wikia.com/?curid=212111
 * http://onepiece.wikia.com/?curid=213252
 * http://onepiece.wikia.com/?curid=215911
 * http://onepiece.wikia.com/?curid=216093
 * http://onepiece.wikia.com/?curid=216094
 * http://onepiece.wikia.com/?curid=219675
 * http://onepiece.wikia.com/?curid=221625
 * http://onepiece.wikia.com/?curid=221626
 * http://onepiece.wikia.com/?curid=222201
 * http://onepiece.wikia.com/?curid=222202
 * http://onepiece.wikia.com/?curid=225111
 * http://onepiece.wikia.com/?curid=225449
 * http://onepiece.wikia.com/?curid=225450
 * http://onepiece.wikia.com/?curid=225451
 * http://onepiece.wikia.com/?curid=228924
 * http://onepiece.wikia.com/?curid=229058
 * http://onepiece.wikia.com/?curid=229338
 * http://onepiece.wikia.com/?curid=229339
 * http://onepiece.wikia.com/?curid=229378
 * http://onepiece.wikia.com/?curid=231456
 * http://onepiece.wikia.com/?curid=231538
 * http://onepiece.wikia.com/?curid=231539
 * http://onepiece.wikia.com/?curid=231540
 * http://onepiece.wikia.com/?curid=231573
 * http://onepiece.wikia.com/?curid=231575
 * http://onepiece.wikia.com/?curid=234289
 * http://onepiece.wikia.com/?curid=234606
 * http://onepiece.wikia.com/?curid=234607
 * http://onepiece.wikia.com/?curid=236516
 * http://onepiece.wikia.com/?curid=236517

--DaNASCAT (help forum | blog) 22:02, January 13, 2014 (UTC)

Discussion
We want to opt out. SeaTerror (talk) 23:48, January 13, 2014 (UTC)

i dont know whether or not we have a choice or not but if we do i agree with st-- 23:52, January 13, 2014 (UTC)

To me it looks like this is just an automated or copy/pasted message, so don't expect a fast response from staff. And judging by the wording of it, I don't think we have a choice in the matter. What we need to do first is find out where the 300,000 transcluded templates are in each of the pages, then work on lessening the number of templates. I've obviously not checked every single linked page, but those that I have checked have all been Episode/OVA pages. Perhaps most of the transclusions lie in the Episode box or the similar templates used on all of those pages? If it comes down to it, we might just have to completely redesign some of our templates from the ground up in order to avoid having so many transclusions. I'm no code genius, so I think we'll need the input of those who actually know what they're talking about before we can really figure out how to deal with this. 00:27, January 14, 2014 (UTC)

I really don't know what's this all about so I'll back off

Joekido (talk) 00:45, January 14, 2014 (UTC)

Did I misunderstood or did he just said that Episode Special 4 has more then 300,000 template transclusions? (included nested and same template calls) How is that possible? And how can you count the template transclusions?

This comment doesn't seems to tell that number...

  &lt;p&gt;Episode 288 &lt;/p&gt; &lt;!-- NewPP limit report Preprocessor node count: 1/1000000 Post‐expand include size: 0/2097152 bytes Template argument size: 0/2097152 bytes Expensive parser function count: 0/100 --&gt;

yeah,we dont see any template transclusion numbers when parsed and the ' alltransclusions' parameter is unrecognized ;(--

Levi is VSTF so he can see them I assume. SeaTerror (talk) 16:44, January 14, 2014 (UTC)

He just said he cant -_---

That has nothing to do with user rights I think, the comment I posted is taken from the page's HTML source code, everybody can check it (like roranoa did). I will question the staff about this though.

I'm not exactly an expert at this stuff, but is this the problem?
 * Episode Box: Uses several templates, including 5 calls to WhichCorresponding
 * WhichCorresponding makes 700 calls to WhichCorresponding/format
 * WhichCorresponding/format makes 24 calls to WhichCorresponding/returner
 * WhichCorresponding/returner makes 4 calls to WhichCorresponding/pageChecker
 * WhichCorresponding/pageChecker makes 2 calls to ChapterPages

If those applied multiplicatively (I'm not sure whether they do or not), it's over 600,000 from just that chain, meaning we'd just have to change the base WhichCorresponding template so it doesn't make 700 calls. 17:01, January 14, 2014 (UTC)

Zodiaque is most likely right. WhichCorresponding is really hardcore. Did someone check whether one of those links are not episode pages?

Something is wrong with our episode pages and they take FOREVER to load. If that wasn't an issue I would have checked them all. SeaTerror (talk) 17:44, January 14, 2014 (UTC)

I'm in the process of checking them all now. At the moment, all are episodes or episode pages, except for We Are!. I'll update this post once I've finished, but like ST said, the episode pages are taking a long time to load. 17:49, January 14, 2014 (UTC)

It's been like that for months now. That's why I rarely edit them right now. Maybe the transclusions is what causes all the lag. On that note chapters also lag at first to load but still load pretty fast. SeaTerror (talk) 18:08, January 14, 2014 (UTC)

Just finished checking the list. All all are episode or episode special pages, including things like Glorious Island, Kyutai Panic Adventure!, etc. And Template:Episode Box itself, but I think we're not not surprised that's over the limit. The only page that doesn't use the Episode box template is We Are!, which I assume means that page needs special attention in order to fix it. And if making these changes does improve the episode pages load speed, we might want to try to apply them to the chapter box too, even though those are under the limit now. 18:29, January 14, 2014 (UTC)

Well, everything sums up quite nicely… By the way, the "We Are" page also use the incriminated template. Anyway, here are some technical details about the problem, for the template-savvy: instead of testing the value of the chapter and writing things accordingly, WhichCorresponding calls its subtemplate for all 700+ possible values, only one of these calls actually writing something. The 'if' statements have to be in the main template, not in the subtemplate(s). This would be orders of magnitude faster/lighter.

Yeah, the "Which..." templates are the problem. We have to rework them if necessary.

I've now removed the use of Template:WhichCorresponding/returner and Template:WhichCorresponding/pageChecker, this has made the code in Template:WhichCorresponding/format incomprehensible though, so I'm keeping all the comprehensible code in a java script that makes it easier to customize the template. It's accessible here: https://www.dropbox.com/s/r84sk8jqj7nc88c/whichCorresponding.java

As you may have noticed, Template:WhichCorresponding calls Template:WhichCorresponding/format once per chapter, this is because it has to look up every instance (if you request a list of chapters) of an episode that a chapter contains, which makes it go through all the calls to Template:WhichCorresponding/format just to make sure no chapters are missing from the list. I don't think there's another way to do it if you want it to look comprehensible and editable and also so you don't have to list both episodes/chapter and what they correspond.

Anyway, you could start by looking how many tranclusions it has now that I've removed the sub-sub-templates.

"...this has made the code in Template:WhichCorresponding/format incomprehensible though" - can you try to use spaces, new lines and/or comments? That way you can actually make it comprehensible. Formatting the code well makes it neat and easy to understand, maybe I'll take a look in to it later.

Well, I barely know myself how it works now, as I originally had it done to be used with 3 templates. All I did was put the code from the templates in the java script and print it from there, making this mess.

I could remove /format entirely but that would create at least one /format code-piece per chapter, that's 700*32,058B = 22,440,600B or 22MB of text. To put that in perspective, the biggest article (Monkey D. Luffy/History) was at 287,724B or 287kB before the split. I don't think that would be the greatest solution.

Just do everything manually again and it's fixed. SeaTerror (talk) 05:05, January 15, 2014 (UTC)

We_Are! This is another article that too a very long time to load. SeaTerror (talk) 19:50, January 15, 2014 (UTC)

To check which pages should be worked on, use Special:ParserSpeed. The "node count field" is the number of transclusions. Only admins can use that special page though...

But you're VSTF. SeaTerror (talk) 21:25, January 16, 2014 (UTC)

One minute of average parsing time for We Are! OMG. Sorry Sewil, but this solution is not tenable… We could use an actual external database, to keep track of the chapter/episode connections and automatically generate chapter and episode infoboxes (and also complete tables, why not?), but I doubt we can use templates as pseudo-databases efficiently enough for this.

Hm... how would it work with an external database? Would it be connected via one of the .js pages? I like that idea actually, but how would normal users access it? Also, could it be done with the rest of the "Which"-type templates, to reduce loading time?

Sff9 that's why I said we should just do it manually again. It would load 100% faster. I noticed chapter pages were also a bit slower to load after Sewil made the changes. SeaTerror (talk) 22:47, January 16, 2014 (UTC)

Static values should never be manual. 22:55, January 16, 2014 (UTC)

Says only you. SeaTerror (talk) 23:13, January 16, 2014 (UTC)

@Sewil, seems I was unclear actually. What I meant was that yeah, databases are perfect for such things. Unfortunately we cannot use an actual database on the wiki, and templates do not work in this case. So it seems ST is right, and values have to be statically written on each episode page. Now, this does not prevent us to use an external database to keep the info, so that infoboxes be easy to generate/update (possibly using a bot). But in the end it would still be statically written in each page.

@Gal, static values should be manual if it avoids pages taking more than 10 seconds to load. And episode pages take more than 40 second on average to be parsed; the loading starts afterwards! That's just not possible.

Seems to me that at the moment, the best solution is to go ahead with re-adding the information manually. Though via bot, or at least through bot accounts, would be best. Perhaps we can think of a solution that lets the values still be automated, but that is going to take time to figure out. What troubles me the most is that the staff member did not give us a deadline for dealing with this problem, and the last thing we need is all of our episode pages becoming unreadable messes in 3 days (one week from the forum's start). We can/should work at a better solution, but we need one that works in the short term for now.

PS: I don't want to see any of these edits done on a non-bot account, since they are the epitome of edit whoring. 03:38, January 17, 2014 (UTC)

From what I've researched the staff seems to be making this change at the end of January.--

I don't really get the point of storing the data offsite if we just need to do everything manually. SeaTerror (talk) 08:58, January 17, 2014 (UTC)

As far as I understand, WhichCorresponding is only used in chapter pages, so what's the point anyway? A template is useful if it's used in a lot of pages, but since for each chapter there is a specific result, we may as well simply edit the chapter page instead of the template.

It's used on Episode pages too, along with openings and specials that contain content from the manga. 16:35, January 17, 2014 (UTC)

It actually works well for chapter pages; episodes are different, probably because the template is used more (content + eyecatcher, it seems).

Chapter pages still load a bit slow too so it doesn't work for them. SeaTerror (talk) 15:38, January 19, 2014 (UTC)

Indeed, it's not perfect either. Something needs to be done ASAP, and I got no time to dig into the templates and figure out what information is supposed to go where…

OK, so I simply removed WhichCorresponding from Episode Box. Now episode pages are useable. But they lack the information about chapters, so it's still urgent that the problem be solved.

Use PX-BOT to add the parameter back on all episodes. SeaTerror (talk) 20:05, January 23, 2014 (UTC)

If it were that simple I would have done it already.

Needs to be done manually since the "Chapter ___" format is better. 20:08, January 23, 2014 (UTC)

OK, I'm going to run PX-Bot on this, so I need to be sure which format to use. I already did it on some songs, e.g., Crazy Rainbow. I used semicolons to separate chapters. Is it OK? Should I use newlines? Should I use a shorter format?

Ew no, there have to be breaks. 23:49, January 24, 2014 (UTC)

Wouldn't breaks though make the list appear longer and leave some empty space on the right? That wouldn't look good either. Unless you have something better in mind. MasterDeva (talk) 13:10, January 25, 2014 (UTC)

Yeah, breaks are not great when the list is long (this can happen, see the data I gathered from the WhichCorresponding "database").

Breaks make it look better than having the lines cut off due to being too long. 16:39, January 25, 2014 (UTC)

Cut off? Do you mean "wrapped", or am I missing something?

this looks good. 02:01, January 26, 2014 (UTC)