[Png-mng-security] Order of building gamma_16_table and png_do_chop

[Glenn Randers-Pehrson]

In pngrtran.c we have this:

#if defined(PNG_READ_GAMMA_SUPPORTED)
   if ((png_ptr->transformations & PNG_GAMMA) &&
#if defined(PNG_READ_BACKGROUND_SUPPORTED)
      !((png_ptr->transformations & PNG_BACKGROUND) &&
      ((png_ptr->num_trans != 0) ||
      (png_ptr->color_type & PNG_COLOR_MASK_ALPHA))) &&
#endif
      (png_ptr->color_type != PNG_COLOR_TYPE_PALETTE))
      png_do_gamma(&(png_ptr->row_info), png_ptr->row_buf + 1,
         png_ptr->gamma_table, png_ptr->gamma_16_table,
         png_ptr->gamma_shift);
#endif

#if defined(PNG_READ_16_TO_8_SUPPORTED)
   if (png_ptr->transformations & PNG_16_TO_8)
      png_do_chop(&(png_ptr->row_info), png_ptr->row_buf + 1);
#endif

Later on, when building gamma tables, we use png_info->depth
to decide whether to build the 16-bit gamma tables or the 8-bit
table.  png_info->depth gets set in png_do_chop().  This means
that the 16-bit gamma tables get built even if the user has
called png_set_strip_16().  That seems to me to be a waste,
and it unnecessarily exposes applications to the current
memory allocation vulnerability.  For example, it makes
mozilla/firefox vulnerable, even though it uses png_set_strip_16(().

Is there any reason not to swap these two blocks of code
in libpng-1.4.0beta, to avoid unnecessarily building the
16-bit gamma tables?

Glenn