[santacruzgalaxy-list] Grackle proposal
Nick Gnedin
gnedin at fnal.gov
Fri Mar 21 09:29:28 PDT 2014
PS. To make further progress, we need a jointly accessible workspace. I
never heard any response to my request, hence I will go ahead and will
create something on BitBucket.
On 03/21/2014 11:23 AM, Nick Gnedin wrote:
>
> At least with ART, the implementation is not what it is supposed to be.
> It actually uses ART sub-cycling solver, and only extracts the cooling
> rate from Grackle. I thought the plan was for everyone to use
> solve_chemistry.
>
> That's why it is important to converge on the API first.
>
>
> On 03/21/2014 11:14 AM, Britton Smith wrote:
>> Hi all,
>>
>> This discussion seems to have ground to a halt, so I'll see if I can get
>> it going again.
>>
>> It looks like the proposed API is pretty similar to what already exists,
>> with mostly cosmetic changes related to what data the user is required
>> to carry around. Since we have heard from so few regarding a new API,
>> it doesn't seem that many people really want it. Perhaps we should
>> establish what groups have completed the Grackle implementation, and
>> then we can focus on what the remaining groups need in order to catch
>> up. I heard a wise man once say, "You can't always get what you want,
>> but if you try sometimes, well you might find you get what you need."
>>
>> We've heard from Ken Nagamine and Junhwan Choi on behalf of the Gadget
>> group. I have done Enzo, and to be clear, the Grackle is slightly
>> different from what is built in to Enzo, so Enzo now explicitly supports
>> Grackle. Sam Leitner has told me that ART is now finished. Are there
>> any other groups out there that can report in?
>>
>> So, it sounds to me that one thing that is /needed/ is a clear path to
>> implementing the Grackle into a purely fortran code, like Ramses. Is
>> that correct? If so, let's talk about how best that can be achieved.
>>
>> To the other groups out there that have not yet finished the
>> implementation process, what are the things that you need to get this done?
>>
>> Britton
>>
>> On Thu, Mar 20, 2014 at 3:07 PM, Junhwan Choi (최준환)
>> <choi.junhwan at gmail.com <mailto:choi.junhwan at gmail.com>> wrote:
>>
>> Hi Romain,
>>
>> For me, it took about two weeks to make it works.
>> But, I spend considerable time to find a way to pass the system unit
>> to grackle properly.
>> There are a few outstanding issues on this adjustment.
>>
>> 1. Grackle use C++ and wrap the FORTRAN cooling routine. It is not
>> trivial to call this C++ wrapped the FORTRAN but there is example in
>> the grackle example directory.
>>
>> 2. In the gadget case, I pass one particle by one particle to the
>> grackle in order to calculate the cooling. The reason is that the
>> gadget cooling and star formation routine is based on particle by
>> particle. Indeed, it is least efficient way but so far I have not
>> experienced significant slow down.
>> [If I use tabulated H and He cooling rates from grackle, it takes
>> almost the same time with the original gadget cooling.]
>>
>>
>> 3. For unit adjustment, we can set as follow:
>> my_units.comoving_coordinates = 0;
>> my_units.density_units = udensity;
>> my_units.length_units = ulength;
>> my_units.time_units = utime;
>> my_units.a_units = 1.0;
>> [udensity, ulength, and utime is the conversion factor from system
>> physical units to cgs.]
>> In this case, you can pass the density, internal energy, and chemistry
>> value in system physical unit.
>>
>> Besides these issues, the incorporating Grackle should not be very
>> difficult.
>>
>> Best,
>> Junhwan
>>
>>
>>
>>
>>
>> On Thu, Mar 20, 2014 at 2:35 AM, Romain Teyssier
>> <romain.teyssier at gmail.com <mailto:romain.teyssier at gmail.com>> wrote:
>> > That's great.
>> > A very important question: how long did it take him to succeed ?
>> > Since in my opinion, many "adjustments" need to be done, I would
>> be curious to
>> > have an estimate on the time required that have this working.
>> >
>> > Cheers,
>> > Romain
>> >
>> > On 20 Mar 2014, at 00:33, Ken Nagamine <kn at physics.unlv.edu
>> <mailto:kn at physics.unlv.edu>> wrote:
>> >
>> >> Dear all,
>> >>
>> >> Let me just mention that, on the SPH side, Junhwan Choi has
>> already succeeded in
>> >> implementing the Grackle package into Gadget3-UNLV version.
>> >> We still need to do more checks, but at least the phase diagram
>> >> looks reasonable compared to the previous TREECOOL table
>> implementation of original Gadget.
>> >> Junhwan can explain the details of his implementation method if
>> it can be of help for this discussion.
>> >>
>> >> cheers,
>> >> Ken
>> >>
>> >>
>> >> On Mar 20, 2014, at 6:44 AM, Romain Teyssier wrote:
>> >>
>> >>>
>> >>> On 19 Mar 2014, at 22:35, Matthew Turk <matthewturk at gmail.com
>> <mailto:matthewturk at gmail.com>> wrote:
>> >>>
>> >>>> On Wed, Mar 19, 2014 at 5:26 PM, Romain Teyssier
>> >>>> <romain.teyssier at gmail.com <mailto:romain.teyssier at gmail.com>>
>> wrote:
>> >>>>> Hi Matt,
>> >>>>>
>> >>>>>
>> >>>>> For example:
>> >>>>>
>> >>>>> int solve_chemistry(chemistry_data &my_chemistry, ???
>> >>>>> code_units &my_units, OK set to 1
>> >>>>> gr_float a_value, gr_float dt_value, why a ?
>> >>>>> gr_int grid_rank, gr_int *grid_dimension, ????
>> >>>>> gr_int *grid_start, gr_int *grid_end, ????
>> >>>>> gr_float *density, gr_float
>> *internal_energy, OK
>> >>>>> gr_float *x_velocity, gr_float *y_velocity,
>> gr_float *z_velocity, ???
>> >>>>> gr_float *HI_density, gr_float
>> *HII_density, gr_float *HM_density, too much
>> >>>>> gr_float *HeI_density, gr_float
>> *HeII_density, gr_float *HeIII_density, too much
>> >>>>> gr_float *H2I_density, gr_float
>> *H2II_density, too much
>> >>>>> gr_float *DI_density, gr_float
>> *DII_density, gr_float *HDI_density, too much
>> >>>>> gr_float *e_density, gr_float
>> *metal_density); OK
>> >>>>>
>> >>>>> What are the *grid related variables ?
>> >>>>
>> >>>> Good question! These, like the code units variables, are
>> designed to
>> >>>> minimize the overhead of any simulation that wants to put its code
>> >>>> into it. For instance, if one had a patch/block-based code (like
>> >>>> Enzo, FLASH, Nyx, etc) or an Octree code where the 2x2x2 zones
>> were
>> >>>> included in (2+2*NGZ, 2+2*NGZ, 2+2*NGZ) blocks and one didn't
>> want to
>> >>>> spend time solving the ghost zones, they can be masked out.
>> These are
>> >>>> the variables:
>> >>>>
>> >>>> grid_start => array of size (grid_rank) indicating the indices to
>> >>>> start at in each rank of dimensionality
>> >>>> grid_end => array of size (grid_rank) indicating the indices
>> to *end*
>> >>>> at in each rank of dimensionality
>> >>>> grid_dimension => size of the block of data in each dimension
>> >>>> grid_rank => dimensionality; if you're supplying a pencil
>> beam, this would be 1.
>> >>>>
>> >>>> While these typically are better for block or patch based
>> solvers, I
>> >>>> think that the primary goal -- *minimizing* the overhead to
>> using a
>> >>>> new code -- is met with them.
>> >>>
>> >>> Well I clearly disagree.
>> >>> It is brain damaging to say the least.
>> >>>
>> >>>>
>> >>>> As I mentioned in our previous email, the velocities are
>> supplied for
>> >>>> the Sobolev approximation. You can see the full discussion on the
>> >>>> public Grackle mailing list here:
>> >>>>
>> >>>>
>> https://groups.google.com/forum/#!searchin/grackle-cooling-users/velocity/grackle-cooling-users/Dr77TM2te9g/225RNzoZADEJ
>> >>>>
>> >>>>>
>> >>>>> If chemistry_data is an external variable, this is a pain
>> because this variable need to be declared in the main code.
>> >>>>> Same thing for code_units.
>> >>>>
>> >>>> The alternative is that it be a global defined in a different
>> >>>> namespace, loading in by the dlloader at runtime. Not sure that's
>> >>>> substantially different.
>> >>>>
>> >>>
>> >>> Well I think this should be completely hidden to the user.
>> >>> This is a very substantial difference.
>> >>>
>> >>>> Code units, if you want to convert to CGS, will just be 1.0
>> for all of
>> >>>> them, which can be constant.
>> >>>>
>> >>>
>> >>> Yes but you still have to create a code_unit type, and set the
>> variable to 1.
>> >>> Unless you change the calling sequence.
>> >>>
>> >>>>> This is all too ENZO specifics.
>> >>>>
>> >>>> I'm not sure I believe that argument. Perhaps the names of the
>> >>>> variables share too much with Enzo naming schemes, but I think
>> it's
>> >>>> rather intuitive to think of the 3D dataset as a base starting
>> point
>> >>>> and then reducing overhead by supplying 1-dimensional arrays.
>> >>>>
>> >>>
>> >>> For a purely local process like cooling and chemistry, the
>> natural data type is 1D.
>> >>> Cells, particles and what not, could then be sent as a 1D array
>> to the chemistry solver.
>> >>> 3D structures are relevant only for patch based codes.
>> >>>
>> >>>
>> >>>> -Matt
>> >>>>
>> >>>>>
>> >>>>> Romain
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On 19 Mar 2014, at 22:08, Matthew Turk <matthewturk at gmail.com
>> <mailto:matthewturk at gmail.com>> wrote:
>> >>>>>
>> >>>>>> Hi all,
>> >>>>>>
>> >>>>>> On Wed, Mar 19, 2014 at 4:58 PM, Romain Teyssier
>> >>>>>> <romain.teyssier at gmail.com
>> <mailto:romain.teyssier at gmail.com>> wrote:
>> >>>>>>>
>> >>>>>>> I totally agree with the 3 routines that need to be used.
>> >>>>>>> Nothing else should be required on the code's side, except
>> passing the required variables.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>> grackle_init_run()
>> >>>>>>>> called once per simulation
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>> The parameters that should be passed here are
>> >>>>>>> - some cosmological model parameters
>> >>>>>>> - UV background model (spectrum, reionization redshift...)
>> >>>>>>> - starting redshift to compute the initial temperature and
>> ionisation state
>> >>>>>>>
>> >>>>>>>> grackle_init_step(aexp,...)
>> >>>>>>>> called once per each time-step, uses the value of the scale
>> >>>>>>>> factor and other parameters as needed
>> >>>>>>>
>> >>>>>>> In principle at this stage, only aexp is required for cosmo
>> runs.
>> >>>>>>> For non cosmo runs, the cooling tables do not need to be
>> modified.
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> grackle_solve_chemistry(dt,den,tem,...)
>> >>>>>>>> called for each resolution element one or more times per one
>> >>>>>>>> global time-step, uses gas properties to update internal
>> energy.
>> >>>>>>>> Input gas properties may be required to have specific
>> units and
>> >>>>>>>> be of specific data type. It should internally sense if it
>> >>>>>>>> runs within an OpenMP construct and support OpenMP
>> >>>>>>>> parallelization.
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>> I would rather give the possibility of passing an array of
>> cell.
>> >>>>>>> Users could also set the array size to 1 to deal with cells
>> one by one.
>> >>>>>>> The required information could be
>> >>>>>>> nH, T or Tovermu or specific energy, metallicity in some
>> units, dt_hydro, ncell
>> >>>>>>> On output, Delta T or Delta Tovermu or Delta specific energy
>> >>>>>>> I suggest we use fixed units (cgs or mks) for all input and
>> output variables.
>> >>>>>>>
>> >>>>>>> These 3 routines are the only one that the user should care
>> about.
>> >>>>>>>
>> >>>>>>
>> >>>>>> This is awfully similar to the existing API:
>> >>>>>>
>> >>>>>>
>> https://bitbucket.org/brittonsmith/grackle/src/642cd133535a2dd3c57626b768c7c8c107096ac7/src/clib/grackle.h?at=default
>> >>>>>>
>> >>>>>> The only necessary functions, according to
>> >>>>>> http://grackle.readthedocs.org/en/latest/Integration.html , are:
>> >>>>>>
>> >>>>>> initialize_chemistry_data
>> >>>>>> solve_chemistry or solve_chemistry for tabular data
>> >>>>>>
>> >>>>>> Looking at the example (non-equilibrium) executable:
>> >>>>>>
>> >>>>>>
>> https://bitbucket.org/brittonsmith/grackle/src/642cd133535a2dd3c57626b768c7c8c107096ac7/src/example/example.C?at=default
>> >>>>>>
>> >>>>>> those are the only *functional* routines. Everything else is to
>> >>>>>> provide additional information, *not* to actually do any
>> computation.
>> >>>>>> I think that we have actually met these standards. All of
>> the units
>> >>>>>> are provided so as the *avoid* any boilerplate -- but if you
>> want to
>> >>>>>> supply in CGS, you can set the input unit conversions to 1.0.
>> >>>>>>
>> >>>>>> To compute the equilibrium tables:
>> >>>>>>
>> >>>>>>
>> https://bitbucket.org/brittonsmith/grackle/src/642cd133535a2dd3c57626b768c7c8c107096ac7/src/example/table_example.C?at=default
>> >>>>>>
>> >>>>>> it's even simpler. The only *active* routine is called on
>> line 120,
>> >>>>>> "solve_chemistry". I am genuinely being earnest when I ask,
>> from the
>> >>>>>> .h file linked above, which arguments to the functions would
>> you like
>> >>>>>> to see removed? I've included the full function signatures
>> below. If
>> >>>>>> you want to get *out* various things, you can utilize the other
>> >>>>>> functions -- like the cooling time function and so on.
>> >>>>>>
>> >>>>>> -Matt
>> >>>>>>
>> >>>>>> int initialize_chemistry_data(chemistry_data &my_chemistry,
>> >>>>>> code_units &my_units, gr_float
>> a_value);
>> >>>>>>
>> >>>>>> int solve_chemistry(chemistry_data &my_chemistry,
>> >>>>>> code_units &my_units,
>> >>>>>> gr_float a_value, gr_float dt_value,
>> >>>>>> gr_int grid_rank, gr_int *grid_dimension,
>> >>>>>> gr_int *grid_start, gr_int *grid_end,
>> >>>>>> gr_float *density, gr_float *internal_energy,
>> >>>>>> gr_float *x_velocity, gr_float *y_velocity, gr_float
>> *z_velocity,
>> >>>>>> gr_float *HI_density, gr_float *HII_density, gr_float
>> *HM_density,
>> >>>>>> gr_float *HeI_density, gr_float *HeII_density, gr_float
>> *HeIII_density,
>> >>>>>> gr_float *H2I_density, gr_float *H2II_density,
>> >>>>>> gr_float *DI_density, gr_float *DII_density, gr_float
>> *HDI_density,
>> >>>>>> gr_float *e_density, gr_float *metal_density);
>> >>>>>>
>> >>>>>> int solve_chemistry(chemistry_data &my_chemistry,
>> >>>>>> code_units &my_units,
>> >>>>>> gr_float a_value, gr_float dt_value,
>> >>>>>> gr_int grid_rank, gr_int *grid_dimension,
>> >>>>>> gr_int *grid_start, gr_int *grid_end,
>> >>>>>> gr_float *density, gr_float *internal_energy,
>> >>>>>> gr_float *x_velocity, gr_float *y_velocity,
>> >>>>>> gr_float *z_velocity,
>> >>>>>> gr_float *metal_density);
>> >>>>>>
>> >>>>>>> Cheers,
>> >>>>>>> Romain
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>> The API can also define its own data types with natutal
>> conversion from
>> >>>>>>>> standard types, for example the following code should be
>> valid:
>> >>>>>>>>
>> >>>>>>>> float f;
>> >>>>>>>> double d;
>> >>>>>>>> gr_float gr_f;
>> >>>>>>>>
>> >>>>>>>> gr_f = f;
>> >>>>>>>> gr_f = d;
>> >>>>>>>>
>> >>>>>>>> An analogous API should be provided for F77.
>> >>>>>>>>
>> >>>>>>>> Then in a code the API will be implemented as follows:
>> >>>>>>>>
>> >>>>>>>> Begin_code
>> >>>>>>>>
>> >>>>>>>> grackle_init_run()
>> >>>>>>>>
>> >>>>>>>> Loop_over_timesteps(aexp)
>> >>>>>>>>
>> >>>>>>>> grackle_init_step(aexp,...)
>> >>>>>>>>
>> >>>>>>>> #pragma omp parallel loop
>> >>>>>>>> Loop_over_resolution_elements(elem)
>> >>>>>>>> {
>> >>>>>>>> gr_float dt = code_time_step*time_unit;
>> >>>>>>>> gr_float den = code_density(elem)*den_unit;
>> >>>>>>>> gr_float tem = code_temperature(elem)*tem_unit;
>> >>>>>>>> grackle_solve_chemistry(dt,den,tem,...)
>> >>>>>>>> }
>> >>>>>>>>
>> >>>>>>>> End_loop
>> >>>>>>>>
>> >>>>>>>> End_code
>> >>>>>>>>
>> >>>>>>>> Let's converge on the API, and then the Grackle team will
>> be able to
>> >>>>>>>> write a couple of wrappers that will suit everyone.
>> >>>>>>>>
>> >>>>>>>> Given the wrappers, if Grackle is installed as an external
>> library and
>> >>>>>>>> provides a proper Linus-style installer (that will handle
>> all HDF5 and
>> >>>>>>>> other dependencies), then it should be trivial to
>> integrate it in any code.
>> >>>>>>>>
>> >>>>>>>> Nick
>> >>>>>>>>
>> >>>>>>>> To unsubscribe from this group and stop receiving emails
>> from it, send an email to santacruzgalaxy-list+unsubscribe at ucsc.edu
>> <mailto:santacruzgalaxy-list%2Bunsubscribe at ucsc.edu>.
>> >>>>>>>> _______________________________________________
>> >>>>>>>> santacruzgalaxy-list mailing list
>> >>>>>>>> santacruzgalaxy-list at ucsc.edu
>> <mailto:santacruzgalaxy-list at ucsc.edu>
>> >>>>>>>> https://lists.ucsc.edu/mailman/listinfo/santacruzgalaxy-list
>> >>>>>>>
>> >>>>>>> --
>> >>>>>>> You received this message because you are subscribed to the
>> Google Groups "Grackle Cooling Library Users" group.
>> >>>>>>> To unsubscribe from this group and stop receiving emails
>> from it, send an email to
>> grackle-cooling-users+unsubscribe at googlegroups.com
>> <mailto:grackle-cooling-users%2Bunsubscribe at googlegroups.com>.
>> >>>>>>> For more options, visit https://groups.google.com/d/optout.
>> >>>
>> >>>
>> >>> To unsubscribe from this group and stop receiving emails from
>> it, send an email to santacruzgalaxy-list+unsubscribe at ucsc.edu
>> <mailto:santacruzgalaxy-list%2Bunsubscribe at ucsc.edu>.
>> >>> _______________________________________________
>> >>> santacruzgalaxy-list mailing list
>> >>> santacruzgalaxy-list at ucsc.edu
>> <mailto:santacruzgalaxy-list at ucsc.edu>
>> >>> https://lists.ucsc.edu/mailman/listinfo/santacruzgalaxy-list
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the
>> Google Groups "Grackle Cooling Library Users" group.
>> > To unsubscribe from this group and stop receiving emails from it,
>> send an email to grackle-cooling-users+unsubscribe at googlegroups.com
>> <mailto:grackle-cooling-users%2Bunsubscribe at googlegroups.com>.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Grackle Cooling Library Users" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to grackle-cooling-users+unsubscribe at googlegroups.com
>> <mailto:grackle-cooling-users%2Bunsubscribe at googlegroups.com>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to santacruzgalaxy-list+unsubscribe at ucsc.edu
>> <mailto:santacruzgalaxy-list+unsubscribe at ucsc.edu>.
> _______________________________________________
> santacruzgalaxy-list mailing list
> santacruzgalaxy-list at ucsc.edu
> https://lists.ucsc.edu/mailman/listinfo/santacruzgalaxy-list
>
More information about the santacruzgalaxy-list
mailing list