[santacruzgalaxy-list] Grackle proposal
Sam Leitner
sam.leitner at gmail.com
Fri Mar 21 09:30:33 PDT 2014
Nick, that is not correct, my ART implementation uses the grackle solver.
We can discuss it in the PR that I issued.
On Fri, Mar 21, 2014 at 12:23 PM, Nick Gnedin <gnedin at fnal.gov> wrote:
>
> At least with ART, the implementation is not what it is supposed to be.
> It actually uses ART sub-cycling solver, and only extracts the cooling
> rate from Grackle. I thought the plan was for everyone to use
> solve_chemistry.
>
> That's why it is important to converge on the API first.
>
>
> On 03/21/2014 11:14 AM, Britton Smith wrote:
> > Hi all,
> >
> > This discussion seems to have ground to a halt, so I'll see if I can get
> > it going again.
> >
> > It looks like the proposed API is pretty similar to what already exists,
> > with mostly cosmetic changes related to what data the user is required
> > to carry around. Since we have heard from so few regarding a new API,
> > it doesn't seem that many people really want it. Perhaps we should
> > establish what groups have completed the Grackle implementation, and
> > then we can focus on what the remaining groups need in order to catch
> > up. I heard a wise man once say, "You can't always get what you want,
> > but if you try sometimes, well you might find you get what you need."
> >
> > We've heard from Ken Nagamine and Junhwan Choi on behalf of the Gadget
> > group. I have done Enzo, and to be clear, the Grackle is slightly
> > different from what is built in to Enzo, so Enzo now explicitly supports
> > Grackle. Sam Leitner has told me that ART is now finished. Are there
> > any other groups out there that can report in?
> >
> > So, it sounds to me that one thing that is /needed/ is a clear path to
> > implementing the Grackle into a purely fortran code, like Ramses. Is
> > that correct? If so, let's talk about how best that can be achieved.
> >
> > To the other groups out there that have not yet finished the
> > implementation process, what are the things that you need to get this
> done?
> >
> > Britton
> >
> > On Thu, Mar 20, 2014 at 3:07 PM, Junhwan Choi (최준환)
> > <choi.junhwan at gmail.com <mailto:choi.junhwan at gmail.com>> wrote:
> >
> > Hi Romain,
> >
> > For me, it took about two weeks to make it works.
> > But, I spend considerable time to find a way to pass the system unit
> > to grackle properly.
> > There are a few outstanding issues on this adjustment.
> >
> > 1. Grackle use C++ and wrap the FORTRAN cooling routine. It is not
> > trivial to call this C++ wrapped the FORTRAN but there is example in
> > the grackle example directory.
> >
> > 2. In the gadget case, I pass one particle by one particle to the
> > grackle in order to calculate the cooling. The reason is that the
> > gadget cooling and star formation routine is based on particle by
> > particle. Indeed, it is least efficient way but so far I have not
> > experienced significant slow down.
> > [If I use tabulated H and He cooling rates from grackle, it takes
> > almost the same time with the original gadget cooling.]
> >
> >
> > 3. For unit adjustment, we can set as follow:
> > my_units.comoving_coordinates = 0;
> > my_units.density_units = udensity;
> > my_units.length_units = ulength;
> > my_units.time_units = utime;
> > my_units.a_units = 1.0;
> > [udensity, ulength, and utime is the conversion factor from system
> > physical units to cgs.]
> > In this case, you can pass the density, internal energy, and
> chemistry
> > value in system physical unit.
> >
> > Besides these issues, the incorporating Grackle should not be very
> > difficult.
> >
> > Best,
> > Junhwan
> >
> >
> >
> >
> >
> > On Thu, Mar 20, 2014 at 2:35 AM, Romain Teyssier
> > <romain.teyssier at gmail.com <mailto:romain.teyssier at gmail.com>>
> wrote:
> > > That's great.
> > > A very important question: how long did it take him to succeed ?
> > > Since in my opinion, many "adjustments" need to be done, I would
> > be curious to
> > > have an estimate on the time required that have this working.
> > >
> > > Cheers,
> > > Romain
> > >
> > > On 20 Mar 2014, at 00:33, Ken Nagamine <kn at physics.unlv.edu
> > <mailto:kn at physics.unlv.edu>> wrote:
> > >
> > >> Dear all,
> > >>
> > >> Let me just mention that, on the SPH side, Junhwan Choi has
> > already succeeded in
> > >> implementing the Grackle package into Gadget3-UNLV version.
> > >> We still need to do more checks, but at least the phase diagram
> > >> looks reasonable compared to the previous TREECOOL table
> > implementation of original Gadget.
> > >> Junhwan can explain the details of his implementation method if
> > it can be of help for this discussion.
> > >>
> > >> cheers,
> > >> Ken
> > >>
> > >>
> > >> On Mar 20, 2014, at 6:44 AM, Romain Teyssier wrote:
> > >>
> > >>>
> > >>> On 19 Mar 2014, at 22:35, Matthew Turk <matthewturk at gmail.com
> > <mailto:matthewturk at gmail.com>> wrote:
> > >>>
> > >>>> On Wed, Mar 19, 2014 at 5:26 PM, Romain Teyssier
> > >>>> <romain.teyssier at gmail.com <mailto:romain.teyssier at gmail.com>>
> > wrote:
> > >>>>> Hi Matt,
> > >>>>>
> > >>>>>
> > >>>>> For example:
> > >>>>>
> > >>>>> int solve_chemistry(chemistry_data &my_chemistry, ???
> > >>>>> code_units &my_units, OK set to 1
> > >>>>> gr_float a_value, gr_float dt_value, why a ?
> > >>>>> gr_int grid_rank, gr_int *grid_dimension,
> ????
> > >>>>> gr_int *grid_start, gr_int *grid_end, ????
> > >>>>> gr_float *density, gr_float
> > *internal_energy, OK
> > >>>>> gr_float *x_velocity, gr_float *y_velocity,
> > gr_float *z_velocity, ???
> > >>>>> gr_float *HI_density, gr_float
> > *HII_density, gr_float *HM_density, too much
> > >>>>> gr_float *HeI_density, gr_float
> > *HeII_density, gr_float *HeIII_density, too much
> > >>>>> gr_float *H2I_density, gr_float
> > *H2II_density, too much
> > >>>>> gr_float *DI_density, gr_float
> > *DII_density, gr_float *HDI_density, too much
> > >>>>> gr_float *e_density, gr_float
> > *metal_density); OK
> > >>>>>
> > >>>>> What are the *grid related variables ?
> > >>>>
> > >>>> Good question! These, like the code units variables, are
> > designed to
> > >>>> minimize the overhead of any simulation that wants to put its
> code
> > >>>> into it. For instance, if one had a patch/block-based code
> (like
> > >>>> Enzo, FLASH, Nyx, etc) or an Octree code where the 2x2x2 zones
> > were
> > >>>> included in (2+2*NGZ, 2+2*NGZ, 2+2*NGZ) blocks and one didn't
> > want to
> > >>>> spend time solving the ghost zones, they can be masked out.
> > These are
> > >>>> the variables:
> > >>>>
> > >>>> grid_start => array of size (grid_rank) indicating the indices
> to
> > >>>> start at in each rank of dimensionality
> > >>>> grid_end => array of size (grid_rank) indicating the indices
> > to *end*
> > >>>> at in each rank of dimensionality
> > >>>> grid_dimension => size of the block of data in each dimension
> > >>>> grid_rank => dimensionality; if you're supplying a pencil
> > beam, this would be 1.
> > >>>>
> > >>>> While these typically are better for block or patch based
> > solvers, I
> > >>>> think that the primary goal -- *minimizing* the overhead to
> > using a
> > >>>> new code -- is met with them.
> > >>>
> > >>> Well I clearly disagree.
> > >>> It is brain damaging to say the least.
> > >>>
> > >>>>
> > >>>> As I mentioned in our previous email, the velocities are
> > supplied for
> > >>>> the Sobolev approximation. You can see the full discussion on
> the
> > >>>> public Grackle mailing list here:
> > >>>>
> > >>>>
> >
> https://groups.google.com/forum/#!searchin/grackle-cooling-users/velocity/grackle-cooling-users/Dr77TM2te9g/225RNzoZADEJ
> > >>>>
> > >>>>>
> > >>>>> If chemistry_data is an external variable, this is a pain
> > because this variable need to be declared in the main code.
> > >>>>> Same thing for code_units.
> > >>>>
> > >>>> The alternative is that it be a global defined in a different
> > >>>> namespace, loading in by the dlloader at runtime. Not sure
> that's
> > >>>> substantially different.
> > >>>>
> > >>>
> > >>> Well I think this should be completely hidden to the user.
> > >>> This is a very substantial difference.
> > >>>
> > >>>> Code units, if you want to convert to CGS, will just be 1.0
> > for all of
> > >>>> them, which can be constant.
> > >>>>
> > >>>
> > >>> Yes but you still have to create a code_unit type, and set the
> > variable to 1.
> > >>> Unless you change the calling sequence.
> > >>>
> > >>>>> This is all too ENZO specifics.
> > >>>>
> > >>>> I'm not sure I believe that argument. Perhaps the names of the
> > >>>> variables share too much with Enzo naming schemes, but I think
> > it's
> > >>>> rather intuitive to think of the 3D dataset as a base starting
> > point
> > >>>> and then reducing overhead by supplying 1-dimensional arrays.
> > >>>>
> > >>>
> > >>> For a purely local process like cooling and chemistry, the
> > natural data type is 1D.
> > >>> Cells, particles and what not, could then be sent as a 1D array
> > to the chemistry solver.
> > >>> 3D structures are relevant only for patch based codes.
> > >>>
> > >>>
> > >>>> -Matt
> > >>>>
> > >>>>>
> > >>>>> Romain
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On 19 Mar 2014, at 22:08, Matthew Turk <matthewturk at gmail.com
> > <mailto:matthewturk at gmail.com>> wrote:
> > >>>>>
> > >>>>>> Hi all,
> > >>>>>>
> > >>>>>> On Wed, Mar 19, 2014 at 4:58 PM, Romain Teyssier
> > >>>>>> <romain.teyssier at gmail.com
> > <mailto:romain.teyssier at gmail.com>> wrote:
> > >>>>>>>
> > >>>>>>> I totally agree with the 3 routines that need to be used.
> > >>>>>>> Nothing else should be required on the code's side, except
> > passing the required variables.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> grackle_init_run()
> > >>>>>>>> called once per simulation
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>> The parameters that should be passed here are
> > >>>>>>> - some cosmological model parameters
> > >>>>>>> - UV background model (spectrum, reionization redshift...)
> > >>>>>>> - starting redshift to compute the initial temperature and
> > ionisation state
> > >>>>>>>
> > >>>>>>>> grackle_init_step(aexp,...)
> > >>>>>>>> called once per each time-step, uses the value of the
> scale
> > >>>>>>>> factor and other parameters as needed
> > >>>>>>>
> > >>>>>>> In principle at this stage, only aexp is required for cosmo
> > runs.
> > >>>>>>> For non cosmo runs, the cooling tables do not need to be
> > modified.
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> grackle_solve_chemistry(dt,den,tem,...)
> > >>>>>>>> called for each resolution element one or more times per
> one
> > >>>>>>>> global time-step, uses gas properties to update internal
> > energy.
> > >>>>>>>> Input gas properties may be required to have specific
> > units and
> > >>>>>>>> be of specific data type. It should internally sense if it
> > >>>>>>>> runs within an OpenMP construct and support OpenMP
> > >>>>>>>> parallelization.
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>> I would rather give the possibility of passing an array of
> > cell.
> > >>>>>>> Users could also set the array size to 1 to deal with cells
> > one by one.
> > >>>>>>> The required information could be
> > >>>>>>> nH, T or Tovermu or specific energy, metallicity in some
> > units, dt_hydro, ncell
> > >>>>>>> On output, Delta T or Delta Tovermu or Delta specific energy
> > >>>>>>> I suggest we use fixed units (cgs or mks) for all input and
> > output variables.
> > >>>>>>>
> > >>>>>>> These 3 routines are the only one that the user should care
> > about.
> > >>>>>>>
> > >>>>>>
> > >>>>>> This is awfully similar to the existing API:
> > >>>>>>
> > >>>>>>
> >
> https://bitbucket.org/brittonsmith/grackle/src/642cd133535a2dd3c57626b768c7c8c107096ac7/src/clib/grackle.h?at=default
> > >>>>>>
> > >>>>>> The only necessary functions, according to
> > >>>>>> http://grackle.readthedocs.org/en/latest/Integration.html ,
> are:
> > >>>>>>
> > >>>>>> initialize_chemistry_data
> > >>>>>> solve_chemistry or solve_chemistry for tabular data
> > >>>>>>
> > >>>>>> Looking at the example (non-equilibrium) executable:
> > >>>>>>
> > >>>>>>
> >
> https://bitbucket.org/brittonsmith/grackle/src/642cd133535a2dd3c57626b768c7c8c107096ac7/src/example/example.C?at=default
> > >>>>>>
> > >>>>>> those are the only *functional* routines. Everything else
> is to
> > >>>>>> provide additional information, *not* to actually do any
> > computation.
> > >>>>>> I think that we have actually met these standards. All of
> > the units
> > >>>>>> are provided so as the *avoid* any boilerplate -- but if you
> > want to
> > >>>>>> supply in CGS, you can set the input unit conversions to 1.0.
> > >>>>>>
> > >>>>>> To compute the equilibrium tables:
> > >>>>>>
> > >>>>>>
> >
> https://bitbucket.org/brittonsmith/grackle/src/642cd133535a2dd3c57626b768c7c8c107096ac7/src/example/table_example.C?at=default
> > >>>>>>
> > >>>>>> it's even simpler. The only *active* routine is called on
> > line 120,
> > >>>>>> "solve_chemistry". I am genuinely being earnest when I ask,
> > from the
> > >>>>>> .h file linked above, which arguments to the functions would
> > you like
> > >>>>>> to see removed? I've included the full function signatures
> > below. If
> > >>>>>> you want to get *out* various things, you can utilize the
> other
> > >>>>>> functions -- like the cooling time function and so on.
> > >>>>>>
> > >>>>>> -Matt
> > >>>>>>
> > >>>>>> int initialize_chemistry_data(chemistry_data &my_chemistry,
> > >>>>>> code_units &my_units, gr_float
> > a_value);
> > >>>>>>
> > >>>>>> int solve_chemistry(chemistry_data &my_chemistry,
> > >>>>>> code_units &my_units,
> > >>>>>> gr_float a_value, gr_float dt_value,
> > >>>>>> gr_int grid_rank, gr_int *grid_dimension,
> > >>>>>> gr_int *grid_start, gr_int *grid_end,
> > >>>>>> gr_float *density, gr_float *internal_energy,
> > >>>>>> gr_float *x_velocity, gr_float *y_velocity, gr_float
> > *z_velocity,
> > >>>>>> gr_float *HI_density, gr_float *HII_density, gr_float
> > *HM_density,
> > >>>>>> gr_float *HeI_density, gr_float *HeII_density, gr_float
> > *HeIII_density,
> > >>>>>> gr_float *H2I_density, gr_float *H2II_density,
> > >>>>>> gr_float *DI_density, gr_float *DII_density, gr_float
> > *HDI_density,
> > >>>>>> gr_float *e_density, gr_float *metal_density);
> > >>>>>>
> > >>>>>> int solve_chemistry(chemistry_data &my_chemistry,
> > >>>>>> code_units &my_units,
> > >>>>>> gr_float a_value, gr_float dt_value,
> > >>>>>> gr_int grid_rank, gr_int *grid_dimension,
> > >>>>>> gr_int *grid_start, gr_int *grid_end,
> > >>>>>> gr_float *density, gr_float
> *internal_energy,
> > >>>>>> gr_float *x_velocity, gr_float *y_velocity,
> > >>>>>> gr_float *z_velocity,
> > >>>>>> gr_float *metal_density);
> > >>>>>>
> > >>>>>>> Cheers,
> > >>>>>>> Romain
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> The API can also define its own data types with natutal
> > conversion from
> > >>>>>>>> standard types, for example the following code should be
> > valid:
> > >>>>>>>>
> > >>>>>>>> float f;
> > >>>>>>>> double d;
> > >>>>>>>> gr_float gr_f;
> > >>>>>>>>
> > >>>>>>>> gr_f = f;
> > >>>>>>>> gr_f = d;
> > >>>>>>>>
> > >>>>>>>> An analogous API should be provided for F77.
> > >>>>>>>>
> > >>>>>>>> Then in a code the API will be implemented as follows:
> > >>>>>>>>
> > >>>>>>>> Begin_code
> > >>>>>>>>
> > >>>>>>>> grackle_init_run()
> > >>>>>>>>
> > >>>>>>>> Loop_over_timesteps(aexp)
> > >>>>>>>>
> > >>>>>>>> grackle_init_step(aexp,...)
> > >>>>>>>>
> > >>>>>>>> #pragma omp parallel loop
> > >>>>>>>> Loop_over_resolution_elements(elem)
> > >>>>>>>> {
> > >>>>>>>> gr_float dt = code_time_step*time_unit;
> > >>>>>>>> gr_float den = code_density(elem)*den_unit;
> > >>>>>>>> gr_float tem = code_temperature(elem)*tem_unit;
> > >>>>>>>> grackle_solve_chemistry(dt,den,tem,...)
> > >>>>>>>> }
> > >>>>>>>>
> > >>>>>>>> End_loop
> > >>>>>>>>
> > >>>>>>>> End_code
> > >>>>>>>>
> > >>>>>>>> Let's converge on the API, and then the Grackle team will
> > be able to
> > >>>>>>>> write a couple of wrappers that will suit everyone.
> > >>>>>>>>
> > >>>>>>>> Given the wrappers, if Grackle is installed as an external
> > library and
> > >>>>>>>> provides a proper Linus-style installer (that will handle
> > all HDF5 and
> > >>>>>>>> other dependencies), then it should be trivial to
> > integrate it in any code.
> > >>>>>>>>
> > >>>>>>>> Nick
> > >>>>>>>>
> > >>>>>>>> To unsubscribe from this group and stop receiving emails
> > from it, send an email to santacruzgalaxy-list+unsubscribe at ucsc.edu
> > <mailto:santacruzgalaxy-list%2Bunsubscribe at ucsc.edu>.
> > >>>>>>>> _______________________________________________
> > >>>>>>>> santacruzgalaxy-list mailing list
> > >>>>>>>> santacruzgalaxy-list at ucsc.edu
> > <mailto:santacruzgalaxy-list at ucsc.edu>
> > >>>>>>>>
> https://lists.ucsc.edu/mailman/listinfo/santacruzgalaxy-list
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>> You received this message because you are subscribed to the
> > Google Groups "Grackle Cooling Library Users" group.
> > >>>>>>> To unsubscribe from this group and stop receiving emails
> > from it, send an email to
> > grackle-cooling-users+unsubscribe at googlegroups.com
> > <mailto:grackle-cooling-users%2Bunsubscribe at googlegroups.com>.
> > >>>>>>> For more options, visit https://groups.google.com/d/optout.
> > >>>
> > >>>
> > >>> To unsubscribe from this group and stop receiving emails from
> > it, send an email to santacruzgalaxy-list+unsubscribe at ucsc.edu
> > <mailto:santacruzgalaxy-list%2Bunsubscribe at ucsc.edu>.
> > >>> _______________________________________________
> > >>> santacruzgalaxy-list mailing list
> > >>> santacruzgalaxy-list at ucsc.edu
> > <mailto:santacruzgalaxy-list at ucsc.edu>
> > >>> https://lists.ucsc.edu/mailman/listinfo/santacruzgalaxy-list
> > >>
> > >
> > > --
> > > You received this message because you are subscribed to the
> > Google Groups "Grackle Cooling Library Users" group.
> > > To unsubscribe from this group and stop receiving emails from it,
> > send an email to grackle-cooling-users+unsubscribe at googlegroups.com
> > <mailto:grackle-cooling-users%2Bunsubscribe at googlegroups.com>.
> > > For more options, visit https://groups.google.com/d/optout.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Grackle Cooling Library Users" group.
> > To unsubscribe from this group and stop receiving emails from it,
> > send an email to grackle-cooling-users+unsubscribe at googlegroups.com
> > <mailto:grackle-cooling-users%2Bunsubscribe at googlegroups.com>.
> > For more options, visit https://groups.google.com/d/optout.
> >
> >
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to santacruzgalaxy-list+unsubscribe at ucsc.edu
> > <mailto:santacruzgalaxy-list+unsubscribe at ucsc.edu>.
> _______________________________________________
> santacruzgalaxy-list mailing list
> santacruzgalaxy-list at ucsc.edu
> https://lists.ucsc.edu/mailman/listinfo/santacruzgalaxy-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.ucsc.edu/pipermail/santacruzgalaxy-list/attachments/20140321/4b20ba39/attachment-0001.html
More information about the santacruzgalaxy-list
mailing list