Stephan Hoyer   @shoyer

ML for science @GoogleAI. @xarray_dev and NumPy core dev. These are my opinions, not my employer's. he/him



















 

  Tweets by Stephan Hoyer  

Stephan Hoyer    @shoyer    7 hours      

One of my regrets with @xarray_dev is that I copied end-inclusive slicing rules from pandas. At this point there's basically no way to fix it without silently breaking lots of code.
  
    1         20



Stephan Hoyer    @shoyer    11/24/2021      

The best part is that you maybe be only one short function decorator away from solving your memory problems for good! If you use Python, take a look at jax.checkpoint, torch.utils.checkpoint or tf.recompute_grad. I'm sure it's easy in Julia, too -- please reply if you know how!
  
          3



Stephan Hoyer    @shoyer    11/24/2021      

Gradient checkpointing (aka rematerialization) is an easy trick that can save massive amounts of memory for calculating gradients. If you differentiate through computation involving long iterative processes (like ODE solving), learn it and make it part of your toolkit! ๐Ÿ‘‡๐Ÿงต
  
    2         3



Stephan Hoyer    @shoyer    11/23/2021      

I'm sure there are sub-optimal things in the PINN method, too, but my guess is that if you fixed these issues, the adjoint method would be 100x faster.
  
          4



Stephan Hoyer    @shoyer    11/23/2021      

This paper "Optimal control of PDEs using physics informed neural networks" looks really nice: https://t.co/GlacmDPL7i Finally, an assessment of PINNs that includes a runtime comparison to classical adjoint methods!
  
    2         23