netket.models.MLP

class netket.models.MLP

Bases: flax.linen.module.Module

A Multi-Layer Perceptron with hidden layers.

This model uses the MLP block with output dimension 1, which is squeezed.

This combines multiple dense layers and activations functions into a single object. It separates the output layer from the hidden layers, since it typically has a different form. One can specify the specific activation functions per layer. The size of the hidden dimensions can be provided as a number, or as a factor relative to the input size (similar as for RBM). The default model is a single linear layer without activations.

Forms a common building block for models such as PauliNet (continuous)

Attributes

hidden_dims: Optional[Union[int, Tuple[int, ...]]] = None

The size of the hidden layers, excluding the output layer.

hidden_dims_alpha: Optional[Union[int, Tuple[int, ...]]] = None

The size of the hidden layers provided as number of times the input size. One must choose to either specify this or the hidden_dims keyword argument

output_activation: Optional[Callable] = None

The nonlinear activation at the output layer. If None is provided, the output layer will be essentially linear.

precision: Optional[jax._src.lax.lax.Precision] = None

Numerical precision of the computation see jax.lax.Precision for details.

use_hidden_bias: bool = True

if True uses a bias in the hidden layer.

use_output_bias: bool = False

if True adds a bias to the output layer.

variables

Returns the variables in this module.

Return type

Mapping[str, Mapping[str, Any]]

Methods

bias_init(shape, dtype=<class 'jax.numpy.float64'>)

An initializer that returns a constant array full of zeros.

The key argument is ignored.

>>> import jax, jax.numpy as jnp
>>> jax.nn.initializers.zeros(jax.random.PRNGKey(42), (2, 3), jnp.float32)
DeviceArray([[0., 0., 0.],
             [0., 0., 0.]], dtype=float32)

Return type

Any

Parameters

• key (jax._src.prng.PRNGKeyArray) –

• shape (Sequence[Union[int, Any]]) –

• dtype (Any) –

has_rng(name)

Returns true if a PRNGSequence with name name exists.

Return type

bool

Parameters

name (str) –

hidden_activations(approximate=True)

Gaussian error linear unit activation function.

If approximate=False, computes the element-wise function:

\[\mathrm{gelu}(x) = \frac{x}{2} \left(1 + \mathrm{erf} \left( \frac{x}{\sqrt{2}} \right) \right)\]

If approximate=True, uses the approximate formulation of GELU:

\[\mathrm{gelu}(x) = \frac{x}{2} \left(1 + \mathrm{tanh} \left( \sqrt{\frac{2}{\pi}} \left(x + 0.044715 x^3 \right) \right) \right)\]

For more information, see Gaussian Error Linear Units (GELUs), section 2.

Parameters

• x (Any) – input array

• approximate (bool) – whether to use the approximate or exact formulation.

Return type

Any

is_initializing()

Returns True if running under self.init(…) or nn.init(…)().

This is a helper method to handle the common case of simple initialization where we wish to have setup logic occur when only called under module.init or nn.init. For more complicated multi-phase initialization scenarios it is better to test for the mutability of particular variable collections or for the presence of particular variables that potentially need to be initialized.

Return type

bool

kernel_init(shape, dtype=<class 'jax.numpy.float64'>)

Return type

Any

Parameters

• key (jax._src.prng.PRNGKeyArray) –

• shape (Sequence[Union[int, Any]]) –

• dtype (Any) –

perturb(name, value, collection='perturbations')

Add an zero-value variable (‘perturbation’) to the intermediate value.

The gradient of value would be the same as the gradient of this perturbation variable. Therefore, if you define your loss function with both params and perturbations as standalone arguments, you can get the intermediate gradients of value by running jax.grad on the perturbation argument.

Note: this is an experimental API and may be tweaked later for better performance and usability. At its current stage, it creates extra dummy variables that occupies extra memory space. Use it only to debug gradients in training.

Example:

import jax
import jax.numpy as jnp
import flax.linen as nn

class Foo(nn.Module):
    @nn.compact
    def __call__(self, x):
        x = nn.Dense(3)(x)
        x = self.perturb('dense3', x)
        return nn.Dense(2)(x)

def loss(params, perturbations, inputs, targets):
  variables = {'params': params, 'perturbations': perturbations}
  preds = model.apply(variables, inputs)
  return jnp.square(preds - targets).mean()

x = jnp.ones((2, 9))
y = jnp.ones((2, 2))
model = Foo()
variables = model.init(jax.random.PRNGKey(0), x)
intm_grads = jax.grad(loss, argnums=1)(variables['params'], variables['perturbations'], x, y)
print(intm_grads['dense3']) # ==> [[-1.456924   -0.44332537  0.02422847]
                            #      [-1.456924   -0.44332537  0.02422847]]

Return type

TypeVar(T)

Parameters

• name (str) –

• value (flax.linen.module.T) –

• collection (str) –

put_variable(col, name, value)

Sets the value of a Variable.

Parameters

• col (str) – the variable collection.

• name (str) – the name of the variable.

• value (Any) – the new value of the variable.

Returns:

tabulate(rngs, *args, depth=None, show_repeated=False, mutable=True, console_kwargs=None, **kwargs)

Creates a summary of the Module represented as a table.

This method has the same signature and internally calls Module.init, but instead of returning the variables, it returns the string summarizing the Module in a table. tabulate uses jax.eval_shape to run the forward computation without consuming any FLOPs or allocating memory.

Example:

import jax
import jax.numpy as jnp
import flax.linen as nn

class Foo(nn.Module):
    @nn.compact
    def __call__(self, x):
        h = nn.Dense(4)(x)
        return nn.Dense(2)(h)

x = jnp.ones((16, 9))

print(Foo().tabulate(jax.random.PRNGKey(0), x))

This gives the following output:

                                Foo Summary
┏━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ path    ┃ module ┃ inputs        ┃ outputs       ┃ params               ┃
┡━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
│         │ Foo    │ float32[16,9] │ float32[16,2] │                      │
├─────────┼────────┼───────────────┼───────────────┼──────────────────────┤
│ Dense_0 │ Dense  │ float32[16,9] │ float32[16,4] │ bias: float32[4]     │
│         │        │               │               │ kernel: float32[9,4] │
│         │        │               │               │                      │
│         │        │               │               │ 40 (160 B)           │
├─────────┼────────┼───────────────┼───────────────┼──────────────────────┤
│ Dense_1 │ Dense  │ float32[16,4] │ float32[16,2] │ bias: float32[2]     │
│         │        │               │               │ kernel: float32[4,2] │
│         │        │               │               │                      │
│         │        │               │               │ 10 (40 B)            │
├─────────┼────────┼───────────────┼───────────────┼──────────────────────┤
│         │        │               │         Total │ 50 (200 B)           │
└─────────┴────────┴───────────────┴───────────────┴──────────────────────┘

                      Total Parameters: 50 (200 B)

Note: rows order in the table does not represent execution order, instead it aligns with the order of keys in variables which are sorted alphabetically.

Parameters

• rngs (Union[Any, Dict[str, Any]]) – The rngs for the variable collections as passed to Module.init.

• *args – The arguments to the forward computation.

• depth (Optional[int]) – controls how many submodule deep the summary can go. By default its None which means no limit. If a submodule is not shown because of the depth limit, its parameter count and bytes will be added to the row of its first shown ancestor such that the sum of all rows always adds up to the total number of parameters of the Module.

• show_repeated (bool) – If True, repeated calls to the same module will be shown in the table, otherwise only the first call will be shown. Default is False.

• mutable (Union[bool, str, Collection[str], DenyList]) – Can be bool, str, or list. Specifies which collections should be treated as mutable: bool: all/no collections are mutable. str: The name of a single mutable collection. list: A list of names of mutable collections. By default all collections except ‘intermediates’ are mutable.

• console_kwargs (Optional[Mapping[str, Any]]) – An optional dictionary with additional keyword arguments that are passed to rich.console.Console when rendering the table. Default arguments are {‘force_terminal’: True, ‘force_jupyter’: False}.

• **kwargs – keyword arguments to pass to the forward computation.

Return type

str

Returns

A string summarizing the Module.